Kafka

The Kafka Connection allows the Intelligence Hub to connect to Apache Kafka brokers for reading (input) and writing (output) of messages. Below are details on the connection, input, and output settings and supported features.

Connection Settings

Setting	Description
Broker URIs	One or more Kafka broker URIs to connect to (e.g., `broker1:9092,broker2:9092`). The connection uses the first broker in the list that successfully connects.
Security Type	Security type used for the Kafka client connection. Plaintext: no security. SSL: enables TLS or mTLS.
Truststore CA Certificate	The broker’s CA certificate imported into the certificate store. Required for standard TLS and mTLS. Only available when Security Type is SSL.
Keystore Client Keypair	The client certificate keypair imported into the certificate store. Required only for mTLS. Only available when Security Type is SSL.

TLS vs. mTLS: In standard TLS, only the Truststore CA Certificate is required — the broker authenticates itself to the client. In mutual TLS (mTLS), both the Truststore CA Certificate and the Keystore Client Keypair are required — the client also authenticates itself to the broker.

Input Settings

Kafka inputs subscribe to topics and consume messages. The following settings are available:

Setting	Description
Topic	The topic to subscribe to.
Group	The consumer group ID to join.
Initial Offset	Where to start consuming if no offset exists (e.g., earliest, latest).
Key Format	The expected format for the message key (Any, JSON, String, Binary). See options below.
Value Format	The expected format for the message value (Any, JSON, String, Binary). See options below.
Max Messages	Maximum number of records per read.
Include Metadata	Include metadata such as topic, partition, offset, timestamp, headers, key.

Key/Value Format Options

Type	Description
Any	The payload can be JSON, String, or Binary
JSON	The payload is deserialized as JSON. Non-JSON payloads are ignored.
String (UTF-8)	The payload is treated as a string.
Binary	The payload is treated as a byte array.

Output Settings

Kafka outputs publish messages to topics. The following settings are available:

Setting	Description
Topic	The topic to publish messages to.
Key	The key to use for the message.
Value	The value to publish (payload).