Kafka

The Kafka Connection allows the Intelligence Hub to connect to Apache Kafka brokers for reading (input) and writing (output) of messages. Below are details on the connection, input, and output settings and supported features.

Connection Settings

SettingDescription
Broker URIsOne or more Kafka broker URIs to connect to (e.g., broker1:9092,broker2:9092). The connection uses the first broker in the list that successfully connects.
Authentication TypeDetermines how the connection secures and authenticates with brokers. Default is None.
None: no encryption or authentication.
TLS: encrypts the connection with TLS, and optionally authenticates the client with mTLS.
SASL: authenticates the client with a username and password, with or without TLS.

TLS Authentication

SettingDescription
Truststore CA CertificateThe broker’s CA certificate imported into the certificate store. Required for standard TLS and mTLS.
Keystore Client KeypairThe client certificate keypair imported into the certificate store. Required only for mTLS.

SASL Authentication

SASL authenticates the client to the broker with a username and password. Use SASL when the Kafka cluster is configured for SCRAM or PLAIN authentication, as is common with managed Kafka services.

SettingDescription
Security ProtocolDetermines the security policy used during authentication. Default is SASL_SSL.
SASL_SSL: SASL authentication over a TLS encrypted connection.
SASL_PLAINTEXT: SASL authentication over an unencrypted connection.
Ignore Self-Signed CertificateWhen enabled, the broker’s certificate and host name are not validated. Only available when Security Protocol is SASL_SSL. Default is disabled.
Truststore CA CertificateThe broker’s CA certificate imported into the certificate store. When blank, the broker’s certificate is validated against the default Java trust store, which works for brokers using certificates from a public CA. Only available when Security Protocol is SASL_SSL and Ignore Self-Signed Certificate is disabled.
SASL MechanismThe SASL mechanism the broker is configured for. PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512. Default is SCRAM-SHA-512.
UsernameThe SASL username. Supports System and Environment references.
PasswordThe SASL password. Supports Secrets.

Input Settings

Kafka inputs subscribe to topics and consume messages. The following settings are available:

SettingDescription
TopicThe topic to subscribe to.
GroupThe consumer group ID to join.
Initial OffsetWhere to start consuming if no offset exists (e.g., earliest, latest).
Key FormatThe expected format for the message key (Any, JSON, String, Binary). See options below.
Value FormatThe expected format for the message value (Any, JSON, String, Binary). See options below.
Max MessagesMaximum number of records per read.
Include MetadataInclude metadata such as topic, partition, offset, timestamp, headers, key.

Key/Value Format Options

TypeDescription
AnyThe payload can be JSON, String, or Binary
JSONThe payload is deserialized as JSON. Non-JSON payloads are ignored.
String (UTF-8)The payload is treated as a string.
BinaryThe payload is treated as a byte array.

Output Settings

Kafka outputs publish messages to topics. The following settings are available:

SettingDescription
TopicThe topic to publish messages to.
KeyThe key to use for the message.
ValueThe value to publish (payload).