Kafka
The Kafka Connection allows the Intelligence Hub to connect to Apache Kafka brokers for reading (input) and writing (output) of messages. Below are details on the connection, input, and output settings and supported features.
Connection Settings
| Setting | Description |
|---|---|
| Broker URIs | One or more Kafka broker URIs to connect to (e.g., broker1:9092,broker2:9092). The connection uses the first broker in the list that successfully connects. |
| Authentication Type | Determines how the connection secures and authenticates with brokers. Default is None. None: no encryption or authentication. TLS: encrypts the connection with TLS, and optionally authenticates the client with mTLS. SASL: authenticates the client with a username and password, with or without TLS. |
TLS Authentication
| Setting | Description |
|---|---|
| Truststore CA Certificate | The broker’s CA certificate imported into the certificate store. Required for standard TLS and mTLS. |
| Keystore Client Keypair | The client certificate keypair imported into the certificate store. Required only for mTLS. |
TLS vs. mTLS: In standard TLS, only the Truststore CA Certificate is required. The broker authenticates itself to the client. In mutual TLS (mTLS), both the Truststore CA Certificate and the Keystore Client Keypair are required, and the client also authenticates itself to the broker.
SASL Authentication
SASL authenticates the client to the broker with a username and password. Use SASL when the Kafka cluster is configured for SCRAM or PLAIN authentication, as is common with managed Kafka services.
| Setting | Description |
|---|---|
| Security Protocol | Determines the security policy used during authentication. Default is SASL_SSL. SASL_SSL: SASL authentication over a TLS encrypted connection. SASL_PLAINTEXT: SASL authentication over an unencrypted connection. |
| Ignore Self-Signed Certificate | When enabled, the broker’s certificate and host name are not validated. Only available when Security Protocol is SASL_SSL. Default is disabled. |
| Truststore CA Certificate | The broker’s CA certificate imported into the certificate store. When blank, the broker’s certificate is validated against the default Java trust store, which works for brokers using certificates from a public CA. Only available when Security Protocol is SASL_SSL and Ignore Self-Signed Certificate is disabled. |
| SASL Mechanism | The SASL mechanism the broker is configured for. PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512. Default is SCRAM-SHA-512. |
| Username | The SASL username. Supports System and Environment references. |
| Password | The SASL password. Supports Secrets. |
With SASL_PLAINTEXT, the connection is not encrypted. When combined with the PLAIN mechanism, the password is sent in clear text. Use SASL_SSL unless the network between the Intelligence Hub and the brokers is trusted.
Input Settings
Kafka inputs subscribe to topics and consume messages. The following settings are available:
| Setting | Description |
|---|---|
| Topic | The topic to subscribe to. |
| Group | The consumer group ID to join. |
| Initial Offset | Where to start consuming if no offset exists (e.g., earliest, latest). |
| Key Format | The expected format for the message key (Any, JSON, String, Binary). See options below. |
| Value Format | The expected format for the message value (Any, JSON, String, Binary). See options below. |
| Max Messages | Maximum number of records per read. |
| Include Metadata | Include metadata such as topic, partition, offset, timestamp, headers, key. |
Key/Value Format Options
| Type | Description |
|---|---|
| Any | The payload can be JSON, String, or Binary |
| JSON | The payload is deserialized as JSON. Non-JSON payloads are ignored. |
| String (UTF-8) | The payload is treated as a string. |
| Binary | The payload is treated as a byte array. |
Output Settings
Kafka outputs publish messages to topics. The following settings are available:
| Setting | Description |
|---|---|
| Topic | The topic to publish messages to. |
| Key | The key to use for the message. |
| Value | The value to publish (payload). |