Configuring operators to communicate with Kafka

The Kafka operators must be configured in order to connect to the Kafka servers.

Configuration can be specified in the following ways:
  • Application configuration - Recommended; you can specify the name of the app config in the appConfigName parameter
  • Property file - you can specify the filename in the propertiesFile parameter
  • Direct operator parameters - you can specify the clientId and groupId parameters. Note: groupId is only needed for the KafkaConsumer operator
If multiple parameters are specified, the precedence is:
  1. Direct operator parameters
  2. Property file
  3. Application configuration (NOTE: An application configuration defined at the instance scope takes precendence over a configuration with the same name at the domain level.)

The only required property is the bootstrap.servers property. This property contains the Kafka server list (for example kafka-0.mydomain:9092,kafka-1.mydomain:9092,kafka-2.mydomain:9092).

All other properties are optional. For a full list of the available properties, see the Kafka producer config and consumer config references.

The following example is a property file in the etc directory of the application's toolkit. Some operator parameters, like groupId, and clientId map directly to properties and may override what's specified in the file if defined in the operator. 


bootstrap.servers=kafka-0.mydomain:9092,kafka-1.mydomain:9092,kafka-2.mydomain:9092
# property files can also contain comments and empty lines

# a consumer group identifier can also specified via 'groupId' operator parameter
group.id=myConsumerGroup