Read MapR Event Store For Apache Kafka Using a Kafka Source

The following parameters are required when configuring Kafka source to read data from MapR Streams topics:
Property Name Description
channels If the source writes events to a channel, this is the name of the channel that the source writes events to.
type This property must be set to
org.apache.flume.source.kafka.KafkaSource
kafka.consumer.group.id A unique identifier of the consumer group. This defaults to flume. Setting the same id in multiple sources or agents indicates that they are part of the same consumer group; when these sources or agents are running at the same time, they will each read a unique set of partitions for the topics in the group.
kafka.topics A comma separated list of MapR Event Store For Apache Kafka topics where each topic is specified with the volume path and stream name. For example:
/volume_path/stream_name:topic_name1,
/volume_path/stream_name:topic_name2
Note: It is critical that the path to the topic starts with a slash(/), as the slash is what distinguishes the topic as a MapR Event Store For Apache Kafka topic.
batchDurationMillis Maximum number of milliseconds to wait before closing a batch. The default value is 1000.

For additional properties that you may want to configure, see the Flume documentation. However, note that kafka.bootstrap.servers is not required for reading MapR Event Store For Apache Kafka.

Tip: To increase throughput, set the batchSize to a higher value. The batchSize is the maximum number of messages written to Channel in one batch. By default, it is set to 1000.