Read MapR-ES Using a Kafka Source

The following parameters are required when configuring Kafka source to read data from MapR Streams topics:
Property Name Description
channels If the source writes events to a channel, this is the name of the channel that the source writes events to.
type This property must be set to
org.apache.flume.source.kafka.KafkaSource A unique identifier of the consumer group. This defaults to flume. Setting the same id in multiple sources or agents indicates that they are part of the same consumer group; when these sources or agents are running at the same time, they will each read a unique set of partitions for the topics in the group.
kafka.topics A comma separated list of MapR-ES topics where each topic is specified with the volume path and stream name. For example:
Note: It is critical that the path to the topic starts with a slash(/), as the slash is what distinguishes the topic as a MapR-ES topic.
batchDurationMillis Maximum number of milliseconds to wait before closing a batch. The default value is 1000.

For additional properties that you may want to configure, see the Flume documentation. However, note that kafka.bootstrap.servers is not required for reading MapR-ES.

Tip: To increase throughput, set the batchSize to a higher value. The batchSize is the maximum number of messages written to Channel in one batch. By default, it is set to 1000.