Write to MapR Streams using a Kafka Sink

The following parameters are required when configuring Kafka sink to write to a MapR Stream:
Property Name Description
type This must be set to
kafka.topic A topic specified with the volume path and stream name. For example: /streaming_data/error_stream:error_log_topic.
Note: It is critical that the path to the topic starts with a slash(/), as the slash is what distinguishes the topic as a MapR Streams topic. The default is "default-flume-topic".
Note: If the header for a given message contains a "topic" field, the message will be published to that topic instead of the topic that you configure here.
For additional properties that you may want to configure for the Kafka sink, see the Flume documentation.
Note: kafka.flumeBatchSize sets the number of messages to process in a batch and the default is 100. To ensure that the events are written in the original order, set kafka.flumeBatchSize=1 and specify a topic with a single partition. If maintaining the order is not required, you can increase the value. However, note that larger batches improve throughput while also adding latency.