Configuration Properties for MapR Event Store For Apache Kafka Python Client

In the instance constructor of a MapR Event Store For Apache Kafka Python application, you can use a dictionary to set the following configuration properties. MapR Event Store For Apache Kafka Python client supports a superset of the configuration properties supported by the MapR Event Store For Apache Kafka C client.

Global Configuration Properties

Property Name Behavior
client.id Same as librdkafka
default.topic.config A dictionary of topic-level configuration properties that are applied to all used topics for the instance.
message.max.bytes Supports a value less than or equal to 10MB (10000000). If this property is set to a value that is higher than 10MB, the client issues a warning and sets the configuration to 10MB. Produce calls fail when the message size is greater than 10MB.
receive.message.max.bytes Same as librdkafka
topic.blacklist Same as librdkafka
error_cb A callback for generic/global error events. This callback is served by poll().
opaque Same as librdkafka.

Consumer Configuration Properties

Property Name Behavior
group.id Same as librdkafka.
enable.auto.commit Same as librdkafka.
auto.commit.interval.ms Same as librdkafka.
rebalance_cb Same as librdkafka.
offset_commit_cb Same as librdkafka.
delivery.report.only.error Same as librdkafka.
dr_msg_cb Same as librdkafka.
on_commit A callback used to indicate success or failure of commit requests.

Topic Configuration Properties

Property Name Behavior
partitioner_cb Same as librdkafka.
auto.offset.reset Supports the following values: earliest, latest, none, smallest, and largest. librdkafka also supports biggest, end and error.

Producer Configuration Properties

Property Name Behavior
on_delivery(kafka.KafkaError, kafka.Message) A Python function reference that is called once for each produced message to indicate the final delivery result (success or failure). This property may also be set per-message by passing callback=callable (or on_delivery=callable) to the confluent_kafka.Producer.produce() function.

MapR-Specific Configuration Properties

Property Name Behavior
streams.consumer.default.stream Specifies the path and name of the stream that the consumer subscribes to if, when subscribing to a topic, the consumer does not specify a stream. For example, the consumer can specify the name of a stream together with the name of a topic to write to, like this: /<stream>:<topic>.
Note: rd_kafka_list groups API uses this consumer configuration to obtain the consumer groups.
streams.parallel.flushers.per.partition Enables the producer may have multiple parallel send requests to the server for each topic partition. If this setting is set to true, it is possible for messages to be sent out of order.
streams.producer.default.stream Specifies the stream that the producer will use by default if the producer does not provide the name of a stream when specifying a topic to write to. For example, the producer can specify the name of a stream together with the name of a topic to write to, like this: /<stream>:<topic>. However, if the stream is not specified, the value of this configuration parameter is assumed to be the stream in which the topic is located. If the producer specifies the name of a topic without also providing the path and name of the stream, and there is no value for this configuration parameter, MapR Event Store For Apache Kafka assumes that the topic specified is in Apache Kafka and does nothing.

Additional Information

Here is a consumer configuration example:
conf = {'group.id': 'mygroup',

'session.timeout.ms': 6000,

'on_commit': my_commit_callback,

'default.topic.config': {'auto.offset.reset': 'smallest'}}

consumer = mapr_streams_python.Consumer(**conf)

For more information about the librdkafka API, see rdkafka.h.