Migrating Kafka Python Applications to MapR-ES

With some modification, you can use existing confluent-kafka python applications to consume and produce topics in MapR-ES. The MapR-ES Python Client is a binding for Apache librdkafka that works with MapR-ES.

  1. Install the MapR-ES Python Client.
    Note: This required that you also install and configure the MapR-ES C Client. See Configuring the MapR-ES C Client.
  2. Do one of the following depending on whether you are using the MEP 5.0 (or higher) MapR-ES Python Client or the MEP 3.0 (or higher) MapR-ES Python Client.
    • If you are using MapR-ES Python MEP 5.0 (or higher), skip this step. The references to confluent_kafka should be retained.
    • If you are using MapR-ES Python MEP 3.0 (or higher), update import statements to refer to the MapR Stream Python API. References to confluent_kafka should be updated to mapr_streams_python.
      Note: For example, update from confluent_kafka import Consumer to from mapr_streams_python import Consumer.
  3. When you refer to a topic in the application code, include the path and name of the stream in which the topic is located:
    /<path and name of stream>:<name of topic>

    For example, you might have a stream in a MapR cluster that is named stream_A, and the stream might be in a volume named IoT and in a directory named automobile_sensors. You want to redirect a producer application to a topic in that stream. The syntax of the path to the topic might look like this: /mapr/IoT/automobile_sensors/stream_A:<name of topic>.

    Note: Optionally, use the streams.consumer.default.stream and streams.producer.default.stream configuration parameters. When you configure these parameters, applications can specify just the topic name to write or read from the default stream.
  4. Review the APIs that are supported and make changes to your application, as needed. See API for MapR-ES Python Client.
  5. See Configuration Properties for MapR-ES Python Client for the list of supported configuration parameters and make changes to your application, as needed.
    Note: SSL-related configuration parameters are ignored. When you set these parameters, the MapR-ES Client issues a warning indicating that the parameters are not supported.