Configure Storm Topology to Work with MapR Streams

For Storm to work with MapR Streams, you must configure the Storm serializer and deserializer properties for each topology.

  1. Verify that the Streams Client (mapr-kafka) is installed on one node in the Storm cluster. See the Installing MapR or MEP Components.
  2. On the node where the Streams Client is installed, create a link between the Storm library and the Streams Client JAR file.
    ln -s /opt/mapr/lib/kafka-clients-0.9.0.0* ${STORM_HOME}/lib/
  3. On any Nimbus node or Supervisor nodes where the Streams client is not installed, copy the Streams Client JAR file into the Storm library.
  4. Create a src/main/resources/kafka.properties file.
  5. Add the following serializer and deserializer properties to the kafka.properties file:
    #Producer
    key.serializer=org.apache.kafka.common.serialization.StringSerializer
    value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
    #Consumer
    key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
    value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
    
  6. When defining the topology in the Storm program, refer to this file for setting the properties of the topology.