Using Kafka Streams for S3 Bucket Event Notifications

You can use Kafka streams to receive event notifications for buckets, for example, when an object is created in a bucket. Install MapR Kafka, create a stream and topic, and then enable notifications in /opt/mapr/objectstore-client/objectstore-client-<version>/conf/config.json. You can then configure S3 event notifications through the S3 REST API, and use the AWS CLI to manage the notifications.

To enable bucket notification functionality, complete the following steps:
  1. Install the mapr-kafka* package:
    zypper|yum|apt-get install mapr-kafka
  2. Create a stream and topic:
    maprcli stream create -path /path/to/stream
    maprcli stream topic create -path /path/to/stream -topic nameOfTopic
  3. Set up /opt/mapr/objectstore-client/objectstore-client-<version>/conf/config.json in block notifications:
    "notify": {
      "kafka": {
        "notifyBlockName": {
          "enable": true,
          "brokers": [""],
          "topic": "/path/to/stream:nameOfTopic"
          }
       }
    }
  4. Restart the MapR Object Store.
Install and configure the AWS CLI to manage notifications from the command line. See Install and Configure the AWS CLI for installation and configuration details.

The following examples demonstrate some useful commands:

  • To read a topic:
    ./kafka/kafka-<version>/bin/kafka-console-consumer.sh --topic /path/to/stream:nameOfTopic --bootstrap-server some:9092 --from-beginning
  • To add notifications to a bucket:
    aws s3api --endpoint-url http://node1.cluster.com:9000 put-bucket-notification-configuration  --notification-configuration file://fileWithNotificationRules.json --bucket bucketName
  • To get notifications for a bucket:
    aws s3api --endpoint-url https://node1.cluster.com:9000 get-bucket-notification-configuration  --bucket bucketName

Configure S3 Event Notifications

You can configure event notifications in the S3 REST API if you want to receive notifications about specific events that happen in a bucket. To enable notifications, add a notification configuration that identifies the events you want published to a topic. For additional details and instruction, see Configuring Amazon S3 Event Notifications.

The following examples demonstrate how to configure event notifications:
Create an object event for all names
{
  "QueueConfigurations": [
  {
   "Id": "1",
   "QueueArn":  "arn:minio:sqs::notifyBlockName:kafka",
   "Events": [
       "s3:ObjectCreated:*"
      ],
   "Filter": {
       "Key": {
       "FilterRules": [
      {
        "Name": "prefix",
        "Value": "*"
       }
     ]
    }
   }
  }
 ]
}   
Remove object events that have names with the mybuck prefix
{
  "QueueConfigurations": [
    {
     "Id": "1", 
     "QueueArn": "arn:minio:sqs::notifyBlockName:kafka", 
     "Events": [
          "s3:ObjectRemoved:*"
         ], 
       "Filter": {
            "Key": {
            "FilterRules": [
              {
                "Name": "prefix", 
                "Value": "mybuck"
              }
            ]
          }
        }
      }
    ]
 }