Supportability of Apache Kafka Java Configuration

Describes the MapR Streams supportability of Apache Kafka Java configuration parameters for producers and consumers.

Support for Apache Kafka's Configuration Parameters for Producers

Name Description Supported for producers in MapR Streams?
bootstrap.servers A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form host1:port1,host2:port2,.... Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down). No. Cluster details are discovered from the file mapr-clusters.conf.
key.serializer Serializer class for key that implements the Serializer interface. Yes
value.serializer Serializer class for value that implements the Serializer interface. Yes
acks The number of acknowledgments the producer requires the leader to have received before considering a request complete. This controls the durability of records that are sent. The following settings are common: Ignored, all writes in MapR Streams are synchronous, and number of replicas is determined at the volume level, with a default of 3.
acks=0

If set to zero then the producer will not wait for any acknowledgment from the server at all. The record will be immediately added to the socket buffer and considered sent. No guarantee can be made that the server has received the record in this case, and the retries configuration will not take effect (as the client won't generally know of any failures). The offset given back for each record will always be set to -1.

acks=1

This will mean the leader will write the record to its local log but will respond without awaiting full acknowledgement from all followers. In this case should the leader fail immediately after acknowledging the record but before the followers have replicated it then the record will be lost.

acks=all

This means the leader will wait for the full set of in-sync replicas to acknowledge the record. This guarantees that the record will not be lost as long as at least one in-sync replica remains alive. This is the strongest available guarantee.

buffer.memory The total bytes of memory the producer can use to buffer records waiting to be sent to the server. If records are sent faster than they can be delivered to the server the producer will either block or throw an exception based on the preference specified by block.on.buffer.full. Yes
This setting should correspond roughly to the total memory the producer will use, but is not a hard bound since not all memory the producer uses is used for buffering. Some additional memory will be used for compression (if compression is enabled) as well as for maintaining in-flight requests.
compression.type The compression type for all data generated by the producer. The default is none (i.e. no compression). Valid values are none, gzip, snappy, or lz4. Compression is of full batches of data, so the efficacy of batching will also impact the compression ratio (more batching means better compression). Ignored. Compression is configured per stream.
retries Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries will potentially change the ordering of records because if two records are sent to a single partition, and the first fails and is retried but the second succeeds, then the second record may appear first. Ignored. MapR Streams always does automatic retries on transient errors.
ssl.key.password The password of the private key in the key store file. This is optional for client. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.keystore.location The location of the key store file. This is optional for client and can be used for two-way authentication for client. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.keystore.password The store password for the key store file.This is optional for client and only needed if ssl.keystore.location is configured. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.truststore.location The location of the trust store file. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.truststore.password The password for the trust store file. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
batch.size The producer will attempt to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. Ignored. MapR always batches records for optimal performance.
No attempt will be made to batch records larger than this size.
Requests sent to brokers will contain multiple batches, one for each partition with data available to be sent.
A small batch size will make batching less common and may reduce throughput (a batch size of zero will disable batching entirely). A very large batch size may use memory a bit more wastefully as we will always allocate a buffer of the specified batch size in anticipation of additional records.
client.id An id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging. Yes
connections.max.idle.ms Close idle connections after the number of milliseconds specified by this config. Ignored.
linger.ms The producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they can be sent out. However in some circumstances the client may want to reduce the number of requests even under moderate load. This setting accomplishes this by adding a small amount of artificial delay—that is, rather than immediately sending out a record the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together. This can be thought of as analogous to Nagle's algorithm in TCP. This setting gives the upper bound on the delay for batching: once we get batch.size worth of records for a partition it will be sent immediately regardless of this setting, however if we have fewer than this many bytes accumulated for this partition we will 'linger' for the specified time waiting for more records to show up. This setting defaults to 0 (i.e. no delay). Setting linger.ms=5, for example, would have the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absense of load. Ignored.
max.block.ms The configuration controls how long {@link KafkaProducer#send()} and {@link KafkaProducer#partitionsFor} will block.These methods can be blocked for multiple reasons. For e.g: buffer full, metadata unavailable.This configuration imposes maximum limit on the total time spent in fetching metadata, serialization of key and value, partitioning and allocation of buffer memory when doing a send(). In case of partitionsFor(), this configuration imposes a maximum time threshold on waiting for metadata Ignored. MapR Streams has a similar parameter: streams.rpc.timeout.ms
max.request.size The maximum size of a request. This is also effectively a cap on the maximum record size. Note that the server has its own cap on record size which may be different from this. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. Ignored.
partitioner.class Partitioner class that implements the Partitioner interface. Use the StreamsPartitioner interface.
receive.buffer.bytes The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. Ignored.
request.timeout.ms The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted. Ignored. MapR Streams has a similar parameter: streams.rpc.timeout.ms
sasl.kerberos.service.name The Kerberos principal name that Kafka runs as. This can be defined either in Kafka's JAAS config or in Kafka's config. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
security.protocol Protocol used to communicate with brokers. Currently only PLAINTEXT and SSL are supported. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
send.buffer.bytes The size of the TCP send buffer (SO_SNDBUF) to use when sending data. Ignored.
ssl.enabled.protocols The list of protocols enabled for SSL connections. TLSv1.2, TLSv1.1 and TLSv1 are enabled by default. Ignored.
ssl.keystore.type The file format of the key store file. This is optional for client. Default value is JKS Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.protocol The SSL protocol used to generate the SSLContext. Default setting is TLS, which is fine for most cases. Allowed values in recent JVMs are TLS, TLSv1.1 and TLSv1.2. SSL, SSLv2 and SSLv3 may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.provider The name of the security provider used for SSL connections. Default value is the default security provider of the JVM. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.truststore.type The file format of the trust store file. Default value is JKS. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
timeout.ms The configuration controls the maximum amount of time the server will wait for acknowledgments from followers to meet the acknowledgment requirements the producer has specified with the acks configuration. If the requested number of acknowledgments are not met when the timeout elapses an error will be returned. This timeout is measured on the server side and does not include the network latency of the request. Ignored. MapR Streams has a similar parameter: streams.rpc.timeout.ms
block.on.buffer.full When our memory buffer is exhausted we must either stop accepting new records (block) or throw errors. By default this setting is true and we block, however in some scenarios blocking is not desirable and it is better to immediately give an error. Setting this to false will accomplish that: the producer will throw a BufferExhaustedException if a recrord is sent and the buffer space is full. Ignored.
max.in.flight.requests.per.connection The maximum number of unacknowledged requests the client will send on a single connection before blocking. Note that if this setting is set to be greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (i.e., if retries are enabled). Ignored. MapR Streams has a similar parameter: streams.parallel.flushers.per.partition
metadata.fetch.timeout.ms The first time data is sent to a topic we must fetch metadata about that topic to know which servers host the topic's partitions. This fetch to succeed before throwing an exception back to the client. Ignored.
metadata.max.age.ms The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions. Yes
metric.reporters A list of classes to use as metrics reporters. Implementing the MetricReporter interface allows plugging in classes that will be notified of new metric creation. The JmxReporter is always included to register JMX statistics. No
metrics.num.samples The number of samples maintained to compute metrics. No.
metrics.sample.window.ms The number of samples maintained to compute metrics. No.
reconnect.backoff.ms The amount of time to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop. This backoff applies to all requests sent by the consumer to the broker. Ignored. MapR Streams has a similar parameter: streams.rpc.timeout.ms
retry.backoff.ms The amount of time to wait before attempting to retry a failed fetch request to a given topic partition. This avoids repeated fetching-and-failing in a tight loop. Ignored. MapR Streams has a similar parameter: streams.rpc.timeout.ms
sasl.kerberos.kinit.cmd Kerberos kinit command path. Default is /usr/bin/kinit Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
sasl.kerberos.min.time.before.relogin Login thread sleep time between refresh attempts. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
sasl.kerberos.ticket.renew.jitter Percentage of random jitter added to the renewal time. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
sasl.kerberos.ticket.renew.window.factor Login thread will sleep until the specified window factor of time from last refresh to ticket's expiry has been reached, at which time it will try to renew the ticket. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.cipher.suites A list of cipher suites. This is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol.By default all the available cipher suites are supported. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.endpoint.identification.algorithm The endpoint identification algorithm to validate server hostname using server certificate. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.keymanager.algorithm The algorithm used by key manager factory for SSL connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.trustmanager.algorithm The algorithm used by trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.

Support for Apache Kafka's Configuration Parameters for Consumers

Name Description Supported for consumers in MapR Streams?
bootstrap.servers A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form host1:port1,host2:port2,.... Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down). No. Cluster details are discovered from the file mapr-clusters.conf.
key.deserializer Deserializer class for key that implements the Deserializer interface. Yes
value.deserializer Deserializer class for value that implements the Deserializer interface. Yes
fetch.min.bytes The minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait for that much data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. Setting this to something greater than 1 will cause the server to wait for larger amounts of data to accumulate which can improve server throughput a bit at the cost of some additional latency. Yes
group.id A unique string that identifies the consumer group this consumer belongs to. This property is required if the consumer uses either the group management functionality by using subscribe(topic) or the Kafka-based offset management strategy. Yes
heartbeat.interval.ms The expected time between heartbeats to the consumer coordinator when using Kafka's group management facilities. Heartbeats are used to ensure that the consumer's session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances. No
max.partition.fetch.bytes The maximum amount of data per-partition the server will return. The maximum total memory used for a request will be #partitions * max.partition.fetch.bytes. This size must be at least as large as the maximum message size the server allows or else it is possible for the producer to send messages larger than the consumer can fetch. If that happens, the consumer can get stuck trying to fetch a large message on a certain partition. Yes
session.timeout.ms The timeout used to detect failures when using Kafka's group management facilities. Ignored
ssl.key.password The password of the private key in the key store file. This is optional for client. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.keystore.location The location of the key store file. This is optional for client and can be used for two-way authentication for client. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.keystore.password The store password for the key store file.This is optional for client and only needed if ssl.keystore.location is configured. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.truststore.location The location of the trust store file. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.truststore.password The password for the trust store file. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
auto.offset.reset

What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted):

earliest
automatically reset the offset to the earliest offset
latest
automatically reset the offset to the latest offset
none
throw exception to the consumer if no previous offset is found for the consumer's group
anything else
throw exception to the consumer.
Yes
connections.max.idle.ms Close idle connections after the number of milliseconds specified by this config. Ignored
enable.auto.commit If true the consumer's offset will be periodically committed in the background. Yes
partition.assignment.strategy The class name of the partition assignment strategy that the client will use to distribute partition ownership amongst consumer instances when group management is used Ignored. MapR Streams distributes partitions equally among the consumers in a group.
receive.buffer.bytes The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. Ignored.
request.timeout.ms The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted. Ignored. MapR Streams has a similar parameter: streams.rpc.timeout.ms
sasl.kerberos.service.name The Kerberos principal name that Kafka runs as. This can be defined either in Kafka's JAAS config or in Kafka's config. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
security.protocol Protocol used to communicate with brokers. Currently only PLAINTEXT and SSL are supported. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
send.buffer.bytes The size of the TCP send buffer (SO_SNDBUF) to use when sending data. Ignored.
ssl.enabled.protocols The list of protocols enabled for SSL connections. TLSv1.2, TLSv1.1 and TLSv1 are enabled by default. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.keystore.type The file format of the key store file. This is optional for client. Default value is JKS Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.protocol The SSL protocol used to generate the SSLContext. Default setting is TLS, which is fine for most cases. Allowed values in recent JVMs are TLS, TLSv1.1 and TLSv1.2. SSL, SSLv2 and SSLv3 may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.provider The name of the security provider used for SSL connections. Default value is the default security provider of the JVM. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.truststore.type The file format of the trust store file. Default value is JKS. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
auto.commit.interval.ms The frequency in milliseconds that the consumer offsets are auto-committed to Kafka if enable.auto.commit is set to true. Yes
check.crcs Automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so it may be disabled in cases seeking extreme performance.

Ignored.

MapR Streams always does end-to-end crc computation and verification.

client.id An id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging. Yes
fetch.max.wait.ms The maximum amount of time the server will block before answering the fetch request if there isn't sufficient data to immediately satisfy the requirement given by fetch.min.bytes. No
metadata.max.age.ms The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions. Yes
metric.reporters A list of classes to use as metrics reporters. Implementing the MetricReporter interface allows plugging in classes that will be notified of new metric creation. The JmxReporter is always included to register JMX statistics. No
metrics.num.samples The number of samples maintained to compute metrics. No
metrics.sample.window.ms The number of samples maintained to compute metrics. No
reconnect.backoff.ms The amount of time to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop. This backoff applies to all requests sent by the consumer to the broker. Ignored. MapR Streams has a similar parameter: streams.rpc.timeout.ms
retry.backoff.ms The amount of time to wait before attempting to retry a failed fetch request to a given topic partition. This avoids repeated fetching-and-failing in a tight loop. Ignored. MapR Streams has a similar parameter: streams.rpc.timeout.ms
sasl.kerberos.kinit.cmd Kerberos kinit command path. Default is /usr/bin/kinit Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
sasl.kerberos.min.time.before.relogin Login thread sleep time between refresh attempts. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
sasl.kerberos.ticket.renew.jitter Percentage of random jitter added to the renewal time. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
sasl.kerberos.ticket.renew.window.factor Login thread will sleep until the specified window factor of time from last refresh to ticket's expiry has been reached, at which time it will try to renew the ticket. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.cipher.suites A list of cipher suites. This is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol.By default all the available cipher suites are supported. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.endpoint.identification.algorithm The endpoint identification algorithm to validate server hostname using server certificate. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.keymanager.algorithm The algorithm used by key manager factory for SSL connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
ssl.trustmanager.algorithm The algorithm used by trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine. Ignored. Authentication and authorization are handled through MapR security. See the Security Overview.
Client Parameter Description Kafka Parameter Replaced
Producer streams.buffer.max.time.ms

Messages are buffered in the producer for at most the specified time. A thread will flush all the messages that have been buffered for more than the time specified.

Default: 3 * 1000 msec

linger.ms
streams.parallel.flushers.per.partition

If enabled, producer may have multiple parallel send requests to the server for each topic partition. If this setting is set to true, it is possible for messages to be sent out of order.

Default: true

max.in.flight.requests.per.connection
streams.partitioner.class The class that implements the StreamsPartitioner interface. This interface lets you write custom algorithms for determining which topic and partition to use for messages that match specific criteria. Use this configuration parameter only for producers that are written in Java. Not applicable.
streams.producer.default.stream

Specifies the stream that the producer will use by default if the producer does not provide the name of a stream when specifying a topic to write to.

For example, the producer can specify the name of a stream together with the name of a topic to write to, like this:

/<stream>/<topic>

However, if the stream is not specified, the value of this configuration parameter is assumed to be the stream in which the topic is located.

If the producer specifies the name of a topic without also providing the path and name of the stream, and there is no value for this configuration parameter, MapR Streams assumes that the topic specified is in Apache Kafka and does nothing.

Not applicable.
streams.rpc.timeout.ms

Specifies the length of time in milliseconds to wait for a response from the MapR Streams server. In effect, this parameter determines whether to use a hard mount or a soft mount to the server.

The default (no value) specifies to use a hard mount.

If you specify a value of 30000 or greater, a soft mount is used.

If you specify a value that is less than 30000, an error is thrown when the producer starts.

If a soft mount is used, the time expires while a producer waits for a response from the MapR Streams server, and the producer used the KafkaProducer.send(ProducerRecord<K,V> record, Callback callback) method, the callback is invoked with the error EAGAIN, which means "Resource temporarily unavailable."

max.block.ms

request.timeout.ms

timeout.ms

reconnect.backoff.ms

retry.backoff.ms

Consumer streams.consumer.default.stream

Specifies the path and name of the stream that the consumer subscribes to if, when subscribing to a topic, the consumer does not specify a stream.

This default value is also used for the KafkaConsumer.listTopics() method.

Not applicable.
streams.rpc.timeout.ms

Specifies the length of time in milliseconds to wait for a response from the MapR Streams server. In effect, this parameter determines whether to use a hard mount or a soft mount to the server.

The default (no value) specifies to use a hard mount.

If you specify a value of 30000 or greater, a soft mount is used.

If you specify a value that is less than 30000, an error is thrown when the consumer starts.

request.timeout.ms

reconnect.backoff.ms

retry.backoff.ms