Parallelism when Consuming Messages

For parallelism when reading messages from topics, you can create consumer groups. These groups consist of consumers that are associated with an ID that you set for each of the partiticipating consumers with the group.id configuration parameter. The partitions in each topic that all of the consumers are subscribed to are assigned dynamically to the consumers in round-robin fashion.

These groups consist of consumers that are associated with an ID that you set for each of the partiticipating consumers with the group.id configuration parameter. The partitions in each topic that all of the consumers are subscribed to are assigned dynamically to the consumers in round-robin fashion.

For example, suppose that there are three consumers in a group and each consumer is subscribed to the same topic. There are five partitions in the topic. MapR-ES assigns each partition to a consumer, with two consumers both being assigned two partitions.

If one of the consumers goes offline, the partitions are reassigned dynamically among the remaining consumers in the group.

If the offline consumer comes back online or a different consumer is added to the group, again the partitions are redistributed among the consumers in the group.

This parallelism and dynamic reassignment is possible only when none of the consumers in a consumer group subscribe to individual partitions.

For example, suppose that of three consumers in a consumer group:
  • Two subscribe to the same topic.
  • One subscribes to a single partition within that topic.
If the topic has five partitions, MapR-ES assigns four of them via round robin to two of the consumers. Only the remaining partition is read from by the third consumer.

If that third consumer fails, MapR-ES does not reassign its partition to either of the other consumers.

Now that you understand how partitions are assigned when the number of partitions is equal to or greater than the number of consumers in a consumer group, you might be wondering what happens if the number of partitions in a topic is less than the number of consumers in a consumer group. The answer is simply that one or more consumers in the consumer group will not be assigned any partitions from the topic.

That does not necessarily mean those consumers will be idle. There could be other topics that the consumer group is subscribed to, and those consumers could be assigned partitions from those other topics.

For example, in this diagram there is a consumer group with four consumers. Topic A has only three partitions, and those are assigned to the first three consumers shown in the group. However, the fourth consumer is not idle. The consumer group also subscribes to Topic B, which has more partitions than there are consumers. Each of the consumers in the group is assigned at least one partition from Topic B.

Moreover, if a consumer that is assigned a partition from Topic A happens to fail, its partition will be reassigned to the fourth consumer.