Monitoring Producers

To get a sense of how quickly producers are sending messages to the producer client library, you can run the command stream topic info at intervals.

Doing so will show you changes over time in the rate at which the values for maxoffset and maxtimestamp increase for the partitions in the topics that your producers are publishing to.

For example, if you have a script that runs the command at intervals of 10 seconds, the change per second would be (Value at first run - Value at second run)/10.

This is an indirect measure of the speed of the producers because the producer client library batches messages before publishing them to partitions. The faster the producers send messages to the client, the faster the client publishes message batches, and the greater the change per second.

If producers do not seem to be sending messages quickly enough, and this problem is not caused by server-side I/O bottlenecks, you can spawn more producer threads.

See stream topic info for the syntax of this command.