Go to All Forums

Plugin to show metrics specific to Kafka consumer group and topic lag partition wise without confluent platform.

We would like to have a plugin to shows the lag for the provided consumer group.

Currently, I need to monitor Kafka consumer and topic message lag. we are not using confluent platform to use the existing plugin. Could you please help us create plugin for this propose which would offer following metrics specific to consumer, topic and partitions used.

(GROUP NAME, TOPIC, PARTITION, CURRENT-OFFSET, LOG-END-OFFSET, LAG, CONSUMER-ID, HOST and CLIENT-ID)

By executing below command could get the result i wanted but plugin for this would be helpful.

bin/kafka-consumer-groups --bootstrap-server <kafka-broker>:9092 --describe --group sample-consumer-group

Like (1) Reply
Replies (2)

Hi Guest,

Thanks for your request. 

We will analyze the Kafka consumer metrics and add them to our Plugins Roadmap. We will update in this thread, once we are done.

Regards,

Muralikrishnan

Product Manager

Like (0) Reply

Hello,

We are happy to introduce Apache Kafka plugin integrations to monitor your entire Kafka stack. You will now be able to monitor critical Kafka producer, consumer, and broker metrics in real time and stay ahead of issues and bottlenecks.

Refer to the below GitHub link for the Kafka monitoring plugin details and setup steps:

https://github.com/site24x7/plugins/tree/master/Kafka_Complete_Monitoring

You can monitor the following components with our Kafka plugins: 

1. Kafka consumer

Install the Kafka Consumer plugin and monitor key consumer metrics.

https://github.com/site24x7/plugins/tree/master/Kafka_Complete_Monitoring/Kafka_Consumer_Monitoring

      Supported metrics: 

  • Records Lag Max(All Partitions)
  • Bytes Consumed Rate(All Topics)
  • Bytes Consumed Rate(Topic Specific)
  • Records Consumed Rate(All Topics)
  • Records Consumed Rate(Topic Specific)
  • Fetch Rate
  • Records Lag
  • Records Lag Max
  • Records Per Request Avg
  • Fetch Throttle Time Avg
  • Fetch Throttle Time Max
  • Topic Name
  • Partition No

2. Kafka producer

Install the Apache Kafka Producer plugin to monitor key Kafka producer metrics and ensure a steady flow of data.

https://github.com/site24x7/plugins/tree/master/Kafka_Complete_Monitoring/Kafka_Producer_Monitoring

      Supported metrics:

  • Compression Rate Avg
  • Response Rate
  • Request Rate
  • Request Latency Avg
  • Outgoing Byte Rate
  • IO Wait Time NS Avg
  • Batch Size Avg

3. Kafka broker

 https://github.com/site24x7/plugins/tree/master/Kafka_Complete_Monitoring/Kafka_Broker_Monitoring

      Supported metrics:

  • Under Replicated Partitions
  • ISR Shrinks Per Sec
  • ISR Expands Per Sec
  • Active Controller Count
  • Offline Partitions Count
  • Leader Election Rate And Time Ms
  • Unclean Leader Elections Per Sec
  • Total Time ms
  • Purgatory Size
  • Bytes In Per Sec
  • Network Request Rate
  • Network Error Rate
  • Total Broker Partitions
  • Young Generation GC Count
  • Young Generation GC Time
  • Old Generation GC Count
  • Old Generation GC Time
  • Log End Offset

Prerequisites for installing the plugins:

  • Install the jmxquery module for Python3.

           pip install jmxquery

Plugin Installation:

If you want to install the Kafka consumer monitoring plugin,

  • Create a directory named "kafka_consumer_monitoring" under the Site24x7 Linux Agent plugin directory:

    Linux    ->   /opt/site24x7/monagent/plugins/kafka_consumer_monitoring
  • Download all the files in the "kafka_consumer_monitoring" folder and place it under the "kafka_consumer_monitoring" directory.

wget https://raw.githubusercontent.com/site24x7/plugins/master/Kafka_Complete_Monitoring/Kafka_Consumer_Monitoring/kafka_consumer_monitoring.py

wget
https://raw.githubusercontent.com/site24x7/plugins/master/Kafka_Complete_Monitoring/Kafka_Consumer_Monitoring/kafka_consumer_monitoring.cfg

  • Provide appropriate configuration in the kafka_consumer_monitoring.cfg file.

To test the plugin manually, execute the below command:

 python3 kafka_consumer_monitoring.py --kafka_consumer_host=<KAFKA_CONSUMER_HOST_NAME> kafka_consumer_jmx_port=<KAFKA_CONSUMER_PORT_NO> --kafka_consumer_partition=<KAFKA_CONSUMER_PARTITION_NO> --kafka_topic_name=<KAFKA_TOPIC_NAME> --kafka_consumer_client_id=<KAFKA_CONSUMER_CLIENT_ID> --logs_enabled=False --log_type_name=None --log_file_path=None

The Kafka consumer plugin will be added for monitoring during the next data collection event of the agent.

Repeat the respective processes for producer and broker plugins as well.

If you have any queries, please reply to this thread.

Regards,

Muralikrishnan
Product Manager

Like (0) Reply

Was this post helpful?