Sharing this here to see if it helps anyone else or to improve it further. I created this tool to do the following:
Automate initial discovery of configuration optimization of both clients and consumers in a full end-to-end scenario from producers to consumers.
For existing clusters, I run multiple instances of latency.py against different topics with different datasets to test load and configuration settings
For training new users on the importance of client settings, I run their settings through and then let the program optimize and return better throughput results.
I use the CSV generated results to graph/visually represent configuration changes as throughput changes.
Show HN: Automated Kafka client and broker optimization
github.com