Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: UI for Apache Kafka

github.com

Post reply on HN
Print view View original

41–50 of 60 posts

Re: Show HN: UI for Apache Kafka

#41

Post by chadhq »

d_t_w wrote: post #36

https://kpow.io - we build kPow for Apache Kafka, a drop-in replacement for Confluent Control Center + much more. https://kpow.io/features - Multi-Cluster, Multi-Schema, Multi-Connect. - Multi-Tenancy, RBAC, all the Authz - LDAP, SAML, etc. - Very fast streaming multi-topic search (~1M messages / minute) - Built in JQ search filters (we implemented a subset of JQ in the browser and JVM) - Live Kafka Streams / Consume…

kRepl is mind blowing. Our engineers have used it very effectively to help diagnose issues in production.

Re: Show HN: UI for Apache Kafka

#42

Post by d_t_w »

dxxvi wrote: post #35

I'm using kafdrop but modify it a bit. Reasons: - the front page is very slow if there are a lot of topics (the kafka cluster I'm using has 6k topics). It's because kafdrop tries to get a lot of information of every topic even though it doesn't display them on the screen. I modified it like this: after display 6k topic names, I send ajax requests to get info (# of partitions, start & end offsets of each partition, nu…

See https://kpow.io - if you would like a trial license let me know. 6k topics is fine, might need slight tuning which is easily done (6k groups slightly more complicated).

Re: Show HN: UI for Apache Kafka

#43

Post by tiew9Vii »

d_t_w wrote: post #36

https://kpow.io - we build kPow for Apache Kafka, a drop-in replacement for Confluent Control Center + much more. https://kpow.io/features - Multi-Cluster, Multi-Schema, Multi-Connect. - Multi-Tenancy, RBAC, all the Authz - LDAP, SAML, etc. - Very fast streaming multi-topic search (~1M messages / minute) - Built in JQ search filters (we implemented a subset of JQ in the browser and JVM) - Live Kafka Streams / Consume…

Is the monitoring functionality independent off the cluster it is monitoring being up to see monitoring?

Control centre eats its own dog food. All its data is on topics. That means if your cluster is unhealthy you can see healthy stats until the control centre consumers have caught up. Which means if you have a lot of data and load, control centre can start lagging and in one case I had some very hard debugging wondering why control centre saying the cluster was unhealthy but the cluster was up. It turned out control centre was lagging by 24 hrs so I was seeing the previous days outage.

Confluent recommended putting control centre on its own cluster for this very reason. One that isn’t cheap, a minimal prod cluster is 3 brokers+zookeepers and secondly what then monitors your monitoring cluster.

Re: Show HN: UI for Apache Kafka

#44

Post by [deleted] »

d_t_w wrote: post #36

https://kpow.io - we build kPow for Apache Kafka, a drop-in replacement for Confluent Control Center + much more. https://kpow.io/features - Multi-Cluster, Multi-Schema, Multi-Connect. - Multi-Tenancy, RBAC, all the Authz - LDAP, SAML, etc. - Very fast streaming multi-topic search (~1M messages / minute) - Built in JQ search filters (we implemented a subset of JQ in the browser and JVM) - Live Kafka Streams / Consume…

[deleted]

Re: Show HN: UI for Apache Kafka

#45

Post by [deleted] »

PhilippGille wrote: post #31

There are several alternatives posted here already, but after comparing several ones maybe a year ago I found Kowl very nice: Nice UI, backend written in Go (quick startup, low memory usage), viewing and filtering of messages, support for schema registry, comes with a Docker image, usable locally and Helm chart for Kubernetes. https://github.com/cloudhut/kowl

[deleted]

Re: Show HN: UI for Apache Kafka

#47

Post by [deleted] »

dxxvi wrote: post #35

I'm using kafdrop but modify it a bit. Reasons: - the front page is very slow if there are a lot of topics (the kafka cluster I'm using has 6k topics). It's because kafdrop tries to get a lot of information of every topic even though it doesn't display them on the screen. I modified it like this: after display 6k topic names, I send ajax requests to get info (# of partitions, start & end offsets of each partition, nu…

[deleted]

Re: Show HN: UI for Apache Kafka

#49

Post by d_t_w »

tiew9Vii wrote: post #43
d_t_w wrote: post #36

https://kpow.io - we build kPow for Apache Kafka, a drop-in replacement for Confluent Control Center + much more. https://kpow.io/features - Multi-Cluster, Multi-Schema, Multi-Connect. - Multi-Tenancy, RBAC, all the Authz - LDAP, SAML, etc. - Very fast streaming multi-topic search (~1M messages / minute) - Built in JQ search filters (we implemented a subset of JQ in the browser and JVM) - Live Kafka Streams / Consume…

Is the monitoring functionality independent off the cluster it is monitoring being up to see monitoring? Control centre eats its own dog food. All its data is on topics. That means if your cluster is unhealthy you can see healthy stats until the control centre consumers have caught up. Which means if you have a lot of data and load, control centre can start lagging and in one case I had some very hard debugging wonde…

Yes, in two ways.

First - kPow also stores all telemetry in internal topics (and also keeps an audit-log in another topic for data governance). Setting up multi-cluster is fairly easy and kPow stores data in the first configured cluster - so like Confluent's suggestion you could have a dedicated kPow cluster. We really only recommend this if/when you get up to a large number of connected Kafka resources (Cluster, Schema, or Connect) and/or you have cluster/s that are very busy with other work.

Second - from v85 (next week) our v3 compute engine make kPow monitoring functionally independent of topic load / performance of the underlying cluster. We do still persist to internal topics for recovery, etc, but live stats and control are only dependent on your Kafka cluster responding to AdminClient requests (a handful every minute).

If you would like a trial license or guided tour, just say.

Re: Show HN: UI for Apache Kafka

#50

Post by weeco »

smsm42 wrote: post #34
PhilippGille wrote: post #31

There are several alternatives posted here already, but after comparing several ones maybe a year ago I found Kowl very nice: Nice UI, backend written in Go (quick startup, low memory usage), viewing and filtering of messages, support for schema registry, comes with a Docker image, usable locally and Helm chart for Kubernetes. https://github.com/cloudhut/kowl

kowl is awesome. The only missing feature I've found is that when looking at groups, only lag is shown but not the actual position.

In the newer versions you can see the actual offset in the group details page :-)
Post reply on HN