Kafka vs Redis. I've only used Redis... what should I know?
Heroku Kafka
11–20 of 120 posts
Re: Heroku Kafka
#12I've wondered why there isn't a "big player" in the cloud space for this. Felt like a hole. My operating theory is that the people who would really make use of something like this have grown beyond managed offerings and would take it in house. For smaller operations Redis is more than enough for pub/sub. Ditto for SQS for externally triggered eventing.
Depending on what you mean by "this" there are offerings by the big players. Google has Cloud Pub/Sub and AWS has Kinesis in addition to SQS, so two of the big players do have offerings. I'm not familiar enough with Azure to know what it has.
Re: Heroku Kafka
#13Kafka vs Redis. I've only used Redis... what should I know?
Redis is an in-memory (with persistence) key-value database that also implements some basic structures like lists, sets and hashes natively.
Kafka is a distributed logging system that can ingest large amounts of data straight to disk, then allows for multiple consumers to read this data through a simple abstraction of topics and partitions. Consumers maintain their own position of where they last read up to (or re-read things if they want) and everything is sequential I/O which creates very high throughput.
Re: Heroku Kafka
#14> What is Kafka? > Apache Kafka is a distributed commit log for fast, fault-tolerant communication between producers and consumers using message based topics. Kafka provides the messaging backbone for building a new generation of distributed applications capable of handling billions of events and millions of transactions Can anyone translate this into meaningful English for me?
You should read this: https://engineering.linkedin.com/distributed-systems/log-wha... - it's long, but it's one of the most impactful essays I've read on software engineering in years.
+1 regarding reading that essay on Kafka. The explanations, illustrations, evolution and use case are all well thought out.
Re: Heroku Kafka
#15Earlier quoted context omitted.
Would love to hear what you don't care for in Kafka and what alterative solution(s) you prefer.
To clarify, my feelings towards Kafka are from the POV of someone who has had to build a managed service on top of it, which is not the common use case (for which many people seem to be happy with). Other people may have more positive experiences. In my experience, Kafka is a solid system when you work in its wheelhouse, which is a relatively static set of servers / topics, that you add to slowly and deliberately. If…
Your comment is highly technical, critical, but still very fair. This is why I love HN.
Re: Heroku Kafka
#16> What is Kafka? > Apache Kafka is a distributed commit log for fast, fault-tolerant communication between producers and consumers using message based topics. Kafka provides the messaging backbone for building a new generation of distributed applications capable of handling billions of events and millions of transactions Can anyone translate this into meaningful English for me?
Kafka is a distributed logging system. Write lots of data very fast by using sequential I/O. Consuming apps can then read this just as fast and maintain their own state (of where they last read up to) which allows for multiple fast and simple consumers and an easy way to have a lasting "log" of all the data.
Re: Heroku Kafka
#17Can somebody provide a real-life use case for Kafka? I've seen comparisons between Redis, but what specifically does Kafka solve that Redis cannot?
Re: Heroku Kafka
#18Can somebody provide a real-life use case for Kafka? I've seen comparisons between Redis, but what specifically does Kafka solve that Redis cannot?
Kafka and Redis are very different things - see this: https://news.ycombinator.com/item?id=11577312
Redis is a database, Kakfa is a data logging system built for scale and throughput. Event processing (of any kind like stocks, ad impressions, ecommerce purchases) are a great fit. Also good as a message queue unless you need ultra low-latency RPC.
Re: Heroku Kafka
#19Can somebody provide a real-life use case for Kafka? I've seen comparisons between Redis, but what specifically does Kafka solve that Redis cannot?
Kafka and Redis are very different things - see this: https://news.ycombinator.com/item?id=11577312 Redis is a database, Kakfa is a data logging system built for scale and throughput. Event processing (of any kind like stocks, ad impressions, ecommerce purchases) are a great fit. Also good as a message queue unless you need ultra low-latency RPC.
Gotcha, so then advantage of Kafka over Logstash + ElasticSearch?
Re: Heroku Kafka
#20I've wondered why there isn't a "big player" in the cloud space for this. Felt like a hole. My operating theory is that the people who would really make use of something like this have grown beyond managed offerings and would take it in house. For smaller operations Redis is more than enough for pub/sub. Ditto for SQS for externally triggered eventing.
Depending on what you mean by "this" there are offerings by the big players. Google has Cloud Pub/Sub and AWS has Kinesis in addition to SQS, so two of the big players do have offerings. I'm not familiar enough with Azure to know what it has.
By "this" I meant a managed Kafka cloud offering. I generally a fan of these types of services as there isn't as tight a binding as proprietary ones. Migrating from Heroku Postgres to RDS or self hosted is well defined. Ditto for Redis migrations.
SQS, Kinesis, and other proprietary ones not so much. You can insulate your code base but if you're really going to leverage the ecosystem of those services then you're going to be stuck there. That's why I find something like this interesting. The "out" is there so it makes it easier to accept getting in.