Live data from Hacker News

Kafka is dead, long live Kafka

warpstream.com

221–230 of 295 posts

Re: Kafka is dead, long live Kafka

#221

Earlier quoted context omitted.

It sounds like there's a sweet spot here. If you are not ACKing Produce requests for 100ms then there's a huge amount latency. If the user want's to reduce that latency from 100ms to say 1ms then their S3 GET requests cost just went up by 100x.

[WarpStream co-founder here] We've done lots of customer research here and, combined with the experience my co-founder and I have, we can confidently say most Kafka users (especially high-throughput users) would happily make a trade off of increased end-to-end latency in exchange for a massive cost reduction and the operational simplicity provided by WarpStream.

Yes kafka is definitely in an awkward latency spot.

Re: Kafka is dead, long live Kafka

#222
post #77

Earlier quoted context omitted.

how do you design avoiding message queues? Or do you use other alternatives around kafka for these things?

RabbitMQ, ActiveMQ, MQtt All of those are fine is you only need pub/sub

Just to clarify MQTT is a pub/sub protocol, while ActiveMQ and RabbitMQ are message broker implementations. As an example ActiveMQ implements MQTT as one of its optional protocols. Though if anyone's looking for a simple MQTT implantation I'd recommend Mosquitto. My 2 cents as someone who's worked with ActiveMQ pub/sub for quite a few years.

Re: Kafka is dead, long live Kafka

#223
post #193

Earlier quoted context omitted.

What do you use instead? Polling APIs? A queue instead of an event stream? Event based architectures definitely add infrastructural overhead, but are positive at a certain scale and/or architectural complexity (multiple decoupled subscribers).

We use a table called "Messages" in our SQL Server database. Everyone talks to the same database. Turns out we don't really need to push extreme message rates or meet aggressive single-digit millisecond budgets, so this works out well in practice. It is also the easiest thing on earth to develop & debug, because you can monitor the table/log and instantly understand the state of the whole system and how it got there.…

For PG users: you can use NOTIFY to avoid polling.

Re: Kafka is dead, long live Kafka

#224
post #193

Earlier quoted context omitted.

What do you use instead? Polling APIs? A queue instead of an event stream? Event based architectures definitely add infrastructural overhead, but are positive at a certain scale and/or architectural complexity (multiple decoupled subscribers).

We use a table called "Messages" in our SQL Server database. Everyone talks to the same database. Turns out we don't really need to push extreme message rates or meet aggressive single-digit millisecond budgets, so this works out well in practice. It is also the easiest thing on earth to develop & debug, because you can monitor the table/log and instantly understand the state of the whole system and how it got there.…

Ah yes, just start with SQL is really the right choice for a lot if not most things. I work on things that are both too high scale and too organizationally complex for the simple solutions.

Re: Kafka is dead, long live Kafka

#225
post #218

Earlier quoted context omitted.

(WarpStream founder) It guarantees correct ordering inside a partition for all acknowledged messages regardless of which batch they originated from. We don't synchronize clocks, the agents call out to our cloud metadata store which runs a per-cluster metadata store that assigns offsets to messages at commit time. Effectively "committing" data involves two steps: 1. Write a file to S3 2. "Commit" that file to the meta…

Then the order is on batch level? Say batch1 is committed with a smaller timestamp than batch2, then all messages in batch1 are considered prior/earlier than any message in batch2?

It’s getting late and I’m not 100% confident I’m sure what you’re asking, but I believe the answer to your question is yes. When you produce a message/batch you get back offsets for each message you produced. If you produce again after receiving that acknowledgement, the next set of offsets you receive are guaranteed to be larger than any of the offsets you received from your previous request.

Re: Kafka is dead, long live Kafka

#226

Earlier quoted context omitted.

Kafka is less message queue (rabbitmq, sqs) and more ordered stream/write ahead log (ala kinesis) > The first is that because of the nature of message queues, they're either empty or full. ... Wat? > that is it's a true "fire and forget" kind of request and direct response time is not a concern, then what does the queue gain you over simply posting it to a database table? Performance is why. The fire and forget aspec…

> > The first is that because of the nature of message queues, they're either empty or full. > ... Wat? I interpreted it as "they are either trending towards empty or full". The statement doesn't seem well thought through. That might be true (either empty or full) most of the time (maybe) _if you squint_, but the entire point of the Message Queue is to provide buffering from the transient state (somewhere between emp…

Yea, I feel like these people never got a chron daily data dump (times N customers). Scaling is dead simple, and there isn't a need (or ability) to instantaneously handle large bursty workloads in like 99% of cases.

Longer SLAs mean it's also easier to hit those SLAs. Giving the clients realistic SLAs is super important.

Re: Kafka is dead, long live Kafka

#227

I think the focus on the $/GB is wrong. The main goal of Kafka is providing certain processing guarantees and connecting consumers with producers of events. What Kafka is not for is dealing with enormous number of events or enormous amount of data. Not that it is particularly slow at it (actually, quite fast) but I see a lot of people needlessly try to push more data and events than they need and then complain they h…

You’ve described the tricks WarpSream does in their implementation, buffering data until a large enough chunk can be written to S3.

Re: Kafka is dead, long live Kafka

#228
post #197

> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…

To me a technology company is not just a company that uses tech (every company does that) but one whose core value proposition is fundamentally technical. And I think most serious companies doing that have a need for highly available data storage, for which Kafka is the least bad option. What are the alternatives? Cassandra is just as operationally complex and harder to fit your dataflow into. The various efforts to…

I think if you dismiss HA setups of SQL dbs as "you won't get around to operating it properly" the same ops culture will also end up getting many less 9's availability than aspired to with Kafka.

(But also of course lots of applications are also fine with the availability that you get from fate-sharing with a single db server)

Re: Kafka is dead, long live Kafka

#229
post #206

Earlier quoted context omitted.

We standardized on Clickhouse for everything. (With its own set of surprising and/or horrifying ops issues.) But at least it is a proper high-load, high-availablity solution, unlike Kafka, Cassandra, et al.

> We standardized on Clickhouse for everything. (With its own set of surprising and/or horrifying ops issues.) Clickhouse I admittedly haven't personally seen quite as much operational unpleasantness as Greenplum or Galera, but at this point I'm dubious of anything in that bucket. > But at least it is a proper high-load, high-availablity solution, unlike Kafka, Cassandra, et al. What went wrong with those for you? In…

Not the parent, but I have some ClickHouse experience. ClickHouse is surprisingly easy to deploy and setup, talks both mysql and postgresql wire protocols (so you can query stuff with your existing relational tools), the query language is SQL (including joins with external data sources, such as S3 files, external relational databases and other clickhouse tables), and it is ACID on specific operations. It assumes your dataset is (mostly) append-only, and inserts work well when done in batch. It is also blazingly fast, and very compact when using the MergeTree family of storage engines.

Development is very active, and some features are experimental. One of the common mistakes is to use latest releases for production environments - you will certainly find odd bugs on specific usage scenarios. Stay away from the bleeding edge and you're fine. Clustering (table replication and sharding of queries) is also a sort-of can of worms by itself, and requires good knowledge of your workload and your data structure to understand all the tradeoffs. Thing is, when designing from scratch, you can often design in such a way where you don't need (clustered) table replication or sharding - again, this also has a learning curve, for both devs and devops.

You can easily spin it on a VM or on your laptop, load a dataset and see for yourself how powerful ClickHouse can be. Honestly, just the data compression alone is good enough to save a s**load of money on storage on an enterprise, compared to most solutions. Couple this with tiered storage - your hot data is eg. in ssd, your historical data is stored on s3, and rotation is done automatically, plus automated ingestion from kafka, and you have a data warehousing system at a fraction of the price of many common alternatives.

Re: Kafka is dead, long live Kafka

#230
post #193

Earlier quoted context omitted.

What do you use instead? Polling APIs? A queue instead of an event stream? Event based architectures definitely add infrastructural overhead, but are positive at a certain scale and/or architectural complexity (multiple decoupled subscribers).

We use a table called "Messages" in our SQL Server database. Everyone talks to the same database. Turns out we don't really need to push extreme message rates or meet aggressive single-digit millisecond budgets, so this works out well in practice. It is also the easiest thing on earth to develop & debug, because you can monitor the table/log and instantly understand the state of the whole system and how it got there.…

fyi, SQL Server has a message broker built-in.

https://learn.microsoft.com/en-us/sql/database-engine/config...

Post reply on HN