Earlier quoted context omitted.
It sounds like there's a sweet spot here. If you are not ACKing Produce requests for 100ms then there's a huge amount latency. If the user want's to reduce that latency from 100ms to say 1ms then their S3 GET requests cost just went up by 100x.
[WarpStream co-founder here] We've done lots of customer research here and, combined with the experience my co-founder and I have, we can confidently say most Kafka users (especially high-throughput users) would happily make a trade off of increased end-to-end latency in exchange for a massive cost reduction and the operational simplicity provided by WarpStream.
Kafka is dead, long live Kafka
221–230 of 295 posts
Re: Kafka is dead, long live Kafka
#222Earlier quoted context omitted.
how do you design avoiding message queues? Or do you use other alternatives around kafka for these things?
RabbitMQ, ActiveMQ, MQtt All of those are fine is you only need pub/sub
Re: Kafka is dead, long live Kafka
#223Earlier quoted context omitted.
What do you use instead? Polling APIs? A queue instead of an event stream? Event based architectures definitely add infrastructural overhead, but are positive at a certain scale and/or architectural complexity (multiple decoupled subscribers).
We use a table called "Messages" in our SQL Server database. Everyone talks to the same database. Turns out we don't really need to push extreme message rates or meet aggressive single-digit millisecond budgets, so this works out well in practice. It is also the easiest thing on earth to develop & debug, because you can monitor the table/log and instantly understand the state of the whole system and how it got there.…
Re: Kafka is dead, long live Kafka
#224Earlier quoted context omitted.
What do you use instead? Polling APIs? A queue instead of an event stream? Event based architectures definitely add infrastructural overhead, but are positive at a certain scale and/or architectural complexity (multiple decoupled subscribers).
We use a table called "Messages" in our SQL Server database. Everyone talks to the same database. Turns out we don't really need to push extreme message rates or meet aggressive single-digit millisecond budgets, so this works out well in practice. It is also the easiest thing on earth to develop & debug, because you can monitor the table/log and instantly understand the state of the whole system and how it got there.…
Re: Kafka is dead, long live Kafka
#225Earlier quoted context omitted.
(WarpStream founder) It guarantees correct ordering inside a partition for all acknowledged messages regardless of which batch they originated from. We don't synchronize clocks, the agents call out to our cloud metadata store which runs a per-cluster metadata store that assigns offsets to messages at commit time. Effectively "committing" data involves two steps: 1. Write a file to S3 2. "Commit" that file to the meta…
Then the order is on batch level? Say batch1 is committed with a smaller timestamp than batch2, then all messages in batch1 are considered prior/earlier than any message in batch2?
Re: Kafka is dead, long live Kafka
#226Earlier quoted context omitted.
Kafka is less message queue (rabbitmq, sqs) and more ordered stream/write ahead log (ala kinesis) > The first is that because of the nature of message queues, they're either empty or full. ... Wat? > that is it's a true "fire and forget" kind of request and direct response time is not a concern, then what does the queue gain you over simply posting it to a database table? Performance is why. The fire and forget aspec…
> > The first is that because of the nature of message queues, they're either empty or full. > ... Wat? I interpreted it as "they are either trending towards empty or full". The statement doesn't seem well thought through. That might be true (either empty or full) most of the time (maybe) _if you squint_, but the entire point of the Message Queue is to provide buffering from the transient state (somewhere between emp…
Longer SLAs mean it's also easier to hit those SLAs. Giving the clients realistic SLAs is super important.
Re: Kafka is dead, long live Kafka
#227I think the focus on the $/GB is wrong. The main goal of Kafka is providing certain processing guarantees and connecting consumers with producers of events. What Kafka is not for is dealing with enormous number of events or enormous amount of data. Not that it is particularly slow at it (actually, quite fast) but I see a lot of people needlessly try to push more data and events than they need and then complain they h…
Re: Kafka is dead, long live Kafka
#228> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…
To me a technology company is not just a company that uses tech (every company does that) but one whose core value proposition is fundamentally technical. And I think most serious companies doing that have a need for highly available data storage, for which Kafka is the least bad option. What are the alternatives? Cassandra is just as operationally complex and harder to fit your dataflow into. The various efforts to…
(But also of course lots of applications are also fine with the availability that you get from fate-sharing with a single db server)
Re: Kafka is dead, long live Kafka
#229Earlier quoted context omitted.
We standardized on Clickhouse for everything. (With its own set of surprising and/or horrifying ops issues.) But at least it is a proper high-load, high-availablity solution, unlike Kafka, Cassandra, et al.
> We standardized on Clickhouse for everything. (With its own set of surprising and/or horrifying ops issues.) Clickhouse I admittedly haven't personally seen quite as much operational unpleasantness as Greenplum or Galera, but at this point I'm dubious of anything in that bucket. > But at least it is a proper high-load, high-availablity solution, unlike Kafka, Cassandra, et al. What went wrong with those for you? In…
Development is very active, and some features are experimental. One of the common mistakes is to use latest releases for production environments - you will certainly find odd bugs on specific usage scenarios. Stay away from the bleeding edge and you're fine. Clustering (table replication and sharding of queries) is also a sort-of can of worms by itself, and requires good knowledge of your workload and your data structure to understand all the tradeoffs. Thing is, when designing from scratch, you can often design in such a way where you don't need (clustered) table replication or sharding - again, this also has a learning curve, for both devs and devops.
You can easily spin it on a VM or on your laptop, load a dataset and see for yourself how powerful ClickHouse can be. Honestly, just the data compression alone is good enough to save a s**load of money on storage on an enterprise, compared to most solutions. Couple this with tiered storage - your hot data is eg. in ssd, your historical data is stored on s3, and rotation is done automatically, plus automated ingestion from kafka, and you have a data warehousing system at a fraction of the price of many common alternatives.
Re: Kafka is dead, long live Kafka
#230Earlier quoted context omitted.
What do you use instead? Polling APIs? A queue instead of an event stream? Event based architectures definitely add infrastructural overhead, but are positive at a certain scale and/or architectural complexity (multiple decoupled subscribers).
We use a table called "Messages" in our SQL Server database. Everyone talks to the same database. Turns out we don't really need to push extreme message rates or meet aggressive single-digit millisecond budgets, so this works out well in practice. It is also the easiest thing on earth to develop & debug, because you can monitor the table/log and instantly understand the state of the whole system and how it got there.…
https://learn.microsoft.com/en-us/sql/database-engine/config...