111–120 of 157 posts
FWIW, the article mentions the book "Designing Data-Intensive Applications" by Martin Kleppmann. I wanted to throw out my own endorsement for the book, it's been instrumental in helping me design my own fairly intensive data pipeline.
Earlier quoted context omitted.
I'd say it isn't appropriate for permanent data storage because the individual brokers don't scale well with the amount of logs present. If you have hundreds of partitions, and millions of logs, then any operation dealing with the indexes (like an unclean startup) will take an extremely long amount of time. So your individual brokers are now down for an hour if they don't shut down cleanly. Which happens often (oom,…
Most of the use-cases you describe wouldn't be resolved by re-reading the entirety of the topics from the beginning of time. They'd resume from their last committed offset and continue where they left off. If you do have to catastrophically recover from the beginning of time, then sure you'd have a rough time. But that's true for any system that would have to do that. It's not Kafka-specific. Now if your consumer was…
> Traditionally, databases have been used as the source of truth ... [but] can be difficult to manage in the long run. First, it’s often tricky to change the schema of a database. Adding and removing fields is not too hard, but more fundamental schema changes can be difficult to organize without downtime. This argument sounds self-contradicting. Kafka doesn't let you change its schema at all! At least postgres gives…
The problem with an RDBMS is that the schema is retroactive -- every time you update the schema, both old and new data must validate against it. There's another approach, which is to also version the schema. Every record is conceptually a pair of [schema, data]. This puts a burden on the application -- every client must be able to understand old schemas -- but the benefits are considerable. The most trivial benefit i…
The very definition of over-engineered. This is just event-sourcing turned into a marketing article for Kafka. It doesn't really matter what the "source of truth" system is, although Kafka doesn't seem quite as mature/stable enough for that. With such little data, they can push into a nice graph database instead, run it entirely in memory and meet 10x any demand they'll ever see along with all the query types they'll…
Why would you reinvent the wheel? What makes you think that Kafka is unstable?
Earlier quoted context omitted.
Unless I'm mistaken, If I were to build out a simple event log represented by a relational DB, I have bottle necks when writing to it, and have lag in terms of processing the events, and if I were also pushing those events to a queue to hydrate aggregate snapshots I would have to have client logic to deal with duplicate events or not acking processed events etc? Intuitively, I guess kafka is "more realtime" and "more…
There's no reason not to make good use of Kafka or similar solutions. The issue is that people use it without understanding it. In this article, they say that Kafka is their system of record and their primary long-term storage. That's very silly.
Earlier quoted context omitted.
> Kafka messages are immutable. Each of those green boxes on the right hand side of the first diagram will need to have special-case logic to unpack the kafka stream, with knowledge of its changes (up until 17 May 2017, treat the data like this, but between then and 19 May 2017 do x, and after that do y). I respectfully disagree. The genius of this approach is that you can make the same transformation on the original…
I still don't understand the hatred around XML. Is it slightly verbose? Yes. Does it support lots of neat functionality that make it great for interoperating between systems, like validations and transformations? Yep. Sure, it's possible to go full architecture astronaut with it, but you can do that with pretty much any programming language. Meanwhile, I'm just sitting over here wondering whether my YAML file is supp…
Kafka logs however are solving a different problem. The mental model is different - they do not record state, but the whole history of transactions, which makes it trivial to change the schema if/when need arises. Saying that the schema should be thought in advance and shouldn't change is not realistic IMHO.
Earlier quoted context omitted.
You can use Kafka as the buffer/processing log before persisting to the database, but with such a small dataset it's just not necessary. It's a news publishing system, not high-frequency trading.
Well my point is that it's probably faster to get to production if I simply used Kafka _when modelling my work flow as an event processing system_ but it took them a year so I don't know now haha
FWIW, the article mentions the book "Designing Data-Intensive Applications" by Martin Kleppmann. I wanted to throw out my own endorsement for the book, it's been instrumental in helping me design my own fairly intensive data pipeline.
6 days to go in the kickstarter map https://www.kickstarter.com/projects/1407076797/a-map-of-the...
The very definition of over-engineered. This is just event-sourcing turned into a marketing article for Kafka. It doesn't really matter what the "source of truth" system is, although Kafka doesn't seem quite as mature/stable enough for that. With such little data, they can push into a nice graph database instead, run it entirely in memory and meet 10x any demand they'll ever see along with all the query types they'll…
Earlier quoted context omitted.
Through its various protocols Kafka topics can be configured to be guaranteed forwards, backwards, or bi-directionally compatible. That is to say: as flexible as XML or an RDBMS schema with long-term, format encoded, data that can explicitly support conflicting clients over time (as desired by the dev). Zero-impact, live, online, updates touching hundreds of active systems without issue... TBH most posters here have…
> Through its various protocols Kafka topics can be configured to be guaranteed forwards, backwards, or bi-directionally compatible. Sure but backwards and bi-directional compatibility inhibits the evolvability of schemas. Something as simple as adding a new required field for example is not backward compatible in avro. I understand why this is, and that in a huge database, it's not that simple in SQL either. But in…
Seriously: that would be a show stopper beyond show stoppers, and would make Kafka useless. Backwards and bi-directional compatibility are the specific facilities that enable safe schema evolution.
From the docs:
"Backward compatibility means that data encoded with an older schema can be read with a newer schema... Forward compatibility means that data encoded with a newer schema can be read with an older schema... Full compatibility means schemas are backward and forward compatible[:]...we can evolve the schemas in a fully compatible way: old data can be read with the new schema, and new data can also be read with the old schema."
A new required field _with a default value_ (important!), is backwards compatible. Same deal for hardcoded SQL updates: no default value, no backwards compatibility. Default value? Backwards compatible because clients that cannot provide a value get one, by default.
And, frankly, if you're thinking about database in the singular form then you're not thinking about an environment that really needs a schema evolution policy... tens and hundreds of parallel databases, active datawarehouse operations, multiple federated service gateways, real-time analytics, and huge batch systems for BigData, all acting in concert 24/7: that's the kind of environment that makes the "95%" simple jobs massively costly due to the logarithmic interaction of complexity when numerous systems are considered at once.
Without some kind of dataflow separation these intrinsic complexities trend towards intractable system configurations, hence the ESB boom and bust. Kafka is not for your applications. Kafka is for your Enterprise with many, many, applications.