Resume driven architecture taken to extreme.
Publishing with Apache Kafka at The New York Times
101–110 of 157 posts
Re: Publishing with Apache Kafka at The New York Times
#102The very definition of over-engineered. This is just event-sourcing turned into a marketing article for Kafka. It doesn't really matter what the "source of truth" system is, although Kafka doesn't seem quite as mature/stable enough for that. With such little data, they can push into a nice graph database instead, run it entirely in memory and meet 10x any demand they'll ever see along with all the query types they'll…
Unless I'm mistaken, If I were to build out a simple event log represented by a relational DB, I have bottle necks when writing to it, and have lag in terms of processing the events, and if I were also pushing those events to a queue to hydrate aggregate snapshots I would have to have client logic to deal with duplicate events or not acking processed events etc? Intuitively, I guess kafka is "more realtime" and "more…
it's exact opposite the main cost is gc of dead rows in MVCC RDBMS since you are never deleting or updating rows performance will be very decent for writes.
Re: Publishing with Apache Kafka at The New York Times
#103The very definition of over-engineered. This is just event-sourcing turned into a marketing article for Kafka. It doesn't really matter what the "source of truth" system is, although Kafka doesn't seem quite as mature/stable enough for that. With such little data, they can push into a nice graph database instead, run it entirely in memory and meet 10x any demand they'll ever see along with all the query types they'll…
Unless I'm mistaken, If I were to build out a simple event log represented by a relational DB, I have bottle necks when writing to it, and have lag in terms of processing the events, and if I were also pushing those events to a queue to hydrate aggregate snapshots I would have to have client logic to deal with duplicate events or not acking processed events etc? Intuitively, I guess kafka is "more realtime" and "more…
You can use Kafka as the buffer/processing log before persisting to the database, but with such a small dataset it's just not necessary. It's a news publishing system, not high-frequency trading.
Re: Publishing with Apache Kafka at The New York Times
#104Earlier quoted context omitted.
Kafka can perfectly keep your data around forever. The only limitation is available disk space (and databases have the same limitation). I'm not implying that it is always the best idea to use Kafka as a long-term storage solution, but likewise a database isn't the silver bullet here either. > so I'd probably be uncomfortable assuming that it won't do something and "tidy up" my very old data at some point in the futu…
>Kafka can perfectly keep your data around forever. In the sense that you can fiddle with it to the point where it doesn't purge things automatically, sure. But RDBMS provides more than the promise that it won't delete your data after a set period of time. If that was all we needed, any filesystem from the last 3 decades would serve fine as a "permanent datastore". MySQL has gone through a lot of grief to get to the…
Kafka is not a messaging queue. It's a log.
The same underlying structure used within our RDBMS to provide all the guarantees Kafka provides.
As for logs and databases, they are duals.
https://www.confluent.io/blog/turning-the-database-inside-ou...
Re: Publishing with Apache Kafka at The New York Times
#105Earlier quoted context omitted.
You've really just laid out feelings rather than concrete technical reasons for why Kafka can't function as a permanent datastore. Can you point to specific design elements in Kafka that would lead you to conclude that it isn't suitable for permanent data storage? Also, Kafka doesn't "do" anything to your old data if you don't want it to. It's also open-source, so these behaviors can be verified.
I'd say it isn't appropriate for permanent data storage because the individual brokers don't scale well with the amount of logs present. If you have hundreds of partitions, and millions of logs, then any operation dealing with the indexes (like an unclean startup) will take an extremely long amount of time. So your individual brokers are now down for an hour if they don't shut down cleanly. Which happens often (oom,…
Most of the use-cases you describe wouldn't be resolved by re-reading the entirety of the topics from the beginning of time.
They'd resume from their last committed offset and continue where they left off.
If you do have to catastrophically recover from the beginning of time, then sure you'd have a rough time. But that's true for any system that would have to do that. It's not Kafka-specific.
Now if your consumer was entirely in-memory and made no use of persistent storage itself and it had to recover from the beginning of time, then I'd say that type of problem is your own architectural failure and I have yet to come across a tool, pattern, framework, or architecture that bullet-proofs your foot.
Re: Publishing with Apache Kafka at The New York Times
#106Earlier quoted context omitted.
Unless I'm mistaken, If I were to build out a simple event log represented by a relational DB, I have bottle necks when writing to it, and have lag in terms of processing the events, and if I were also pushing those events to a queue to hydrate aggregate snapshots I would have to have client logic to deal with duplicate events or not acking processed events etc? Intuitively, I guess kafka is "more realtime" and "more…
You can use Kafka as the buffer/processing log before persisting to the database, but with such a small dataset it's just not necessary. It's a news publishing system, not high-frequency trading.
Well my point is that it's probably faster to get to production if I simply used Kafka _when modelling my work flow as an event processing system_ but it took them a year so I don't know now haha
Re: Publishing with Apache Kafka at The New York Times
#107Earlier quoted context omitted.
>Kafka can perfectly keep your data around forever. In the sense that you can fiddle with it to the point where it doesn't purge things automatically, sure. But RDBMS provides more than the promise that it won't delete your data after a set period of time. If that was all we needed, any filesystem from the last 3 decades would serve fine as a "permanent datastore". MySQL has gone through a lot of grief to get to the…
It may be unfair to describe setting a documented configuration parameter as "fiddling." Retention is seven days by default. It is trivial to set it to arbitrarily long periods of time. To my knowledge, this functionality isn't really in question. Whether logs are a good unifying abstraction on which to build systems is in dispute among reasonable people, but whether Kafka randomly deletes stuff is not. :)
I don't claim that Kafka randomly deletes things. Just that it automatically does so.
The danger is not that Kafka will choose not to respect the configuration value. It is that the default setting will find a way to creep back in without the admin noticing it, and then a quick reboot, maybe even an unplanned one caused by a power trip or a kernel crash, will be sayonara to the system of record. Sure, there are backups, but who needs that aggravation? (p.s.: there probably aren't actually any workable backups)
In the RDBMS world, MySQL's automatic and silent truncation of VARCHARs down to the character limit of the column was seen as a sign of its badness. That demonstrates the difference in paradigm.
Anyway, the argument doesn't really hinge on whether or not there's an automatic eviction model in the software. It's just a clear, loud signal that the software is not really intended for long-term storage, and that you are, at best, entrusting decades of mission-critical data to a less-tested configuration on a young, maturing product. This should not be appealing by itself, and that's a very optimistic perspective on the choice to forgo the data integrity features provided by a traditional RDBMS.
Developers just cannot seem to grok that just because something appears to store data across server restarts does not mean it is necessarily a safe permanent parking spot. I'm not a DBA, and I've had my share of serious squabbles with them, but when this is what happens with unsupervised developers at the helm, it's hard not to be sympathetic to their aggressive, almost hostile, feelings around developer input into the data model.
Re: Publishing with Apache Kafka at The New York Times
#108This is a flawed architecture. It will work at release, but it will be difficult to manoeuvre with, and they will grow to hate it. As your business changes, your data changes. Imagine if on day one, they had one author per article. On day 1000, they change this to be a list of authors. Kafka messages are immutable. Each of those green boxes on the right hand side of the first diagram will need to have special-case lo…
> Kafka messages are immutable. Each of those green boxes on the right hand side of the first diagram will need to have special-case logic to unpack the kafka stream, with knowledge of its changes (up until 17 May 2017, treat the data like this, but between then and 19 May 2017 do x, and after that do y). I respectfully disagree. The genius of this approach is that you can make the same transformation on the original…
I still don't understand the hatred around XML. Is it slightly verbose? Yes. Does it support lots of neat functionality that make it great for interoperating between systems, like validations and transformations? Yep. Sure, it's possible to go full architecture astronaut with it, but you can do that with pretty much any programming language.
Meanwhile, I'm just sitting over here wondering whether my YAML file is supposed to have certain indents here or not, "-" or not, and trying to go figure out which magic incantation I need to get it to handle a multi-line string the way I'm expecting.
Re: Publishing with Apache Kafka at The New York Times
#109Earlier quoted context omitted.
You can use Kafka as the buffer/processing log before persisting to the database, but with such a small dataset it's just not necessary. It's a news publishing system, not high-frequency trading.
Well my point is that it's probably faster to get to production if I simply used Kafka _when modelling my work flow as an event processing system_ but it took them a year so I don't know now haha
[deleted]
Re: Publishing with Apache Kafka at The New York Times
#110Earlier quoted context omitted.
I skimmed the article but I imagined they were using it as a secondary data store. I think they want to it to be durable in the sense that even if the events are already consumed they can still play them back to reindex elastic search (which is a thing you need to do periodically).
> I think they want to it to be durable in the sense that even if the events are already consumed they can still play them back to reindex elastic search (which is a thing you need to do periodically). That (replaying if needed) is exactly what Kafka allows you to do, unless I misunderstood what you wrote.
No, you understand. Just not sure what failure mode would make kafka a bad store, especially if all its logs are created from other services.