Publishing with Apache Kafka at The New York Times
1–10 of 157 posts
Re: Publishing with Apache Kafka at The New York Times
#2[deleted]
Re: Publishing with Apache Kafka at The New York Times
#3Interesting, we use rabbitmq instead of kafka and we have a re-indexation system... not sure if it's more complex for what I see.
Re: Publishing with Apache Kafka at The New York Times
#4Excellent, well written article. The key take away seems to be that instead of an temporary event stream log, since the number of news articles (and associated assets) is finite and cannot explode, they store all the "logs" forever (I'm using the term log as is defined in the article, as a unit of a time-ordered data structure).
I wonder if NYT can help other news websites by making their code open source? I'm a huge fan of NYT and their jump to digital has just been amazing. However, I would also like my local newspaper (which covers more regional news) to be able to serve quality digital content.
Re: Publishing with Apache Kafka at The New York Times
#5>We need the log to retain all events forever, otherwise it is not possible to recreate a data store from scratch.
SIGH. Cue the facepalm, head in hands, etc.
I'm not going to get into a big thing here. But if you find yourself saying "I need to keep this thing forever no matter what" and then you try to use something that even entertains the notion of automatic eviction/deletion semantics as the system of record, you're doing it wrong.
Not to burst the bubble of the techno-hipsters, but Kafka is "durable" relative to message brokers like RabbitMQ, not relative to a system actually designed to store decades of mission-critical data. Those systems are called "RDBMS".
Elsewhere in the article he says that they have less than 100GB of data and that it's mostly text. This is massive overarchitecture that isn't even covering the basic flanks that it thinks it is, such as data permanence.
I would really like to read the article that discusses why Postgres or MySQL couldn't have served this purpose equally well.
Re: Publishing with Apache Kafka at The New York Times
#6>We need the log to retain all events forever, otherwise it is not possible to recreate a data store from scratch. SIGH . Cue the facepalm, head in hands, etc. I'm not going to get into a big thing here. But if you find yourself saying "I need to keep this thing forever no matter what" and then you try to use something that even entertains the notion of automatic eviction/deletion semantics as the system of record, y…
OOC what makes a RDBMS more durable that a Kafka? Both of them are systems for representing data on disk. I'd love to hear why one representation system is better at disaster recovery than another.
Re: Publishing with Apache Kafka at The New York Times
#7>We need the log to retain all events forever, otherwise it is not possible to recreate a data store from scratch. SIGH . Cue the facepalm, head in hands, etc. I'm not going to get into a big thing here. But if you find yourself saying "I need to keep this thing forever no matter what" and then you try to use something that even entertains the notion of automatic eviction/deletion semantics as the system of record, y…
OOC what makes a RDBMS more durable that a Kafka? Both of them are systems for representing data on disk. I'd love to hear why one representation system is better at disaster recovery than another.
That's a bit of an oversimplification. Production grade RDBMS systems have far more guard rails, testing and work put in to them than Kafka. It's relatively straight forward to lose data in Kafka, I've done it (its usually control plane bugs, not data plane).
Re: Publishing with Apache Kafka at The New York Times
#8>We need the log to retain all events forever, otherwise it is not possible to recreate a data store from scratch. SIGH . Cue the facepalm, head in hands, etc. I'm not going to get into a big thing here. But if you find yourself saying "I need to keep this thing forever no matter what" and then you try to use something that even entertains the notion of automatic eviction/deletion semantics as the system of record, y…
OOC what makes a RDBMS more durable that a Kafka? Both of them are systems for representing data on disk. I'd love to hear why one representation system is better at disaster recovery than another.
GP is clueless. Kafka is in fact more resilient and much more performant than systems like MySQL. It's the difference between having one writer and many writers.
More importantly, adding yet another database to an enterprise that's already overrun databases is not going to fix anything. Systems like Kafka are deployed precisely because there are already many domain-specific databases at work and now there's a need to tie them together. You can't have a single database that works for all domains, for everything from elastic search text to billing. That's just dumb. So you have many databases and tie them together using a transactional event log.
BTW, the NYTimes, like virtually any mature enterprise, probably has a robust data warehouse and data retention strategy. I kind of doubt they're in the habit of losing data given their archives go back more than a century. But it's important to distinguish front-office transactional concerns (actually handing real-time requests from content producers and consumers) from back-office concerns (reporting, back-ups). They are very different domains as many enterprises are slowly discovering.
Re: Publishing with Apache Kafka at The New York Times
#9>We need the log to retain all events forever, otherwise it is not possible to recreate a data store from scratch. SIGH . Cue the facepalm, head in hands, etc. I'm not going to get into a big thing here. But if you find yourself saying "I need to keep this thing forever no matter what" and then you try to use something that even entertains the notion of automatic eviction/deletion semantics as the system of record, y…
I skimmed the article but I imagined they were using it as a secondary data store. I think they want to it to be durable in the sense that even if the events are already consumed they can still play them back to reindex elastic search (which is a thing you need to do periodically).
Re: Publishing with Apache Kafka at The New York Times
#10>We need the log to retain all events forever, otherwise it is not possible to recreate a data store from scratch. SIGH . Cue the facepalm, head in hands, etc. I'm not going to get into a big thing here. But if you find yourself saying "I need to keep this thing forever no matter what" and then you try to use something that even entertains the notion of automatic eviction/deletion semantics as the system of record, y…
I think the POV he's taken is that the kafka stream is the one true datasource (for all time), with all other dbs being derivatives. This insane strategy seems to be over engineered to get around db migrations... though i'm sure the event stream will also change over time, and he'll have to write migration-like code anyways.