Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Publishing with Apache Kafka at The New York Times

confluent.io

Post reply on HN
Print view View original

41–50 of 157 posts

Re: Publishing with Apache Kafka at The New York Times

#41

Post by pcsanwald »

dswalter wrote: post #35
jdcarter wrote: post #14

FWIW, the article mentions the book "Designing Data-Intensive Applications" by Martin Kleppmann. I wanted to throw out my own endorsement for the book, it's been instrumental in helping me design my own fairly intensive data pipeline.

It's such a wonderful book. Reading it pushed me from thinking in terms of what I had worked with to building systems based on what was needed. I cannot recommend it highly enough, for pretty much anyone in {frontend, backend, data science, etc}.

I third this recommendation. I've worked on a ton of data intensive applications on all kinds of stacks over the years, and this book gives you lessons learned as well as a very valuable historical perspective on relational databases that is missing from a lot of the popular literature today.

Re: Publishing with Apache Kafka at The New York Times

#42

Post by pcsanwald »

BenoitEssiambre wrote: post #40
toomim wrote: post #36

> Traditionally, databases have been used as the source of truth ... [but] can be difficult to manage in the long run. First, it’s often tricky to change the schema of a database. Adding and removing fields is not too hard, but more fundamental schema changes can be difficult to organize without downtime. This argument sounds self-contradicting. Kafka doesn't let you change its schema at all! At least postgres gives…

I tend to be the one arguing this, to stick to postgres for most things but even I will admit it does depend on scale. I'm not sure what the NYT requirements are but from my understanding of Kafka, its persistent redundant distributed queues scale automatically horizontally across machines to support colossal amounts of data. It's possible that they had difficulty fitting everything in a postgres instance.

The author states it's about 100GB of data.

Re: Publishing with Apache Kafka at The New York Times

#43

Post by miguno »

scaryclam wrote: post #13
bognition wrote: post #6

Earlier quoted context omitted.

OOC what makes a RDBMS more durable that a Kafka? Both of them are systems for representing data on disk. I'd love to hear why one representation system is better at disaster recovery than another.

From what I can tell, Kafka isn't designed for long term data storage. RDBMS systems are designed for this. Kafka is more for streaming data and events, so I'd probably be uncomfortable assuming that it won't do something and "tidy up" my very old data at some point in the future. Since it's supposed to do this from time to time out of the box, you'd have to be very careful not to let anyone tweak the custom config t…

Kafka can perfectly keep your data around forever. The only limitation is available disk space (and databases have the same limitation). I'm not implying that it is always the best idea to use Kafka as a long-term storage solution, but likewise a database isn't the silver bullet here either.

> so I'd probably be uncomfortable assuming that it won't do something and "tidy up" my very old data at some point in the future

Kafka doesn't "tidy up" your data unless it is configured to do so. What's true is that, by default, Kafka will keep your data around for a week "only" but that's a configuration. And most people change it to whatever fits their use case (some lower it to a few hours, some increase it to months or years; others configure it to keep data around forever, typically in combination with Kafka's so-called "log compaction" functionality).

> Since it's supposed to do this from time to time out of the box, you'd have to be very careful not to let anyone tweak the custom config to revert back to this behaviour. RDBMS won't delete things unless you tell it to more explicitly.

The DBAs I worked with would now say "Hold my beer..." ;-)

> While I think storing things in Kafka is fine generally, there's no way I'd not have a more perminant store of the data somewhere so that I can recreate the Kafka data store if I need to.

What's interesting is that more and more users (from my experience) are actually beginning to treat Kafka as the source of truth, and rather recreate other data stores -- like RDBMS, Elastic indexes, etc. -- from it. If you like RDBMS, you can think of Kafka as the DB's transaction log.

IMHO a lot of these discussion is about personal preferences, the situation that you are in (cough legacy cough), team skills, etc. There are often good reasons to use Kafka rather than RDBMS (in this context) but also vice versa, or completely different technologies of course (like blob stores, e.g. S3).

Re: Publishing with Apache Kafka at The New York Times

#44

Post by theossuary »

BenoitEssiambre wrote: post #40
toomim wrote: post #36

> Traditionally, databases have been used as the source of truth ... [but] can be difficult to manage in the long run. First, it’s often tricky to change the schema of a database. Adding and removing fields is not too hard, but more fundamental schema changes can be difficult to organize without downtime. This argument sounds self-contradicting. Kafka doesn't let you change its schema at all! At least postgres gives…

I tend to be the one arguing this, to stick to postgres for most things but even I will admit it does depend on scale. I'm not sure what the NYT requirements are but from my understanding of Kafka, its persistent redundant distributed queues scale automatically horizontally across machines to support colossal amounts of data. It's possible that they had difficulty fitting everything in a postgres instance.

See, that's where I'm confused. I'm no Kafka expert, but they say they use a "single-partition topic" which I believe means the only way they can replicate the data is by replicating the entire log, they can't shard because it's a single partition. The reasoning behind this is because Kafka doesn't support ordering between partitions.

Also I've never thought of Kafka as a persistent data storage solution, it's interesting Confluent is supporting Kafka being used in this way.

Re: Publishing with Apache Kafka at The New York Times

#45

Post by knowtheory »

pm90 wrote: post #4

Excellent, well written article. The key take away seems to be that instead of an temporary event stream log, since the number of news articles (and associated assets) is finite and cannot explode, they store all the "logs" forever (I'm using the term log as is defined in the article, as a unit of a time-ordered data structure). I wonder if NYT can help other news websites by making their code open source? I'm a huge…

> I wonder if NYT can help other news websites by making their code open source?

Hey! I, and a number of other news nerds have been encouraging FOSS for the past decade or so. And in fact a number of major open source projects have come out of news related projects, including Django, Backbone.js/Underscore.js, Rich Harris's work on Svelt.js, and a whole lot more.

Most often the problem with local news organizations are operational constraints. The news biz has seen a huge downturn over this same period of time. Most orgs, both on the reporting side and on the tech side are super strapped for people-time.

It's not enough to have FOSS software, you also have to have folks doing devops and maintaining systems often at below-market salaries.

Re: Publishing with Apache Kafka at The New York Times

#46

Post by teej »

jdcarter wrote: post #14

FWIW, the article mentions the book "Designing Data-Intensive Applications" by Martin Kleppmann. I wanted to throw out my own endorsement for the book, it's been instrumental in helping me design my own fairly intensive data pipeline.

Dear HN reader - if you're not quite ready to buy the book, take a listen to this episode of Software Engineering Daily (https://softwareengineeringdaily.com/2017/05/02/data-intensi...). It will give you a sense of what Martin Kleppmann is all about and how he thinks about problems. I ordered my copy of "Designing Data-Intensive Applications" after listening to this episode.

Re: Publishing with Apache Kafka at The New York Times

#47

Post by cturner »

amenod wrote: post #21
cturner wrote: post #20

This is a flawed architecture. It will work at release, but it will be difficult to manoeuvre with, and they will grow to hate it. As your business changes, your data changes. Imagine if on day one, they had one author per article. On day 1000, they change this to be a list of authors. Kafka messages are immutable. Each of those green boxes on the right hand side of the first diagram will need to have special-case lo…

> Kafka messages are immutable. Each of those green boxes on the right hand side of the first diagram will need to have special-case logic to unpack the kafka stream, with knowledge of its changes (up until 17 May 2017, treat the data like this, but between then and 19 May 2017 do x, and after that do y). I respectfully disagree. The genius of this approach is that you can make the same transformation on the original…

"Voila - you only have y."

Thanks. My mental model had a long-lived channel, but I follow your explanation.

Re: Publishing with Apache Kafka at The New York Times

#48

Post by rhizome »

_lex wrote: post #10
cookiecaper wrote: post #5

>We need the log to retain all events forever, otherwise it is not possible to recreate a data store from scratch. SIGH . Cue the facepalm, head in hands, etc. I'm not going to get into a big thing here. But if you find yourself saying "I need to keep this thing forever no matter what" and then you try to use something that even entertains the notion of automatic eviction/deletion semantics as the system of record, y…

I think the POV he's taken is that the kafka stream is the one true datasource (for all time), with all other dbs being derivatives. This insane strategy seems to be over engineered to get around db migrations... though i'm sure the event stream will also change over time, and he'll have to write migration-like code anyways.

If datasource engineers are for some reason implementing "ontogeny recapitulates phylogeny," they won't be creating migration-like code, they'll be writing ETL filters. Break out your Members Only jackets!

Re: Publishing with Apache Kafka at The New York Times

#49

Post by LgWoodenBadger »

cookiecaper wrote: post #28
skybrian wrote: post #25

Earlier quoted context omitted.

Well, except that Kafka doesn't automatically delete stuff?

https://kafka.apache.org/documentation/ Name: cleanup.policy Description: A string that is either "delete" or "compact". This string designates the retention policy to use on old log segments. The default policy ("delete") will discard old segments when their retention time or size limit has been reached. ??? How is this not automatic deletion of stuff? I don't have to worry about someone setting a "delete all data o…

You may not have to worry about somebody setting a "delete all data older and/or bigger than Y or Z" but you have to worry about someone running "DELETE FROM table" without a WHERE clause. Which is easier to prevent? The one that can be done through the same mechanism as non-destructive queries? Or the one that can only be modified through a file-system configuration, completely separate from its API?

Regardless, it's a different paradigm with different "don't do that" behaviors that you need to know about.

In Kafka, if you want the persistent, append-only, write-ahead log to not delete stuff, then configure the retention period to keep things forever.

Re: Publishing with Apache Kafka at The New York Times

#50

Post by LgWoodenBadger »

_lex wrote: post #10
cookiecaper wrote: post #5

>We need the log to retain all events forever, otherwise it is not possible to recreate a data store from scratch. SIGH . Cue the facepalm, head in hands, etc. I'm not going to get into a big thing here. But if you find yourself saying "I need to keep this thing forever no matter what" and then you try to use something that even entertains the notion of automatic eviction/deletion semantics as the system of record, y…

I think the POV he's taken is that the kafka stream is the one true datasource (for all time), with all other dbs being derivatives. This insane strategy seems to be over engineered to get around db migrations... though i'm sure the event stream will also change over time, and he'll have to write migration-like code anyways.

There's not just one reason for it.

Kafka lets dependent consumers transform that data into whatever model is appropriate for their use case. There's no one-size-fits-all E-R model for data for all use cases.

It's not just working around db migrations. It's also providing you with the ability to model the data as many ways as you require. How else would you do it, materialized views? How often are they materialized? Regular views? How performant are they?

There are many benefits to this approach, and a lot of them require a different way of thinking. It's a different paradigm.

Post reply on HN