Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Publishing with Apache Kafka at The New York Times

confluent.io

Post reply on HN
Print view View original

61–70 of 157 posts

Re: Publishing with Apache Kafka at The New York Times

#61

Post by kod »

toomim wrote: post #36

> Traditionally, databases have been used as the source of truth ... [but] can be difficult to manage in the long run. First, it’s often tricky to change the schema of a database. Adding and removing fields is not too hard, but more fundamental schema changes can be difficult to organize without downtime. This argument sounds self-contradicting. Kafka doesn't let you change its schema at all! At least postgres gives…

Kafka doesn't have a schema per message, messages are just bytes that you can serialize / deserialize however you want to. The article refers to using protobuf for messages, which does support adding fields.

If you're equating kafka topics with the idea of a schema, you can add topics.

Re: Publishing with Apache Kafka at The New York Times

#62

Post by robertlagrant »

amenod wrote: post #21
cturner wrote: post #20

This is a flawed architecture. It will work at release, but it will be difficult to manoeuvre with, and they will grow to hate it. As your business changes, your data changes. Imagine if on day one, they had one author per article. On day 1000, they change this to be a list of authors. Kafka messages are immutable. Each of those green boxes on the right hand side of the first diagram will need to have special-case lo…

> Kafka messages are immutable. Each of those green boxes on the right hand side of the first diagram will need to have special-case logic to unpack the kafka stream, with knowledge of its changes (up until 17 May 2017, treat the data like this, but between then and 19 May 2017 do x, and after that do y). I respectfully disagree. The genius of this approach is that you can make the same transformation on the original…

Not sure how just having a new API on top of a database doesn't achieve the same?

Re: Publishing with Apache Kafka at The New York Times

#63

Post by bonesss »

skybrian wrote: post #23
cturner wrote: post #20

This is a flawed architecture. It will work at release, but it will be difficult to manoeuvre with, and they will grow to hate it. As your business changes, your data changes. Imagine if on day one, they had one author per article. On day 1000, they change this to be a list of authors. Kafka messages are immutable. Each of those green boxes on the right hand side of the first diagram will need to have special-case lo…

They're using protobufs, which seem just about as flexible as XML as far as schema updates are concerned and are considerably less ambiguous. So I don't see how XML would help?

Through its various protocols Kafka topics can be configured to be guaranteed forwards, backwards, or bi-directionally compatible.

That is to say: as flexible as XML or an RDBMS schema with long-term, format encoded, data that can explicitly support conflicting clients over time (as desired by the dev). Zero-impact, live, online, updates touching hundreds of active systems without issue...

TBH most posters here have completely missed the forest for the trees. The point is not to avoid DB migrations. The point is to support hundreds of DB migrations in connected systems simultaneously with no schema-related down-time or centralized point of failure or intractable CAP challenges.

Trying to solve these kind of operation issues with an RDBMS in an Enterprise context is _literally_ the "big ball of mud" design pattern.

Kafka, warts and all, is an operational answer to how 1 client can feed 1 MM real-time connections, how massive resource unlimited batch systems can integrate with real-time feeds, how your data warehouse can keep growing without painful forced restructuring, and how data architects can mandate standards across multiple systems built by external teams with human sized budgets.

Data is part of it. Protocol, format guarantees, and loosely coupled systems are where the wins lie.

Re: Publishing with Apache Kafka at The New York Times

#64

Post by chuhlomin »

iooi wrote: post #60

> Because the topic is single-partition, it needs to be stored on a single disk, due to the way Kafka stores partitions. This is not a problem for us in practice, since all our content is text produced by humans — our total corpus right now is less than 100GB, and disks are growing bigger faster than our journalists can write. Before this line, the author mentions they also store images. There's no way that all their…

I think they mean image metadata (caption, paths to image files, credits, etc.), not the images itself.

Re: Publishing with Apache Kafka at The New York Times

#65

Post by kod »

iooi wrote: post #60

> Because the topic is single-partition, it needs to be stored on a single disk, due to the way Kafka stores partitions. This is not a problem for us in practice, since all our content is text produced by humans — our total corpus right now is less than 100GB, and disks are growing bigger faster than our journalists can write. Before this line, the author mentions they also store images. There's no way that all their…

More than likely they store references to images in kafka, with the actual image bytes being in a different store.

Re: Publishing with Apache Kafka at The New York Times

#69

Post by hugofirth »

jdcarter wrote: post #14

FWIW, the article mentions the book "Designing Data-Intensive Applications" by Martin Kleppmann. I wanted to throw out my own endorsement for the book, it's been instrumental in helping me design my own fairly intensive data pipeline.

An upvote was an insufficient form of agreement. Go buy this book people!

Re: Publishing with Apache Kafka at The New York Times

#70

Post by pram »

BenoitEssiambre wrote: post #40
toomim wrote: post #36

> Traditionally, databases have been used as the source of truth ... [but] can be difficult to manage in the long run. First, it’s often tricky to change the schema of a database. Adding and removing fields is not too hard, but more fundamental schema changes can be difficult to organize without downtime. This argument sounds self-contradicting. Kafka doesn't let you change its schema at all! At least postgres gives…

I tend to be the one arguing this, to stick to postgres for most things but even I will admit it does depend on scale. I'm not sure what the NYT requirements are but from my understanding of Kafka, its persistent redundant distributed queues scale automatically horizontally across machines to support colossal amounts of data. It's possible that they had difficulty fitting everything in a postgres instance.

Kafka by default is not persistent, the logs expire after 7 days. You can increase it on a per topic basis. It also doesn't scale automatically. If you have three replicas on a single partition topic, they will live on their assigned broker forever unless you manually reassign them. Adding new nodes does not kick off rebalancing of partitions. Its automatic cluster management is very primitive compared to something like elasticsearch.

For example, if you lose a broker, the replica will just be gone forever. Unless you replace the broker with the same id.

Post reply on HN