Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Publishing with Apache Kafka at The New York Times

confluent.io

Post reply on HN
Print view View original

21–30 of 157 posts

Re: Publishing with Apache Kafka at The New York Times

#21

Post by amenod »

cturner wrote: post #20

This is a flawed architecture. It will work at release, but it will be difficult to manoeuvre with, and they will grow to hate it. As your business changes, your data changes. Imagine if on day one, they had one author per article. On day 1000, they change this to be a list of authors. Kafka messages are immutable. Each of those green boxes on the right hand side of the first diagram will need to have special-case lo…

> Kafka messages are immutable. Each of those green boxes on the right hand side of the first diagram will need to have special-case logic to unpack the kafka stream, with knowledge of its changes (up until 17 May 2017, treat the data like this, but between then and 19 May 2017 do x, and after that do y).

I respectfully disagree. The genius of this approach is that you can make the same transformation on the original Kafka stream to change its schema and prepare the new feed. Once you are satisfied with the results and you have switched all subscribers to the new feed, just turn off the old one. Voila - you only have y.

> This is a rare case where use of XML makes sense.

Sorry, but no. Just no.

Re: Publishing with Apache Kafka at The New York Times

#22

Post by [deleted] »

scaryclam wrote: post #13
bognition wrote: post #6

Earlier quoted context omitted.

OOC what makes a RDBMS more durable that a Kafka? Both of them are systems for representing data on disk. I'd love to hear why one representation system is better at disaster recovery than another.

From what I can tell, Kafka isn't designed for long term data storage. RDBMS systems are designed for this. Kafka is more for streaming data and events, so I'd probably be uncomfortable assuming that it won't do something and "tidy up" my very old data at some point in the future. Since it's supposed to do this from time to time out of the box, you'd have to be very careful not to let anyone tweak the custom config t…

[deleted]

Re: Publishing with Apache Kafka at The New York Times

#23

Post by skybrian »

cturner wrote: post #20

This is a flawed architecture. It will work at release, but it will be difficult to manoeuvre with, and they will grow to hate it. As your business changes, your data changes. Imagine if on day one, they had one author per article. On day 1000, they change this to be a list of authors. Kafka messages are immutable. Each of those green boxes on the right hand side of the first diagram will need to have special-case lo…

They're using protobufs, which seem just about as flexible as XML as far as schema updates are concerned and are considerably less ambiguous. So I don't see how XML would help?

Re: Publishing with Apache Kafka at The New York Times

#25

Post by skybrian »

cookiecaper wrote: post #18
_m8fo wrote: post #11

I wonder how much of this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied. Are there any articles that supplement this that explain how much business value is added/lost by the existence/removal of these kind of features? In the case of NYT I suspect its popularity is maintained because of the perception (real or not) of high quality journali…

>I wonder how much of this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied. That's a ton of it. Like it or not, publishing a digital newspaper is not a hard or unsolved problem; it's one of the web's core competencies. If you hire people who want to build cool stuff to supervise a CMS, well, you get this kind of outcome. The raw cost is under…

Well, except that Kafka doesn't automatically delete stuff?

Re: Publishing with Apache Kafka at The New York Times

#26

Post by feelin_googley »

_m8fo wrote: post #11

I wonder how much of this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied. Are there any articles that supplement this that explain how much business value is added/lost by the existence/removal of these kind of features? In the case of NYT I suspect its popularity is maintained because of the perception (real or not) of high quality journali…

"I wonder how much this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied."

Do you mean only the developers (==very smart people) hired by the NYT or those hired at other companies, too?

Re: Publishing with Apache Kafka at The New York Times

#27

Post by 1wheel »

_m8fo wrote: post #11

I wonder how much of this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied. Are there any articles that supplement this that explain how much business value is added/lost by the existence/removal of these kind of features? In the case of NYT I suspect its popularity is maintained because of the perception (real or not) of high quality journali…

> How much would be lost if NYT was just implemented as text articles that are cached and styled with some CSS.

https://www.nytimes.com/interactive/2016/12/28/us/year-in-in...

Re: Publishing with Apache Kafka at The New York Times

#28

Post by cookiecaper »

skybrian wrote: post #25
cookiecaper wrote: post #18

Earlier quoted context omitted.

>I wonder how much of this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied. That's a ton of it. Like it or not, publishing a digital newspaper is not a hard or unsolved problem; it's one of the web's core competencies. If you hire people who want to build cool stuff to supervise a CMS, well, you get this kind of outcome. The raw cost is under…

Well, except that Kafka doesn't automatically delete stuff?

https://kafka.apache.org/documentation/

Name: cleanup.policy

Description: A string that is either "delete" or "compact". This string designates the retention policy to use on old log segments. The default policy ("delete") will discard old segments when their retention time or size limit has been reached.

???

How is this not automatic deletion of stuff? I don't have to worry about someone setting a "delete all data older and/or bigger than Y or Z" flag on a RDBMS, because such a flag doesn't exist.

---

EDIT: Since my other post got a few downvotes, and my account has been sanctioned in the past for posting similar thoughts about tech-hipsters diving head-first into Kubernetes without understanding any of the effects of doing so, I can no longer post replies for a period of some hours.

As such, I will copy my reply to child comment from snuxoll below, which I was not allowed to post, and then log out for a while.

Kafka devs, seek investment from YC, you may get them to sanction accounts that are too critical of the inappropriate adoption of your project, as they've done to me for criticizing Docker/k8s.

reply:

>Log compaction is generally what you want, which will preserve the most recent message for every key in a topic. Event streams expanding boundlessly is something very few will ever need or want, so you'll toss messages into an "event" topic of some kind, apply the event to the most recent entry in the "model" topic and store a new version (which will be kept after log compaction, the original messages can be pruned after you need to free up storage).

Yeah, I didn't explicitly detail the compaction option because it's still potentially deleting huge amounts of data. It promises to keep the latest state, which, yes, is better than deleting everything. But why not just not delete anything? The article says they're working with 100GB of data. There are VIDEO GAMES that get downloaded to consumer hard disks that occupy more than that (also not really excusably, but still).

I understand why Kafka provides these semantics and I think it's great that they do. The issue is that developers choose to ignore them, and seriously imperil their data's well-being by doing so.

Re: Publishing with Apache Kafka at The New York Times

#29

Post by skybrian »

cookiecaper wrote: post #12
bognition wrote: post #6

Earlier quoted context omitted.

OOC what makes a RDBMS more durable that a Kafka? Both of them are systems for representing data on disk. I'd love to hear why one representation system is better at disaster recovery than another.

In Postgres, I never have to worry that the server will be accidentally loaded with `retention.bytes` or `retention.days` set too low and, as a result, choose to delete everything in the database, generating a wholly artificial "disaster" that can result in long periods of disruption or downtime (at a minimum; worst case is permanent data loss). It is true that someone could issue `DROP DATABASE`, `rm -rf` the filesy…

So, remove the flag that you'll never use and recompile?

I'm not sure this is worse than using a Unix box with a working "rm" command.

Post reply on HN