Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Publishing with Apache Kafka at The New York Times

confluent.io

Post reply on HN
Print view View original

131–140 of 157 posts

Re: Publishing with Apache Kafka at The New York Times

#131

Post by qaq »

elnygren wrote: post #89
toomim wrote: post #36

> Traditionally, databases have been used as the source of truth ... [but] can be difficult to manage in the long run. First, it’s often tricky to change the schema of a database. Adding and removing fields is not too hard, but more fundamental schema changes can be difficult to organize without downtime. This argument sounds self-contradicting. Kafka doesn't let you change its schema at all! At least postgres gives…

Postgres could in fact be used here by creating an append-only table similar to: id, data (JSON field) However, Postgres doesn't have good support for creating derived append-only logs (=streams) from that table. Kafka has Kafka Streams and KafkaSQL. And producer+consumer APIs that are a good fit for NYTs use case. > Am I missing something? Remember that in addition to schema changes, NYT also wants to avoid row chan…

However, Postgres doesn't have good support for creating derived append-only logs ???? A one line trigger will give you derived append only log that is transactionally consistent.

Re: Publishing with Apache Kafka at The New York Times

#132

Post by qaq »

BenoitEssiambre wrote: post #40
toomim wrote: post #36

> Traditionally, databases have been used as the source of truth ... [but] can be difficult to manage in the long run. First, it’s often tricky to change the schema of a database. Adding and removing fields is not too hard, but more fundamental schema changes can be difficult to organize without downtime. This argument sounds self-contradicting. Kafka doesn't let you change its schema at all! At least postgres gives…

I tend to be the one arguing this, to stick to postgres for most things but even I will admit it does depend on scale. I'm not sure what the NYT requirements are but from my understanding of Kafka, its persistent redundant distributed queues scale automatically horizontally across machines to support colossal amounts of data. It's possible that they had difficulty fitting everything in a postgres instance.

They are storing 100GB :). You can do ok with 5TB +/- on something as crappy as RDS.

Re: Publishing with Apache Kafka at The New York Times

#133

Post by elnygren »

qaq wrote: post #131
elnygren wrote: post #89

Earlier quoted context omitted.

Postgres could in fact be used here by creating an append-only table similar to: id, data (JSON field) However, Postgres doesn't have good support for creating derived append-only logs (=streams) from that table. Kafka has Kafka Streams and KafkaSQL. And producer+consumer APIs that are a good fit for NYTs use case. > Am I missing something? Remember that in addition to schema changes, NYT also wants to avoid row chan…

However, Postgres doesn't have good support for creating derived append-only logs ???? A one line trigger will give you derived append only log that is transactionally consistent.

so how does that one-liner transform data from log A to log B in realtime?

are talking about creating a new table or materialised views or stored procedures or...?

Re: Publishing with Apache Kafka at The New York Times

#134

Post by qaq »

elnygren wrote: post #133
qaq wrote: post #131

Earlier quoted context omitted.

However, Postgres doesn't have good support for creating derived append-only logs ???? A one line trigger will give you derived append only log that is transactionally consistent.

so how does that one-liner transform data from log A to log B in realtime? are talking about creating a new table or materialised views or stored procedures or...?

In your post you are proposing to have a table be equivalent to a kafka topic. So derived "log"/topic can be another table updated with a trigger.

Re: Publishing with Apache Kafka at The New York Times

#135

Post by sunnykgupta »

jdcarter wrote: post #14

FWIW, the article mentions the book "Designing Data-Intensive Applications" by Martin Kleppmann. I wanted to throw out my own endorsement for the book, it's been instrumental in helping me design my own fairly intensive data pipeline.

+1 for DDIA by Kleppmann!

Re: Publishing with Apache Kafka at The New York Times

#136

Post by tlberglund »

cookiecaper wrote: post #107
tlberglund wrote: post #98

Earlier quoted context omitted.

It may be unfair to describe setting a documented configuration parameter as "fiddling." Retention is seven days by default. It is trivial to set it to arbitrarily long periods of time. To my knowledge, this functionality isn't really in question. Whether logs are a good unifying abstraction on which to build systems is in dispute among reasonable people, but whether Kafka randomly deletes stuff is not. :)

I don't claim that Kafka randomly deletes things. Just that it automatically does so. The danger is not that Kafka will choose not to respect the configuration value. It is that the default setting will find a way to creep back in without the admin noticing it, and then a quick reboot, maybe even an unplanned one caused by a power trip or a kernel crash, will be sayonara to the system of record. Sure, there are backu…

Agreed that this is a newer architectural paradigm and a younger product. Of that there is no doubt, and there is always risk there. Also return, of course; someone had to deploy an RDBMs for the first time too—and everyone is glad they did.

But I still don't follow the argument. If the eviction model is a loud, clear signal that this is the wrong solution, why isn't the mutability of RDBMS data the same sort of signal? Claiming that the presence of a DELETE statement in SQL rules out relational databases as durable data stores would not get me too far. And nor should it!

You are 100% right that this is a new approach. You are also right that it is possible to make configuration errors that will break the system. But this is true of all nontrivial systems. At the end of all of this, we still have a very interesting sequence of events (all NYT content ever) stored in an immutable log. This seems reasonable. Maybe the NYT team is blazing a trail, it's not prima facie a crazy one. :)

Re: Publishing with Apache Kafka at The New York Times

#137

Post by thinkMOAR »

pm90 wrote: post #4

Excellent, well written article. The key take away seems to be that instead of an temporary event stream log, since the number of news articles (and associated assets) is finite and cannot explode, they store all the "logs" forever (I'm using the term log as is defined in the article, as a unit of a time-ordered data structure). I wonder if NYT can help other news websites by making their code open source? I'm a huge…

It would make me very sad if this would be the case, that every newspaper is going to be running the same kind of software stack. It's then 1 step away from just having only 1 news paper in total.

Diversity, choice, innovation, etc up to different reporters reporting on the same story with their point of view, will all disappear if all newspapers are going to run the same software stack.

Re: Publishing with Apache Kafka at The New York Times

#138

Post by miguno »

cookiecaper wrote: post #90
miguno wrote: post #43

Earlier quoted context omitted.

Kafka can perfectly keep your data around forever. The only limitation is available disk space (and databases have the same limitation). I'm not implying that it is always the best idea to use Kafka as a long-term storage solution, but likewise a database isn't the silver bullet here either. > so I'd probably be uncomfortable assuming that it won't do something and "tidy up" my very old data at some point in the futu…

>Kafka can perfectly keep your data around forever. In the sense that you can fiddle with it to the point where it doesn't purge things automatically, sure. But RDBMS provides more than the promise that it won't delete your data after a set period of time. If that was all we needed, any filesystem from the last 3 decades would serve fine as a "permanent datastore". MySQL has gone through a lot of grief to get to the…

> No offense, but this just shows your ignorance of the functionality that you're leaving on the table by considering SQL a cough-legacy-cough solution.

No reason to get ad-hominem.

When I said cough-legacy-cough, I was referring to the situation you happen to be in when e.g. taking on a new project. If, for example, your company has been fully committed to Postgres since several years, your chances to change this and move to a different architecture (or even sticking to the same architecture but using a different RDBMS like MySQL) are pretty low.

Re: Publishing with Apache Kafka at The New York Times

#139

Post by awinder »

cookiecaper wrote: post #18
_m8fo wrote: post #11

I wonder how much of this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied. Are there any articles that supplement this that explain how much business value is added/lost by the existence/removal of these kind of features? In the case of NYT I suspect its popularity is maintained because of the perception (real or not) of high quality journali…

>I wonder how much of this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied. That's a ton of it. Like it or not, publishing a digital newspaper is not a hard or unsolved problem; it's one of the web's core competencies. If you hire people who want to build cool stuff to supervise a CMS, well, you get this kind of outcome. The raw cost is under…

There are non-negligible problems in the news space like:

  1. Supporting full-text search for a fair number of concurrent users
  2. Availability of the system with minimal downtime
  3. Scalability within the day and year, traffic patterns around e.g., breaking news events will far surpass 2AM traffic
  4. Notifications
I could go on and on but honestly, it's just a tone-deaf response.

Parting pot-shot: "No one is going to admit to their boss that the reason a worldwide news organization can't publish any stories is because their one postgres master node went down, or is waiting on a state transfer to a fallback master"

Re: Publishing with Apache Kafka at The New York Times

#140

Post by neeleshs »

manigandham wrote: post #82

The very definition of over-engineered. This is just event-sourcing turned into a marketing article for Kafka. It doesn't really matter what the "source of truth" system is, although Kafka doesn't seem quite as mature/stable enough for that. With such little data, they can push into a nice graph database instead, run it entirely in memory and meet 10x any demand they'll ever see along with all the query types they'll…

I was about to comment that while any database can serve as a log replay and may even be feasible for these volumes, scaling it at high volume later would be extremely hard - I have experience with this in a multi-petabyte set up and its a nightmare using an RDBMS.

But, the article says "In Apache Kafka, the Monolog is implemented as a single-partition topic" - losing all the goodness Kafka provides around scaling. This setup now is no better than an RDBMS with master/slave replication.

Post reply on HN