Publishing with Apache Kafka at The New York Times • Hacker News Board

Re: Publishing with Apache Kafka at The New York Times

#121

Post by manigandham » Thu, Sep 07, 2017, 7:17 AM UTC

ma2rten wrote: post #114

manigandham wrote: post #82

The very definition of over-engineered. This is just event-sourcing turned into a marketing article for Kafka. It doesn't really matter what the "source of truth" system is, although Kafka doesn't seem quite as mature/stable enough for that. With such little data, they can push into a nice graph database instead, run it entirely in memory and meet 10x any demand they'll ever see along with all the query types they'll…

You could use any database, but Kafka has apis specifically for that. Why would you reinvent the wheel? What makes you think that Kafka is unstable?

Event sourcing is unnecessary here. What they want is a versioned history of their content with flexible schemas and arbitrary queries. Instead of using a strong fast graph database as a perfect fit, they chose to implement it poorly using Kafka which I see as much more time wasted on reinventing the wheel.

The dataset is small and all of the "very different" use-cases are just downstream apps that query a database. Why use Kafka to then materialize several different databases when a single graph database can serve all these downstream apps? Remember Kafka consumers themselves are just polling queries against a log.

A processing log is one thing but event-sourced source of truth in custom database logic in Kafka is borderline ridiculous. This project is more moving parts and less functionality while cleaning none of the existing mess. Effort would've been better spent consolidating all their systems instead (which they still have to do since these standard schemas need to be used somehow).

Re: Publishing with Apache Kafka at The New York Times

#122

Post by josephg » Thu, Sep 07, 2017, 7:20 AM UTC

puranjay wrote: post #119

manigandham wrote: post #82

The very definition of over-engineered. This is just event-sourcing turned into a marketing article for Kafka. It doesn't really matter what the "source of truth" system is, although Kafka doesn't seem quite as mature/stable enough for that. With such little data, they can push into a nice graph database instead, run it entirely in memory and meet 10x any demand they'll ever see along with all the query types they'll…

Reminds me of the NYT "Snowfall" story website that NYT spent $100k+ on producing, and some startup remade within a couple of days on Wordpress

That doesn't prove that the NYT's money was poorly spent. Decisions are always slow and expensive, and typing is quick. Copying an existing design is almost always cheaper than making the original.

And in many cases, copying is dramatically cheaper. My favorite example is the iOS game Threes[1], which took 14 months to make and then was cloned on android in ~20 days. It was cloned so quickly that the original developers were accused of copying the cloned version!

But just because the cloners made a functionally identical product doesn't mean they did the same work as the original designers. They got to skip designing anything - which is usually the hardest and slowest part.

[1] http://asherv.com/threes/threemails/

Re: Publishing with Apache Kafka at The New York Times

#123

Post by amygdyl » Thu, Sep 07, 2017, 7:52 AM UTC

emmelaich wrote: post #75

jdcarter wrote: post #14

FWIW, the article mentions the book "Designing Data-Intensive Applications" by Martin Kleppmann. I wanted to throw out my own endorsement for the book, it's been instrumental in helping me design my own fairly intensive data pipeline.

Had to reparse "throw out ..." a few times before I understood what you saying. Initially I thought you were withdrawing your former endorsement.

I parsed and reparsed just like you and I stalled again, while I imagined the sentence again as "throw up", trying to make a quick fix. OP has picked a tricky one..

Re: Publishing with Apache Kafka at The New York Times

#124

Post by mehh » Thu, Sep 07, 2017, 7:52 AM UTC

Why not just have the 'logs' in s3 in sorted/indexed buckets.

Re: Publishing with Apache Kafka at The New York Times

#125

Post by srigi » Thu, Sep 07, 2017, 8:22 AM UTC

pm90 wrote: post #4

Excellent, well written article. The key take away seems to be that instead of an temporary event stream log, since the number of news articles (and associated assets) is finite and cannot explode, they store all the "logs" forever (I'm using the term log as is defined in the article, as a unit of a time-ordered data structure). I wonder if NYT can help other news websites by making their code open source? I'm a huge…

What would be a bigger thing is if they opened access to their monolog.

Re: Publishing with Apache Kafka at The New York Times

#126

Post by LaFolle » Thu, Sep 07, 2017, 9:07 AM UTC

If all articles are put into monolog, what is the procedure to fetch all articles, lets say, published in year 1857? Will that be a O(n) operation (assuming all messages published in monolog have a timestamp field).

Re: Publishing with Apache Kafka at The New York Times

#127

Post by seanp2k2 » Thu, Sep 07, 2017, 9:10 AM UTC

manigandham wrote: post #86

robbyt wrote: post #83

Earlier quoted context omitted.

Why do you think it's over engineered? Don't you think the Times have considerable capacity requirements?

They claim in the article to have 100GB of text data. Let's bump that up 2 magnitudes for all text and metadata ever (outside of media files) and you can still run the entire thing on a single rack of servers and meet any performance needs. Many industries and applications are leagues ahead in both data size and speed - this isn't an example of such.

100GB...let me go find the largest MicroSD card in my house to put that on. That would actually be a good fit, since it'd work in an rPi3 which could likely serve their data publishing needs (assuming only a few updates to articles per second..not mentioned in the article, but I'd be surprised if there's more than that given the data sources) vs what they've done.

Honestly, what kind of RPS are they talking here? Requests per minute, if that, seems like.

Re: Publishing with Apache Kafka at The New York Times

#128

Post by seanp2k2 » Thu, Sep 07, 2017, 9:14 AM UTC

lobster_johnson wrote: post #76

toomim wrote: post #36

> Traditionally, databases have been used as the source of truth ... [but] can be difficult to manage in the long run. First, it’s often tricky to change the schema of a database. Adding and removing fields is not too hard, but more fundamental schema changes can be difficult to organize without downtime. This argument sounds self-contradicting. Kafka doesn't let you change its schema at all! At least postgres gives…

The problem with an RDBMS is that the schema is retroactive -- every time you update the schema, both old and new data must validate against it. There's another approach, which is to also version the schema. Every record is conceptually a pair of [schema, data]. This puts a burden on the application -- every client must be able to understand old schemas -- but the benefits are considerable. The most trivial benefit i…

CouchDB is cool if you want to mess with revisions: http://docs.couchdb.org/en/2.1.0/intro/api.html

When you're talking billions - trillions of events, check out something like https://www.slideshare.net/AmazonWebServices/how-netflix-use...

Re: Publishing with Apache Kafka at The New York Times

#129

Post by qaq » Thu, Sep 07, 2017, 10:52 AM UTC

lobster_johnson wrote: post #76

toomim wrote: post #36

> Traditionally, databases have been used as the source of truth ... [but] can be difficult to manage in the long run. First, it’s often tricky to change the schema of a database. Adding and removing fields is not too hard, but more fundamental schema changes can be difficult to organize without downtime. This argument sounds self-contradicting. Kafka doesn't let you change its schema at all! At least postgres gives…

The problem with an RDBMS is that the schema is retroactive -- every time you update the schema, both old and new data must validate against it. There's another approach, which is to also version the schema. Every record is conceptually a pair of [schema, data]. This puts a burden on the application -- every client must be able to understand old schemas -- but the benefits are considerable. The most trivial benefit i…

id uuid ts timestsmp data jsonb

solved it for you :) and thats will have ACID and all the other features of RDBMS that kafka does not have. It will also allow you to create transactionally consistent "views" representing whatever you need implemented in a few lines of SQL vs complex standalone services.

Re: Publishing with Apache Kafka at The New York Times

#130

Post by qaq » Thu, Sep 07, 2017, 11:00 AM UTC

sidlls wrote: post #55

toomim wrote: post #36

> Traditionally, databases have been used as the source of truth ... [but] can be difficult to manage in the long run. First, it’s often tricky to change the schema of a database. Adding and removing fields is not too hard, but more fundamental schema changes can be difficult to organize without downtime. This argument sounds self-contradicting. Kafka doesn't let you change its schema at all! At least postgres gives…

I agree. It's rare that a company needs something like Kafka. Kafka introduces a number of issues related to the development of client code and data stores (if any) and the maintenance of these things. It's important that the actual scale justifies the expense incurred.

It's not rare, using Kafka in place of a prober db is crazy though. There are tones of use cases that benefit from having Kafka a very common one is putting Kafka in front of the data processing pipeline to absorb spikes without data loss.