Publishing with Apache Kafka at The New York Times • Hacker News Board

Re: Publishing with Apache Kafka at The New York Times

#31

Post by _m8fo » Wed, Sep 06, 2017, 6:57 PM UTC

1wheel wrote: post #27

_m8fo wrote: post #11

I wonder how much of this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied. Are there any articles that supplement this that explain how much business value is added/lost by the existence/removal of these kind of features? In the case of NYT I suspect its popularity is maintained because of the perception (real or not) of high quality journali…

> How much would be lost if NYT was just implemented as text articles that are cached and styled with some CSS. https://www.nytimes.com/interactive/2016/12/28/us/year-in-in...

That's neat, but most of those aren't anything that can't be implemented with just CSS and HTML, unlike some of the stuff you've done for the NYT. Though even in that case, isn't all of this stuff just static assets? Hasn't this problem already been "solved"?

Re: Publishing with Apache Kafka at The New York Times

#32

Post by robohamburger » Wed, Sep 06, 2017, 7:00 PM UTC

Diederich wrote: post #16

robohamburger wrote: post #9

Earlier quoted context omitted.

I skimmed the article but I imagined they were using it as a secondary data store. I think they want to it to be durable in the sense that even if the events are already consumed they can still play them back to reindex elastic search (which is a thing you need to do periodically).

"With the log as the source of truth, there is no longer any need for a single database that all systems have to use. Instead, every system can create its own data store (database) – its own materialized view – representing only the data it needs, in the form that is the most useful for that system. This massively simplifies the role of databases in an architecture, and makes them more suited to the need of each appl…

Fair enough. It seems like it still ought to be able to burn the kafka+elasticsearch world down and submit everything to kafka with such a setup (and thus elasticsearch). I would certainly not sleep very well at night if I could not.

Re: Publishing with Apache Kafka at The New York Times

#33

Post by snuxoll » Wed, Sep 06, 2017, 7:07 PM UTC

cookiecaper wrote: post #28

skybrian wrote: post #25

Earlier quoted context omitted.

Well, except that Kafka doesn't automatically delete stuff?

https://kafka.apache.org/documentation/ Name: cleanup.policy Description: A string that is either "delete" or "compact". This string designates the retention policy to use on old log segments. The default policy ("delete") will discard old segments when their retention time or size limit has been reached. ??? How is this not automatic deletion of stuff? I don't have to worry about someone setting a "delete all data o…

Log compaction is generally what you want, which will preserve the most recent message for every key in a topic. Event streams expanding boundlessly is something very few will ever need or want, so you'll toss messages into an "event" topic of some kind, apply the event to the most recent entry in the "model" topic and store a new version (which will be kept after log compaction, the original messages can be pruned after you need to free up storage).

Re: Publishing with Apache Kafka at The New York Times

#34

Post by cookiecaper » Wed, Sep 06, 2017, 7:07 PM UTC

skybrian wrote: post #29

cookiecaper wrote: post #12

Earlier quoted context omitted.

In Postgres, I never have to worry that the server will be accidentally loaded with `retention.bytes` or `retention.days` set too low and, as a result, choose to delete everything in the database, generating a wholly artificial "disaster" that can result in long periods of disruption or downtime (at a minimum; worst case is permanent data loss). It is true that someone could issue `DROP DATABASE`, `rm -rf` the filesy…

So, remove the flag that you'll never use and recompile? I'm not sure this is worse than using a Unix box with a working "rm" command.

For what it's worth, I've known sysadmins who strip their boxes to the bones and take pains to ensure that the "rm" command won't be able to be accidentally invoked, primarily by ensuring it doesn't exist on the box. They carry their utilities from box to box, and take them with them when they leave.

That said, any slightly-sane permission or access control scheme, including the defaults mandated by almost all RDBMS distributions (which want a system user dedicated to their use), would make it rather difficult to rm the database folder. Just opening a shell to a RDBMS's underlying server should be a rare event in itself, to say nothing of actually elevating to root, or running a sloppy/careless rm command that is uncaught by the numerous potential failsafes that sysadmins have been installing for decades now (constraining superuser access to a pre-defined set of commands, for example).

Again, the point is not that RDBMS systems are invincible. It's just that they're much sturdier, and actually designed to serve this purpose.

In what universe is "Well, hack out the dangerous parts" a reasonable answer? Talk about reckless disregard for data integrity! Do you really want to use Kafka that bad that you'd develop, maintain, and thoroughly test a custom patchset that circumvents its eviction routines, rather than just using the systems that already excel at not deleting stuff?

Secondly, eviction is a core part of a message queue's design, on purpose. It's actually a needed thing, and while I'm not a Kafka dev, I seriously doubt that it's so simple that a single flag can be disabled and we can move on.

Re: Publishing with Apache Kafka at The New York Times

#35

Post by dswalter » Wed, Sep 06, 2017, 7:14 PM UTC

jdcarter wrote: post #14

FWIW, the article mentions the book "Designing Data-Intensive Applications" by Martin Kleppmann. I wanted to throw out my own endorsement for the book, it's been instrumental in helping me design my own fairly intensive data pipeline.

It's such a wonderful book.

Reading it pushed me from thinking in terms of what I had worked with to building systems based on what was needed.

I cannot recommend it highly enough, for pretty much anyone in {frontend, backend, data science, etc}.

Re: Publishing with Apache Kafka at The New York Times

#36

Post by toomim » Wed, Sep 06, 2017, 7:16 PM UTC

> Traditionally, databases have been used as the source of truth ... [but] can be difficult to manage in the long run. First, it’s often tricky to change the schema of a database. Adding and removing fields is not too hard, but more fundamental schema changes can be difficult to organize without downtime.

This argument sounds self-contradicting. Kafka doesn't let you change its schema at all! At least postgres gives you the option.

It seems that the author is excited about having a single source of truth that doesn't change, and didn't realize that he could do that with a database, if he just never used the schema-changing features.

Am I missing something? It seems like the author could be totally happy with a bunch of derived postgres databases sitting in front of a "source of truth" database, where he never changes the source of truth database's schema.

Why use kafka?

Re: Publishing with Apache Kafka at The New York Times

#37

Post by look_lookatme » Wed, Sep 06, 2017, 7:20 PM UTC

This is very similar to a normalized model in a relational database, with many-to-many relationships between the assets.

In the example we have two articles that reference other assets. For instance, the byline is published separately, and then referenced by the two articles. All assets are identified using URIs of the form nyt://article/577d0341-9a0a-46df-b454-ea0718026d30. We have a native asset browser that (using an OS-level scheme handler) lets us click on these URIs, see the asset in a JSON form, and follow references. The assets themselves are published to the Monolog as protobuf binaries.

When consuming this data do you have to programatically do relationship fetching on the client side or is eager loading/joins available in some way in Kafka?

Additionally there seems to be a focus on point-in-time specific views of this data, but are you able to construct views using arbitrary values/functions? Let's say each article is annotated with some geo data, can you construct regional versions of these materialized views of articles at the Kafka level? If not it seems like you are pushing a fair amount of existing sophisticated behavior at the RDBMS level up into custom built application servers.

Re: Publishing with Apache Kafka at The New York Times

#38

Post by miguno » Wed, Sep 06, 2017, 7:24 PM UTC

robohamburger wrote: post #9

cookiecaper wrote: post #5

>We need the log to retain all events forever, otherwise it is not possible to recreate a data store from scratch. SIGH . Cue the facepalm, head in hands, etc. I'm not going to get into a big thing here. But if you find yourself saying "I need to keep this thing forever no matter what" and then you try to use something that even entertains the notion of automatic eviction/deletion semantics as the system of record, y…

I skimmed the article but I imagined they were using it as a secondary data store. I think they want to it to be durable in the sense that even if the events are already consumed they can still play them back to reindex elastic search (which is a thing you need to do periodically).

> I think they want to it to be durable in the sense that even if the events are already consumed they can still play them back to reindex elastic search (which is a thing you need to do periodically).

That (replaying if needed) is exactly what Kafka allows you to do, unless I misunderstood what you wrote.

Re: Publishing with Apache Kafka at The New York Times

#39

Post by lima » Wed, Sep 06, 2017, 7:25 PM UTC

cookiecaper wrote: post #12

bognition wrote: post #6

Earlier quoted context omitted.

OOC what makes a RDBMS more durable that a Kafka? Both of them are systems for representing data on disk. I'd love to hear why one representation system is better at disaster recovery than another.

In Postgres, I never have to worry that the server will be accidentally loaded with `retention.bytes` or `retention.days` set too low and, as a result, choose to delete everything in the database, generating a wholly artificial "disaster" that can result in long periods of disruption or downtime (at a minimum; worst case is permanent data loss). It is true that someone could issue `DROP DATABASE`, `rm -rf` the filesy…

Kafka is a fine replacement for a RDBMS if it fits your particular use case. It has very strong data consistency guarantees - stronger than most RDBMS - if you configure it properly (acks=1 et al). It won't even lose data if the leader of a partition crashes during a commit.

It has been explicitly designed for these use cases and even has features like compaction:

https://kafka.apache.org/documentation/#compaction

Now, I agree with you that in most cases, using Kafka as your primary data store instead of a RDBMS is madness - but that doesn't mean it's a bad idea in general.

Re: Publishing with Apache Kafka at The New York Times

#40

Post by BenoitEssiambre » Wed, Sep 06, 2017, 7:31 PM UTC

toomim wrote: post #36

> Traditionally, databases have been used as the source of truth ... [but] can be difficult to manage in the long run. First, it’s often tricky to change the schema of a database. Adding and removing fields is not too hard, but more fundamental schema changes can be difficult to organize without downtime. This argument sounds self-contradicting. Kafka doesn't let you change its schema at all! At least postgres gives…

I tend to be the one arguing this, to stick to postgres for most things but even I will admit it does depend on scale.

I'm not sure what the NYT requirements are but from my understanding of Kafka, its persistent redundant distributed queues scale automatically horizontally across machines to support colossal amounts of data. It's possible that they had difficulty fitting everything in a postgres instance.