Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Publishing with Apache Kafka at The New York Times

confluent.io

Post reply on HN
Print view View original

51–60 of 157 posts

Re: Publishing with Apache Kafka at The New York Times

#51

Post by [deleted] »

teej wrote: post #46
jdcarter wrote: post #14

FWIW, the article mentions the book "Designing Data-Intensive Applications" by Martin Kleppmann. I wanted to throw out my own endorsement for the book, it's been instrumental in helping me design my own fairly intensive data pipeline.

Dear HN reader - if you're not quite ready to buy the book, take a listen to this episode of Software Engineering Daily ( https://softwareengineeringdaily.com/2017/05/02/data-intensi... ). It will give you a sense of what Martin Kleppmann is all about and how he thinks about problems. I ordered my copy of "Designing Data-Intensive Applications" after listening to this episode.

[deleted]

Re: Publishing with Apache Kafka at The New York Times

#52

Post by [deleted] »

BenoitEssiambre wrote: post #40
toomim wrote: post #36

> Traditionally, databases have been used as the source of truth ... [but] can be difficult to manage in the long run. First, it’s often tricky to change the schema of a database. Adding and removing fields is not too hard, but more fundamental schema changes can be difficult to organize without downtime. This argument sounds self-contradicting. Kafka doesn't let you change its schema at all! At least postgres gives…

I tend to be the one arguing this, to stick to postgres for most things but even I will admit it does depend on scale. I'm not sure what the NYT requirements are but from my understanding of Kafka, its persistent redundant distributed queues scale automatically horizontally across machines to support colossal amounts of data. It's possible that they had difficulty fitting everything in a postgres instance.

[deleted]

Re: Publishing with Apache Kafka at The New York Times

#53

Post by dreamfactored »

LgWoodenBadger wrote: post #50
_lex wrote: post #10

Earlier quoted context omitted.

I think the POV he's taken is that the kafka stream is the one true datasource (for all time), with all other dbs being derivatives. This insane strategy seems to be over engineered to get around db migrations... though i'm sure the event stream will also change over time, and he'll have to write migration-like code anyways.

There's not just one reason for it. Kafka lets dependent consumers transform that data into whatever model is appropriate for their use case. There's no one-size-fits-all E-R model for data for all use cases. It's not just working around db migrations. It's also providing you with the ability to model the data as many ways as you require. How else would you do it, materialized views? How often are they materialized?…

Isn't that just skinny model, fat client?

Re: Publishing with Apache Kafka at The New York Times

#54

Post by dreamfactored »

lima wrote: post #39
cookiecaper wrote: post #12

Earlier quoted context omitted.

In Postgres, I never have to worry that the server will be accidentally loaded with `retention.bytes` or `retention.days` set too low and, as a result, choose to delete everything in the database, generating a wholly artificial "disaster" that can result in long periods of disruption or downtime (at a minimum; worst case is permanent data loss). It is true that someone could issue `DROP DATABASE`, `rm -rf` the filesy…

Kafka is a fine replacement for a RDBMS if it fits your particular use case. It has very strong data consistency guarantees - stronger than most RDBMS - if you configure it properly (acks=1 et al). It won't even lose data if the leader of a partition crashes during a commit. It has been explicitly designed for these use cases and even has features like compaction: https://kafka.apache.org/documentation/#compaction No…

Isn't that what 'in general' means?

Re: Publishing with Apache Kafka at The New York Times

#55

Post by sidlls »

toomim wrote: post #36

> Traditionally, databases have been used as the source of truth ... [but] can be difficult to manage in the long run. First, it’s often tricky to change the schema of a database. Adding and removing fields is not too hard, but more fundamental schema changes can be difficult to organize without downtime. This argument sounds self-contradicting. Kafka doesn't let you change its schema at all! At least postgres gives…

I agree.

It's rare that a company needs something like Kafka.

Kafka introduces a number of issues related to the development of client code and data stores (if any) and the maintenance of these things. It's important that the actual scale justifies the expense incurred.

Re: Publishing with Apache Kafka at The New York Times

#56

Post by poooogles »

theossuary wrote: post #44
BenoitEssiambre wrote: post #40

Earlier quoted context omitted.

I tend to be the one arguing this, to stick to postgres for most things but even I will admit it does depend on scale. I'm not sure what the NYT requirements are but from my understanding of Kafka, its persistent redundant distributed queues scale automatically horizontally across machines to support colossal amounts of data. It's possible that they had difficulty fitting everything in a postgres instance.

See, that's where I'm confused. I'm no Kafka expert, but they say they use a "single-partition topic" which I believe means the only way they can replicate the data is by replicating the entire log, they can't shard because it's a single partition. The reasoning behind this is because Kafka doesn't support ordering between partitions. Also I've never thought of Kafka as a persistent data storage solution, it's intere…

>it's interesting Confluent is supporting Kafka being used in this way.

If it earns them money, I think they'll pretty much support anything. Jay has rubber stamped it on SO [1], but he's got a bit of a vested interest on selling Kafka.

1. https://stackoverflow.com/questions/17708489/using-kafka-as-...

Re: Publishing with Apache Kafka at The New York Times

#57

Post by skybrian »

cookiecaper wrote: post #34
skybrian wrote: post #29

Earlier quoted context omitted.

So, remove the flag that you'll never use and recompile? I'm not sure this is worse than using a Unix box with a working "rm" command.

For what it's worth, I've known sysadmins who strip their boxes to the bones and take pains to ensure that the "rm" command won't be able to be accidentally invoked, primarily by ensuring it doesn't exist on the box. They carry their utilities from box to box, and take them with them when they leave. That said, any slightly-sane permission or access control scheme, including the defaults mandated by almost all RDBMS…

Disabling a flag is likely a one-line change, assuming a reasonable flag library. But yes, maintaining a custom fork at all is not something to take on lightly. It would make more sense to talk to the Kafka developers about how to make it safer to use in this scenario.

Re: Publishing with Apache Kafka at The New York Times

#58

Post by mayank »

scaryclam wrote: post #13
bognition wrote: post #6

Earlier quoted context omitted.

OOC what makes a RDBMS more durable that a Kafka? Both of them are systems for representing data on disk. I'd love to hear why one representation system is better at disaster recovery than another.

From what I can tell, Kafka isn't designed for long term data storage. RDBMS systems are designed for this. Kafka is more for streaming data and events, so I'd probably be uncomfortable assuming that it won't do something and "tidy up" my very old data at some point in the future. Since it's supposed to do this from time to time out of the box, you'd have to be very careful not to let anyone tweak the custom config t…

You've really just laid out feelings rather than concrete technical reasons for why Kafka can't function as a permanent datastore.

Can you point to specific design elements in Kafka that would lead you to conclude that it isn't suitable for permanent data storage?

Also, Kafka doesn't "do" anything to your old data if you don't want it to. It's also open-source, so these behaviors can be verified.

Re: Publishing with Apache Kafka at The New York Times

#59

Post by robertlagrant »

toomim wrote: post #36

> Traditionally, databases have been used as the source of truth ... [but] can be difficult to manage in the long run. First, it’s often tricky to change the schema of a database. Adding and removing fields is not too hard, but more fundamental schema changes can be difficult to organize without downtime. This argument sounds self-contradicting. Kafka doesn't let you change its schema at all! At least postgres gives…

Heh this was exactly my thought. I've no idea why they couldn't just store the data in a database! Or if they really have several ways of storing data, then have a database for each and a common API/ESB layer over the top. Seems normal.

Re: Publishing with Apache Kafka at The New York Times

#60

Post by iooi »

> Because the topic is single-partition, it needs to be stored on a single disk, due to the way Kafka stores partitions. This is not a problem for us in practice, since all our content is text produced by humans — our total corpus right now is less than 100GB, and disks are growing bigger faster than our journalists can write.

Before this line, the author mentions they also store images. There's no way that all their text + images is <100GB right? Something is inconsistent here.

Post reply on HN