Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Publishing with Apache Kafka at The New York Times

confluent.io

Post reply on HN
Print view View original

91–100 of 157 posts

Re: Publishing with Apache Kafka at The New York Times

#91

Post by vikiomega9 »

manigandham wrote: post #82

The very definition of over-engineered. This is just event-sourcing turned into a marketing article for Kafka. It doesn't really matter what the "source of truth" system is, although Kafka doesn't seem quite as mature/stable enough for that. With such little data, they can push into a nice graph database instead, run it entirely in memory and meet 10x any demand they'll ever see along with all the query types they'll…

Unless I'm mistaken, If I were to build out a simple event log represented by a relational DB, I have bottle necks when writing to it, and have lag in terms of processing the events, and if I were also pushing those events to a queue to hydrate aggregate snapshots I would have to have client logic to deal with duplicate events or not acking processed events etc?

Intuitively, I guess kafka is "more realtime" and "more available" when compared to the home-brew event log?

EDIT: obviously those constraints in my home-brew event log are relaxed when my problem domain is amenable to things like associative operators, idempotency, inverses etc.

Re: Publishing with Apache Kafka at The New York Times

#92

Post by cookiecaper »

vikiomega9 wrote: post #91
manigandham wrote: post #82

The very definition of over-engineered. This is just event-sourcing turned into a marketing article for Kafka. It doesn't really matter what the "source of truth" system is, although Kafka doesn't seem quite as mature/stable enough for that. With such little data, they can push into a nice graph database instead, run it entirely in memory and meet 10x any demand they'll ever see along with all the query types they'll…

Unless I'm mistaken, If I were to build out a simple event log represented by a relational DB, I have bottle necks when writing to it, and have lag in terms of processing the events, and if I were also pushing those events to a queue to hydrate aggregate snapshots I would have to have client logic to deal with duplicate events or not acking processed events etc? Intuitively, I guess kafka is "more realtime" and "more…

There's no reason not to make good use of Kafka or similar solutions. The issue is that people use it without understanding it. In this article, they say that Kafka is their system of record and their primary long-term storage. That's very silly.

Re: Publishing with Apache Kafka at The New York Times

#93

Post by sidlls »

Diederich wrote: post #16
robohamburger wrote: post #9

Earlier quoted context omitted.

I skimmed the article but I imagined they were using it as a secondary data store. I think they want to it to be durable in the sense that even if the events are already consumed they can still play them back to reindex elastic search (which is a thing you need to do periodically).

"With the log as the source of truth, there is no longer any need for a single database that all systems have to use. Instead, every system can create its own data store (database) – its own materialized view – representing only the data it needs, in the form that is the most useful for that system. This massively simplifies the role of databases in an architecture, and makes them more suited to the need of each appl…

And then you end up with a different flavor of data store for every team, complete with its own idioms and (probably duplicative) business logic.

Unless their devops/SRE staff is up to the task this "architecture" is a nightmare waiting to happen.

Re: Publishing with Apache Kafka at The New York Times

#94

Post by tlberglund »

theossuary wrote: post #44
BenoitEssiambre wrote: post #40

Earlier quoted context omitted.

I tend to be the one arguing this, to stick to postgres for most things but even I will admit it does depend on scale. I'm not sure what the NYT requirements are but from my understanding of Kafka, its persistent redundant distributed queues scale automatically horizontally across machines to support colossal amounts of data. It's possible that they had difficulty fitting everything in a postgres instance.

See, that's where I'm confused. I'm no Kafka expert, but they say they use a "single-partition topic" which I believe means the only way they can replicate the data is by replicating the entire log, they can't shard because it's a single partition. The reasoning behind this is because Kafka doesn't support ordering between partitions. Also I've never thought of Kafka as a persistent data storage solution, it's intere…

You are not even a little bit confused! I think you have it perfectly right. And you are not alone in not thinking of Kafka as persistent storage, but when you get down to it there is no reason not to, and people are indeed using it in just that way. And yes, Confluent does give its +1 to this practice. :)

Re: Publishing with Apache Kafka at The New York Times

#95

Post by cookiecaper »

LgWoodenBadger wrote: post #49
cookiecaper wrote: post #28

Earlier quoted context omitted.

https://kafka.apache.org/documentation/ Name: cleanup.policy Description: A string that is either "delete" or "compact". This string designates the retention policy to use on old log segments. The default policy ("delete") will discard old segments when their retention time or size limit has been reached. ??? How is this not automatic deletion of stuff? I don't have to worry about someone setting a "delete all data o…

You may not have to worry about somebody setting a "delete all data older and/or bigger than Y or Z" but you have to worry about someone running "DELETE FROM table" without a WHERE clause. Which is easier to prevent? The one that can be done through the same mechanism as non-destructive queries? Or the one that can only be modified through a file-system configuration, completely separate from its API? Regardless, it'…

If someone runs `DELETE FROM table` without a WHERE clause, I expect:

a) the query to be tested and scripted on non-production environments first, making this a non-issue;

b) user doesn't have DELETE permissions on that table and/or the rows not intended to be deleted;

c) referential integrity to kick in and prevent the deletion of interdependent records (which is most records in a database);

d) CHECK constraints, triggers, and other validation routines to prevent this clearly-excessive operation;

e) the person executing and inspecting these queries within an ad-hoc transaction to roll it back before committing;

f) if, in the event this does occur and commit, which itself means there's a big problem with your procedure, streaming binlog archives can facilitate a point-in-time backup, audit tables can be used to rebuild the data, etc.; these aren't typically included by default (streaming point-in-time backups are on AWS Aurora) but they're conventional for many professionally-run RDBMS installations.

and I'm sure there are failsafes that I'm forgetting, and since I'm not a DBA, some I'm probably not even aware of.

How many of these can I expect to help me out when a packaging bug (or, simply a mistaken "y" on the prompt asking if I want to override the package config) clobbers the Kafka config file?

Re: Publishing with Apache Kafka at The New York Times

#96

Post by tlberglund »

elnygren wrote: post #87
TwelveNights wrote: post #81

Earlier quoted context omitted.

Is 100GB of data really enough to warrant this type of architecture?

if you read more closely, this article is not about the amount of data, but the way how it is managed.

Precisely. While "all NYT data since 1851" sounds like a lot, and >8660 days sounds like a long retention period for a Kafka topic, this--like most systems in the world--is not a Big Data application. One of the key insights from the post is that there are interesting architectural considerations that have nothing to do with data size that make immutable logs a good idea.

Re: Publishing with Apache Kafka at The New York Times

#97

Post by mvc »

bonesss wrote: post #63
skybrian wrote: post #23

Earlier quoted context omitted.

They're using protobufs, which seem just about as flexible as XML as far as schema updates are concerned and are considerably less ambiguous. So I don't see how XML would help?

Through its various protocols Kafka topics can be configured to be guaranteed forwards, backwards, or bi-directionally compatible. That is to say: as flexible as XML or an RDBMS schema with long-term, format encoded, data that can explicitly support conflicting clients over time (as desired by the dev). Zero-impact, live, online, updates touching hundreds of active systems without issue... TBH most posters here have…

> Through its various protocols Kafka topics can be configured to be guaranteed forwards, backwards, or bi-directionally compatible.

Sure but backwards and bi-directional compatibility inhibits the evolvability of schemas. Something as simple as adding a new required field for example is not backward compatible in avro.

I understand why this is, and that in a huge database, it's not that simple in SQL either. But in 95% of databases it actually is quite simple.

Re: Publishing with Apache Kafka at The New York Times

#98

Post by tlberglund »

cookiecaper wrote: post #90
miguno wrote: post #43

Earlier quoted context omitted.

Kafka can perfectly keep your data around forever. The only limitation is available disk space (and databases have the same limitation). I'm not implying that it is always the best idea to use Kafka as a long-term storage solution, but likewise a database isn't the silver bullet here either. > so I'd probably be uncomfortable assuming that it won't do something and "tidy up" my very old data at some point in the futu…

>Kafka can perfectly keep your data around forever. In the sense that you can fiddle with it to the point where it doesn't purge things automatically, sure. But RDBMS provides more than the promise that it won't delete your data after a set period of time. If that was all we needed, any filesystem from the last 3 decades would serve fine as a "permanent datastore". MySQL has gone through a lot of grief to get to the…

It may be unfair to describe setting a documented configuration parameter as "fiddling." Retention is seven days by default. It is trivial to set it to arbitrarily long periods of time. To my knowledge, this functionality isn't really in question. Whether logs are a good unifying abstraction on which to build systems is in dispute among reasonable people, but whether Kafka randomly deletes stuff is not. :)

Re: Publishing with Apache Kafka at The New York Times

#99

Post by tlberglund »

cookiecaper wrote: post #12
bognition wrote: post #6

Earlier quoted context omitted.

OOC what makes a RDBMS more durable that a Kafka? Both of them are systems for representing data on disk. I'd love to hear why one representation system is better at disaster recovery than another.

In Postgres, I never have to worry that the server will be accidentally loaded with `retention.bytes` or `retention.days` set too low and, as a result, choose to delete everything in the database, generating a wholly artificial "disaster" that can result in long periods of disruption or downtime (at a minimum; worst case is permanent data loss). It is true that someone could issue `DROP DATABASE`, `rm -rf` the filesy…

I don't mean to be cavalier about misconfiguration, but it's not like the retention period is the Sword of Damocles. It's a configuration setting, and Kafka honors it reliably. As other commenters have pointed out, there are other bad things you can do to cause data loss with any system no matter how hard you try. These stories will continue to grace post-mortem blog posts long after we are gone, but stories of Kafka accidentally not retaining data do not seem to be thick on the ground. Any non-trivial system has its rough edges, but this just doesn't seem to be one of them for Kafka.

Re: Publishing with Apache Kafka at The New York Times

#100

Post by tlberglund »

pram wrote: post #70
BenoitEssiambre wrote: post #40

Earlier quoted context omitted.

I tend to be the one arguing this, to stick to postgres for most things but even I will admit it does depend on scale. I'm not sure what the NYT requirements are but from my understanding of Kafka, its persistent redundant distributed queues scale automatically horizontally across machines to support colossal amounts of data. It's possible that they had difficulty fitting everything in a postgres instance.

Kafka by default is not persistent, the logs expire after 7 days. You can increase it on a per topic basis. It also doesn't scale automatically. If you have three replicas on a single partition topic, they will live on their assigned broker forever unless you manually reassign them. Adding new nodes does not kick off rebalancing of partitions. Its automatic cluster management is very primitive compared to something l…

Your statements about data retention are true, but if I may, I think they are only vacuously true. There are definitely ways to configure data retention incorrectly, but it seems like the folks at NYT have cracked the nut. Indeed, this is a trivial thing to set up.

Your points about auto-scaling are well-taken, though. It does not automatically rebalance when you add a new node. Confluent obvious agrees that this is a problem, since this is part of what Confluent Enterprise does. #shamelessplug (Which of course I do not intend shamelessly; just saying you have a point. :) )

Post reply on HN