71–80 of 157 posts
> Traditionally, databases have been used as the source of truth ... [but] can be difficult to manage in the long run. First, it’s often tricky to change the schema of a database. Adding and removing fields is not too hard, but more fundamental schema changes can be difficult to organize without downtime. This argument sounds self-contradicting. Kafka doesn't let you change its schema at all! At least postgres gives…
Earlier quoted context omitted.
From what I can tell, Kafka isn't designed for long term data storage. RDBMS systems are designed for this. Kafka is more for streaming data and events, so I'd probably be uncomfortable assuming that it won't do something and "tidy up" my very old data at some point in the future. Since it's supposed to do this from time to time out of the box, you'd have to be very careful not to let anyone tweak the custom config t…
You've really just laid out feelings rather than concrete technical reasons for why Kafka can't function as a permanent datastore. Can you point to specific design elements in Kafka that would lead you to conclude that it isn't suitable for permanent data storage? Also, Kafka doesn't "do" anything to your old data if you don't want it to. It's also open-source, so these behaviors can be verified.
It scales linearly.
> Traditionally, databases have been used as the source of truth ... [but] can be difficult to manage in the long run. First, it’s often tricky to change the schema of a database. Adding and removing fields is not too hard, but more fundamental schema changes can be difficult to organize without downtime. This argument sounds self-contradicting. Kafka doesn't let you change its schema at all! At least postgres gives…
Kafka doesn't have a schema per message, messages are just bytes that you can serialize / deserialize however you want to. The article refers to using protobuf for messages, which does support adding fields. If you're equating kafka topics with the idea of a schema, you can add topics.
FWIW, the article mentions the book "Designing Data-Intensive Applications" by Martin Kleppmann. I wanted to throw out my own endorsement for the book, it's been instrumental in helping me design my own fairly intensive data pipeline.
Initially I thought you were withdrawing your former endorsement.
> Traditionally, databases have been used as the source of truth ... [but] can be difficult to manage in the long run. First, it’s often tricky to change the schema of a database. Adding and removing fields is not too hard, but more fundamental schema changes can be difficult to organize without downtime. This argument sounds self-contradicting. Kafka doesn't let you change its schema at all! At least postgres gives…
There's another approach, which is to also version the schema. Every record is conceptually a pair of [schema, data]. This puts a burden on the application -- every client must be able to understand old schemas -- but the benefits are considerable.
The most trivial benefit is that no data is ever lost, even if the business case is taken away. Over time, the burden of supporting obsolete features mean that you want to remove columns that don't apply anymore. But that does remove interesting and potentially valuable historical data. If the schema evolves together with the data, you can keep even the cruft, at little expense.
In practice, the evolution of the schema means that data often has to be transformed from one schema to another (e.g. v1 -> v2). In an RDBMS, this transformation (schema migration) is performed once, and if it contains bugs, you have to resort to backups to restore the old data. But with a versioned schema, the transformations are simply functions on immutable data. You can go back and re-run transformations on old data to get new data, non-destructively.
This is a flawed architecture. It will work at release, but it will be difficult to manoeuvre with, and they will grow to hate it. As your business changes, your data changes. Imagine if on day one, they had one author per article. On day 1000, they change this to be a list of authors. Kafka messages are immutable. Each of those green boxes on the right hand side of the first diagram will need to have special-case lo…
They're using protobufs, which seem just about as flexible as XML as far as schema updates are concerned and are considerably less ambiguous. So I don't see how XML would help?
(1) Protobufs are not self-describing. There is a meta-model but you can't really understand the data unless you have access to the schema. This means just to be safe you might end up storing the schema beside every message or at least a reference to the schema. Welcome to the schema management business.
(2) Protobufs aren't extensible. In practice this means any time a system wants to introduce a new field they have to submit a proposal to some sort of highly centralized "Architecture Group." This is followed by lots of debate. Then, if you're lucky, it gets put in and a few systems adopt it. With XML you can slice off namespaces and let people innovate in those namespaces.
(3) Protobufs aren't human readable. At the end of the day this means you need special tools to do anything with them. Meanwhile XML can actually be imported directly into Excel.
There's a whole ecosystem of powerful technologies around XML that make it work very well in this case. People underestimate how valuable this is because they're still not thinking of the log as its own first-class product.
That said, perfect is the enemy of good. This could be a good step in the right direction and migrating to XML in the future would be pretty easy because it becomes a very simple event transformation.
I wonder how much of this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied. Are there any articles that supplement this that explain how much business value is added/lost by the existence/removal of these kind of features? In the case of NYT I suspect its popularity is maintained because of the perception (real or not) of high quality journali…
I can't speak for the situation at the NYT but the actual public site for online papers are often pretty simple things with most of the complexity being ad logic. The systems here almost entirely deal with writing and content retrieval pipelines for stuff that was written years ago in other systems that isn't tagged or stored in sympathetic ways, and there will also be the very old school print pipeline to have to de…
Excellent, well written article. The key take away seems to be that instead of an temporary event stream log, since the number of news articles (and associated assets) is finite and cannot explode, they store all the "logs" forever (I'm using the term log as is defined in the article, as a unit of a time-ordered data structure). I wonder if NYT can help other news websites by making their code open source? I'm a huge…
Family-owned papers like the one I work at (The Spokesman-Review in Spokane, WA) are the one of the few news orgs that could actually put something like this into production within this century, but even we still have to deal with manpower issues.
Earlier quoted context omitted.
I can't speak for the situation at the NYT but the actual public site for online papers are often pretty simple things with most of the complexity being ad logic. The systems here almost entirely deal with writing and content retrieval pipelines for stuff that was written years ago in other systems that isn't tagged or stored in sympathetic ways, and there will also be the very old school print pipeline to have to de…
How much time/effort does it take to just update all this old stuff once and for all?