Live data from Hacker News

Publishing with Apache Kafka at The New York Times

confluent.io

141–150 of 157 posts

Re: Publishing with Apache Kafka at The New York Times

#141
post #130
post #55

Earlier quoted context omitted.

I agree. It's rare that a company needs something like Kafka. Kafka introduces a number of issues related to the development of client code and data stores (if any) and the maintenance of these things. It's important that the actual scale justifies the expense incurred.

It's not rare, using Kafka in place of a prober db is crazy though. There are tones of use cases that benefit from having Kafka a very common one is putting Kafka in front of the data processing pipeline to absorb spikes without data loss.

It's rare that a company has spikes of that sort that can't be handled in far more mundane ways. Most companies do not operate at a scale that merits this sort of solution.

Look at the engineering blogs on LinkedIn for the reasoning behind creating it. It's over-kill for most use cases.

Re: Publishing with Apache Kafka at The New York Times

#142
post #21

Earlier quoted context omitted.

> Kafka messages are immutable. Each of those green boxes on the right hand side of the first diagram will need to have special-case logic to unpack the kafka stream, with knowledge of its changes (up until 17 May 2017, treat the data like this, but between then and 19 May 2017 do x, and after that do y). I respectfully disagree. The genius of this approach is that you can make the same transformation on the original…

I still don't understand the hatred around XML. Is it slightly verbose? Yes. Does it support lots of neat functionality that make it great for interoperating between systems, like validations and transformations? Yep. Sure, it's possible to go full architecture astronaut with it, but you can do that with pretty much any programming language. Meanwhile, I'm just sitting over here wondering whether my YAML file is supp…

> trying to go figure out which magic incantation I need to get it to handle a multi-line string the way I'm expecting.

Shameless plug: https://yaml-multiline.info is a little website I wrote to help with exactly this. I'd love to hear what other people think of it!

Re: Publishing with Apache Kafka at The New York Times

#143

Earlier quoted context omitted.

I don't claim that Kafka randomly deletes things. Just that it automatically does so. The danger is not that Kafka will choose not to respect the configuration value. It is that the default setting will find a way to creep back in without the admin noticing it, and then a quick reboot, maybe even an unplanned one caused by a power trip or a kernel crash, will be sayonara to the system of record. Sure, there are backu…

Agreed that this is a newer architectural paradigm and a younger product. Of that there is no doubt, and there is always risk there. Also return, of course; someone had to deploy an RDBMs for the first time too—and everyone is glad they did. But I still don't follow the argument. If the eviction model is a loud, clear signal that this is the wrong solution, why isn't the mutability of RDBMS data the same sort of sign…

SQL provides users with a lot of facilities and mechanisms to limit, control, supervise, and if performed within a transaction, even undo overzealous DELETE statements. I discussed some of these at https://news.ycombinator.com/item?id=15188619.

AFAIK, with Postgres, there are no known circumstances where restarting your server will result in the purge of your database; that's really just icing on the cake, not the core of the argument. The core of the argument is that SQL provides not only a rational design paradigm for long-term storage and choices that reflect they take it seriously, but also an extremely strong feature set for data management and integrity.

As I've said numerous times now, SQL isn't invincible. But it's inarguably more resilient than Kafka, and it provides the controls necessary to keep some sanity over data in the long run.

Re: Publishing with Apache Kafka at The New York Times

#144

Earlier quoted context omitted.

Well, except that Kafka doesn't automatically delete stuff?

https://kafka.apache.org/documentation/ Name: cleanup.policy Description: A string that is either "delete" or "compact". This string designates the retention policy to use on old log segments. The default policy ("delete") will discard old segments when their retention time or size limit has been reached. ??? How is this not automatic deletion of stuff? I don't have to worry about someone setting a "delete all data o…

I'm not quite sure what you're insinuating about YC here but I suppose a standard "no, we don't do anything like that" is in order.

We sometimes rate-limit HN accounts when they have a habit of posting low-quality comments too quickly or (especially) getting involved in flamewars. Since we've discussed this more than once before, I assume you remember it, but other people might not know. That's all that's happening here and of course it has nothing to do with your opinions, about Kafka or YC or anything else.

Re: Publishing with Apache Kafka at The New York Times

#145

When they publish hateful garbage like the following, I have no interest in their supposed technical prowess: https://www.nytimes.com/2017/09/04/us/texas-storm-federal-ai...

Could you please stop violating the guidelines with gratuitous off-topic flamebait?

https://news.ycombinator.com/newsguidelines.html

Re: Publishing with Apache Kafka at The New York Times

#146
post #144

Earlier quoted context omitted.

https://kafka.apache.org/documentation/ Name: cleanup.policy Description: A string that is either "delete" or "compact". This string designates the retention policy to use on old log segments. The default policy ("delete") will discard old segments when their retention time or size limit has been reached. ??? How is this not automatic deletion of stuff? I don't have to worry about someone setting a "delete all data o…

I'm not quite sure what you're insinuating about YC here but I suppose a standard "no, we don't do anything like that" is in order. We sometimes rate-limit HN accounts when they have a habit of posting low-quality comments too quickly or (especially) getting involved in flamewars. Since we've discussed this more than once before, I assume you remember it, but other people might not know. That's all that's happening h…

"Flame wars" in the sense of actually responding to the people who are trying to have a discussion on the technical points? Suggesting that my posts involve flaming virtually ever is a flagrant mischaracterization, and we both know it.

This thread contains about the "worst" you'll find from me, with "No offense, but this reveals your ignorance...". This is a flame only insofar as "flame" constitutes any disagreement at all.

When the account was rate-limited, the complaint was a) that HN had received out-of-band complaints from people who were getting mad that I discouraged others from deploying database clusters on Kubernetes; and b) that my posts on the subject were "trite". Maybe I don't waste enough time on HN, but I'd never seen that before.

I concede there was one post in that thread that could be interpreted as borderline incendiary, suggesting that the only reason to run a database on top of k8s is to win "GCool Points" (a position I continue to maintain), but it was directed at no one in particular, intended to provide some levity to non-techno-hipsters, and prevent some of the routine noise we see every time I make that type of post from people who really don't have any counter-point except that "Google does it!". Hardly a habit of engaging in flame wars. And it was used by HN/YC as an excuse to "detach" the entire thread, in which a real technical debate was occurring, as is occurring here, instead of just the single borderline post.

I understand that you feel the need to go on the record with a denial. I hope you can appreciate that I feel that need too, whether the accusation is implied through the rate limit that prevents me from replying and may mislead others into believing that my position is indefensible, or explicit, as it is now.

Re: Publishing with Apache Kafka at The New York Times

#147

Earlier quoted context omitted.

>Kafka can perfectly keep your data around forever. In the sense that you can fiddle with it to the point where it doesn't purge things automatically, sure. But RDBMS provides more than the promise that it won't delete your data after a set period of time. If that was all we needed, any filesystem from the last 3 decades would serve fine as a "permanent datastore". MySQL has gone through a lot of grief to get to the…

Kafka is not a messaging queue. It's a log. The same underlying structure used within our RDBMS to provide all the guarantees Kafka provides. As for logs and databases, they are duals. https://www.confluent.io/blog/turning-the-database-inside-ou...

>Kafka is not a messaging queue. It's a log.

Kafka bills itself as a "messaging system".

>The same underlying structure used within our RDBMS to provide all the guarantees Kafka provides.

Is there perhaps a reason that people use the RDBMS scaffold atop this "underlying structure"? The point I'm making is that Kafka is not safe for use cases that demand robust data storage and integrity, at least not in comparison to the standard of safety set by the traditional systems of record for important data (RDBMS).

Re: Publishing with Apache Kafka at The New York Times

#148

Earlier quoted context omitted.

>I wonder how much of this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied. That's a ton of it. Like it or not, publishing a digital newspaper is not a hard or unsolved problem; it's one of the web's core competencies. If you hire people who want to build cool stuff to supervise a CMS, well, you get this kind of outcome. The raw cost is under…

There are non-negligible problems in the news space like: 1. Supporting full-text search for a fair number of concurrent users 2. Availability of the system with minimal downtime 3. Scalability within the day and year, traffic patterns around e.g., breaking news events will far surpass 2AM traffic 4. Notifications I could go on and on but honestly, it's just a tone-deaf response. Parting pot-shot: "No one is going to…

"We can't publish right now because the database had to enter an unplanned maintenance period" is a lot different from "our authoritative archive is gone and we have to try to rebuild it from all these separate 'materialized views', woops."

Re: Publishing with Apache Kafka at The New York Times

#149
post #141
post #130

Earlier quoted context omitted.

It's not rare, using Kafka in place of a prober db is crazy though. There are tones of use cases that benefit from having Kafka a very common one is putting Kafka in front of the data processing pipeline to absorb spikes without data loss.

It's rare that a company has spikes of that sort that can't be handled in far more mundane ways. Most companies do not operate at a scale that merits this sort of solution. Look at the engineering blogs on LinkedIn for the reasoning behind creating it. It's over-kill for most use cases.

Kafka is fairly mundane piece of software aside from periodic ops issues with Zookeeper.

Re: Publishing with Apache Kafka at The New York Times

#150
post #145

When they publish hateful garbage like the following, I have no interest in their supposed technical prowess: https://www.nytimes.com/2017/09/04/us/texas-storm-federal-ai...

Could you please stop violating the guidelines with gratuitous off-topic flamebait? https://news.ycombinator.com/newsguidelines.html

I stated an opinion. You don't like it? Tough.
Post reply on HN