Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a Relational Database Using Kafka

yokota.blog

Post reply on HN
Print view View original

1–10 of 34 posts

Re: Building a Relational Database Using Kafka

#6

Post by thinkersilver »

humbleMouse wrote: post #2

This is fetishizing complexity for no reason. Makes for an interesting blog post, but if any company told me they were doing this I would run the other direction immediately.

Great article. I think you might be taking a subtle point in the article for granted. Namely, the ability to assemble a fairly complex distributed system from open source components.

Democratisation of complex machinery like the raft and consensus algorithms, schedulers, append logs, query optimisers and so on is a superb thing. It is something that has only been possible in the last 3/4 years and would have been devilishly difficult before then without significant upfront work.

Fetishising complexity might be a bit strong here. Seeing more posts like this is a good thing.

Re: Building a Relational Database Using Kafka

#7

Post by swagonomixxx »

tener wrote: post #3

Interesting article, but I think it is missing a solid comparison of such "overlay SQL" with more traditional database engine. It isn't enough to show something can be done. One must also think if it should be done.

The author is doing this purely for fun (I think, I'm not the author). They link to this: https://www.confluent.io/product/ksql/ which is apparently a production tested system that provides a SQL interface to a Kafka backend.

Re: Building a Relational Database Using Kafka

#8

Post by mLuby »

thinkersilver wrote: post #6
humbleMouse wrote: post #2

This is fetishizing complexity for no reason. Makes for an interesting blog post, but if any company told me they were doing this I would run the other direction immediately.

Great article. I think you might be taking a subtle point in the article for granted. Namely, the ability to assemble a fairly complex distributed system from open source components. Democratisation of complex machinery like the raft and consensus algorithms, schedulers, append logs, query optimisers and so on is a superb thing. It is something that has only been possible in the last 3/4 years and would have been dev…

You make a good counterpoint: it's awesome that this can be done concisely enough these days to fit in a hobby blog post. That's new and demonstrates the approachability of these tools.

At the same time, OP is right: a business doing this is making poor engineering decisions.

Re: Building a Relational Database Using Kafka

#9

Post by foxyv »

humbleMouse wrote: post #2

This is fetishizing complexity for no reason. Makes for an interesting blog post, but if any company told me they were doing this I would run the other direction immediately.

If you are using this for your shopping website I would definitely agree. If you are building a managed database service like DynamoDB or Aurora then complexity may be warranted. Great for companies like Google and Amazon that need something that will work best for them and have the engineering talent to make it stick.

Projects like this are how we get solutions that blow existing stuff out of the water. Although a lot just end up being fun projects that amount to nothing but an entertaining blog article.

Re: Building a Relational Database Using Kafka

#10

Post by pram »

From personal experience, considering the reliability of Kafka, I’d implement this in reverse. Most relational databases are far more resilient in operation.

One idea I’ve had in the past is to implement a Kafka and Zookeeper layer for FoundationDB.

Post reply on HN