Building a Relational Database Using Kafka • Hacker News Board

Re: Building a Relational Database Using Kafka

#11

Post by humbleMouse » Mon, Sep 23, 2019, 5:06 PM UTC

thinkersilver wrote: post #6

humbleMouse wrote: post #2

This is fetishizing complexity for no reason. Makes for an interesting blog post, but if any company told me they were doing this I would run the other direction immediately.

Great article. I think you might be taking a subtle point in the article for granted. Namely, the ability to assemble a fairly complex distributed system from open source components. Democratisation of complex machinery like the raft and consensus algorithms, schedulers, append logs, query optimisers and so on is a superb thing. It is something that has only been possible in the last 3/4 years and would have been dev…

I agree my comment may be a knee-jerk reaction, and it is quite a good technical article with clear examples.

I also agree that the open source tech used in this solution is very powerful and it is great we are at a point where a solution like this could exist. I love avro schema registries and kafka just as much as the next guy.

Re: Building a Relational Database Using Kafka

#12

Post by xjoins » Mon, Sep 23, 2019, 5:26 PM UTC

xmpir wrote: post #4

Why would I want a relational database living in Kafka? MySQL or Postgres are great at what they do.

I came here to ask this question. It's a really solid technical article. But apart from the sake of doing it, why would anyone ever want to? I'm not suggesting there's no scenario in which one would want to build a SQL database using Kafka, I'm just asking if anybody knows of one.

Re: Building a Relational Database Using Kafka

#13

Post by wefarrell » Mon, Sep 23, 2019, 5:40 PM UTC

xjoins wrote: post #12

xmpir wrote: post #4

Why would I want a relational database living in Kafka? MySQL or Postgres are great at what they do.

I came here to ask this question. It's a really solid technical article. But apart from the sake of doing it, why would anyone ever want to? I'm not suggesting there's no scenario in which one would want to build a SQL database using Kafka, I'm just asking if anybody knows of one.

One use case for Kafka is log aggregation. Using SQL to analyze logs would be one use case.

Re: Building a Relational Database Using Kafka

#14

Post by yowlingcat » Mon, Sep 23, 2019, 6:36 PM UTC

foxyv wrote: post #9

humbleMouse wrote: post #2

This is fetishizing complexity for no reason. Makes for an interesting blog post, but if any company told me they were doing this I would run the other direction immediately.

If you are using this for your shopping website I would definitely agree. If you are building a managed database service like DynamoDB or Aurora then complexity may be warranted. Great for companies like Google and Amazon that need something that will work best for them and have the engineering talent to make it stick. Projects like this are how we get solutions that blow existing stuff out of the water. Although a l…

Agreed. To expand on your point, I believe this is pretty close to exactly what Amazon did to build Aurora, which is probably the most exciting development I've seen as an engineering leader in the past decade. Suddenly, the one crucial, stateful component of my system that could ever present sharding issues at scale became something I could pay a premium to not have to think about (to a point -- believe it's still 64TB for Aurora).

What's extra cool about this for me is how it illustrates to me how far open source components that build on the Apache Kafka ecosystem have come, to the degree that each fairly complex component of relational databases that expose a full RDBMS system from the basic building block of a log have been fully developed (to some production capacity) in the Kafka ecosystem. That's amazing! What does this mean?

It means that you could theoretically (please don't hurt me) implement a relational DB backed by a twitter DB log, or a blockchain, or any other event log that you can't necessarily use traditionally as a WAL. It also introduces a lot of really interesting possibilities for data integration. Definitely has my mind reeling a bit.

Re: Building a Relational Database Using Kafka

#15

Post by lichtenberger » Mon, Sep 23, 2019, 6:47 PM UTC

I like the idea, to use Apache Kafka as the underlying log. It's in principle what Martin Kleppmann suggests.

As I also want to use a distributed log in the future: Do you know Apache Pulsar or the underlying BookKeeper, which I think was especially made for a distributed WAL?

I might either want to use Apache BookKeeper or Apache Pulsar for a distributed log for scaling my Open Source temporal database, too. Furthermore I'd like to expose the API for streaming changes into the Browser or wherever you want to :-)

Re: Building a Relational Database Using Kafka

#16

Post by dominotw » Mon, Sep 23, 2019, 6:51 PM UTC

humbleMouse wrote: post #2

This is fetishizing complexity for no reason. Makes for an interesting blog post, but if any company told me they were doing this I would run the other direction immediately.

how else would one implement microservices then?

Do you run away from microservices too?

Re: Building a Relational Database Using Kafka

#17

Post by dominotw » Mon, Sep 23, 2019, 6:52 PM UTC

swagonomixxx wrote: post #7

tener wrote: post #3

Interesting article, but I think it is missing a solid comparison of such "overlay SQL" with more traditional database engine. It isn't enough to show something can be done. One must also think if it should be done.

The author is doing this purely for fun (I think, I'm not the author). They link to this: https://www.confluent.io/product/ksql/ which is apparently a production tested system that provides a SQL interface to a Kafka backend.

> provides a SQL interface to a Kafka backend.

Not sure if thats an accurate description of ksql.

Re: Building a Relational Database Using Kafka

#18

Post by eternalban » Mon, Sep 23, 2019, 6:57 PM UTC

lichtenberger wrote: post #15

I like the idea, to use Apache Kafka as the underlying log. It's in principle what Martin Kleppmann suggests. As I also want to use a distributed log in the future: Do you know Apache Pulsar or the underlying BookKeeper, which I think was especially made for a distributed WAL? I might either want to use Apache BookKeeper or Apache Pulsar for a distributed log for scaling my Open Source temporal database, too. Further…

I've been itching for a while to build something on top of BookKeeper's DistributedLog. I suggest ditching the Pulsar and just using the DL.

Re: Building a Relational Database Using Kafka

#19

Post by lmm » Mon, Sep 23, 2019, 7:03 PM UTC

xmpir wrote: post #4

Why would I want a relational database living in Kafka? MySQL or Postgres are great at what they do.

A lot of the low-level behaviour is hard to control, at least in terms of having a well-known public interface. E.g. transaction isolation level is database-global, limited control over when updates to indices happen, limited direct control over MVCC. The internals of MySQL or Postgres look a lot like Kafka, but the event-transforming parts are hidden inside a black box.

Re: Building a Relational Database Using Kafka

#20

Post by lichtenberger » Mon, Sep 23, 2019, 7:21 PM UTC

eternalban wrote: post #18

lichtenberger wrote: post #15

I like the idea, to use Apache Kafka as the underlying log. It's in principle what Martin Kleppmann suggests. As I also want to use a distributed log in the future: Do you know Apache Pulsar or the underlying BookKeeper, which I think was especially made for a distributed WAL? I might either want to use Apache BookKeeper or Apache Pulsar for a distributed log for scaling my Open Source temporal database, too. Further…

I've been itching for a while to build something on top of BookKeeper's DistributedLog. I suggest ditching the Pulsar and just using the DL.

I've already written with one of the core committers to BookKeeper and yes, I think I'll use it.

That said, I'm always looking for users and contributors to https://sirix.io/ or https://github.com/sirixdb/sirix. That would be super awesome, but I'm sure you have your own ideas already for using BookKeeper.

I'm currently not sure if I first want to build a frontend (I'm a backend engineer, but would like to learn some TypeScript along with using Vue.js and D3js), to interact with SirixDB and to build interactive visualizations to compare revisions of JSON- or XML-resources in SirixDB (stored in a binary format of course highly optimized for space-efficient snapshots).

I think as I'm lacking users it might be more useful, but for sure I'm at least as eager to put forth the idea of scalable SirixDB databases :-)

So, I'd like to use BookKeeper most probably (single writer, read your own write consistency, using synchronous -- for a quorum -- and asynchronous replication for the rest, exactly once semantics...). The thing I don't like is that we also need ZooKeeper, but yeah.

BTW: Why do you think BookKeeper is better than Kafka for this purpose? :-)