Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a Relational Database Using Kafka

yokota.blog

Post reply on HN
Print view View original

21–30 of 34 posts

Re: Building a Relational Database Using Kafka

#21

Post by mamcx »

xmpir wrote: post #4

Why would I want a relational database living in Kafka? MySQL or Postgres are great at what they do.

I love PG and rdbms in general, but certainly exist SO MANY things that could be improved with RDBMS.

1- RDBMS implement an imperfect view of the relational model. Is like to say Java is the only OO inplemented in the world.

2- SQL is the standard... and is not practical to say to eliminate it... BUT RDBMS are constrained for the subpar language SQL is. Modern additions are nice... but that is to say that postcript is a nice way to do apps.

Also, a much better language could expand the role of RDBMS.

This is not weird. I live this way when working in FoxPro:

- UI on fox lang. - DB on fox lang. - Reports on fox lang. - Scripting on fox lang. - Web on fox lang - Triggers on fox lang - OO on fox lang.

And all that still with clean separation of logic and components.

3- RDBMS were made by a certain mindset and use case of the 80s. Still so good that can be valid today, but it could get extended. JSON support is just a tiny example of that.

4- Why you can't do "SELECT .. FROM index"? Why you need to create a table to get an index or FTS? Is weird limitation.

The relational model is fine too for KV stores.

----

"Relational database" is a database made on top of the relational MODEL. You could get MANY IMPLEMENTATIONS of that with different use cases. Similar how the functional MODEL and the OO MODEL is not frozen with a single language... that is not that good for app development.

With a better lang, you could eliminate massive ORMs in a nice way!

P.D: Remember, this IS NOT teory. Is proven. This is how was with the dbase family of langs!

Re: Building a Relational Database Using Kafka

#22

Post by rad_gruchalski »

lichtenberger wrote: post #20
eternalban wrote: post #18

Earlier quoted context omitted.

I've been itching for a while to build something on top of BookKeeper's DistributedLog. I suggest ditching the Pulsar and just using the DL.

I've already written with one of the core committers to BookKeeper and yes, I think I'll use it. That said, I'm always looking for users and contributors to https://sirix.io/ or https://github.com/sirixdb/sirix . That would be super awesome, but I'm sure you have your own ideas already for using BookKeeper. I'm currently not sure if I first want to build a frontend (I'm a backend engineer, but would like to learn som…

> BTW: Why do you think BookKeeper is better than Kafka for this purpose? :-)

One of the reasons is that your storage is not tied to ingest. With Kafka, your broker sits where the data sits. That has advantages and disadvantages. The disadvantages are that it's really difficult to do unlimited history topics / partitions in Kafka edit for significant volume topics. You can have only so much storage and only so many disks attached to your machine. It's not possible to add volumes at runtime. I could be wrong but I think Kafka still requires a broker restart after adding volumes to the server properties.

With BookKeeper, you just add nodes with more storage. BookKeeper tracks who was the owner of that partition at the time of write. It's possible to spread partitions across multiple machines.

Re: Building a Relational Database Using Kafka

#23

Post by outworlder »

I am amazed by the number of negative comments this article is getting.. on Hacker News of all places.

I don't see the article claiming people should drop PostgreSQL and use this, it's just asking people to give it a try if they are interested.

Much more interesting would be to discuss what capabilities would be different if this were a mature product.

For instance, this sounds interesting:

> One advantage of using Kafka is that multiple servers can all “tail” the same set of topics. This allows multiple KarelDB servers to run as a cluster, with no single-point of failure. In this case, one of the servers will be elected as the leader while the others will be followers (or replicas). When a follower receives a JDBC request, it will use the Avatica JDBC driver to forward the JDBC request to the leader. If the leader fails, one of the followers will be elected as a new leader.

Re: Building a Relational Database Using Kafka

#24

Post by EdwardDiego »

pram wrote: post #10

From personal experience, considering the reliability of Kafka, I’d implement this in reverse. Most relational databases are far more resilient in operation. One idea I’ve had in the past is to implement a Kafka and Zookeeper layer for FoundationDB.

Interesting, what were the failure modes you encountered?

Re: Building a Relational Database Using Kafka

#25

Post by eternalban »

lichtenberger wrote: post #20
eternalban wrote: post #18

Earlier quoted context omitted.

I've been itching for a while to build something on top of BookKeeper's DistributedLog. I suggest ditching the Pulsar and just using the DL.

I've already written with one of the core committers to BookKeeper and yes, I think I'll use it. That said, I'm always looking for users and contributors to https://sirix.io/ or https://github.com/sirixdb/sirix . That would be super awesome, but I'm sure you have your own ideas already for using BookKeeper. I'm currently not sure if I first want to build a frontend (I'm a backend engineer, but would like to learn som…

This is a great review of Pulsar by Jack Vanlightly. If you haven't read it, I highly recommend it:

https://jack-vanlightly.com/blog/2018/10/2/understanding-how...

> The thing I don't like is that we also need BookKeeper, but yeah.

Word on the street is now you can swap the consensus module in ZooKeeper. [update: just re-checked Pulsar's dev channel. Apparently one can use etcd instead of zookeeper.]

> sirixdb

Sirix sounds interesting but is a bit close to home to a (currently dormant) temporal DB project of my own. But the real issue is I usually do significant OSS contribution if I'm on the bench and not working.

Wish you success in your effort!

Re: Building a Relational Database Using Kafka

#26

Post by atombender »

eternalban wrote: post #25
lichtenberger wrote: post #20

Earlier quoted context omitted.

I've already written with one of the core committers to BookKeeper and yes, I think I'll use it. That said, I'm always looking for users and contributors to https://sirix.io/ or https://github.com/sirixdb/sirix . That would be super awesome, but I'm sure you have your own ideas already for using BookKeeper. I'm currently not sure if I first want to build a frontend (I'm a backend engineer, but would like to learn som…

This is a great review of Pulsar by Jack Vanlightly. If you haven't read it, I highly recommend it: https://jack-vanlightly.com/blog/2018/10/2/understanding-how... > The thing I don't like is that we also need BookKeeper, but yeah. Word on the street is now you can swap the consensus module in ZooKeeper. [update: just re-checked Pulsar's dev channel. Apparently one can use etcd instead of zookeeper.] > sirixdb Sirix…

Being able to use Etcd instead of ZooKeeper is huge. Etcd is considerably easier to set up and manage, in my experience; not needing to manage another JVM component is also a big win. Thanks for sharing!

Re: Building a Relational Database Using Kafka

#27

Post by dominotw »

xjoins wrote: post #12
xmpir wrote: post #4

Why would I want a relational database living in Kafka? MySQL or Postgres are great at what they do.

I came here to ask this question. It's a really solid technical article. But apart from the sake of doing it, why would anyone ever want to? I'm not suggesting there's no scenario in which one would want to build a SQL database using Kafka, I'm just asking if anybody knows of one.

> SQL database

what is your definition of sql database?

Re: Building a Relational Database Using Kafka

#28

Post by eternalban »

atombender wrote: post #26
eternalban wrote: post #25

Earlier quoted context omitted.

This is a great review of Pulsar by Jack Vanlightly. If you haven't read it, I highly recommend it: https://jack-vanlightly.com/blog/2018/10/2/understanding-how... > The thing I don't like is that we also need BookKeeper, but yeah. Word on the street is now you can swap the consensus module in ZooKeeper. [update: just re-checked Pulsar's dev channel. Apparently one can use etcd instead of zookeeper.] > sirixdb Sirix…

Being able to use Etcd instead of ZooKeeper is huge. Etcd is considerably easier to set up and manage, in my experience; not needing to manage another JVM component is also a big win. Thanks for sharing!

To be clear, it is BookKeeper that can alternatively use etcd not Pulsar.

https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-sto...

https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed...

Re: Building a Relational Database Using Kafka

#29

Post by atombender »

eternalban wrote: post #28
atombender wrote: post #26

Earlier quoted context omitted.

Being able to use Etcd instead of ZooKeeper is huge. Etcd is considerably easier to set up and manage, in my experience; not needing to manage another JVM component is also a big win. Thanks for sharing!

To be clear, it is BookKeeper that can alternatively use etcd not Pulsar. https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-sto... https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed...

Absolutely. Pulsar looks nice, too, but I think BookKeeper not needing ZK gives it an edge over Kafka.

I have to admit that I'm not entirely clear on what Pulsar adds, given that BookKeeper implements the distributed log. Pulsar looks like it's a slightly higher-level abstraction that gives you a more traditional topic-based pub/sub mssaging API on top of BK, plus things like schemas and "functions"?

From what I can tell, BK supports log streaming just like Kafka, so you can use it to build a pub/sub system if you don't need all of Pulsar's features — right?

Re: Building a Relational Database Using Kafka

#30

Post by eternalban »

atombender wrote: post #29
eternalban wrote: post #28

Earlier quoted context omitted.

To be clear, it is BookKeeper that can alternatively use etcd not Pulsar. https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-sto... https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed...

Absolutely. Pulsar looks nice, too, but I think BookKeeper not needing ZK gives it an edge over Kafka. I have to admit that I'm not entirely clear on what Pulsar adds, given that BookKeeper implements the distributed log. Pulsar looks like it's a slightly higher-level abstraction that gives you a more traditional topic-based pub/sub mssaging API on top of BK, plus things like schemas and "functions"? From what I can…

Pulsar is built on top of BK. BK has it own consensus mechanism (ZK, etcd), and Pulsar also has its own (low traffic) consensus via ZK. The article cited above does a very good job of addressing the advantages of Pulsar's architecture over Kafka's. Pulsar also provides for user defined functions (that can run on the broker). This is reminiscent of Postgres's User Defined Functions.

BK provides ledgers, DistributedLog provides log semantics on top of those ledgers. Pulsar builds a full blown messaging/streaming system on top of that.

Per this thread's OP, it seems to make more sense to use BK/DL directly if you want a WAL component to build a DB.

Post reply on HN