Building a Relational Database Using Kafka • Hacker News Board

Re: Building a Relational Database Using Kafka

#31

Post by kjeetgill » Tue, Sep 24, 2019, 3:55 AM UTC

mLuby wrote: post #8

thinkersilver wrote: post #6

Earlier quoted context omitted.

Great article. I think you might be taking a subtle point in the article for granted. Namely, the ability to assemble a fairly complex distributed system from open source components. Democratisation of complex machinery like the raft and consensus algorithms, schedulers, append logs, query optimisers and so on is a superb thing. It is something that has only been possible in the last 3/4 years and would have been dev…

You make a good counterpoint: it's awesome that this can be done concisely enough these days to fit in a hobby blog post. That's new and demonstrates the approachability of these tools. At the same time, OP is right: a business doing this is making poor engineering decisions.

We don't know that yet! The post explains what they did but not the why of the problem. Maybe they have a good reason, maybe they don't.

But no need to jump to conclusions. I've seen plenty of bs fragile ducktape databases. Some of them exist because because someone wanted to hand-roll everything. And some exist because they were afraid to invent anything and ended up hacking together a bunch of garbage MySQL nodes. Because they didn't want "reinvent the wheel".

Specifics matter!

Re: Building a Relational Database Using Kafka

#32

Post by kjeetgill » Tue, Sep 24, 2019, 4:02 AM UTC

humbleMouse wrote: post #11

thinkersilver wrote: post #6

Earlier quoted context omitted.

Great article. I think you might be taking a subtle point in the article for granted. Namely, the ability to assemble a fairly complex distributed system from open source components. Democratisation of complex machinery like the raft and consensus algorithms, schedulers, append logs, query optimisers and so on is a superb thing. It is something that has only been possible in the last 3/4 years and would have been dev…

I agree my comment may be a knee-jerk reaction, and it is quite a good technical article with clear examples. I also agree that the open source tech used in this solution is very powerful and it is great we are at a point where a solution like this could exist. I love avro schema registries and kafka just as much as the next guy.

So question: is there a good reason to use Avro over protobufs?

Just from a hand wavy aesthetic perspective I like protobufs but I'd like to get a clear picture.

Re: Building a Relational Database Using Kafka

#33

Post by atombender » Tue, Sep 24, 2019, 4:11 AM UTC

eternalban wrote: post #30

atombender wrote: post #29

Earlier quoted context omitted.

Absolutely. Pulsar looks nice, too, but I think BookKeeper not needing ZK gives it an edge over Kafka. I have to admit that I'm not entirely clear on what Pulsar adds, given that BookKeeper implements the distributed log. Pulsar looks like it's a slightly higher-level abstraction that gives you a more traditional topic-based pub/sub mssaging API on top of BK, plus things like schemas and "functions"? From what I can…

Pulsar is built on top of BK. BK has it own consensus mechanism (ZK, etcd), and Pulsar also has its own (low traffic) consensus via ZK. The article cited above does a very good job of addressing the advantages of Pulsar's architecture over Kafka's. Pulsar also provides for user defined functions (that can run on the broker). This is reminiscent of Postgres's User Defined Functions. BK provides ledgers, DistributedLog…

Thanks. Right, I know Pulsar is built on top of BK — it's just unclear how much extra value it adds.

I'm looking at using BK for a WAL-like component for a document store. It does involve multiple (many!) consumers tailing the log, but unlike your typical Kafka setup, there's need for no partitioning.

I was unsure of whether BK provides an API that's nice to use for non-Java languages like Go and Rust, which is what we use. Apache projects written in Java (or Scala), particularly those in the Hadoop ecosystem, tend to be very JVM-centric, and often they're impossible to use from other languages because there's no non-Java API. Kafka is an outlier here, but it certainly started out with a Java-only client and it took years for non-Java clients to mature enough to be usable.

I thought I saw that BK had a REST API, but now I see that it's just an admin API. From what I can tell, it's exactly like every other Apache Java project — there's a big monolithic Java client that's tightly coupled with the server, and to write a client in any other language you'd have to reverse-engineer it and essentially port the whole thing, because it's a "fat client" — it looks like it talks directly to ZooKeeper and "bookies", and has a lot of logic built in. The Bookie protocol looks like it's binary and hand-coded (not gRPC or anything nice). So disappointing! /rant

Edit: Looks like the files I was looking at are generated from Protobuf. So, yay. Unfortunately, that's just the Bookie protocol. The main client is still fat and in Java.

Re: Building a Relational Database Using Kafka

#34

Post by lichtenberger » Tue, Sep 24, 2019, 6:40 AM UTC

atombender wrote: post #33

eternalban wrote: post #30

Earlier quoted context omitted.

Pulsar is built on top of BK. BK has it own consensus mechanism (ZK, etcd), and Pulsar also has its own (low traffic) consensus via ZK. The article cited above does a very good job of addressing the advantages of Pulsar's architecture over Kafka's. Pulsar also provides for user defined functions (that can run on the broker). This is reminiscent of Postgres's User Defined Functions. BK provides ledgers, DistributedLog…

Thanks. Right, I know Pulsar is built on top of BK — it's just unclear how much extra value it adds. I'm looking at using BK for a WAL-like component for a document store. It does involve multiple (many!) consumers tailing the log, but unlike your typical Kafka setup, there's need for no partitioning. I was unsure of whether BK provides an API that's nice to use for non-Java languages like Go and Rust, which is what…

Regarding the question if Apache Pulsar or Apache BookKeeper for me it's not so clear which of both to use. I think it adds a lot of value if you add the WAL to the public API. Forgot how it's called, I think change data types (CDTs) without searching on google!? So basically you have the stream of events and expose these for use by applications :-)