Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small Kafka: Tansu and SQLite on a free t3.micro

blog.tansu.io

Post reply on HN
Print view View original

21–27 of 27 posts

Re: Small Kafka: Tansu and SQLite on a free t3.micro

#21

Post by kitd »

apgwoz wrote: post #9

Any good and honest tansu experience reports out there? Would be nice to understand how “bleeding edge” this actually is, in practice. The idea of a kafka compatible, but trivial to run, system like this is very intriguing!

My thoughts too.

> kafka compatible

Kafka is not a straightforward protocol and has a few odd niches. Not to mention that message formats have changed over the years. Even the base product has recently dropped support for some of the oldest API versions. And there are still plenty of clients out there using old versions of librdkafka (he says from experience).

So I'd be interested how (backward-)compatible they are.

Re: Small Kafka: Tansu and SQLite on a free t3.micro

#22

Post by sdoering »

mjmas wrote: post #8
tombert wrote: post #7

Earlier quoted context omitted.

I love the idea of SQLite, but I actually really dislike using it. I think part of my issue is that a lot of uses of it end up having a big global lock on the database file (see: older versions of Emby/Jellyfin) so you can't use it with multiple threads or processes, but I also haven't really ever find a case to use it over other options. I've never really felt the need to do anything like a JOIN or a UNION when doin…

PRAGMA journal_mode = WAL; And set the busy timeout tunction as well. https://www.sqlite.org/c3ref/busy_timeout.html

Curious, what do you think about

> PRAGMA synchronous = NORMAL;

I am just not experienced enough to form an opinion.

Re: Small Kafka: Tansu and SQLite on a free t3.micro

#23

Post by krsoninikhil »

I love sqlite backed system, one less component to worry about. But when using Tansu with sqlite storage, what are my options for horizonal scaling and keeping Tansu HA?

Also, are there any benchmark on how Tansu with S3 storage would perform in comparison to Kafka or something like WarpStream?

Re: Small Kafka: Tansu and SQLite on a free t3.micro

#24

Post by shortishly »

kitd wrote: post #21
apgwoz wrote: post #9

Any good and honest tansu experience reports out there? Would be nice to understand how “bleeding edge” this actually is, in practice. The idea of a kafka compatible, but trivial to run, system like this is very intriguing!

My thoughts too. > kafka compatible Kafka is not a straightforward protocol and has a few odd niches. Not to mention that message formats have changed over the years. Even the base product has recently dropped support for some of the oldest API versions. And there are still plenty of clients out there using old versions of librdkafka (he says from experience). So I'd be interested how (backward-)compatible they are.

I agree that it isn't straight forward! Tansu uses the JSON protocol descriptors from Apache Kafka, generating ~60k LoC of Rust to represent the structures. It then uses a custom Serde encoder/decoder to implement the protocol: original, flexible and tag buffers formats for every API version (e.g., the 18 just in FETCH). It is based off spending the past ~10 years using Kafka, and writing/maintaining an Erlang client (there are no "good" Kafka clients for Erlang!). It also uses a bunch of collected protocol examples, to encode/decode during the tests. Tansu is also a Kafka proxy, which is also used to feed some of those tests.

Some of the detail: https://blog.tansu.io/articles/serde-kafka-protocol

However, there are definitely cases I am sure where Tansu isn't compatible. For example, Kafka UI (kafbat) reports a strange error when doing a fetch (despite actually showing the fetched data), which I've yet to get to the bottom of.

If you find any compatibility issues, then please raise an issue, and I can take a look.

Re: Small Kafka: Tansu and SQLite on a free t3.micro

#25

Post by shortishly »

krsoninikhil wrote: post #23

I love sqlite backed system, one less component to worry about. But when using Tansu with sqlite storage, what are my options for horizonal scaling and keeping Tansu HA? Also, are there any benchmark on how Tansu with S3 storage would perform in comparison to Kafka or something like WarpStream?

You could use the proxy to spread topics over a number of brokers. The broker and proxy share a number of services and layers, that could be used to route:

https://blog.tansu.io/articles/route-layer-service

My itch for SQLite was smaller scale (and reproducible) environments, e.g., development, test/integration (with a single file to reset the environment). PostgreSQL was intended for "larger scale", with (database level) partitioning of Kafka records on each topic/partition, and replication for leader/follower setups, which might work better for HA. S3 for environments where latency is less of any issue (though with the SlateDB/S3 engine that might change).

S3: Not yet. I've been working through tuning each engine, S3 is next on the list.

Re: Small Kafka: Tansu and SQLite on a free t3.micro

#26

Post by ncb9094 »

nchmy wrote: post #15
ncb9094 wrote: post #11

To me it sounds like NATS Jetstream but with Rust. I wonder what the reliability looks like when it is prod ready

Jetstream isn't kafka-compatible, nor does it have pluggable storage of s3, sqlite, Postgres etc...

I think jetstream storage is about to get s3 api support https://github.com/nats-io/nats-server/discussions/5486 . also you can use bento connector to connect it to any pipeline you could possibly want. It is easy to manage and works great

Re: Small Kafka: Tansu and SQLite on a free t3.micro

#27

Post by nchmy »

ncb9094 wrote: post #26
nchmy wrote: post #15

Earlier quoted context omitted.

Jetstream isn't kafka-compatible, nor does it have pluggable storage of s3, sqlite, Postgres etc...

I think jetstream storage is about to get s3 api support https://github.com/nats-io/nats-server/discussions/5486 . also you can use bento connector to connect it to any pipeline you could possibly want. It is easy to manage and works great

Great to see. Hopefully something comes of it. Thanks for sharing
Post reply on HN