Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small Kafka: Tansu and SQLite on a free t3.micro

blog.tansu.io

Post reply on HN
Print view View original

1–10 of 27 posts

Re: Small Kafka: Tansu and SQLite on a free t3.micro

#2

Post by randito »

Great link. I've always been drawn to sqlite3 just from a simplicity and operational point of view. And with tools like "make it easy to replcate" Litestream and "make it easy to use" sqlite-utils, it just becomes easier.

And one of the first patterns I wanted to use was this. Just a read-only event log that's replicated, that is very easy to understand and operate. Kafka is a beast to manage and run. We picked it at my last company -- and it was a mistake, when a simple DB would have sufficed.

https://github.com/simonw/sqlite-utils https://litestream.io/

Re: Small Kafka: Tansu and SQLite on a free t3.micro

#4

Post by 8organicbits »

Quite cool. 7000 records per second is usable for a lot of projects.

One note on the backup/migrate, I think you need a shared lock on the database before you copy the database. If you dont, the database can corrupt. SQLite docs have other recommendations too:

https://sqlite.org/backup.html

Re: Small Kafka: Tansu and SQLite on a free t3.micro

#7

Post by tombert »

randito wrote: post #2

Great link. I've always been drawn to sqlite3 just from a simplicity and operational point of view. And with tools like "make it easy to replcate" Litestream and "make it easy to use" sqlite-utils, it just becomes easier. And one of the first patterns I wanted to use was this. Just a read-only event log that's replicated, that is very easy to understand and operate. Kafka is a beast to manage and run. We picked it at…

I love the idea of SQLite, but I actually really dislike using it.

I think part of my issue is that a lot of uses of it end up having a big global lock on the database file (see: older versions of Emby/Jellyfin) so you can't use it with multiple threads or processes, but I also haven't really ever find a case to use it over other options. I've never really felt the need to do anything like a JOIN or a UNION when doing local configurations, and for anything more complicated than a local configuration, I likely have access to Postgres or something. I mean, the executable for Postgres is only ten megs or twenty on Linux, so it's not even that much bigger than SQLite for modern computers.

Re: Small Kafka: Tansu and SQLite on a free t3.micro

#8

Post by mjmas »

tombert wrote: post #7
randito wrote: post #2

Great link. I've always been drawn to sqlite3 just from a simplicity and operational point of view. And with tools like "make it easy to replcate" Litestream and "make it easy to use" sqlite-utils, it just becomes easier. And one of the first patterns I wanted to use was this. Just a read-only event log that's replicated, that is very easy to understand and operate. Kafka is a beast to manage and run. We picked it at…

I love the idea of SQLite, but I actually really dislike using it. I think part of my issue is that a lot of uses of it end up having a big global lock on the database file (see: older versions of Emby/Jellyfin) so you can't use it with multiple threads or processes, but I also haven't really ever find a case to use it over other options. I've never really felt the need to do anything like a JOIN or a UNION when doin…

  PRAGMA journal_mode = WAL;
And set the busy timeout tunction as well.

https://www.sqlite.org/c3ref/busy_timeout.html

Re: Small Kafka: Tansu and SQLite on a free t3.micro

#10

Post by nchmy »

apgwoz wrote: post #9

Any good and honest tansu experience reports out there? Would be nice to understand how “bleeding edge” this actually is, in practice. The idea of a kafka compatible, but trivial to run, system like this is very intriguing!

I wonder how it compares to Redpanda
Post reply on HN