Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Limbo: A complete rewrite of SQLite in Rust

turso.tech

Post reply on HN
Print view View original

61–70 of 238 posts

Re: Limbo: A complete rewrite of SQLite in Rust

#61

Post by wis »

masklinn wrote: post #41
danieljanes wrote: post #12

Given the code quality and rigid testing, SQLite is probably the last project that should be rewritten. It'd be great to see all other C code rewritten first!

Seems like a potentially interesting project to get rid of sqlite's compatibility baggage e.g. non-strict tables, opt-in foreign keys, the oddities around rowid tables, etc... as well as progress the dialect a bit (types and domains for instance).

But the article mentions that they intend to have full compatibility:

  > Our goal is to build a reimplementation of SQLite from scratch, fully compatible at the language and file format level, with the same or higher reliability SQLite is known for, but with full memory safety and on a new, modern architecture.

Re: Limbo: A complete rewrite of SQLite in Rust

#62

Post by pkolaczk »

Ne02ptzero wrote: post #17
AlotOfReading wrote: post #7

Earlier quoted context omitted.

The benchmarks in the post indicate that Limbo is more performant than SQLite, not less.

> Executing cargo bench on Limbo’s main directory, we can compare SQLite running SELECT * FROM users LIMIT 1 (620ns on my Macbook Air M2), with Limbo executing the same query (506ns), which is 20% faster. Faster on a single query, returning a single result, on a single computer. That's not how database performance should be measured or compared. In any case, the programming language should have little to no impact on…

> In any case, the programming language should have little to no impact on the database performance, since the majority of the time is spent waiting on io anyway

That was true maybe 30 years ago with spinning disks and 100 mbit ethernet. Currently, with storage easily approaching speeds of 10 GB/s and networks at 25+ Gbit/s it is quite hard to saturate local I/O in a database system. Like, you need not just a fast language (C, C++, Rust) but also be very smart about how you write code.

Re: Limbo: A complete rewrite of SQLite in Rust

#63

Post by _ugfj »

jey wrote: post #35
cogman10 wrote: post #23

Earlier quoted context omitted.

Good luck for sure, but browsing their compatibility matrix, it looks like they are a LONG way off. By the looks of it, they have mostly read compatibility with little write capabilities (no alter table, for example).

That's fully in line with what they're announcing here. It's the announcement of a new project that has passed the prototyping stage, but one that has not reached the 1.0 stage.

And it never will

The first 90% is easy, it's the second 90% that is very hard.

Re: Limbo: A complete rewrite of SQLite in Rust

#64

Post by pkolaczk »

glommer wrote: post #48
Ne02ptzero wrote: post #17

Earlier quoted context omitted.

> Executing cargo bench on Limbo’s main directory, we can compare SQLite running SELECT * FROM users LIMIT 1 (620ns on my Macbook Air M2), with Limbo executing the same query (506ns), which is 20% faster. Faster on a single query, returning a single result, on a single computer. That's not how database performance should be measured or compared. In any case, the programming language should have little to no impact on…

The goal here is not to claim that it is faster, though (it isn't, in a lot of other things it is slower and if you run cargo bench you will see) It is to highlight that we already reached a good level of performance this early in the project. Your claim about the programming language having no impact is just false, though. It's exactly what people said back in 2015 when we released Scylla. It was already false then,…

> It is to highlight that we already reached a good level of performance this early in the project.

This is the right thing to do. It's a pity so many projects don't keep an eye on performance from the very first day. Getting high performing product is a process, not a single task you apply at the end. Especially in a performance critical system like a database, if you don't pay attention to performance and instead you delay optimizing till the end, at the end of the day you'll need to do a major rewrite.

Re: Limbo: A complete rewrite of SQLite in Rust

#65

Post by wmf »

bruce511 wrote: post #59
cryptonector wrote: post #21

Earlier quoted context omitted.

That was my take when LibSQL was announced. And it still is and would be my take if LibSQL remains C-coded. But a Rust-coded rewrite of SQLite3 or LibSQL is a different story. The SQLite3 business model is that SQLite3 is open source but the best test suite for it is proprietary, and they don't accept contributions to any of either. This incentivizes anyone who needs support and/or new features in SQLite3 to join the…

>> The SQLite3 business model is that SQLite3 is open source This is going to sound pedantic, but SQLite is not Open Source. It's Public Domain. The distinction is subtle, but it is important.

Public domain is a form of open source.

Re: Limbo: A complete rewrite of SQLite in Rust

#66

Post by simonw »

glommer wrote: post #54
simonw wrote: post #39

Clearly still in very early days: uv run --with pylimbo --python 3.13 python Then: >>> import limbo >>> con = limbo.connect("/tmp/content.db") thread ' ' panicked at core/schema.rs:186:18: not yet implemented: Expected CREATE TABLE statement note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace With that environment variable: stack backtrace: 0: _rust_begin_unwind 1: core::panicking::panic_fmt…

Hey Simon! However early you think it is, I can guarantee it is even earlier =) If this is just a standard sqlite database that you are trying to open, though, I'd have expected it to work.

It's this database from here: https://datasette.io/content.db - it uses the SQLite FTS extension though so it's not surprising there was something in there that caused problems!

Re: Limbo: A complete rewrite of SQLite in Rust

#68

Post by encoderer »

nine_k wrote: post #52
galangalalgol wrote: post #28

Earlier quoted context omitted.

Also the very rigid testing makes the rewrite a lot easier to validate.

...as long as you can persuade the keepers of the proprietary test suite to agree to run it against your code.

Even without that, it’s helpful. It means there is less (no?) undefined behavior that you will need to emulate to maintain compatibility. You can just follow the spec.

Re: Limbo: A complete rewrite of SQLite in Rust

#69

Post by wgjordan »

> For maximum performance, users have to choose WAL mode over journal mode, disable POSIX advisory locks, etc.

Speaking of "wal" mode, is "wal2" mode [1] on your radar for this project to prevent wal files from growing indefinitely in a busy system?

[1] https://sqlite.org/cgi/src/doc/wal2/doc/wal2.md

Re: Limbo: A complete rewrite of SQLite in Rust

#70

Post by thih9 »

bruce511 wrote: post #59
cryptonector wrote: post #21

Earlier quoted context omitted.

That was my take when LibSQL was announced. And it still is and would be my take if LibSQL remains C-coded. But a Rust-coded rewrite of SQLite3 or LibSQL is a different story. The SQLite3 business model is that SQLite3 is open source but the best test suite for it is proprietary, and they don't accept contributions to any of either. This incentivizes anyone who needs support and/or new features in SQLite3 to join the…

>> The SQLite3 business model is that SQLite3 is open source This is going to sound pedantic, but SQLite is not Open Source. It's Public Domain. The distinction is subtle, but it is important.

Details if anyone is interested: https://opensource.org/blog/public-domain-is-not-open-source
Post reply on HN