Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Limbo: A complete rewrite of SQLite in Rust

turso.tech

Post reply on HN
Print view View original

21–30 of 238 posts

Re: Limbo: A complete rewrite of SQLite in Rust

#21

Post by cryptonector »

danieljanes wrote: post #12

Given the code quality and rigid testing, SQLite is probably the last project that should be rewritten. It'd be great to see all other C code rewritten first!

That was my take when LibSQL was announced. And it still is and would be my take if LibSQL remains C-coded. But a Rust-coded rewrite of SQLite3 or LibSQL is a different story.

The SQLite3 business model is that SQLite3 is open source but the best test suite for it is proprietary, and they don't accept contributions to any of either. This incentivizes anyone who needs support and/or new features in SQLite3 to join the SQLite Consortium. It's a great business model -- I love it. But there are many users who want more of a say than even being a consortium member would grant them, and they want to contribute. For those users only a fork would make sense. But a fork would never gain much traction given that test suite being proprietary, and the SQLite3 team being so awesome.

However, a memory-safe language re-implementation of SQLite3 is a very different story. The U.S. government wants everyone to abandon C/C++ -- how will they do this if they depend on SQLite3? Apart from that there's also just a general interest and need to use memory-safe languages.

That said, you're right that there are many other projects that call for a rewrite in Rust way before SQLite3. The thing is: if you have the need and the funding, why wouldn't you rewrite the things you need first? And if SQLite3 is the first thing you need rewritten, why not?

Re: Limbo: A complete rewrite of SQLite in Rust

#22

Post by avinassh »

Ne02ptzero wrote: post #17
AlotOfReading wrote: post #7

Earlier quoted context omitted.

The benchmarks in the post indicate that Limbo is more performant than SQLite, not less.

> Executing cargo bench on Limbo’s main directory, we can compare SQLite running SELECT * FROM users LIMIT 1 (620ns on my Macbook Air M2), with Limbo executing the same query (506ns), which is 20% faster. Faster on a single query, returning a single result, on a single computer. That's not how database performance should be measured or compared. In any case, the programming language should have little to no impact on…

> In any case, the programming language should have little to no impact on the database performance, since the majority of the time is spent waiting on io anyway

may be! However, Rust makes some things easier. It is also easy to maintain and reiterate

Re: Limbo: A complete rewrite of SQLite in Rust

#23

Post by cogman10 »

cryptonector wrote: post #9

When the initial SQLite3->LibSQL fork was announced I was pretty negative about it because SQLite3 has a wonderful, 100% branch coverage test suite that is proprietary , and so without access to that any fork would be bound to fail. However, if there's a big product behind the fork, and even better, a rewrite to a memory-safe language, then the fork begins to make a lot of sense. So, hats off to y'all for pulling thi…

Good luck for sure, but browsing their compatibility matrix, it looks like they are a LONG way off. By the looks of it, they have mostly read compatibility with little write capabilities (no alter table, for example).

Re: Limbo: A complete rewrite of SQLite in Rust

#24

Post by cryptonector »

xyc wrote: post #18
cryptonector wrote: post #13

Earlier quoted context omitted.

I do see a need for multiple implementations of SQLite3. First there's the need for multiple implementations for the reasons given by the LibSQL folks, second there's the need for a memory-safe language implementation of SQLite3, and third there's the need for a native language implementation for languages whose runtimes really want not to have C involved (e.g., Go).

The fact that there's no alternative implementation of SQLite also seems to play a part in preventing standardization of WebSQL. https://www.w3.org/TR/webdatabase/ "The specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple independent implementations to proceed along a standardisation path."

Indeed! This sort of thing is a problem. It's the same with Internet protocols: you need at least two implementations to get to Standard.

Re: Limbo: A complete rewrite of SQLite in Rust

#25

Post by cryptonector »

cogman10 wrote: post #23
cryptonector wrote: post #9

When the initial SQLite3->LibSQL fork was announced I was pretty negative about it because SQLite3 has a wonderful, 100% branch coverage test suite that is proprietary , and so without access to that any fork would be bound to fail. However, if there's a big product behind the fork, and even better, a rewrite to a memory-safe language, then the fork begins to make a lot of sense. So, hats off to y'all for pulling thi…

Good luck for sure, but browsing their compatibility matrix, it looks like they are a LONG way off. By the looks of it, they have mostly read compatibility with little write capabilities (no alter table, for example).

As long as they have the funding I'm sure they can get there.

Re: Limbo: A complete rewrite of SQLite in Rust

#26

Post by avinassh »

edweis wrote: post #6

I am not sure how feasible it is, but can't SQLite be partially rewritten step by step on the main branch instead of being forked? As the article mentioned, a complete rewrite will not be as stable as the original.

As other commenters have already pointed out, SQLite does not take outside contributions.

We already have a fork, called libSQL. However, the goals of Limbo are far more ambitious and we cannot rewrite some parts step by step. We want to have DST ( Deterministic Simulation Testing), a testing methodology pioneered by Foundation DB and TigerBeetle. It is not easy to do that in an existing codebase

Re: Limbo: A complete rewrite of SQLite in Rust

#27

Post by AlotOfReading »

jsheard wrote: post #15
AlotOfReading wrote: post #7

Earlier quoted context omitted.

The benchmarks in the post indicate that Limbo is more performant than SQLite, not less.

Related: https://old.reddit.com/r/rust/comments/1ha7uyi/memorysafe_pn... C libraries aren't automatically the fastest option, there's a lot of C code which has stagnated on the performance front but is still widely used because it's battle tested and known to be robust by C standards.

There's still an element of truth in the idea that C is going to be faster by default. There's simply a much lower bar to writing fast (and unsafe) C. Fast Rust demands considerably more thoughtfulness from the programmer (at least for me).

Re: Limbo: A complete rewrite of SQLite in Rust

#28

Post by galangalalgol »

cryptonector wrote: post #21
danieljanes wrote: post #12

Given the code quality and rigid testing, SQLite is probably the last project that should be rewritten. It'd be great to see all other C code rewritten first!

That was my take when LibSQL was announced. And it still is and would be my take if LibSQL remains C-coded. But a Rust-coded rewrite of SQLite3 or LibSQL is a different story. The SQLite3 business model is that SQLite3 is open source but the best test suite for it is proprietary, and they don't accept contributions to any of either. This incentivizes anyone who needs support and/or new features in SQLite3 to join the…

Also the very rigid testing makes the rewrite a lot easier to validate.

Re: Limbo: A complete rewrite of SQLite in Rust

#30

Post by gwbas1c »

danieljanes wrote: post #12

Given the code quality and rigid testing, SQLite is probably the last project that should be rewritten. It'd be great to see all other C code rewritten first!

From https://news.ycombinator.com/item?id=42379402

> It uses Asynchronous I/O, considers WASM as first class, and has Deterministic Simulation Testing support from the beginning.

These are all very hard to do in straight C if your goal is to program in Rust.

Post reply on HN