Is there any big open soure, long term, community contributed, in rust?
Edit: Is there any big open source project, long term, community contributed, in rust?
Limbo: A complete rewrite of SQLite in Rust
221–230 of 238 posts
Re: Limbo: A complete rewrite of SQLite in Rust
#222sqlite3 is 1.6MB while limbo is 6MB, size matters for many low-end but huge-volume embedded boards.
Re: Limbo: A complete rewrite of SQLite in Rust
#223Earlier quoted context omitted.
The test code of sqlite is not public.
Yes and no. Part of it is public, just not the "best" part: https://www.sqlite.org/testing.html
Thanks for the link. It looks like the public part is 27k lines of code (vs the 92,000k lines of code in the proprietary closed-source part).
Re: Limbo: A complete rewrite of SQLite in Rust
#224Earlier quoted context omitted.
Why do you believe that? I'm guessing you're thinking about in-browser or sandboxed lambdas?
Local-first primarily.
Ok I understand that, but why wasm size? Local-first works pretty well for a shared library, and it can already be compiled to wasm. But you're predicting based on the _size_ of the wasm bundle being the determining factor which is a really interesting opinion so are you able to explain that?
I could understand if you said 'the fastest' or 'the safest' but 'the smallest' is what I'm hung up on
Re: Limbo: A complete rewrite of SQLite in Rust
#225Earlier quoted context omitted.
Local-first primarily.
Ok I understand that, but why wasm size? Local-first works pretty well for a shared library, and it can already be compiled to wasm. But you're predicting based on the _size_ of the wasm bundle being the determining factor which is a really interesting opinion so are you able to explain that? I could understand if you said 'the fastest' or 'the safest' but 'the smallest' is what I'm hung up on
I'm biased for sure, but the biggest thing keeping me from using sqlite or pglite in the browser is the size of the WASM payloads. They dwarf every other part of a well designed app, at least for the simple things I like to build.
Re: Limbo: A complete rewrite of SQLite in Rust
#226Earlier quoted context omitted.
Ok I understand that, but why wasm size? Local-first works pretty well for a shared library, and it can already be compiled to wasm. But you're predicting based on the _size_ of the wasm bundle being the determining factor which is a really interesting opinion so are you able to explain that? I could understand if you said 'the fastest' or 'the safest' but 'the smallest' is what I'm hung up on
I'm biased for sure, but the biggest thing keeping me from using sqlite or pglite in the browser is the size of the WASM payloads. They dwarf every other part of a well designed app, at least for the simple things I like to build.
Ah I see! If your use case was read-only you might be able to cut out some of the binary size, but it sounds like Web SQL would've been what you need unfortunately
Re: Limbo: A complete rewrite of SQLite in Rust
#227Earlier quoted context omitted.
That was my take when LibSQL was announced. And it still is and would be my take if LibSQL remains C-coded. But a Rust-coded rewrite of SQLite3 or LibSQL is a different story. The SQLite3 business model is that SQLite3 is open source but the best test suite for it is proprietary, and they don't accept contributions to any of either. This incentivizes anyone who needs support and/or new features in SQLite3 to join the…
> The U.S. government wants everyone to abandon C/C++ That's the position of two federal agencies, namely, FBI and CISA. They don't describe how this change will reduce CVEs or why the languages they prefer still produce projects with CVEs. I don't particularly hold the technical or social acumen of FBI or CISA in particularly high regard and I'm not sure why anyone would by default either. Mostly because they say th…
Well Google's Rust report is already out with great results.. so..
Re: Limbo: A complete rewrite of SQLite in Rust
#228Earlier quoted context omitted.
author here: fully compatible at the language and file format level. Further down the post I actually call out explicitly that we do intend to get rid of some of the baggage.
If you "intend to get rid of some of the baggage" you won't be fully compatible. libSQL already isn't fully compatible: as soon as you add a RANDOM ROWID table, you get "malformed database schema" when using the (e.g.) sqlite3 shell to open your file (also Litestream doesn't work, etc). And that's fine, as there probably is no better way of doing what you needed to do. But it's also taking what SQLite offers and brea…
That's fine though. Full compatibility doesn't have to mean full backwards compatibility. I think of it as what's Typescript to Javascript.
Re: Limbo: A complete rewrite of SQLite in Rust
#229Earlier quoted context omitted.
You can get more total IO throughput (at the cost of latency) by queueing up multiple reads and writes concurrently. You can do this with threads, but io_uring should theoretically go faster (but don't take my word for it, let's wait for benchmarks). I'm personally interested in the potential for async bindings for Python. Making fast async wrappers for blocking APIs in Python-land is painful (although it might impro…
They had been talking about making the high-level interface to sqlite async (sqlite3_step()). With io_uring you're talking about the low-level, where blocks are actually read and written. As-is, sqlite is agnostic on that point. It doesn't do I/O directly, but uses an OS abstraction layer, called VFS. VFS implementations for common platforms are built-in, but you can create your own that handles storage IO any way yo…
The VFS interface is synchronous, I don't see how a custom VFS could meaningfully implement asynchronous IO.
> Well, it's perfectly possible to do that with the current sqlite.
If you want to wrap a blocking API in python, with actual parallelism, you have to use multiple processes with communication between them. The main advantage of sqlite in the first place is that it's in-process, and you'd lose that.
Re: Limbo: A complete rewrite of SQLite in Rust
#230Earlier quoted context omitted.
If you "intend to get rid of some of the baggage" you won't be fully compatible. libSQL already isn't fully compatible: as soon as you add a RANDOM ROWID table, you get "malformed database schema" when using the (e.g.) sqlite3 shell to open your file (also Litestream doesn't work, etc). And that's fine, as there probably is no better way of doing what you needed to do. But it's also taking what SQLite offers and brea…
That's fine though. Full compatibility doesn't have to mean full backwards compatibility. I think of it as what's Typescript to Javascript.
Once you compile your Typescript to Javascript, Javascript runtimes can run it, Javascript code can call it, etc. Even source maps work.
Once you start using libSQL features, SQLite tools will simply stop working with your databases.
That means the sqlite3 shell stops working, backup solutions like Litestream and sqlite-rsync stop working, SQLite GUIs like SQLiteStudio stop working, forensic and data recovery tools start giving will have a harder time working, etc.
Maybe it's all worth it, but it's not full compatibility, and it should at least be documented.