211–220 of 238 posts
Earlier quoted context omitted.
Well, Java during its initial life was controlled to a degree through the control of the tests: https://en.wikipedia.org/wiki/Technology_Compatibility_Kit .
Huh, I wasn't aware that Java was initially open source
I'm not buying the rationale in the "async IO" section. First, there's no need to rewrite anything to add an async interface to sqlite if you want (many clients do, whether local or remote). The issue with sqlite's synchronous interface is leaving a thread idle while you wait for IO. But I wonder how much of an issue that really is. sqlite is designed to run very locally to the storage, and can make use of native fil…
I'm personally interested in the potential for async bindings for Python. Making fast async wrappers for blocking APIs in Python-land is painful (although it might improve in the future with nogil).
Earlier quoted context omitted.
But the article mentions that they intend to have full compatibility: > Our goal is to build a reimplementation of SQLite from scratch, fully compatible at the language and file format level, with the same or higher reliability SQLite is known for, but with full memory safety and on a new, modern architecture.
author here: fully compatible at the language and file format level. Further down the post I actually call out explicitly that we do intend to get rid of some of the baggage.
libSQL already isn't fully compatible: as soon as you add a RANDOM ROWID table, you get "malformed database schema" when using the (e.g.) sqlite3 shell to open your file (also Litestream doesn't work, etc).
And that's fine, as there probably is no better way of doing what you needed to do. But it's also taking what SQLite offers and breaking the ecosystem, under the covers of "we're compatible" without ever calling out what compromises are being made.
Note how the SQLite documentation that introduces STRICT tables very clearly documents the backwards compatibility issues of the feature and how to get around them: https://sqlite.org/stricttables.html#backwards_compatibility
You also never got round to documenting the internal Virtual WAL APIs you exposed. This is something where SQLite is lacking, where you could've made an impact without any compatibility issues, and pressure upstream to release something by doing it first/better. Alas, you did it for Turso's exclusive benefit.
Earlier quoted context omitted.
If I were to fork SQLite, drawing users away from the main project would be a non-goal. The goal would be to get strict tables and foreign key constraints enforced 100% of the time.
Yeah, I would assume that any project like this would strive to be a soft fork that just has a few minimal patches to address specific needs, not something that actually tries to compete with the original.
Compatibility for libSQL is a one way street. I don't expect Limbo to be any different.
Earlier quoted context omitted.
Here is the STRICT table type page: https://www.sqlite.org/stricttables.html It is fairly straightforward: you just have to add STRICT to your table definition and you have it. And the FOREIGN KEY support is here: https://www.sqlite.org/foreignkeys.html The two requirements are that your build not have it disabled, and that you execute `PRAGMA foreign_keys = ON;` when you open the database (every time you open the da…
I don't view opt-in as a very good defaults for these.
As for STRICT: if you make your tables STRICT, there's no opt-out.
So why is this an issue? Do you want them to break the file format to say "from this version forward, all tables are STRICT"? What does that really buy you?
It's an embed database: anyone who can mess with your database and circumvert integrity can also open the file and corrupt it.
Earlier quoted context omitted.
I don't view opt-in as a very good defaults for these.
Then build with SQLITE_DEFAULT_FOREIGN_KEYS=1 to make it opt-out (and to opt-out you'd need to inject SQL). As for STRICT: if you make your tables STRICT, there's no opt-out. So why is this an issue? Do you want them to break the file format to say "from this version forward, all tables are STRICT"? What does that really buy you? It's an embed database: anyone who can mess with your database and circumvert integrity…
It removes a footgun for new users. That's not an insignificant benefit.
Probably not worth the backwards compatibility cost, but it definitely is an issue.
I'm not buying the rationale in the "async IO" section. First, there's no need to rewrite anything to add an async interface to sqlite if you want (many clients do, whether local or remote). The issue with sqlite's synchronous interface is leaving a thread idle while you wait for IO. But I wonder how much of an issue that really is. sqlite is designed to run very locally to the storage, and can make use of native fil…
You can get more total IO throughput (at the cost of latency) by queueing up multiple reads and writes concurrently. You can do this with threads, but io_uring should theoretically go faster (but don't take my word for it, let's wait for benchmarks). I'm personally interested in the potential for async bindings for Python. Making fast async wrappers for blocking APIs in Python-land is painful (although it might impro…
With io_uring you're talking about the low-level, where blocks are actually read and written.
As-is, sqlite is agnostic on that point. It doesn't do I/O directly, but uses an OS abstraction layer, called VFS. VFS implementations for common platforms are built-in, but you can create your own that handles storage IO any way you like, including queuing reads and writes concurrently using io_uring.
So that's not a reason to rewrite sqlite.
(In fact, I'd be surprised if they weren't looking at io_uring, and, if it seemed likely to generally improve performance, to provide an option to use it, either in the existing linux-vfs or in some other way.)
> I'm personally interested in the potential for async bindings for Python.
Well, it's perfectly possible to do that with the current sqlite. It may be painful, as you say, but not even remotely at the level of pain a complete rewrite entails.
disclosure: I work here. I am happy to answer any questions tl;dr We are rewriting SQLite in Rust. It uses Asynchronous I/O, considers WASM as first class, and has Deterministic Simulation Testing support from the beginning. source: https://github.com/tursodatabase/limbo
How much has Turso raised so far? Rewriting SQLite sounds like it may cost a sum.
Earlier quoted context omitted.
Then build with SQLITE_DEFAULT_FOREIGN_KEYS=1 to make it opt-out (and to opt-out you'd need to inject SQL). As for STRICT: if you make your tables STRICT, there's no opt-out. So why is this an issue? Do you want them to break the file format to say "from this version forward, all tables are STRICT"? What does that really buy you? It's an embed database: anyone who can mess with your database and circumvert integrity…
> What does that really buy you It removes a footgun for new users. That's not an insignificant benefit. Probably not worth the backwards compatibility cost, but it definitely is an issue.
This is easy to do for any project using the amalgamation.