Live data from Hacker News

Deep dive into Turso, the “SQLite rewrite in Rust”

kerkour.com

131–140 of 154 posts

Re: Deep dive into Turso, the “SQLite rewrite in Rust”

#131

Earlier quoted context omitted.

> 2) They have a paid cloud option to drive income from: I’ve been confused by this for a while. What is it competing with? Surely not SQLite, being client server defeats all the latency benefits. I feel it would be considered as an alternative to cloud Postgres offerings, and it seems unlikely they could compete on features. Genuinely curious, but is there any sensible use case for this product, or do they just catc…

The article talks about this. If you have a project that starts small and an in-process DB is fine, but you end up needing to scale up then you don't have to switch DBs.

That's a valid, but very tiny, use case.

After all, if you can tell in advance that you might hit the limits of SQLite, you'd simply start with postgresql on day one, not with a new unproven DB vendor with a product that has been through the trial by fire of existing DBs.

Re: Deep dive into Turso, the “SQLite rewrite in Rust”

#132

Earlier quoted context omitted.

IMHO breaking free of SQLite's proprietary test suite is a bigger driver than C vs Rust. Turso's Limbo announcement says exactly that: they couldn't confidently make large architectural changes without access to the tests. The rewrite lets them build Deterministic Simulation Testing from scratch, which they argue can exceed SQLite's reliability by simulating unlikely scenarios and reproducing failures deterministical…

Having seen way too many "we're going to rewrite $xyz but make it BETTERER!!", I don't give this one much chance of success. SQLite is a high-quality product with a quarter-century of development history and huge amounts of effort, both by the devs and via public use, of testing. So this let's-reinvent-it-in-Rust effort will have to beat an already very good product that's had a staggering amount of development effor…

How do you want to define success for this project relative to SQLite? Because they already have concurrent writes working for their rust implementation. It's currently marked experimental, but it does already work. And for a lot of people, that's all they want or need.

https://turso.tech/blog/beyond-the-single-writer-limitation-...

Re: Deep dive into Turso, the “SQLite rewrite in Rust”

#133

Earlier quoted context omitted.

In other words, they are creating their own database and hitching on to the SQLite brand to market it. (That's fine though).

I don't think that's fine at all, it's quite a shitty thing to do hoenstly and I'm not surprised it's a VC backed company doing it.

How would you do it then?

Re: Deep dive into Turso, the “SQLite rewrite in Rust”

#135
post #74

Earlier quoted context omitted.

Without the test suite isn’t even more likely to have stability problems?

Not likely. The alternative was for them to modify SQLite without the test suite and no obvious indication of what they would need to do to try to fill in the gaps. Modifying SQLite with its full test suite would be the best choice, of course, but one that is apparently[1] not on the table for them. Since they have to reimagine the test suite either way, they believe they can do a better job if the tests are written…

I don't get this. In their own rust implementation they have to write and use their own test and they still don't have access to the proprietary sqlite tests. So their implementation will necessarily be whatever they implement + whatever passes their tests. Same as it would be if they forked sqlite in C. (Plus they would have the open source tests). Am I missing something?

Re: Deep dive into Turso, the “SQLite rewrite in Rust”

#136
post #18

I’d imagine this will go a bit like the rust rewrite of sudo etc. Despite the memory safety advantages at least towards the start it still ends up more fragile because the incumbent has years of testing and fixing behind it

Of all the projects which may benefit from a rewrite or re-imagining in a memory-safe language, I'm really puzzled why it's heavily-tested, near-universally-deployed software such as sudo (use oBSD doas instead?), the coreutils, and sqlite.

Doas supports a subset of sudo functionality by design. Your comment is exactly what I said when I first heard about the rust linux utils thing. The best they can do is have new bugs.

Re: Deep dive into Turso, the “SQLite rewrite in Rust”

#138

This is very shallow for a supposed deep dive. I'm not ready to entertain Turso as an alternative to something that is as battle tested as Sqlite.

Taking feature lists and plans at face value is offensively shallow; the typical Rust fan arrogance pattern can be an explanation (if the Rust rewrite is "better", it doesn't have to be compatible with the rest of the world who uses the actual C SQLite).

Re: Deep dive into Turso, the “SQLite rewrite in Rust”

#139

Earlier quoted context omitted.

where do you see these opportunities? i didnt see a lot of issues personally rust would be better at than C in this domain. care to elaborate? (genuinely curious!) personally i see more benefit in rust for example as ORM and layers that talk to the database. (those are often useful to have in such an ecossystem so you can use the database safe and sanely, like python or so but then u know, fast and secure.)

You need to be crazy to use an ORM. I personally think that even SQL is redundant. I would like to see a high quality embedded database written in Rust.

Yep, exactly this.

It's painful having to switch to another language to talk to the database, and ORMs are the worst kind of leaky abstractions. With Rust, we've finally got a systems language that's expressive enough to do a really good job with the API to an embedded database.

The only thing that's really missing is language support for properly ergonomic Cap'n Proto support - Swift has stuff in this vein already. That'd mean serializable ~native types with no serialization/deserialization overhead, and it's applicable to a lot of things; Swift developed the support so they could do proper dynamically linked libraries (including handling version skew).

If I might plug my project yet again (as if I don't do that enough :) - bcachefs has a high quality embedded database at its core, and one of the dreams has always been to turn that into a real general purpose database. Much of the remaining stuff for making it truly general purpose is stuff that we're going to want sooner or later for the filesystem anyways, and while it's all C today I've done a ton of work on refactoring and modernizing the codebase to hopefully make a Rust conversion tractable, in the not too distant future.

(Basically, with the cleanup attribute in modern C, you can do pseudo RAII that's good enough to eliminate goto error handling in most code. That's been the big obstacle to transitioning a C codebase to be "close enough" to what the Rust version would look like to make the conversion mostly syntactic, not a rewrite, and that work is mostly done in bcachefs).

The database project is very pie in the sky, but if the project gets big enough (it's been growing, slowly but steadily), that's the dream. One of them, anyways.

A big obstacle towards codebases that we can continue to understand, maintain and continue to improve over the next 100 years is giant monorepos, and anything we can do to split giant monorepos apart into smaller, cleaner reusable components is pure gold.

Re: Deep dive into Turso, the “SQLite rewrite in Rust”

#140
post #135
post #74

Earlier quoted context omitted.

Not likely. The alternative was for them to modify SQLite without the test suite and no obvious indication of what they would need to do to try to fill in the gaps. Modifying SQLite with its full test suite would be the best choice, of course, but one that is apparently[1] not on the table for them. Since they have to reimagine the test suite either way, they believe they can do a better job if the tests are written…

I don't get this. In their own rust implementation they have to write and use their own test and they still don't have access to the proprietary sqlite tests. So their implementation will necessarily be whatever they implement + whatever passes their tests. Same as it would be if they forked sqlite in C. (Plus they would have the open source tests). Am I missing something?

You are missing that HN accounts needlessly overthink everything, perhaps?

Otherwise, I doubt it. They have to write the tests again no matter what. Given that, there is no downside to reimplementing it while they are at it. All while there is a big upside to doing that: Trying to test something after the implementation is already written never ends well.

That does not guarantee that their approach will succeed. It is hard problem no matter how you slice it. But trying to reverse engineer the tests for the C version now that all knowledge of what went into it in the first place is lost is all but guaranteed to fail. Testing after the fact never ends well. Rewriting the implementation and tests in parallel increases the chances of success.

Post reply on HN