Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turso is an in-process SQL database, compatible with SQLite

github.com

Post reply on HN
Print view View original

91–100 of 109 posts

Re: Turso is an in-process SQL database, compatible with SQLite

#91

Post by menaerus »

heliumtera wrote: post #87
menaerus wrote: post #60

Earlier quoted context omitted.

It goes to show how future of SE will look like. Loose reviews (a big win for the industry) with each person capable of doing 10x more work, and basically orchestrating/overlooking the work done by the agents. The bottleneck eventually at some point, if not already, will become a fatigue caused by having to process/acknowledge/understand the sheer volume of code spitted out at speeds that we could never have imagined…

If you are doing loose reviews I guess you're not strictly more productive, because this would be a different kind of work...a more "loose" work, but a scalar I'm volume.

Loose work? No, quality of work being done does not correlate with reviews in almost any way, unless you're a lousy engineer doing lost work anyways. High quality reviews are exceptionally rare.

Re: Turso is an in-process SQL database, compatible with SQLite

#92

Post by namibj »

anentropic wrote: post #73
bawolff wrote: post #48

Earlier quoted context omitted.

I find it crazy that people would ask you in that direction. I would wonder why anyone would chose Turso over SQLite. SQLite is some of the most battle tested software in the world. Turso seems like relatively new software that is still a bit experimental. You do not want your databases to be new or trendy. You want your databases to be solid as a rock.

> I would wonder why anyone would chose Turso over SQLite well, Turso adds features otherwise yeah, there'd be no reason

Ok, `io_uring` (like NVMe but for IO commands from application to kernel) and DBSP (high-grade framework for differential (as in, based on Delta streams/diffs not full updates) compression of "incremental view maintenance", it can keep materialized views synchronously up-to-date with a cost proportional to just the diff (for most typical ones; certain queries can of course be doing things at an intermediate stage that blow up and collapse again right after)).

At least notably; not sure about the MVCC `BEGIN CONCURRENT`'s practical relevance though; I am just already familiar enough with the other two big ones to chime in without having to dive into what Turso does about them...

Re: Turso is an in-process SQL database, compatible with SQLite

#93

Post by tracker1 »

rendaw wrote: post #71
jauntywundrkind wrote: post #58

Earlier quoted context omitted.

Sqlite had such a stellar stellar reputation, for so many excellent reasons. I still find it absolutely freakish & abominable that people are so incredibly touchy & reflexively mean & vile to Turso. I've seen a couple Turso centric YouTube's recently and there are dozens and dozens of up votes for what just seems like the most petulant vacuous reflexive bitter viewed comments, dominating the comments. Sqlite deserves…

I definitely feel this. So many "I made an alternative to X that fixes these issues, or is better in these ways" met with "Well X is fine for me, and I don't need those things, so why change?" These posts are obviously meant for adventurers, people looking to improve on the status quo, have some experimental budget left, etc. Reading the repo, I'm not sure what it offers. It's still CGO for Go (edit: it's not, it's p…

The way I see it there are a few goals for Turso as opposed to SQLite...

One is to be more open to contribution, which is of arguable value for a pretty "complete" project.

Another is to be able to better support a client-server and distribution model for resilience over only in-process options, which is harder. This is while being file compatible with SQLite for the database itself.

Another aspect is multi-threaded support (mutli-read in particular), which is part of the impetus for rewriting in Rust over the fork, for what may well be a dramatic performance improvement.

Cloudflare and Turso as companies are both using SQLite's interfaces and structure at a core piece of their distributed database offerings... There's definitely different characteristics for use/scale if you're going that route. I've also found CockroachDB to be interesting along with the now deprecated RethinkDB's approach. That doesn't even get into the more prominent distributed cloud db options out there.

In the end they're all just different approaches to solving similar issues.

Re: Turso is an in-process SQL database, compatible with SQLite

#94

Post by tracker1 »

bawolff wrote: post #74
jauntywundrkind wrote: post #66

Earlier quoted context omitted.

Pretty good vector processing built-in. Time series capabilities. Nice Change-Data-Capture table that I've used & loved. Rust which is easy as hell to embed. Underlying libsqlite is very useful too. The CLI has far better ergonomics than sqlite & good formatting. Async & concurrent writes. Backwards compatibility. Just so ragingly badass. Tries. Isn't narrow & conservative. Amazing test suite. The discussion didn't s…

> Amazing test suite. Lol, is that a joke. Seesh. [To be clear, i think sqlite is the hands down winner on this front, no contest. Does the Turso test suite qualify it to be used in safety critical applications? I don't think so]. To your other points - look if it works for you i'm not here to tell you you can't use it. However these features sound more trendy than useful. To me these sound like negatives. A bunch of…

Have you seen the SQLite test suite? AFAIK, it's private/secret/closed. I think that is a part of the argument against SQLite.

Re: Turso is an in-process SQL database, compatible with SQLite

#95

Post by tracker1 »

d1l wrote: post #10

I’d imagine there’s an extremely long tail of features and quirks that will take time to iron out even after SQL compatibility is achieved. Looks like it’s still missing some important features like savepoints (!!!), windows and attach database. I’d be more excited and imagine it would be more marketable if it focused instead on being simply an embedded sql db that allowed multiple writers (for example), or some othe…

I think that async support for multi-read(and write) are part of the reason for the separate Turso library in Rust over the C fork (libSQL). I also wouldn't be surprised if baking in better support for replication was a design goal as well. Being file-format compatible with SQLite is really useful as well.

In the end, the company is a distributed DBaaS provider using a SQLite interface to do so... this furthers that goal... being SQLite compatible in terms of final file structure just eases backup/recovery and duplication options.

I think being able to self-host either in an open/free and commercial/paid setting is also going to be important to potential users and customers... I'm not going to comment on the marketing spin, as it is definitely that.

Re: Turso is an in-process SQL database, compatible with SQLite

#96

Post by tracker1 »

oofbey wrote: post #42
doanbactam wrote: post #40

We switched our main API from Postgres to Turso last month and haven't looked back. The automatic schema migrations are a nice touch, but I wish the documentation on vector embeddings was a bit more robust. It's wild how much of the modern web is moving back to file-based databases. We switched our main API from Postgres to Turso last month and the cold start times are basically zero now. Are there any plans to suppo…

How do you scale your front end horizontally with a file based database? Do you put the files on a shared file system that the app layer all mounts and lock when writing? Or do you shard and route by user? Or do you build a big vertically scaled API server with the database in it?

There's a few options... they could be using separate database instances per client to isolate workloads... if there are fewer than 10k or so users at each client, it's pretty doable without a lot of effort, you can further isolate types of data into separate databases as well... other operations can rely on heavy caching, such as maintaining the list of dbs for each client, etc.

You CAN use a single database instance or file for everything, you can also use multiples to scale without falling strictly into a heavy vertical or horizontally scaleable database system. Especially if those db instances are distributed over a network channel to different physical servers on the back-end (as is with Turso).

I've worked on a lot of systems where I had advocated the use of separate DBs either on a oer-client or per-project basis... Even from a single management server, you can do a lot... from a small cluster operating against SQLite databases you can do more. With Turso's efforts to improve concurrency, it can go further still.

I'm not an employee, related to, or even an active customer right now... but I do understand the model and how it can work in a lot of use cases.

Re: Turso is an in-process SQL database, compatible with SQLite

#97

Post by gritspants »

thayne wrote: post #55
heyts wrote: post #33

This looks like a solution in search of a problem already solved by SQLite

I think the problems with sqlite: - sqlite can't do concurrent writes, which is a performance bottleneck for certain workflows - sqlite is synchronous, which isn't ideal in applications that are async - while sqlite itself is open source, the test suite is proprietary, which means forking it and adding your own features or bug fixes isn't really practical. Of course, that is also a significant barrier to turso having…

Proprietary? Are you referring to the test suite they created for avionics that last I checked they've never really sold? I think it's highly misleading to make that claim if so.

Re: Turso is an in-process SQL database, compatible with SQLite

#98

Post by oofbey »

tracker1 wrote: post #96
oofbey wrote: post #42

Earlier quoted context omitted.

How do you scale your front end horizontally with a file based database? Do you put the files on a shared file system that the app layer all mounts and lock when writing? Or do you shard and route by user? Or do you build a big vertically scaled API server with the database in it?

There's a few options... they could be using separate database instances per client to isolate workloads... if there are fewer than 10k or so users at each client, it's pretty doable without a lot of effort, you can further isolate types of data into separate databases as well... other operations can rely on heavy caching, such as maintaining the list of dbs for each client, etc. You CAN use a single database instanc…

If each client has its own database, does that mean you're automating the infra to turn on a new instance for each database, and then routing requests appropriately? Not too hard with k8, I suppose. Or if you add new clients infrequently enough it's a manual task. But you wouldn't do this for each user - it assumes you have some higher level of organization (client/tenant/org/etc).

Sounds like these are different ways of manually sharding the database?

Re: Turso is an in-process SQL database, compatible with SQLite

#99

Post by thayne »

gritspants wrote: post #97
thayne wrote: post #55

Earlier quoted context omitted.

I think the problems with sqlite: - sqlite can't do concurrent writes, which is a performance bottleneck for certain workflows - sqlite is synchronous, which isn't ideal in applications that are async - while sqlite itself is open source, the test suite is proprietary, which means forking it and adding your own features or bug fixes isn't really practical. Of course, that is also a significant barrier to turso having…

Proprietary? Are you referring to the test suite they created for avionics that last I checked they've never really sold? I think it's highly misleading to make that claim if so.

I thought most of the test cases were proprietary, but it looks like you are right and it actually is just the TH3 test suite, and the fuzzer.

Re: Turso is an in-process SQL database, compatible with SQLite

#100

Post by sudhirb »

oofbey wrote: post #45
memothon wrote: post #36

Earlier quoted context omitted.

Turso offers a cloud-hosted sqlite product. The idea is you can easily spin up per-tenant databases and have a "serverless" sqlite interface. It feels like it has a lot of the same downsides of hosted databases so I'm not sure what the specific value is. From their site they really emphasize local first syncing (so mobile apps and electron/tauri apps) and multi tenancy (hard database boundaries)

Huh. That sucks but also isn’t surprising. Seems like putting this in the cloud eliminates most of its benefits though. If you’re going to wait for that much latency you might as well use a “real” (traditional big complicated) rdbms.

For me, a lot of the draw is that it's cheaper than managed db services for small/toy projects of mine (that I don't want to use dynamo db for) - that and in a previous job it was useful as relatively temporary multi-tenant storage.
Post reply on HN