Live data from Hacker News

Soul: A SQLite REST and Realtime Server

thevahidal.github.io

21–30 of 53 posts

Re: Soul: A SQLite REST and Realtime Server

#21
post #17

No benchmark = no download. Plenty of these SQLite web front ends exist, including more enticing single static binaries in go, and they are always far too slow to use in anything but toy projects. Should be standard to bench against a standard SQLite integration (on modern cpu with nvme). Even PocketBase, for as nice as the UI is, an order of magnitude slower than SQLite directly. Here's some rough numbers I've found…

Go calls to C are slow https://vancan1ty.com/blog/post/52

Re: Soul: A SQLite REST and Realtime Server

#22
Nice and a possible alternative to the very polished PocketBase.

I'm working on something similar for PostgreSQL [0], with an API compatible with the excellent PostgREST [1].

I believe these tools can be of great utility in many projects and represent a generalization compared to the dedicated middleware that was popular a few years ago. Companies like Supabase are demonstrating this.

[0] https://github.com/sted/smoothdb [1] https://github.com/PostgREST/postgrest

Re: Soul: A SQLite REST and Realtime Server

#23
post #17

No benchmark = no download. Plenty of these SQLite web front ends exist, including more enticing single static binaries in go, and they are always far too slow to use in anything but toy projects. Should be standard to bench against a standard SQLite integration (on modern cpu with nvme). Even PocketBase, for as nice as the UI is, an order of magnitude slower than SQLite directly. Here's some rough numbers I've found…

I think this one is more for development or test endpoints. So installation from NPM is big plus, so that I can add it to devDependencies, have sources in node_modules and patch them with patch-package.

Re: Soul: A SQLite REST and Realtime Server

#24
post #2

I thought the whole point of sqlite is that you don't have to deal with the cruft and overhead of having to send queries over a network connection. Why tack that back on?

I regularly use an embedded sqlite database in my projects, and the existence of tools such as soul and pocketbase means I can easily make those databases available to other members of my team without having to write a custom front-end or component of my application to deal with their unwillingness to install an sqlite GUI on systems they may not necessarily be able to do so, in the first place.

Re: Soul: A SQLite REST and Realtime Server

#25
post #17

No benchmark = no download. Plenty of these SQLite web front ends exist, including more enticing single static binaries in go, and they are always far too slow to use in anything but toy projects. Should be standard to bench against a standard SQLite integration (on modern cpu with nvme). Even PocketBase, for as nice as the UI is, an order of magnitude slower than SQLite directly. Here's some rough numbers I've found…

Hi, PocketBase author here.

Keep in mind that PocketBase do a lot more than just executing a raw DB query. We perform data validation, normalization, serialization, enriching, auto fail-retry to handle additional SQLITE_BUSY errors, etc. All of this comes with some cost and will always have an effect when doing microbenchmarks like this.

The performance would also depend on what version of PocketBase did you try (before or after v0.10), whether you used CGO or the pure Go driver, etc.

For a benchmark closer to "real world" scenarios tested on various servers you can check the results from https://github.com/pocketbase/benchmarks.

There is definitely room for improvements (I haven't done any detailed profiling yet) but the current performance is "good enough" for the purposes the applications PocketBase is intended for (I've shared some numbers regarding a PocketBase app on production in https://github.com/pocketbase/pocketbase/discussions/4254).

Hope the above helps.

Re: Soul: A SQLite REST and Realtime Server

#26
post #2

I thought the whole point of sqlite is that you don't have to deal with the cruft and overhead of having to send queries over a network connection. Why tack that back on?

Frankly, you thought wrong. The whole point of sqlite was to build a database for naval vessels. Aside from that, given it's open source, the "point" of it doesn't really matter anymore.

That is inaccurate. Ginko's statement is closer to truth.

I was inspired to write SQLite while working with Informix on DDG-79 and I saw how useful an embedded database would be in some situations, compared to a client/server solution. So I went off and wrote SQLite on my own, while the development contract was on hiatus. There was never a request for SQLite or anything like it coming from the the navy (or more precisely, Bath Iron Works) as they were both very happy with Informix on the ship and Oracle on land and had zero desire for anything new or different. The development team I worked on ended up using SQLite some for prototyping and testing on that project, but it was never deployed to the ship, as far as I know.

So yes, the whole point of SQLite was to build a database that operated as a library linked into the application, rather than as a separate server, as ginko postulates. Design issues on a single system within DDG-79 (Automated Common Diagrams) were the inspiration for that idea, but to say that SQLite was designed for DDG-79 is not true. There was never a request for SQLite coming from the navy or the ship designers. Indeed, there is was a lot of pushback against SQLite. SQLite was just a crazy idea coming from a rogue developer who happened to be working on one of the many on-board systems at that time.

Re: Soul: A SQLite REST and Realtime Server

#28
I am not sure what Realtime means here. But if it means something close to what I used to [1], then REST is a horrible idea. Javascript is not a good idea either.

[1] " Real-time computing (RTC) is the computer science term for hardware and software systems subject to a "real-time constraint", for example from event to system response.[1] Real-time programs must guarantee response within specified time constraints, often referred to as "deadlines".[2] "

Re: Soul: A SQLite REST and Realtime Server

#29

I am not sure what Realtime means here. But if it means something close to what I used to [1], then REST is a horrible idea. Javascript is not a good idea either. [1] " Real-time computing (RTC) is the computer science term for hardware and software systems subject to a "real-time constraint", for example from event to system response.[1] Real-time programs must guarantee response within specified time constraints, o…

Realtime seems to mean - in this context - that you can open up a web socket and subscribe to changes to rows. i.e. get changes to rows pushed to you as a client instead of pulling.

Re: Soul: A SQLite REST and Realtime Server

#30
post #19
post #17

No benchmark = no download. Plenty of these SQLite web front ends exist, including more enticing single static binaries in go, and they are always far too slow to use in anything but toy projects. Should be standard to bench against a standard SQLite integration (on modern cpu with nvme). Even PocketBase, for as nice as the UI is, an order of magnitude slower than SQLite directly. Here's some rough numbers I've found…

Btw ws4sqlite rewrite in rust that's allegedly faster: https://github.com/proofrock/sqliterg

Ran the same test, same server (Ubuntu 22.04 + 32 thread 7950X + Gen 4 NVME + 64GB DDR5) -- nearly zero difference sqliterg vs ws4sqlite.

sqliterg (CURL)

20,000 / 2,500 / 237

For those wondering about write performance...its poor except for direct SQLite. If you're write heavy, look elsewhere, or use SQLite directly (preferably with multiple databases and WAL2).

INSERT INTO users (id) VALUES (..); per second ( 1x / 10x / 100x )

SQLite WAL

11,000 / 3,000 / 300

SQLite WAL2

14,500 / 3,000 / 760

SQLite WAL2 (3x Database files)

29,000 / 6,000 / 1,400

sqliterg (CURL)

1,750 / 181 / blocks indefinitely

Post reply on HN