Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Is Serverless

sqlite.org

Post reply on HN
Print view View original

181–190 of 453 posts

Re: SQLite Is Serverless

#181

Post by nickthemagicman »

idclip wrote: post #85
EGreg wrote: post #31

Earlier quoted context omitted.

Waaaaait wouldn’t that mean the file system is the server, with some binary API and responsible for handling concurrent access and locks for the entire file? LOL.

That just circle jerk. There is an agreed upon def of serverless. If we reduce things to the absurd we stop being able to reason about things. https://en.m.wikipedia.org/wiki/Serverless_computing

Thank you for saying this. People seem to argue semantics to appear clever and it really pollutes communication and reasoning.

Re: SQLite Is Serverless

#182

Post by blattimwind »

oldgradstudent wrote: post #23

> Of those that are serverless, SQLite is the only one known to this author that allows multiple applications to access the same database at the same time. IIRC, MS Access allowed that, which explained a lot of its popularity.

Since Access 97 or so it could actually use MS SQL as a backend which largely removed those issues; the Access "database" was then merely a VBS GUI for SQL Server. This works pretty well.

Re: SQLite Is Serverless

#183

Post by lazyant »

bane wrote: post #149

I love SQLite, but people approach it from a classic RDBMS angle which confuses them. Here's the deal: SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file. That's it. Stop overthinking it. Can you write a microservice that stores its data in a big JSON file that you've built some code around to read/write to? Yes. It's just a file, but you have to build all the read/w…

Sure. Or a light RDBMS where the main not supported case is concurrent writes.

Re: SQLite Is Serverless

#184

Post by andreareina »

bane wrote: post #149

I love SQLite, but people approach it from a classic RDBMS angle which confuses them. Here's the deal: SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file. That's it. Stop overthinking it. Can you write a microservice that stores its data in a big JSON file that you've built some code around to read/write to? Yes. It's just a file, but you have to build all the read/w…

In their own words: "SQLite does not compete with client/server databases. SQLite competes with fopen()."

[1] https://www.sqlite.org/whentouse.html

Re: SQLite Is Serverless

#185

Post by nicholaides »

I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are just files that are easy to copy, store, version, etc and don't require a centralized service).

I'm not saying ALWAYS use SQLite for these cases, but in the right scenario it can simplify things significantly.

Another similar use case would be AI/ML models that require a bunch of data to operate (e.g. large random forests). If you store that data in Postgres, Mongo or Redis, it becomes hard to ship your model alongside with updated data sets. If you store the data in memory (e.g. if you just serialize your model after training it), it can be too large to fit in memory. SQLite (or other embedded database, like BerkleyDB) can give the best of both worlds-- fast random access, low memory usage, and easy to ship.

Re: SQLite Is Serverless

#186

Post by mrmekon »

filleduchaos wrote: post #176
aikah wrote: post #138

Earlier quoted context omitted.

> Does this mean that you can hack some database storage (w/ sqlite) together on frontend only hosting platforms like Github or Netlify? You can run SQLlite in the client via webassembly therefore open a SQLite file in the browser to query it yes, you just can't write anything in it and expect it to persist somehow on the static hosting service itself. > The best I came up with: let's say you have a toy project, and…

"SQLite is serverless" is by no means a meaningless buzzword. The term had a clear meaning before it was coopted by the current web dev fad. SQLite does not operate on a client-server architecture the way e.g. MySQL or PostgreSQL do.

Just to back that up, it had a clear meaning because -less is a valid suffix to append to English words. When grandma runs out of cookies she is cookieless (when a website doesn't use cookies it is also cookieless). It doesn't have to be "in the dictionary" to make sense in conversation.

A quick search of usenet shows "serverless" being used in 1994. It wasn't a term or a buzzword, it wasn't common, it was just English: https://groups.google.com/d/msg/comp.os.linux.misc/r76oNl98C...

Re: SQLite Is Serverless

#187

Post by nickthemagicman »

bane wrote: post #149

I love SQLite, but people approach it from a classic RDBMS angle which confuses them. Here's the deal: SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file. That's it. Stop overthinking it. Can you write a microservice that stores its data in a big JSON file that you've built some code around to read/write to? Yes. It's just a file, but you have to build all the read/w…

If there's no Inter Process Communications and it doesn't use the OS to write to the filesystem...how does an application write to SQLlite db?

I'm a noob and just curious.

Re: SQLite Is Serverless

#188

Post by bane »

derefr wrote: post #172
bane wrote: post #149

I love SQLite, but people approach it from a classic RDBMS angle which confuses them. Here's the deal: SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file. That's it. Stop overthinking it. Can you write a microservice that stores its data in a big JSON file that you've built some code around to read/write to? Yes. It's just a file, but you have to build all the read/w…

Consider: it’s totally possible to strip down Postgres until all you have left is an embedded RDBMS of the style of SQLite. (I’m not sure why nobody has done this yet, actually.) Would you call the result “just a file format”? Such an instance of “embedded Postgres” would still have a huge sprawling catalog (PG_DATA) directory attached to each use of it, so it wouldn’t be contained to a single file. But neither is SQ…

SQLite is a file format with a familiar API and uses SQL as the logic for searching/adding data to the file.

Approach it exactly the same way you'd approach using a CSV file and all the confusion and overthinking about it goes away. Approach it as a stripped down RDBMS and you end up with all kinds of questions about support for this or that RDBMS familiar service.

You can write your own SQLite file reader/writer. Here's the specs (includes the specs for the Journal and WAL files and semantics as well) https://www.sqlite.org/fileformat.html

Here's an example of somebody who's done this. https://sqljet.com/ - this is not a wrapper on the sqlite C code, this is a re-implementation of that code that is binary compatible with SQLite files.

The Journal file only exists as a temporary file until transactions complete. The .sqlite file you make is the entire atomic file that follows the SQLite file format. The Journal has its own file format. Same goes for the Write-Ahead log.

RDBMSs also manage connection queues, account management, rights and permissions, and so on. Many overcome various OS limitations by providing their own entire file management, fopen(), virtual memory and other subsystems that are tuned to their workloads.

SQLite is a file format. SQLite uses familiar relational paradigms to make it easy to read/write data to the format without having to learn yet another API and domain language. The API code is extraordinarily well tested, and it makes simple complex logic like transaction journaling, indexing and so on.

>Consider: it’s totally possible to strip down Postgres until all you have left is an embedded RDBMS of the style of SQLite.

No! SQLite is not an embedded RDBMS. It's a file format.

If there was a library you could import, and it provided methods to read and write directly to files that PostgreSQL could read/write to and there was nothing else to install, no runtime, no daemons, no servers, etc., then we could pass around self-contained PostgreSQL files to each other. Then PostgreSQL files would be a file format as well.

Have you ever used a library to read/write from a CSV, JSON, JPEG? It's no different than doing so for a SQLite file!

SQLite is a file format.

Re: SQLite Is Serverless

#189

Post by bshipp »

pm90 wrote: post #166

What I really want: SQLite storage backend driver for s3/gcs. No need for disks then. I haven’t been able to find such a solution though; and am not technically proficient enough in C (the Lang SQLite is written in) to do so myself.

that would be cool! as an interim step, if your data is small enough, perhaps you could run an in-memory sqlite db and periodically backup to a permanent s3 file?

APSW exposes the sqlite backup api so you could do them online without shutting down the database.

Re: SQLite Is Serverless

#190

Post by Geeflow »

imafish wrote: post #49

My understanding of serverless = easily scalable, managed service. But somehow the word annoys people. Maybe we should find a better word?

I always explain serverless as "Instead of one server you have countless servers, but they are managed by someone else".

Yes, a better word would be great. I think that ship is sailed though...

Post reply on HN