Live data from Hacker News

SQLite Is Serverless

sqlite.org

171–180 of 453 posts

Re: SQLite Is Serverless

#171

Earlier quoted context omitted.

"Serverless" is a garbage marketing term in general. It sounds sexy to nontechnical management folks who are used to hearing a bunch of expensive costs and the word "server" associated with them in some fashion. From that view, anything that gets rid of those pesky "servers" sounds like a win.

It is, I did Azure Functions which is their "serverless" solution. It's either a type of CGI 'trigger' which is not as impressive however, another 'function' can be triggered based on Database actions or file uploads in their Blob Storage. So it's just handy if you're all in on Azure, and probably other clouds because you can have code run immediately when events happen for certain services. Outside of that, I find i…

> The other buzzword is microservices. I was working a gig that involved running like 10 on a MBP with 16GB of RAM and it froze up the laptop. Java is not a language for micro anything, at least not in regards to memory. Was fun but dreadful to work with more than 3 local services at once. The creators of the framework suggested to mock services, but it was just messy to do that too.

This seems like a strange criticism. It appears that you want a full deployment of the platform on your local machine. But Microservices aren’t optimized to run on your local machine; they’re meant to be deployed to kubernetes/docker swarm/mesosphere.

I’m also no fan of Java but I don’t think it’s the language that is preventing you here? All speculation since I don’t know the details but it sounds suspiciously like the JVM is getting stuck trying to allocate heap during startup (this is just a guess).

Re: SQLite Is Serverless

#172
post #149

I love SQLite, but people approach it from a classic RDBMS angle which confuses them. Here's the deal: SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file. That's it. Stop overthinking it. Can you write a microservice that stores its data in a big JSON file that you've built some code around to read/write to? Yes. It's just a file, but you have to build all the read/w…

Consider: it’s totally possible to strip down Postgres until all you have left is an embedded RDBMS of the style of SQLite. (I’m not sure why nobody has done this yet, actually.) Would you call the result “just a file format”?

Such an instance of “embedded Postgres” would still have a huge sprawling catalog (PG_DATA) directory attached to each use of it, so it wouldn’t be contained to a single file. But neither is SQLite contained to a single file—SQLite maintains a journal and/or WAL in a second file.

And, yes, this “embedded Postgres” would require things like vacuuming. But... so does SQLite. Have you never maintained an application that maintains a long-lived “project” as a single SQLite file, where changes are written into this project file repeatedly over a long period? (Think: the “library” databases of music/photo library management software.) SQLite database files experience performance degradation from dead tuples too, and need all the same maintenance. Often “database version migrations” of such software is written to either rewrite the SQLite file into a clean state, or—if it has the possibility of being too big for that to be a quick task—to call regular VACUUM-like SQL commands to clean the database state up.

——

Now, I get what you’re trying to say; the point that you’re trying to make—that SQLite might be a relational database, but it’s not a relational database management system in the sense of sitting around online+idle where it can do maintenance tasks like auto-vacuuming. Unlike an RDBMS, SQLite doesn’t have its own “thread of execution”: it is a library whose functioning only “happens” when something calls into it. By analogy, regular RDBMSes are like regular OS kernels, while SQLite is like a library kernel or exokernel.

But that doesn’t mean that SQLite is a file format! It can be used as one, certainly, but what SQLite is is exactly the analogy above: the kernel of an RDBMS, externalized to a library. As long as you “run” said kernel from your application, and your application is a daemon with its own thread of execution, then your application is an RDBMS.

This can be seen most directly in systems like ActorDB, that simply act as a “transaction server” routing requests to SQLite. ActorDB is, pretty obviously, an RDBMS; but it achieves that not due to its own features, but 99% due to embedding SQLite. All it does is call into SQLite, which already has the “management system” part of an RDBMS built in, just not called unless you use it—just like exokernels often already have things like a scheduler, just not called into unless you as the application layer do so.

Re: SQLite Is Serverless

#173
post #149

I love SQLite, but people approach it from a classic RDBMS angle which confuses them. Here's the deal: SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file. That's it. Stop overthinking it. Can you write a microservice that stores its data in a big JSON file that you've built some code around to read/write to? Yes. It's just a file, but you have to build all the read/w…

I'm no SQLite expert but by this logic aren't all single-host databases that tick the "Durability" ACID checkbox "just file formats" in the sense that yeah, the bytes we care about exist somewhere in the filesystem? Moreover I'm having trouble coming up with things that I'd associate with a RDBMS and not "just a file format" that SQLite doesn't support. Transactions? SQLite has them. Relational constraints? SQLite ha…

Look at the other comments just in this post.

Re: SQLite Is Serverless

#174
post #168
post #149

I love SQLite, but people approach it from a classic RDBMS angle which confuses them. Here's the deal: SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file. That's it. Stop overthinking it. Can you write a microservice that stores its data in a big JSON file that you've built some code around to read/write to? Yes. It's just a file, but you have to build all the read/w…

I think most rmdbs systems are just a file (or set of distributed files) when you strip away all the helper functions. The fact that Sqlite squirrels everything into a single archive doesn't mean many of the abilities housed within a more comprehensive database aren't there. You can implement JSON columns and full text searching and numerous other fancy systems. The main argument I hear for why Sqlite deserves second…

> The main argument I hear for why Sqlite deserves second class status in the DB marketplace is the difficulty in handling multiple writes simultaneously.

SQLite is not in the DB marketplace. It's in the file format marketplace. It handles multiple writes in exactly the same way CSV handles multiple writes. If you want to handle multiple writes with SQLite, you handle it the same way as CSV.

Re: SQLite Is Serverless

#176
post #138

Does this mean that you can hack some database storage (w/ sqlite) together on frontend only hosting platforms like Github or Netlify? I think not, but I wonder if some hack is available by virtue of it simply being a file that you can read (and somehow) write to. The best I came up with: let's say you have a toy project, and you call the Github API and replace the file upon every write. Implementing a read is easier…

> Does this mean that you can hack some database storage (w/ sqlite) together on frontend only hosting platforms like Github or Netlify? You can run SQLlite in the client via webassembly therefore open a SQLite file in the browser to query it yes, you just can't write anything in it and expect it to persist somehow on the static hosting service itself. > The best I came up with: let's say you have a toy project, and…

"SQLite is serverless" is by no means a meaningless buzzword. The term had a clear meaning before it was coopted by the current web dev fad. SQLite does not operate on a client-server architecture the way e.g. MySQL or PostgreSQL do.

Re: SQLite Is Serverless

#177
post #172
post #149

I love SQLite, but people approach it from a classic RDBMS angle which confuses them. Here's the deal: SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file. That's it. Stop overthinking it. Can you write a microservice that stores its data in a big JSON file that you've built some code around to read/write to? Yes. It's just a file, but you have to build all the read/w…

Consider: it’s totally possible to strip down Postgres until all you have left is an embedded RDBMS of the style of SQLite. (I’m not sure why nobody has done this yet, actually.) Would you call the result “just a file format”? Such an instance of “embedded Postgres” would still have a huge sprawling catalog (PG_DATA) directory attached to each use of it, so it wouldn’t be contained to a single file. But neither is SQ…

Great comment! Respectful of GP and constructively critical.

The other thing I would mention is that SQLite can operate totally in memory which makes it useful without even using it to persist data (say you have a language with a slow dataframe API, just use SQLite in memory to process your data).

Re: SQLite Is Serverless

#178
post #85
post #31

Earlier quoted context omitted.

Waaaaait wouldn’t that mean the file system is the server, with some binary API and responsible for handling concurrent access and locks for the entire file? LOL.

That just circle jerk. There is an agreed upon def of serverless. If we reduce things to the absurd we stop being able to reason about things. https://en.m.wikipedia.org/wiki/Serverless_computing

That is not the definition of serverless in question.

Re: SQLite Is Serverless

#179

Earlier quoted context omitted.

"Serverless" is a garbage marketing term in general. It sounds sexy to nontechnical management folks who are used to hearing a bunch of expensive costs and the word "server" associated with them in some fashion. From that view, anything that gets rid of those pesky "servers" sounds like a win.

It is, I did Azure Functions which is their "serverless" solution. It's either a type of CGI 'trigger' which is not as impressive however, another 'function' can be triggered based on Database actions or file uploads in their Blob Storage. So it's just handy if you're all in on Azure, and probably other clouds because you can have code run immediately when events happen for certain services. Outside of that, I find i…

Java has been used on servers since 256MB was common to use — Java is not a limiting factor in how lightweight you can get on modern Intel hardware, though it might need some tuning.

Re: SQLite Is Serverless

#180
post #174
post #168

Earlier quoted context omitted.

I think most rmdbs systems are just a file (or set of distributed files) when you strip away all the helper functions. The fact that Sqlite squirrels everything into a single archive doesn't mean many of the abilities housed within a more comprehensive database aren't there. You can implement JSON columns and full text searching and numerous other fancy systems. The main argument I hear for why Sqlite deserves second…

> The main argument I hear for why Sqlite deserves second class status in the DB marketplace is the difficulty in handling multiple writes simultaneously. SQLite is not in the DB marketplace. It's in the file format marketplace. It handles multiple writes in exactly the same way CSV handles multiple writes. If you want to handle multiple writes with SQLite, you handle it the same way as CSV.

I disagree. CSVs don't have write-ahead logging and are not able to selectively lock portions file for writes.

And I'd say that not only is sqlite in the db marketplace (albeit for a specific subset of database application types) it's one of the largest players.

Post reply on HN