I don't think we fully appreciate just how pervasive SQLite is in our computing devices. Apple recognized its power and utility and embedded it in macOS and iOS, and Google apparently followed suit in Android. The universality of SQLite is due in no small measure to its stellar quality and the trustworthiness of its author, D. Richard Hipp.
SQLite Is Serverless
311–320 of 453 posts
Re: SQLite Is Serverless
#312Isn’t it easier to say “embedded” and “severless”
Not in this thread apparently. I guarantee nobody here as ever used the term "serverless" over "embedded" or "in-process" in their entire career but apparently the purity and nostalgia of SQLite overrides it.
Re: SQLite Is Serverless
#313Earlier quoted context omitted.
I know, but have you anyone calling "embedded databases" "serverless databases" in the last 10 years? It's like someone found this page and felt very smart about it, because they stick it to the serverless crowd.
"embedded database" does not imply "serverless database". Other embedded RDBMSes run servers; they just run the server as a separate thread rather than a separate process. SQLite is different in that there is no separate thread of control. SQLite runs in the same thread as the application that calls it. There is no separate thread hanging around to clean up or handle background tasks after an SQLite function call ret…
[deleted]
Re: SQLite Is Serverless
#314Earlier quoted context omitted.
Multiple processes is an instant anti-pattern for a single SQLite database. I would stop and reconsider your approach before trying to build this solution using it. The way I see it there are 3 options: 1) Implement another process which will have exclusive ownership of the shared SQLite database, and then use some IPC scheme to delegate database operations from multiple processes. 2) Give each process its own copy o…
It's ironic how using a severless database require your app to be the server and do top-level access management to the database
Nothing of the sort is required. If you want to write a simple single-threaded synchronous program, sqlite works great for that.
Re: SQLite Is Serverless
#315I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…
I have been using SQLite as a format to move data between steps in a complicated batch processing pipeline. With the right pragmas it is both faster and more compact than JSON. It is also much more "human readable" than gigabytes of JSON. I only wish there was a way to open an http-fetched SQLite database from memory so I don't have to write it to disk first.
If the language's sqlite bindings don't offer a way to load a database from a string, if you're on a modern linux kernel (3.17+) you can make use of the memfd_create syscall: it creates an anonymous memory-backed file descriptor equivalent to a tmpfs file, but no tmpfs filesystem needs to be mounted and there's no need to think about file paths.
Re: SQLite Is Serverless
#316Earlier quoted context omitted.
> The other buzzword is microservices. I was working a gig that involved running like 10 on a MBP with 16GB of RAM and it froze up the laptop. Java is not a language for micro anything, at least not in regards to memory. Was fun but dreadful to work with more than 3 local services at once. The creators of the framework suggested to mock services, but it was just messy to do that too. This seems like a strange critici…
And when you need to interact with other services in a disconnected way? You have to run them somewhere... and local should generally be an option, even if that's to a local single-node (mini)kube cluster. There's nothing wrong with needing to run more than one thing while developing/testing and interacting. There are other options, but it's not a clearly bad approach. That said, I would definitely lean towards Go or…
In general, staging environments are the solution to cross service testing for Microservices. The expectation that a Microservice must be testable end to end locally is not one I would like to encourage. The whole point of splitting out a service this way is that individual services have very narrowly scoped, independent function and can be tested on their own, perhaps with mocked data.
One wouldn’t expect to run all of a companys entire pipeline locally on a single laptop. Expecting that for Microservices is similar.
Re: SQLite Is Serverless
#317> It is important to understand these two different definitions for "serverless". When a database claims to be "serverless", be sure to discern whether they mean "classic serverless" or "neo-serverless". It's really not important to understand that distinction, because this author seems to be the only one making it. Everyone knows what "serverless" means at this point, and it's not an embedded DB.
Everyone know that “serverless” anything doesn’t run on a server, be it AWS or Azure instances or what have you. That would be both ironic and silly, like someone used the wrong word or something.
Serverless databases and what have you have been around longer than the current batch of folks trying to redefine things (or more charitably, ran out names to call things). Like or not, there is a distinction even if the old definition was before your time.
Re: SQLite Is Serverless
#318> It is important to understand these two different definitions for "serverless". When a database claims to be "serverless", be sure to discern whether they mean "classic serverless" or "neo-serverless". It's really not important to understand that distinction, because this author seems to be the only one making it. Everyone knows what "serverless" means at this point, and it's not an embedded DB.
The "neo-serverless" thing is strange. But then, so is "serverless". "Serverless" systems are just time-shared servers. But only if they use new, cool technology. Shared hosting with vendor managed MySQL doesn't count, apparently.
Re: SQLite Is Serverless
#319> Of those that are serverless, SQLite is the only one known to this author that allows multiple applications to access the same database at the same time. IIRC, MS Access allowed that, which explained a lot of its popularity.
FoxPro/Foxbase as well. In the Foxbase days predating FoxPro, there was even a “multi-user” version that, unsurprisingly, cost more than the standard single user.
And plenty of other file-based DBs such as Borland’s Paradox and the db engine, dBase; pretty sure FileMaker, too.
Re: SQLite Is Serverless
#320I love SQLite, but people approach it from a classic RDBMS angle which confuses them. Here's the deal: SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file. That's it. Stop overthinking it. Can you write a microservice that stores its data in a big JSON file that you've built some code around to read/write to? Yes. It's just a file, but you have to build all the read/w…
I've spent half an hour or so looking at this thread, and my take away is that you're largely just confusing matters even further. > It's just a file format. This is clearly incorrect. It does encompass a file format, but it also contains code to manage that file. The existence of sqljet does not change this, it's merely a different database management system that uses the same file format. You also seem to mostly ig…
It's a file format with a very mature and nice library around it. At the end of a day it's just fwrite() and fread() with SQL syntax.