Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Is Serverless

sqlite.org

Post reply on HN
Print view View original

241–250 of 453 posts

Re: SQLite Is Serverless

#241

Post by bob1029 »

vdfs wrote: post #204
bob1029 wrote: post #119

Earlier quoted context omitted.

Multiple processes is an instant anti-pattern for a single SQLite database. I would stop and reconsider your approach before trying to build this solution using it. The way I see it there are 3 options: 1) Implement another process which will have exclusive ownership of the shared SQLite database, and then use some IPC scheme to delegate database operations from multiple processes. 2) Give each process its own copy o…

It's ironic how using a severless database require your app to be the server and do top-level access management to the database

It may be ironic but there are benefits to rolling your own. Having your persistence layer talk in terms of your business models instead of raw SQL could be seen as a benefit in some contexts. The process which is the "server" can be written to allow for relaxed consistency based on specific business operation being performed (e.g. no need for transactions around log entries). This could be used to leverage substantial performance gains. The benefits afforded by guarantees of exclusivity between application and database are difficult to overstate.

These advantages of purpose-built functionality would also extend into the arena of handling replication and clustering. I.e. multiple distributed processes each with independent databases synchronized via some custom protocol that operates in terms of specific business models and processes.

Re: SQLite Is Serverless

#242

Post by pdimitar »

logicallee wrote: post #127
pilif wrote: post #118

Earlier quoted context omitted.

«just write your own server on top of serverless embedded SQLite to get to acceptable performance without needing a client/server database» (Honestly, once you are at the point where concurrency causes performance issues with SQLite, you are better off moving to databases designed to handle concurrency rather than trying to cobble together your own workaround - you have reached the point where the drawback of SQLite‘…

As you quoted, people in this discussion are saying that writing a dummy program that has its own sqlite and does nothing but pass messages to it that it receives from all other processes on the system that need to talk to that sqlite file results in much better performance than accessing that file "directly" from the separate processes. so if everyone's saying this, is there such a standard dummy program?

There shouldn’t be should a such standard program. It’s up to you to implement it in-process so you reap all the benefits of sqlite’s speed and lack of dedicated server process.

In the case of Erlang/Elixir where I most actively work in the last few years, it’s really easy to centralise write access (dedicated actor receiving messages). With other languages it should also be fairly easy.

Re: SQLite Is Serverless

#243

Post by tmpz22 »

nicholaides wrote: post #185

I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…

How does this work with container/ephemeral services such as typical K8s deployments? Can I trust the file system mounting via resources like StatefulSets or FS mounts? For that matter, Heroku, App Engine, Cloud Functions, whatever?

Our current setup is having all our services in kubernetes but our databases in stateful VMs. I do occasionally stuff job-reports and similar data into postgres rows since it's already there, but I've been unhappy with our ETL setup and would be interested in hearing techniques to improve it.

Re: SQLite Is Serverless

#244

Post by qes »

filleduchaos wrote: post #176
aikah wrote: post #138

Earlier quoted context omitted.

> Does this mean that you can hack some database storage (w/ sqlite) together on frontend only hosting platforms like Github or Netlify? You can run SQLlite in the client via webassembly therefore open a SQLite file in the browser to query it yes, you just can't write anything in it and expect it to persist somehow on the static hosting service itself. > The best I came up with: let's say you have a toy project, and…

"SQLite is serverless" is by no means a meaningless buzzword. The term had a clear meaning before it was coopted by the current web dev fad. SQLite does not operate on a client-server architecture the way e.g. MySQL or PostgreSQL do.

> does not operate on a client-server architecture

That was never a common usage of the term serverless.

Re: SQLite Is Serverless

#245

Post by nebulous1 »

bane wrote: post #149

I love SQLite, but people approach it from a classic RDBMS angle which confuses them. Here's the deal: SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file. That's it. Stop overthinking it. Can you write a microservice that stores its data in a big JSON file that you've built some code around to read/write to? Yes. It's just a file, but you have to build all the read/w…

I've spent half an hour or so looking at this thread, and my take away is that you're largely just confusing matters even further.

> It's just a file format.

This is clearly incorrect. It does encompass a file format, but it also contains code to manage that file. The existence of sqljet does not change this, it's merely a different database management system that uses the same file format.

You also seem to mostly ignore that the data it's managing is a relational database, not some other form of data. This is why you can't compare it to python's csv module or python-docx, neither of these do anything to restrict you to a relational data model, nor do they provide a system to query them as if they were a relational database. On the other hand, if for some unknown reason you rewrite the storage engine of postgresql to use the either the sqlite file format or csv/docx, I assume you agree it would still be an RDBMS.

Ultimately I think you're just adding to the confusion by saying that a project with 139,000 lines of C code is "just a file format".

Re: SQLite Is Serverless

#246

Post by hinkley »

> Neo-Serverless: The database engine runs in a separate namespace from the application, probably on a separate machine, but the database is provided as a turn-key service by the hosting provider, requires no management or administration by the application owners, and is so easy to use that the developers can think of the database as being serverless even if it really does use a server under the covers.

I’m having a failure of imagination here.

What would I use a database for that I could reasonably assert to my peers and superiors that no maintenance whatsoever is required? Backups and restores count as maintenance. Multi region is now common, if not pervasive.

Are there public datasets that are so common that it would be worth it to provide it as a service? What other service would behave like S3 but look like SQLite?

I get that one might be safe to assume that “serverless” isn’t just pure functions. It could reach out to other services that are not serverless and still not consume (further) resources on a set of machines while not in use.

But a severless database... I’d have to have something aggressively read-mostly, written to S3 at intervals and read from serverless processes. But is that a new thing or reading data from S3?

Re: SQLite Is Serverless

#247

Post by filleduchaos »

qes wrote: post #244
filleduchaos wrote: post #176

Earlier quoted context omitted.

"SQLite is serverless" is by no means a meaningless buzzword. The term had a clear meaning before it was coopted by the current web dev fad. SQLite does not operate on a client-server architecture the way e.g. MySQL or PostgreSQL do.

> does not operate on a client-server architecture That was never a common usage of the term serverless.

The linked article was written in 2007, for your information.

Software development didn't begin or end with web dev and the cloud.

Re: SQLite Is Serverless

#248

Post by hinkley »

at_a_remove wrote: post #191
nicholaides wrote: post #185

I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…

Using SQLite in my ETL processes is something I have done for over a decade. It's just so convenient and, at the end, I have this file that can be examined and queried to see where something might have gone wrong. All of my "temporary" tables are right there for me to look at. It is wonderful!

Do you generate the file from scratch every time or do you modify the previous one as new data arrives?

Re: SQLite Is Serverless

#249

Post by jtms »

Frost1x wrote: post #228
jtms wrote: post #222

Earlier quoted context omitted.

"managed" is an AWS instance or a digital ocean droplet. "serverless" is an entirely different thing.

"Serverless" is another level of abstraction and management, that's it. Are you proposing that a term collision adds too much confusion? "Managed" hosting has been around for decades as have "managed" services, in general, long before Amazon even existed. EC2 brought a new level/layer of automation and management ontop of what colo datacenters used to do, provide scaling etc. To be clear I'm not saying it's simple bu…

I would argue that the architectural implications of using a "run individual functions on a remote pool of computing resources and build your client application around this pattern and only pay for the exact amount of compute time you use" is not sufficiently covered by the term "managed". I think there could have perhaps been a better name than "serverless" chosen, but that's the name enough people have agreed on to use when talking about this specific architecture that it would be difficult to use another one.

Re: SQLite Is Serverless

#250

Post by hinkley »

cle wrote: post #196

> It is important to understand these two different definitions for "serverless". When a database claims to be "serverless", be sure to discern whether they mean "classic serverless" or "neo-serverless". It's really not important to understand that distinction, because this author seems to be the only one making it. Everyone knows what "serverless" means at this point, and it's not an embedded DB.

Meh. If we allow serverless to make REST calls, is accessing a file system any different?

I had more trouble with the assertion that the embedded DB would be maintenance-free. I started a top level comment asking for someone to explain how that would work. The part of my brain that protects me from scams is screaming “something for nothing”.

Post reply on HN