Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Is Serverless

sqlite.org

Post reply on HN
Print view View original

251–260 of 453 posts

Re: SQLite Is Serverless

#251

Post by filleduchaos »

mrmekon wrote: post #186
filleduchaos wrote: post #176

Earlier quoted context omitted.

"SQLite is serverless" is by no means a meaningless buzzword. The term had a clear meaning before it was coopted by the current web dev fad. SQLite does not operate on a client-server architecture the way e.g. MySQL or PostgreSQL do.

Just to back that up, it had a clear meaning because -less is a valid suffix to append to English words. When grandma runs out of cookies she is cookieless (when a website doesn't use cookies it is also cookieless). It doesn't have to be "in the dictionary" to make sense in conversation. A quick search of usenet shows "serverless" being used in 1994. It wasn't a term or a buzzword, it wasn't common, it was just Engli…

Yes, this is what I was trying to say. It wasn't a thing people would throw around like a buzzword but if someone used it in conversation and especially in context (like the SQLite page, which was written over a decade ago, does) people would understand what you meant.

Re: SQLite Is Serverless

#252

Post by tracker1 »

tmpz22 wrote: post #243
nicholaides wrote: post #185

I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…

How does this work with container/ephemeral services such as typical K8s deployments? Can I trust the file system mounting via resources like StatefulSets or FS mounts? For that matter, Heroku, App Engine, Cloud Functions, whatever? Our current setup is having all our services in kubernetes but our databases in stateful VMs. I do occasionally stuff job-reports and similar data into postgres rows since it's already th…

Use a volume container mounted against a persistent storage engine on the node and do pod mounting from those containers. Stateful VMs are often a better choice for production imo.

I'm in favor of leveraging ISP dbaas and persistence offerings over trying to home grow something. It just depends on where you are coming from and/or what you are trying to do... K8s alone avoids so much lock in, and as long as whatever storage option (container mount) or dbaas you use is portable, I don't think it's so bad in either case.

Re: SQLite Is Serverless

#253

Post by idclip »

finnthehuman wrote: post #97
idclip wrote: post #85

Earlier quoted context omitted.

That just circle jerk. There is an agreed upon def of serverless. If we reduce things to the absurd we stop being able to reason about things. https://en.m.wikipedia.org/wiki/Serverless_computing

>There is an agreed upon def of serverless. Sorta but not really. The fact people have worked backwards from marketing names to try and constructively define inherently self-contradictory branding (rather than create a descriptive category into which we place questionable names and ignore them) is an embarrassment for everyone except the marketing departments.

I honestly too dislike all this naming fad and feel the internet’s been taken over by the management and marketting folk .. but still, i try not to hyperbole about it - too bad, but its sorta ok, and it ultimately irrelevant for the job, dbConnection is simply remote.

Re: SQLite Is Serverless

#254

Post by tenant »

paulie_a wrote: post #104
tenant wrote: post #81

Earlier quoted context omitted.

Access is actually designed to work in a multiuser situation over a LAN for concurrent read and write. SQLite isn't afaik. As long as the LAN was cabled I never saw any issues. The only reason it was necessary to move to a server type database was because people were insisting on wifi networking.

It might have been designed for it. It was poorly designed for it. It was a clustfuck of corruption when actually utilized

It depended. It suited a small cabled office with 10 to 15 computers fine. I saw such setups work fine for 12 years without corruption. But when wifi came along people started connecting that way, sometimes unintentionally, and corruption became an issue. So then we bit the bullet and moved the backend to a server database. Not a single database corruption since.

Re: SQLite Is Serverless

#255

Post by tracker1 »

hinkley wrote: post #248
at_a_remove wrote: post #191

Earlier quoted context omitted.

Using SQLite in my ETL processes is something I have done for over a decade. It's just so convenient and, at the end, I have this file that can be examined and queried to see where something might have gone wrong. All of my "temporary" tables are right there for me to look at. It is wonderful!

Do you generate the file from scratch every time or do you modify the previous one as new data arrives?

Depends on what you want... if you have a separate db project, you can have the output of that project be a clean database for testing other things, or a set of migration scripts for existing deployments.

I've been working on doing similar with containerized dababase servers for testing, while still having versioned scripts for prod (multiple separate deployments).

Re: SQLite Is Serverless

#256

Post by pm90 »

tn1 wrote: post #192
pm90 wrote: post #166

What I really want: SQLite storage backend driver for s3/gcs. No need for disks then. I haven’t been able to find such a solution though; and am not technically proficient enough in C (the Lang SQLite is written in) to do so myself.

The challenge with this is that S3 and friends are object stores, meaning you upload or download the whole file each time. As you can imagine, this will cost you tremendous bandwidth even to insert one row. Furthermore, it doesn't solve the multiple writers problem, because (afaik) there's no way to lock a file on S3.

> As you can imagine, this will cost you tremendous bandwidth even to insert one row.

Is this the cost of network transfers changed by AWS or GCP? What if I'm hosting my app on e.g. EKS or GKE respectively?

Re: SQLite Is Serverless

#257

Post by erezsh »

siscia wrote: post #226
mettamage wrote: post #126

Does this mean that you can hack some database storage (w/ sqlite) together on frontend only hosting platforms like Github or Netlify? I think not, but I wonder if some hack is available by virtue of it simply being a file that you can read (and somehow) write to. The best I came up with: let's say you have a toy project, and you call the Github API and replace the file upon every write. Implementing a read is easier…

I will shamelessly plug myself again. I am the main author of RediSQL [1] and I am about to launch a managed service for it. It will allow to write SQL (SQLite dialect) against an HTTP or Redis protocol, to make thing clearer: https://simplesql.carrd.co/ Eventually it will upload your database to an S3 bucket for backup. [1]: https://redisql.com/

Looks pretty cool!

I can imagine it would be useful for data-science projects, if the pricing is right.

Re: SQLite Is Serverless

#258

Post by closeparen »

tmpz22 wrote: post #243
nicholaides wrote: post #185

I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…

How does this work with container/ephemeral services such as typical K8s deployments? Can I trust the file system mounting via resources like StatefulSets or FS mounts? For that matter, Heroku, App Engine, Cloud Functions, whatever? Our current setup is having all our services in kubernetes but our databases in stateful VMs. I do occasionally stuff job-reports and similar data into postgres rows since it's already th…

ETL workers themselves are typically ephemeral, plumbing batches between remote storage systems like Postgres, S3, and Hive. You might use local disk as scratch space during the batch, but not as a sink.

Re: SQLite Is Serverless

#259

Post by tracker1 »

derefr wrote: post #172
bane wrote: post #149

I love SQLite, but people approach it from a classic RDBMS angle which confuses them. Here's the deal: SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file. That's it. Stop overthinking it. Can you write a microservice that stores its data in a big JSON file that you've built some code around to read/write to? Yes. It's just a file, but you have to build all the read/w…

Consider: it’s totally possible to strip down Postgres until all you have left is an embedded RDBMS of the style of SQLite. (I’m not sure why nobody has done this yet, actually.) Would you call the result “just a file format”? Such an instance of “embedded Postgres” would still have a huge sprawling catalog (PG_DATA) directory attached to each use of it, so it wouldn’t be contained to a single file. But neither is SQ…

FirebirdSQL is pretty close to what you're talking about... it's library can do either embedded or you can run a shared server instance. It's really pretty neat, but on the one side, PostgreSQL is probably better, and on the other SQLite is nicer.

I worked on a project a few years ago, where I chose Firebird so I could use literally the same database on potentially offline sites that regularly sync up to a main office (shared) deployment. I worked pretty well and was still a lot of work.

Re: SQLite Is Serverless

#260

Post by wvenable »

bane wrote: post #202
catalogia wrote: post #195

Earlier quoted context omitted.

> No! SQLite is not an embedded RDBMS. It's a file format. You keep saying it's a file format, but it's quite possible to use sqlite without persisting anything to a file at all. rc = sqlite3_open(":memory:", &db);

You can read/write CSV files, JSON, JPEG, WAV, MP3, MP4, etc. into memory as well. That doesn't make any of them an RDBMS. SQLite is a file format. It has a nice API and uses SQL as the domain language for read/write logic. If it didn't use SQL for the logic, would you still be confused?

All other RDBMS products also store their information inside files. I can take a mdb file from SQL Server and copy it to another server and attach it there.

SQLite is obviously a library that stores it's data to a file like millions of other libraries. SQLite is to it's data file the way LAME is to MP3. LAME is not the file format, it's the library.

Post reply on HN