Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Is Serverless

sqlite.org

Post reply on HN
Print view View original

261–270 of 453 posts

Re: SQLite Is Serverless

#261

Post by pm90 »

siscia wrote: post #223
pm90 wrote: post #166

What I really want: SQLite storage backend driver for s3/gcs. No need for disks then. I haven’t been able to find such a solution though; and am not technically proficient enough in C (the Lang SQLite is written in) to do so myself.

I am working on RediSQL[1] and I am about to launch a managed version. The interface will be either HTTP or Redis protocol, you create your database, and daily I will back it up on S3. (If interested you can subscribe for updated here: https://simplesql.carrd.co/ [1]: https://redisql.com/

Redisql looks pretty sweet! To be frank, and this is only my personal opinion, I don't think I would want to pay for an api, but rather run my own. The business model of providing everything OSS but sending telemetry seems rather intriguing; as a hobbyist user I am ok with such telemetry being collected. If I were to run it in production for a business app though, I wouldn't even bother considering the unpaid version for the following reasons:

1. I do not want my production instance to shut down for WHATEVER reason. This is just not an acceptable risk for most businesses. The only time a DB can go down is something goes wrong.

2. As an engineer, I understand that 3 counters that are not accurate aren't a big deal. I can even look into the source and see that they really do as you say. Justifying this to a security org will be a complete nightmare, as most security orgs in enterprises are staffed with barely technical folks masquerading as "security".

So, it seems like a pretty good way to coerce enterprises to pay up while letting hobbyists continue using it. Very smart, I wish you the very best!

Re: SQLite Is Serverless

#262

Post by tracker1 »

bane wrote: post #202
catalogia wrote: post #195

Earlier quoted context omitted.

> No! SQLite is not an embedded RDBMS. It's a file format. You keep saying it's a file format, but it's quite possible to use sqlite without persisting anything to a file at all. rc = sqlite3_open(":memory:", &db);

You can read/write CSV files, JSON, JPEG, WAV, MP3, MP4, etc. into memory as well. That doesn't make any of them an RDBMS. SQLite is a file format. It has a nice API and uses SQL as the domain language for read/write logic. If it didn't use SQL for the logic, would you still be confused?

SQLite is a a library the implements an embedded database engine. The file that it stores this data in is an artifact, not the interface.

Re: SQLite Is Serverless

#263

Post by kgwgk »

bane wrote: post #149

I love SQLite, but people approach it from a classic RDBMS angle which confuses them. Here's the deal: SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file. That's it. Stop overthinking it. Can you write a microservice that stores its data in a big JSON file that you've built some code around to read/write to? Yes. It's just a file, but you have to build all the read/w…

> SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file.

is a collection of bits with a nice interface that uses a query language as the paradigm for reading/writing data.

Re: SQLite Is Serverless

#264

Post by tracker1 »

zachthewf wrote: post #165
bane wrote: post #149

I love SQLite, but people approach it from a classic RDBMS angle which confuses them. Here's the deal: SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file. That's it. Stop overthinking it. Can you write a microservice that stores its data in a big JSON file that you've built some code around to read/write to? Yes. It's just a file, but you have to build all the read/w…

I'm no SQLite expert but by this logic aren't all single-host databases that tick the "Durability" ACID checkbox "just file formats" in the sense that yeah, the bytes we care about exist somewhere in the filesystem? Moreover I'm having trouble coming up with things that I'd associate with a RDBMS and not "just a file format" that SQLite doesn't support. Transactions? SQLite has them. Relational constraints? SQLite ha…

edit - removed

Re: SQLite Is Serverless

#265

Post by tracker1 »

pilif wrote: post #118
pdimitar wrote: post #116

Earlier quoted context omitted.

Centralise your writes to be only done through a single thread/process and then you can read from as many threads/processes you like with no noticeable difference in performance (at least for the several hobby dataset importing projects I tried).

«just write your own server on top of serverless embedded SQLite to get to acceptable performance without needing a client/server database» (Honestly, once you are at the point where concurrency causes performance issues with SQLite, you are better off moving to databases designed to handle concurrency rather than trying to cobble together your own workaround - you have reached the point where the drawback of SQLite‘…

Depends on what you're doing... If you're working in descrete projects that need to be run, then archived.. using an RDBMS vs a service over SQLite is an actual consideration... if you have a remote API interface that talks to different SQLite db files on a per project basis, then backup and archival become trivial matters... if you're using a classic RDBMS then it can become much more complicated.

Aside, adjusting schema over time also becomes easier as archived projects don't need to be updated, they just continue to exist with the older schema.

Re: SQLite Is Serverless

#266

Post by tenant »

paulie_a wrote: post #150
slfnflctd wrote: post #117

Earlier quoted context omitted.

You basically had to engineer around the corruption risks for anything actually in production. Data redundancy, old-school 'Save' buttons (your work isn't properly saved until you click it, because it sends copies of that work to multiple backends) and isolating users as much as possible to their own 'shards' was part of how I saw it kludged through in the real world. There was still a lot of weird behavior, though,…

Penny wise pound foolish. I had a president waste two to three hours a day instead of using a 40 dollar service to take care of the incredibly simple and repetitive task. Eventually when I was planning to leave anyways I asked him what his time was with per hour. I guess it was less than minimum wage at the time. (5.25 an hour) I'm probably going to have nightmares filled with screens of the access database corruptio…

What's anyone's time worth? Might have kept him from doing something foolish.

Re: SQLite Is Serverless

#267

Post by pjot »

at_a_remove wrote: post #191
nicholaides wrote: post #185

I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…

Using SQLite in my ETL processes is something I have done for over a decade. It's just so convenient and, at the end, I have this file that can be examined and queried to see where something might have gone wrong. All of my "temporary" tables are right there for me to look at. It is wonderful!

Would you mind elaborating on your ETL process a little more? Im a junior DE and curious about how I would implement this

Re: SQLite Is Serverless

#268

Post by skrebbel »

cle wrote: post #196

> It is important to understand these two different definitions for "serverless". When a database claims to be "serverless", be sure to discern whether they mean "classic serverless" or "neo-serverless". It's really not important to understand that distinction, because this author seems to be the only one making it. Everyone knows what "serverless" means at this point, and it's not an embedded DB.

I agree! Also the cryptography people should stop calling their hobby "crypto", everybody knows crypto means bitcoin and stuff. Snobs.

(here's SQLite's "Serverless" page the way it was in 2007: https://web.archive.org/web/20071115173112/https://www.sqlit...)

Re: SQLite Is Serverless

#269

Post by tracker1 »

logicallee wrote: post #127
pilif wrote: post #118

Earlier quoted context omitted.

«just write your own server on top of serverless embedded SQLite to get to acceptable performance without needing a client/server database» (Honestly, once you are at the point where concurrency causes performance issues with SQLite, you are better off moving to databases designed to handle concurrency rather than trying to cobble together your own workaround - you have reached the point where the drawback of SQLite‘…

As you quoted, people in this discussion are saying that writing a dummy program that has its own sqlite and does nothing but pass messages to it that it receives from all other processes on the system that need to talk to that sqlite file results in much better performance than accessing that file "directly" from the separate processes. so if everyone's saying this, is there such a standard dummy program?

Given the extensions that people compile into SQLite, I don't think one can even say that... there's mention of ActorDB in another thread that seems to cover this...

Wrapping a typical API in either GRPC or even over HTTP isn't too hard.. you take in a parameterized query with parameters using library X and return JSON stringified to that library... you may want more specific interfaces, or even GraphQL over the database for that matter... it really depends on your needs.

Re: SQLite Is Serverless

#270

Post by celticmusic »

oldgradstudent wrote: post #23

> Of those that are serverless, SQLite is the only one known to this author that allows multiple applications to access the same database at the same time. IIRC, MS Access allowed that, which explained a lot of its popularity.

JET database are what MS Access used underneath, and attempting multiple concurrent usage on JET databases is a good way to corrupt them.
Post reply on HN