Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Is Serverless

sqlite.org

Post reply on HN
Print view View original

231–240 of 453 posts

Re: SQLite Is Serverless

#231

Post by chrisweekly »

at_a_remove wrote: post #191
nicholaides wrote: post #185

I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…

Using SQLite in my ETL processes is something I have done for over a decade. It's just so convenient and, at the end, I have this file that can be examined and queried to see where something might have gone wrong. All of my "temporary" tables are right there for me to look at. It is wonderful!

Yes! Along these lines I heartily recommend `lnav` ^1, a fantastic, lightweight, scriptable CLI mini-ETL tool w embedded sqlite engine, ideally suited for working with moderately-sized data sets (ie, millions of rows not billions) ... so useful!

1. https://lnav.org

Re: SQLite Is Serverless

#232

Post by bane »

jturpin wrote: post #227
bane wrote: post #149

I love SQLite, but people approach it from a classic RDBMS angle which confuses them. Here's the deal: SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file. That's it. Stop overthinking it. Can you write a microservice that stores its data in a big JSON file that you've built some code around to read/write to? Yes. It's just a file, but you have to build all the read/w…

Sqlite implements all of the features you would expect from a relational database, including indexes, transactions, write ahead logging, consistency, and provides a SQL wrapper on top of its file API. _This is what makes it a database and not just a file format_. To the client, whether or not the API is happening over a socket or a locally linked library is irrelevant. How connection pooling works is also not relevan…

https://sqljet.com/

Re: SQLite Is Serverless

#233

Post by 0xff00ffee »

cle wrote: post #196

> It is important to understand these two different definitions for "serverless". When a database claims to be "serverless", be sure to discern whether they mean "classic serverless" or "neo-serverless". It's really not important to understand that distinction, because this author seems to be the only one making it. Everyone knows what "serverless" means at this point, and it's not an embedded DB.

"neo-serverless"

LOL GTFOUTTAHERE!

Such an arrogant programmer. I see this on HN every now and then: a programmer discovers something and thinks "I bet I'm first to notice this!", then proceeds to invent new lexicon to feel special.

Re: SQLite Is Serverless

#234

Post by yyyk »

stuartc842 wrote: post #221
lucideer wrote: post #213

Earlier quoted context omitted.

> Have you tried to figure out where to install the server or asked what the system requirements were for it? Not exactly, but asking "where to install the server & client and what are the system requirements for each" is not that different to asking "where to install the client and what are the system requirements for it", even when there is no server. > Have you grown concerned that once the system moves into produ…

bane said "not stupid"

That was a later Edit. The original did not have the word "not".

I make these errors sometimes too and need to edit. HN's method beats Twitter, but it would be nice if we could see a versioned history of each message...

Re: SQLite Is Serverless

#235

Post by gpvos »

verdverm wrote: post #2

Choose your own definitions... but SQLite is not serverless by common parlance. Author creates two definitions for serverless which don't match the common usage. Serverless is more about DevOps / deploy experience than how the program leverages OS processes internally. Apparently MS and AWS are ISPs? Maybe SQLite could be serverless if you defined it as incapable or running as a server on its own?

Serverless used to have a very clear definition: not having or using a server, so SQLite is a perfect example of serverlessness. I was extremely confused when the newfangled definition (called neo-serverless on TFA) showed up, or your definition for that matter. Who ever thought up theses confusing meanings for a word that used to be perfectly clear?

You don't need an ISP to have a server. Any computer or program that listens to a network port is a server.

Re: SQLite Is Serverless

#236

Post by cma »

bane wrote: post #224
cma wrote: post #217

Earlier quoted context omitted.

It can open multiple files (the transaction log).

Almost any modern programming language can open multiple files.

At a minimum it is multiple file formats. But really it is multiple file formats + a fairly intricate library for dealing with them along with locks etc.

According to their home page the SQLite database file format is a file format, and SQLite is a library.

Re: SQLite Is Serverless

#237

Post by bshipp »

slfnflctd wrote: post #110
wenc wrote: post #95

Earlier quoted context omitted.

MS Access “allowed” it but depending on the version it could be quite problematic. We had a use case where Tableau connected to an Access file read only (but which another program used as a data store and wrote to it often) on an Windows file share, and once in a while the lock files would get screwy and we would have to manually delete the lock files to get things working again. Deleting lock files could be a huge c…

> Access is really meant for single-user scenarios In a lot of cases, I've found it to be the best tool for a temporary or one-off (preferably smaller scale) data mining/massaging project. The query-building interface was the way I originally learned the basics of relational databases, and it also helped me get a better grasp of SQL-- the ability to flip back & forth from the GUI query builder to the SQL it generates…

My recollection was that the automated wizard loved to nest statements and it was a huge chore to manually massage stuff after the fact; almost incomprehensible to parse. but I agree that it was a real godsend for people trying to branch out from the limitations of excel.

Re: SQLite Is Serverless

#238

Post by cma »

bane wrote: post #225
cma wrote: post #214

Earlier quoted context omitted.

SQLite is a whole library with complicated locking protocol options etc. and works with multiple files of different formats (e.g. separate transaction log).

Since the file format specification is available, you can write your own code that directly reads/writes to SQLite files. You can't even have to use the logs and journaling options. You don't even have to use SQL.

That’s the SQLite database file format. SQLite itself is a library that can deal with multiple files in different formats (transaction log and database file). It also reads .sqliterc and maybe others.

Re: SQLite Is Serverless

#239

Post by hinkley »

bane wrote: post #149

I love SQLite, but people approach it from a classic RDBMS angle which confuses them. Here's the deal: SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file. That's it. Stop overthinking it. Can you write a microservice that stores its data in a big JSON file that you've built some code around to read/write to? Yes. It's just a file, but you have to build all the read/w…

I think you’re stuck in 90/10 rule territory here. But even so, SQLite was 220,000 lines the last time they measured, which was five years ago. You can do a lot of functionality in 22kloc, even ignoring the other 90%, which you shouldn’t

Lodash, for instance, is much smaller than 22k lines, and it “just” manipulates objects and lists.

If you downplay others like this, I wonder how you feel about your own work. Have you been working hard for years on something that “just” accomplishes a straightforward task? Are you happy? I know I wasn’t.

Re: SQLite Is Serverless

#240

Post by gwd »

bane wrote: post #200
lazyant wrote: post #183

Earlier quoted context omitted.

Sure. Or a light RDBMS where the main not supported case is concurrent writes.

No, SQLite provides no RDBMS functionality. It is not an RDBMS. It is a file format. Saying that a program that opens and reads/writes a file through a file format API is a light RDBMS turns almost every program in history into an RDBMS. If SQLite didn't force you to use SQL as the read/write logic, absolutely nobody would confuse it for an RDBMS. That's because it's a file format.

> No, SQLite provides no RDBMS functionality.

I think your comment would be more comprehensible if you gave some examples of the kind of functionality you think is missing.

I don't know what you mean by "RDBMS", but JSON and XML don't do joins, don't do views, don't do efficient query plans, and so on. It's either ignorance or obstinacy to say SQLite is just a file format.

Post reply on HN