Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Is Serverless

sqlite.org

Post reply on HN
Print view View original

361–370 of 453 posts

Re: SQLite Is Serverless

#361

Post by PaulDavisThe1st »

cle wrote: post #329
jiofih wrote: post #322

Earlier quoted context omitted.

This such an air-headed comment to make. You must realize that page describing how SQLite is “serverless” has been up probably longer than your entire adult life. It is important in its context, they are not trying to “redefine” (lol) the term.

It doesn’t matter. It is not the generally accepted definition of serverless. The meaning of a word is based on how it is generally used, and almost nobody means this when they use the word “serverless”. At this point trying to use the word in this way just creates a bunch of unnecessary confusion. Call it something else so we can move on to more important (and clear) discussions. (Also there are a lot of assumptions…

No doubt you're a little surprised that hacker news doesn't contain that many posts about people breaking into computer information systems, amirite?

Re: SQLite Is Serverless

#362

Post by rakoo »

palotasb wrote: post #308
sly010 wrote: post #300

Earlier quoted context omitted.

I have been using SQLite as a format to move data between steps in a complicated batch processing pipeline. With the right pragmas it is both faster and more compact than JSON. It is also much more "human readable" than gigabytes of JSON. I only wish there was a way to open an http-fetched SQLite database from memory so I don't have to write it to disk first.

You can use the memvfs module to load in-memory databases if you're using the C API. I'm not sure how many higher-level APIs support it though. [1] https://stackoverflow.com/a/53453338/3063 [2] https://www.sqlite.org/loadext.html#example_extensions [3] https://www.sqlite.org/src/file/ext/misc/memvfs.c

A very interesting approach is sqltorrent (https://github.com/bittorrent/sqltorrent): the sqlite file is shared in a torrent, and all queries will touch a specific part of the file, which is downloaded on-demand.

Also check https://github.com/lmatteis/torrent-net

Re: SQLite Is Serverless

#363

Post by elyobo »

nicholaides wrote: post #185

I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…

We do something like this; one of the outputs of the data pipeline is an sqlite file that's deployed nightly along with code to App Engine. The sqlite stuff is all read only, read/write data for the app is stored in firestore instead.

We initially used json but ran in to memory issues; sqlite is more memory efficient and being able to use SQL instead of the wild SQL-esque is both faster and more reliable.

Re: SQLite Is Serverless

#364

Post by catalogia »

manigandham wrote: post #358
filleduchaos wrote: post #335

Earlier quoted context omitted.

This article was written in 2007, so yes, people have been using the word "serverless" before managed service providers decided to take it up as a buzzword.

Nobody used it before. Even the about page says: "SQLite is an embedded SQL database engine." "serverless" is a marketing term, then and now. Not sure why there's such a big defense of it. If you have to argue this much over the provenance of a term that clearly isn't used the same way today then it's a good sign that it's not very useful.

> Nobody used it before.

The archive.org links already provided to you prove that you're flat wrong about that. SQLite used the term many years before it was a webshit buzzword. They used it in the intuitive straight forward english sense; somebody who has no hat is hatless, somebody who has no home is homeless, and a database system that has no server process is serverless. In 2007 when it was written, nobody would have batted at eye at this term, the meaning would have been immediately clear to anybody who had any familiarity with databases.

Re: SQLite Is Serverless

#365

Post by adamcharnock »

wpietri wrote: post #148
geokon wrote: post #24

a bit tangential, but when do you move form using in-application data structures (maps, trees, vector/arrays) to using a database? Is it basically when the data doesn't fit in memory? I've been programming for almost a decade and I've never come across needing a database... (for context, it's ten years without anything web related) I'm interested in them and I'd love to learn SQL but I can't even think of a use case…

I'm sure people will give you the orthodox answer, so let me give you mine: when other things become more important than performance, ease of development, and code clarity. Many years ago I happened to meet the creator of Prevayler, an open-source persistence framework that provided ACID guarantees and was thousands of times faster than a database as long as your data fit in RAM. I tried it out for a project and we l…

Prevayler sounds a lot like Redis, unless I’m missing something.

I’m a big fan of Redis, but I was bitten many years ago when I tried to use it as a replacement for an RDBMS. There were two reasons for this: 1) lack of development libraries and operational tools, and 2) lack of data integrity checks.

1 has changed these days, but 2 is still very much the case (and rightly so, IMHO). Perhaps Prevayler had this?

But yep, it was fast at a time when our competition’s software was slow and clunky. It made a difference.

Re: SQLite Is Serverless

#366

Post by manigandham »

catalogia wrote: post #364
manigandham wrote: post #358

Earlier quoted context omitted.

Nobody used it before. Even the about page says: "SQLite is an embedded SQL database engine." "serverless" is a marketing term, then and now. Not sure why there's such a big defense of it. If you have to argue this much over the provenance of a term that clearly isn't used the same way today then it's a good sign that it's not very useful.

> Nobody used it before. The archive.org links already provided to you prove that you're flat wrong about that. SQLite used the term many years before it was a webshit buzzword. They used it in the intuitive straight forward english sense; somebody who has no hat is hat less , somebody who has no home is home less , and a database system that has no server process is server less . In 2007 when it was written, nobody…

Just because a single page on the internet used it does not mean the term was used in the industry. It also doesn't matter if it technically makes sense, although it's a stretch (because there is a server process, you just share it).

It's a marketing term, and a poor one at that. That's why SQLite even describes itself as a embedded, in-process database without client/server architecture. Because that's the common jargon.

I'm surprised at the endless defense of a marketing buzzword and the argument over which marketing definition is the "real" one. This is bikeshedding at its finest.

Re: SQLite Is Serverless

#367

Post by jjgreen »

FridgeSeal wrote: post #356
sly010 wrote: post #300

Earlier quoted context omitted.

I have been using SQLite as a format to move data between steps in a complicated batch processing pipeline. With the right pragmas it is both faster and more compact than JSON. It is also much more "human readable" than gigabytes of JSON. I only wish there was a way to open an http-fetched SQLite database from memory so I don't have to write it to disk first.

This is a amazing, I think you may have just solved and headed-off a huge number odd problems for me. Could you talk more about what Pragmas you’ve been using and why?

Not the OP, but I find `PRAGMA synchronous = OFF` makes the creation of DBs vastly faster ...

Re: SQLite Is Serverless

#368

Post by quasarj »

cle wrote: post #329
jiofih wrote: post #322

Earlier quoted context omitted.

This such an air-headed comment to make. You must realize that page describing how SQLite is “serverless” has been up probably longer than your entire adult life. It is important in its context, they are not trying to “redefine” (lol) the term.

It doesn’t matter. It is not the generally accepted definition of serverless. The meaning of a word is based on how it is generally used, and almost nobody means this when they use the word “serverless”. At this point trying to use the word in this way just creates a bunch of unnecessary confusion. Call it something else so we can move on to more important (and clear) discussions. (Also there are a lot of assumptions…

For sure. Everyone knows hackers are criminals now. So why are we all here on a public criminal forum discussing our crime?

Re: SQLite Is Serverless

#369

Post by quasarj »

cle wrote: post #338
mikestew wrote: post #317

Earlier quoted context omitted.

Everyone know that “serverless” anything doesn’t run on a server , be it AWS or Azure instances or what have you. That would be both ironic and silly, like someone used the wrong word or something. Serverless databases and what have you have been around longer than the current batch of folks trying to redefine things (or more charitably, ran out names to call things). Like or not, there is a distinction even if the o…

Nobody called them serverless DBs, and nobody does. There is a vocal minority of reductionists here on HN who dismiss the accepted definition of serverless because the literal meaning doesn’t make sense. It’s just noise though. “Serverless” does have a specific meaning and it’s not “there are no servers anywhere”. I was around when there was no serverless. Things change, new words arise. Time for us to get with the t…

nah

Re: SQLite Is Serverless

#370

Post by platform »

elyobo wrote: post #363
nicholaides wrote: post #185

I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…

We do something like this; one of the outputs of the data pipeline is an sqlite file that's deployed nightly along with code to App Engine. The sqlite stuff is all read only, read/write data for the app is stored in firestore instead. We initially used json but ran in to memory issues; sqlite is more memory efficient and being able to use SQL instead of the wild SQL-esque is both faster and more reliable.

Yes, I have been doing same thing, only with LMDB.

I do not think LMDB could load from in-memory only object (as it has to have file to memory-map to), however.

But same design reasons, I wanted something that

a) I can move across host architectures

b) something that can act as key-val cache, as soon as the processes using it are restarted (so no cache hydrating delay)

c) something that I can diff/archive/restore/modify in place

We tested sqllite for the above purpose at the time, and writing speed and ( b ) - lmdb was significantly faster.

So we lost the flexibility of SQLite, but I felt it was a reasonable tradeoff, given our needs.

I also know that one of the Intel's python toolkits for image recognition/ai, uses LMDB (optionally) store images that processing routines do not have incur the cost of directory lookups when touching millions of small images. (forgot the name of the toolkit though)…

Overall, this a very valid practice/pattern in data processing pipelines, kudos to you for mentioning it.

Post reply on HN