Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Is Serverless

sqlite.org

Post reply on HN
Print view View original

421–430 of 453 posts

Re: SQLite Is Serverless

#421

Post by dredmorbius »

fctorial wrote: post #386
sly010 wrote: post #300

Earlier quoted context omitted.

I have been using SQLite as a format to move data between steps in a complicated batch processing pipeline. With the right pragmas it is both faster and more compact than JSON. It is also much more "human readable" than gigabytes of JSON. I only wish there was a way to open an http-fetched SQLite database from memory so I don't have to write it to disk first.

> I only wish there was a way to open an http-fetched SQLite database from memory so I don't have to write it to disk first. Ramfs?

tmpfs is the better-behaved option should you run out of resources, see:

https://www.jamescoyle.net/knowledge/951-the-difference-betw...

I'm still remembering old-school ramdisks under Linux which were finite in both number and size, both to quite small extents. I think there were 8 (or 12 or 16?) total ramdisks available, of only 2-4 MB each, configurable with LILO boot options.

That's now ... mostly taking up valuable storage in my own brain for no useful effect.

Re: SQLite Is Serverless

#422

Post by skrebbel »

mcpeepants wrote: post #339
yegle wrote: post #276

Earlier quoted context omitted.

I'm not sure if this comment missed a /s somewhere...

it wouldn't be wrong to include one, but it didn't necessarily need it. the link in parentheses serves the same purpose.

nitpick: it would be wrong to include one, because sarcasm isn't sarcasm if you say it's sarcasm.

personally i feel like even including the archive.org link was pushing it over the edge :)

Re: SQLite Is Serverless

#423

Post by vturner »

nicholaides wrote: post #185

I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…

Terrific! Data pipelines I've built have had JSON as their intermediary steps which I'm growing weary of.

Re: SQLite Is Serverless

#424

Post by fctorial »

dredmorbius wrote: post #421
fctorial wrote: post #386

Earlier quoted context omitted.

> I only wish there was a way to open an http-fetched SQLite database from memory so I don't have to write it to disk first. Ramfs?

tmpfs is the better-behaved option should you run out of resources, see: https://www.jamescoyle.net/knowledge/951-the-difference-betw... I'm still remembering old-school ramdisks under Linux which were finite in both number and size, both to quite small extents. I think there were 8 (or 12 or 16?) total ramdisks available, of only 2-4 MB each, configurable with LILO boot options. That's now ... mostly taking up valua…

It looks like a good intro, thanks. I wasn't aware of these technologies, but I knew it was possible to build an FS in RAM. So I just put these two keywords together.

Re: SQLite Is Serverless

#425

Post by siscia »

erezsh wrote: post #420
siscia wrote: post #342

Earlier quoted context omitted.

Hummm, what you mean by data science? Like running analysis in a Jupiter notebook? Indeed I am not sure this case is a good fit...

Yes, that's what I meant. And possibly for ML preprocessing. Just out of curiosity, who do you imagine your users will be?

The problem with data science is that usually you have relatively big datasets, you care more about throughput than latency and you work in secure an environment where you can definitely have access to the database credentials.

Streaming over the network the result of a big select is not ideal, moreover I believe that data scientist prefer to work with common technologies. I mean that there are already adapter for SQLite or PG or MySQL, while for RediSQL it won't be as straightforward.

I am thinking to developers for the JAM stack would be interested to this sort of API. Or people that want a database without having to think too much about it.

Re: SQLite Is Serverless

#426

Post by zubairq »

Yep SQLite is in fact serverless. We based our open source web based IDE on SQLite and it works fantastically. The best part is that you can even take a SQLite database and use sql.js and run it offline in the browser!

Re: SQLite Is Serverless

#427

Post by ldiracdelta »

szatkus wrote: post #320
nebulous1 wrote: post #245

Earlier quoted context omitted.

I've spent half an hour or so looking at this thread, and my take away is that you're largely just confusing matters even further. > It's just a file format. This is clearly incorrect. It does encompass a file format, but it also contains code to manage that file. The existence of sqljet does not change this, it's merely a different database management system that uses the same file format. You also seem to mostly ig…

It's a file format with a very mature and nice library around it. At the end of a day it's just fwrite() and fread() with SQL syntax.

And perhaps python is just glorified assembly, but it seems to me to be a "difference in degree that leads to a difference in kind."

Re: SQLite Is Serverless

#428

Post by nomadluap »

bob1029 wrote: post #69
geokon wrote: post #57

Earlier quoted context omitted.

will you store large objects like images or audio file directly in SQL?

In most cases I will. Pulling a blob out of a row is a lot faster than opening an additional file handle. There aren't really any downsides to this either unless you are running table scans. SQLite does support indexes (even full-text) and they do work miracles so do use them when necessary. I would go so far as to argue that SQLite could be used to store all of the assets for any large piece of software (I.e. a AAA…

SQLite has a page measuring performance of internal blobs vs external files: https://www.sqlite.org/intern-v-extern-blob.html

Re: SQLite Is Serverless

#429

Post by HelloNurse »

catalogia wrote: post #143
FpUser wrote: post #137

Earlier quoted context omitted.

Let's say there are concurrent writes to a single table with few indexes. Process of updating said table and indices eventually boils down to update of a single B+ tree from a multiple threads. In general until serialized it is physically impossible without corrupting the tree. Sure there might be other specially crafted implementation of table more friendly to concurrent writes but there will be trade-offs. There ar…

If you need concurrent/parallel writes, SQLite is not the right tool for you. You may as well lament that hammers are no good for driving screws.

Some concurrent writes are "a transaction can wait for others to finish, up to how long it takes for a person to grow bored" and some concurrent writes are "if a transaction cannot finish without waiting the application is already struggling with a severe performance problem" (for example, because it needs multiple concurrent writes on different disks to keep up with incoming data).

Re: SQLite Is Serverless

#430

Post by at_a_remove »

julianee wrote: post #406
at_a_remove wrote: post #297

Earlier quoted context omitted.

I am not a very talented programmer so I stick very close to what is common, standard, and easy to understand. It usually means I am on the downslope of the hype cycle and it limits some opportunities but I have become okay with that. I have gotten some CS students who were about to shoot flies with various cannons turned on to SQLite. I kept a couple of the decent books about it nearby and would shove it into their…

Do you still have the titles of those books at hand? I'd love to take a look at them.

They are The Definitive Guide to SQLite by Mike Owens and Using SQLite by Jay A. Kreibich. I am quite sure they are more book than I needed, I only plumbed a fraction of SQLite's immense capabilities.
Post reply on HN