Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Is Serverless

sqlite.org

Post reply on HN
Print view View original

121–130 of 453 posts

Re: SQLite Is Serverless

#121

Post by bernawil »

geokon wrote: post #24

a bit tangential, but when do you move form using in-application data structures (maps, trees, vector/arrays) to using a database? Is it basically when the data doesn't fit in memory? I've been programming for almost a decade and I've never come across needing a database... (for context, it's ten years without anything web related) I'm interested in them and I'd love to learn SQL but I can't even think of a use case…

> when do you move from using in-application data structures?

as soon as you need any of these:

* a relational model -because you need to model your data that way or you are required to because someone wants to consume it with tableau-.

* concurrently read/read data between 1+ instances.

* a standard way of doing backups.

moving from that to a database would probably equal a rewrite.

Re: SQLite Is Serverless

#122

Post by pdimitar »

pilif wrote: post #118
pdimitar wrote: post #116

Earlier quoted context omitted.

Centralise your writes to be only done through a single thread/process and then you can read from as many threads/processes you like with no noticeable difference in performance (at least for the several hobby dataset importing projects I tried).

«just write your own server on top of serverless embedded SQLite to get to acceptable performance without needing a client/server database» (Honestly, once you are at the point where concurrency causes performance issues with SQLite, you are better off moving to databases designed to handle concurrency rather than trying to cobble together your own workaround - you have reached the point where the drawback of SQLite‘…

Sure, I don't disagree. I've ditched sqlite for Postgres many times.

It's just that for 99% of the projects I ever worked on, writes are like 100x less than reads so wrapping the writes in a queue of sorts has been quite okay and performant.

It can and it has been coming to a point when it's easier to use a full-blown database server, too.

I was simply pointing out that for a lot of classic workflows wrapping/centralising writes works quite fine.

Re: SQLite Is Serverless

#123

Post by sethev »

oldgradstudent wrote: post #23

> Of those that are serverless, SQLite is the only one known to this author that allows multiple applications to access the same database at the same time. IIRC, MS Access allowed that, which explained a lot of its popularity.

Didn't prevent it is closer to my recollection. My first job was on a system that used this idea pretty heavily - what a nightmare! We only got that code stabilized once we removed any sharing (and later removed Access completely).

Re: SQLite Is Serverless

#124

Post by lvh »

GordonS wrote: post #107
dana321 wrote: post #91

It does allow multiple applications to access the same database at the same time, but when you do so it really hurts performance. I noticed this when i wrote a web crawler in go and used sqlite as the backend. As soon as i connect using the command line interface, it slows down significantly. Just something to bear in mind if you want to use it with multiple processes!

Presumably multiple readers are fine, with only multiple writers being an issue? I vaguely recall trying to use multiple threads to write to an SQLite DB some years ago, and I think it actually locked the entire file for writes. I might remembering wrongly, but I think I switched to reader/writer locks in c# instead, and seeing a huge perf boost.

WAL mode means readers do not block writers and a writer does not block readers. If it locked the entire database you likely weren’t using WAL.

Re: SQLite Is Serverless

#125

Post by peterwwillis »

"Microsoft Azure Cosmo DB and Amazon S3 are examples of a neo-serverless databases."

Can we back away a bit from the bandwagoning of misused terminology? Serverless literally means "running your apps on somebody else's server". S3 is not a server you run your apps on, it is SaaS that you manipulate through an API - you don't put your apps on it. If S3 is serverless, then literally every network service of any kind is serverless.

Re: SQLite Is Serverless

#126

Post by mettamage »

Does this mean that you can hack some database storage (w/ sqlite) together on frontend only hosting platforms like Github or Netlify?

I think not, but I wonder if some hack is available by virtue of it simply being a file that you can read (and somehow) write to.

The best I came up with: let's say you have a toy project, and you call the Github API and replace the file upon every write. Implementing a read is easier as you know where the file is located. This hack shows that you somehow need write access to get any form of performance out of it, because this hack is super slow.

Re: SQLite Is Serverless

#127

Post by logicallee »

pilif wrote: post #118
pdimitar wrote: post #116

Earlier quoted context omitted.

Centralise your writes to be only done through a single thread/process and then you can read from as many threads/processes you like with no noticeable difference in performance (at least for the several hobby dataset importing projects I tried).

«just write your own server on top of serverless embedded SQLite to get to acceptable performance without needing a client/server database» (Honestly, once you are at the point where concurrency causes performance issues with SQLite, you are better off moving to databases designed to handle concurrency rather than trying to cobble together your own workaround - you have reached the point where the drawback of SQLite‘…

As you quoted, people in this discussion are saying that writing a dummy program that has its own sqlite and does nothing but pass messages to it that it receives from all other processes on the system that need to talk to that sqlite file results in much better performance than accessing that file "directly" from the separate processes.

so if everyone's saying this, is there such a standard dummy program?

Re: SQLite Is Serverless

#128

Post by derision »

ComodoHacker wrote: post #113
lxgr wrote: post #36

Earlier quoted context omitted.

Good point. Berkeley DB also supports multiple processes accessing a database concurrently, as far as I know. I was wondering if the authors were referring to SQL-like databases, but MS Access seems to be one?

Yes, MS Access supports both SQL and non-SQL API, and not mentioning it isn't very professional from the author.

Is there anyone actually choosing MS Access for new projects in 2020?

Re: SQLite Is Serverless

#129

Post by kubanczyk »

imafish wrote: post #49

My understanding of serverless = easily scalable, managed service. But somehow the word annoys people. Maybe we should find a better word?

'Instanceless'. Because the difference is that you outsource and forget any questions about the instance(s) of the service (and whatever supports it, like the os process, the kernel, the vm, the real machine, the datacenter).
Post reply on HN