Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Is Serverless

sqlite.org

Post reply on HN
Print view View original

271–280 of 453 posts

Re: SQLite Is Serverless

#271

Post by tracker1 »

pm90 wrote: post #171
giancarlostoro wrote: post #155

Earlier quoted context omitted.

It is, I did Azure Functions which is their "serverless" solution. It's either a type of CGI 'trigger' which is not as impressive however, another 'function' can be triggered based on Database actions or file uploads in their Blob Storage. So it's just handy if you're all in on Azure, and probably other clouds because you can have code run immediately when events happen for certain services. Outside of that, I find i…

> The other buzzword is microservices. I was working a gig that involved running like 10 on a MBP with 16GB of RAM and it froze up the laptop. Java is not a language for micro anything, at least not in regards to memory. Was fun but dreadful to work with more than 3 local services at once. The creators of the framework suggested to mock services, but it was just messy to do that too. This seems like a strange critici…

And when you need to interact with other services in a disconnected way? You have to run them somewhere... and local should generally be an option, even if that's to a local single-node (mini)kube cluster.

There's nothing wrong with needing to run more than one thing while developing/testing and interacting. There are other options, but it's not a clearly bad approach.

That said, I would definitely lean towards Go or Rust myself if given the option (even though I have limited exposure to either) over Java. C#/.Net Core is in the middle imho. Even node works pretty well, but the tons of files thing bugs me sometimes. We're leveraging more node and C# where I work and I definitely prefer node... but for more services I do think that Go and Rust are probably better options.

Re: SQLite Is Serverless

#272

Post by GordonS »

lvh wrote: post #124
GordonS wrote: post #107

Earlier quoted context omitted.

Presumably multiple readers are fine, with only multiple writers being an issue? I vaguely recall trying to use multiple threads to write to an SQLite DB some years ago, and I think it actually locked the entire file for writes. I might remembering wrongly, but I think I switched to reader/writer locks in c# instead, and seeing a huge perf boost.

WAL mode means readers do not block writers and a writer does not block readers. If it locked the entire database you likely weren’t using WAL.

I was definitely using WAL - I know I mentioned using reader/write locks, but that was so the C# locking mechanism worked the way I wanted; I realise that SQLite doesn't block readers on writes.

My point, not well made :), was that a lightweight locking mechanism worked much faster than SQLite's file-based locking mechanism. This was on Windows, mind, so things might be very different on Linux.

Re: SQLite Is Serverless

#273

Post by tracker1 »

jtms wrote: post #159
Frost1x wrote: post #151

Earlier quoted context omitted.

"Serverless" is a garbage marketing term in general. It sounds sexy to nontechnical management folks who are used to hearing a bunch of expensive costs and the word "server" associated with them in some fashion. From that view, anything that gets rid of those pesky "servers" sounds like a win.

This is overly reductive. “Serverless” might be overhyped, but amongst devs and ops it is a perfectly good way to succinctly describe a very specific architecture with real merit. Paying for exactly the compute time i use is pretty sexy to me when I want to deploy and scale a side project.

I would say (X)aaS (whatever as a service) is probably a better term... like DBaaS in this case, vs Serverless as in in-process, which is probably a closer and better use of the term.

Re: SQLite Is Serverless

#274

Post by makapuf »

tenant wrote: post #102
wolf550e wrote: post #87

Earlier quoted context omitted.

Maybe disconnect and reconnect cause windows network share to lose locks or something like that.

Yes, for Access (and sqlite?) it is the client computers that do the edits directly to the database file, rather than via a mediating server side process, so they are particularly vulnerable to network interruptions while in the middle of an edit. Cable is far more reliable than wifi and so far fewer interruptions. I think that's it.

Ok but you cant really say you support network access if you rely on it to be flawless. Working fast, maybe but not reliably working if your network is not super reliable is not really support.

Re: SQLite Is Serverless

#275

Post by [deleted] »

tracker1 wrote: post #264
zachthewf wrote: post #165

Earlier quoted context omitted.

I'm no SQLite expert but by this logic aren't all single-host databases that tick the "Durability" ACID checkbox "just file formats" in the sense that yeah, the bytes we care about exist somewhere in the filesystem? Moreover I'm having trouble coming up with things that I'd associate with a RDBMS and not "just a file format" that SQLite doesn't support. Transactions? SQLite has them. Relational constraints? SQLite ha…

edit - removed

[deleted]

Re: SQLite Is Serverless

#276

Post by yegle »

skrebbel wrote: post #268
cle wrote: post #196

> It is important to understand these two different definitions for "serverless". When a database claims to be "serverless", be sure to discern whether they mean "classic serverless" or "neo-serverless". It's really not important to understand that distinction, because this author seems to be the only one making it. Everyone knows what "serverless" means at this point, and it's not an embedded DB.

I agree! Also the cryptography people should stop calling their hobby "crypto", everybody knows crypto means bitcoin and stuff. Snobs. (here's SQLite's "Serverless" page the way it was in 2007: https://web.archive.org/web/20071115173112/https://www.sqlit... )

I'm not sure if this comment missed a /s somewhere...

Re: SQLite Is Serverless

#277

Post by simonw »

tracker1 wrote: post #264
zachthewf wrote: post #165

Earlier quoted context omitted.

I'm no SQLite expert but by this logic aren't all single-host databases that tick the "Durability" ACID checkbox "just file formats" in the sense that yeah, the bytes we care about exist somewhere in the filesystem? Moreover I'm having trouble coming up with things that I'd associate with a RDBMS and not "just a file format" that SQLite doesn't support. Transactions? SQLite has them. Relational constraints? SQLite ha…

edit - removed

The R in RDBMS stands for Relational, not Remote.

Re: SQLite Is Serverless

#278

Post by at_a_remove »

pjot wrote: post #267
at_a_remove wrote: post #191

Earlier quoted context omitted.

Using SQLite in my ETL processes is something I have done for over a decade. It's just so convenient and, at the end, I have this file that can be examined and queried to see where something might have gone wrong. All of my "temporary" tables are right there for me to look at. It is wonderful!

Would you mind elaborating on your ETL process a little more? Im a junior DE and curious about how I would implement this

It's pretty straightforward, really.

I construct the .sqlite database from scratch each time in Python, building out table after table as I like it.

Some configuration data is loaded in from files first. This could be some default values or even test records for later injection.

The input data is loaded into the appropriate tables and then indexed as appropriate (or if appropriate). It is as "raw" as I can get it.

Each successive transformation occurs on a new table. This is so I can always go back one step for any post-mortem if I need to. Also, I can reference something that might be DELETEd in an a later table.

Often (and this is task-dependent), I will have to pull in data from other server-based databases, typically the target. They get their own tables. Then I can mark certain records as not being present in the target database, so they must be INSERTed. If a record is not present in my input and is there in the target, that would suggest a DELETE. Finally, I can compare records where some ID is present in my input and my .sqlite, they might be good for an UPDATE. All of this is so I can make only the changes that need to be made. Speed is not important to me here, only understanding what changes needed to be made and having a record of what they were and why.

I am happy to say that an ETL process I wrote using this general method back around 2009 is probably still running. I haven't had to touch it in years. Occasionally I will receive questions as to "why did this happen?" and I can just start running queries on the resultant .sqlite database file, kept with the logs, for answers.

Similarly, I can use these sorts of techniques when I am analyzing other datasets. The value here is that I can just refresh one table when the relevant data comes in, rather than having to run the ingest process for everything all over again. This can save me a lot of time.

Re: SQLite Is Serverless

#279

Post by tracker1 »

Gys wrote: post #68
threatripper wrote: post #67

Earlier quoted context omitted.

No, in this case you always have to use rqlite/dqlite because they manage the network synchronization. They use SQLite as storage engine (one SQLite database per server instance).

I understand that in those cases rqlite/dqlite is used. But that it just a technical detail. My point is that I am running two servers: one with the app and Xqlite and another one with Xqlite. In case of a neo-serveless setup, I also have two servers: one with the app, the other one with the db server. So what are the benefits of the Xqlite setup? I looked into that before and for one thing, Xqlite is slower (obvious…

Not the author, or knowing of all the technical details... simplistic replication structure and redundancy/failover without an expensive or more complex RDBMS solution while still self-hosting the service.

There are still a lot of instances where you cannot use a cloud provider for your app or database.

To be honest, I'd probably lean more towards a nosql database that has in the box, relatively easy replication strategy, though that might mean 3+ db servers for good performance. (RethinkDB, MongoDB, Cassandra/ScyllaDB, Cockroach). Just depends on the budget and resources.

Re: SQLite Is Serverless

#280

Post by siscia »

pm90 wrote: post #261
siscia wrote: post #223

Earlier quoted context omitted.

I am working on RediSQL[1] and I am about to launch a managed version. The interface will be either HTTP or Redis protocol, you create your database, and daily I will back it up on S3. (If interested you can subscribe for updated here: https://simplesql.carrd.co/ [1]: https://redisql.com/

Redisql looks pretty sweet! To be frank, and this is only my personal opinion, I don't think I would want to pay for an api, but rather run my own. The business model of providing everything OSS but sending telemetry seems rather intriguing; as a hobbyist user I am ok with such telemetry being collected. If I were to run it in production for a business app though, I wouldn't even bother considering the unpaid version…

Thanks! That was exactly my reasoning.

Make it available and simple for hobbyist and small companies, ask money to who can afford it to sustain the development.

Post reply on HN