Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Is Serverless

sqlite.org

Post reply on HN
Print view View original

321–330 of 453 posts

Re: SQLite Is Serverless

#321

Post by m0xte »

JasonFruit wrote: post #216
m0xte wrote: post #135

Earlier quoted context omitted.

It was ok pre-windows 2000. I had access 97 it on 9 workstations on NT with never a problem! Moment office 2000 and windows 2000 came out it went to hell. Moved to SQL Server then. I am not sure what changed. It was quite frankly the most productive custom business software package I have ever used. Literally would take 10 people a week to do something custom you could do in an afternoon with 1 person in Access. I su…

We had a much heavier multiuser setup on Access 97, and it worked fine. We had the same breakdown as you did, and had to do a registry edit on all users' machines to keep it limping along until we could move to SQL Server, which was the right thing to do anyway.

Interesting. Nice to know it wasn't just me. There was no one to talk to about it back then so you had to think on your toes.

SQL Server was a crap load easier to back up reliably

Re: SQLite Is Serverless

#322

Post by jiofih »

cle wrote: post #196

> It is important to understand these two different definitions for "serverless". When a database claims to be "serverless", be sure to discern whether they mean "classic serverless" or "neo-serverless". It's really not important to understand that distinction, because this author seems to be the only one making it. Everyone knows what "serverless" means at this point, and it's not an embedded DB.

This such an air-headed comment to make. You must realize that page describing how SQLite is “serverless” has been up probably longer than your entire adult life. It is important in its context, they are not trying to “redefine” (lol) the term.

Re: SQLite Is Serverless

#323

Post by kondro »

pm90 wrote: post #256
tn1 wrote: post #192

Earlier quoted context omitted.

The challenge with this is that S3 and friends are object stores, meaning you upload or download the whole file each time. As you can imagine, this will cost you tremendous bandwidth even to insert one row. Furthermore, it doesn't solve the multiple writers problem, because (afaik) there's no way to lock a file on S3.

> As you can imagine, this will cost you tremendous bandwidth even to insert one row. Is this the cost of network transfers changed by AWS or GCP? What if I'm hosting my app on e.g. EKS or GKE respectively?

It costs roughly $5/million PUTs and $0.40/million GETs on S3 in addition to the bandwidth and storage you use.

S3 objects are also immutable. Once they’re written, they can’t be updated.

A read-only version of this might be useful, but probably wouldn’t work in-place.

Something that might be of interest is S3 SELECT support that lets you query a single (optionally compressed) CSV, JSON or Parquet file server-side at the same cost of a regular S3 GET.

https://docs.aws.amazon.com/AmazonS3/latest/dev/selecting-co...

And if you really want relational (i.e. JOINs, aggregations and sub-queries) semantics on a bucket full of CSV, JSON, Parquet, ORC or regular-expression-describable files in a cost effective way that has great performance on buckets containing 100’s of TBs of data, definitely look at Athena which is only $5/TB of data scanned during a query.

Re: SQLite Is Serverless

#324

Post by bane »

nebulous1 wrote: post #245
bane wrote: post #149

I love SQLite, but people approach it from a classic RDBMS angle which confuses them. Here's the deal: SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file. That's it. Stop overthinking it. Can you write a microservice that stores its data in a big JSON file that you've built some code around to read/write to? Yes. It's just a file, but you have to build all the read/w…

I've spent half an hour or so looking at this thread, and my take away is that you're largely just confusing matters even further. > It's just a file format. This is clearly incorrect. It does encompass a file format, but it also contains code to manage that file. The existence of sqljet does not change this, it's merely a different database management system that uses the same file format. You also seem to mostly ig…

You can read and write sqlite files entirely without the library. The file format is available on the sqlite site. SQLJet proves the point that you don't need 139,000 lines of C code to use sqlite files.

Re: SQLite Is Serverless

#325

Post by erezsh »

siscia wrote: post #309
erezsh wrote: post #307

Earlier quoted context omitted.

I'm far from an expert on such things, but I like the adjustable fixed-price model (like digital ocean). Whenever I see a price like 0.0002$ per hour, I feel like they're trying to mess with my intuition. That's especially applicable to data-science, because you're not worried so much about automatic scaling, you just don't want to be surprised at the end of the month. I don't know if you can match digital ocean's pr…

I was thinking something based on size or performance. 1 req for second, free. 50 req for second, 5€. Unmetered, 30€. Maybe not counting for seconds but for hour or day so to accommodate burst. Another option would be on features. But that will requires time.

If you have full support for SQL, it means some queries can run for a few seconds, possibly even minutes (even sqlite supports some form of bfs). So will you just time-out those requests? That would make a lot of possible uses suddenly impossible.

I might consider the unmetered option, if the performance gain is worth the 6x cost of other providers.

For web and games, maybe your existing model could work, but then always having a recent backup also becomes more important (and for most sites, the lack of acid is a deal-breaker).

The nice thing about data-science is that you interact with the workspace yourself, so you know exactly when you want to save a snapshot.

Re: SQLite Is Serverless

#326

Post by bane »

cma wrote: post #238
bane wrote: post #225

Earlier quoted context omitted.

Since the file format specification is available, you can write your own code that directly reads/writes to SQLite files. You can't even have to use the logs and journaling options. You don't even have to use SQL.

That’s the SQLite database file format. SQLite itself is a library that can deal with multiple files in different formats (transaction log and database file). It also reads .sqliterc and maybe others.

> That’s the SQLite database file format.

You got it! SQLite is a file format.

> SQLite itself is a library that can deal with multiple files in different formats (transaction log and database file). It also reads .sqliterc and maybe others.

Yup you also got it! SQLite is not an RDBMS and shouldn't be approached that way.

The more you try to ram RDBMS ideas into what SQLite is, the more it won't be that. The more you try to treat it like a file format, the more it will be what you want.

Re: SQLite Is Serverless

#327

Post by giancarlostoro »

wwright wrote: post #179
giancarlostoro wrote: post #155

Earlier quoted context omitted.

It is, I did Azure Functions which is their "serverless" solution. It's either a type of CGI 'trigger' which is not as impressive however, another 'function' can be triggered based on Database actions or file uploads in their Blob Storage. So it's just handy if you're all in on Azure, and probably other clouds because you can have code run immediately when events happen for certain services. Outside of that, I find i…

Java has been used on servers since 256MB was common to use — Java is not a limiting factor in how lightweight you can get on modern Intel hardware, though it might need some tuning.

The bloat was in the framework used, not as much on Java itself.

Re: SQLite Is Serverless

#328

Post by coldtea »

cle wrote: post #196

> It is important to understand these two different definitions for "serverless". When a database claims to be "serverless", be sure to discern whether they mean "classic serverless" or "neo-serverless". It's really not important to understand that distinction, because this author seems to be the only one making it. Everyone knows what "serverless" means at this point, and it's not an embedded DB.

>Everyone knows what "serverless" means at this point, and it's not an embedded DB.

Serverless is a marketing term at this point (as it was when it started). This post brings some welcome definitions and expands it to something that has many of the same attributes but wasn't appreciated as such.

Re: SQLite Is Serverless

#329

Post by cle »

jiofih wrote: post #322
cle wrote: post #196

> It is important to understand these two different definitions for "serverless". When a database claims to be "serverless", be sure to discern whether they mean "classic serverless" or "neo-serverless". It's really not important to understand that distinction, because this author seems to be the only one making it. Everyone knows what "serverless" means at this point, and it's not an embedded DB.

This such an air-headed comment to make. You must realize that page describing how SQLite is “serverless” has been up probably longer than your entire adult life. It is important in its context, they are not trying to “redefine” (lol) the term.

It doesn’t matter. It is not the generally accepted definition of serverless. The meaning of a word is based on how it is generally used, and almost nobody means this when they use the word “serverless”.

At this point trying to use the word in this way just creates a bunch of unnecessary confusion. Call it something else so we can move on to more important (and clear) discussions.

(Also there are a lot of assumptions and snarky comments in these responses about my age, which is quite rude and pretty elucidating.)

Re: SQLite Is Serverless

#330

Post by bane »

gwd wrote: post #240
bane wrote: post #200

Earlier quoted context omitted.

No, SQLite provides no RDBMS functionality. It is not an RDBMS. It is a file format. Saying that a program that opens and reads/writes a file through a file format API is a light RDBMS turns almost every program in history into an RDBMS. If SQLite didn't force you to use SQL as the read/write logic, absolutely nobody would confuse it for an RDBMS. That's because it's a file format.

> No, SQLite provides no RDBMS functionality. I think your comment would be more comprehensible if you gave some examples of the kind of functionality you think is missing. I don't know what you mean by "RDBMS", but JSON and XML don't do joins, don't do views, don't do efficient query plans, and so on. It's either ignorance or obstinacy to say SQLite is just a file format.

> ...but JSON and XML don't do joins, don't do views, don't do efficient query plans, and so on. It's either ignorance or obstinacy to say SQLite is just a file format.

Sure they do. If you write the logic to do so, and put it behind a nice API, you can make all of this come true. In fact, millions of people every day do joins with JSON and XML in their code every day. You can probably just use Apache Drill as the "library" in this sense to facilitate joins and whatnot. The creators of the SQLite library simply build that stuff into their library for you.

SQLite is a file format. It has a nice library full of wonderful utility functions for reading/writing that file format and a simple to use API that is operated by sending SQL to it.

It is exactly the same as reading and writing any other file format with any other API and library. The more you understand SQLite as a file format with a nice reference API implementation, the more it makes sense.

It is not the same as using an RDBMS and offers almost none of the things and RDBMS might offer. The more you try to figure out how it's not like PostgreSQL or Oracle or MongoDB, the more confused you'll make yourself.

It's no more an RDBMS than a .docx file is.

Post reply on HN