Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Is Serverless

sqlite.org

Post reply on HN
Print view View original

391–400 of 453 posts

Re: SQLite Is Serverless

#391

Post by bane »

imtringued wrote: post #376
bane wrote: post #351

Earlier quoted context omitted.

No, an RDBMS is a piece of software for managing databases in the relational model, access to the databases (such as users and permissions) and provides services such as a server, connection pooling, and so on. SQLite provides almost no RDBMS features. This isn't just semantics. A car is not an engine. A fork is not a kitchen. A SQLite file is not a DBMS.

https://en.wikipedia.org/wiki/Database#Database_management_s... >Connolly and Begg define database management system (DBMS) as a "software system that enables users to define, create, maintain and control access to the database".[24] >The functionality provided by a DBMS can vary enormously. The core functionality is the storage, retrieval and update of data. Codd proposed the following functions and services a fully…

The zip utility.

[x] Data storage, retrieval and update

[x] User accessible catalog or data dictionary describing the metadata

[ ] Support for transactions and concurrency

[x] Facilities for recovering the database should it become damaged

[x] Support for authorization of access and update of data

[ ] Access support from remote locations

[x] Enforcing constraints to ensure data in the database abides by certain rules

Congratulations, apparently zip files are as much of an RDBMS as SQLite. If I bundle zip with ssh (Access support from remote locations) and Linux (Support for transactions and concurrency) did I just create a new RDBMS?

How many checkboxes do I have to have in order to call anything an RDBMS? Is SSH an RDBMS (Access support from remote locations). Can I just put a catalog in a .txt file and check that box? Is XML an RDBMS because it enforces constraints and supports storage retrieval and update? Are chmod and chown an RDBMS because they support authorization of access and update of data?

> SQLite manages are the database, not just merely a file format.

It turns out databases can be just files. Those files must follow a described file format. SQLite files are relational databases that are instantiations of the file format specification for SQLite files. The SQLite library implements that file format as well as some clever logic to support SQL and ACID compliance. Some SQLite libraries do not support these things.

Re: SQLite Is Serverless

#392

Post by bane »

imtringued wrote: post #378
bane wrote: post #337

Earlier quoted context omitted.

Those are relational database features, not RDBMS features. The SQLite file format specifies a way to organize, store and retrieve fairly arbitrary data using a relational database model. The library knows how to handle SQL to describe the work being done. The SQL is optional, one can, with the specification, read/write SQLite files in many other ways. There are almost no RDBMS features in SQLite. There are many many…

You're making some basic assumptions that do not make sense. The database can be physically stored in any arbitrary format. One can build an RDBMS that stores all its data in tar or JSON files. No matter how inefficient as long as software exists that manages the database. >Are .tar files RDBMSs? This question doesn't make sense because you are asking if databases can be management systems which is obviously false by…

> This question doesn't make sense because you are asking if databases can be management systems which is obviously false by definition.

Yes! And by extension the tar utility is not a DBMS even if it checks some of the boxes for one. And thus SQLite files are not RDBMSs. Looks like you and I agree.

> The database can be physically stored in any arbitrary format. One can build an RDBMS that stores all its data in tar or JSON files. No matter how inefficient as long as software exists that manages the database.

Sure! One can come up with all kinds of very terrible software. But what's the distinction between some random software that just allows CRUD operations on a file format and an RDBMS by your definition? Because you've defined something close to 100% of all software as an RDBMS which makes the distinction between software and RDBMSs meaningless.

There has to be something more than just that to be an RDBMS doesn't there?

Re: SQLite Is Serverless

#393

Post by 0xbadcafebee »

sly010 wrote: post #300
nicholaides wrote: post #185

I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…

I have been using SQLite as a format to move data between steps in a complicated batch processing pipeline. With the right pragmas it is both faster and more compact than JSON. It is also much more "human readable" than gigabytes of JSON. I only wish there was a way to open an http-fetched SQLite database from memory so I don't have to write it to disk first.

  $ mount -t tmpfs none /some/path
  $ write db.sqlite /some/path/db.sqlite
  $ read db.sqlite
We've been abusing tmpfs for more than 10 years to get around the IO layer's failings. It's probably still a valid pattern.

Re: SQLite Is Serverless

#394

Post by pupdogg »

siscia wrote: post #223
pm90 wrote: post #166

What I really want: SQLite storage backend driver for s3/gcs. No need for disks then. I haven’t been able to find such a solution though; and am not technically proficient enough in C (the Lang SQLite is written in) to do so myself.

I am working on RediSQL[1] and I am about to launch a managed version. The interface will be either HTTP or Redis protocol, you create your database, and daily I will back it up on S3. (If interested you can subscribe for updated here: https://simplesql.carrd.co/ [1]: https://redisql.com/

this looks very interesting...can you please post some benchmarks in your docs for reference?

Re: SQLite Is Serverless

#395

Post by kevas »

rakoo wrote: post #362
palotasb wrote: post #308

Earlier quoted context omitted.

You can use the memvfs module to load in-memory databases if you're using the C API. I'm not sure how many higher-level APIs support it though. [1] https://stackoverflow.com/a/53453338/3063 [2] https://www.sqlite.org/loadext.html#example_extensions [3] https://www.sqlite.org/src/file/ext/misc/memvfs.c

A very interesting approach is sqltorrent ( https://github.com/bittorrent/sqltorrent ): the sqlite file is shared in a torrent, and all queries will touch a specific part of the file, which is downloaded on-demand. Also check https://github.com/lmatteis/torrent-net

Incredibly odd, but so awesome

Re: SQLite Is Serverless

#396

Post by EGreg »

m-p-3 wrote: post #74
EGreg wrote: post #31

Earlier quoted context omitted.

Waaaaait wouldn’t that mean the file system is the server, with some binary API and responsible for handling concurrent access and locks for the entire file? LOL.

Serverless in this situation means that you don't really have to provision or setup an actual server to handle the database, the client itself just need the ability to read and write an SQLite file.

Because many of the locking things are outsourced to the file system which acts as server for concurrent threads.

A server is something that listens for commands from various clients and executes them.

Re: SQLite Is Serverless

#397

Post by tonyarkles »

cle wrote: post #338
mikestew wrote: post #317

Earlier quoted context omitted.

Everyone know that “serverless” anything doesn’t run on a server , be it AWS or Azure instances or what have you. That would be both ironic and silly, like someone used the wrong word or something. Serverless databases and what have you have been around longer than the current batch of folks trying to redefine things (or more charitably, ran out names to call things). Like or not, there is a distinction even if the o…

Nobody called them serverless DBs, and nobody does. There is a vocal minority of reductionists here on HN who dismiss the accepted definition of serverless because the literal meaning doesn’t make sense. It’s just noise though. “Serverless” does have a specific meaning and it’s not “there are no servers anywhere”. I was around when there was no serverless. Things change, new words arise. Time for us to get with the t…

> I was around when there was no serverless.

We used to call it /cgi-bin/ ;)

Re: SQLite Is Serverless

#398

Post by cnst »

ForHackernews wrote: post #349
cle wrote: post #336

Earlier quoted context omitted.

Serverless refers to platforms with specific and well-understood properties, such as managed servers, autoscaling, usage-based pricing, etc. What parts of SQLite have these properties?

SQLite has specific, well-understood pricing and scaling properties.

This so much!

But is it web-scale?! /s

Re: SQLite Is Serverless

#399

Post by [deleted] »

SQLite wrote: post #382
sly010 wrote: post #300

Earlier quoted context omitted.

I have been using SQLite as a format to move data between steps in a complicated batch processing pipeline. With the right pragmas it is both faster and more compact than JSON. It is also much more "human readable" than gigabytes of JSON. I only wish there was a way to open an http-fetched SQLite database from memory so I don't have to write it to disk first.

> I only wish there was a way to open an http-fetched SQLite database from memory so I don't have to write it to disk first. The sqlite3_deserialize() interface was created for this very purpose. https://www.sqlite.org/c3ref/deserialize.html

[deleted]

Re: SQLite Is Serverless

#400

Post by int_19h »

cle wrote: post #338
mikestew wrote: post #317

Earlier quoted context omitted.

Everyone know that “serverless” anything doesn’t run on a server , be it AWS or Azure instances or what have you. That would be both ironic and silly, like someone used the wrong word or something. Serverless databases and what have you have been around longer than the current batch of folks trying to redefine things (or more charitably, ran out names to call things). Like or not, there is a distinction even if the o…

Nobody called them serverless DBs, and nobody does. There is a vocal minority of reductionists here on HN who dismiss the accepted definition of serverless because the literal meaning doesn’t make sense. It’s just noise though. “Serverless” does have a specific meaning and it’s not “there are no servers anywhere”. I was around when there was no serverless. Things change, new words arise. Time for us to get with the t…

I remember Firebird called "serverless" sometime around 2006, in the same meaning that SQLite uses. It was pretty common terminology for RDBMS.
Post reply on HN