Live data from Hacker News

SQLite Is Serverless

sqlite.org

381–390 of 453 posts

Re: SQLite Is Serverless

#381
post #142

Earlier quoted context omitted.

SQL queries tend to be much smaller than their equivalent data-structure traversal procedures. This can be beneficial even when you still want the data in your process's address space, hence embedded database engines like sqlite. Libraries like Linq can also provide the same expressive power over a programming language's native objects and collections. As to why you'd want a separate DB server process: long lived mut…

" long lived mutable structures tend to drift into unexpected states " Why are those "tend to drift" ? As an example I wrote sort of like game server for one of my applications. Internally it has those exact forever lived mutable structures. I've never observed it to drift into any unpredictable state. Works like a charm and running for many month. I only reboot it when I need to update it to a new version. The only…

Consider why 3rd normal form exists to begin with.

In the real world, let's say you have a shipping address and a billing address for a customer, and they are usually (but not always) the same.

Eventually, a customer moves, changing both of their addresses. But the user forgets to change their billing address with their delivery address.

A proper database would have a 'billing address is same as delivery address' logic, following the principle of DRY (don't repeat yourself).

----------

There are lots of examples here of what can go wrong when you repeat yourself in a database application. The user may have an error when repeating themselves over the dataset (delivery address is correct, but zip code on billing address has a typo).

Dealing with these issues at scale, with hundreds of thousands of customers, is certainly a problem. Normal forms can formalize these issues and help the business owner avoid the problems.

Where do you verify the existence of zip codes and cities? Where do you check for typos? How do you prevent contradictions on the submitted information?

Your human customers will make many mistakes. Your logic must hold up even in the presence of faulty data.

Re: SQLite Is Serverless

#382
post #300

I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…

I have been using SQLite as a format to move data between steps in a complicated batch processing pipeline. With the right pragmas it is both faster and more compact than JSON. It is also much more "human readable" than gigabytes of JSON. I only wish there was a way to open an http-fetched SQLite database from memory so I don't have to write it to disk first.

> I only wish there was a way to open an http-fetched SQLite database from memory so I don't have to write it to disk first.

The sqlite3_deserialize() interface was created for this very purpose. https://www.sqlite.org/c3ref/deserialize.html

Re: SQLite Is Serverless

#383

I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…

I’d love a SQLite to macOS Excel (or any macOS spreadsheet application) workflow so less technical users can do analysis. Has anybody pulled this off?

You can do that with powerquery

Re: SQLite Is Serverless

#384
post #149

I love SQLite, but people approach it from a classic RDBMS angle which confuses them. Here's the deal: SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file. That's it. Stop overthinking it. Can you write a microservice that stores its data in a big JSON file that you've built some code around to read/write to? Yes. It's just a file, but you have to build all the read/w…

I'm no SQLite expert but by this logic aren't all single-host databases that tick the "Durability" ACID checkbox "just file formats" in the sense that yeah, the bytes we care about exist somewhere in the filesystem? Moreover I'm having trouble coming up with things that I'd associate with a RDBMS and not "just a file format" that SQLite doesn't support. Transactions? SQLite has them. Relational constraints? SQLite ha…

Sorry for the previous terse reply.

Longer answer:

SQLite files do not guarantee ACID compliance. You can write code tomorrow that produces SQLite files, and so long as you follow the specification (https://www.sqlite.org/fileformat.html) it will be readable by any other code that implements the specification (e.g. https://sqljet.com/)

An RDBMS is not a database, nor is it SQL, nor is it data. It is a kind of DBMS software that manages relational databases, and access to the data (such as users and user rights). Most modern RDBMSs run as servers and offer network connectivity, connection pooling, advanced buffering options, various memory usage schemes. Many have their own memory allocation and file handling routines that are separate from the OS. Some offer clustering, partitioning and so on. SQLite does not offer any of these things. If you were to write a comprehensive list of things that Oracle, MS SQL Server, DB2, PostgreSQL, MySQL and SQLite offer, SQLite would offer almost none of the features that the rest do.

A relational database uses the relational model to store data. SQL is the most common language for describing what you want to put into or retrieve from the relational database, but it is not required.

There are many kinds of databases. Some of them store data in memory, in a file, in multiple files, and so on. Some of them follow various models, some of them are unique. If you have the file format for a database that stores its data in files, you can read/write to the file freely without any management system and without ACID compliance. SQLite files are examples of a kind of database file that stores data using a relational model. So are MDB files that Microsoft Access uses.

By conflating a file format with an RDBMS, it's like conflating a fork for a restaurant, or a chair for a house.

ACID compliance is not something guaranteed by the file format. SQLite files do not guarantee ACID compliance. If you write some code tomorrow that can read/write SQLite files based on the spec, you haven't created and ACID compliant SQLite file, nor is your code ACID compliant.

The SQLite library implements the properties that make SQLite ACID compliant. It does so by various clever means like a journal file format, and a write-ahead-log file format and various other well thought out approaches. If you were to write your own code that implemented the SQLite file spec, and you wished your code to also offer ACID compliance, you would have to implement those things yourself -- and you are under no obligation to use the SQLite journal and WAL file formats nor the internal logic that the SQLite library uses. You can do it entirely your own way!

Re: SQLite Is Serverless

#385
post #343

Earlier quoted context omitted.

I think you’re stuck in 90/10 rule territory here. But even so, SQLite was 220,000 lines the last time they measured, which was five years ago. You can do a lot of functionality in 22kloc, even ignoring the other 90%, which you shouldn’t Lodash, for instance, is much smaller than 22k lines, and it “just” manipulates objects and lists. If you downplay others like this, I wonder how you feel about your own work. Have y…

I'm not downplaying anybody's work. Many people approach SQLite as something in the RDBMS territory. It's not. Almost all of the confusion I've ever seen related to SQLite comes from starting from that basis. If one simply thinks of it as an alternative to fopen() then it makes very simple and intuitive sense. The people in this thread seem to be very resistant to this simple clarity of thought, but whatever, they ca…

The impression given here is that you only use it as a dumb store of data. My experience is that it's more like:

  file-format::DBMS

Re: SQLite Is Serverless

#386
post #300

I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…

I have been using SQLite as a format to move data between steps in a complicated batch processing pipeline. With the right pragmas it is both faster and more compact than JSON. It is also much more "human readable" than gigabytes of JSON. I only wish there was a way to open an http-fetched SQLite database from memory so I don't have to write it to disk first.

> I only wish there was a way to open an http-fetched SQLite database from memory so I don't have to write it to disk first.

Ramfs?

Re: SQLite Is Serverless

#387
post #354
post #330

Earlier quoted context omitted.

> ...but JSON and XML don't do joins, don't do views, don't do efficient query plans, and so on. It's either ignorance or obstinacy to say SQLite is just a file format. Sure they do. If you write the logic to do so, and put it behind a nice API, you can make all of this come true. In fact, millions of people every day do joins with JSON and XML in their code every day. You can probably just use Apache Drill as the "l…

First of all, you still haven't answered the question: What is it that an RDBMS has that SQLite doesn't have? > It's no more an RDBMS than a .docx file is. Thanks for the idea. Your argument is like saying this: Microsoft Word is not a word processor -- it's a file format. I mean, yes, Word has a file format; but it's far more than just a format specification. > Sure they do. If you write the logic to do so, Right, b…

> First of all, you still haven't answered the question: What is it that an RDBMS has that SQLite doesn't have?

And RDBMS is a well defined thing and is literally what the acronym expands to mean. This is very old technology with an interesting history and I really implore you and anybody reading this to go read up on it. It's not just whatever we assume it to be or some kind of data bucket with SQL.

> Microsoft Word is not a word processor -- it's a file format.

No, don't be obtuse. I'm saying that .docx is a file format.

Word is both an application for editing documents and contains a reference implementation for reading/writing .docx formatted files. There are many libraries that can read .docx files and some of them are also part of document editing software.

> Right, but you don't have to write the logic if you're using SQLite. That's the point. SQLite is a library, which provides a way to do SQL operations on data. Like Word, SQLite has a file format, but it is far more.

I just don't get where you're coming from. Do you not know that the SQLite libray can actually do complex SQL queries on data? Or do you think that people shouldn't do that for some reason? Or do you just value SQL queries so little that you don't see any difference between being able to do complex queries and doing `file.Write(json.Marshal(data))`? What is it you're trying to accomplish with this line of argument?

Precision of thought. People don't go around calling fish oceans, or forks restaurants. The SQLite library does what you've described to SQLite files. But you don't need the SQLite library to work on SQLite files. You don't need even need SQL. e.g. https://sqljet.com/

Just because a library offers SQL as a convenient tool to read/write data into its file format everybody loses their minds and starts to think the library is some kind feature reduced Oracle cluster. Go back to my first post. People are approach what SQLite is from the wrong direction (RDBMS) and its confusing the fuck out of everybody who gets near it.

This is important. IT departments and governments make very large, very expensive decisions based on if people know that SQLite is closer to CSV files than to Oracle databases.

I literally sat in a meeting last week where a senior decision-maker at a client wouldn't accept delivery of some software because it used SQLite and didn't want to add maintenance of yet another database to their overworked DBA staff and didn't want to hire a dedicated person to manage it. So now, instead of just taking delivery of the software, some of it has to be rewritten to use the client's RDBMS system, which in turn actually will add workload to the overworked DBA staff and will also perform worse.

SQLite IS A FILE FORMAT with a really nice library for reading/writing to that format.

Re: SQLite Is Serverless

#388
post #385
post #343

Earlier quoted context omitted.

I'm not downplaying anybody's work. Many people approach SQLite as something in the RDBMS territory. It's not. Almost all of the confusion I've ever seen related to SQLite comes from starting from that basis. If one simply thinks of it as an alternative to fopen() then it makes very simple and intuitive sense. The people in this thread seem to be very resistant to this simple clarity of thought, but whatever, they ca…

The impression given here is that you only use it as a dumb store of data. My experience is that it's more like: file-format: :DBMS

Why, because it offers SQL support? That just makes it a relational database that supports SQL. MS Access supports SQL.

If you were to draw up a feature list of Oracle, PostgreSQL, MS SQL and SQLite, SQLite would have almost none of the features of any of the actual RDBMSs.

Here's some examples from MS SQL:

- Support to PMEM devices and bypassing OS storage mechanisms for optimal file read/write access

- Availability Groups and synchronous replica pairs

- Users and permissions

- Secure Enclaves

- Certificate management functionality

- BI tools

- Database tuning advisor

- Machine Learning services

- Service Broker

- Replication services

- Analysis Services

- Reporting services

- Notification services

- Integration services

and so on.

Draw up a set of features for JSON files and jq and compare to SQLite. Is it closer to MS SQL or JSON?

Re: SQLite Is Serverless

#389
post #343

Earlier quoted context omitted.

I'm not downplaying anybody's work. Many people approach SQLite as something in the RDBMS territory. It's not. Almost all of the confusion I've ever seen related to SQLite comes from starting from that basis. If one simply thinks of it as an alternative to fopen() then it makes very simple and intuitive sense. The people in this thread seem to be very resistant to this simple clarity of thought, but whatever, they ca…

There is no such spectrum. SQLite is a piece of software that implements some but not all commonly expected RDBMS features. Software is not a file format but software may be written with the expectation that a given file follows the requirements of a certain file format and the software may be written in a way that it produces files that follow the file format specification. Since SQLite - the software - is an RDBMS…

I just drew the spectrum. It exists now.

SQLite is a file format. You can read/write SQLite databases without the SQLite library. You can write your own custom reader/writer/creator. You don't have to use SQL. You don't have to be ACID compliant. I can make for you right now a SQLite database that did not touch any SQLite software, put data into it and you can open it with another piece of software that implements the SQLite file format specification.

Likewise, you can use the SQLite library software to create a SQLite database file, put data into it, and I can read it/update it using any other software that implements the SQLite file format specification.

The SQLite library offers some very very basic features, such as ACID compliance, and so on, but those are not part of or guarantees of the file format or the database files. The software that you write that implements the SQLite file format specification does not have to do any of these things to work with or produce a valid SQLite file.

An RDBMS is a kind of DBMS for managing relational databases and providing access to the databases (for example users and permissions). Modern RDBMSs offer extensive features (look at an Oracle or MSSQL Server spec sheet) that are not even hinted at with the SQLite library software.

This is because SQLite is not an RDBMS, it's a file format.

Re: SQLite Is Serverless

#390
post #196

> It is important to understand these two different definitions for "serverless". When a database claims to be "serverless", be sure to discern whether they mean "classic serverless" or "neo-serverless". It's really not important to understand that distinction, because this author seems to be the only one making it. Everyone knows what "serverless" means at this point, and it's not an embedded DB.

Meh. If we allow serverless to make REST calls, is accessing a file system any different? I had more trouble with the assertion that the embedded DB would be maintenance-free. I started a top level comment asking for someone to explain how that would work. The part of my brain that protects me from scams is screaming “something for nothing”.

> "I had more trouble with the assertion that the embedded DB would be maintenance-free. I started a top level comment asking for someone to explain how that would work."

When is the last time a certed-up DBA had to do any maintenance on your Firefox install's places.sqlite database? That's what's meant by maintenance free; you can reasonably employ sqlite databases on users' computers, without users having the foggiest idea of what a database even is.

Post reply on HN