Live data from Hacker News

First Contact with SQLite

brandur.org

51–60 of 99 posts

Re: First Contact with SQLite

#51
post #12

Earlier quoted context omitted.

The wacky approach to column types came from SQLite's origins of being closely integrated with Tcl. Knowing that doesn't somehow mean it was a good default worth carrying on for decades.

Firstly, while it's the default, it's not mandatory. As the author discovered you can use STRICT to make the columns typed, and types to be enforced. The reason it remains unaltered by default is because one of the goals (and accomplishments) of SQLite is that the on-disk-data-file is completely backwards compatible, and cross-platform. This is a very important feature in some situations, and not lightly tossed aside…

Put another way, "default to what's best for all users collectively" is not strictly equivalent to "default to what's most intuitive to unfamiliar users".

The latter is the luxury of end-user software unburdened by decades of legacy compatibility obligations.

Re: First Contact with SQLite

#52

It’s interesting to read a lot of push back to the points here. I recently built a product with the backend using SQLite as the data store and ran into all these issues and many more. It is frustrating. I use SQLAlchemy and Alembic. It seemed everywhere I turned, the docs said “it works this way in all databases, except SQLite where X isn’t supported or you have to do Y differently.” I think with litestream and D1 an…

>> I would stick to the most boring stack Just use sql. Very straightforward and easy to use program understand maintain.

I think this is the best advice. The further you are away from your data store, the more you are relying on the used cases others have considered. Just because a library needs features that something like SQLite doesn’t provide doesn’t mean that you need that feature. But every extra abstraction in your stack has this danger.

I also think that the closer you are to your DB, more you appreciate the features you actually need are. A developer should know how their data is ultimately stored. And if an ORM hides this from you, that’s a problem.

Note: and ORM doesn’t need to hide this and can be a rational way to manage DB storage. But it’s then up to the developer to manage that abstraction. I’ve done this before where I used an ORM but I still knew exactly what SQL was going to be generated. But for many, especially new devs, the ORM is a black box.

Re: First Contact with SQLite

#53

It’s interesting to read a lot of push back to the points here. I recently built a product with the backend using SQLite as the data store and ran into all these issues and many more. It is frustrating. I use SQLAlchemy and Alembic. It seemed everywhere I turned, the docs said “it works this way in all databases, except SQLite where X isn’t supported or you have to do Y differently.” I think with litestream and D1 an…

I'm frustrated by someone reaching the conclusion that there can ever be a "best database". I spent several hours today writing a bash script that jammed CSV files into sqlite3 and ran some queries on them. Using postgres for this would have been insane, and I can imagine writing the exact opposite of this article if I had tried to do so: "It stores everything all over the place! There's no easily configurable in-memory storage option! All the docs and packages expect you to be running it as a service! Running my script takes an entire ansible playbook just to set things up! Why the hell is there a user table already? This thing uses hundreds of megs of memory at rest sometimes!"

Sqlite is not just for embedded data stores. However, it makes trade-offs to achieve ease of use and performance on certain workloads. If it tried to be postgres then it would be an inferior competitor; instead, it is an alternative that suits some use cases better and others worse. If you're trying to build a web app that can be deployed and backed up as two files, an executable and a db, then you'll probably want sqlite. If you're trying not to shoot yourself in the foot with "oh god why can't I do a full outer join and why are all the dates weird" then use postgres.

I imagine programmers would also find cockroachdb frustrating compared to postgres, if they didn't benefit from anything it offered and used it anyway.

Re: First Contact with SQLite

#54
My decision on SQLite vs Postgres is usually something like fixed and variable costs. For each little thing, Postgres is typically nicer to use. But the fixed cost of standing up and maintaining a dedicated database is high enough that I rarely need to go that route. (Hosted services just make that fixed cost into explicit $) And now with litestream, the server-side story has gotten nicer for SQLite, so I expect the percentage of time I need pg to continue to drop.

Re: First Contact with SQLite

#55

I think it is time for a real version/generation 4 of SQLite which drops some backwards compatibility (e.g. file format) and has 1) strict always enforced. 2) full datatypes (ints, floats, datetime, jsonb) 3) all "ALTER TABLE" functionality, even if it has to rewrite the table

Not to be contrary but... why? Why make a new vesion that breaks compatibility with the old version? Why make a new version just so it behaves like all the other database engines out there? Isn't having difference the point of having choices?

Agreed. Most of that can be achieved with SQL (how hard is it to add STRICT?) or the code you use to interact with SQLite. I don't see how it's worth sacrificing backwards compatibility to achieve those things, which are mostly non-issues in practice.

Re: First Contact with SQLite

#56
post #29
post #17

Earlier quoted context omitted.

The author is both ignorant of SQLite history, and philosophy as also ignorant of the real use-cases SQLite solves for embedding a lightweight SQL-ish database into applications. Looking into his "about" section he worked mostly with web APIs (Stripe, Heroku) and is a self-proclaimed fan of Postgres. Maybe when he acquires some experience working with embedded applications without boatloads of resources available nor…

None of the things mentioned in the article appear to have any significant impact on the resource usage and or embeddability if they were implemented, so it's totally fair to draw comparisons here. If want to have more of an apples-to-apples comparison you could swap out Postgres for DuckDB (which aims to follow Postgres in SQL dialect), and all of the mentioned points should still hold.

> None of the things mentioned in the article appear to have any significant impact on the resource usage and or embeddability if they were implemented, so it's totally fair to draw comparisons here.

Which is why I mention "history and philosophy" on the same line of my comment.

The `ALTER TABLE` documentation on SQLite [0] has a very clear reasoning on why it's implemented the way it is, and gives steps on how to reproduce more common usage of `ALTER TABLE` in other SQL engines.

To directly quote from their docs:

> Why ALTER TABLE is such a problem for SQLite

> Most SQL database engines store the schema already parsed into various system tables. On those database engines, ALTER TABLE merely has to make modifications to the corresponding system tables.

> SQLite is different in that it stores the schema in the sqlite_schema table as the original text of the CREATE statements that define the schema. Hence ALTER TABLE needs to revise the text of the CREATE statement. Doing so can be tricky for certain "creative" schema designs.

> The SQLite approach of storing the schema as text has advantages for an embedded relational database. For one, it means that the schema takes up less space in the database file. This is important since a common SQLite usage pattern is to have many small, separate database files instead of putting everything in one big global database file, which is the usual approach for client/server database engines. Since the schema is duplicated in each separate database file, it is important to keep the schema representation compact.

> Storing the schema as text rather than as parsed tables also give flexibility to the implementation. Since the internal parse of the schema is regenerated each time the database is opened, the internal representation of the schema can change from one release to the next. This is important, as sometimes new features require enhancements to the internal schema representation. Changing the internal schema representation would be much more difficult if the schema representation was exposed in the database file. So, in other words, storing the schema as text helps maintain backwards compatibility, and helps ensure that older database files can be read and written by newer versions of SQLite.

> Storing the schema as text also makes the SQLite database file format easier to define, document, and understand. This helps make SQLite database files a recommended storage format for long-term archiving of data.

> The downside of storing schema a text is that it can make the schema tricky to modify. And for that reason, the ALTER TABLE support in SQLite has traditionally lagged behind other SQL database engines that store their schemas as parsed system tables that are easier to modify.

[0] https://www.sqlite.org/lang_altertable.html

Re: First Contact with SQLite

#57
I needed to use SQLite from php to develop a WordPress plugin. (Yeah, yeah, I have heard most if not all the jokes about that particular stack.)

Here are some notes about my experience, offered to anybody who might be able to use them. https://www.plumislandmedia.net/reference/sqlite3-in-php-som...

Re: First Contact with SQLite

#58
post #8
post #6

I feel like the recent hype around SQLite made people use it for a lot of stuff that is not really suitable for SQLite. It has too many caveats especially around data types. I'm not saying it does not deserve the attention, it is a fantastic piece of software but if I had the option to use PostgreSQL for something I'd never ever get close to choosing SQLite over it. It shines when you don't need or want something mor…

> if I had the option to use PostgreSQL for something I'd never ever get close to choosing SQLite over it I had the option for a recent project. It's a niche forum-like application with around 2,000 users. Went with a monolithic design and vertical scaling (if needed), so SQLite was perfect. Every dynamic HTML page renders in under 1ms. Litestream for live-replication to a couple S3 buckets. Running PostgreSQL for so…

Is that 10ms of network latency? I remember Postgres being pretty low latency if you run it on the same host, although the point about maintenance makes complete sense.

Re: First Contact with SQLite

#59
post #6

I feel like the recent hype around SQLite made people use it for a lot of stuff that is not really suitable for SQLite. It has too many caveats especially around data types. I'm not saying it does not deserve the attention, it is a fantastic piece of software but if I had the option to use PostgreSQL for something I'd never ever get close to choosing SQLite over it. It shines when you don't need or want something mor…

> if I had the option to use PostgreSQL for something I'd never ever get close to choosing SQLite over it.

I'm using both Postgres and SQLite for active projects. Postgres is great for a multi-user blog I run where the DB is hosted, backed up, etc. The same site would run fine and slightly faster with local SQLite (which it used to) but having Postgres lets me use Render.com's built in management features, which are nice.

SQLite works great for anything app-like. I have a script that OCRs certain video game screenshots and saves the OCR data in a searchable database, for example. Postgres would be complete overkill for this and would add nothing but hassle. I don't want to bother with keeping a separate database going for that in my Postgres server. I just want a folder of files, and SQLite works perfectly for that. I could use Postgres but it would offer zero useful benefits and might be a bit slower (even with a local server) due to my sloppy code.

They are just different tools for different tasks, with some overlap.

Re: First Contact with SQLite

#60

How do you typically deal with "missing features" from SQLite (e.g. stored procedures)? Do you use extensions like https://github.com/nalgeon/sqlean in production?

Don't Application-Defined SQL functions fulfill the role of stored procedures?

https://www.sqlite.org/appfunc.html

Post reply on HN