Live data from Hacker News

Prefer strict tables in SQLite

evanhahn.com

171–180 of 188 posts

Re: Prefer strict tables in SQLite

#171
post #95

> Unfortunately, I don’t think there’s a way to ALTER a table to make it strict. I think you have to copy the data out of the non-strict table into the strict one. This inspired me to add a feature to my sqlite-utils Python library and CLI tool, so you can now use it to transform non-strict tables to strict (and vice-versa) like this: uvx sqlite-utils transform data.db mytable --strict Or in Python: import sqlite_uti…

> This inspired me to add a feature to my sqlite-utils Python library "me" == "ChatGPT", apparently: > Can the .transform() internal Python method turn a non-strict table into a strict table? > No. Table.transform() preserves the table’s existing strictness; it cannot change it. Its signature has no strict= parameter > add an optional strict= boolean parameter to the transform() method - if it is None (the default) t…

Right, the transcript is linked from the issue: https://github.com/simonw/sqlite-utils/issues/787#issuecomme...

As far as I can tell I've shared more prompt transcripts than anyone else. Happy to be proven wrong about that.

Look through my commit history on https://github.com/simonw/tools/commits and https://github.com/simonw/sqlite-utils/commits and https://github.com/simonw/datasette/commits and you'll see that commits that were AI-assisted almost all link to a transcript or include a prompt or both.

Re: Prefer strict tables in SQLite

#172
post #95

> Unfortunately, I don’t think there’s a way to ALTER a table to make it strict. I think you have to copy the data out of the non-strict table into the strict one. This inspired me to add a feature to my sqlite-utils Python library and CLI tool, so you can now use it to transform non-strict tables to strict (and vice-versa) like this: uvx sqlite-utils transform data.db mytable --strict Or in Python: import sqlite_uti…

> This inspired me to add a feature to my sqlite-utils Python library "me" == "ChatGPT", apparently: > Can the .transform() internal Python method turn a non-strict table into a strict table? > No. Table.transform() preserves the table’s existing strictness; it cannot change it. Its signature has no strict= parameter > add an optional strict= boolean parameter to the transform() method - if it is None (the default) t…

What are you trying to achieve with this comment? Does everything need to turn into a sniping contest about whether or not something is ai-produced? This is the world we live in now.

Re: Prefer strict tables in SQLite

#174

Earlier quoted context omitted.

What the hell? This is the first I'm learning of this. Why did they do that? Is it owned by a private company?

That's one way they make money. That's the reason sqlite exists.

In a 2021 podcast interview[0], Dr. Hipp noted they've sold zero copies of the TH3 (extensive, proprietary) test suite in SQLite's history.

By the time TH3 was added in 2008, SQLite had gained a fair bit of traction across multiple industries. Though I totally agree that the comprehensive coverage is a leading reason why they've been so stable over the last ~20 years.

So indirectly, TH3 is why they (continue to) exist and (are able to) make money, but it isn't a direct line as one might assume.

[0] https://corecursive.com/066-sqlite-with-richard-hipp/

[1] https://youtu.be/5zQdYx-fqJg?t=300

Re: Prefer strict tables in SQLite

#175
post #141
post #54

Earlier quoted context omitted.

Right, and that's the problem. There should be DATE and BOOL as well, especially in strict mode.

I don't understand what this has to do with strict mode? Yes those types should exist, but the workaround is to use an integer column for bool and a text column for date, whether or not you're using strict mode.

The point of strict mode is that the RDBMS performs data type validation when inserting/updating data. If you use a text column for storing dates, you can store any string in the column, not just valid dates. That's not strict.

Re: Prefer strict tables in SQLite

#176

Earlier quoted context omitted.

But one DB one app is fairly common with Postgres too, particularly if you're adhering to a services deliniation. Guess if your code enforces types at DB insert time, the DB doesn't need to, but one of those is more likely to change than the other.

> But one DB one app is fairly common with Postgres too That is common today, but remember that Postgres is now 40 years old. It is so old that it was originally based on QUEL rather than SQL. Back then database servers were designed to be what we now think of as the "API server". That necessitates runtime input validation same as your "API server" needs input validation today. If you were designing Postgres from scr…

Even with a statically typed language, two separate codepaths or code versions could write to the same tables and disagree on the types.

Re: Prefer strict tables in SQLite

#177
post #149

Earlier quoted context omitted.

The largest table where I've done this had over 50 of these booleans. That's one 64-bit number, or 50 different columns.

But if you do the function based index, you actually increase the stored size, right? Seems like a microoptimization to me that I would only do if I really have to.

For the searched column only?

If you are greatly concerned, SQL Server and the Sybase database from which it emerged have a native boolean data type.

Another benefit of this scheme is that adding another boolean means using the next power of 2 in the existing integer, assuming room remains. No new column necessary.

Re: Prefer strict tables in SQLite

#178
post #174

Earlier quoted context omitted.

That's one way they make money. That's the reason sqlite exists.

In a 2021 podcast interview[0], Dr. Hipp noted they've sold zero copies of the TH3 (extensive, proprietary) test suite in SQLite's history. By the time TH3 was added in 2008, SQLite had gained a fair bit of traction across multiple industries. Though I totally agree that the comprehensive coverage is a leading reason why they've been so stable over the last ~20 years. So indirectly, TH3 is why they (continue to) exis…

They sell several niche things, for hopefully hundreds of thousands of dollars, to just a few people each. TH3 is just one instance of the general pattern.

Re: Prefer strict tables in SQLite

#179

Earlier quoted context omitted.

> But one DB one app is fairly common with Postgres too That is common today, but remember that Postgres is now 40 years old. It is so old that it was originally based on QUEL rather than SQL. Back then database servers were designed to be what we now think of as the "API server". That necessitates runtime input validation same as your "API server" needs input validation today. If you were designing Postgres from scr…

Even with a statically typed language, two separate codepaths or code versions could write to the same tables and disagree on the types.

[deleted]

Re: Prefer strict tables in SQLite

#180

Earlier quoted context omitted.

> But one DB one app is fairly common with Postgres too That is common today, but remember that Postgres is now 40 years old. It is so old that it was originally based on QUEL rather than SQL. Back then database servers were designed to be what we now think of as the "API server". That necessitates runtime input validation same as your "API server" needs input validation today. If you were designing Postgres from scr…

Even with a statically typed language, two separate codepaths or code versions could write to the same tables and disagree on the types.

How? Consider the following pseudocode:

Schema:

    CREATE TABLE foo (
        bar INTEGER
    );
Program:

    fn main() {
        stmt := "INSERT INTO foo (bar) VALUES (?)"
        if (now() % 2 == 0) {
            db.exec(stmt, [1]); 
        } else {
            db.exec(stmt, ["Baz"]); // Static analysis fails here. Input is not an integer.
        }
    }
If a different code version comes along and, say, changes the schema then:

Schema:

    CREATE TABLE foo (
        bar TEXT
    );
Program:

    fn main() {
        stmt := "INSERT INTO foo (bar) VALUES (?)"
        if (now() % 2 == 0) {
            db.exec(stmt, [1]); // Static analysis fails here. Input is not text.
        } else {
            db.exec(stmt, ["Baz"]);
        }
    }
Perhaps what you are imagining is when the data is provided externally, where the target schema isn't known at compile time? That is a possible use-case, but not what SQLite was primarily designed for and not what we are talking about. If that is what you need that is what strict tables are there for. Different tools for different jobs.
Post reply on HN