Live data from Hacker News

SQLite's File Format

sqlite.org

81–86 of 86 posts

Re: SQLite's File Format

#81
post #18

Any recommendations from HN for a write-once (literally once), data storage format that's suitable for network storage? sqlite docs recommend avoiding using it on network storage, though from what I can gather, it's less of an issue if you're truly only doing reads (meaning I could create it locally and then copy it to network storage). Apache Parquet seems promising, and it seems to support indexing now which is an…

SQLite over NFS works if you have one writer and many readers.

Re: SQLite's File Format

#82
post #32

My only question is if you really need a prefix before every value to say what type it is.

Any field in SQLite can contain any type, even if the schema says that a field should be INTEGER, it could have a TEXT, so it's necessary to specify what's the type of every single value

Indeed, unless it's a strict table, you can put gibberish in the type field (or forego giving a column a type altogether).

Re: SQLite's File Format

#83
post #46

Earlier quoted context omitted.

I've had multiple flaky issues with SQLite (e.g. non-HA Grafana) on Azure Files using NFS v4.1 leading to locked DBs. Perhaps some implementations work, I'm not gonna rely on it or advise others to do so.

Yeah trying to write from several hosts will certainly fail if you don't have advisory locks working, which is not a given, so you are right of course

These were singular containers, let alone hosts.

Re: SQLite's File Format

#84
post #73

Earlier quoted context omitted.

>> It was designed to be a DB for Tcl at a time when that language didn't have typed objects. Its SQL implementation reflects that. > It actually does have typed values Now. As the article points out, they were not part of the initial design, because of the Tcl heritage.

AFAIK it has always had typed values. Don’t confuse column types (which constrain a column to containing only values of a specified type) with value types (which enable it to treat the string “12” and the integer 12 and the floating point 12.0 as three distinct values) Tcl has value types. Tcl 7.x and earlier only had one data type, the string-so adding two integers required two string-to-int conversions followed by…

Tcl still only has one data type, which is the string. It has different internal representations to allow representing integers etc more efficiently, but since that is not part of the formal semantics of the language, those representations are not data types.

Re: SQLite's File Format

#85
post #72

The fact this fits in a few pages and is so approachable is a testament to its simplicity. I think I'd find it a lot harder to grok the file format of, for example, a Word doc/docx file.

I wouldn't put .doc and .docx next to one another, as they're only tangentially related. I'd bet getting the hello, world of .docx would be some silliness, but would not be hard to grok. I couldn't readily find a browsable copy of ECMA 376 4th Ed online but https://github.com/PumasAI/WriteDocx.jl/blob/v1.2.0/docs/src... was in the ballpark of what I expected to find in some section of the actual spec

They are much more closely related than you might think - .docx is basically an XML serialization of the same object graph as .doc, but the latter is a COM Structured Storage file with binary objects inside. Anyway, most of the cognitive load would be in the semantics of various objects, not their representation.

Re: SQLite's File Format

#86
post #84

Earlier quoted context omitted.

AFAIK it has always had typed values. Don’t confuse column types (which constrain a column to containing only values of a specified type) with value types (which enable it to treat the string “12” and the integer 12 and the floating point 12.0 as three distinct values) Tcl has value types. Tcl 7.x and earlier only had one data type, the string-so adding two integers required two string-to-int conversions followed by…

Tcl still only has one data type, which is the string. It has different internal representations to allow representing integers etc more efficiently, but since that is not part of the formal semantics of the language, those representations are not data types.

At the level of its C API-which is the level at which it integrates with sqlite-it indeed has multiple data types, it even has a structure to represent them, Tcl_ObjType: https://www.tcl-lang.org/man/tcl8.4/TclLib/ObjectType.htm
Post reply on HN