Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite File Format Viewer

sqlite-internal.pages.dev

Post reply on HN
Print view View original

31–40 of 45 posts

Re: SQLite File Format Viewer

#31

Post by jchw »

burntcaramel wrote: post #30
jchw wrote: post #13

This really would've come in handy when I was debugging my own SQLite parser a couple weeks ago. One thing that initially confused me was how exactly the pages worked w.r.t. the first page on disk... I misunderstood the SQLite documentation in different ways, but it's really rather simple: the very first page is just treated as containing the file header in it, and it pushes down the rest of the data, making the page…

Yes I’m also working on a SQLite parser, mine is in raw WebAssembly. Is yours open source too? This tool will be so useful. I have basic page reading and parsing of the CREATE TABLE schema: https://github.com/RoyalIcing/SilverOrb/blob/9dacad0ce521b0d... My plan is to create a miniature .wasm module to read .sqlite files that works in the browser. It will be in the tens of kilobytes rather than the 1 megabyte that the…

Mine is in TypeScript and for the purpose of parsing a file that happens to use sqlite, so I don't think I'll bother parsing the CREATE TABLE schema unless I have to. It's not currently posted anywhere but will be open source. It works, but the code isn't particularly great :)

Re: SQLite File Format Viewer

#32

Post by hamstergene »

This is super cool and will really help anyone studying how to build their own storage engines.

A note: the table that describes the data structures (Field # | Description | Value) should also list the data type (uint8 uint16 int32 text etc.). If it's an enum ("Table Leaf"), reference other possible enum values. If its length depends on another field, reference that field No.

Re: SQLite File Format Viewer

#33

Post by [deleted] »

hinkley wrote: post #24
Retr0id wrote: post #21

Earlier quoted context omitted.

sqlite itself supports a binary encoding of JSON: https://sqlite.org/jsonb.html

When I said binary JSON I didn’t mean literal JSON. I meant “common denominator interchange format”. It’s too chatty by far and has dismal performance for queries. So you’re better off asking a specific question and getting a larger document that could answer many questions that you do t yet have. For CDNs things like this matter a lot.

[deleted]

Re: SQLite File Format Viewer

#35

Post by [deleted] »

hinkley wrote: post #18
jchw wrote: post #13

This really would've come in handy when I was debugging my own SQLite parser a couple weeks ago. One thing that initially confused me was how exactly the pages worked w.r.t. the first page on disk... I misunderstood the SQLite documentation in different ways, but it's really rather simple: the very first page is just treated as containing the file header in it, and it pushes down the rest of the data, making the page…

I really want a data format that is effectively binary JSON. What is the subset of all of the features of SQLite that makes either a read-only or an updatable data set that is compact. But better searchability than a streaming parser.

[deleted]

Re: SQLite File Format Viewer

#36

Post by Timon3 »

senkora wrote: post #19
NoSalt wrote: post #12

Really nice, looks great! However, I am not a fan of uploading databases to "strange" sites on the internet, so I will probably never use this.

I wish that we had a browser version of “pledge” that would: 1. Permanently restrict the browser tab from accessing the network. 2. Show an indication of this in the browser UI outside of the content area. It would be perfect for this kind of app.

I've long been wishing for the same! With service workers etc. this would be a perfect approach for many small utilities. I've started building things as single HTML files using vite-plugin-singlefile[0], but users either have to manually block network requests, or they have to trust me. It would be awesome if they could be sure that nothing will be uploaded anywhere.

The pledge would have to be persistent, which would make e.g. service worker updates difficult - ideally the browser should show you that an update is available, and offer to delete all stored information so nothing can be saved to localStorage etc. to be uploaded afterwards.

[0]: https://www.npmjs.com/package/vite-plugin-singlefile

Re: SQLite File Format Viewer

#37

Post by roelschroeven »

I tried with a simple database, and it just shows me a blank screen. The console in developer tools shows an exception (both in Firefox and Chromium):

    index-CGfYebmt.js:85 Uncaught RangeError: Offset is outside the bounds of the DataView
        at DataView.prototype.getUint32 ()
        at R0 (index-CGfYebmt.js:85:4683)
        at X0 (index-CGfYebmt.js:85:7966)
        at index-CGfYebmt.js:85:8157
        at Array.map ()
        at Q0 (index-CGfYebmt.js:85:8150)
        at index-CGfYebmt.js:85:8240
        at Object.useMemo (index-CGfYebmt.js:48:46087)
        at My.$.useMemo (index-CGfYebmt.js:17:7276)
        at V0 (index-CGfYebmt.js:85:8228)
    R0 @ index-CGfYebmt.js:85
    X0 @ index-CGfYebmt.js:85
    (anonymous) @ index-CGfYebmt.js:85
    Q0 @ index-CGfYebmt.js:85
    (anonymous) @ index-CGfYebmt.js:85
    useMemo @ index-CGfYebmt.js:48
    My.$.useMemo @ index-CGfYebmt.js:17
    V0 @ index-CGfYebmt.js:85
    tc @ index-CGfYebmt.js:48
    bc @ index-CGfYebmt.js:48
    lo @ index-CGfYebmt.js:48
    No @ index-CGfYebmt.js:48
    wm @ index-CGfYebmt.js:48
    Xc @ index-CGfYebmt.js:48
    Ao @ index-CGfYebmt.js:48
    Qo @ index-CGfYebmt.js:48
    me @ index-CGfYebmt.js:25

Re: SQLite File Format Viewer

#39

Post by invisal »

roelschroeven wrote: post #37

I tried with a simple database, and it just shows me a blank screen. The console in developer tools shows an exception (both in Firefox and Chromium): index-CGfYebmt.js:85 Uncaught RangeError: Offset is outside the bounds of the DataView at DataView.prototype.getUint32 ( ) at R0 (index-CGfYebmt.js:85:4683) at X0 (index-CGfYebmt.js:85:7966) at index-CGfYebmt.js:85:8157 at Array.map ( ) at Q0 (index-CGfYebmt.js:85:8150…

Probably I might made mistake somewhere in my parser. Do you have small reproducible database with no sensitive database and add to the https://github.com/invisal/sqlite-internal/issues

I will try to fix it

Re: SQLite File Format Viewer

#40

Post by invisal »

jchw wrote: post #13

This really would've come in handy when I was debugging my own SQLite parser a couple weeks ago. One thing that initially confused me was how exactly the pages worked w.r.t. the first page on disk... I misunderstood the SQLite documentation in different ways, but it's really rather simple: the very first page is just treated as containing the file header in it, and it pushes down the rest of the data, making the page…

Glad you like it.
Post reply on HN