Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hosting SQLite Databases on GitHub Pages (2021)

phiresky.github.io

Post reply on HN
Print view View original

11–20 of 21 posts

Re: Hosting SQLite Databases on GitHub Pages (2021)

#11

Post by Groxx »

cwmma wrote: post #6
misthop wrote: post #5

It would be a it more compelling if every example, at least for me, didn't return `[error: NetworkError: A network error occurred.]`

I'm assuming your using firefox, if so it's this bug[1] that basically prevents range requests from working. Basically firefox says it will accept gziped data even though it's a range request and github pages dudifly sends back an unreadable slice of a gziped file. 1. https://bugzilla.mozilla.org/show_bug.cgi?id=1874840

huh. is this due to ambiguity in whether you want gzipped content vs gzipped transport (of arbitrary content), and/or which range the bytes are requesting? I can see both being useful, but I don't know what headers are available for these intentions...

Re: Hosting SQLite Databases on GitHub Pages (2021)

#13

Post by ngc6677 »

cwmma wrote: post #6
misthop wrote: post #5

It would be a it more compelling if every example, at least for me, didn't return `[error: NetworkError: A network error occurred.]`

I'm assuming your using firefox, if so it's this bug[1] that basically prevents range requests from working. Basically firefox says it will accept gziped data even though it's a range request and github pages dudifly sends back an unreadable slice of a gziped file. 1. https://bugzilla.mozilla.org/show_bug.cgi?id=1874840

this exact same example used to work in firefox a few years back, i guess some change introduced this bug in between

Re: Hosting SQLite Databases on GitHub Pages (2021)

#14

Post by mrtimo »

pspeter3 wrote: post #2

I wonder if the author would use DuckDB WASM now?

I'm using DuckDB WASM on github pages. This will take about 10 seconds to load [1] and shows business trends in my county (Spokane County). This site is built using data-explorer [2] which uses many other open-source projects including malloy and malloy-explorer. One cool thing... if you use the UI to make a query on the data - you can share the URL with someone and they will see the same result / query (it's all embedded in the URL).

[1] - https://mrtimo.github.io/spokane-co-biz/#/model/businesses/e... [2] - https://github.com/aszenz/data-explorer

Re: Hosting SQLite Databases on GitHub Pages (2021)

#15

Post by seligman99 »

ngc6677 wrote: post #13
cwmma wrote: post #6

Earlier quoted context omitted.

I'm assuming your using firefox, if so it's this bug[1] that basically prevents range requests from working. Basically firefox says it will accept gziped data even though it's a range request and github pages dudifly sends back an unreadable slice of a gziped file. 1. https://bugzilla.mozilla.org/show_bug.cgi?id=1874840

this exact same example used to work in firefox a few years back, i guess some change introduced this bug in between

Indeed, I lost the history in a shuffle, but a similar use case broke in some Firefox update, and it's the exact reason behind this comment:

https://github.com/seligman/podcast_to_text/blob/master/sear...

In my case, loading the entire file is loading a tiny bit more data, so this fallback doesn't hurt, but it's still annoying, and broke any hope I had of doing something more clever with the dataset.

Re: Hosting SQLite Databases on GitHub Pages (2021)

#16

Post by cwmma »

Groxx wrote: post #11
cwmma wrote: post #6

Earlier quoted context omitted.

I'm assuming your using firefox, if so it's this bug[1] that basically prevents range requests from working. Basically firefox says it will accept gziped data even though it's a range request and github pages dudifly sends back an unreadable slice of a gziped file. 1. https://bugzilla.mozilla.org/show_bug.cgi?id=1874840

huh. is this due to ambiguity in whether you want gzipped content vs gzipped transport (of arbitrary content), and/or which range the bytes are requesting? I can see both being useful, but I don't know what headers are available for these intentions...

my understanding is that technically only gzipped content is supported, not gzipped transport of arbitrary content. Due to ambiguity around the word 'append' [1] in the spec, firefox adds 'identity' (aka don't compress) to the end of the list of compressions supported while most other browsers replace the list with 'identity'. Also it should be noted that this is not a user configurable header so you can't actually try to override it.

There is a second layer to the bug in that github pages should almost certainly not be sending back slices of compressed files even if gzip is listed before identity and some change to something in the github stack probably exposed the bug that was there in firefox all along.

I literally just stumbled on this last week while doing a side project that involved browser range requests so this is fresh in my head.

1. it comes down to whether it was meant that 'Accept-Encoding:identity' should be appended to the list of header values possibly overwriting the one that already was there or if 'identity' should be appended to the list of values already in the 'Accept-Encoding' header. Firefox does the latter, everyone else does the former.

Re: Hosting SQLite Databases on GitHub Pages (2021)

#17

Post by cwmma »

ngc6677 wrote: post #13
cwmma wrote: post #6

Earlier quoted context omitted.

I'm assuming your using firefox, if so it's this bug[1] that basically prevents range requests from working. Basically firefox says it will accept gziped data even though it's a range request and github pages dudifly sends back an unreadable slice of a gziped file. 1. https://bugzilla.mozilla.org/show_bug.cgi?id=1874840

this exact same example used to work in firefox a few years back, i guess some change introduced this bug in between

The actual regression might be with githup pages where firefox was sending the same ambiguous headers the whole time but something in github's stack started interpreting them differently.

Re: Hosting SQLite Databases on GitHub Pages (2021)

#19

Post by westurner »

pspeter3 wrote: post #2

I wonder if the author would use DuckDB WASM now?

From "Show HN: TeaTime – distributed book library powered by SQLite, IPFS and GitHub" https://news.ycombinator.com/item?id=42264274 :

>> phiresky/sql.js-httpvfs: https://github.com/phiresky/sql.js-httpvfs

>> mmomtchev/sqlite-wasm-http: https://github.com/mmomtchev/sqlite-wasm-http

>> This project is inspired from @phiresky/sql.js-httpvfs but uses the new official SQLite WASM distribution

duckdb/duckdb-wasm: https://github.com/duckdb/duckdb-wasm

Re: Hosting SQLite Databases on GitHub Pages (2021)

#20

Post by westurner »

westurner wrote: post #19
pspeter3 wrote: post #2

I wonder if the author would use DuckDB WASM now?

From "Show HN: TeaTime – distributed book library powered by SQLite, IPFS and GitHub" https://news.ycombinator.com/item?id=42264274 : >> phiresky/sql.js-httpvfs: https://github.com/phiresky/sql.js-httpvfs >> mmomtchev/sqlite-wasm-http: https://github.com/mmomtchev/sqlite-wasm-http >> This project is inspired from @phiresky/sql.js-httpvfs but uses the new official SQLite WASM distribution duckdb/duckdb-wasm: https://g…

"PSA: SQLite WAL checksums fail silently and may lose data" so that's probably not how to sync sqlite; https://news.ycombinator.com/item?id=44672902

electric-sql/electric: https://github.com/electric-sql/electric :

> Specifically, Electric is a read-path sync engine for Postgres. It syncs data out of Postgres into ... anything you like. The core sync protocol is based on a low-level HTTP API. This integrates with CDNs for highly-scalable data delivery.

electric-sql/pglite: https://github.com/electric-sql/pglite :

> Embeddable Postgres with real-time, reactive bindings.

"Using Postgres for Everything" https://news.ycombinator.com/item?id=40420474

Post reply on HN