Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: 22 GB of Hacker News in SQLite

hackerbook.dosaygo.com

Post reply on HN
Print view View original

51–60 of 229 posts

Re: Show HN: 22 GB of Hacker News in SQLite

#51

Post by nextaccountic »

simonw wrote: post #22

Don't miss how this works. It's not a server-side application - this code runs entirely in your browser using SQLite compiled to WASM, but rather than fetching a full 22GB database it instead uses a clever hack that retrieves just "shards" of the SQLite database needed for the page you are viewing. I watched it in the browser network panel and saw it fetch: https://hackerbook.dosaygo.com/static-shards/shard_1636.sqli…

Is there anything more production grade built around the same idea of HTTP range requests like that sqlite thing? This has so much potential

Re: Show HN: 22 GB of Hacker News in SQLite

#52

Post by simonw »

nextaccountic wrote: post #51
simonw wrote: post #22

Don't miss how this works. It's not a server-side application - this code runs entirely in your browser using SQLite compiled to WASM, but rather than fetching a full 22GB database it instead uses a clever hack that retrieves just "shards" of the SQLite database needed for the page you are viewing. I watched it in the browser network panel and saw it fetch: https://hackerbook.dosaygo.com/static-shards/shard_1636.sqli…

Is there anything more production grade built around the same idea of HTTP range requests like that sqlite thing? This has so much potential

There was a UK government GitHub repo that did something interesting with this kind of trick against S3 but I checked just now and the repo is a 404. Here are my notes about what it did: https://simonwillison.net/2025/Feb/7/sqlite-s3vfs/

Looks like it's still on PyPI though: https://pypi.org/project/sqlite-s3vfs/

You can see inside it with my PyPI package explorer: https://tools.simonwillison.net/zip-wheel-explorer?package=s...

Re: Show HN: 22 GB of Hacker News in SQLite

#54

Post by ericd »

nextaccountic wrote: post #51
simonw wrote: post #22

Don't miss how this works. It's not a server-side application - this code runs entirely in your browser using SQLite compiled to WASM, but rather than fetching a full 22GB database it instead uses a clever hack that retrieves just "shards" of the SQLite database needed for the page you are viewing. I watched it in the browser network panel and saw it fetch: https://hackerbook.dosaygo.com/static-shards/shard_1636.sqli…

Is there anything more production grade built around the same idea of HTTP range requests like that sqlite thing? This has so much potential

This is somewhat related to a large dataset browsing service a friend and I worked on a while back - we made index files, and the browser ran a lightweight query planner to fetch static chunks which could be served from S3/torrents/whatever. It worked pretty well, and I think there’s a lot of potential for this style of data serving infra.

Re: Show HN: 22 GB of Hacker News in SQLite

#55

Post by Sn0wCoder »

Site does not load on Firefox console error says 'Uncaught (in promise) TypeError: can't access property "wasm", sqlite3 is null'

Guess its common knowledge that SharedArrayBuffer (SQLite wasm) does not work with FF due to Cross-Origin Attacks (i just found out ;).

Once the initial chunk of data loads the rest load almost instantly on Chrome. Can you please fix the GitHub link (current 404) would like to peak at the code. Thank you!

Re: Show HN: 22 GB of Hacker News in SQLite

#57

Post by dspillett »

Is there a public dump of the data anywhere that this is based upon, or have they scraped it themselves?

Such as DB might be entertaining to play with, and the threadedness of comments would be useful for beginners to practise efficient recursive queries (more so than the StackExchange dumps, for instance).

Re: Show HN: 22 GB of Hacker News in SQLite

#58

Post by rplnt »

mburns wrote: post #47
jesprenj wrote: post #39

Earlier quoted context omitted.

do you mean single database? it'd be quite hard if not impossible to make applications using a single table (no relations). reddit did it though, they have a huge table of "things" iirc.

That is a common misconception. > Next, we've got more than just two tables. The quote/paraphrase doesn't make it clear, but we've got two tables per thing. That means Accounts have an "account_thing" and an "account_data" table, Subreddits have a "subreddit_thing" and "subreddit_data" table, etc. https://www.reddit.com/r/programming/comments/z9sm8/comment/...

And the important lesson from that the k/v-like aspect of it. That the "schema" is horizontal (is that a thing?) and not column-based. But I actually only read it on their blog IIRC and never even got the full details - that there's still a third ID column. Thanks for the link.

Re: Show HN: 22 GB of Hacker News in SQLite

#59

Post by [deleted] »

Sn0wCoder wrote: post #55

Site does not load on Firefox console error says 'Uncaught (in promise) TypeError: can't access property "wasm", sqlite3 is null' Guess its common knowledge that SharedArrayBuffer (SQLite wasm) does not work with FF due to Cross-Origin Attacks (i just found out ;). Once the initial chunk of data loads the rest load almost instantly on Chrome. Can you please fix the GitHub link (current 404) would like to peak at the…

[deleted]

Re: Show HN: 22 GB of Hacker News in SQLite

#60

Post by jacquesm »

yread wrote: post #49

I wonder how much smaller it could get with some compression. You could probably encode "This website hijacks the scrollbar and I don't like it" comments into just a few bits.

That's at least 45%, then you can leave out all of my comments and you're left with only 5!
Post reply on HN