Looks like the repo was taken down (404).
That's too bad, I'd like to see the inner-working with a subset of data, even with placeholders for the posts and comments.
Show HN: 22 GB of Hacker News in SQLite
71–80 of 229 posts
Re: Show HN: 22 GB of Hacker News in SQLite
#72How do I download it? That repo is a 404.
Re: Show HN: 22 GB of Hacker News in SQLite
#73Earlier quoted context omitted.
There was a UK government GitHub repo that did something interesting with this kind of trick against S3 but I checked just now and the repo is a 404. Here are my notes about what it did: https://simonwillison.net/2025/Feb/7/sqlite-s3vfs/ Looks like it's still on PyPI though: https://pypi.org/project/sqlite-s3vfs/ You can see inside it with my PyPI package explorer: https://tools.simonwillison.net/zip-wheel-explorer?p…
I recovered it from https://archive.softwareheritage.org/browse/origin/directory... and pushed a fresh copy to GitHub here: https://github.com/simonw/sqlite-s3vfs This comment was helpful in figuring out how to get a full Git clone out of the heritage archive: https://news.ycombinator.com/item?id=37516523#37517378 Here's a TIL I wrote up of the process: https://til.simonwillison.net/github/software-archive-recove...
didn't you do something similar for Datasette, Simon?
Re: Show HN: 22 GB of Hacker News in SQLite
#74What a reminder on how text is so much more efficient than video, its crazy! Could you imagine the same amount of knowledge (or dribble) but in video form? I wonder how large that would be.
That's what's so sad about youtube. 20 minute videos to encode a hundred words of usable content to get you to click on a link. The inefficiency is just staggering.
Youtube can be excellent for explanations. A picture's worth a thousand words, and you can fit a lot of decent pictures in a 20 minute video. The signal-to-noise can be high, of course.
Re: Show HN: 22 GB of Hacker News in SQLite
#75Looks like the repo was taken down (404). That's too bad, I'd like to see the inner-working with a subset of data, even with placeholders for the posts and comments.
That was fast. I was looking into recent HN datasets, and they are impossible find.
Re: Show HN: 22 GB of Hacker News in SQLite
#76Looks like the repo was taken down (404). That's too bad, I'd like to see the inner-working with a subset of data, even with placeholders for the posts and comments.
That was fast. I was looking into recent HN datasets, and they are impossible find.
Complete and continuously updated: https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0a...
Re: Show HN: 22 GB of Hacker News in SQLite
#77Earlier quoted context omitted.
I recovered it from https://archive.softwareheritage.org/browse/origin/directory... and pushed a fresh copy to GitHub here: https://github.com/simonw/sqlite-s3vfs This comment was helpful in figuring out how to get a full Git clone out of the heritage archive: https://news.ycombinator.com/item?id=37516523#37517378 Here's a TIL I wrote up of the process: https://til.simonwillison.net/github/software-archive-recove...
didn't you do something similar for Datasette, Simon?
Nothing smart with HTTP range requests yet - I have https://lite.datasette.io which runs the full Python server app in the browser via WebAssembly and Pyodide but it still works by fetching the entire SQLite file at once.
Re: Show HN: 22 GB of Hacker News in SQLite
#78Don't miss how this works. It's not a server-side application - this code runs entirely in your browser using SQLite compiled to WASM, but rather than fetching a full 22GB database it instead uses a clever hack that retrieves just "shards" of the SQLite database needed for the page you are viewing. I watched it in the browser network panel and saw it fetch: https://hackerbook.dosaygo.com/static-shards/shard_1636.sqli…
Is there anything more production grade built around the same idea of HTTP range requests like that sqlite thing? This has so much potential
[deleted]
Re: Show HN: 22 GB of Hacker News in SQLite
#79Earlier quoted context omitted.
didn't you do something similar for Datasette, Simon?
Nothing smart with HTTP range requests yet - I have https://lite.datasette.io which runs the full Python server app in the browser via WebAssembly and Pyodide but it still works by fetching the entire SQLite file at once.
oh! I must've been confused with your TIL where you linked to an explainer of this technique
https://simonwillison.net/2021/May/2/hosting-sqlite-database...
https://phiresky.github.io/blog/2021/hosting-sqlite-database...
Re: Show HN: 22 GB of Hacker News in SQLite
#80That repo is throwing up a 404 for me. Question - did you consider tradeoffs between duckdb (or other columnar stores) and SQLite?
No, I just went straight to sqlite. What is duckdb?
"What is duckdb?"
duckdb is a 45M dynamically-linked binary (amd64)
sqlite3 1.7M static binary (amd64)
DuckDB is a 6yr-old project
SQLite is a 25yr-old project