[flagged]
Show HN: 22 GB of Hacker News in SQLite
21–30 of 229 posts
Re: Show HN: 22 GB of Hacker News in SQLite
#22Don't miss how this works. It's not a server-side application - this code runs entirely in your browser using SQLite compiled to WASM, but rather than fetching a full 22GB database it instead uses a clever hack that retrieves just "shards" of the SQLite database needed for the page you are viewing.
I watched it in the browser network panel and saw it fetch:
https://hackerbook.dosaygo.com/static-shards/shard_1636.sqlite.gz
https://hackerbook.dosaygo.com/static-shards/shard_1635.sqlite.gz
https://hackerbook.dosaygo.com/static-shards/shard_1634.sqlite.gz
It's reminiscent of that brilliant SQLite.js VFS trick from a few years ago: https://github.com/phiresky/sql.js-httpvfs - only that one used HTTP range headers, this one uses sharded files instead.
The interactive SQL query interface at https://hackerbook.dosaygo.com/?view=query asks you to select which shards to run the query against, there are 1636 total.
Re: Show HN: 22 GB of Hacker News in SQLite
#23That repo is throwing up a 404 for me. Question - did you consider tradeoffs between duckdb (or other columnar stores) and SQLite?
No, I just went straight to sqlite. What is duckdb?
One interesting feature of DuckDB is that it can run queries against HTTP ranges of a static file hosted via HTTPS, and there's an official WebAssembly build of it that can do that same trick.
So you can dump e.g. all of Hacker News in a single multi-GB Parquet file somewhere and build a client-side JavaScript application that can run queries against that without having to fetch the whole thing.
You can run searches on https://lil.law.harvard.edu/data-gov-archive/ and watch the network panel to see DuckDB in action.
Re: Show HN: 22 GB of Hacker News in SQLite
#24What a reminder on how text is so much more efficient than video, its crazy! Could you imagine the same amount of knowledge (or dribble) but in video form? I wonder how large that would be.
Re: Show HN: 22 GB of Hacker News in SQLite
#25[flagged]
Why do you say that?
Re: Show HN: 22 GB of Hacker News in SQLite
#26This would be awesome as a cross platform app.
Re: Show HN: 22 GB of Hacker News in SQLite
#27Re: Show HN: 22 GB of Hacker News in SQLite
#28The link seems to be down, was it taken down?
Re: Show HN: 22 GB of Hacker News in SQLite
#29Earlier quoted context omitted.
Why do you say that?
Because anything that even slightly differs from the standard American phrasing of something must be "LLM generated" these days.
With the em dashes I see you. But at this point idrc so long as it reads well. Everyone uses spell check…
Re: Show HN: 22 GB of Hacker News in SQLite
#30[flagged]
There's a thing in soccer at the moment where a tackle looks fine in realtime but when the video referee shows it to the onpitch referee, they show the impact in slo-mo over and over again and it always looks way worse.
I wonder if there's something like this going on here. I never thought it was LLM on first read, and I still don't, but when you take snippets and point at them it makes me think maybe they are