Show HN: 22 GB of Hacker News in SQLite • Hacker News Board

Re: Show HN: 22 GB of Hacker News in SQLite

#191

Post by Sn0wCoder » Wed, Dec 31, 2025, 3:02 PM UTC

keepamovin wrote: post #134

Sn0wCoder wrote: post #55

Site does not load on Firefox console error says 'Uncaught (in promise) TypeError: can't access property "wasm", sqlite3 is null' Guess its common knowledge that SharedArrayBuffer (SQLite wasm) does not work with FF due to Cross-Origin Attacks (i just found out ;). Once the initial chunk of data loads the rest load almost instantly on Chrome. Can you please fix the GitHub link (current 404) would like to peak at the…

Damn. Will try to fix for FF. edit: I just tested with FF latest, seems to be working.

Strange now the first few days load (getting a new error) 'Ignoring inability to install OPFS sqlite3_vfs: Cannot install OPFS: Missing SharedArrayBuffer and/or Atomics. The server must emit the COOP/COEP response headers to enable those. See https://sqlite.org/wasm/doc/trunk/persistence.md#coop-coep'

But when go back to the 26th none of the shards will load, error out.

Using Windows 11, FF 146.0.1

Since you tested it seems its just a me problem and thanks for fixing the GitHub link

Re: Show HN: 22 GB of Hacker News in SQLite

#192

Post by keepamovin » Wed, Dec 31, 2025, 3:22 PM UTC

hyperbolablabla wrote: post #168

simonw wrote: post #68

Earlier quoted context omitted.

PMTiles is absurdly great software.

My only gripe is that the tile metadata is stored as JSON, which I get is for compatibility reasons with existing software, but for e.g. a simple C program to implement the full spec you need to ship a JSON parser on top of the PMTiles parser itself.

How would you store it?

Re: Show HN: 22 GB of Hacker News in SQLite

#193

Post by jacquesm » Wed, Dec 31, 2025, 3:44 PM UTC

keepamovin wrote: post #170

jacquesm wrote: post #165

Earlier quoted context omitted.

Pretty neat project. I never thought you could do this in the first place, very much inspiring. I've made a little project that stores all of its data locally but still runs in the browser to protect against take downs and because I don't think you should store your precious data online more than you have to, eventually it all rots away. Your project takes this to the next level.

Thanks, bud, that means a lot! Would like to see your versions of the data stored offline idea, it's very cool.

pianojacq.com

It's super simple, really, far less impressive than what you've built there.

Re: Show HN: 22 GB of Hacker News in SQLite

#194

Post by 3eb7988a1663 » Wed, Dec 31, 2025, 4:07 PM UTC

peheje wrote: post #167

3eb7988a1663 wrote: post #10

Earlier quoted context omitted.

While I suspect DuckDB would compress better, given the ubiquity of SQLite, it seems a fine standard choice.

the data is dominated by big unique TEXT columns, unsure how that can much compress better when grouped - but would be interesting to know

I was thinking more the numeric columns which have pre-built compression mechanisms to handle incrementing columns or long runs of identical values. For sure less total data than the text, but my prior is that the two should perform equivalently on the text, so the better compression on numbers should let duckdb pull ahead.

I had to run a test for myself, and using sqlite2duckdb (no research, first search hit), and using randomly picked shard 1636, the sqlite.gz was 4.9MB, but the duckdb.gz was 3.7MB.

The uncompressed sizes favor sqlite, which does not make sense to me, so not sure if duckdb keeps around more statistics information. Uncompressed sqlite 12.9MB, duckdb 15.5MB

Re: Show HN: 22 GB of Hacker News in SQLite

#195

Post by fao_ » Wed, Dec 31, 2025, 4:14 PM UTC

rantingdemon wrote: post #25

fao_ wrote: post #21

[flagged]

Why do you say that?

It feels like a LLM doing it's usual "gushing out appreciations"

Re: Show HN: 22 GB of Hacker News in SQLite

#196

Post by fao_ » Wed, Dec 31, 2025, 4:15 PM UTC

naikrovek wrote: post #40

JavGull wrote: post #29

Earlier quoted context omitted.

With the em dashes I see you. But at this point idrc so long as it reads well. Everyone uses spell check…

I add em dashes to everything I write now, solely to throw people who look for them off. Lots of editors add them automatically when you have two sequential dashes between words — a common occurrence, like that one. And this is is Chrome on iOS doing it automatically. Ooh, I used “sequential”, ooh, I used an em dash. ZOMG AI IS COMING FOR US ALL

I also use Em-Dashes, this is about how weird the thing is tonally

Re: Show HN: 22 GB of Hacker News in SQLite

#197

Post by fao_ » Wed, Dec 31, 2025, 4:16 PM UTC

jesprenj wrote: post #38

fao_ wrote: post #21

[flagged]

I doubt it. "hacker news" spelled lowercase? comma after "beauty"? missing "in" after "it's"? i doubt an LLM would make such syntax mistakes. it's just good writing, that's also possible these days.

> it's just good writing, that's also possible these days.

As someone reskilling into being a writer, I really do not think that is "good writing".

Re: Show HN: 22 GB of Hacker News in SQLite

#198

Post by keepamovin » Wed, Dec 31, 2025, 4:24 PM UTC

Sn0wCoder wrote: post #191

keepamovin wrote: post #134

Earlier quoted context omitted.

Damn. Will try to fix for FF. edit: I just tested with FF latest, seems to be working.

Strange now the first few days load (getting a new error) 'Ignoring inability to install OPFS sqlite3_vfs: Cannot install OPFS: Missing SharedArrayBuffer and/or Atomics. The server must emit the COOP/COEP response headers to enable those. See https://sqlite.org/wasm/doc/trunk/persistence.md#coop-coep ' But when go back to the 26th none of the shards will load, error out. Using Windows 11, FF 146.0.1 Since you tested…

No I've seen that error too, on Safari. I think it's related to the wasm being sent with wrong headers. CF pages _headers file should be ensuring correctness. Can you try busting your cache (or wait for a new Dec 29 Data dump version coming in a couple minutes), or from incognito to see if that fixes the issue? It's possible an earlier version had stale headers or sth. Idk.

Re: Show HN: 22 GB of Hacker News in SQLite

#199

Post by keepamovin » Wed, Dec 31, 2025, 4:24 PM UTC

rcarmo wrote: post #187

Nice. I wonder if there’s any way to quickly get a view for a whole year.

You mean like stuff ranked per year?

Edit: Good idea! I implemented a "year" selector so all main views (front/show/ask/jobs) will be from that entire year rather than just a single day.

Re: Show HN: 22 GB of Hacker News in SQLite

#200

Post by ComputerGuru » Wed, Dec 31, 2025, 5:39 PM UTC

Rendello wrote: post #74

jacquesm wrote: post #61

Earlier quoted context omitted.

That's what's so sad about youtube. 20 minute videos to encode a hundred words of usable content to get you to click on a link. The inefficiency is just staggering.

Youtube can be excellent for explanations. A picture's worth a thousand words, and you can fit a lot of decent pictures in a 20 minute video. The signal-to-noise can be high, of course.

Unfortunately even the videos that do contain helpful imagery are still dominated by huge sections of low entropy.

For example, one of the most useful applications of video over text is appliance or automotive repair, but the ideal format would be an article interspersed with short video sections, not a video with a talking head and some ~static shaky cam taking up most of the time as the individual drones on about mostly unrelated topics or unimportant details yet you can’t skip past it in case there is something actually pertinent covered in that time.