Show HN: 22 GB of Hacker News in SQLite • Hacker News Board

Re: Show HN: 22 GB of Hacker News in SQLite

#101

Post by fsiefken » Wed, Dec 31, 2025, 1:16 AM UTC

deskamess wrote: post #46

fsiefken wrote: post #41

Earlier quoted context omitted.

one could use a video llm to generate the video, diagrams or the stills automatically based on the text. except when it's boardgames playthroughs or programming i just transcribe to text, summarise and read youtube video's.

How do you read youtube videos? Very curious as I have been wanting to watch PDF's scroll by slowly on a large TV. I am interested in the workflow of getting a pdf/document into a scrolling video format. These days NotebookLM may be an option but I am curious if there is something custom. If I can get it into video form (mp4) then I can even deliver it via plex.

I use yt-dlp to download the transcript, and if it's not available i can get the audio file and run it through parakeet locally. Then I have the plain text, which could be read out loud (kind of defeating the purpose), but perhaps at triple speed with a computer voice that's still understandble at that speed. I could also summarize it with an llm. With pandoc or typst I can convert to single column or mult column pdf to print or watch on tv or my smart glasses. If I strip the vowels and make the font smaller I can fit more!

One could convert the Markdown/PDF to a very long image first with pandoc+wkhtml, then use ffmpeg to crop and move the viewport slowly over the image, this scrolls at 20 pixels per second for 30s - with the mpv player one could change speed dynamically through keys.

ffmpeg -loop 1 -i long_image.png -vf "crop=iw:ih/10:0:t*20" -t 30 -pix_fmt yuv420p output.mp4

Alternatively one could use a Rapid Serial Visual Presentation / Speedreading / Spritz technique to output to mp4 or use dedicated rsvp program where one can change speed.

One could also output to a braille 'screen'.

Scrolling mp4 text on the the TV or Laptop to read is a good idea for my mother and her macula degeneration, or perhaps I should make use of an easier to see/read magnification browser plugin tool.

Re: Show HN: 22 GB of Hacker News in SQLite

#102

Post by nextaccountic » Wed, Dec 31, 2025, 1:21 AM UTC

6510 wrote: post #96

nextaccountic wrote: post #51

Earlier quoted context omitted.

Is there anything more production grade built around the same idea of HTTP range requests like that sqlite thing? This has so much potential

I want to see a bittorrent version :P

Maybe webtorrent-based?

Re: Show HN: 22 GB of Hacker News in SQLite

#103

Post by nextaccountic » Wed, Dec 31, 2025, 1:23 AM UTC

Humphrey wrote: post #62

nextaccountic wrote: post #51

Earlier quoted context omitted.

Is there anything more production grade built around the same idea of HTTP range requests like that sqlite thing? This has so much potential

Yes — PMTiles is exactly that: a production-ready, single-file, static container for vector tiles built around HTTP range requests. I’ve used it in production to self-host Australia-only maps on S3. We generated a single ~900 MB PMTiles file from OpenStreetMap (Australia only, up to Z14) and uploaded it to S3. Clients then fetch just the required byte ranges for each vector tile via HTTP range requests. It’s fast, sc…

That's neat, but.. is it just for cartographic data?

I want something like a db with indexes

Re: Show HN: 22 GB of Hacker News in SQLite

#104

Post by omneity » Wed, Dec 31, 2025, 1:33 AM UTC

nextaccountic wrote: post #51

simonw wrote: post #22

Don't miss how this works. It's not a server-side application - this code runs entirely in your browser using SQLite compiled to WASM, but rather than fetching a full 22GB database it instead uses a clever hack that retrieves just "shards" of the SQLite database needed for the page you are viewing. I watched it in the browser network panel and saw it fetch: https://hackerbook.dosaygo.com/static-shards/shard_1636.sqli…

Is there anything more production grade built around the same idea of HTTP range requests like that sqlite thing? This has so much potential

I tried to implement something similar to optimize sampling semi-random documents from (very) large datasets on Huggingface, unfortunately their API doesn't support range requests well.

Re: Show HN: 22 GB of Hacker News in SQLite

#105

Post by 3eb7988a1663 » Wed, Dec 31, 2025, 1:40 AM UTC

naikrovek wrote: post #40

JavGull wrote: post #29

Earlier quoted context omitted.

With the em dashes I see you. But at this point idrc so long as it reads well. Everyone uses spell check…

I add em dashes to everything I write now, solely to throw people who look for them off. Lots of editors add them automatically when you have two sequential dashes between words — a common occurrence, like that one. And this is is Chrome on iOS doing it automatically. Ooh, I used “sequential”, ooh, I used an em dash. ZOMG AI IS COMING FOR US ALL

Anyone demonstrating above a high-school vocabulary/reading level is obviously a machine.

Re: Show HN: 22 GB of Hacker News in SQLite

#106

Post by 3eb7988a1663 » Wed, Dec 31, 2025, 1:41 AM UTC

Did anyone get a copy of this before it was pulled? If GitHub is not keen, could it be uploaded to HuggingFace or some other service which hosts large assets?

I have always known I could scrape HN, but I would much rather take a neat little package.

Re: Show HN: 22 GB of Hacker News in SQLite

#107

Post by modeless » Wed, Dec 31, 2025, 1:45 AM UTC

It's really a shame that comment scores are hidden forever. Would the admins consider publishing them after stories are old enough that voting is closed? It would be great to have them for archives and search indices and projects like this.

Re: Show HN: 22 GB of Hacker News in SQLite

#108

Post by deadbabe » Wed, Dec 31, 2025, 1:49 AM UTC

ssl-3 wrote: post #95

deadbabe wrote: post #34

Earlier quoted context omitted.

Sometimes I want to write more creatively, but then worry I’ll be accused of being an LLM. So I dumb it down. Remove the colorful language. Conform.

Fuck 'em. Always write what you want, however you want to write it. If some reader somewhere decides to be judgemental because of — you know — an em dash or an X/Y comparison or a complement or some other thing that they think pins you down as being a bot, then that's entirely their own problem. Not yours. They observe the reality that they deserve.

You’re absolutely right. It’s not my problem, it’s their problem.

Re: Show HN: 22 GB of Hacker News in SQLite

#109

Post by 1vuio0pswjnm7 » Wed, Dec 31, 2025, 1:51 AM UTC

1vuio0pswjnm7 wrote: post #80

keepamovin wrote: post #7

Earlier quoted context omitted.

No, I just went straight to sqlite. What is duckdb?

"What is duckdb?" duckdb is a 45M dynamically-linked binary (amd64) sqlite3 1.7M static binary (amd64) DuckDB is a 6yr-old project SQLite is a 25yr-old project

I like SQLite

Re: Show HN: 22 GB of Hacker News in SQLite

#110

Post by meander_water » Wed, Dec 31, 2025, 2:18 AM UTC

simonw wrote: post #22

Don't miss how this works. It's not a server-side application - this code runs entirely in your browser using SQLite compiled to WASM, but rather than fetching a full 22GB database it instead uses a clever hack that retrieves just "shards" of the SQLite database needed for the page you are viewing. I watched it in the browser network panel and saw it fetch: https://hackerbook.dosaygo.com/static-shards/shard_1636.sqli…

I love this so much, on my phone this is much faster than actual HN (I know it's only a read-only version).

Where did you get the 22GB figure from? On the site it says:

> 46,399,072 items, 1,637 shards, 8.5GB, spanning Oct 9, 2006 to Dec 28, 2025