Don't miss how this works. It's not a server-side application - this code runs entirely in your browser using SQLite compiled to WASM, but rather than fetching a full 22GB database it instead uses a clever hack that retrieves just "shards" of the SQLite database needed for the page you are viewing. I watched it in the browser network panel and saw it fetch: https://hackerbook.dosaygo.com/static-shards/shard_1636.sqli…
Show HN: 22 GB of Hacker News in SQLite
211–220 of 229 posts
Re: Show HN: 22 GB of Hacker News in SQLite
#212What a reminder on how text is so much more efficient than video, its crazy! Could you imagine the same amount of knowledge (or dribble) but in video form? I wonder how large that would be.
Average high quality 1080p60 video has bitrate of 5Mbps, which is equivalent to 120k English words per second. With average English speech being 150wpm, we end up with text being 50 thousand times more space efficient. Converting 22GB of uncompressed text into video essay lands us at ~1PB or 1000TB.
Re: Show HN: 22 GB of Hacker News in SQLite
#213I wonder how much smaller it could get with some compression. You could probably encode "This website hijacks the scrollbar and I don't like it" comments into just a few bits.
Re: Show HN: 22 GB of Hacker News in SQLite
#214Earlier quoted context omitted.
So cool! Would it be impossible fro me to use them on the Archive stats page ( https://hackerbook.dosaygo.com/?view=archive ) ? If you're okay with that any links/credit line details? Totally cool if not, just super interesting!
yes, please, that'd be cool! you can link my site grimridge.net if you'd like. here are the numbers so you can plot graphs that fit the site's style: mean (all time): https://katb.in/yutupojerux mean (since 2020): https://katb.in/omoyibisava median (all time): https://katb.in/kilopofivet median (since 2020): https://katb.in/ukefetuyuhi
Re: Show HN: 22 GB of Hacker News in SQLite
#215Earlier quoted context omitted.
Really? Let me know how I can help. What would you like to see fixed?
Just following the ordinary guidelines when doing responsive designs, like increasing the text size and sizes of buttons and inputs, so my fat fingers don't missklick every other try. HN has gotten better, but is still below average, hence why I thought it was some kind of aesthetic choice.
Re: Show HN: 22 GB of Hacker News in SQLite
#216Earlier quoted context omitted.
Your page is served over sqlitevfs with Range queries? Let's try this here.
I did a similar VFS in Go. It doesn't run client-side in a browser. But you can use it (e.g.) in a small VPS to access a multi-TB database directly from S3.
Re: Show HN: 22 GB of Hacker News in SQLite
#217Re: Show HN: 22 GB of Hacker News in SQLite
#218Earlier quoted context omitted.
A JSON parser is less than a thousand lines of code.
And where most of CPU time will be wasted in, if you care about profiling/improving responsiveness.
Re: Show HN: 22 GB of Hacker News in SQLite
#219Earlier quoted context omitted.
So cool! Would it be impossible fro me to use them on the Archive stats page ( https://hackerbook.dosaygo.com/?view=archive ) ? If you're okay with that any links/credit line details? Totally cool if not, just super interesting!
yes, please, that'd be cool! you can link my site grimridge.net if you'd like. here are the numbers so you can plot graphs that fit the site's style: mean (all time): https://katb.in/yutupojerux mean (since 2020): https://katb.in/omoyibisava median (all time): https://katb.in/kilopofivet median (since 2020): https://katb.in/ukefetuyuhi
Re: Show HN: 22 GB of Hacker News in SQLite
#220Earlier quoted context omitted.
what dumb phone do you use? and why do you want wikipedia in your pocket, but not a smartphone? where do you draw the line? (doing a lot of work in that area, so i am asking to learn from someone who might think alike)
I use the Mudita Kompakt specifically cause it allows sideloading so I can still have a few extras. Right now I have Kiwix and Libby. It works really well. I have a $10 a month plan from US cellular with only 2gigs so I try to keep everything offline that I can. Honestly it's mostly the news... so I draw the line at browser, I'll never install a browser, that's basically something I can do when I sit down at a PC. I…
your input would be very valuable.