Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: 22 GB of Hacker News in SQLite

hackerbook.dosaygo.com

Post reply on HN
Print view View original

221–229 of 229 posts

Re: Show HN: 22 GB of Hacker News in SQLite

#221

Post by keepamovin »

kamranjon wrote: post #97

It'd be great if you could add it to Kiwix[1] somehow (not sure what the process is for that but 100rabbits figured it out for their site) - I use it all the time now that I have a dumb phone - I have the entirety of wikipedia, wiktionary and 100rabbits all offline. https://kiwix.org/en/

This is a good idea — we should do it.

I also want to make sure we can build this in CI. My goal is to have this updated every day using the BigQuery update process, so it becomes a 1–2 day delayed static archive of the current state of Hacker News, which is honestly very cool.

I can probably run the build for free on GitHub Actions runners, as long as the runner has about 40 GB of disk space available. If needed, I can free up space on the runner before the build starts.

I’ll also write to GitHub support and ask if they can sponsor the cost of a larger runner, mainly because I need the extra disk space to run the build reliably.

Re: Show HN: 22 GB of Hacker News in SQLite

#222

Post by keepamovin »

yread wrote: post #49

I wonder how much smaller it could get with some compression. You could probably encode "This website hijacks the scrollbar and I don't like it" comments into just a few bits.

Dear it is already compressed using G zip – nine for every SQLlight shard and manifest

22 GB is uncompressed and compressed the entire things about 9 GB

Re: Show HN: 22 GB of Hacker News in SQLite

#223

Post by nextaccountic »

jtbaker wrote: post #161
nextaccountic wrote: post #103

Earlier quoted context omitted.

That's neat, but.. is it just for cartographic data? I want something like a db with indexes

Look into using duckdb with remote http/s3 parquet files. The parquet files are organized as columnar vectors, grouped into chunks of rows. Each row group stores metadata about the set it contains that can be used to prune out data that doesn’t need to be scanned by the query engine. https://duckdb.org/docs/stable/guides/performance/indexing LanceDB has a similar mechanism for operating on remote vector embeddings/te…

> Look into using duckdb with remote http/s3 parquet files. The parquet files are organized as columnar vectors, grouped into chunks of rows. Each row group stores metadata about the set it contains that can be used to prune out data that doesn’t need to be scanned by the query engine. https://duckdb.org/docs/stable/guides/performance/indexing

But, when using this on frontend, are portions of files fetched specifically with http range requests? I tried to search for it but couldn't find details

Re: Show HN: 22 GB of Hacker News in SQLite

#224

Post by keepamovin »

jacquesm wrote: post #193
keepamovin wrote: post #170

Earlier quoted context omitted.

Thanks, bud, that means a lot! Would like to see your versions of the data stored offline idea, it's very cool.

pianojacq.com It's super simple, really, far less impressive than what you've built there.

That's really cool, man. The music notation is beautiful. I hit play but couldn't get it to progress past the first note. Maybe I need to plug in a midi keyboard? It would be cool if I could "play" with my ASCII keyboard.

Listen was nice. That's really cool, actually. I encourage you to do it.

Re: Show HN: 22 GB of Hacker News in SQLite

#225

Post by jtbaker »

nextaccountic wrote: post #223
jtbaker wrote: post #161

Earlier quoted context omitted.

Look into using duckdb with remote http/s3 parquet files. The parquet files are organized as columnar vectors, grouped into chunks of rows. Each row group stores metadata about the set it contains that can be used to prune out data that doesn’t need to be scanned by the query engine. https://duckdb.org/docs/stable/guides/performance/indexing LanceDB has a similar mechanism for operating on remote vector embeddings/te…

> Look into using duckdb with remote http/s3 parquet files. The parquet files are organized as columnar vectors, grouped into chunks of rows. Each row group stores metadata about the set it contains that can be used to prune out data that doesn’t need to be scanned by the query engine. https://duckdb.org/docs/stable/guides/performance/indexing But, when using this on frontend, are portions of files fetched specifical…

Yes, you should be able to see the byte range requests and 206 responses from an s3 compatible bucket or http server that supports those access patterns.

Re: Show HN: 22 GB of Hacker News in SQLite

#227

Post by adamszakal »

keepamovin wrote: post #215
adamszakal wrote: post #204

Earlier quoted context omitted.

Just following the ordinary guidelines when doing responsive designs, like increasing the text size and sizes of buttons and inputs, so my fat fingers don't missklick every other try. HN has gotten better, but is still below average, hence why I thought it was some kind of aesthetic choice.

I tried to keep to HN look. But I’m not a mobile design expert. I will give it a once over see if i can make it better for you

You rock dude, hope it didn't come out as condescending! It's an awesome project.

Re: Show HN: 22 GB of Hacker News in SQLite

#228

Post by WadeGrimridge »

keepamovin wrote: post #219
WadeGrimridge wrote: post #206

Earlier quoted context omitted.

yes, please, that'd be cool! you can link my site grimridge.net if you'd like. here are the numbers so you can plot graphs that fit the site's style: mean (all time): https://katb.in/yutupojerux mean (since 2020): https://katb.in/omoyibisava median (all time): https://katb.in/kilopofivet median (since 2020): https://katb.in/ukefetuyuhi

Added in the latest: https://hackerbook.dosaygo.com/?view=archive

just saw this, thanks!
Post reply on HN