131–140 of 229 posts
The query tab looks quite complex with all these content shards: https://hackerbook.dosaygo.com/?view=query I have a much simpler database: https://play.clickhouse.com/play?user=play#U0VMRUNUIHRpbWUsI...
Does your database also runs offline/locally in the browser? Seems to be the reason for the large number of shards.
Earlier quoted context omitted.
No, I just went straight to sqlite. What is duckdb?
One interesting feature of DuckDB is that it can run queries against HTTP ranges of a static file hosted via HTTPS, and there's an official WebAssembly build of it that can do that same trick. So you can dump e.g. all of Hacker News in a single multi-GB Parquet file somewhere and build a client-side JavaScript application that can run queries against that without having to fetch the whole thing. You can run searches…
It would be an interesting experiment to add the duckdb hackend
Site does not load on Firefox console error says 'Uncaught (in promise) TypeError: can't access property "wasm", sqlite3 is null' Guess its common knowledge that SharedArrayBuffer (SQLite wasm) does not work with FF due to Cross-Origin Attacks (i just found out ;). Once the initial chunk of data loads the rest load almost instantly on Chrome. Can you please fix the GitHub link (current 404) would like to peak at the…
edit: I just tested with FF latest, seems to be working.
What a reminder on how text is so much more efficient than video, its crazy! Could you imagine the same amount of knowledge (or dribble) but in video form? I wonder how large that would be.
It's really a shame that comment scores are hidden forever. Would the admins consider publishing them after stories are old enough that voting is closed? It would be great to have them for archives and search indices and projects like this.
Is there a public dump of the data anywhere that this is based upon, or have they scraped it themselves? Such as DB might be entertaining to play with, and the threadedness of comments would be useful for beginners to practise efficient recursive queries (more so than the StackExchange dumps, for instance).
Apparently the comment counts are only the top-level comments? It would be nice for the thread pages to show a comment count.
Don't miss how this works. It's not a server-side application - this code runs entirely in your browser using SQLite compiled to WASM, but rather than fetching a full 22GB database it instead uses a clever hack that retrieves just "shards" of the SQLite database needed for the page you are viewing. I watched it in the browser network panel and saw it fetch: https://hackerbook.dosaygo.com/static-shards/shard_1636.sqli…
Is there anything more production grade built around the same idea of HTTP range requests like that sqlite thing? This has so much potential
Wonder if you could turn this into a .zim file for offline browsing with an offline browser like Kiwix, etc. [0] I've been taking frequent "offline-only-day" breaks to consolidate whatever I've been learning, and Kiwix has been a great tool for reference (offline Wikipedia, StackOverflow and whatnot). [0] https://kiwix.org/en/the-new-kiwix-library-is-available/