Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
1–10 of 178 posts
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#2Is is possible to only download a subset? e.g. Show HNs or HN Whoishiring. The Show HNs and HN Whoishiring are very useful for classroom data science i.e. a very useful set of data for students to learn the basic of data cleaning and engineering.
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#3Is is possible to only download a subset? e.g. Show HNs or HN Whoishiring. The Show HNs and HN Whoishiring are very useful for classroom data science i.e. a very useful set of data for students to learn the basic of data cleaning and engineering.
It’s date partitioned, you could download just a date range. It’s also parquet, so you can download just specific columns with the right client
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#4what’s the license? “do whatever the fuck you want with the data as long as you don’t get caught”? or does that only work for massive corporations
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#5> At midnight UTC, the entire current month is refetched from the source as a single authoritative Parquet file, and today's individual 5-minute blocks are removed from the today/ directory.
Wouldn't that lose deleted/moderated comments?
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#6is the legal page a placeholder, do words have no meaning?
https://www.ycombinator.com/legal/
Mods, enforce your license terms, you're playing fast and loose with the law (GDPR/CPRA)
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#7is the legal page a placeholder, do words have no meaning? https://www.ycombinator.com/legal/ Mods, enforce your license terms, you're playing fast and loose with the law (GDPR/CPRA)
They already refuse to comply with CPRA, instead electing to replace your username with a random 6(?) character string, prefixed with `_`, if I remember correctly.
I know, because I've been here since maybe 2015 or so, but this account was created in 2019.
So any PII you have mentioned in your comments is permanent on Hacker News.
I would appreciate it if they gave users the ability to remove all of their personal data, but in correspondence and in writing here on Hacker News itself, Dan has suggested that they value the posterity of conversations over the law.
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#8Under the Known Limitations section
deleted and dead are integers. They are stored as 0/1 rather than booleans.
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#9My Hacker News items table in ClickHouse has 47,428,860 items, and it's 5.82 GB compressed and 18.18 GB uncompressed. What makes Parquet compression worse here, when both formats are columnar?
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#10is the legal page a placeholder, do words have no meaning? https://www.ycombinator.com/legal/ Mods, enforce your license terms, you're playing fast and loose with the law (GDPR/CPRA)
Which terms are not being enforced? (not disagreeing I just don't feel like reading a large legal document)