The best source for this data used to be Clickhouse (https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0a...), but it hasn't updated since 2025-12-26.
Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
11–20 of 178 posts
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#12My Hacker News items table in ClickHouse has 47,428,860 items, and it's 5.82 GB compressed and 18.18 GB uncompressed. What makes Parquet compression worse here, when both formats are columnar?
Parquet has a few compression option. Not sure which one they are using.
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#13is the legal page a placeholder, do words have no meaning? https://www.ycombinator.com/legal/ Mods, enforce your license terms, you're playing fast and loose with the law (GDPR/CPRA)
How is is he breaking gdpr here?
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#14My Hacker News items table in ClickHouse has 47,428,860 items, and it's 5.82 GB compressed and 18.18 GB uncompressed. What makes Parquet compression worse here, when both formats are columnar?
Sorting, compression algorithm +level, and data types can all have an impact. I noted elsewhere that a Boolean is getting represented as an integer. That’s one bit vs 1-4 bytes.
There is also flexibility in what you define as the dataset. Skinnier, but more focused tables could be space saving vs a wide table that covers everything -will probably break compressible runs of data.
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#15is the legal page a placeholder, do words have no meaning? https://www.ycombinator.com/legal/ Mods, enforce your license terms, you're playing fast and loose with the law (GDPR/CPRA)
Which terms are not being enforced? (not disagreeing I just don't feel like reading a large legal document)
None that I could see:
Your submissions to, and comments you make on, the Hacker News site are not Personal Information and are not "HN Information" as defined in this Privacy Policy.
Other Users: certain actions you take may be visible to other users of the Services.
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#16is the legal page a placeholder, do words have no meaning? https://www.ycombinator.com/legal/ Mods, enforce your license terms, you're playing fast and loose with the law (GDPR/CPRA)
Which terms are not being enforced? (not disagreeing I just don't feel like reading a large legal document)
> By uploading any User Content you hereby grant and will grant Y Combinator and its affiliated companies
The user content is supposed to be licensed only Y Combinator and (bleah) its affiliated companies (which are many, all the startups they fund, for example).
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#17Earlier quoted context omitted.
Which terms are not being enforced? (not disagreeing I just don't feel like reading a large legal document)
None that I could see: Your submissions to, and comments you make on, the Hacker News site are not Personal Information and are not "HN Information" as defined in this Privacy Policy. Other Users: certain actions you take may be visible to other users of the Services.
I mean, just because they say the comments are not PI doesn't make it so.
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#18My Hacker News items table in ClickHouse has 47,428,860 items, and it's 5.82 GB compressed and 18.18 GB uncompressed. What makes Parquet compression worse here, when both formats are columnar?
Parquet has a few compression option. Not sure which one they are using.
Plus isn't the least wasteful format, native duckdb for instance compacts better. That's not just down to the compression algorithm, which as you say got three main options for parquet.
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#19Earlier quoted context omitted.
None that I could see: Your submissions to, and comments you make on, the Hacker News site are not Personal Information and are not "HN Information" as defined in this Privacy Policy. Other Users: certain actions you take may be visible to other users of the Services.
I mean, just because they say the comments are not PI doesn't make it so.
That’s a good point. I’m only referring to the terms they used in the privacy policy.
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#20Static web content and dynamic data?
> The archive currently spans from 2006-10 to 2026-03-16 23:55 UTC, with 47,358,772 items committed.
That’s more than 5 minutes ago by a day or two. No big deal, but a little bit depressing this is still how we do things in 2026.