Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m • Hacker News Board

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#51

Post by [deleted] » Wed, Mar 18, 2026, 7:34 PM UTC

voxic11 wrote: post #48

john_strinlai wrote: post #46

Earlier quoted context omitted.

the data updates every 5 minutes, but the description on huggingface says the last update was 2 days ago. they are suggesting that the huggingface description should be automatically updating the date & item count when the data gets updated.

No that is the date at which the bulk archive ends and the 5 minute update files begin, so it should not be updated.

[deleted]

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#52

Post by robotswantdata » Wed, Mar 18, 2026, 7:35 PM UTC

Where’s the opt out ?

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#53

Post by john_strinlai » Wed, Mar 18, 2026, 7:38 PM UTC

robotswantdata wrote: post #52

Where’s the opt out ?

hackernews is very upfront that they do not really care about deletion requests or anything of that sort, so, the opt out is to not use hackernews.

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#54

Post by imhoguy » Wed, Mar 18, 2026, 7:41 PM UTC

alstonite wrote: post #26

What happened between 2023 and 2024 to cause the usage dropoff?

Return to office

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#55

Post by brtkwr » Wed, Mar 18, 2026, 7:44 PM UTC

This comment should make it into the download in a few mins.

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#56

Post by waynesonfire » Wed, Mar 18, 2026, 7:44 PM UTC

lokimoon wrote: post #32

[dead]

Your reward is the endorphin hit from writing this comment.

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#57

Post by vovavili » Wed, Mar 18, 2026, 7:45 PM UTC

Replacing an 11.6GB Parquet file every 5 minutes strikes me as a bit wasteful. I would probably use Apache Iceberg here.

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#58

Post by tantalor » Wed, Mar 18, 2026, 7:45 PM UTC

brtkwr wrote: post #55

This comment should make it into the download in a few mins.

As should this reply

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#59

Post by tantalor » Wed, Mar 18, 2026, 7:46 PM UTC

robotswantdata wrote: post #52

Where’s the opt out ?

The back button

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#60

Post by fabmilo » Wed, Mar 18, 2026, 7:47 PM UTC

vovavili wrote: post #57

Replacing an 11.6GB Parquet file every 5 minutes strikes me as a bit wasteful. I would probably use Apache Iceberg here.

Was thinking the same thing. probably once a day would be more than enough. if you really want a minute by minute probably a delta file from the previous day should be more than enough.