Live data from Hacker News

Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

huggingface.co

41–50 of 178 posts

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#41

is the legal page a placeholder, do words have no meaning? https://www.ycombinator.com/legal/ Mods, enforce your license terms, you're playing fast and loose with the law (GDPR/CPRA)

They already refuse to comply with CPRA, instead electing to replace your username with a random 6(?) character string, prefixed with `_`, if I remember correctly. I know, because I've been here since maybe 2015 or so, but this account was created in 2019. So any PII you have mentioned in your comments is permanent on Hacker News. I would appreciate it if they gave users the ability to remove all of their personal da…

[dead]

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#42
As someone who had made a project analysing hackernews who had used clickhouse, I really feel like this is a project made for me (especially the updated every 5 minute aspect which could've helped my project back then too!)

Your project actually helps me out a ton in making one of the new project ideas that I had about hackernews that I had put into the back-burner.

I had thought of making a ping website where people can just @Username and a service which can detect it and then send mail to said username if the username has signed up to the service (similar to a service run by someone from HN community which mails you everytime someone responds to your thread directly, but this time in a sort of ping)

[The previous idea came as I tried to ping someone to show them something relevant and thought that wait a minute, something like ping which mails might be interesting and then tried to see if I can use algolia or any service to hook things up but not many/any service made much sense back then sadly so I had the idea in back of my mind but this service sort of solves it by having it being updated every 5 minutes]

Your 5 minute updates really make it possible. I will look what I can do with that in some days but I am seeing some discrepancy in the 5 minute update as last seems to be 16 march in the readme so I would love to know more about if its being updated every 5 minutes because it truly feels phenomenal if true and its exciting to think of some new possibilities unlocked with it.

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#43
post #20

Static web content and dynamic data? > The archive currently spans from 2006-10 to 2026-03-16 23:55 UTC, with 47,358,772 items committed. That’s more than 5 minutes ago by a day or two. No big deal, but a little bit depressing this is still how we do things in 2026.

I don't get what you meant with this comment.

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#45
post #5

> At midnight UTC, the entire current month is refetched from the source as a single authoritative Parquet file, and today's individual 5-minute blocks are removed from the today/ directory. Wouldn't that lose deleted/moderated comments?

I guess that's the point.

Can't someone create an automatic script which can just copy the files say 5 minutes before midnight UTC?

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#46
post #20

Static web content and dynamic data? > The archive currently spans from 2006-10 to 2026-03-16 23:55 UTC, with 47,358,772 items committed. That’s more than 5 minutes ago by a day or two. No big deal, but a little bit depressing this is still how we do things in 2026.

I don't get what you meant with this comment.

the data updates every 5 minutes, but the description on huggingface says the last update was 2 days ago.

they are suggesting that the huggingface description should be automatically updating the date & item count when the data gets updated.

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#47
post #20

Static web content and dynamic data? > The archive currently spans from 2006-10 to 2026-03-16 23:55 UTC, with 47,358,772 items committed. That’s more than 5 minutes ago by a day or two. No big deal, but a little bit depressing this is still how we do things in 2026.

That is just the archive part, if you just would finish reading the paragraph you would know that updates since 2026-03-16 23:55 UTC are "are fetched every 5 minutes and committed directly as individual Parquet files through an automated live pipeline, so the dataset stays current with the site itself."

So to get all the data you need to grab the archive and all the 5 minute update files.

archive data is here https://huggingface.co/datasets/open-index/hacker-news/tree/...

update files are here (I know that its called "today" but it actually includes all the update files which span multiple days at this point) https://huggingface.co/datasets/open-index/hacker-news/tree/...

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#48

Earlier quoted context omitted.

I don't get what you meant with this comment.

the data updates every 5 minutes, but the description on huggingface says the last update was 2 days ago. they are suggesting that the huggingface description should be automatically updating the date & item count when the data gets updated.

No that is the date at which the bulk archive ends and the 5 minute update files begin, so it should not be updated.

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#49
post #21

is the legal page a placeholder, do words have no meaning? https://www.ycombinator.com/legal/ Mods, enforce your license terms, you're playing fast and loose with the law (GDPR/CPRA)

Eh, fuck that agreement. I'm kind of old school in that I believe if you put it on the internet without an auth-wall, people should be allowed to do whatever they want with it. The AI companies seem to agree. Then again, I'm not the guy that is going to get sued...

[flagged]

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#50
post #47
post #20

Static web content and dynamic data? > The archive currently spans from 2006-10 to 2026-03-16 23:55 UTC, with 47,358,772 items committed. That’s more than 5 minutes ago by a day or two. No big deal, but a little bit depressing this is still how we do things in 2026.

That is just the archive part, if you just would finish reading the paragraph you would know that updates since 2026-03-16 23:55 UTC are "are fetched every 5 minutes and committed directly as individual Parquet files through an automated live pipeline, so the dataset stays current with the site itself." So to get all the data you need to grab the archive and all the 5 minute update files. archive data is here https:/…

>if you just would finish reading the paragraph

probably uncalled for

Post reply on HN