Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

huggingface.co

Post reply on HN
Print view View original

71–80 of 178 posts

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#71

Post by tomrod »

zerocrates wrote: post #66
vovavili wrote: post #57

Replacing an 11.6GB Parquet file every 5 minutes strikes me as a bit wasteful. I would probably use Apache Iceberg here.

"The dataset is organized as one Parquet file per calendar month, plus 5-minute live files for today's activity. Every 5 minutes, new items are fetched from the source and committed directly as a single Parquet block. At midnight UTC, the entire current month is refetched from the source as a single authoritative Parquet file, and today's individual 5-minute blocks are removed from the today/ directory." So it's not…

Parquet is a very efficient storage approach. Data interfaces tend to treat paths as partitions, if logical.

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#72

Post by politician »

This is great. I've soured on this site over the past few years due to the heavy partisanship that wasn't as present in the early days (eternal September), but there are still quite a few people whose opinions remain thought-provoking and insightful. I'm going to use this corpus to make a local self-hosted version of HN with the ability to a) show inline article summaries and b) follow those folks.

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#73

Post by socksy »

fdghrtbrt wrote: post #68
ratg13 wrote: post #64

Earlier quoted context omitted.

Create a new account every so often, don’t leave any identifying information, occasionally switch up the way you spell words (British/US English), and alternate using different slang words and shorthand.

And do what I do - paste everything into ChatGPT and have it rephrase it. Not because I need help writing, but because I’d rather not have my writing style used against me.

I can't stand this and will actively discriminate against comments I notice in that voice. Even this one has "Not because [..], but because [..]"

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#74

Post by btown »

ai-inquisitor wrote: post #65
vovavili wrote: post #57

Replacing an 11.6GB Parquet file every 5 minutes strikes me as a bit wasteful. I would probably use Apache Iceberg here.

It's not doing that. If you look at the repository, it's adding a new commit with tiny parquet files every 5 minutes. This recent one only was a 20.9 KB parquet file: https://huggingface.co/datasets/open-index/hacker-news/commi... and the ones before it were a median of 5 KB: https://huggingface.co/datasets/open-index/hacker-news/tree/... The bigger concern is how large the git history is going to get on the reposito…

I recall that this became a big problem for the Homebrew project in terms of load on the repo, to the extent that Github asked them not to recommend/default-enable shallow clones for their users: https://github.com/Homebrew/brew/issues/15497#issuecomment-1...

This is likely to be lower traffic, and the history should (?) scale only linearly with new data, so likely not the worst thing. But it's something to be cognizant of when using SCM software in unexpected ways!

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#76

Post by BowBun »

tonymet wrote: post #31

what's the license for HN content?

We have LLMs and links to TOS, this is easily answerable by _anyone_ on the internet at this point.

Comments+posts are defined as user generated content, you have no right to its privacy/control in any capacity once you post it - https://www.ycombinator.com/legal/

YC in theory has the right to go after unauthorized 3rd parties scraping this data. YC funds startups and is deeply vested in the AI space. Why on Earth would they do that.

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#77

Post by fatty_patty89 »

john_strinlai wrote: post #50
voxic11 wrote: post #47

Earlier quoted context omitted.

That is just the archive part, if you just would finish reading the paragraph you would know that updates since 2026-03-16 23:55 UTC are "are fetched every 5 minutes and committed directly as individual Parquet files through an automated live pipeline, so the dataset stays current with the site itself." So to get all the data you need to grab the archive and all the 5 minute update files. archive data is here https:/…

> if you just would finish reading the paragraph probably uncalled for

not really since original comment completely missed it

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#79

Post by cyanydeez »

tomrod wrote: post #70
vovavili wrote: post #69

Earlier quoted context omitted.

This makes more sense. I still wonder if the author isn't just effectively recreating Apache Iceberg manually here.

Are they paying for the repo space, I wonder?

someones paying to keep name dropping Iceberg(tm)

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#80

Post by Bewelge »

zamadatix wrote: post #24
GeoAtreides wrote: post #16

Earlier quoted context omitted.

> By uploading any User Content you hereby grant and will grant Y Combinator and its affiliated companies The user content is supposed to be licensed only Y Combinator and (bleah) its affiliated companies (which are many, all the startups they fund, for example).

If you carry on the quote two more words: > ... a nonexclusive I.e. this section is talking to additional rights to the content you post to ALSO go to YC, not that YC is guaranteeing it (+friends) will be the only one to hold these rights or will enforce who else should hold the rights to your publicly shared content for you. There's a more intricate conversation to be had with GDPR and public data on forums in gener…

I think that's incorrect. Exclusivity would be something you grant to YC. These terms need to make sense to be valid. Claiming exclusive rights would mean they are forbidding YOU from licensing YOUR rights to anyone else.

Imagine Facebook claiming that by uploading images you are granting them exclusive usage rights to that image. It would mean you couldn't upload it to any other site with similar terms anymore.

Post reply on HN