Wonder why median votes count is 0, seems every post is getting at least a few votes - maybe this was not the case in the past
Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
61–70 of 178 posts
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#62Yay! So much knowledge in just 11GB. Adding to my end of the World hoarding stash!
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#63Wonder why median votes count is 0, seems every post is getting at least a few votes - maybe this was not the case in the past
Ahhh I get it the moment I asked, there are usually no votes on comments
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#64Where’s the opt out ?
Create a new account every so often, don’t leave any identifying information, occasionally switch up the way you spell words (British/US English), and alternate using different slang words and shorthand.
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#65Replacing an 11.6GB Parquet file every 5 minutes strikes me as a bit wasteful. I would probably use Apache Iceberg here.
It's not doing that. If you look at the repository, it's adding a new commit with tiny parquet files every 5 minutes. This recent one only was a 20.9 KB parquet file: https://huggingface.co/datasets/open-index/hacker-news/commi... and the ones before it were a median of 5 KB: https://huggingface.co/datasets/open-index/hacker-news/tree/...
The bigger concern is how large the git history is going to get on the repository.
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#66Replacing an 11.6GB Parquet file every 5 minutes strikes me as a bit wasteful. I would probably use Apache Iceberg here.
"The dataset is organized as one Parquet file per calendar month, plus 5-minute live files for today's activity. Every 5 minutes, new items are fetched from the source and committed directly as a single Parquet block. At midnight UTC, the entire current month is refetched from the source as a single authoritative Parquet file, and today's individual 5-minute blocks are removed from the today/ directory."
So it's not really one big file getting replaced all the time. Though a less extreme variation of that is happening day to day.
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#67Could you also release the source code behind the automatic update system?
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#68Where’s the opt out ?
Create a new account every so often, don’t leave any identifying information, occasionally switch up the way you spell words (British/US English), and alternate using different slang words and shorthand.
And do what I do - paste everything into ChatGPT and have it rephrase it. Not because I need help writing, but because I’d rather not have my writing style used against me.
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#69Replacing an 11.6GB Parquet file every 5 minutes strikes me as a bit wasteful. I would probably use Apache Iceberg here.
It's not doing that. If you look at the repository, it's adding a new commit with tiny parquet files every 5 minutes. This recent one only was a 20.9 KB parquet file: https://huggingface.co/datasets/open-index/hacker-news/commi... and the ones before it were a median of 5 KB: https://huggingface.co/datasets/open-index/hacker-news/tree/... The bigger concern is how large the git history is going to get on the reposito…
This makes more sense. I still wonder if the author isn't just effectively recreating Apache Iceberg manually here.
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#70Earlier quoted context omitted.
It's not doing that. If you look at the repository, it's adding a new commit with tiny parquet files every 5 minutes. This recent one only was a 20.9 KB parquet file: https://huggingface.co/datasets/open-index/hacker-news/commi... and the ones before it were a median of 5 KB: https://huggingface.co/datasets/open-index/hacker-news/tree/... The bigger concern is how large the git history is going to get on the reposito…
This makes more sense. I still wonder if the author isn't just effectively recreating Apache Iceberg manually here.
Are they paying for the repo space, I wonder?