Good for demo but every 5 minutes? Why?
It can have some good use cases I can think of. Personally I really appreciate the 5 minute update.
5 minutes is the sweet spot for many (enterprise) data pipelines too, in my experience.
Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
161–170 of 178 posts
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#162Static web content and dynamic data? > The archive currently spans from 2006-10 to 2026-03-16 23:55 UTC, with 47,358,772 items committed. That’s more than 5 minutes ago by a day or two. No big deal, but a little bit depressing this is still how we do things in 2026.
That’s a silly bug in the "dynamic" README, fixing it now.
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#163From YC /legal > Except as expressly authorized by Y Combinator, you agree not to modify, copy, frame, scrape, rent, lease, loan, sell, distribute or create derivative works based on the Site or the Site Content, in whole or in part Not to pretend this isn't widely happening behind the curtains already, but coming from a "Show HN" seems daring.
This site offers a public, non rate-limted API. IANAL but I'm reasonably certain that's authorization for anyone to use the data as long as they do so through the API. It certainly isn't the case that you need explicit legal permission to use Hacker News comment data in your project.
There have been tons of alternative frontends and projects using HN data over the years, posted to Show HN without an issue. I think their primary concern is interfering with the YCombinator brand itself. "the site" and "site content" referring to YCombinator and not HN specifically.
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#164Earlier quoted context omitted.
Unless I'm missing something, this uses a simple synchronous for loop: for text in texts: key = (text, model) if key not in pickle_cache: pickle_cache[key] = openai_client.create_embedding(text, model=model) embeddings.append(pickle_cache[key]) operations.save_pickle_cache(pickle_cache, pickle_path) return embeddings At the throughput rates I was seeing of one embedding per second, a million comments would take over…
Hey, idk if that helps but I developed something similar to the wrapper you're mentioning as an open-source python library. Just plug any async function into the provided async context manager and you get Batch APIs in two lines of code with any existing framework you currently have: https://github.com/vienneraphael/batchling Let me know if you have any questions, looking forward to having your feedback!
Looks very nice! This is exactly what I was thinking of doing, except that I work mostly with C# in enterprise settings.
Looking at your approach, the equivalent in .NET land would be if the Microsoft.AI.Extensions package added some sort of batch abstraction side-by-side (or on top of) their existing IChatClient or IEmbeddingGenerator interfaces.
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#165Earlier quoted context omitted.
If you carry on the quote two more words: > ... a nonexclusive I.e. this section is talking to additional rights to the content you post to ALSO go to YC, not that YC is guaranteeing it (+friends) will be the only one to hold these rights or will enforce who else should hold the rights to your publicly shared content for you. There's a more intricate conversation to be had with GDPR and public data on forums in gener…
I think that's incorrect. Exclusivity would be something you grant to YC. These terms need to make sense to be valid. Claiming exclusive rights would mean they are forbidding YOU from licensing YOUR rights to anyone else. Imagine Facebook claiming that by uploading images you are granting them exclusive usage rights to that image. It would mean you couldn't upload it to any other site with similar terms anymore.
Yes, this is what I mean in the above - the rights are non-exclusive so YC is also granted rights but not in a way that any of those other things listed after are true.
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#166Earlier quoted context omitted.
And do what I do - paste everything into ChatGPT and have it rephrase it. Not because I need help writing, but because I’d rather not have my writing style used against me.
I can't stand this and will actively discriminate against comments I notice in that voice. Even this one has "Not because [..], but because [..]"
The good rephrasing will not include that voice.
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#167[Author here] The whole pipeline runs on a single ~$10/month VPS, but it can process hundreds of TB even with just 12GB RAM and a 200GB SSD. The main reason I built this was to have HN data that is easy to query and always up to date, without needing to run your own pipeline first. There are also some interesting ideas in the pipeline, like what I call "auto-heal". Happy to share more if anyone is interested :) A lot…
Where are you getting a ~$10/month VPS with 12GB RAM from?
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#168Earlier quoted context omitted.
I did a show hn a month or so back like this: https://hackerbook.dosaygo.com/ https://news.ycombinator.com/item?id=46435308 https://github.com/DOSAYGO-STUDIO/HackerBook The mods and community had no problem with it Differences: Sharded SQLITE, used bigquery export, build script is open on GitHub, interactive “archived website” view of HN, updated weekly (each build takes a couple dollars on a custom GitHub runner)
@keepamovin thanks, your project was a big inspiration for this. I built my own pipeline with a slightly different setup. I use Go to download and process the data, and update it every 5 minutes using the HN API, trying to stay within fair use. It is also easy to tweak if someone wants faster or slower updates. One part I really like is the "dynamic" README on Hugging Face. It is generated automatically by the code a…
Wow tamnd that is lovely to hear. I’m so glad you told me it was an inspiration.
Your big download plus quick refreshes is smart. Is your Background in data/AI?
Because i don’t know much about huggingface beyond its a hub for that.
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#169[Author here] The whole pipeline runs on a single ~$10/month VPS, but it can process hundreds of TB even with just 12GB RAM and a 200GB SSD. The main reason I built this was to have HN data that is easy to query and always up to date, without needing to run your own pipeline first. There are also some interesting ideas in the pipeline, like what I call "auto-heal". Happy to share more if anyone is interested :) A lot…
Where are you getting a ~$10/month VPS with 12GB RAM from?
https://contabo.com/en/vps/cloud-vps-20/ - $8/month, 6 vCPU, 12 GB RAM, 200 GB SSD (or Hetzner servers, which offer good hourly pricing).
In my ongoing project, with 10 servers like this, I could index the large part internet (about 10 billion pages) using vector and full-text search.
Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
#170Earlier quoted context omitted.
@keepamovin thanks, your project was a big inspiration for this. I built my own pipeline with a slightly different setup. I use Go to download and process the data, and update it every 5 minutes using the HN API, trying to stay within fair use. It is also easy to tweak if someone wants faster or slower updates. One part I really like is the "dynamic" README on Hugging Face. It is generated automatically by the code a…
Wow tamnd that is lovely to hear. I’m so glad you told me it was an inspiration. Your big download plus quick refreshes is smart. Is your Background in data/AI? Because i don’t know much about huggingface beyond its a hub for that.
Connecting directly with the author of the project that inspired me is awesome.
Let's collaborate and see how we can make our two projects work together. DuckDB has a feature that can write to SQLite: https://duckdb.org/docs/stable/core_extensions/sqlite. Starting from Parquet files, we could use DuckDB to write into SQLite databases. This could reduce ingress time to around five minutes instead of a week.
If I have some free time this weekend, I would definitely like to contribute to your project. Would you be interested?
As for my background, I focus on data engineering and data architecture. I help clients build very large-scale data pipelines, ranging from near real-time systems (under 10 ms) to large batch processing systems (handling up to 1 billion business transactions per day across thousands of partners). Some of these systems use mathematical models I developed, particularly in graph theory.
Happy to chat.