Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

Post by Wowfunhappy » Thu, Mar 19, 2026, 2:36 AM UTC

stackghost wrote: post #137

Wowfunhappy wrote: post #136

Earlier quoted context omitted.

I agree with this policy, deleting comments isn't fair to all the other people who replied to that comment. I don't see how this goes against what I said?

I was responding to your statement that you don't like that your comments are the property of YC. I was elaborating on how they hold our content (that we author) hostage because it looks pretty. Not wanting your comments to be property of YC but then also being okay with them refusing to delete your content doesn't make sense to me. Those seem like fundamentally-opposed viewpoints. Now I'm thinking about it, I wonder…

Post by pjot » Thu, Mar 19, 2026, 2:56 AM UTC

jiggawatts wrote: post #140

pjot wrote: post #135

Earlier quoted context omitted.

I did this but used duckdb as the vector store. Works really well, quite fast too. https://github.com/patricktrainer/duckdb-embedding-search

Unless I'm missing something, this uses a simple synchronous for loop: for text in texts: key = (text, model) if key not in pickle_cache: pickle_cache[key] = openai_client.create_embedding(text, model=model) embeddings.append(pickle_cache[key]) operations.save_pickle_cache(pickle_cache, pickle_path) return embeddings At the throughput rates I was seeing of one embedding per second, a million comments would take over…

Post by politician » Thu, Mar 19, 2026, 3:25 AM UTC

neom wrote: post #131

politician wrote: post #72

This is great. I've soured on this site over the past few years due to the heavy partisanship that wasn't as present in the early days (eternal September), but there are still quite a few people whose opinions remain thought-provoking and insightful. I'm going to use this corpus to make a local self-hosted version of HN with the ability to a) show inline article summaries and b) follow those folks.

"heavy partisanship" - I've seen this claim a few times and I find it a bit odd. Certainly I feel HN leans left, but I've never seen what I would consider a strong preference for any particular political party? When the American daggers do come out - it seems fairly split? Even the post about the Canadian meta data law the other day, left leaning maybe, but I see when partisan comments came out directly, it looked ab…

Post by vienneraphael » Thu, Mar 19, 2026, 4:05 AM UTC

jiggawatts wrote: post #140

pjot wrote: post #135

Earlier quoted context omitted.

I did this but used duckdb as the vector store. Works really well, quite fast too. https://github.com/patricktrainer/duckdb-embedding-search

Unless I'm missing something, this uses a simple synchronous for loop: for text in texts: key = (text, model) if key not in pickle_cache: pickle_cache[key] = openai_client.create_embedding(text, model=model) embeddings.append(pickle_cache[key]) operations.save_pickle_cache(pickle_cache, pickle_path) return embeddings At the throughput rates I was seeing of one embedding per second, a million comments would take over…

Post by neom » Thu, Mar 19, 2026, 4:37 AM UTC

politician wrote: post #145

neom wrote: post #131

Earlier quoted context omitted.

"heavy partisanship" - I've seen this claim a few times and I find it a bit odd. Certainly I feel HN leans left, but I've never seen what I would consider a strong preference for any particular political party? When the American daggers do come out - it seems fairly split? Even the post about the Canadian meta data law the other day, left leaning maybe, but I see when partisan comments came out directly, it looked ab…

I think we'll be able to quantify sentiment from the data, and I look forward to doing so. There's a few other datasets that I want to look at such as whether there is evidence of participation suppression via rate limiting on a per-profile basis.