Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

huggingface.co

Post reply on HN
Print view View original

171–178 of 178 posts

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#171

Post by keepamovin »

tamnd wrote: post #170
keepamovin wrote: post #168

Earlier quoted context omitted.

Wow tamnd that is lovely to hear. I’m so glad you told me it was an inspiration. Your big download plus quick refreshes is smart. Is your Background in data/AI? Because i don’t know much about huggingface beyond its a hub for that.

Connecting directly with the author of the project that inspired me is awesome. Let's collaborate and see how we can make our two projects work together. DuckDB has a feature that can write to SQLite: https://duckdb.org/docs/stable/core_extensions/sqlite . Starting from Parquet files, we could use DuckDB to write into SQLite databases. This could reduce ingress time to around five minutes instead of a week. If I have…

One of the things that i got interested in from the comments on my show was parquet. Everyone raving about it. Happy to see a project using that today.

Would be happy to connect more :)

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#172

Post by albedoa »

endofreach wrote: post #121
albedoa wrote: post #109

Earlier quoted context omitted.

By "to do this" do you mean to not use booleans? It's because the value does not represent a binary true or false but rather a means by which the item is deleted or dead. So not only would it not make sense semantically, it would break if a third means were introduced.

> It's because the value does not represent a binary true or false but rather a means by which the item is deleted or dead. "Deleted" and "dead" are separate columns. > So not only would it not make sense semantically, it would break if a third means were introduced. If that was the intention, it would seem like a bad design decision to me. And actually what you assume to be the reasoning, is exactly what should be a…

> "Deleted" and "dead" are separate columns.

Yes, we know!

> If that was the intention, it would seem like a bad design decision to me. And actually what you assume to be the reasoning, is exactly what should be avoided. Which makes it a bad thing.

That is should be avoided is what makes it a bad thing? I am probably on board with the idea that it is bad design, I just don't know what reasoning you are referring to here. How would you design it?

> This is a limitation not because of having the bool value be represented by an int (or rather "be presented as"), but because of the t y p e , being an integer.

What bool value? As designed, it is an int. I'm sure that I am just missing what you are saying.

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#173

Post by YVoyiatzis »

Wowfunhappy wrote: post #133
6thbit wrote: post #128

From YC /legal > Except as expressly authorized by Y Combinator, you agree not to modify, copy, frame, scrape, rent, lease, loan, sell, distribute or create derivative works based on the Site or the Site Content, in whole or in part Not to pretend this isn't widely happening behind the curtains already, but coming from a "Show HN" seems daring.

I can't comment on what is legal, but I very much dislike the idea that my comments are the property of Y Combinator. I assume that by writing here, I am putting information out into the world for anyone to use as they wish.

Your comments, yes. But the contextual thread as a while, no.

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#174

Post by BowBun »

pkilgore wrote: post #118
BowBun wrote: post #78

Earlier quoted context omitted.

By posting comments on this site, you are relinquishing your right to that content. It belongs to YC and it is theirs to enforce, not yours. https://www.ycombinator.com/legal/

Is this legal advice?

Do you want it to be? I think it's safe to assume that most comments are _not_ legal advice.

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#176

Post by keepamovin »

tamnd wrote: post #170
keepamovin wrote: post #168

Earlier quoted context omitted.

Wow tamnd that is lovely to hear. I’m so glad you told me it was an inspiration. Your big download plus quick refreshes is smart. Is your Background in data/AI? Because i don’t know much about huggingface beyond its a hub for that.

Connecting directly with the author of the project that inspired me is awesome. Let's collaborate and see how we can make our two projects work together. DuckDB has a feature that can write to SQLite: https://duckdb.org/docs/stable/core_extensions/sqlite . Starting from Parquet files, we could use DuckDB to write into SQLite databases. This could reduce ingress time to around five minutes instead of a week. If I have…

If you have time I welcome your contribution.

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#177

Post by imhoguy »

sockaddr wrote: post #100
imhoguy wrote: post #62

Yay! So much knowledge in just 11GB. Adding to my end of the World hoarding stash!

Your family is starving and your dog died of radiation poisoning from the fallout but at least your local LLM can browse this and recommend a good software stack for your automated booby traps.

Not really, AI slop flood is also some kind of end of World. It gets harder and harder to hoard pre-2020 content.

Re: Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m

#178

Post by maxloh »

6thbit wrote: post #128

From YC /legal > Except as expressly authorized by Y Combinator, you agree not to modify, copy, frame, scrape, rent, lease, loan, sell, distribute or create derivative works based on the Site or the Site Content, in whole or in part Not to pretend this isn't widely happening behind the curtains already, but coming from a "Show HN" seems daring.

For what it's worth, there is an official, daily updated public dataset of all posts and comments. Therefore, the data clearly isn't something they consider a trade secret.

https://console.cloud.google.com/marketplace/details/y-combi...

Post reply on HN