Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun • Hacker News Board

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#161

Post by Foreignborn » Fri, May 10, 2024, 10:45 PM UTC

gaauch wrote: post #43

A long term side project of mine is to try to build a recommendation algorithm trained on HN data. I trained a model to predict if a given post will reach the front page, get flagged etc, I collected over a 1000 RSS feeds and rank the RSS entries with my ranking models. I submit the high ranking entries on HN to test out my models and I can reach the front page consistently sometimes having multiple entries on the fr…

Could you explain more about what you mean by modeling interactions between comments and entities?

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#162

Post by stavros » Sat, May 11, 2024, 12:34 AM UTC

b800h wrote: post #107

oersted wrote: post #27

Earlier quoted context omitted.

"Hundreds of dollars" sounds a bit painful as an EU engineer and entrepreneur :), but I guess it's all relative. We would think twice about investing this much manpower and compute for such an exploratory project even in a commercial setting if it was not directly funded by a client. But your technical skill is obvious and very impressive. If you want to read more, my old bachelor's thesis is somewhat related, from w…

As an EU-based engineer, you wouldn't do this, it's a massive GDPR violation (failure to notify data subjects of data processing), which does actually have extraterritoriality, although I somehow doubt that the information commissioners are going to be coming after OP.

Processing comments on a forum being a violation of the GDPR? That's crazy, the OP is neither the data controller (HN is) nor a data processor on behalf of the controller. If you post your data in public, it's not a GDPR violation for people to use it for things.

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#163

Post by carte_blanche » Sat, May 11, 2024, 6:23 AM UTC

Getting "Argo tunnel error" on the page

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#164

Post by wilsonzlin » Sat, May 11, 2024, 6:37 AM UTC

carte_blanche wrote: post #163

Getting "Argo tunnel error" on the page

Thanks for the heads up, just fixed this.

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#165

Post by tarasglek » Sat, May 11, 2024, 8:56 AM UTC

zX41ZdbW wrote: post #62

thyrox wrote: post #2

Very nice. Since Hn data spawns so many such fun projects, there should be a monthly or weekly updates zip file or torrent with this data, which hackers can just download instead of writing a scraper and starting from scratch all the time.

It is very easy to get this dataset directly from HN API. Let me just post it here: Table definition: CREATE TABLE hackernews_history ( update_time DateTime DEFAULT now(), id UInt32, deleted UInt8, type Enum('story' = 1, 'comment' = 2, 'poll' = 3, 'pollopt' = 4, 'job' = 5), by LowCardinality(String), time DateTime, text String, dead UInt8, parent UInt32, poll UInt32, kids Array(UInt32), url String, score Int32, title…

May I hijack this thread for a related q. I love the public up-to-date hn dataset.

I saw recursive cte blog post..but this doesn't seem to work your hn dataset

https://play.clickhouse.com/play?user=play#V0lUSCBSRUNVUlNJV...

Are recursive ctes disabled on this instance or am i doing something wrong?

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#166

Post by zX41ZdbW » Sun, May 12, 2024, 12:48 AM UTC

tarasglek wrote: post #165

zX41ZdbW wrote: post #62

Earlier quoted context omitted.

It is very easy to get this dataset directly from HN API. Let me just post it here: Table definition: CREATE TABLE hackernews_history ( update_time DateTime DEFAULT now(), id UInt32, deleted UInt8, type Enum('story' = 1, 'comment' = 2, 'poll' = 3, 'pollopt' = 4, 'job' = 5), by LowCardinality(String), time DateTime, text String, dead UInt8, parent UInt32, poll UInt32, kids Array(UInt32), url String, score Int32, title…

May I hijack this thread for a related q. I love the public up-to-date hn dataset. I saw recursive cte blog post..but this doesn't seem to work your hn dataset https://play.clickhouse.com/play?user=play#V0lUSCBSRUNVUlNJV... Are recursive ctes disabled on this instance or am i doing something wrong?

This is unclear to me, I will ask the author.

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#167

Post by stavros » Sun, May 12, 2024, 10:45 AM UTC

This search engine is amazing. I was looking for an old story about curing acid reflux by some exercise, Google/DDG/Kagi/HN's Algolia were completely useless, this found it first hit. Well done, this is the HN search engine I've always wanted.

Is it possible to keep it up to date?

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#168

Post by madacol » Mon, May 13, 2024, 7:04 AM UTC

Thorrez wrote: post #137

rantymcrant wrote: post #136

Earlier quoted context omitted.

> However, the "Show HN" phrase is only used for self promotion I think, so even without the "I", anyone familiar with the convention will know it's self promotion. I think that's an extremely cynical view though a common one. I've never thought of "Show HN" as self promotion if it doesn't include "I" unless I go through to the actual product/library/post and find it full of self promotion. I agree with you that a po…

>Show HN is for sharing your personal work and has special rules. https://news.ycombinator.com/newsfaq.html

Interestingly, the special rules are more in favor of the more "self-promotionals" variants

> On topic: things people can run on their computers or hold in their hands. For hardware, you can post a video or detailed article. For books, a sample chapter is ok.

> Off topic: blog posts, sign-up pages, newsletters, lists, and other reading material. Those can't be tried out, so can't be Show HNs. Make a regular submission instead.

https://news.ycombinator.com/showhn.html

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#169

Post by zX41ZdbW » Tue, May 14, 2024, 4:09 PM UTC

zX41ZdbW wrote: post #166

tarasglek wrote: post #165

Earlier quoted context omitted.

May I hijack this thread for a related q. I love the public up-to-date hn dataset. I saw recursive cte blog post..but this doesn't seem to work your hn dataset https://play.clickhouse.com/play?user=play#V0lUSCBSRUNVUlNJV... Are recursive ctes disabled on this instance or am i doing something wrong?

This is unclear to me, I will ask the author.

The reason is trivial - I disabled the new feature flag on the playground service long ago (when it was in development). I will enable it back and send an example.

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#170

Post by zX41ZdbW » Tue, May 14, 2024, 4:17 PM UTC

tarasglek wrote: post #165

zX41ZdbW wrote: post #62

Earlier quoted context omitted.

It is very easy to get this dataset directly from HN API. Let me just post it here: Table definition: CREATE TABLE hackernews_history ( update_time DateTime DEFAULT now(), id UInt32, deleted UInt8, type Enum('story' = 1, 'comment' = 2, 'poll' = 3, 'pollopt' = 4, 'job' = 5), by LowCardinality(String), time DateTime, text String, dead UInt8, parent UInt32, poll UInt32, kids Array(UInt32), url String, score Int32, title…

May I hijack this thread for a related q. I love the public up-to-date hn dataset. I saw recursive cte blog post..but this doesn't seem to work your hn dataset https://play.clickhouse.com/play?user=play#V0lUSCBSRUNVUlNJV... Are recursive ctes disabled on this instance or am i doing something wrong?

Done, and now it works perfectly.