Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Subreddit Finder - Trained on 4M Reddit Posts from 4K Subreddits

valohai.com

Post reply on HN
Print view View original

11–20 of 82 posts

Re: Show HN: Subreddit Finder - Trained on 4M Reddit Posts from 4K Subreddits

#13

Post by Arimbr »

toyg wrote: post #8
minimaxir wrote: post #6

You should probably add how exactly you retrieved the 4 million Reddit posts.

Probably with the official Reddit API? There are several libraries for it.

Right! I used the official Reddit API. I created an APP, got the credentials for the API. Then used the Python library PRAW to consume the API. https://praw.readthedocs.io/en/latest/

It took me 36 hours to collect the 4M posts. Reddit API returns results in batches of 100 results, and then sleeps for 2 seconds.

You can find some more details on how it was built here https://blog.valohai.com/machine-learning-pipeline-classifyi...

I can publish on Github the repository that runs two commands to collect the data if your are interested.

Re: Show HN: Subreddit Finder - Trained on 4M Reddit Posts from 4K Subreddits

#18

Post by minimaxir »

dan1234 wrote: post #15
minimaxir wrote: post #6

You should probably add how exactly you retrieved the 4 million Reddit posts.

The official API is well documented. https://www.reddit.com/dev/api/

True, but 4M posts is heavy (even when bypassing the API itself and using unauthenticated requests), so was curious.

Re: Show HN: Subreddit Finder - Trained on 4M Reddit Posts from 4K Subreddits

#19

Post by minimaxir »

Arimbr wrote: post #13
toyg wrote: post #8

Earlier quoted context omitted.

Probably with the official Reddit API? There are several libraries for it.

Right! I used the official Reddit API. I created an APP, got the credentials for the API. Then used the Python library PRAW to consume the API. https://praw.readthedocs.io/en/latest/ It took me 36 hours to collect the 4M posts. Reddit API returns results in batches of 100 results, and then sleeps for 2 seconds. You can find some more details on how it was built here https://blog.valohai.com/machine-learning-pipeline-…

Thanks for the answer!

I was curious since it seems like it was using the BigQuery dataset, but PRAW works too.

Re: Show HN: Subreddit Finder - Trained on 4M Reddit Posts from 4K Subreddits

#20

Post by ChefboyOG »

dehrmann wrote: post #10

Anyone remember /r/reddit.com? That was around the time reddit looked like Hacker News and people were embarrassed to admit they use it.

HN doesn't get enough credit for the tight rope they walk maintaining this community. I see some people post that HN should expand to other topics a la Reddit, but the team does a great job of maintaining focus.

It's not just HN's aesthetic that is minimal and no-nonsense, it's their moderation policies and the tone they set for the community. There is perfect alignment between their approach to content, community, and UX—no fluff, no nonsense, no manipulation, just the simplest, most valuable material possible.

If they expand the scope of acceptable content, it will be really hard not to tweak moderation policies, and eventually you end up with something like reddit, each subreddit might as well be its own (typically under-staffed) site.

EDIT: This is not critical of any post on this thread, just seeing the above comment about old-school reddit got me thinking.

Post reply on HN