How F5Bot Slurps All of Reddit
intoli.com
How F5Bot Slurps All of Reddit
1–10 of 91 posts
Re: How F5Bot Slurps All of Reddit
#2Aho-Corasick is really great. It’s a bit complicated to set up, but once you have the modified true set up it’s really fast. By the way,
> Basically I use the selftext, subreddit, permalink, url and title. The other 95% of it is just wasted bandwidth.
It’d probably be better for Reddit if they allowed for specifying the fields we care about rather than just returning the whole thing…
Re: How F5Bot Slurps All of Reddit
#3I made something just like this that worked on forums. Basically any forum that was using the tapatalk plugin (pretty much any busy forum uses it these days) you could subscribe too. It doesn't look like this will handle mispellings of works, or anything like that. I was handling that, however it took a LOT of processing power and quickly realized that the more people used it, the more it wasn't going to scale really well. Good luck with your project.
Re: How F5Bot Slurps All of Reddit
#4edit: cool
Re: How F5Bot Slurps All of Reddit
#5Do the social share buttons literally cover the first few paragraphs of content for anyone else?
Re: How F5Bot Slurps All of Reddit
#6It is difficult for me to describe just how angry it makes me that reddit doesn't provide a way for users to even do basic things like "see all of my own comments" or "see all of the posts made to the subreddit I moderate". They keep nerfing the search APIs and claim it is so they could make the indexes more efficient, but while that might make sense for a full-text search interface, that is entirely unreasonable for basic functionality like "I'm scrolling back through time on my own user page" (where the efficient index is pretty obvious). Both of "see all of the content I posted" and "see all of the content I'm supposedly responsible for" seems like it should be basic, if not required, functionality for any website.
https://www.reddit.com/r/changelog/comments/7tus5f/update_to...
https://www.reddit.com/r/redditdev/comments/7qpn0h/how_to_re...
https://www.reddit.com/r/help/comments/1u0scj/get_full_post_...
Re: How F5Bot Slurps All of Reddit
#7Related: Jason Baumgartner has maintained a Reddit scraping pipeline for a few years now, and wrote up some notes about making it robust: https://pushshift.io
Re: How F5Bot Slurps All of Reddit
#8[deleted]
Re: How F5Bot Slurps All of Reddit
#9edit: cool
> Don't be that guy
She/He's not. One request can supply 100 IDs.
> So here are 2,000 posts, spread out over 20 batches of 100 that we download simultaneously. It assumes you’ve already got the last post ID loaded into $id base-36.
Re: How F5Bot Slurps All of Reddit
#10edit: cool
He's able to ask for multiple posts in one request so the actual request rate is fairly low:
> Here’s how we do it. We find the starting post ID, and then we get posts individually from https://api.reddit.com/api/info.json?id=. We’ll need to add a big list of post IDs to the end of that API URL.