Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How F5Bot Slurps All of Reddit

intoli.com

Post reply on HN
Print view View original

11–20 of 91 posts

Re: How F5Bot Slurps All of Reddit

#11

Post by minimaxir »

allenz wrote: post #7

Related: Jason Baumgartner has maintained a Reddit scraping pipeline for a few years now, and wrote up some notes about making it robust: https://pushshift.io

You can also use the Pushshift real-time feed in BigQuery to query for keywords in submissions in real time (unfortunately the comments feed broke last month)

Example query which searches for 'f5bot' in the past day and correctly finds the corresponding posts on Reddit:

   #standardSQL
   SELECT title, subreddit, permalink
   FROM `pushshift.rt_reddit.submissions`
   WHERE created_utc > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY)
   AND REGEXP_CONTAINS(LOWER(title), r'f5bot')

Re: How F5Bot Slurps All of Reddit

#13

Post by wolco »

"You may think PHP is slow"

Why would we think php is slow? PHP is blazing fast certain applications (looking at you sugarcrm) make this into a mockery by rewriting queries and loading unnecessary data into each page request.

Nice to see a php related show and tell.

Re: How F5Bot Slurps All of Reddit

#14

Post by 333c »

> I mean do you really want subreddit name and subreddit_name_prefixed? They’re the same, one just has an “r/” in front of it.

This is (unfortunately) not quite true. Since Reddit introduced "profile posts," there can be a post where the subreddit name is something like "u_Shitty_Watercolour" but the subreddit_name_prefixed is actually "u/Shitty_Watercolour", rather than "r/u_Shitty_Watercolour".

Example: https://www.reddit.com/user/Shitty_Watercolour/comments/84nh...

Re: How F5Bot Slurps All of Reddit

#15

Post by mustacheemperor »

saurik wrote: post #6

It is difficult for me to describe just how angry it makes me that reddit doesn't provide a way for users to even do basic things like "see all of my own comments" or "see all of the posts made to the subreddit I moderate". They keep nerfing the search APIs and claim it is so they could make the indexes more efficient, but while that might make sense for a full-text search interface, that is entirely unreasonable for…

Reddit doesn't even allow users to save more than 1000 posts, and worse does not visibly document this or provide any kind of warning that the limit has been exceeded. Anecdotally, I've read users say that revisiting the saved pages will still show an "unsave" button so the information is recorded somewhere. But once a user exceeds 1000 entries on their "saved" page, adding new ones will silently vaporize old ones.

https://www.reddit.com/r/help/comments/6nxqjm/maximum_of_100...

Re: How F5Bot Slurps All of Reddit

#16

Post by bufferoverflow »

wolco wrote: post #13

"You may think PHP is slow" Why would we think php is slow? PHP is blazing fast certain applications (looking at you sugarcrm) make this into a mockery by rewriting queries and loading unnecessary data into each page request. Nice to see a php related show and tell.

PHP used to be very slow, it got better with v7.0.

It's still quite slow compared to C/C++/Rust/Go, more than 10x slower:

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Re: How F5Bot Slurps All of Reddit

#17

Post by Puer »

mustacheemperor wrote: post #15
saurik wrote: post #6

It is difficult for me to describe just how angry it makes me that reddit doesn't provide a way for users to even do basic things like "see all of my own comments" or "see all of the posts made to the subreddit I moderate". They keep nerfing the search APIs and claim it is so they could make the indexes more efficient, but while that might make sense for a full-text search interface, that is entirely unreasonable for…

Reddit doesn't even allow users to save more than 1000 posts, and worse does not visibly document this or provide any kind of warning that the limit has been exceeded. Anecdotally, I've read users say that revisiting the saved pages will still show an "unsave" button so the information is recorded somewhere. But once a user exceeds 1000 entries on their "saved" page, adding new ones will silently vaporize old ones. h…

Saving posts is also notoriously unreliable, in my experience. I've had posts where I've clicked the "save" button and it's flipped to "unsave" but upon refreshing the page the button resets as if it was never pressed (and it never gets added.)

Re: How F5Bot Slurps All of Reddit

#18

Post by minimaxir »

saurik wrote: post #6

It is difficult for me to describe just how angry it makes me that reddit doesn't provide a way for users to even do basic things like "see all of my own comments" or "see all of the posts made to the subreddit I moderate". They keep nerfing the search APIs and claim it is so they could make the indexes more efficient, but while that might make sense for a full-text search interface, that is entirely unreasonable for…

The big concern I have is that the endpoints used in the article are pretty old and given that Reddit is now making their site more modern like Facebook and Twitter, they'll just close the endpoints holes like they did.

Re: How F5Bot Slurps All of Reddit

#19

Post by cxseven »

saagarjha wrote: post #2

Aho-Corasick is really great. It’s a bit complicated to set up, but once you have the modified true set up it’s really fast. By the way, > Basically I use the selftext, subreddit, permalink, url and title. The other 95% of it is just wasted bandwidth. It’d probably be better for Reddit if they allowed for specifying the fields we care about rather than just returning the whole thing…

Also, imagine the joules saved worldwide if known key names didn't have to be sent each time, or, better yet, data was packed in a format that optimized for size and processing speed rather than readability.

I mean, I enjoy the idea of human readability as much as Jon Postel, but at certain scales you have to wonder about the hidden cost of petabytes of human-readable data flying over the wire, never to be seen by anything but computers.

Re: How F5Bot Slurps All of Reddit

#20

Post by sieabahlpark »

bufferoverflow wrote: post #16
wolco wrote: post #13

"You may think PHP is slow" Why would we think php is slow? PHP is blazing fast certain applications (looking at you sugarcrm) make this into a mockery by rewriting queries and loading unnecessary data into each page request. Nice to see a php related show and tell.

PHP used to be very slow, it got better with v7.0. It's still quite slow compared to C/C++/Rust/Go, more than 10x slower: https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

An interpreted language is slower than a compiled binary? Color me shocked.
Post reply on HN