This is mostly why I left Reddit. The API allows far too much control and I started questioning what was even real. Being able to quickly find keywords and then have a network of bots that creates replies/upvotes/downvotes is very disturbing thought to me. I can't even imagine something like that on a large scale to change opinions.
How F5Bot Slurps All of Reddit
31–40 of 91 posts
Re: How F5Bot Slurps All of Reddit
#32"Turns out that Reddit [API] has a limit. It'll only show you 100 posts at a time."
100 sounds like a typical "max-requests" pipelining limit.
He does not mention CURLMOPT_PIPELINING.
Does this mean he makes 100 TCP connections in order to make 100 HTTP requests?
Re: How F5Bot Slurps All of Reddit
#33It is difficult for me to describe just how angry it makes me that reddit doesn't provide a way for users to even do basic things like "see all of my own comments" or "see all of the posts made to the subreddit I moderate". They keep nerfing the search APIs and claim it is so they could make the indexes more efficient, but while that might make sense for a full-text search interface, that is entirely unreasonable for…
Their API is so fucked that it is literally impossible to get more than a couple hundred posts from a subreddit.
Thankfully, services like pushshift[1] exist, which has a sane API and the option to use plain elasticsearch.
Re: How F5Bot Slurps All of Reddit
#34It is difficult for me to describe just how angry it makes me that reddit doesn't provide a way for users to even do basic things like "see all of my own comments" or "see all of the posts made to the subreddit I moderate". They keep nerfing the search APIs and claim it is so they could make the indexes more efficient, but while that might make sense for a full-text search interface, that is entirely unreasonable for…
Reddit doesn't even allow users to save more than 1000 posts, and worse does not visibly document this or provide any kind of warning that the limit has been exceeded. Anecdotally, I've read users say that revisiting the saved pages will still show an "unsave" button so the information is recorded somewhere. But once a user exceeds 1000 entries on their "saved" page, adding new ones will silently vaporize old ones. h…
It's a bit weirder than that. It actually does save all the posts, but the "saved" page (like almost every other page on the site) will only show you 1000 items, so there's just no way to access all the older items once they've been "pushed off the end".
I posted some more information about it a while ago here: https://www.reddit.com/r/help/comments/7en0uu/my_saved_posts...
Re: How F5Bot Slurps All of Reddit
#35Earlier quoted context omitted.
Except that the data stream is, I should hope, compressed so that the data is actually packed into a format optimized for size.
Not necessarily, and even so not for free. (Client must specify compression support)
Most clients support it implicitly; you probably have to go out of your way to get an uncompressed stream. Now compressing a verbose text string is not optimal but given the past attempts I'd hesitate against using a pre-packed format. Historically that has not worked out well. Compressing the text format is ultimately the worse-is-better solution.
Re: How F5Bot Slurps All of Reddit
#36Earlier quoted context omitted.
An interpreted language is slower than a compiled binary? Color me shocked.
> An interpreted language is slower than a compiled binary? Color me shocked. That same benchmark shows JS code that’s 6x faster than PHP.
Yes, but JS is faster than Ruby and Python, and indeed most of your go-to scripting options.
Re: How F5Bot Slurps All of Reddit
#37It is difficult for me to describe just how angry it makes me that reddit doesn't provide a way for users to even do basic things like "see all of my own comments" or "see all of the posts made to the subreddit I moderate". They keep nerfing the search APIs and claim it is so they could make the indexes more efficient, but while that might make sense for a full-text search interface, that is entirely unreasonable for…
(I worked at reddit)
It's not really related to search, most of the cause is a pretty bizarre optimization method that reddit decided to implement fairly early on. The database structure is unusual and not very conducive to indexing (it's similar to an EAV model), so at some point they decided to basically write their own "secondary indexing"-like system that stores the "listing indexes" in memcached/Cassandra (with the data itself in PostgreSQL).
Whenever something happens that affects any listings (new post created, voting, etc), the site figures out all the listings it needs to update, and where in each listing the affected post now belongs and updates them all. So for example, if you make a new submission to /r/pics, it will go through and add the post's ID in the right spot to the "new posts in /r/pics" listing, the "hot posts in /r/pics" listing, the "new posts by yourusername" listing, and so on. As it's going through and updating all these lists, it also trims each one down to 1000 items.
It's conceptually pretty similar to a normal database indexing system, but basically maintains all the indexes "manually" and restricts them all to the top 1000 items.
If you're curious enough to dig around in the code, this is probably the main relevant file: https://github.com/reddit-archive/reddit/blob/master/r2/r2/l...
Re: How F5Bot Slurps All of Reddit
#38Earlier quoted context omitted.
Not necessarily, and even so not for free. (Client must specify compression support)
Most clients support it implicitly; you probably have to go out of your way to get an uncompressed stream. Now compressing a verbose text string is not optimal but given the past attempts I'd hesitate against using a pre-packed format. Historically that has not worked out well. Compressing the text format is ultimately the worse-is-better solution.
They are using libcurl, for which you need to request compression explicitly:
Re: How F5Bot Slurps All of Reddit
#39"Turns out that Reddit [API] has a limit. It'll only show you 100 posts at a time." 100 sounds like a typical "max-requests" pipelining limit. He does not mention CURLMOPT_PIPELINING. Does this mean he makes 100 TCP connections in order to make 100 HTTP requests?
The 100 has nothing to do with HTTP pipelining, it's just a standard REST style "&limit=100" hard limit
Re: How F5Bot Slurps All of Reddit
#40Earlier quoted context omitted.
PHP used to be very slow, it got better with v7.0. It's still quite slow compared to C/C++/Rust/Go, more than 10x slower: https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
An interpreted language is slower than a compiled binary? Color me shocked.
PHP is JIT-compiled, which helps.