How F5Bot Slurps All of Reddit • Hacker News Board

Re: How F5Bot Slurps All of Reddit

#71

Post by textmode » Tue, Jul 31, 2018, 5:46 AM UTC

meritt wrote: post #39

textmode wrote: post #32

"Turns out that Reddit [API] has a limit. It'll only show you 100 posts at a time." 100 sounds like a typical "max-requests" pipelining limit. He does not mention CURLMOPT_PIPELINING. Does this mean he makes 100 TCP connections in order to make 100 HTTP requests?

The 100 has nothing to do with HTTP pipelining, it's just a standard REST style "&limit=100" hard limit

You might be right.

With the "&limit" parameter he can change how many items he receives per HTTP request. This has nothing to do with a limit on how many HTTP requests he can make per TCP connection (pipelining). Maybe that is the "100" he is complaining about, i.e., 100 items per HTTP request.

However you failed to answer my question: Is he making 100 TCP connections to make 100 HTTP requests?

Does the Reddit server set a limit on how many HTTP requests he can make per connection? (100 is a common limit for web servers)

Sometimes the server admins may set a limit of 1 HTTP request per TCP connection. This prevents users from pipelining outside the browser, e.g., with libcurl or some other method.

Re: How F5Bot Slurps All of Reddit

#72

Post by Deimorz » Tue, Jul 31, 2018, 5:55 AM UTC

ralfd wrote: post #68

Deimorz wrote: post #55

Earlier quoted context omitted.

Nope, because the old one's already been pushed off the end of the list and it doesn't re-generate the list when you unsave, just removes the item from the list. Imagine I have a list of max length 5, when I initially fill it up it looks like [5, 4, 3, 2, 1]. If I save one more thing, it adds 6 at the front, then truncates the list and removes the 1 from the end, so now you have [6, 5, 4, 3, 2]. At that point, if I u…

What if you unsave 2 items and save 1. Is then the list remade?

No, the list is never remade. The new item just gets inserted into the list, exactly the same as if it had never reached the limit in the first place.

Re: How F5Bot Slurps All of Reddit

#73

Post by _ps6d » Tue, Jul 31, 2018, 5:58 AM UTC

rotred wrote: post #64

Deimorz wrote: post #55

Earlier quoted context omitted.

Nope, because the old one's already been pushed off the end of the list and it doesn't re-generate the list when you unsave, just removes the item from the list. Imagine I have a list of max length 5, when I initially fill it up it looks like [5, 4, 3, 2, 1]. If I save one more thing, it adds 6 at the front, then truncates the list and removes the 1 from the end, so now you have [6, 5, 4, 3, 2]. At that point, if I u…

why would they make the limit so low?

Not sure, the 1000 limit is in the file from its very first version 10 years ago: https://github.com/reddit-archive/reddit/commit/33fd4e9684ca...

1000 probably seemed like a lot at the time or had reasonable performance, and it's just never been changed.

Re: How F5Bot Slurps All of Reddit

#74

Post by anonytrary » Tue, Jul 31, 2018, 6:24 AM UTC

barrkel wrote: post #36

hk__2 wrote: post #28

Earlier quoted context omitted.

> An interpreted language is slower than a compiled binary? Color me shocked. That same benchmark shows JS code that’s 6x faster than PHP.

Yes, but JS is faster than Ruby and Python, and indeed most of your go-to scripting options.

JS has the advantage of being the thing that makes your website run faster on the screen of the user whose clicks are earning you ad dollars. JS is in a good spot.

Re: How F5Bot Slurps All of Reddit

#75

Post by meritt » Tue, Jul 31, 2018, 6:49 AM UTC

textmode wrote: post #71

meritt wrote: post #39

Earlier quoted context omitted.

The 100 has nothing to do with HTTP pipelining, it's just a standard REST style "&limit=100" hard limit

You might be right. With the "&limit" parameter he can change how many items he receives per HTTP request. This has nothing to do with a limit on how many HTTP requests he can make per TCP connection (pipelining). Maybe that is the "100" he is complaining about, i.e., 100 items per HTTP request. However you failed to answer my question: Is he making 100 TCP connections to make 100 HTTP requests? Does the Reddit serve…

I didn't feel the need to answer your question because it was abundantly clear in the code that it's not using pipelining. You posted the exact curl option that he's not using.

Re: How F5Bot Slurps All of Reddit

#76

Post by TrolTure » Tue, Jul 31, 2018, 9:30 AM UTC

prolikewh0a wrote: post #31

This is mostly why I left Reddit. The API allows far too much control and I started questioning what was even real. Being able to quickly find keywords and then have a network of bots that creates replies/upvotes/downvotes is very disturbing thought to me. I can't even imagine something like that on a large scale to change opinions.

Out of interest in this I made Https://linksforreddit.com. You can view who links to certain articles. I backed up and truncated the 2017 data because the VPS ran out of space but there where and are interesting patterns. Nothing proof like but the same pdfs sometimes get linked close to a hundred times by similar themed and structured essays but textually different comments.

Re: How F5Bot Slurps All of Reddit

#77

Post by friendzis » Tue, Jul 31, 2018, 10:26 AM UTC

anarazel wrote: post #48

codeplea wrote: post #45

Earlier quoted context omitted.

Thanks for the explanation. The APIs make a lot more sense in light of it. It seems odd they didn't just use Postgres indexing though. Do you know if they benchmarked it at any point?

For a lot of queries EAV type schemas are really hard to index efficiently. E.g. searching for something like a = ? and b = ? where a and b are dynamic attributes you can't just have a multi-column index when using EAV. So you instead end up with two intermediate query results that then are intersected. If you need even semi-efficient querying EAV usually isn't the answer.

But it is possible to use hybrid data store. Extract important attributes you want to search on efficiently and store them in properly structured and indexed relational database. Other attributes can still be stored EAV style in different schema/database.

Re: How F5Bot Slurps All of Reddit

#78

Post by jwilk » Tue, Jul 31, 2018, 10:32 AM UTC

tgtweak wrote: post #57

koolba wrote: post #49

Earlier quoted context omitted.

Decompression is usually cheaper (CPU-wise) than compression so technically you’re trading their CPU for less of your bandwidth and CPU. If you do it right you can even keep the content stored compressed without re-compressing by saving the compressed byte stream directly.

Also use pigz instead of gzip so it can use multiple threads on decoding (although you're probably Limited by sending bandwidth)

pigz doesn't help much with decompression:

https://github.com/madler/pigz/issues/36#issuecomment-249041...

Decompression can’t be parallelized, at least not without specially prepared deflate streams for that purpose. As a result, pigz uses a single thread (the main thread) for decompression, but will create three other threads for reading, writing, and check calculation, which can speed up decompression under some circumstances.

Re: How F5Bot Slurps All of Reddit

#79

Post by testplzignore » Tue, Jul 31, 2018, 10:48 AM UTC

Which API do most Reddit bots use? Do they use the Reddit APIs directly, or do they use one of the third-party services (F5Bot, pushshift)? And are there any other options for getting a firehose of new Reddit posts/comments?

Re: How F5Bot Slurps All of Reddit

#80

Post by bicubic » Tue, Jul 31, 2018, 11:02 AM UTC

testplzignore wrote: post #79

Which API do most Reddit bots use? Do they use the Reddit APIs directly, or do they use one of the third-party services (F5Bot, pushshift)? And are there any other options for getting a firehose of new Reddit posts/comments?

https://praw.readthedocs.io/en/latest/code_overview/reddit_i...

It's pretty easy to get a firehose.