Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How F5Bot Slurps All of Reddit

intoli.com

Post reply on HN
Print view View original

81–90 of 91 posts

Re: How F5Bot Slurps All of Reddit

#81

Post by bluetidepro »

mustacheemperor wrote: post #15
saurik wrote: post #6

It is difficult for me to describe just how angry it makes me that reddit doesn't provide a way for users to even do basic things like "see all of my own comments" or "see all of the posts made to the subreddit I moderate". They keep nerfing the search APIs and claim it is so they could make the indexes more efficient, but while that might make sense for a full-text search interface, that is entirely unreasonable for…

Reddit doesn't even allow users to save more than 1000 posts, and worse does not visibly document this or provide any kind of warning that the limit has been exceeded. Anecdotally, I've read users say that revisiting the saved pages will still show an "unsave" button so the information is recorded somewhere. But once a user exceeds 1000 entries on their "saved" page, adding new ones will silently vaporize old ones. h…

A good work around tip is to set up an https://ifttt.com/ to trigger when you save to offload your saves to somewhere else. I save all my reddit saves to Evernote, personally. It can help with this known issue!

Re: How F5Bot Slurps All of Reddit

#82

Post by textmode »

meritt wrote: post #75
textmode wrote: post #71

Earlier quoted context omitted.

You might be right. With the "&limit" parameter he can change how many items he receives per HTTP request. This has nothing to do with a limit on how many HTTP requests he can make per TCP connection (pipelining). Maybe that is the "100" he is complaining about, i.e., 100 items per HTTP request. However you failed to answer my question: Is he making 100 TCP connections to make 100 HTTP requests? Does the Reddit serve…

I didn't feel the need to answer your question because it was abundantly clear in the code that it's not using pipelining. You posted the exact curl option that he's not using.

I apologise if I confused you. I was simply wondering why he is not using pipelining, which IME can be ideal for the sort of text retrieval he is performing.

Re: How F5Bot Slurps All of Reddit

#83

Post by bmn__ »

saagarjha wrote: post #2

Aho-Corasick is really great. It’s a bit complicated to set up, but once you have the modified true set up it’s really fast. By the way, > Basically I use the selftext, subreddit, permalink, url and title. The other 95% of it is just wasted bandwidth. It’d probably be better for Reddit if they allowed for specifying the fields we care about rather than just returning the whole thing…

> It’s a bit complicated to set up […] it’s really fast

Sigh, just use Perl. Writing code with the general regex engine took me only one minute of effort, but it runs already nearly 500× faster than codeplea's optimised special purpose code.

Why 100000 loops and not 10 like in the original code? Otherwise Benchmark.pm will show "(warning: too few iterations for a reliable count)".

----

benchmark.php 100000 loops:

    Loaded 3000 keywords to search on a text of 19377 characters.

    Searching with aho corasick...
    time: 329.3522541523
----

benchmark.pl 100000 loops:

    Benchmark: timing 100000 iterations of regex...
    regex: 0.691561 wallclock secs ( 0.69 usr +  0.00 sys =  0.69 CPU) @ 144927.54/s (n=100000)
----

benchmark.pl (fill in the abbreviated ... parts from benchmark_setup.php):

    #!/usr/bin/env perl
    use Benchmark qw(timethese :hireswallclock);
    require Time::HiRes;
    my @needles = qw(
    abandonment abashed abashments abduction ...
    );
    my $haystack = 'unscathed grampus ...
    heroically';
    my $n = join '|', @needles;
    timethese 100000, {
        regex => sub {
            my @found;
            while ($haystack =~ /($n)/cg) {
                push @found, [$1, pos $haystack];
            }
            return @found;
        },
        index => sub {
            my @found;
            for (@needles) {
                my $pos = index $haystack, $_;
                push @found, [$_, $pos] if -1

Re: How F5Bot Slurps All of Reddit

#85

Post by codeplea »

bmn__ wrote: post #83
saagarjha wrote: post #2

Aho-Corasick is really great. It’s a bit complicated to set up, but once you have the modified true set up it’s really fast. By the way, > Basically I use the selftext, subreddit, permalink, url and title. The other 95% of it is just wasted bandwidth. It’d probably be better for Reddit if they allowed for specifying the fields we care about rather than just returning the whole thing…

> It’s a bit complicated to set up […] it’s really fast Sigh, just use Perl. Writing code with the general regex engine took me only one minute of effort, but it runs already nearly 500× faster than codeplea's optimised special purpose code. Why 100000 loops and not 10 like in the original code? Otherwise Benchmark.pm will show "(warning: too few iterations for a reliable count)". ---- benchmark.php 100000 loops: Loa…

Is your solution broken for the cases where keywords are prefixes or suffixes of each other? This situation is very common in my use-case. Also, does your solution work if a keyword appears multiple times?

I get what you're saying, but it's not quite as easy as you imply.

Pulling in an entire programming language is a much bigger dependency and maintenance cost than spending a couple hours writing an algorithm. It would make more sense to just use a C extension.

I did try PHP's regex. It was much, much slower.

Re: How F5Bot Slurps All of Reddit

#86

Post by igouy »

bufferoverflow wrote: post #16
wolco wrote: post #13

"You may think PHP is slow" Why would we think php is slow? PHP is blazing fast certain applications (looking at you sugarcrm) make this into a mockery by rewriting queries and loading unnecessary data into each page request. Nice to see a php related show and tell.

PHP used to be very slow, it got better with v7.0. It's still quite slow compared to C/C++/Rust/Go, more than 10x slower: https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Just curious - why did you choose fasta as your comparison?

Re: How F5Bot Slurps All of Reddit

#87

Post by NVRM »

sieabahlpark wrote: post #20
bufferoverflow wrote: post #16

Earlier quoted context omitted.

PHP used to be very slow, it got better with v7.0. It's still quite slow compared to C/C++/Rust/Go, more than 10x slower: https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

An interpreted language is slower than a compiled binary? Color me shocked.

Not always! https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Re: How F5Bot Slurps All of Reddit

#89

Post by bmn__ »

codeplea wrote: post #85
bmn__ wrote: post #83

Earlier quoted context omitted.

> It’s a bit complicated to set up […] it’s really fast Sigh, just use Perl. Writing code with the general regex engine took me only one minute of effort, but it runs already nearly 500× faster than codeplea's optimised special purpose code. Why 100000 loops and not 10 like in the original code? Otherwise Benchmark.pm will show "(warning: too few iterations for a reliable count)". ---- benchmark.php 100000 loops: Loa…

Is your solution broken for the cases where keywords are prefixes or suffixes of each other? This situation is very common in my use-case. Also, does your solution work if a keyword appears multiple times? I get what you're saying, but it's not quite as easy as you imply. Pulling in an entire programming language is a much bigger dependency and maintenance cost than spending a couple hours writing an algorithm. It wo…

You are right, the solution is broken. I can't make it work, so I take back what I said.

I learnt something valuable, thank you for that.

Re: How F5Bot Slurps All of Reddit

#90

Post by igouy »

NVRM wrote: post #87
sieabahlpark wrote: post #20

Earlier quoted context omitted.

An interpreted language is slower than a compiled binary? Color me shocked.

Not always! https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Furthermore ;-)

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Post reply on HN