Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How F5Bot Slurps All of Reddit

intoli.com

Post reply on HN
Print view View original

61–70 of 91 posts

Re: How F5Bot Slurps All of Reddit

#61

Post by stuck_in_the_ma »

minimaxir wrote: post #11
allenz wrote: post #7

Related: Jason Baumgartner has maintained a Reddit scraping pipeline for a few years now, and wrote up some notes about making it robust: https://pushshift.io

You can also use the Pushshift real-time feed in BigQuery to query for keywords in submissions in real time (unfortunately the comments feed broke last month) Example query which searches for 'f5bot' in the past day and correctly finds the corresponding posts on Reddit: #standardSQL SELECT title, subreddit, permalink FROM `pushshift.rt_reddit.submissions` WHERE created_utc > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVA…

There has been a lot of interest expressed in getting this working and dependable. It's part of my plan when releasing the new API. There is A LOT of internal code managing everything. I've got terabytes of indexes alone just to handle the 5 million API requests I'm currently getting each month to the Pushshift API (I have around 20 terabytes of SSD / NVMe space and around 512 GB of ram behind this project).

Re: How F5Bot Slurps All of Reddit

#62

Post by Bromskloss »

jersully72 wrote: post #60
Bromskloss wrote: post #51

Earlier quoted context omitted.

How do you change opinions using that? Would it be an illegitimate way to change someone's opinion?

I think what they're describing are propaganda bots. If so, what you're asking is how does propaganda work, and is it not legitimate?

Yes. Voting patterns, for example, how do those change people's opinions? It would seem that the most they could do is give an impression of what comments are and are not well received. Do people base much of their opinions on that? In which direction?

Re: How F5Bot Slurps All of Reddit

#63

Post by [deleted] »

lucb1e wrote: post #54
RSZC wrote: post #4

edit: cool

Please don't remove your old text when it turns out you're wrong or have an unpopular opinion. You're not the only person in this thread who missed the batch part of individual ID requests and I initially was confused as well, but removing your post breaks the comment thread, and after 3h you can't edit anymore so I'll have to downvote.

[deleted]

Re: How F5Bot Slurps All of Reddit

#64

Post by rotred »

Deimorz wrote: post #55
lucb1e wrote: post #52

Earlier quoted context omitted.

If you unsave a newer one, wouldn't the older one pop up again because it updates the index with limit 1000 again? If the data is there, it should find all posts and then truncate, in that order, if I'm understanding this correctly.

Nope, because the old one's already been pushed off the end of the list and it doesn't re-generate the list when you unsave, just removes the item from the list. Imagine I have a list of max length 5, when I initially fill it up it looks like [5, 4, 3, 2, 1]. If I save one more thing, it adds 6 at the front, then truncates the list and removes the 1 from the end, so now you have [6, 5, 4, 3, 2]. At that point, if I u…

why would they make the limit so low?

Re: How F5Bot Slurps All of Reddit

#66

Post by jack9 »

jonknee wrote: post #21
wolco wrote: post #13

"You may think PHP is slow" Why would we think php is slow? PHP is blazing fast certain applications (looking at you sugarcrm) make this into a mockery by rewriting queries and loading unnecessary data into each page request. Nice to see a php related show and tell.

Because for a long time (critically, when PHP was very popular) it was quite slow. It has gotten better, but a lot of people haven't used it since it was slow.

Quite slow compared to what?

PHP was popular when it had MySQL bindings that were always up to date with MySQL changes and apache PHP setup was simplistic. Thus the LAMP stack, right? It has always been rather fast, in the larger ecosystem of web scripting languages (especially if you avoided particular BIFs) but it got a LOT faster in 7 as the slowest bits were optimized alongside everything else.

Re: How F5Bot Slurps All of Reddit

#67

Post by jack9 »

prolikewh0a wrote: post #31

This is mostly why I left Reddit. The API allows far too much control and I started questioning what was even real. Being able to quickly find keywords and then have a network of bots that creates replies/upvotes/downvotes is very disturbing thought to me. I can't even imagine something like that on a large scale to change opinions.

> I started questioning what was even real.

The votes aren't real and they don't matter. This is true for FB likes or whatever else you can imagine. Reddit goes to a LOT of trouble to counter bad actors, but if you're in some subreddit that you think is full of abuse, then it's like any other part of the internet. Go somewhere else. This has little to do with Reddit as a whole, so the proclamation seems unnecessarily volatile.

Re: How F5Bot Slurps All of Reddit

#68

Post by ralfd »

Deimorz wrote: post #55
lucb1e wrote: post #52

Earlier quoted context omitted.

If you unsave a newer one, wouldn't the older one pop up again because it updates the index with limit 1000 again? If the data is there, it should find all posts and then truncate, in that order, if I'm understanding this correctly.

Nope, because the old one's already been pushed off the end of the list and it doesn't re-generate the list when you unsave, just removes the item from the list. Imagine I have a list of max length 5, when I initially fill it up it looks like [5, 4, 3, 2, 1]. If I save one more thing, it adds 6 at the front, then truncates the list and removes the 1 from the end, so now you have [6, 5, 4, 3, 2]. At that point, if I u…

What if you unsave 2 items and save 1. Is then the list remade?
Post reply on HN