Avoiding bot detection: How to scrape the web without getting blocked? • Hacker News Board

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#171

Post by [deleted] » Mon, Nov 01, 2021, 3:48 AM UTC

Mister_Snuggles wrote: post #31

bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…

[deleted]

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#172

Post by oefrha » Mon, Nov 01, 2021, 3:51 AM UTC

marginalia_nu wrote: post #36

If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally questionable, a serious pain in the ass for the site owner and makes websites more prone to attempt to block bots in general. Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching…

> Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching some resource that is expensive that bots in general refuse to lay off. Bots with seconds between the requests rarely get blocked even by CDNs.

Tell that to any Cloudflare site on security level High or "I'm Under Attack!" year round.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#173

Post by ufmace » Mon, Nov 01, 2021, 4:01 AM UTC

What I really enjoy about this thread is all of the completely different perspectives. Lots of people doing anti-abuse research bemoaning that this stuff exists, and lots of people working against what are from their perspective ham-handed anti-abuse tech blocking legitimate useful automation trading tips on how to do it better. I guess the other sides of those we don't see much. People doing actual black-hat work probably don't post about it on public forums, and most of the over-broad anti-abuse is probably a side effect of taking some anti-abuse tech and blindly applying it to the whole site just because that's simpler, often no tech people may be really involved at all.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#174

Post by AlexanderTheGr8 » Mon, Nov 01, 2021, 4:04 AM UTC

notriddle wrote: post #104

janmo wrote: post #57

Earlier quoted context omitted.

Or, you can spoof the google bot or Bing bot user agent and try to scrape the site that way.

Impersonating Googlebot is a great way to get blocked. Real Googlebot only comes from certain IP addresses.

almost all sites uses "host" command to see if the IP belongs to the googlebot (this is recommended way by Google). And you can easily find many IPs which return googlebot when you "host" them. So you can use any of these IP addresses to spoof GoogleBot.

To find more IPs, make your own website, and wait until GoogleBot eventually shows up :)

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#175

Post by fragmede » Mon, Nov 01, 2021, 4:10 AM UTC

abadger9 wrote: post #115

ryandrake wrote: post #110

Earlier quoted context omitted.

Do you have any factual corrections? Your post reminds me of those "I'm getting a kick out of these replies" copypasta--declaring someone wrong and claiming authoritative knowledge, but without actually correcting any of the errors of fact.

Sorry about that, we're unable to discuss our projects with adjacent teams within the company. What you're saying is a valid frustration, the motivation behind the original comment was to put a thermometer on the repo.

Adjacent teams? It makes sense not to discuss it with totally unrelated teams, but a large Internet company wants to have its cake and eat it too - scrape all of the worlds websites, while simultaneously denying others from scraping its results. Limiting communication between adjacent teams (ie the scraper team and the bot blocker team) really seems like it would be a hinder rather than help.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#176

Post by wumpus » Mon, Nov 01, 2021, 4:15 AM UTC

charcircuit wrote: post #141

azalemeth wrote: post #106

Earlier quoted context omitted.

You very much missed the false positive rate! I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech. I had to try to log in to my bank about ten times today because their js-crap website didn't like me ( grumble why does it even need to ask for my desktop's accelerometer data via js...) Stuff like this is a pain beyond pain. I rea…

>I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech. Have you tried not using these things? Anonymity is exactly what bots want. They want to be able to post a spam message every single second and be impossible to ban since they are anonymous. The internet can't function if people are allowed to be anonymous.

This is not quite up there with "won't someone think about the children!!!!", but still, it's sad.

Fortunately, almost all of the websites I visit with my anonymized browser aren't places that I wish to attempt to post a message. Unfortunately, I can easily run into defenses of an entire site when the problem is spam sending.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#177

Post by drivebycomment » Mon, Nov 01, 2021, 4:19 AM UTC

intricatedetail wrote: post #96

By the way - is it possible to stop Google bot from scrapping without maintaining a list of IP addresses? Google doesn't publish these and it's not good to run reverse DNS as it slows down legitimate clients. I know you can put a meta tag, but bot still has to make a request to read it. I would like to completely cut off Google from scrapping.

Just put robot.txt and block Googlebot from there. Google obeys robot.txt.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#178

Post by catlifeonmars » Mon, Nov 01, 2021, 4:31 AM UTC

toomuchtodo wrote: post #75

catlifeonmars wrote: post #66

Earlier quoted context omitted.

I’m curious. What is link-only login?

A link is generated, emailed to the user, and clicking the link logs them in.

This sounds more or less like OTP.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#179

Post by AlexanderTheGr8 » Mon, Nov 01, 2021, 4:32 AM UTC

gzer0 wrote: post #137

abadger9 wrote: post #102

I'm a lead engineer on the search team of a publicly traded company who's bread and butter is this domain. I was curious about this list, it candidly misses the mark- the tech mentioned in this blog is what you might get if you hired a competent consultant to build out a service without having domain knowledge. In my experience, what's being used on the bleeding edge is two steps ahead of this.

I have a considerable amount of experience in the industry. Some of these so-called "advanced" techniques: * We use our own mobile emulation software (similiar to bluestacks). Turns out, mobile helps with a lot of things (below). * We use mobile IPs only. Mobile LTE data users are behind CGNATfor IPV4. You can't block one ip without possibly blocking hundreds of innocent IPs using the same exit point. * All you need…

This list is pretty interesting. If you don't mind me asking, what do you work on that you requires such sophisticated stuff?

Also, does this work only for browsers or also for mobile apps? I have always assumed that it is always theoretically possible to get data from browsers (very extreme resort is save the browser page / (screenshot + computer vision)); but it can be impossible to get data from apps (especially ios). Are my assumptions correct?

Also can you explain mobile IPs more? If they are such a big vulnerability, why is there no potential solution to them?

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#180

Post by fragmede » Mon, Nov 01, 2021, 4:32 AM UTC

azalemeth wrote: post #106

oxymoron wrote: post #15

Earlier quoted context omitted.

Yeah, I used to work for one of the major anti-bot vendors. Customers weren't clueless. Nobody buys these solutions because they're so much fun, it's a cost center and they monitor their ROI quite closely. Credit card charge backs, impact to infrastructure, extra incurred cost due to underlying api's (like in the Airline industry in particular) etc are all reasons why bot mitigation is a better option than nothing fo…

You very much missed the false positive rate! I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech. I had to try to log in to my bank about ten times today because their js-crap website didn't like me ( grumble why does it even need to ask for my desktop's accelerometer data via js...) Stuff like this is a pain beyond pain. I rea…

You could always go into your local bank branch instead of accessing it over the Internet. Your desktop's accelerometer helps add to your computers 'run by a human' score. Normally I'd take more issue with whatever possible privacy issue there, but my bank is where I keep my money so I'm really okay with them trying hard to keep bots out of my account.