Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoiding bot detection: How to scrape the web without getting blocked?

github.com

Post reply on HN
Print view View original

41–50 of 312 posts

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#41

Post by TedDoesntTalk »

melony wrote: post #3
bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

The gold standard is residential IP. It is not cheap but its effectiveness is indisputable.

Not anymore. Now it’s mobile IP addresses.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#42

Post by TruthWillHurt »

marginalia_nu wrote: post #36

If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally questionable, a serious pain in the ass for the site owner and makes websites more prone to attempt to block bots in general. Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching…

You're forgetting a case where a website offers a garbage API that doesn't provide all the data available via the web interface,

either due to neglegance (Apple Store developer console),

Security (Google Play Store accounting data),

Or financial gain (AppsFlyer "premium API").

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#43

Post by ChuckMcM »

I am always amazed when otherwise intelligent people assert without data that the marginal cost of serving web traffic to scrapers/bots is zero. It is kind of like people who say "Why don't they put more fuel in the rocket so it can get all the way into orbit with just one stage?"

It sounds great but it is a completely ignorant thing to say.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#44

Post by jonatron »

marginalia_nu wrote: post #36

If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally questionable, a serious pain in the ass for the site owner and makes websites more prone to attempt to block bots in general. Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching…

Legitimate uses of scraping include price comparison, and finding copyright or trademark infringement.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#45

Post by 1cvmask »

ixs wrote: post #30
Gigachad wrote: post #8

Earlier quoted context omitted.

2FA should be a requirement on everything now. And if your site can't for some reason or you don't want to deal with it, then limit your site to external login providers only. 2FA, especially app based, has been proven to work really really well.

It does not. There are myriad ways of extracting the TOTP seed from these apps... Or you just reverse engineer the setup/confirmation process and then you can generate/trigger your own tokens from your automation workflow. 2FA is a good security feature but it does not help against web scraping. Credential stuffing and other 3rd party attacks? Yes, it _can_ help. But it does not always help. There's a phishing group…

There are different methods of 2FA like scanning encrypted barcodes that show that you require intent.

It seems that the Duo core app is a variant of HOTP?

What's the name of the phishing group and any details on them? There was a Defcon or Black Hat video where they would constantly send a push approval to the mobile which was not PIN protected and most people would click on it. Don't remember which OTP generator it was.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#46

Post by peterburkimsher »

2 of my social media accounts have fallen victim to bot detection, despite not using scripts. There are other websites for which I have used scripts, and sometimes ran into CAPTCHA restrictions, but was able to adjust the rate to stay within limits.

CouchSurfing blocked me after I manually searched for the number of active hosts in each country (191 searches), and posted the results on Facebook. Basically I questioned their claim that they have 15 million users - although that may be their total number of registered accounts, the real number of users is about 350k. They didn't like that I said that (on Facebook) so they banned my CouchSurfing account. They refused to give a reason, but it was a month after gathering the data, so I know that it was retaliation for publication.

LinkedIn blocked me 10 days ago, and I'm still trying to appeal to get my account back.

A colleague was leaving, and his manager asked me to ask people around the company to sign his leaving card. Rather than go to 197 people directly, I intentionally wanted to target those who could also help with the software language translation project (my actual work). So I read the list of names, cut it down to 70 "international" people, and started searching for their names on Google. Then I clicked on the first result, usually LinkedIn or Facebook.

The data was useful, and I was able to find willing volunteers for Malay, Russian, and Brazilian Portuguese!

After finding the languages from 55 colleagues over 2 hours, LinkedIn asked for an identity verification: upload a photo of my passport. No problem, I uploaded it. I also sent them a full explanation of what I was doing, why, how it was useful, and a proof of my Google search history.

But rather than reactivate my account, LinkedIn have permanently banned me, and will not explain why.

"We appreciate the time and effort behind your response to us. However, LinkedIn has reviewed your request to appeal the restriction placed on your account and will be maintaining our original decision. This means that access to the account will remain restricted.

We are not at liberty to share any details around investigations, or interpret the terms of service for you."

So when the CAPTCHA says "Are you a robot?", I'm really not sure. Like Pinocchio, "I'm a real boy!"

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#48

Post by marginalia_nu »

soheil wrote: post #40
marginalia_nu wrote: post #36

If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally questionable, a serious pain in the ass for the site owner and makes websites more prone to attempt to block bots in general. Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching…

Google can access any site without being blocked. They dominate the search space and give little incentive for site owners to allow other bots. I'd say bypassing these measures is fair game while there is a monopoly in search space. We don't want a web that only Google can access. By the way great work on Marginalia search engine, I love it.

I've honestly not had much problem at all crawling the web as an indie search engine operator. If you want to get past CloudFlare you can register your bot fingerprint with them.

A small number of sites has blocked my crawler , but that's almost always been my own fault, and happened a few instances when the crawler was misbehaving and actually fetching too aggressively (or repeatedly). In every case just sending an email to the site explaining what happened and humbly asking for a second chance been enough to be allowed back in.

Most website owners don't seem to mind small search engines at all, what they don't want is scrapers that aggressively scrape their entire site 10 times a day, ignoring robots.txt, and being a general nuisance.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#49

Post by [deleted] »

kevin_thibedeau wrote: post #37
Mister_Snuggles wrote: post #31

Earlier quoted context omitted.

I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…

At a minimum any scraper that doesn't execute JS needs to impersonate a screen reader user agent. Locking out disabled people has to be many levels of illegal in most countries.

[deleted]

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#50

Post by Dah00n »

jonatron wrote: post #44
marginalia_nu wrote: post #36

If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally questionable, a serious pain in the ass for the site owner and makes websites more prone to attempt to block bots in general. Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching…

Legitimate uses of scraping include price comparison, and finding copyright or trademark infringement.

>Legitimate uses of scraping include price comparison

"Legitimate uses" is what the site operator says it is, nothing more nothing less. There are no laws that says you can scrape a site and circumvent their protection against doing so.

Post reply on HN