Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoiding bot detection: How to scrape the web without getting blocked?

github.com

Post reply on HN
Print view View original

261–270 of 312 posts

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#261

Post by no_time »

softwaredoug wrote: post #164

A lot of web scraping is annoying often because there’s *an explicit API built for the scrapers needs*. Instead of looking for an API, many think to first use web scraping. This in turn puts load and complexity on the user facing web app that must now tell scraper from real users.

Using the API almost always has more "strings attached". Like you have to register and get an API token or something. Or even pay. If you want people to use your API, don't make it less convinient than scraping the page.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#262

Post by defer »

vidarh wrote: post #226
defer wrote: post #206

Earlier quoted context omitted.

Not exactly. Some people are in the business of gathering and selling valid credit cards. They won't cash out on them or buy items. Instead, they'll collect cards from a source (skimming, hacking, whatever), validate them by adding them to a website that does an authorization (those $1 checks that never get committed). They can then sell them wholesale for a premium compared to non-verified cards.

You'd think so, but I've personally had to find ways of blocking people who were buying premium services for an online service to validate cards.

Oh, for sure. It's definitely not either/or. I've worked in fraud prevention software in the past and our clients would definitely see both.

I've been away from that world for a while but remember that more serious operations will separate the cashing out part (either money or goods) from their acquiring / validating operation because the former carries more risk.

There's also an interesting episode of the darknet diaries podcast (https://darknetdiaries.com/episode/85/) about card cloning which I found interesting.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#263

Post by matheusmoreira »

ChuckMcM wrote: post #43

I am always amazed when otherwise intelligent people assert without data that the marginal cost of serving web traffic to scrapers/bots is zero. It is kind of like people who say "Why don't they put more fuel in the rocket so it can get all the way into orbit with just one stage?" It sounds great but it is a completely ignorant thing to say.

What could my scraper which makes 1 HTTP request per day possibly cost the webmaster?

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#264

Post by matheusmoreira »

ufmace wrote: post #173

What I really enjoy about this thread is all of the completely different perspectives. Lots of people doing anti-abuse research bemoaning that this stuff exists, and lots of people working against what are from their perspective ham-handed anti-abuse tech blocking legitimate useful automation trading tips on how to do it better. I guess the other sides of those we don't see much. People doing actual black-hat work pr…

When these companies endeavor to stop abuse, they trample all over our freedoms. Suddenly we can't have non-browser user agents anymore. Suddenly we can't root our smartphones anymore. They want nothing to do with us unless it's 100% on their terms with us completely under their control.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#265

Post by smichel17 »

Tor3 wrote: post #258
fragmede wrote: post #180

Earlier quoted context omitted.

You could always go into your local bank branch instead of accessing it over the Internet. Your desktop's accelerometer helps add to your computers 'run by a human' score. Normally I'd take more issue with whatever possible privacy issue there, but my bank is where I keep my money so I'm really okay with them trying hard to keep bots out of my account.

The physical presence of banks is going away. Where I live you can't do any kind of monetary transaction in the local branch offices of any of the banks anymore. You can a) apply for a loan (and even that may go away soon), and b) identify yourself and get a physical token used for accessing the bank via the net. You can't withdraw money, you can't pay bills, you can't exchange currency. I haven't been inside my bank…

Where do you live (country or US state)?

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#266

Post by matheusmoreira »

marginalia_nu wrote: post #36

If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally questionable, a serious pain in the ass for the site owner and makes websites more prone to attempt to block bots in general. Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching…

> If someone is signalling to you you that they do not want your bot on their site, then maybe respect that?

Maybe respect user freedom? If I can access the data using my browser, why can't I access it using my script?

Why is Google the only one who can do it? Must they have yet another monopoly?

> Bots with seconds between the requests rarely get blocked even by CDNs.

I've had scripts that made 1 request per day get blocked for no reason. Not to mention the endless cloudflare javascript bullshit they made me support for it to even work.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#267

Post by leppr »

charcircuit wrote: post #141
azalemeth wrote: post #106

Earlier quoted context omitted.

You very much missed the false positive rate! I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech. I had to try to log in to my bank about ten times today because their js-crap website didn't like me ( grumble why does it even need to ask for my desktop's accelerometer data via js...) Stuff like this is a pain beyond pain. I rea…

>I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech. Have you tried not using these things? Anonymity is exactly what bots want. They want to be able to post a spam message every single second and be impossible to ban since they are anonymous. The internet can't function if people are allowed to be anonymous.

With many of these big anti-bot services like Google ReCaptcha, it's not even specialized anonymity tools that can cause shadow banning, just unusual user-agents.

All of these have independently caused me to get into endless ReCaptcha loops: firefox on android, smartphone with unusual screen resolution, clean browser profile with VPN.

It's so common that I now default to using duckduckgo, which never blocks me. I doubt DDG has a lower DDoS/Resources ratio than Google. Some companies are just lazier and less principled than others.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#268

Post by marginalia_nu »

herbst wrote: post #242
marginalia_nu wrote: post #235

Earlier quoted context omitted.

Usually means they have large ongoing problems with aggressive bots.

Or they using it as lazy approach to fix long-dos attacks by competitors.

What is the non-lazy alternative that stops a botnet DDOS without affecting regular visitors?

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#269

Post by raxxorrax »

azalemeth wrote: post #106
oxymoron wrote: post #15

Earlier quoted context omitted.

Yeah, I used to work for one of the major anti-bot vendors. Customers weren't clueless. Nobody buys these solutions because they're so much fun, it's a cost center and they monitor their ROI quite closely. Credit card charge backs, impact to infrastructure, extra incurred cost due to underlying api's (like in the Airline industry in particular) etc are all reasons why bot mitigation is a better option than nothing fo…

You very much missed the false positive rate! I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech. I had to try to log in to my bank about ten times today because their js-crap website didn't like me ( grumble why does it even need to ask for my desktop's accelerometer data via js...) Stuff like this is a pain beyond pain. I rea…

Cloudflare is pretty guilty of this if you use some more exotic approaches to request info. How often have I seen their captcha that is intended for bots...

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#270

Post by dxdm »

vidarh wrote: post #226
defer wrote: post #206

Earlier quoted context omitted.

Not exactly. Some people are in the business of gathering and selling valid credit cards. They won't cash out on them or buy items. Instead, they'll collect cards from a source (skimming, hacking, whatever), validate them by adding them to a website that does an authorization (those $1 checks that never get committed). They can then sell them wholesale for a premium compared to non-verified cards.

You'd think so, but I've personally had to find ways of blocking people who were buying premium services for an online service to validate cards.

Doing tiny transactions to validate cards is a thing. I've seen this happen, it's a known problem, and that at other times it's larger transactions does not make it go away.
Post reply on HN