Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoiding bot detection: How to scrape the web without getting blocked?

github.com

Post reply on HN
Print view View original

231–240 of 312 posts

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#231

Post by Mordisquitos »

fragmede wrote: post #157
jameshart wrote: post #116

Earlier quoted context omitted.

Bots aren't just trying credential stuffing. They are: - committing clickfraud to game ad and referral revenue systems - posting fake or spam reviews and comments - generating fake behavioral signals to help bypass CAPTCHAs to help create accounts on other sites that can post spam comments - validating stolen credit card details - screwing with your metrics collection if you can't identify them as bots All of that is…

> -validating stolen credit card details To be clear, "validating" is an industry euphemism for stealing, just for a different purpose. How do you validate the card is live? Run a real transaction through it and mark it based on the result. But what do you run for this real transaction? Well, whatever you want. Typically it'll be something to avoid suspicion as much as possible, but the thief gets to pick what they t…

> How do you validate the card is live? Run a real transaction through it and mark it based on the result.

Not necessarily. Some online services, particularly shops that do home delivery, may give their customers the possibility of adding a card to their wallet and perform a verification as part of the process. As a result, it becomes possible to validate a stolen card number without performing an actual transaction.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#232

Post by intricatedetail »

drivebycomment wrote: post #177
intricatedetail wrote: post #96

By the way - is it possible to stop Google bot from scrapping without maintaining a list of IP addresses? Google doesn't publish these and it's not good to run reverse DNS as it slows down legitimate clients. I know you can put a meta tag, but bot still has to make a request to read it. I would like to completely cut off Google from scrapping.

Just put robot.txt and block Googlebot from there. Google obeys robot.txt.

Not all Google crawlers obey it. Also if Google already indexed something, only way is to let it crawl again and see meta noindex. It's a mess.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#233

Post by kzrdude »

jonatron wrote: post #44
marginalia_nu wrote: post #36

If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally questionable, a serious pain in the ass for the site owner and makes websites more prone to attempt to block bots in general. Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching…

Legitimate uses of scraping include price comparison, and finding copyright or trademark infringement.

I would say legitimate use includes archiving, and nothing more.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#234

Post by marginalia_nu »

hattmall wrote: post #181
marginalia_nu wrote: post #128

Earlier quoted context omitted.

Fine, send a 403 then. > The client does not have access rights to the content; that is, it is unauthorized, so the server is refusing to give the requested resource. Machines don't have any legal responsibility, bot-operators do. Which is why respecting these things is sort of important. At any rate, 40x does not mean "try again with a different user agent and another IP"

403 is per request, not requester. I get random 403s when just browsing some websites. Does that mean I should close the browser and not hit refresh for fear of breaking some wire fraud unauthorized access law?

If you go by the semantics of what the 403 code means, absolutely, that's excatly what the status code means.

In practice there's of course nuance, like anyone will occasionally type in the wrong password on a log-in screen, maybe try again and then realize it was the wrong log-in prompt. That's mostly fine.

That's different from deliberate trying to circumvent a measure like this. If you are doing the stuff in the link, you are absolutely crossing a line and you know it.

There's a large difference between "I got a 403 so I hit F5 once" and "I got a 403 so I used a residential proxy and spoofed my user-agent".

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#235

Post by marginalia_nu »

oefrha wrote: post #172
marginalia_nu wrote: post #36

If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally questionable, a serious pain in the ass for the site owner and makes websites more prone to attempt to block bots in general. Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching…

> Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching some resource that is expensive that bots in general refuse to lay off. Bots with seconds between the requests rarely get blocked even by CDNs. Tell that to any Cloudflare site on security level High or "I'm Under Attack!" year round.

Usually means they have large ongoing problems with aggressive bots.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#236

Post by herbst »

dzhiurgis wrote: post #211
varenc wrote: post #54

Earlier quoted context omitted.

This! Mobile IPs are far more lucrative. Many services will drop captchas and other anti-bot stuff for consumer mobile IPs. I recall Plaid at some point would run their bank scraping through mobile IPs. This sketchy company lets mobile app developers monetize user base by letting other people pay $$ to route requests through random people’s mobile IPs: https://brightdata.com/

Brightdata is formerly known as as Luminati who is owned by same company as Hola VPN. Similarly NordVPN owns Oxylabs (who mostly hack routers and cameras and sell those as residential IP’s).

I never checked their business practice. But I use Luminati rotating residential proxies for a lot of my scraping work.

Mostly to avoid hitting 'ddos protections' or other security bullshit that doesn't really make sense on 1 daily request or so.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#237

Post by herbst »

sparkling wrote: post #67
melony wrote: post #3

Earlier quoted context omitted.

The gold standard is residential IP. It is not cheap but its effectiveness is indisputable.

There are services that detect residential IPs being used for scraping nowadays. Plus there are other ways of detecting scraping: browser fingerprinting, aggressive rate-limiting and CAPTCHAs etc.

It's always just a question of detecting these things and code them in.

It's like writing a game bot with Java robots and pixel detection. It may is inefficient, may takes longer to make than a network solution. But I have yet to be detected anywhere

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#238

Post by herbst »

sparkling wrote: post #73
walrus01 wrote: post #72

Earlier quoted context omitted.

Captcha solving services are a thing, can be as crude as something that takes a screenshot, sends an image to a click farm worker getting paid $300 a month sitting in a cubicle in Bangladesh. There's various captcha solving services where you pay in bulk per captcha and submit data via an api.

Yup, such click farms exist. But driving up the costs and/or technical implementation efforts for bots/scrapers can be a part of your anti-bot strategy.

1000 captchas usually cost around a dollar or two. Honestly captchas never stopped me from anything.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#239

Post by aembleton »

jonatron wrote: post #80
Dah00n wrote: post #50

Earlier quoted context omitted.

>Legitimate uses of scraping include price comparison "Legitimate uses" is what the site operator says it is, nothing more nothing less. There are no laws that says you can scrape a site and circumvent their protection against doing so.

If you look at Ryanair, they've had legal battles with scrapers for more than 10 years, so it's not that simple.

I found Ryanair one of the more friendly ones to scrape, albeit for my own personal project. When you query for flights, they make a GET request with a JSON object in response, complete with flight times and prices.

For example:

  curl "https://www.ryanair.com/api/booking/v4/en-gb/availability?ADT=1&CHD=0&DateIn=&DateOut=2021-11-15&Destination=BER&Disc=0&INF=0&Origin=MAN&TEEN=0&promoCode=&IncludeConnectingFlights=false&FlexDaysBeforeOut=2&FlexDaysOut=2&ToUs=AGREED" | jq

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#240

Post by herbst »

rstupek wrote: post #119
kodah wrote: post #78

Earlier quoted context omitted.

When I worked in e-commerce as a SRE, bots were doing two things: - trying to disrupt business processes (eg: false referral listings, gift card scams, etc) - trying to disrupt systems I'm sure there are folks who use bots and scrapers for home automation, but these users generate marginal traffic in comparison. The real cost, aside from successfully achieving the points above, is the bandwidth and hardware costs tha…

Or bots using ecommerce sites to test credit card validity for use elsewhere

For whatever reason in years of running a SaaS this only happened maybe 3 times and never with my online shop. I guess using stripe and a few basic security settings keeps them away mostly
Post reply on HN