Live data from Hacker News

Avoiding bot detection: How to scrape the web without getting blocked?

github.com

181–190 of 312 posts

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#181
post #124

Earlier quoted context omitted.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#cli... > Although the HTTP standard specifies "unauthorized", semantically this response means "unauthenticated". That is, the client must authenticate itself to get the requested response. So it would seem that it actually doesn't positively imply that you're NOT authorized. Which kind of makes sense; machines can't detect legality of things, just that certain…

Fine, send a 403 then. > The client does not have access rights to the content; that is, it is unauthorized, so the server is refusing to give the requested resource. Machines don't have any legal responsibility, bot-operators do. Which is why respecting these things is sort of important. At any rate, 40x does not mean "try again with a different user agent and another IP"

403 is per request, not requester. I get random 403s when just browsing some websites. Does that mean I should close the browser and not hit refresh for fear of breaking some wire fraud unauthorized access law?

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#182

Earlier quoted context omitted.

Do you have any factual corrections? Your post reminds me of those "I'm getting a kick out of these replies" copypasta--declaring someone wrong and claiming authoritative knowledge, but without actually correcting any of the errors of fact.

Not the OP but when I was running Operations at Blekko (a search engine) I spent part of my time dealing with scrapers. When treated like a puzzle it can be really interesting. So I thought I'd share a few tidbits. 1) We did a simple 'speed' test, how many queries per second were coming from an IP, and auto-ban on the limit being exceeded, started at 100qps and watched as the traffic moved down to 99.5qps. Pushed to…

Love the idea of all the links going to fbi.gov.

If you don't mind, can you elaborate on the 2nd point? Specifically, what do you mean by hijacking browsers with toolbars?

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#183

Earlier quoted context omitted.

Not the OP but when I was running Operations at Blekko (a search engine) I spent part of my time dealing with scrapers. When treated like a puzzle it can be really interesting. So I thought I'd share a few tidbits. 1) We did a simple 'speed' test, how many queries per second were coming from an IP, and auto-ban on the limit being exceeded, started at 100qps and watched as the traffic moved down to 99.5qps. Pushed to…

Love the idea of all the links going to fbi.gov. If you don't mind, can you elaborate on the 2nd point? Specifically, what do you mean by hijacking browsers with toolbars?

Well if you missed it, back in the day, places like Download.com or Source Forge would have a little pre-clicked link to "add search toolbar to my browser!" which, after you downloaded your think would leave you with a browser that always redirected your search requests to what ever company paid to send you there.

Lots of people did it, even Blekko (although we stopped after we figured out it was just a scam), and the way it worked is toolbar company X would approach your internet site and say "we can send you a lot of traffic, just use this toolbar, we'll even pay you every time one gets installed."

Anyway, the toolbar would hook itself into the address bar and search selection hook in the browser and redirect every search to where ever it was told to. The nefarious part is that these tool bars were often shipped to the company as binaries not source, so you didn't really know everything they were doing. Once we figured out they were scumbags we also found out that they did some really scumbaggish things.

We stopped using them but lots of people did and finally the browser makers re-wrote the browsers to make what they did either impossible or easy for the user to revert and those guys rolled up their shops and went on to become some other type of scam.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#184
post #136

Earlier quoted context omitted.

I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…

Unfortunately, the days of reliable non-JavaScript capable scraping are over. Fortunately there are plenty of tools to handle this, and at a hobby level not particularly resource intensive. Something like this is simple and reliable in many cases: https://github.com/berstend/puppeteer-extra/tree/master/pack...

> Unfortunately, the days of reliable non-JavaScript capable scraping are over.

Not really. In a lot of cases websites use JavaScript to call some API along with some on the fly generated token to prevent abuse.

As long as that token isn't captcha you can reverse engineer the site to do scraping without javascript and that is so much faster than browser based scraping.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#185

If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally questionable, a serious pain in the ass for the site owner and makes websites more prone to attempt to block bots in general. Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching…

> Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive

Obviously you haven't used Instagram recently.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#186
post #6
post #3

Earlier quoted context omitted.

The gold standard is residential IP. It is not cheap but its effectiveness is indisputable.

A residential IP would help for IP based detection. As the Readme mentions, there's also Javascript based detection. If, for example, your browser has navigator.webdriver set incorrectly, then you can still get blocked even on a residential IP.

The point is that both can be required. You can have the most sophisticated user emulating browser, bit if all you have access to run it on are low quality IPs that have been blocked or that are often used for abuse, you won't get far. You can have residential IPs and if you're just wrapping curl, you also might find you're blocked.

Together, there's little to detect different than a regular user. The reason why the residential IPs is given heavy importance though is that it's the one part that costs a lot of money if you need enough of them you need to use a proxy service and you transfer a lot of data. Entry level pricing is over $15/GB for high quality services.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#187
post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

Bots aren't just trying credential stuffing. They are: - committing clickfraud to game ad and referral revenue systems - posting fake or spam reviews and comments - generating fake behavioral signals to help bypass CAPTCHAs to help create accounts on other sites that can post spam comments - validating stolen credit card details - screwing with your metrics collection if you can't identify them as bots All of that is…

All very valid reasons (except to a degree the "screwing with metrics" one). There a lot of websites which do not really face any of the aforementioned issues, simply because they do not sell you anything, are not an ad network or run referral programs, do not even have user-generated content, etc. And even the sites that do, it's usually a rather small part of the "surface" that needs such protections e.g. the actual API call to make a checkout or post a comment.

And yet if you dare browse the web with TOR or a VPN or sometimes just happen to be on a small ISP[0][1] then you're being punished immediately. You solve your cloudflare-supplied captcha because you may be a bot (you're not, and the dangerous bots will not be defeated by this anyway, but some humans will be), and then you get an error from the website itself because it runs a secondary bot detection thing. And you weren't even anywhere near anything "dangerous" like a checkout page.

[0] My parents use a regional small ISP (but locally very popular) that serves around 50K customers. My parents also use a regional bank (a Volksbank, and those are members of a national association that provides all kinds of services). Suddenly that bank would not even let them see the bank's front page. After some back and forth on the phone support line it turned out the bank had recently deployed some "advanced" bot detection, one that had a whitelist of residential-ISP-associated AS/IP ranges, and of course whoever compiled and maintained that list had forgot to include that small local ISP. For that regional bank it meant they had just shut out a very significant number of their customers (and potential customers just trying to look up what the bank offers), as there was very likely a huge overlap of people using that regional ISP and that regional bank (both are regional, after all). It also was something they couldn't fix themselves, as the "online banking" stuff was not in-house but was run by the national association (which probably used some bot-detection as a service provider). It took the bank (or rather, the national association) a few weeks to fix. Mind you, the last few years that bank has been heavily marketing a cheaper "online only" account, only online banking, online support, and access to the self-serve ATMs and banking terminals, but no face-to-face or even ear-to-ear human interaction. Try contacting "online support" about "the website outright refuses me" when the "online support" is only available on that website. Kudos is you're smart enough to switch to their mobile app, as your phone uses a different ISP, unless you forget to turn off wifi. That's the advice my parents got from the phone support (they sill have an account type that not online-only).

[1] When I recently visited my parents, from the wifi [same ISP as in 0] I couldn't open the website of a bakery too look up if and when they would be open on a Sunday. Some error message about "this website is not available in your network" (English text, for a German bakery... suspicious :P). I could open it via my mobile, tho. I could open it from my regular ISP when back home (in another city) again. Mind you, that website is purely informational and has no "interactive" features let alone let's you buy anything. It's just static text and some pictures.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#188

Earlier quoted context omitted.

Yeah, I used to work for one of the major anti-bot vendors. Customers weren't clueless. Nobody buys these solutions because they're so much fun, it's a cost center and they monitor their ROI quite closely. Credit card charge backs, impact to infrastructure, extra incurred cost due to underlying api's (like in the Airline industry in particular) etc are all reasons why bot mitigation is a better option than nothing fo…

You very much missed the false positive rate! I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech. I had to try to log in to my bank about ten times today because their js-crap website didn't like me ( grumble why does it even need to ask for my desktop's accelerometer data via js...) Stuff like this is a pain beyond pain. I rea…

Doesn't your bank already know who you are?

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#189
post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

On a site I used to run, there was no content which needed protection. So, it was not much of a pain except that there would be a lot of bot- filled contact forms. Slowly the problem became severe enough that bandwidth fees started becoming an issue. Finally had to use cloudflare in the front to reduce bandwidth usage. It worked but the side-effect was that some valid users may now get blocked.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#190
post #136

Earlier quoted context omitted.

Unfortunately, the days of reliable non-JavaScript capable scraping are over. Fortunately there are plenty of tools to handle this, and at a hobby level not particularly resource intensive. Something like this is simple and reliable in many cases: https://github.com/berstend/puppeteer-extra/tree/master/pack...

> Unfortunately, the days of reliable non-JavaScript capable scraping are over. Not really. In a lot of cases websites use JavaScript to call some API along with some on the fly generated token to prevent abuse. As long as that token isn't captcha you can reverse engineer the site to do scraping without javascript and that is so much faster than browser based scraping.

I agree with this. This is what I see on a lot of sites I scrape. Reverse engineering the JS to figure out how the fuck the token was generated is a bitch though.
Post reply on HN