Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoiding bot detection: How to scrape the web without getting blocked?

github.com

Post reply on HN
Print view View original

121–130 of 312 posts

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#121

Post by nieve »

toomuchtodo wrote: post #75
catlifeonmars wrote: post #66

Earlier quoted context omitted.

I’m curious. What is link-only login?

A link is generated, emailed to the user, and clicking the link logs them in.

I.e. what Facebook does if you don't log in for long enough. Two days ago I got a pair of messages to the same address with links to completely bypass login and verbiage about how they'd seen I was having trouble logging in followed an sms message with the same to a phone number they're not supposed to be using. It looks a lot like phishing, but it comes out of Facebook's servers and they've done it to me before.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#122

Post by calvinmorrison »

toomuchtodo wrote: post #75
catlifeonmars wrote: post #66

Earlier quoted context omitted.

I’m curious. What is link-only login?

A link is generated, emailed to the user, and clicking the link logs them in.

So SSO but you have to trust the email provider instead of another random SaaS

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#124

Post by jhgb »

marginalia_nu wrote: post #83
Aeolun wrote: post #77

Earlier quoted context omitted.

The server says nothing of the kind. The response that was previously positive is now broken, and it happens to be fixed if you access it from a different IP. Maybe we need a status code that means ‘lay off all the requests made from this entire system’?

How do you interpret a 401 Unauthorized to mean you are authorized to access the resource?

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#cli...

> Although the HTTP standard specifies "unauthorized", semantically this response means "unauthenticated". That is, the client must authenticate itself to get the requested response.

So it would seem that it actually doesn't positively imply that you're NOT authorized.

Which kind of makes sense; machines can't detect legality of things, just that certain procedural niceties haven't been observed.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#125

Post by ChuckMcM »

ryandrake wrote: post #110
abadger9 wrote: post #102

I'm a lead engineer on the search team of a publicly traded company who's bread and butter is this domain. I was curious about this list, it candidly misses the mark- the tech mentioned in this blog is what you might get if you hired a competent consultant to build out a service without having domain knowledge. In my experience, what's being used on the bleeding edge is two steps ahead of this.

Do you have any factual corrections? Your post reminds me of those "I'm getting a kick out of these replies" copypasta--declaring someone wrong and claiming authoritative knowledge, but without actually correcting any of the errors of fact.

Not the OP but when I was running Operations at Blekko (a search engine) I spent part of my time dealing with scrapers.

When treated like a puzzle it can be really interesting. So I thought I'd share a few tidbits.

1) We did a simple 'speed' test, how many queries per second were coming from an IP, and auto-ban on the limit being exceeded, started at 100qps and watched as the traffic moved down to 99.5qps. Pushed to 10qps and watched the traffic follow it down. Even at 3qps you would get traffic at bit over 2 qps trying to limbo in under the limit.

2) At that time, lots of people who highjacked browsers with toolbars sold scraping as a service to third parties. Their toolbar would check in to see if it should do a query and it would launch a query and return the results without the user even knowing. One company, 80 legs, was pretty up front about their "service", SEO types would use it to scrape Google results to see how their SEO campaigns were doing.

3) The majority of the traffic had criminal intent, looking for metadata on web pages to indicate they were running an unpatched version of some store software or had sql injection bugs. These would often come from PCs that had been compromised for other purposes or "zombie" PCs. We could rapidly map out these networks when we got 100 queries from 100 different IPs looking for "joomla version x.y",p=1 through "joomla version x.y",p=100. We briefly played around with sending them official looking SERPs but all the links went to fbi.gov though an obfuscator.

One of more effective strategies was to field a "black hole" server, basically it was an http server that answered like you had gotten hold of it but then it never sent any data. With some simple kernel mods these TCP connections were silently removed on our end so they took no resources and the client would wait basically forever. We ack'd all keep alive packets with "Yup, we're here." so they just kept waiting and waiting.

It really was a never ending game. We mass banned an entire Ukranian ISP because out of billions of queries not a single one was legitimate.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#126

Post by kristjansson »

marginalia_nu wrote: post #48
soheil wrote: post #40

Earlier quoted context omitted.

Google can access any site without being blocked. They dominate the search space and give little incentive for site owners to allow other bots. I'd say bypassing these measures is fair game while there is a monopoly in search space. We don't want a web that only Google can access. By the way great work on Marginalia search engine, I love it.

I've honestly not had much problem at all crawling the web as an indie search engine operator. If you want to get past CloudFlare you can register your bot fingerprint with them. A small number of sites has blocked my crawler , but that's almost always been my own fault, and happened a few instances when the crawler was misbehaving and actually fetching too aggressively (or repeatedly). In every case just sending an…

There's also the alignment of interests between you and the site operators, even for a small search engine. Cooperation improves outcomes for both sides. Whereas the more aggressive scrapers almost certainly have interests that conflict with the site operators, regardless of the costs to serve the traffic itself.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#127

Post by EMIRELADERO »

dragonwriter wrote: post #62
EMIRELADERO wrote: post #58

Earlier quoted context omitted.

On the contrary, there are no laws that say you can't scrape a site. If it's available to the public internet, it's legally scrapable.

> On the contrary, there are no laws that say you can't scrape a site. You are both wrong: copyright law both says you can't (in some cases for some uses) and that you can (under implicit license, fair use, and other rules) in others.

Depends on what exactly is being scraped. If it's something like price data or exact values then it isn't protected by copyright at all.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#128

Post by marginalia_nu »

jhgb wrote: post #124
marginalia_nu wrote: post #83

Earlier quoted context omitted.

How do you interpret a 401 Unauthorized to mean you are authorized to access the resource?

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#cli... > Although the HTTP standard specifies "unauthorized", semantically this response means "unauthenticated". That is, the client must authenticate itself to get the requested response. So it would seem that it actually doesn't positively imply that you're NOT authorized. Which kind of makes sense; machines can't detect legality of things, just that certain…

Fine, send a 403 then.

> The client does not have access rights to the content; that is, it is unauthorized, so the server is refusing to give the requested resource.

Machines don't have any legal responsibility, bot-operators do. Which is why respecting these things is sort of important. At any rate, 40x does not mean "try again with a different user agent and another IP"

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#129

Post by eppp »

walrus01 wrote: post #59
Mister_Snuggles wrote: post #31

Earlier quoted context omitted.

I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…

Out of curiosity how is that you have electricity rates that change every month? Are you buying power through a third party organization? The vast majority of place I've seen have a fixed tariff for residential use that changes no more often than every 12-24 months.

TVA in the United States has a variable rate per month due to fuel surcharges.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#130

Post by Scoundreller »

ChuckMcM wrote: post #125
ryandrake wrote: post #110

Earlier quoted context omitted.

Do you have any factual corrections? Your post reminds me of those "I'm getting a kick out of these replies" copypasta--declaring someone wrong and claiming authoritative knowledge, but without actually correcting any of the errors of fact.

Not the OP but when I was running Operations at Blekko (a search engine) I spent part of my time dealing with scrapers. When treated like a puzzle it can be really interesting. So I thought I'd share a few tidbits. 1) We did a simple 'speed' test, how many queries per second were coming from an IP, and auto-ban on the limit being exceeded, started at 100qps and watched as the traffic moved down to 99.5qps. Pushed to…

> One of more effective strategies was to field a "black hole" server, basically it was an http server that answered like you had gotten hold of it but then it never sent any data. With some simple kernel mods these TCP connections were silently removed on our end so they took no resources and the client would wait basically forever. We ack'd all keep alive packets with "Yup, we're here." so they just kept waiting and waiting.

Mailinator would do a similar thing with their custom email server hardware. Since they didn't really use sockets in the traditional sense, they were happy to give slooow replies and never disconnect "bad" connections.

Post reply on HN