Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoiding bot detection: How to scrape the web without getting blocked?

github.com

Post reply on HN
Print view View original

111–120 of 312 posts

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#111

Post by ComputerGuru »

ryandrake wrote: post #110
abadger9 wrote: post #102

I'm a lead engineer on the search team of a publicly traded company who's bread and butter is this domain. I was curious about this list, it candidly misses the mark- the tech mentioned in this blog is what you might get if you hired a competent consultant to build out a service without having domain knowledge. In my experience, what's being used on the bleeding edge is two steps ahead of this.

Do you have any factual corrections? Your post reminds me of those "I'm getting a kick out of these replies" copypasta--declaring someone wrong and claiming authoritative knowledge, but without actually correcting any of the errors of fact.

Yes, but it’s also understandable why they wouldn’t want to expand.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#113

Post by jrockway »

spookthesunset wrote: post #60
walrus01 wrote: post #55

Google "residential proxies for sale" if you want to see the weird shady grey market for proxies when you need your traffic to come from things like cablemodem operator ASNs' DHCP pools

Wonder what fraction of that traffic is from p0wned IoT refrigerators, smoke detectors or WiFi enabled light bulbs… probably more than anybody cares to admit…

Some amount, probably, but some of these "residential IP" providers just buy IP blocks from small residential or business ISPs. (Or even less shadily, buy IP transit from them and get assigned a block of IPs that the databases think are normal residential/business users.)

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#114

Post by hcta »

marginalia_nu wrote: post #36

If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally questionable, a serious pain in the ass for the site owner and makes websites more prone to attempt to block bots in general. Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching…

I'm a bot, you're a bot, we are all bots, so what does this even mean. If someone is trying to discriminate between one set of visitors that they welcome and another that they deny service to, by definition they are the ones creating an adversarial environment. They are going to have to work to get the terms they want. Asking the players on the opposite side of the table to comply with their imagined rules out of altruism is a non-starter.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#115

Post by abadger9 »

ryandrake wrote: post #110
abadger9 wrote: post #102

I'm a lead engineer on the search team of a publicly traded company who's bread and butter is this domain. I was curious about this list, it candidly misses the mark- the tech mentioned in this blog is what you might get if you hired a competent consultant to build out a service without having domain knowledge. In my experience, what's being used on the bleeding edge is two steps ahead of this.

Do you have any factual corrections? Your post reminds me of those "I'm getting a kick out of these replies" copypasta--declaring someone wrong and claiming authoritative knowledge, but without actually correcting any of the errors of fact.

Sorry about that, we're unable to discuss our projects with adjacent teams within the company. What you're saying is a valid frustration, the motivation behind the original comment was to put a thermometer on the repo.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#116

Post by jameshart »

bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

Bots aren't just trying credential stuffing. They are:

- committing clickfraud to game ad and referral revenue systems

- posting fake or spam reviews and comments

- generating fake behavioral signals to help bypass CAPTCHAs to help create accounts on other sites that can post spam comments

- validating stolen credit card details

- screwing with your metrics collection if you can't identify them as bots

All of that is enough reason for sites to use bot detection and blocking technology. The fact that the same tech also has some utility against accidental or malicious traffic-based DoS is also a bonus.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#117

Post by jrockway »

intricatedetail wrote: post #96

By the way - is it possible to stop Google bot from scrapping without maintaining a list of IP addresses? Google doesn't publish these and it's not good to run reverse DNS as it slows down legitimate clients. I know you can put a meta tag, but bot still has to make a request to read it. I would like to completely cut off Google from scrapping.

You can buy databases of who owns which IP blocks.

If you really care but don't want to spend the money, just block the subnet each time you see a Googlebot request. "whois w.x.y.z" returns an entire CIDR, and it seems unlikely to me that Google is scraping from a bunch of disconnected /24s.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#118

Post by marginalia_nu »

gzer0 wrote: post #107
marginalia_nu wrote: post #48

Earlier quoted context omitted.

I've honestly not had much problem at all crawling the web as an indie search engine operator. If you want to get past CloudFlare you can register your bot fingerprint with them. A small number of sites has blocked my crawler , but that's almost always been my own fault, and happened a few instances when the crawler was misbehaving and actually fetching too aggressively (or repeatedly). In every case just sending an…

How did you go about registering your bot fingerprint with Cloudflare? I did a quick search but cannot find anything with regards to that.

Good question, I can't find the link now either. I found it somewhere on Cloudflare's site.

--edit--

Nevermind, here it is: https://support.cloudflare.com/hc/en-us/articles/36003538743...

It should skip to "I run a good bot...", but if it doesn't, that's where you want to scroll.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#119

Post by rstupek »

kodah wrote: post #78
ChuckMcM wrote: post #43

I am always amazed when otherwise intelligent people assert without data that the marginal cost of serving web traffic to scrapers/bots is zero. It is kind of like people who say "Why don't they put more fuel in the rocket so it can get all the way into orbit with just one stage?" It sounds great but it is a completely ignorant thing to say.

When I worked in e-commerce as a SRE, bots were doing two things: - trying to disrupt business processes (eg: false referral listings, gift card scams, etc) - trying to disrupt systems I'm sure there are folks who use bots and scrapers for home automation, but these users generate marginal traffic in comparison. The real cost, aside from successfully achieving the points above, is the bandwidth and hardware costs tha…

Or bots using ecommerce sites to test credit card validity for use elsewhere

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#120

Post by jsnell »

ryandrake wrote: post #110
abadger9 wrote: post #102

I'm a lead engineer on the search team of a publicly traded company who's bread and butter is this domain. I was curious about this list, it candidly misses the mark- the tech mentioned in this blog is what you might get if you hired a competent consultant to build out a service without having domain knowledge. In my experience, what's being used on the bleeding edge is two steps ahead of this.

Do you have any factual corrections? Your post reminds me of those "I'm getting a kick out of these replies" copypasta--declaring someone wrong and claiming authoritative knowledge, but without actually correcting any of the errors of fact.

Abuse is the kind of problem area where anyone seriously working on either side will be unlikely to go into the details. It's the same whether it's blocking scraping, stopping spam, preventing account takeovers, or detecting payment fraud.

In this specific case, the people wanting to detect bots want to avoid having their signals burned, the scrapers don't want the defenders to know which signals they're able to cloak since it will spur new signals development. So what gets disclosed publicly is just the really simple stuff.

It's kind of sad. There's a huge pipeline for getting people up to speed on security engineering, since there's a lot of incentive for everyone in the ecosystem to share information as publicly as possible except for relatively short responsible disclosure windows.

In contrast, the only way to learn abuse engineering is to happen to work in an organization with an abuse problem (and live with the frustration of an abnormally long ramp up period), or to go black hat. And likewise it's quite hard for the good guys to actually learn from each other, since they're spread across so many companies and it's thus hard for them to exchange information on what works and what doesn't.

Post reply on HN