Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoiding bot detection: How to scrape the web without getting blocked?

github.com

Post reply on HN
Print view View original

201–210 of 312 posts

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#201

Post by dzhiurgis »

oxymoron wrote: post #15
bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

Yeah, I used to work for one of the major anti-bot vendors. Customers weren't clueless. Nobody buys these solutions because they're so much fun, it's a cost center and they monitor their ROI quite closely. Credit card charge backs, impact to infrastructure, extra incurred cost due to underlying api's (like in the Airline industry in particular) etc are all reasons why bot mitigation is a better option than nothing fo…

Do you know much about airline api pricing more?

I’ve noticed most sites won’t let you search business fares efficiently, so I made my own for Google Flights which only worked for like 6months until they added bunch of changes that made it near impossible to scrape.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#202

Post by cute_boi »

walrus01 wrote: post #61
TedDoesntTalk wrote: post #41

Earlier quoted context omitted.

Not anymore. Now it’s mobile IP addresses.

One of the reasons for this is that the vast majority of the time, mobile LTE data users are behind cgnat for ipv4. You can't block one ip without possibly blocking hundreds of innocent IPs using the same exit point. As a scraper operator on a mobile data connection all you need is a new useragent and browser fingerprint, there's no easy way for a scraper-blocker-operator to tell that you're not a totally new person.

This is reason why most of services uses App instead of Browser. When there is App it can use lot of thing like phone fingerprint derived from various sources.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#203

Post by kingcharles »

zarzavat wrote: post #168
dpryden wrote: post #88

It always amazes me how people believe they have a right to retrive data from a website. The HTTP protocol calls it a request for a reason: you are asking for data. The server is allowed to say no, for any reason it likes, even a reason you don't agree with. This whole field of scraping and anti-bot technology is an arms race: one side gets better at something, the other side gets better at countering it. An arms rac…

Because often that data is only available through scraping. Nobody wants to scrape, it's messy and fickle and a general pain in the backside. But sometimes the data you need exists only in that form. If you run a website and you have a problem with scrapers, then make all that data available through an API and say what acceptable rate limits are. If cost is an issue, then charge a proportionate fee, my time writing a…

God. This. The number of times I've spent 2 days of my very expensive time coding a scraper to get data I'll use once, when I would have paid a few dollars just to download it in a text file.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#204

Post by nextaccountic »

bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

> Nobody pays those vendors $10m/year to frustrate web crawler enthusiasts, they do it to stop credential stuffing.

I don't know about $10m/year, but many sites block bots just because they don't want competitors to access publicly available data. Which is bullshit.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#205

Post by geysersam »

arp242 wrote: post #158
temp8964 wrote: post #148

Earlier quoted context omitted.

I really don't believe "The site owner decides what you can and cannot do." statement. What is the base of this? This does not seem to apply to anything in the real world. In most cases you have very limited ability to decide what other people cannot do. And other people has mostly infinite choices of what they can do. I never heard anything as broad as you said. What you said is like a person standing on the street…

> I really don't believe "The site owner decides what you can and cannot do." statement. What is the base of this? This does not seem to apply to anything in the real world. It kind of does though; if I own a store and say that only people with hats can enter, then I'm free to do so. Silly? Yes. Legal? Also yes. There are some circumstances where it's not legal, mostly centred around discrimination. Details on this d…

Interesting discussion.

To me it seems sending a http request is somewhere inbetween looking (legal) and entering (illegal if not permitted).

However, most important is that the web as a system makes positive interactions easy and negative difficult. We have already found some set of constraints achieving this for interactions in public city streets. But it's not obvious the same rules (that we have internalized) have the same effect in another medium of communication.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#206

Post by defer »

fragmede wrote: post #157
jameshart wrote: post #116

Earlier quoted context omitted.

Bots aren't just trying credential stuffing. They are: - committing clickfraud to game ad and referral revenue systems - posting fake or spam reviews and comments - generating fake behavioral signals to help bypass CAPTCHAs to help create accounts on other sites that can post spam comments - validating stolen credit card details - screwing with your metrics collection if you can't identify them as bots All of that is…

> -validating stolen credit card details To be clear, "validating" is an industry euphemism for stealing, just for a different purpose. How do you validate the card is live? Run a real transaction through it and mark it based on the result. But what do you run for this real transaction? Well, whatever you want. Typically it'll be something to avoid suspicion as much as possible, but the thief gets to pick what they t…

Not exactly. Some people are in the business of gathering and selling valid credit cards.

They won't cash out on them or buy items. Instead, they'll collect cards from a source (skimming, hacking, whatever), validate them by adding them to a website that does an authorization (those $1 checks that never get committed). They can then sell them wholesale for a premium compared to non-verified cards.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#207

Post by maven29 »

rndgermandude wrote: post #187
jameshart wrote: post #116

Earlier quoted context omitted.

Bots aren't just trying credential stuffing. They are: - committing clickfraud to game ad and referral revenue systems - posting fake or spam reviews and comments - generating fake behavioral signals to help bypass CAPTCHAs to help create accounts on other sites that can post spam comments - validating stolen credit card details - screwing with your metrics collection if you can't identify them as bots All of that is…

All very valid reasons (except to a degree the "screwing with metrics" one). There a lot of websites which do not really face any of the aforementioned issues, simply because they do not sell you anything, are not an ad network or run referral programs, do not even have user-generated content, etc. And even the sites that do, it's usually a rather small part of the "surface" that needs such protections e.g. the actua…

It doesn't help that VPN providers in the consumer space pay small ISPs to front their traffic so that netflix et al doesn't drop their traffic coming from a datacenter and they still get to keep up the false pretenses and get valued at billions of dollars.

Smaller ISPs are also more likely to have issues with CPE getting compromised and routers running botnets within the comfort of your home. There are services which invite people to sell their residential bandwidth in return for money, this can potentially have a disproportionate impact at smaller sample sizes.

Networks can declare themselves to be ISPs. You could check if your ISP shows up as an ISP in peeringDB.

https://networkengineering.stackexchange.com/questions/44585...

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#208

Post by at_compile_time »

Mister_Snuggles wrote: post #151
cebert wrote: post #133

Earlier quoted context omitted.

Have you considered using Playwright to automate that instead?

I had never heard of this, but it looks like a reasonable option. My go-to for this type of thing would be Python+Selenium+Firefox, but only due to familiarity with those.

I'm glad that I'm not the only one not using Selenium for its stated purpose.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#210

Post by TedDoesntTalk »

cute_boi wrote: post #200
varenc wrote: post #54

Earlier quoted context omitted.

This! Mobile IPs are far more lucrative. Many services will drop captchas and other anti-bot stuff for consumer mobile IPs. I recall Plaid at some point would run their bank scraping through mobile IPs. This sketchy company lets mobile app developers monetize user base by letting other people pay $$ to route requests through random people’s mobile IPs: https://brightdata.com/

This is really bad. Imagine if someone plants these proxy inside app how user are even going to know? I think every OS should come with firewall so if app tries to make connection it should prompt with Accept | Accept Forever | Deny | Deny Forever. I think these companies used to go for extension developer now it seems they have found new idea to implant malware on apps which is not easy to detect.

This is EXACTLY how mobile proxy companies like Luminati and OxyLabs acquire their ip address pool. They pay devs to embed a lib inside their app.
Post reply on HN