* Scrape open proxy websites for open proxies, then use those proxies, cycle which proxies you use frequently.
* Change your user-agent to a real user-agent, cycle it frequently.
* Done.
Avoiding bot detection: How to scrape the web without getting blocked?
71–80 of 312 posts
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#72Earlier quoted context omitted.
The gold standard is residential IP. It is not cheap but its effectiveness is indisputable.
There are services that detect residential IPs being used for scraping nowadays. Plus there are other ways of detecting scraping: browser fingerprinting, aggressive rate-limiting and CAPTCHAs etc.
Captcha solving services are a thing, can be as crude as something that takes a screenshot, sends an image to a click farm worker getting paid $300 a month sitting in a cubicle in Bangladesh.
There's various captcha solving services where you pay in bulk per captcha and submit data via an api.
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#73Earlier quoted context omitted.
There are services that detect residential IPs being used for scraping nowadays. Plus there are other ways of detecting scraping: browser fingerprinting, aggressive rate-limiting and CAPTCHAs etc.
Captcha solving services are a thing, can be as crude as something that takes a screenshot, sends an image to a click farm worker getting paid $300 a month sitting in a cubicle in Bangladesh. There's various captcha solving services where you pay in bulk per captcha and submit data via an api.
Yup, such click farms exist. But driving up the costs and/or technical implementation efforts for bots/scrapers can be a part of your anti-bot strategy.
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#74Earlier quoted context omitted.
https://en.m.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn That’s one of the bigger ones. Unfortunately recent events means scraping is still a gray area.
Legal gray areas are perfect for growth hacking. Just look at Uber and AirBnb.
And even bigger growth hack for a lot of companies would be having scraping protected by law.
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#75Earlier quoted context omitted.
Link-only login is the most underused security option, even more so for low-profile sites that need a minimal user account but do not really need full-on security.
I’m curious. What is link-only login?
A link is generated, emailed to the user, and clicking the link logs them in.
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#76I am always amazed when otherwise intelligent people assert without data that the marginal cost of serving web traffic to scrapers/bots is zero. It is kind of like people who say "Why don't they put more fuel in the rocket so it can get all the way into orbit with just one stage?" It sounds great but it is a completely ignorant thing to say.
Anyone who has a minor website knows that majority of the traffic are bot.
Imagine if the goal is images and videos, now you've got yourself some heavy duty scraper that could cost the website owner lots of data fees.
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#77Earlier quoted context omitted.
On the contrary, there are no laws that say you can't scrape a site. If it's available to the public internet, it's legally scrapable.
There are laws against unauthorized computer access. This is a scenario where you have a server explicitly saying "Stop! You are not permitted to access this computer!", and yet you persist in circumventing that by hiding your identity and accessing it anyway. Those are some murky waters.
The server says nothing of the kind. The response that was previously positive is now broken, and it happens to be fixed if you access it from a different IP.
Maybe we need a status code that means ‘lay off all the requests made from this entire system’?
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#78I am always amazed when otherwise intelligent people assert without data that the marginal cost of serving web traffic to scrapers/bots is zero. It is kind of like people who say "Why don't they put more fuel in the rocket so it can get all the way into orbit with just one stage?" It sounds great but it is a completely ignorant thing to say.
When I worked in e-commerce as a SRE, bots were doing two things:
- trying to disrupt business processes (eg: false referral listings, gift card scams, etc)
- trying to disrupt systems
I'm sure there are folks who use bots and scrapers for home automation, but these users generate marginal traffic in comparison. The real cost, aside from successfully achieving the points above, is the bandwidth and hardware costs that become overhead. Bots are usually coded with retry mechanisms and ways to change connection criteria on subsequent retries.
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#79> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…
2FA should be a requirement on everything now. And if your site can't for some reason or you don't want to deal with it, then limit your site to external login providers only. 2FA, especially app based, has been proven to work really really well.
thats great till you're in a foreign country and your phone suddnely decides to die leaving you stranded and unable to access bank accounts or prove your identity. (happened to me)
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#80Earlier quoted context omitted.
Legitimate uses of scraping include price comparison, and finding copyright or trademark infringement.
>Legitimate uses of scraping include price comparison "Legitimate uses" is what the site operator says it is, nothing more nothing less. There are no laws that says you can scrape a site and circumvent their protection against doing so.
If you look at Ryanair, they've had legal battles with scrapers for more than 10 years, so it's not that simple.