61–70 of 196 posts
Earlier quoted context omitted.
I'm right there with you. I'm the lead engineer for an automotive SaaS provider (with ~6000 customers and ~4 billion requests per month) and we recently started moving all our services to Cloudflare's WAF to take advantage of their bot protection. We were getting scrapes from botnets in the 100000+ per minute range that was affecting performance. We chose to switch to the JS challenge screen as it requires no human i…
What were they scraping, if I can ask? Was it targeted or just wget -r style?
The irony for us as a provider is that it's one of our customers (party A) paying a third party to scrape data from another one of our customers (party B) which in turn affects the performance of party A's site. We've started blocking these third parties and directing them to paid APIs that we offer.
Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…
I'm right there with you. I'm the lead engineer for an automotive SaaS provider (with ~6000 customers and ~4 billion requests per month) and we recently started moving all our services to Cloudflare's WAF to take advantage of their bot protection. We were getting scrapes from botnets in the 100000+ per minute range that was affecting performance. We chose to switch to the JS challenge screen as it requires no human i…
Earlier quoted context omitted.
I'm right there with you. I'm the lead engineer for an automotive SaaS provider (with ~6000 customers and ~4 billion requests per month) and we recently started moving all our services to Cloudflare's WAF to take advantage of their bot protection. We were getting scrapes from botnets in the 100000+ per minute range that was affecting performance. We chose to switch to the JS challenge screen as it requires no human i…
I'm really surprised that the JS challenges helped so much, given that there are open source libraries for bypassing them (e.g. cloudscraper[0]). [0]: https://github.com/venomous/cloudscraper
I used to lead Sys Eng for a FTSE 100 company. Our data was valuable but only for a short amount of time. We were constantly scraped which cost us in hosting etc. We even seen competitors use our figures (good ones used it to offset their prices, bad ones just used it straight). As the article suggest, we couldn't block mobile operator IPs, some had over 100k customers behind them. Forcing the users to login did litt…
Earlier quoted context omitted.
Why would Amazon wish to provide you with easy to access data on their products and prices when you could either be a competitor wishing to undercut those prices, or be a scraper company hired by such a competitor? In what universe is providing such a straightforward way of helping a competitor considered sane business practice?
Man your comment is hilarious because in fact Amazon DOES provide an API for exactly that
> We've once had to scrape Amazon product pages because they have a lot of API endpoints, but those didn't contain the data we needed.
...only a couple of comments up.
Earlier quoted context omitted.
> You can think of it this way, the prices and product data are publicly visible already on the website, there are no real secrets, none of it is password protected. There's the problem right there. The prices and product data are publicy visible - because there is a target audience of /humans/ for whom the site is designed and intended to be used by. The site is not there to cater for a competitor's scrapers. I don'…
I agree 100%, but it is a fact of life, and sometimes it's better to just minimize the fuzz and focus on the things that matter. Your argument is perfectly valid and applies to offline activities as well (what stops a competitor from walking through the aisles of a Walmart or Costco?), but this is a battle that can't be won, there are too many parasitic actors. It is human nature.
That's a significant portion of Nielsen's business model.
We use it for data-entry on a government website. A human would average around 10 minutes of clicking and typing, where the bot takes maybe 10 seconds. Last year we did 12000 entries. Good bot.
Earlier quoted context omitted.
Well, you don't need an api, just a CSV file with a catalog. The scraping company WILL use the API/CSV file... they will probably also still charge their customer for scraping, so it's a win-win :D You can think of it this way, the prices and product data are publicly visible already on the website, there are no real secrets, none of it is password protected. You can be principled and insist on blocking bots and spen…
> You can think of it this way, the prices and product data are publicly visible already on the website, there are no real secrets, none of it is password protected. There's the problem right there. The prices and product data are publicy visible - because there is a target audience of /humans/ for whom the site is designed and intended to be used by. The site is not there to cater for a competitor's scrapers. I don'…
But overall, information is one of those goods that has intrinsic properties like no other. It can be copied, infinitely. And we haven't yet figured out the dynamics of how to reason about it, so it feels like we're pretending they're physical goods.
Edit. Side note. I'd go further and say that some of the data is even worse, it's "offered" with the real intention being to confuse the users into performing non-optimally in the market. Look at Amazon/Ebay/AliExpress/Google listings for evidence of that. Just Google - Google is a ML and scraping power house, and the best they can muster is to be spammed with fake websites and duplicate/confusing listings.
Earlier quoted context omitted.
Well, you don't need an api, just a CSV file with a catalog. The scraping company WILL use the API/CSV file... they will probably also still charge their customer for scraping, so it's a win-win :D You can think of it this way, the prices and product data are publicly visible already on the website, there are no real secrets, none of it is password protected. You can be principled and insist on blocking bots and spen…
> You can think of it this way, the prices and product data are publicly visible already on the website, there are no real secrets, none of it is password protected. There's the problem right there. The prices and product data are publicy visible - because there is a target audience of /humans/ for whom the site is designed and intended to be used by. The site is not there to cater for a competitor's scrapers. I don'…
There's a whole ethical subthread here of websites trying to making the experience for those humans miserable, and taking away the agency necessary to protect oneself from that. A browser is a user agent. So is a screen reader. So is a script one writes to not deal with bullshit fluff, when all one wants is a simple table of products, features and prices.