51–60 of 123 posts
"How do I stop all these dinner guests from eating this lovely pie I set out on the table?" I remember working hard on a project for a year, then releasing the data and visualizations online. I was very proud. It was very cool. Almost immediately, we saw grad students and research assistants across the globe scraping our site. I started brainstorming clever ways to fend off the scrapers with a colleague when my boss…
The company I work for does a large amount of scraping of partner websites, with whom we have contracts that allow us to do it and that someone in their company signed off, but we still get blocked and throttled by tech teams who think they are helping by blocking bots. If we can't scrape a site we just turn off the partner, and that means lost business for them.
I've built crawlers that retrieve billions of web pages every month. We had a whole team working modifying the crawlers to resolve website changes, to reverse engineer ajax requests and solve complex problems like captcha solvers. Bottom line, if someone wants to crawl your website they will. What you can do, however, is make it hard so that the vast majority of developers can't do it (e.g. My tech crawl billions of…
Do you feel bad at all about apparently making a business out of crawlers, but still apparently viewing it as bad enough that you want countermeasures against it? Don't you feel a slight bit hypocritical about this?
One of the alternatives is charging for my service but bots are not my users problem, they are mine.
http://www.inmotionhosting.com/support/website/restricting-b...
I've built crawlers that retrieve billions of web pages every month. We had a whole team working modifying the crawlers to resolve website changes, to reverse engineer ajax requests and solve complex problems like captcha solvers. Bottom line, if someone wants to crawl your website they will. What you can do, however, is make it hard so that the vast majority of developers can't do it (e.g. My tech crawl billions of…
> I've built crawlers that retrieve billions of web pages every month. Wow, what were you doing with the data?
Crawling thousands of website, mashing up the data to analyze competitiveness between them, and selling it back.
For example, cost of flights. Different websites provide different prices for the same flight. The technology crawls all the prices, combines the data, then resells it back to the websites. Everyone knows everyones prices, keeps competition high, lower prices for consumers.
I've built crawlers that retrieve billions of web pages every month. We had a whole team working modifying the crawlers to resolve website changes, to reverse engineer ajax requests and solve complex problems like captcha solvers. Bottom line, if someone wants to crawl your website they will. What you can do, however, is make it hard so that the vast majority of developers can't do it (e.g. My tech crawl billions of…
BTW, interested in learning a bit more about your stack, we are on the same route but at smaller scale.
"How do I stop all these dinner guests from eating this lovely pie I set out on the table?" I remember working hard on a project for a year, then releasing the data and visualizations online. I was very proud. It was very cool. Almost immediately, we saw grad students and research assistants across the globe scraping our site. I started brainstorming clever ways to fend off the scrapers with a colleague when my boss…
I run a property website that lists properties for sale, rent, etc. A big part of my job is importing feeds, scraping sites (with permission) - and preventing others scraping our site. I know that some people will scrape, but I make I try to make it unprofitable for them to do so. We do a bunch of other stuff, like adding fake properties so we can check who is scraping our content, and using tarpits. Developers alway…
If you want to charge rent (via a subscription service etc), then do that, and be clear that you're in the business of charging rent. Don't conflate selling with renting - that just leads to a product gimping death spiral.
One thing I've thought about but never had the chance to put into practice would be to randomize CSS classes and IDs. Most web scraping relies on these to identify the content they are looking for. Imagine if everyday they changed? It would make things a lot more difficult. There would be disadvantages to actual users with this method like caching wouldn't work very well but maybe this alternative site could be displ…
Think about it this way: you could render the whole page as a JPEG with no computer readable text, but someone could still ocr it.
One thing I've thought about but never had the chance to put into practice would be to randomize CSS classes and IDs. Most web scraping relies on these to identify the content they are looking for. Imagine if everyday they changed? It would make things a lot more difficult. There would be disadvantages to actual users with this method like caching wouldn't work very well but maybe this alternative site could be displ…