81–90 of 123 posts
I've built crawlers that retrieve billions of web pages every month. We had a whole team working modifying the crawlers to resolve website changes, to reverse engineer ajax requests and solve complex problems like captcha solvers. Bottom line, if someone wants to crawl your website they will. What you can do, however, is make it hard so that the vast majority of developers can't do it (e.g. My tech crawl billions of…
Earlier quoted context omitted.
> I've built crawlers that retrieve billions of web pages every month. Wow, what were you doing with the data?
Competitive intelligence. Crawling thousands of website, mashing up the data to analyze competitiveness between them, and selling it back. For example, cost of flights. Different websites provide different prices for the same flight. The technology crawls all the prices, combines the data, then resells it back to the websites. Everyone knows everyones prices, keeps competition high, lower prices for consumers.
Earlier quoted context omitted.
I run a property website that lists properties for sale, rent, etc. A big part of my job is importing feeds, scraping sites (with permission) - and preventing others scraping our site. I know that some people will scrape, but I make I try to make it unprofitable for them to do so. We do a bunch of other stuff, like adding fake properties so we can check who is scraping our content, and using tarpits. Developers alway…
You are deliberately devoting energy and resources towards removing value that already existed in your product. If you want to charge rent (via a subscription service etc), then do that, and be clear that you're in the business of charging rent. Don't conflate selling with renting - that just leads to a product gimping death spiral.
I've built crawlers that retrieve billions of web pages every month. We had a whole team working modifying the crawlers to resolve website changes, to reverse engineer ajax requests and solve complex problems like captcha solvers. Bottom line, if someone wants to crawl your website they will. What you can do, however, is make it hard so that the vast majority of developers can't do it (e.g. My tech crawl billions of…
How do you bypass google recaptcha
I assume you are concerned about crawlers that do not respect the robots.txt file (which is the polite way to restrict them from indexing your side, but does not provide any actual protection if crawlers chose to ignore the file). Cloudflare has a tool for doing this (now part of their core service): https://blog.cloudflare.com/introducing-scrapeshield-discove... There's a nice Github repo with some advice on blockin…
The going rate for CAPTCHA solving is about 1/10 of a USD penny.
Earlier quoted context omitted.
This is what we used to do. Then send a large zipfile with schreenshots and other data to the lawyers to handle the contact. Shortly after the scraping usually stopped. The contact and sell access wasnt an option because it was competitors taking the data.
I did some scraping for a lawyer back in like 01 from other lawyers. He got a c&d and told me to turn it off (we were done anyway). Funny part was the lawyer on the other side wanted us to return all of the content on disk. Not show what we had copied but literally return it. My lawyer laughed about it. The other lawyer was smart/savvy enough to be effectively using the internet in 01 but didn't really understand the…
I actually had a client that asked me to record a screenshot of me deleting their files from my computer. (and this was actually a developer making the request)
One thing I've thought about but never had the chance to put into practice would be to randomize CSS classes and IDs. Most web scraping relies on these to identify the content they are looking for. Imagine if everyday they changed? It would make things a lot more difficult. There would be disadvantages to actual users with this method like caching wouldn't work very well but maybe this alternative site could be displ…
Don't prevent them. The same data you let humans access for free should be accessible via bots. If you only want to give out a "reasonable" amount of data, that humans wouldn't usually exceed but bots would, then define a rate-limit that wouldn't inconvenience humans and then apply it for everyone - bot or not. That way you're discriminating based on the amount of data instead of whether it's a bot or not. It will th…
They kill your bandwidth. For a client's catalog site we discovered crawlers were more than half of the used bandwidth costs.
My solution would solve the issue in a fair way by having a "reasonable usage" limit applied to everyone, bot or not. This also means it can't be defeated by someone paying humans to do the dirty work to bypass bot restrictions.
Earlier quoted context omitted.
You are deliberately devoting energy and resources towards removing value that already existed in your product. If you want to charge rent (via a subscription service etc), then do that, and be clear that you're in the business of charging rent. Don't conflate selling with renting - that just leads to a product gimping death spiral.
Value to whom? If a competitor scrapes all my content and then gets the leads instead of me, they've removed value from my bank account, and that's the main value I'm concerned about. This is not a hypothetical concern - it happens a lot.
To me, it sounds like your business lacks some fundamentals.
This is for Drupal sites. It has a strong firewall (csf) and it has a lot of crawler detections on the nginx configurations. It checks the load and when on high load it blocks the crawlers.