If I were in OSVDB's shoes, I would call these people out in an email and ask them to pay a licencing fee. McAfee have always had a shady past, even when they shook themselves clean of John, they have a history of scam-like behaviour to make a quick-buck. You should be rate-limiting how many requests free API users can make, like; Twitter, Facebook and every other Internet provider does via their API. Make it harder…
The Scraping Problem and Ethics
11–20 of 130 posts
Re: The Scraping Problem and Ethics
#12https://web.archive.org/web/20130714002216/http://www.osvdb....
I guess they were hoping no one would notice the subtle but substantial change to their service
Re: The Scraping Problem and Ethics
#13If I were in OSVDB's shoes, I would call these people out in an email and ask them to pay a licencing fee. McAfee have always had a shady past, even when they shook themselves clean of John, they have a history of scam-like behaviour to make a quick-buck. You should be rate-limiting how many requests free API users can make, like; Twitter, Facebook and every other Internet provider does via their API. Make it harder…
On the other hand, large (even non-profit) organisations are precisely the ones who have the resources to scrape steathily and widely, as would a loosely-organised community of users... it's not hard to come up with algorithms to respect the rate limits, balancing the load across multiple IPs and accounts, and producing access patterns that don't look any different from the rest of the site traffic.
At one point the effort to circumvent would cost more in man-hours than just buying the product.
Re: The Scraping Problem and Ethics
#14Re: The Scraping Problem and Ethics
#15This doesn't even rise to the level of what Weev did.
Re: The Scraping Problem and Ethics
#16Earlier quoted context omitted.
Did you read the article? In both cases the offending parties were told to contact RBS to obtain a proper license/account.
That's exactly the problem that GP is talking about. McAfee were just being the usual assholes, but the first guy mentioned in the blog post could have been converted into a paying customer if the pricing scheme were clearly outlined on the web. Since "Contact us" account tiers are usually reserved for the very high end, he probably assumed that it would cost him an arm and a leg. It shouldn't be too difficult to com…
I understand the sales psychology in making sure you enter into a proper discussion with people to make sure their needs are right, and extracting the maximum consumer surplus from them. But there is a non-trivial pricing point where this makes sense, and by not having any anonymous-sign up, you're cutting off your sales curve below this point.
I think it's very easy to convince yourself it's easier just to sign up customers over the phone, but doing so without at least testing takeup of simpler tiers is an incomplete picture.
Re: The Scraping Problem and Ethics
#17Shouldn't you at least be disallowing /show/* in robots.txt? Not that scrapers are necessarily going to respect this... but the way your set up it seems like this is semi-legit behavior.
Re: The Scraping Problem and Ethics
#18Re: The Scraping Problem and Ethics
#19Web scraping isn't a crime - the simple act of downloading the data should not be a problem here. (The reuse of the data might be, depending, but we don't have that information right now.) This doesn't even rise to the level of what Weev did.
Re: The Scraping Problem and Ethics
#20This is straight from the open security foundation website:
"We believe that security information and services should be easily accessible for all who have the need for such information and services"