Live data from Hacker News

The Scraping Problem and Ethics

blog.osvdb.org

11–20 of 130 posts

Re: The Scraping Problem and Ethics

#11

If I were in OSVDB's shoes, I would call these people out in an email and ask them to pay a licencing fee. McAfee have always had a shady past, even when they shook themselves clean of John, they have a history of scam-like behaviour to make a quick-buck. You should be rate-limiting how many requests free API users can make, like; Twitter, Facebook and every other Internet provider does via their API. Make it harder…

On the other hand, large (even non-profit) organisations are precisely the ones who have the resources to scrape steathily and widely, as would a loosely-organised community of users... it's not hard to come up with algorithms to respect the rate limits, balancing the load across multiple IPs and accounts, and producing access patterns that don't look any different from the rest of the site traffic.

Re: The Scraping Problem and Ethics

#12
I see that the "Open Source" Vulnerability Database changed it's name to the "Open Sourced" Vulnerability Database in July 2013.

https://web.archive.org/web/20130714002216/http://www.osvdb....

I guess they were hoping no one would notice the subtle but substantial change to their service

Re: The Scraping Problem and Ethics

#13

If I were in OSVDB's shoes, I would call these people out in an email and ask them to pay a licencing fee. McAfee have always had a shady past, even when they shook themselves clean of John, they have a history of scam-like behaviour to make a quick-buck. You should be rate-limiting how many requests free API users can make, like; Twitter, Facebook and every other Internet provider does via their API. Make it harder…

On the other hand, large (even non-profit) organisations are precisely the ones who have the resources to scrape steathily and widely, as would a loosely-organised community of users... it's not hard to come up with algorithms to respect the rate limits, balancing the load across multiple IPs and accounts, and producing access patterns that don't look any different from the rest of the site traffic.

like much security, it's not about making it impossible, it's about making it a lot less convenient/a bit harder.

At one point the effort to circumvent would cost more in man-hours than just buying the product.

Re: The Scraping Problem and Ethics

#14
Shouldn't you at least be disallowing /show/* in robots.txt? Not that scrapers are necessarily going to respect this... but the way your set up it seems like this is semi-legit behavior.

Re: The Scraping Problem and Ethics

#15
Web scraping isn't a crime - the simple act of downloading the data should not be a problem here. (The reuse of the data might be, depending, but we don't have that information right now.)

This doesn't even rise to the level of what Weev did.

Re: The Scraping Problem and Ethics

#16
post #7
post #5

Earlier quoted context omitted.

Did you read the article? In both cases the offending parties were told to contact RBS to obtain a proper license/account.

That's exactly the problem that GP is talking about. McAfee were just being the usual assholes, but the first guy mentioned in the blog post could have been converted into a paying customer if the pricing scheme were clearly outlined on the web. Since "Contact us" account tiers are usually reserved for the very high end, he probably assumed that it would cost him an arm and a leg. It shouldn't be too difficult to com…

So many times I say this to people. Sometimes the reason people aren't buying is because they can't find the price, and aren't willing to pay the mental price of talking to a sales person to find out.

I understand the sales psychology in making sure you enter into a proper discussion with people to make sure their needs are right, and extracting the maximum consumer surplus from them. But there is a non-trivial pricing point where this makes sense, and by not having any anonymous-sign up, you're cutting off your sales curve below this point.

I think it's very easy to convince yourself it's easier just to sign up customers over the phone, but doing so without at least testing takeup of simpler tiers is an incomplete picture.

Re: The Scraping Problem and Ethics

#17
post #14

Shouldn't you at least be disallowing /show/* in robots.txt? Not that scrapers are necessarily going to respect this... but the way your set up it seems like this is semi-legit behavior.

Maybe they want Google to be able to crawl their database (which it has clearly done, as you'll see if you do a search.) That also raises some questions...

Re: The Scraping Problem and Ethics

#18
Why not put all information behind a login page and force people to sign up? It looks like this site will be used by only a few people anyways. You can then also track scrapers by login.

Re: The Scraping Problem and Ethics

#19

Web scraping isn't a crime - the simple act of downloading the data should not be a problem here. (The reuse of the data might be, depending, but we don't have that information right now.) This doesn't even rise to the level of what Weev did.

Scraping isn't a crime, but by the same token, neither is rate limiting and banning ips.

Re: The Scraping Problem and Ethics

#20
What about Aaron Swartz? He essentially did the same exact thing to the computers at MIT and he is somehow a freedom fighter when someone doing the same thing to the osvdb website is considered "unethical".

This is straight from the open security foundation website:

"We believe that security information and services should be easily accessible for all who have the need for such information and services"

Post reply on HN