Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Scraping Problem and Ethics

blog.osvdb.org

Post reply on HN
Print view View original

21–30 of 130 posts

Re: The Scraping Problem and Ethics

#21

Post by codezero »

paulhauggis wrote: post #20

What about Aaron Swartz? He essentially did the same exact thing to the computers at MIT and he is somehow a freedom fighter when someone doing the same thing to the osvdb website is considered "unethical". This is straight from the open security foundation website: "We believe that security information and services should be easily accessible for all who have the need for such information and services"

While your sentiment is reasonable, I think the main difference here is that McAfee and others mentioned host their own private vuln databases and do not share them with anyone, so they were scraping to increase their own private resources for commercial use.

Aaron was scraping private resources to share publicly.

Re: The Scraping Problem and Ethics

#22

Post by codezero »

kylemaxwell wrote: post #15

Web scraping isn't a crime - the simple act of downloading the data should not be a problem here. (The reuse of the data might be, depending, but we don't have that information right now.) This doesn't even rise to the level of what Weev did.

It can be a crime if it is a violation of the site's terms of service and the requests violate the host's robots.txt.

I'm not sure whether there is a legal precedent though, in some cases you could call it a denial of service if the requests are not rate limited, and in other cases it might be considered an inappropriate access (see Weev, though he eventually won appeal).

Re: The Scraping Problem and Ethics

#23

Post by pmorici »

paulhauggis wrote: post #20

What about Aaron Swartz? He essentially did the same exact thing to the computers at MIT and he is somehow a freedom fighter when someone doing the same thing to the osvdb website is considered "unethical". This is straight from the open security foundation website: "We believe that security information and services should be easily accessible for all who have the need for such information and services"

You might argue the difference is that the information that Aaron was after was already paid for with public money. The OSVDB sounds like it is a bunch of people who aren't otherwise paid to maintain this information.

Re: The Scraping Problem and Ethics

#24

Post by cproctor »

I can't help recalling a post here a couple of years ago about the concept of "hellbanning" scammers on ecommerce sites--in short, making it look like everything is going fine, while actually isolating them completely from your business logic. Orders with stolen cards appear to go through, and send confirmation emails, but no real order is generated... In this case, you could transparently poison the results served to identified scrapers with 5% bogus vulnerabilities.

Or is that about as ethical as spiking trees to prevent illegal logging?

Re: The Scraping Problem and Ethics

#25

Post by lukejduncan »

cproctor wrote: post #24

I can't help recalling a post here a couple of years ago about the concept of "hellbanning" scammers on ecommerce sites--in short, making it look like everything is going fine, while actually isolating them completely from your business logic. Orders with stolen cards appear to go through, and send confirmation emails, but no real order is generated... In this case, you could transparently poison the results served t…

I've definitely seen this done in practice

Re: The Scraping Problem and Ethics

#26

Post by lukejduncan »

duey wrote: post #3

The OSVDB website contains no signup page for commercial access. No pricing either, purely sign up via contacting someone. From my experience whenever I see this, I just refuse to use the service and look elsewhere. Contacting someone is annoying and opens you up to repeat sales calls. Perhaps they should make commercial API access easier to access rather than complain about scrapers.

It's not really reasonable to say "I don't like the way you market your goods... so you really shouldn't be concerned with people stealing them."

Re: The Scraping Problem and Ethics

#27

Post by stronglikedan »

kylemaxwell wrote: post #15

Web scraping isn't a crime - the simple act of downloading the data should not be a problem here. (The reuse of the data might be, depending, but we don't have that information right now.) This doesn't even rise to the level of what Weev did.

He's just making the point that it's unethical, which it is. Even if it boils down to simple bandwidth theft.

Re: The Scraping Problem and Ethics

#28

Post by ChuckMcM »

This is one of the more interesting policy questions on the web. Our search engine crawls a lot of blogs and what not on the web, criminals who want to find unpatched wordpress sites try to acrape our crawl by sending automated (scripted) queries to find them. We have developed a number of defenses over the years and pretty regularly ban them[1]. Here is the weird part though, if they hired 300 people on mechanical turk and paid them each a dollar to do one search, it probably would get them more information. Look at folks like 80legs or other 'distributed' scrapers. They exist almost solely to subvert these service terms. Are they evil? Creative?

One of the things that stands out is called out in this article. The people involved really want this information, so much that they are willing to expend time and effort to construct scraping bots and what have you . Why not just buy it? How is it that someone gets a request from their boss to get some information, but their boss expects them to get it for free? Can you imagine if they said, "We need pens, pencils, notebooks, staplers, the works for the office here. Oh and you can't spend any money getting that stuff, just get it here." Would they construct some elaborate raid on a nearby Office supplies store using a mercenary army of criminals? Why do that with information?

We did an experiment where we would 'grep the web' for you, basically run a regex over a multi-billion page crawl, give you the first 50 results for "free" and you could buy the complete set. I think we sold exactly one of those.

It is a weird thing, the OP captured it perfectly.

[1] Its a violation of the terms of service.

Re: The Scraping Problem and Ethics

#29

Post by [deleted] »

codezero wrote: post #22
kylemaxwell wrote: post #15

Web scraping isn't a crime - the simple act of downloading the data should not be a problem here. (The reuse of the data might be, depending, but we don't have that information right now.) This doesn't even rise to the level of what Weev did.

It can be a crime if it is a violation of the site's terms of service and the requests violate the host's robots.txt. I'm not sure whether there is a legal precedent though, in some cases you could call it a denial of service if the requests are not rate limited, and in other cases it might be considered an inappropriate access (see Weev, though he eventually won appeal).

[deleted]

Re: The Scraping Problem and Ethics

#30

Post by jamra »

codezero wrote: post #21
paulhauggis wrote: post #20

What about Aaron Swartz? He essentially did the same exact thing to the computers at MIT and he is somehow a freedom fighter when someone doing the same thing to the osvdb website is considered "unethical". This is straight from the open security foundation website: "We believe that security information and services should be easily accessible for all who have the need for such information and services"

While your sentiment is reasonable, I think the main difference here is that McAfee and others mentioned host their own private vuln databases and do not share them with anyone, so they were scraping to increase their own private resources for commercial use. Aaron was scraping private resources to share publicly.

More than that, he was scraping private resources that were freely populated. He was not robbing content creators of their money. He was circumventing a paywall to what should be free data.
Post reply on HN