The Scraping Problem and Ethics

Is OSVDB any different? "Open Source" Vulnerability Database implies openness. The maintainers are trying to implement a paywall.

I feel bad for OSVDB from a sysadmin perspective, but if Aaron's case was so polarizing for essentially the same thing, why isn't everyone jumping on the Hate Train here?

This is exactly the argument MPAA makes against people torrenting movies and tv shows, using words such as "criminals", "evil", in an attempt to polarize people who are simply consuming information over the internet in the most efficient way possible, without any 3rd party oversight and censorship, the way information on the internet is always bound to be, the way equilibrium is achieved.

The argument that if you really want this information, they should pay for it doesn't work. Basically, you are punishing people for automating manual labor. The fact that you can hire bunch of people and tell them to manually copy data from the website and achieve the same result means that it shouldn't be any different from automating the process itself.

Terms of service on a website is not the law. If there is something you don't want people to have access to, then don't publish it online at all.

i) no, search engines articulate the web. They are essentially 'pre-paid' computation.

Take the classic example, a search for 'bilbo baggins'. What that is, is a request to identify documents on the web that have referred to Bilbo Baggins and return their locations.

1) It is absolutely true that you could sit down at your computer and look at each site, from aol to zillow, read all their pages, and note the ones that mention Bilbo. Then you could go back and order the list by the ones that had more of the information you were looking for to the ones with less useful information. Along the way you would find some sites that would not open up to you unless you had an account, those you could not visit.

2) A search engine can look at all the sites, it can note which sites mention Bilbo along with a bunch of other terms and can essentially "pre-compute" that list you were looking for. Along the way it will find sites that, through their robots.txt file, will say "We'd rather you not look here." and it will respect that, thus not indexing those sites.

In both cases figuring out which web pages have information about Bilbo on them is creating 'new' information out of existing data. You can do it on your own and it will cost you time, or you can do it with a few thousand machines and it will cost you money. Either way you get a list of possible sites.

That list forms a distribution, where there are a lot of sites that don't care one way or the other if you read them, sites that won't show up because they asked to be excluded, and sites that will show up because they paid to be included. Some sites really want you to find them, some sites really don't. A good search engine caters to both types.

ii) If a search engine was taking the page, copying it, and then showing that instead of showing the page (this is what got Google's news product in trouble) then its pretty clear that they should not do that. But in terms of location information? The sites themselves derive a huge economic benefit from being in the index that isn't reflected at all back to the search engine that sent traffic there [1], so on a pure economic basis the search engine is on the losing side of that transaction. However, the marginal cost of additional transactions is small (search engines are general purpose) so they make a small amount on large volumes.

To put your question in more specifics, where is the economic value in the list; bobs middle earth atlas, wikipedia entry on bilbo baggins, middle earth web ring, imdb pages on characters in the "Lord of the Rings" movie.

Is it that Bob has an atlas of Middle Earth? Or is it the list itself? Who made the list? Bob or the search engine? (or some human curator of a bookmarks page[2])

iii) It is completely up to the site to allow or disallow access to its content by search engines. Some sites do only allow themselves to be indexed by Google and they find they get less search engine directed traffic that way. Some sites don't allow anyone to index them and they get no traffic (sometimes they are surprised by this, sometimes they don't care, sometimes they are angry that the only way for people to find them is to be in a search engine index)

[1] Google broke that by creating AdSense for Content and created a pretty interesting conflict of interest for themselves.

[2] Good luck finding a book marks page these days :-)

but how is it stealing when one does not lose inventory? If one person scrapes a page, did you lose the source code? Does it not become available for the next visitor? What possible loss do you incur that is directly tied to your data? When you make data public with the intent of being readily accessible by the public, how can you claim theft when you are achieving what you set out to do? Does the accelerated rate of access suddenly become a theft? Does one need to pay a third party to avoid the pain associated with manual hand labor for simply hosting the data which is available to the public? Help me understand.

Apropos of nothing: customers often have an exaggerated notion on how important it is to e.g. an enterprise software company that that company land their account.

A conversation I've had a few times:

"We need it to do $THING_IT_WON'T_DO."

"In that case, it probably isn't a great fit for your needs."

"You don't understand. I won't buy it if it doesn't do that."

"I think I do understand. That's fine. You might consider trying $COMPETITOR, although you should know their minimum spend is $1,000 a month."

"That's outrageous. You have a $29 plan."

"Yes. So you should go with the competitor if that requirement is worth $971 a month to you."

"No, I want to spend $29, but I absolutely need that."

"I understand where you're coming from, but we do not offer that feature, and if we did, we would charge prices close to what our competitor does for it."

"You're not working with me here."

"I'm trying to find a resolution which works for you, but including that feature at $29 doesn't make business sense for me, so I won't do it."

"Put me on the phone with your boss."

"I'm afraid that isn't possible, as I sort of run things around here."

"What sort of businessman turns customers away."

"You're not a customer. If you were, you would be purchasing a product I sell for the amount I sell it for. That isn't happening. That's fine. Have a nice day."