Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Scraping Problem and Ethics

blog.osvdb.org

Post reply on HN
Print view View original

41–50 of 130 posts

Re: The Scraping Problem and Ethics

#41

Post by andrewfong »

userbinator wrote: post #11
DigitalSea wrote: post #8

If I were in OSVDB's shoes, I would call these people out in an email and ask them to pay a licencing fee. McAfee have always had a shady past, even when they shook themselves clean of John, they have a history of scam-like behaviour to make a quick-buck. You should be rate-limiting how many requests free API users can make, like; Twitter, Facebook and every other Internet provider does via their API. Make it harder…

On the other hand, large (even non-profit) organisations are precisely the ones who have the resources to scrape steathily and widely, as would a loosely-organised community of users... it's not hard to come up with algorithms to respect the rate limits, balancing the load across multiple IPs and accounts, and producing access patterns that don't look any different from the rest of the site traffic.

Large organizations also tend to have risk-averse lawyers. I'm not a fan of the CFAA, but if what Aaron Swartz did was illegal, then so is this.

Re: The Scraping Problem and Ethics

#42

Post by hueving »

pmorici wrote: post #23
paulhauggis wrote: post #20

What about Aaron Swartz? He essentially did the same exact thing to the computers at MIT and he is somehow a freedom fighter when someone doing the same thing to the osvdb website is considered "unethical". This is straight from the open security foundation website: "We believe that security information and services should be easily accessible for all who have the need for such information and services"

You might argue the difference is that the information that Aaron was after was already paid for with public money. The OSVDB sounds like it is a bunch of people who aren't otherwise paid to maintain this information.

>You might argue the difference is that the information that Aaron was after was already paid for with public money.

This is naive. Just because research is backed by public money doesn't mean the publications are automatically free to the public. If your argument was valid, you could use it to demand access to the emails of every FBI employee. Just because something is funded by the public doesn't automatically make every component of it open to the public.

Re: The Scraping Problem and Ethics

#43

Post by wpietri »

hueving wrote: post #42
pmorici wrote: post #23

Earlier quoted context omitted.

You might argue the difference is that the information that Aaron was after was already paid for with public money. The OSVDB sounds like it is a bunch of people who aren't otherwise paid to maintain this information.

>You might argue the difference is that the information that Aaron was after was already paid for with public money. This is naive. Just because research is backed by public money doesn't mean the publications are automatically free to the public. If your argument was valid, you could use it to demand access to the emails of every FBI employee. Just because something is funded by the public doesn't automatically make…

http://en.wikipedia.org/wiki/Freedom_of_Information_Act_%28U...

http://en.wikipedia.org/wiki/Government_in_the_Sunshine_Act

http://en.wikipedia.org/wiki/Brown_Act

Re: The Scraping Problem and Ethics

#44

Post by yaur »

yhager wrote: post #37
userbinator wrote: post #17

Earlier quoted context omitted.

Maybe they want Google to be able to crawl their database (which it has clearly done, as you'll see if you do a search.) That also raises some questions...

Not completely by the specification, but I think this one works as expected. user-agent: * disallow: / user-agent: Googlebot allow: /

I think that userbinator's point is that the DO conditionally allow scraping which makes their position even more tenuous IMO.

Re: The Scraping Problem and Ethics

#45

Post by nemothekid »

brc wrote: post #16
kijin wrote: post #7

Earlier quoted context omitted.

That's exactly the problem that GP is talking about. McAfee were just being the usual assholes, but the first guy mentioned in the blog post could have been converted into a paying customer if the pricing scheme were clearly outlined on the web. Since "Contact us" account tiers are usually reserved for the very high end, he probably assumed that it would cost him an arm and a leg. It shouldn't be too difficult to com…

So many times I say this to people. Sometimes the reason people aren't buying is because they can't find the price, and aren't willing to pay the mental price of talking to a sales person to find out. I understand the sales psychology in making sure you enter into a proper discussion with people to make sure their needs are right, and extracting the maximum consumer surplus from them. But there is a non-trivial prici…

I get this line of thinking if your you can sell your offering for $100 bucks a month. But what if your minimum offering is $2500/mo? Correct me if I'm wrong, but I've yet to see a company - that only sales through enterprise - put up a sign saying where their plans start at $2,500.

In most cases I don't believe these services are forcing customers to talk on the phone because they think they will convert them. Chances are that sign up isn't as easy as a 1-2-click, and involved set up and understanding of the customers needs is required before services can be rendered.

Re: The Scraping Problem and Ethics

#46

Post by danielweber »

duey wrote: post #36
lukejduncan wrote: post #26

Earlier quoted context omitted.

It's not really reasonable to say "I don't like the way you market your goods... so you really shouldn't be concerned with people stealing them."

Not in the long term obviously, but you can't make a product extremely difficult to buy and then complain that everyone is taking the easy route.

Talking to someone on the phone is not "extremely difficult." Large companies buy stuff by talking to people on the phone all the time.

Re: The Scraping Problem and Ethics

#47

Post by mehwoot »

lukejduncan wrote: post #26
duey wrote: post #3

The OSVDB website contains no signup page for commercial access. No pricing either, purely sign up via contacting someone. From my experience whenever I see this, I just refuse to use the service and look elsewhere. Contacting someone is annoying and opens you up to repeat sales calls. Perhaps they should make commercial API access easier to access rather than complain about scrapers.

It's not really reasonable to say "I don't like the way you market your goods... so you really shouldn't be concerned with people stealing them."

Tell that to the 95% of the population on this site who torrent TV shows, movies and music.

Re: The Scraping Problem and Ethics

#48

Post by thefreeman »

yaur wrote: post #44
yhager wrote: post #37

Earlier quoted context omitted.

Not completely by the specification, but I think this one works as expected. user-agent: * disallow: / user-agent: Googlebot allow: /

I think that userbinator's point is that the DO conditionally allow scraping which makes their position even more tenuous IMO.

By a single search engine which probably provides the vast majority of their traffic.

While I don't necessarily agree with the concept of only allowing google to index your site, comparing a search engine which feeds you business to a company reselling your data with no attribution is not really fair in my opinion.

Re: The Scraping Problem and Ethics

#49

Post by DanBC »

danielweber wrote: post #46
duey wrote: post #36

Earlier quoted context omitted.

Not in the long term obviously, but you can't make a product extremely difficult to buy and then complain that everyone is taking the easy route.

Talking to someone on the phone is not "extremely difficult." Large companies buy stuff by talking to people on the phone all the time.

And it sucks as a method of buying something.
Post reply on HN