The Scraping Problem and Ethics • Hacker News Board

Re: The Scraping Problem and Ethics

#91

Post by the_ancient » Thu, May 08, 2014, 2:01 PM UTC

danielweber wrote: post #46

duey wrote: post #36

Earlier quoted context omitted.

Not in the long term obviously, but you can't make a product extremely difficult to buy and then complain that everyone is taking the easy route.

Talking to someone on the phone is not "extremely difficult." Large companies buy stuff by talking to people on the phone all the time.

and I have a policy that if the company does not practice open pricing by publishing their prices online I will not do business with them..

Fuck that non-sense of pricing based on what ever they think they can scam me out of

Re: The Scraping Problem and Ethics

#92

Post by hnal943 » Thu, May 08, 2014, 2:02 PM UTC

alexnking wrote: post #35

I'm more and more concerned that the legal and cultural environment for web scraping would make it hard for a company like Google or Yahoo to be founded today. The internet isn't about "don't take my stuff", it's about spreading that stuff around. I'm confused by people who want to make their data public, but want to control exactly how people access it.

I'm confused by people who want to make their data public, but want to control exactly how people access it.

You mean like TV? or Radio? or print....

Re: The Scraping Problem and Ethics

#93

Post by danielweber » Thu, May 08, 2014, 2:08 PM UTC

the_ancient wrote: post #91

danielweber wrote: post #46

Earlier quoted context omitted.

Talking to someone on the phone is not "extremely difficult." Large companies buy stuff by talking to people on the phone all the time.

and I have a policy that if the company does not practice open pricing by publishing their prices online I will not do business with them.. Fuck that non-sense of pricing based on what ever they think they can scam me out of

That's fine. 'patio11 has lots of stories about that.

However, the lack of up-front pricing data isn't being used to justify "I won't do business with you." It's justifying "I'm going to take all your stuff anyway."

Re: The Scraping Problem and Ethics

#94

Post by NoMoreNicksLeft » Thu, May 08, 2014, 2:18 PM UTC

Maybe it's the supervillain in me, but I think I would try to come up with software to recognize the scraping attempts, and rather than ban them just have it generate fake data on the fly.

Re: The Scraping Problem and Ethics

#95

Post by detroitcoder » Thu, May 08, 2014, 2:19 PM UTC

Amateurs. If you are going to crawl distribute it across unique IP addresses.

Re: The Scraping Problem and Ethics

#96

Post by yp_all » Thu, May 08, 2014, 2:33 PM UTC

ChuckMcM wrote: post #28

This is one of the more interesting policy questions on the web. Our search engine crawls a lot of blogs and what not on the web, criminals who want to find unpatched wordpress sites try to acrape our crawl by sending automated (scripted) queries to find them. We have developed a number of defenses over the years and pretty regularly ban them[1]. Here is the weird part though, if they hired 300 people on mechanical t…

Pot calling the kettle black.

Crawling others' websites then selling ads. Scraping others' websites then selling ads. Selling access to user-generated content. Amusing to watch these folks argue about ethics.

Who owns the copyrights in this data? Surely not the one who is demanding that you pay a license fee. These "services" are middlemen, plain and simple.

This might be why McAfee was wondering about how much manual curation is done.

Maybe that is the only possible theory of how OSVDB could assert any rights in the data (and only in a select few jurisdictions).

Re: The Scraping Problem and Ethics

#97

Post by johnhess » Thu, May 08, 2014, 2:36 PM UTC

lukejduncan wrote: post #26

duey wrote: post #3

The OSVDB website contains no signup page for commercial access. No pricing either, purely sign up via contacting someone. From my experience whenever I see this, I just refuse to use the service and look elsewhere. Contacting someone is annoying and opens you up to repeat sales calls. Perhaps they should make commercial API access easier to access rather than complain about scrapers.

It's not really reasonable to say "I don't like the way you market your goods... so you really shouldn't be concerned with people stealing them."

I have to disagree. As creators, we should expect to see the behavior our designs encourage.

The fact is, if it's harder to buy something, would-be buyers choose another route.

Imagine something of trivial value that was very arduous to obtain. Say, the scores to last weekend's football game are only available via sending the NFL 2 cents taped onto a postcard then getting a user account and password back in the mail. Yes, you could apply for an account and who cares about the 2 cents? But most people wouldn't and who could blame them? There'd certainly be a market for pirated 'score data'.

But, if you just put ads on your site (the NFL does), voila! You make the same amount of money and people are happy to use your service. Netflix, Hulu, iTunes Store and Spotify all get this. If you make it a pain to do something, you can't feign surprise when everyone goes around you.

Is it legal to circumvent something just because it's arduous? No. Is it ethical? Not completely, but downloading pirated football scores wouldn't keep me from running for office.

Re: The Scraping Problem and Ethics

#98

Post by falcolas » Thu, May 08, 2014, 2:36 PM UTC

dspillett wrote: post #67

gadders wrote: post #56

Earlier quoted context omitted.

If I were in their shows I would track their IPs and send them bogus data along the lines of "Please pay for a commercial license."

You need to be careful sending bogus data: in some jurisdictions this could be argued to be deliberate targeted commercial sabotage. You would no doubt eventually win any resulting legal argument, assuming you could afford to carry the argument on to that conclusion . Sending no data, or limited data, would be safe though. A better method would be to set "default" pricing (something high but not ridiculous, that coul…

Hasn't the mapping industry already set a precedent in this field? They provide maps with small, wrong roads/trails specifically to identify who is stealing their data.

Re: The Scraping Problem and Ethics

#99

Post by im3w1l » Thu, May 08, 2014, 2:46 PM UTC

im3w1l wrote: post #79

I was once developing a program for automated trading for my own personal use. In the beginning I made periodical requests to my broker, scraped out the price info, and made orders if the price was right. Once I had something sort of working I decided that I wanted to do this the right way and ask them for permission. They told me I would have to subscribe to a feed through some other program. After many hours of rea…

paying [the site in question] for scraping I mean

Re: The Scraping Problem and Ethics

#100

Post by johnward » Thu, May 08, 2014, 3:05 PM UTC

hnal943 wrote: post #92

alexnking wrote: post #35

I'm more and more concerned that the legal and cultural environment for web scraping would make it hard for a company like Google or Yahoo to be founded today. The internet isn't about "don't take my stuff", it's about spreading that stuff around. I'm confused by people who want to make their data public, but want to control exactly how people access it.

I'm confused by people who want to make their data public, but want to control exactly how people access it. You mean like TV? or Radio? or print....

To be fair those mediums all kind of suck in their own way.