Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Scraping Problem and Ethics

blog.osvdb.org

Post reply on HN
Print view View original

71–80 of 130 posts

Re: The Scraping Problem and Ethics

#71

Post by andreasvc »

dspillett wrote: post #67
gadders wrote: post #56

Earlier quoted context omitted.

If I were in their shows I would track their IPs and send them bogus data along the lines of "Please pay for a commercial license."

You need to be careful sending bogus data: in some jurisdictions this could be argued to be deliberate targeted commercial sabotage. You would no doubt eventually win any resulting legal argument, assuming you could afford to carry the argument on to that conclusion . Sending no data, or limited data, would be safe though. A better method would be to set "default" pricing (something high but not ridiculous, that coul…

You don't have to return bogus data, you can return an HTTP error code: 402 payment required.

Re: The Scraping Problem and Ethics

#73

Post by stuaxo »

meowface wrote: post #38
ChuckMcM wrote: post #28

This is one of the more interesting policy questions on the web. Our search engine crawls a lot of blogs and what not on the web, criminals who want to find unpatched wordpress sites try to acrape our crawl by sending automated (scripted) queries to find them. We have developed a number of defenses over the years and pretty regularly ban them[1]. Here is the weird part though, if they hired 300 people on mechanical t…

I'm assuming that the vast majority of the time, it's a case of an individual or small group that wants the information, not a real corporation. Usually they have no intention of paying for anything or playing by the rules. I have absolutely no explanation regarding McAfee though, considering the billions McAfee and its parent company, Intel, makes in revenue yearly.

Could this be a company culture thing at McAfee ?

Re: The Scraping Problem and Ethics

#74

Post by test1235 »

Mikushi wrote: post #69
rtpg wrote: post #13

Earlier quoted context omitted.

like much security, it's not about making it impossible, it's about making it a lot less convenient/a bit harder. At one point the effort to circumvent would cost more in man-hours than just buying the product.

You can get scrapping libraries fairly easily. In my more shady past I developed a library like that and shared it, HTTP client library with automatic proxy rotation and rate limiting friendly. When I used it (which was almost a decade ago) never ran into problems, plug a list of 10,000 proxy and scrap away. Not condoning that, which is a bit hypocrite of me, at the time I was mostly doing what I was told and I thoug…

Speaking as devil's advocate, it might just be more convenient to steal the data.

Maybe I want to use your data casually once, and I don't want to sign up and give you all my contact details and subscribe to your annual plan with all the other optional extras.

Tough shit, you say? I'll just steal it then, and not because I can't afford it, but because you're making it hard to pay.

Re: The Scraping Problem and Ethics

#75

Post by TheLoneWolfling »

alexnking wrote: post #35

I'm more and more concerned that the legal and cultural environment for web scraping would make it hard for a company like Google or Yahoo to be founded today. The internet isn't about "don't take my stuff", it's about spreading that stuff around. I'm confused by people who want to make their data public, but want to control exactly how people access it.

Agreed.

I talked about this a while back: https://news.ycombinator.com/item?id=6572937

Someone decided to ban all bots from accessing their site except for Google and Bing. So much for "if you're so worried about Google just use another search engine".

Re: The Scraping Problem and Ethics

#77

Post by wslh »

ChuckMcM wrote: post #28

This is one of the more interesting policy questions on the web. Our search engine crawls a lot of blogs and what not on the web, criminals who want to find unpatched wordpress sites try to acrape our crawl by sending automated (scripted) queries to find them. We have developed a number of defenses over the years and pretty regularly ban them[1]. Here is the weird part though, if they hired 300 people on mechanical t…

Since you work at Blekko, there are more points to discuss that were not addressed in the article.

For example:

i) Are search engines web scrapers?

ii) Should search engines pay the scraped sites if they are charging to access their indexed data? probably some of the scraped sites has a specific license forbidding the search engine to sell their information in any way.

iii) Regarding Internet policies, is it fair/unfair that a site has a robots.txt configuration to avoid being indexed by a search engine other than Google? I would call this "search neutrality".

Re: The Scraping Problem and Ethics

#78

Post by pbhjpbhj »

dspillett wrote: post #67
gadders wrote: post #56

Earlier quoted context omitted.

If I were in their shows I would track their IPs and send them bogus data along the lines of "Please pay for a commercial license."

You need to be careful sending bogus data: in some jurisdictions this could be argued to be deliberate targeted commercial sabotage. You would no doubt eventually win any resulting legal argument, assuming you could afford to carry the argument on to that conclusion . Sending no data, or limited data, would be safe though. A better method would be to set "default" pricing (something high but not ridiculous, that coul…

>"You need to be careful sending bogus data: in some jurisdictions this could be argued to be deliberate targeted commercial sabotage." //

That sounds pretty ludicrous, do you have anything to back it up - caselaw, settlement report? It would be analagous to serving a fake image to combat hotlinking; or a fake page to combat framing.

Re: The Scraping Problem and Ethics

#79

Post by im3w1l »

I was once developing a program for automated trading for my own personal use. In the beginning I made periodical requests to my broker, scraped out the price info, and made orders if the price was right. Once I had something sort of working I decided that I wanted to do this the right way and ask them for permission. They told me I would have to subscribe to a feed through some other program. After many hours of reading the manual and trying to figure out how to transfer the data to my application I again got it sort of working. But I had already started to lose interest and other things had come up in my life.

How is this relevant? Well I would have much preferred paying for scraping, than trying to learn some new api. It increased the transaction cost. If you further have to negotiate with a partner company, that sounds like even more transaction cost, from having to send emails back and forth and the mental effort of negotiating.

Re: The Scraping Problem and Ethics

#80

Post by wslh »

im3w1l wrote: post #79

I was once developing a program for automated trading for my own personal use. In the beginning I made periodical requests to my broker, scraped out the price info, and made orders if the price was right. Once I had something sort of working I decided that I wanted to do this the right way and ask them for permission. They told me I would have to subscribe to a feed through some other program. After many hours of rea…

Beyond the moral discussion, I think a market for web scraping is a good thing because currently there are a lot of unconnected people trying to buy/sell this service.

Freelancer sites has a lot of offerings for web scraping but this niche has its own issues.

Post reply on HN