Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Scraping Problem and Ethics

blog.osvdb.org

Post reply on HN
Print view View original

31–40 of 130 posts

Re: The Scraping Problem and Ethics

#31

Post by acrefoot »

[deleted] wrote: post #29
codezero wrote: post #22

Earlier quoted context omitted.

It can be a crime if it is a violation of the site's terms of service and the requests violate the host's robots.txt. I'm not sure whether there is a legal precedent though, in some cases you could call it a denial of service if the requests are not rate limited, and in other cases it might be considered an inappropriate access (see Weev, though he eventually won appeal).

[deleted]

Actually, Terms of Service violations fall under the Computer Fraud and Abuse Act, since ToS agreements can lay out under which circumstances that authorization for access to computer systems is given. That sort of obscene generality is the reason for proposals such as Aaron's Law, but to my knowledge there are no such protections today.

Re: The Scraping Problem and Ethics

#32

Post by freshhawk »

cproctor wrote: post #24

I can't help recalling a post here a couple of years ago about the concept of "hellbanning" scammers on ecommerce sites--in short, making it look like everything is going fine, while actually isolating them completely from your business logic. Orders with stolen cards appear to go through, and send confirmation emails, but no real order is generated... In this case, you could transparently poison the results served t…

Heh, I've always called this the map makers trick (found an article about it here: https://theweek.com/article/index/241967/trap-streets-the-cr...) although I guess that is specific to putting a small amount of fake data in your dataset to prove someone else used it. The hellbanning metaphor does fit for return large amounts of poison results.

It could be like spiking trees I guess, but that depends on the potential for harm. I just looked it up and was surprised to find that only one injury has ever been reported due to tree spiking. I guess it's a better talking point than actual tactic.

And like lukejduncan said, this is definitely done in practice.

Re: The Scraping Problem and Ethics

#34

Post by bizark1342 »

ChuckMcM wrote: post #28

This is one of the more interesting policy questions on the web. Our search engine crawls a lot of blogs and what not on the web, criminals who want to find unpatched wordpress sites try to acrape our crawl by sending automated (scripted) queries to find them. We have developed a number of defenses over the years and pretty regularly ban them[1]. Here is the weird part though, if they hired 300 people on mechanical t…

Looks like 80legs that you mention rebranded themselves as "Datafiniti" at some point recently.

http://blog.datafiniti.net/?p=230

Re: The Scraping Problem and Ethics

#35

Post by alexnking »

I'm more and more concerned that the legal and cultural environment for web scraping would make it hard for a company like Google or Yahoo to be founded today.

The internet isn't about "don't take my stuff", it's about spreading that stuff around. I'm confused by people who want to make their data public, but want to control exactly how people access it.

Re: The Scraping Problem and Ethics

#36

Post by duey »

lukejduncan wrote: post #26
duey wrote: post #3

The OSVDB website contains no signup page for commercial access. No pricing either, purely sign up via contacting someone. From my experience whenever I see this, I just refuse to use the service and look elsewhere. Contacting someone is annoying and opens you up to repeat sales calls. Perhaps they should make commercial API access easier to access rather than complain about scrapers.

It's not really reasonable to say "I don't like the way you market your goods... so you really shouldn't be concerned with people stealing them."

Not in the long term obviously, but you can't make a product extremely difficult to buy and then complain that everyone is taking the easy route.

Re: The Scraping Problem and Ethics

#37

Post by yhager »

userbinator wrote: post #17
yaur wrote: post #14

Shouldn't you at least be disallowing /show/* in robots.txt? Not that scrapers are necessarily going to respect this... but the way your set up it seems like this is semi-legit behavior.

Maybe they want Google to be able to crawl their database (which it has clearly done, as you'll see if you do a search.) That also raises some questions...

Not completely by the specification, but I think this one works as expected.

    user-agent: *
    disallow: /

    user-agent: Googlebot
    allow: /

Re: The Scraping Problem and Ethics

#38

Post by meowface »

ChuckMcM wrote: post #28

This is one of the more interesting policy questions on the web. Our search engine crawls a lot of blogs and what not on the web, criminals who want to find unpatched wordpress sites try to acrape our crawl by sending automated (scripted) queries to find them. We have developed a number of defenses over the years and pretty regularly ban them[1]. Here is the weird part though, if they hired 300 people on mechanical t…

I'm assuming that the vast majority of the time, it's a case of an individual or small group that wants the information, not a real corporation. Usually they have no intention of paying for anything or playing by the rules.

I have absolutely no explanation regarding McAfee though, considering the billions McAfee and its parent company, Intel, makes in revenue yearly.

Re: The Scraping Problem and Ethics

#39

Post by drewcrawford »

ChuckMcM wrote: post #28

This is one of the more interesting policy questions on the web. Our search engine crawls a lot of blogs and what not on the web, criminals who want to find unpatched wordpress sites try to acrape our crawl by sending automated (scripted) queries to find them. We have developed a number of defenses over the years and pretty regularly ban them[1]. Here is the weird part though, if they hired 300 people on mechanical t…

I think there is a great meta-question in here, about business models for digital data and software.

Here you have a great case study, about an organization that tried to do a volunteer model, and it didn't work. Then they pivoted to a commercial model, but fundamentally they still believe in a free tier. But they have to cripple that free tier pretty thoroughly, and even still people abuse it.

I have a product I'm working on, that some people are apparently willing to spend lots of money on. Ideally I would have some kind of low tier, so that people without lots of money would be able to use it too. But I can't figure out a way to segment the product so that everybody pays what they can afford without bad apples abusing the low tier and ruining it for everybody. The result is that I may end up only selling it to customers with deep pockets, even though the product is much more broadly applicable.

Re: The Scraping Problem and Ethics

#40

Post by quesera »

acrefoot wrote: post #31
[deleted] wrote: post #29

Earlier quoted context omitted.

[deleted]

Actually, Terms of Service violations fall under the Computer Fraud and Abuse Act, since ToS agreements can lay out under which circumstances that authorization for access to computer systems is given. That sort of obscene generality is the reason for proposals such as Aaron's Law, but to my knowledge there are no such protections today.

Totally correct. Parent comment deleted for lack of usefulness. Thank you for the correction.
Post reply on HN