Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Scraping Problem and Ethics

blog.osvdb.org

Post reply on HN
Print view View original

51–60 of 130 posts

Re: The Scraping Problem and Ethics

#51

Post by yaur »

thefreeman wrote: post #48
yaur wrote: post #44

Earlier quoted context omitted.

I think that userbinator's point is that the DO conditionally allow scraping which makes their position even more tenuous IMO.

By a single search engine which probably provides the vast majority of their traffic. While I don't necessarily agree with the concept of only allowing google to index your site, comparing a search engine which feeds you business to a company reselling your data with no attribution is not really fair in my opinion.

Neither of the two parties mentioned is likely to resell the data directly, both are likely to to create a derivative work which they will exploit commercially as will Google. What part of the ToS make one OK while and the other forbidden? How does this work when a smart scraper can just pull from the Google or archive.org cache?

Re: The Scraping Problem and Ethics

#52

Post by [deleted] »

freshhawk wrote: post #32
cproctor wrote: post #24

I can't help recalling a post here a couple of years ago about the concept of "hellbanning" scammers on ecommerce sites--in short, making it look like everything is going fine, while actually isolating them completely from your business logic. Orders with stolen cards appear to go through, and send confirmation emails, but no real order is generated... In this case, you could transparently poison the results served t…

Heh, I've always called this the map makers trick (found an article about it here: https://theweek.com/article/index/241967/trap-streets-the-cr... ) although I guess that is specific to putting a small amount of fake data in your dataset to prove someone else used it. The hellbanning metaphor does fit for return large amounts of poison results. It could be like spiking trees I guess, but that depends on the potential…

[deleted]

Re: The Scraping Problem and Ethics

#53

Post by npizzolato »

jamra wrote: post #30
codezero wrote: post #21

Earlier quoted context omitted.

While your sentiment is reasonable, I think the main difference here is that McAfee and others mentioned host their own private vuln databases and do not share them with anyone, so they were scraping to increase their own private resources for commercial use. Aaron was scraping private resources to share publicly.

More than that, he was scraping private resources that were freely populated. He was not robbing content creators of their money. He was circumventing a paywall to what should be free data.

That's quite the mental leap. He was circumventing a paywall, but that's not robbing anyone of money because it should have been free in the first place? Well, it wasn't free, even if you think you it should be. Thus the paywall.

Re: The Scraping Problem and Ethics

#54

Post by kijin »

nemothekid wrote: post #45
brc wrote: post #16

Earlier quoted context omitted.

So many times I say this to people. Sometimes the reason people aren't buying is because they can't find the price, and aren't willing to pay the mental price of talking to a sales person to find out. I understand the sales psychology in making sure you enter into a proper discussion with people to make sure their needs are right, and extracting the maximum consumer surplus from them. But there is a non-trivial prici…

I get this line of thinking if your you can sell your offering for $100 bucks a month. But what if your minimum offering is $2500/mo? Correct me if I'm wrong, but I've yet to see a company - that only sales through enterprise - put up a sign saying where their plans start at $2,500. In most cases I don't believe these services are forcing customers to talk on the phone because they think they will convert them. Chanc…

If your signup process is understandably complicated and you require a minimum commitment of $2500/mo, then it's probably okay for you to tell potential customers to give you a call.

If your signup process can be automated and you only charge $25/mo, and you still tell people to give you a call, then you will lose business because 1) the friction of a phone call is worth more than the difference between your offer and a competing offer; 2) it's impossible to tell whether your offer is worth the friction in the first place, because your pricing is unknown; and 3) people just assume that you'll charge $2500/mo because that's the usual price point where people say "contact us". If someone else comes along and offers an inferior product for $35/mo, they'll get all the business because monkey psychology.

Re: The Scraping Problem and Ethics

#55

Post by codezero »

npizzolato wrote: post #53
jamra wrote: post #30

Earlier quoted context omitted.

More than that, he was scraping private resources that were freely populated. He was not robbing content creators of their money. He was circumventing a paywall to what should be free data.

That's quite the mental leap. He was circumventing a paywall, but that's not robbing anyone of money because it should have been free in the first place? Well, it wasn't free, even if you think you it should be. Thus the paywall.

To that point, the people who submitted to the journal did so knowing that it costs money to access, it wasn't as if they were tricked into contributing to a private pay-wall journal.

Re: The Scraping Problem and Ethics

#56

Post by gadders »

DigitalSea wrote: post #8

If I were in OSVDB's shoes, I would call these people out in an email and ask them to pay a licencing fee. McAfee have always had a shady past, even when they shook themselves clean of John, they have a history of scam-like behaviour to make a quick-buck. You should be rate-limiting how many requests free API users can make, like; Twitter, Facebook and every other Internet provider does via their API. Make it harder…

If I were in their shows I would track their IPs and send them bogus data along the lines of "Please pay for a commercial license."

Re: The Scraping Problem and Ethics

#57

Post by Blahah »

codezero wrote: post #55
npizzolato wrote: post #53

Earlier quoted context omitted.

That's quite the mental leap. He was circumventing a paywall, but that's not robbing anyone of money because it should have been free in the first place? Well, it wasn't free, even if you think you it should be. Thus the paywall.

To that point, the people who submitted to the journal did so knowing that it costs money to access, it wasn't as if they were tricked into contributing to a private pay-wall journal.

For most scientists there isn't another system, and certainly when the system was established nobody was envisaging a time when publishers would make hundreds of % markup on every access of a paper. The prices reflect print publishing costs and the historical lack of a cheap distribution system like the internet. Now we have a hangover where people's careers are judged on their ability to publish in high-impact journals, all of which charge several thousand pounds extra to make the paper Open Access. So we're not tricked, but we do it under duress.

The system is changing, Aaron helped.

Re: The Scraping Problem and Ethics

#58

Post by __david__ »

stronglikedan wrote: post #27
kylemaxwell wrote: post #15

Web scraping isn't a crime - the simple act of downloading the data should not be a problem here. (The reuse of the data might be, depending, but we don't have that information right now.) This doesn't even rise to the level of what Weev did.

He's just making the point that it's unethical, which it is. Even if it boils down to simple bandwidth theft.

I feel the opposite. I don't think it's unethical, even if it might be illegal.

Even calling this "bandwidth theft" is quite the hyperbole—if the server can't handle the bandwidth, then rate limit the requests.

I think if you're serving out pages to the public, you don't really get to tell me what kind of browser I'm allowed to download it with. As long as I'm speaking HTTP, it seems fair.

Sadly, that law has been slowly creeping against this mentality... Lately I feel like I'm some old internet hippy with these views. On a site called "Hacker News", no less.

Re: The Scraping Problem and Ethics

#59

Post by ibmthrowaway218 »

__david__ wrote: post #58
stronglikedan wrote: post #27

Earlier quoted context omitted.

He's just making the point that it's unethical, which it is. Even if it boils down to simple bandwidth theft.

I feel the opposite. I don't think it's unethical, even if it might be illegal. Even calling this "bandwidth theft" is quite the hyperbole—if the server can't handle the bandwidth, then rate limit the requests. I think if you're serving out pages to the public, you don't really get to tell me what kind of browser I'm allowed to download it with. As long as I'm speaking HTTP, it seems fair. Sadly, that law has been sl…

> Sadly, that law has been slowly creeping against this mentality... Lately I feel like I'm some old internet hippy with these views. On a site called "Hacker News", no less.

I guess it's because everyday more and more people on here are finding themselves on the other side of the fence, i.e. finding that some of their users are ripping off their content/site.

Re: The Scraping Problem and Ethics

#60

Post by thaumasiotes »

danielweber wrote: post #46
duey wrote: post #36

Earlier quoted context omitted.

Not in the long term obviously, but you can't make a product extremely difficult to buy and then complain that everyone is taking the easy route.

Talking to someone on the phone is not "extremely difficult." Large companies buy stuff by talking to people on the phone all the time.

Perhaps a method used exclusively by very large companies is that way for a reason? Of course purchasing by talking to someone on the phone is extremely difficult. The only reason people sell that way is to make sure they can hassle you as much as possible during the sale.
Post reply on HN