Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Scraping Problem and Ethics

blog.osvdb.org

Post reply on HN
Print view View original

111–120 of 130 posts

Re: The Scraping Problem and Ethics

#111

Post by buro9 »

ChuckMcM wrote: post #28

This is one of the more interesting policy questions on the web. Our search engine crawls a lot of blogs and what not on the web, criminals who want to find unpatched wordpress sites try to acrape our crawl by sending automated (scripted) queries to find them. We have developed a number of defenses over the years and pretty regularly ban them[1]. Here is the weird part though, if they hired 300 people on mechanical t…

> We did an experiment where we would 'grep the web' for you, basically run a regex over a multi-billion page crawl, give you the first 50 results for "free" and you could buy the complete set. I think we sold exactly one of those.

Sounds like you didn't advertise it right. I've been looking for a "grep the web" service for a while.

We're going to use https://builtwith.com/ to achieve that exact thing. It will cost me $295 : https://builtwith.com/plans

For that $295 I will get a list of all domains using a rival technology... a list of sales leads.

I'm still short of what I want to have... a prioritised list of sales leads.

So I will write my own scraper to go through those results and scrape every one of those so that I can pull some info from the HTML page to tell me how large those customers might be. I'm not aiming for the largest (easy to find, costly to win), nor the smallest (time consuming and pointless to win), but the median.

I would definitely pay for a "grep the web" that allowed me to match pages by text signatures in the HTML, and then extract part of the DOM as values, and return the list of "url + extracted values" for the matching hits.

I'd consider that to be worth similar amounts to what BuiltWith are chaging, but I'd add more and would go up to $500 assuming that the results come with the extracted values as a CSV file of some kinda and the quality and completeness of the report is high.

People will pay for "grep the web", especially if you sell it to them as something they know they really want: "sales leads".

Re: The Scraping Problem and Ethics

#112

Post by jerf »

notastartup wrote: post #108
test1235 wrote: post #74

Earlier quoted context omitted.

Speaking as devil's advocate, it might just be more convenient to steal the data. Maybe I want to use your data casually once, and I don't want to sign up and give you all my contact details and subscribe to your annual plan with all the other optional extras. Tough shit, you say? I'll just steal it then, and not because I can't afford it, but because you're making it hard to pay.

but how is it stealing when one does not lose inventory? If one person scrapes a page, did you lose the source code? Does it not become available for the next visitor? What possible loss do you incur that is directly tied to your data? When you make data public with the intent of being readily accessible by the public, how can you claim theft when you are achieving what you set out to do? Does the accelerated rate of…

"but how is it stealing when one does not lose inventory?"

Scraping is not necessarily a no-victim situation. Even today after this stuff has gotten cheaper, you're costing them bandwidth fees, and likely increasing their server storage and CPU fees if it's on a metered hosting service, which is quite likely nowadays. If you degrade their site's functionality, you may chase away paying customers.

We need not hypothesize crazy third-order effects; you are taking money out of their pockets by the act of scraping itself, independent of the question of the value of the content.

"What about Google? etc." - robots.txt-honoring scrapers that don't hammer the sites at least have a plausible claim to permission. Scrapers are quite likely to be ignoring the robots.txt.

Re: The Scraping Problem and Ethics

#113

Post by pbreit »

It's hard to get excited about this. The data owner supposedly has a mission to make the data available but then is concerned with such a thing. It doesn't strike me as that difficult or expensive to make that sized database available.

Re: The Scraping Problem and Ethics

#114

Post by scott_karana »

yp_all wrote: post #96
ChuckMcM wrote: post #28

This is one of the more interesting policy questions on the web. Our search engine crawls a lot of blogs and what not on the web, criminals who want to find unpatched wordpress sites try to acrape our crawl by sending automated (scripted) queries to find them. We have developed a number of defenses over the years and pretty regularly ban them[1]. Here is the weird part though, if they hired 300 people on mechanical t…

Pot calling the kettle black. Crawling others' websites then selling ads. Scraping others' websites then selling ads. Selling access to user-generated content. Amusing to watch these folks argue about ethics. Who owns the copyrights in this data? Surely not the one who is demanding that you pay a license fee. These "services" are middlemen, plain and simple. This might be why McAfee was wondering about how much manua…

If they're a middleman, and it's such an easy "service", why didn't McAfee just bypass them and get the data from the original sources, rather than do the wrong thing?

Re: The Scraping Problem and Ethics

#115

Post by jmzbond »

This also goes way beyond scraping, in the sense of small start-ups doing all kinds of hacks or clever manips to stay within the bounds of a free trial service.

As someone who sees both sides… I don't know what to say. I'm running test landing pages on Heroku with New Relic that pings the sites every minute to ensure the dynes keep spinning and my users don't experience downtime. While I'm careful to stay within fair use, this is at best obnoxious, because if everyone did this, Heroku would certainly need to redefine what's free. From my POV though, I am a bootstrapped entrepreneur and supporting 5 landing pages. I simply don't have resources to pay for a dyno and test everything I have in my head, especially not combined with the many other resources I'd need to start paying for as well.

Or consider the kid in Florida who used Parse' free account for hundreds of thousands of users. [1] (The article was on HN a few weeks ago, this was not it's central point, just something I took away relevant to this comment.)

Part of the cause I think is that we live in a world where we're so used to having things be free, it becomes an entitlement. Another is that all these examples of start-up hacks and hustle stories, we kind of laud, don't we? Everyone talks about how Airbnb scraped Craigslist and got a huge boon that way, but few in critical tones. Should we? Or is that how competition and new products get created (i.e., if the scraping hadn't happened, perhaps Airbnb and the whole sharing economy would be less successful today).

These are philosophical questions, and I don't really have a solution, but they are things to think about.

[1] http://pando.com/2014/04/30/how-a-florida-kids-stupid-app-sa...

Re: The Scraping Problem and Ethics

#116

Post by ForHackernews »

ibmthrowaway218 wrote: post #59
__david__ wrote: post #58

Earlier quoted context omitted.

I feel the opposite. I don't think it's unethical, even if it might be illegal. Even calling this "bandwidth theft" is quite the hyperbole—if the server can't handle the bandwidth, then rate limit the requests. I think if you're serving out pages to the public, you don't really get to tell me what kind of browser I'm allowed to download it with. As long as I'm speaking HTTP, it seems fair. Sadly, that law has been sl…

> Sadly, that law has been slowly creeping against this mentality... Lately I feel like I'm some old internet hippy with these views. On a site called "Hacker News", no less. I guess it's because everyday more and more people on here are finding themselves on the other side of the fence, i.e. finding that some of their users are ripping off their content/site.

Maybe people need to come up with a better business model for websites than "Put up some 'content' and sell ads next to it."

Sell a product or service.

Re: The Scraping Problem and Ethics

#117

Post by parktheredcar »

ksk wrote: post #68
newaccountfool wrote: post #66

Earlier quoted context omitted.

>Is that a bad thing? Yes, no question about it. More players in the market means more competition and more competition means a better service.

>More players in the market means more competition and more competition means a better service I'm not sure if this is a joke. So, I'll refrain from replying.

Could you explain your point of view? I'd like to understand both sides here and I think I understand why more competition would be good, but now why it wouldn't be.

Re: The Scraping Problem and Ethics

#118

Post by calroc »

freshhawk wrote: post #32
cproctor wrote: post #24

I can't help recalling a post here a couple of years ago about the concept of "hellbanning" scammers on ecommerce sites--in short, making it look like everything is going fine, while actually isolating them completely from your business logic. Orders with stolen cards appear to go through, and send confirmation emails, but no real order is generated... In this case, you could transparently poison the results served t…

Heh, I've always called this the map makers trick (found an article about it here: https://theweek.com/article/index/241967/trap-streets-the-cr... ) although I guess that is specific to putting a small amount of fake data in your dataset to prove someone else used it. The hellbanning metaphor does fit for return large amounts of poison results. It could be like spiking trees I guess, but that depends on the potential…

The point of spiking trees isn't to hurt people, in fact the tree-huggers tell the loggers that the trees have been spiked so that they won't attempt to cut the trees. The point of the tactic is to prevent damage to the forest, not to actually hurt anyone.

Re: The Scraping Problem and Ethics

#119

Post by yhager »

muyuu wrote: post #70
yhager wrote: post #37

Earlier quoted context omitted.

Not completely by the specification, but I think this one works as expected. user-agent: * disallow: / user-agent: Googlebot allow: /

Not sure it's a good idea to block all search engines but Google.

Probably not, but if they don't want to be scraped, the first place they should notify about this is robots.txt. I was just stating an example that you can allow some bots and not others if you like.

Of course, forging your user agent, disobeying robots.txt or scraping after you were told not to, is wrong ethically.

Post reply on HN