Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in 2016

franciskim.co

Post reply on HN
Print view View original

181–190 of 402 posts

Re: Web Scraping in 2016

#181

Post by Mikushi »

hobs wrote: post #13
minimaxir wrote: post #2

Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…

This post is kind of crazy, aggrandizing bad behavior and misuse of other's resources against their will. Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread.

Instagram or Facebook, they thrive on stolen or relinked content and monetize it day in day out.

Being amazed at this kind of bad behaviour where the targets are some of the most despicable companies on the web is a bit ironic. Scrape away, these companies hurt the web, let's hurt them (even though, all the scraping in the world won't have any impact).

Re: Web Scraping in 2016

#182

Post by Mikushi »

hobs wrote: post #13
minimaxir wrote: post #2

Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…

This post is kind of crazy, aggrandizing bad behavior and misuse of other's resources against their will. Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread.

Ah, Instagram or Facebook, they thrive on stolen or relinked content and monetize it day in day out. Scrapping is the way of the web, that you like it or not.

Really, being amazed at this kind of bad behaviour where the targets are some of the most despicable companies on the web is a bit ironic. Scrape away, these companies hurt the web, let's hurt them.

Re: Web Scraping in 2016

#183

Post by hobs »

cookiecaper wrote: post #178
muglug wrote: post #166

Earlier quoted context omitted.

Search engines respect robots.txt – not sure many scrapers do.

First, search engines are scrapers. No need to make a distinction. Second, search engines don't always respect robots.txt. They sometimes do. Even Google itself says it may still contact a page that has disallowed it. [0] Third, robots.txt is just a convention. There's no reason to assume it has any binding authority. Users should be able to access public HTTP resources with any non-disruptive HTTP client, regardless…

And the original point of my comment was that doing this is extremely rude and not appropriate, not that it couldn't be done or that others weren't doing it.

Feel free to send any request to any server you want, it is certainly up to them to decide whether or not to serve it, but that doesnt absolve you of guilt from scraping someone's site when they explicitly ask you not to.

Re: Web Scraping in 2016

#184

Post by malz »

hobs wrote: post #13
minimaxir wrote: post #2

Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…

This post is kind of crazy, aggrandizing bad behavior and misuse of other's resources against their will. Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread.

> misuse of other's resources against their will

Nonsense, there is no implication that this activity is illicit. Many sites (I have worked with hundreds) are happy to be included in my service, but don't have the technical ability to provide a data feed. They were delighted when I told them I could aggregate their content without any extra work on their part.

We respect TOS, we respect robots.txt and so on. Just because you study scraping techniques doesn't mean you intend to break the law.

> Breaking captchas and the like is basically blackhat work

Um, captchas only work if they work. If breaking them is trivial, they shouldn't exist. Don't shoot the messenger for pointing out the front door is unlocked.

Re: Web Scraping in 2016

#185

Post by hobs »

Mikushi wrote: post #181
hobs wrote: post #13

Earlier quoted context omitted.

This post is kind of crazy, aggrandizing bad behavior and misuse of other's resources against their will. Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread.

Instagram or Facebook, they thrive on stolen or relinked content and monetize it day in day out. Being amazed at this kind of bad behaviour where the targets are some of the most despicable companies on the web is a bit ironic. Scrape away, these companies hurt the web, let's hurt them (even though, all the scraping in the world won't have any impact).

So its moral to continue bad behavior because someone else did it?

Re: Web Scraping in 2016

#187

Post by downandout »

minimaxir wrote: post #2

Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…

Obviously it's a good idea to follow TOS. But as a practical matter, they have to know that you're doing it before they can take action. You wouldn't want to put up a site announcing that you're selling scraped LinkedIn data, for example. But if that data is valuable to your business - collecting names of people that work in certain positions at certain companies so that you can do targeted snail mail campaigns for example - you could quietly scrape and use the data without issue. Use proxies and prosper.

Re: Web Scraping in 2016

#188

Post by cookiecaper »

DashRattlesnake wrote: post #162
cookiecaper wrote: post #131

Earlier quoted context omitted.

It's true that there's no way to know that for sure, but it doesn't make sense that a scraper, by virtue of its being a scraper, is incurring additional load. A scraper is only making requests that a person with a desktop browser or any other appliance that speaks HTTP could make. What's the difference between a user clicking the same button on the page 50 times or holding down F5 and a scraper that pings a page once…

> but it doesn't make sense that a scraper, by virtue of its being a scraper, is incurring additional load It does, because scrapers don't have normal usage patterns. They're robots and behave like robots. > What's the difference between a user clicking the same button on the page 50 times or holding down F5 and a scraper that pings a page once a minute? Typical users aren't usually in the habit of mashing F5, especi…

Scrapers are usually coded to have as regular of a usage pattern as possible, so that the data they retrieve is as much like the data the end user would receive as possible.

For example, Googlebot does everything in its power to ensure that it sees pages the same way that end users sees them, executing JavaScript and performing OCR to try to read information conveyed in images. Google also has non-Googlebot scans to try to determine if a page is serving different content to Googlebot-labeled scans, and they penalize sites that they suspect of doing this.

While it is true that someone could write a scraper that obviously behaved robotically, it is also true that someone could use their desktop browser in a robotic way. Mashing F5 is so common that there are many ancient memes referring to and making jokes about that activity. There are extensions that end users use to record browser macros, behaviors they want their browser to repeat over and over again.

However, this conversation about whether scrapers behave robotically or not is moot because a web site shouldn't break down under load when someone uses it in a slightly-irregular way. The obvious, crappy scrapers are trivial to block. The ones that blend into the traffic are no harm, no foul. If you can't tell the difference between an optimized browser like a scraper and a general-purpose browser like Chrome, why shouldn't it be allowed to talk to your site?

Re: Web Scraping in 2016

#189

Post by slig »

user5994461 wrote: post #129
slig wrote: post #11

I wonder how effective the CloudFlare anti-scrapper protection is against this approach of breaking captchas. Also, I find it interesting that big websites don't just block all traffic from AWS IPs as they do with Tor.

There can be legitimate traffic coming from AWS, if not the site itself. It's especially true when the site provides an API and is meant to be integrated by people/companies. In which case, the AWS traffic is likely to include major and/or important and/or paying customers. You really don't want to block that. On the other hand, Tor is likely to be 90% evil. When in doubt, just block it. (That makes me think, I shoul…

> There can be legitimate traffic coming from AWS, if not the site itself.

The traffic from the site itself, if it's hosted there, would come from the intranet IP address, right? Not the public facing one.

> It's especially true when the site provides an API and is meant to be integrated by people/companies. In which case, the AWS traffic is likely to include major and/or important and/or paying customers. You really don't want to block that.

Agreed, but it's fairly easy to block the AWS IP traffic on web endpoints and not on the API endpoints.

Re: Web Scraping in 2016

#190

Post by Asooka »

mindslight wrote: post #88
minimaxir wrote: post #82

Earlier quoted context omitted.

That analogy is not equitable. If you take photographs of a building while on the building's property , they have the right to tell you to stop, or call the police to escort you off if you refuse to do so.

Sure, but they do not have the right to retroactively declare you as having been trespassing, nor even to preemptively put up a "no photography" sign and have you arrested for trespassing if you disobey it. The entire point of protocols is to precisely define the terms of communication. The status code is '200 OK', not '200 OK/Asterisk'. But of course if lawlers didn't force themselves into the situation, they'd be o…

Well, it's just a technical response code. 200 OK - everything went as normal, here's your data. By the same margin, the door on a shop doesn't stop you walking out without paying and the road markings don't stop you from driving in the wrong lane.

I think imbuing technical protocols with legal implications would be even worse than the current situation since then changing anything on a protocol would require changing the law and getting a protocol implementation slightly wrong would carry real-world legal repercussions on the order of licensing your work in the public domain rather than retaining copyright. Let the lawyers make the law and check the human terms of service before using the data. Trying to out-lawyer the lawyers is like challenging a hedgehog to a butt-kicking brawl.

Post reply on HN