Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in 2016

franciskim.co

Post reply on HN
Print view View original

171–180 of 402 posts

Re: Web Scraping in 2016

#171

Post by buzzdenver »

ysavir wrote: post #130
minimaxir wrote: post #82

Earlier quoted context omitted.

That analogy is not equitable. If you take photographs of a building while on the building's property , they have the right to tell you to stop, or call the police to escort you off if you refuse to do so.

Let's go with a more apt analogy: If you're entering a country, do its laws not apply to you until you've seen a copy of them? "Oh, sorry, no one told me theft is illegal here. Where does it say that? Oh, I see. Okay. I'll stop now. Thanks for letting me know." If you cross the border without necessary documents, does that country have no right to detain you, simply because you haven't checked the laws? Just because…

That is not a good analogy. There is such a thing as reasonable expectations when visiting a website, so you do not need to read the TOS. Otherwise I could put "you own me $1000 for visiting my site" into the TOS. In other words, just clicking on a page does not constitute entering into a contract with the website. Registering and accepting the TOS does, but that still doesn't mean that anything in the TOS is enforceable.

Re: Web Scraping in 2016

#172

Post by paulryanrogers »

ramblenode wrote: post #145
franciskim wrote: post #63

Earlier quoted context omitted.

Selenium grid runs in docker, so it's easy to have multiple instances running. Better control.

What are the advantages of this versus a thread pool of web drivers? I'm not really familiar with Selenium Grid.

Grid can dynamically dispatch based on the browser and capabilities you want when you create the session.

Re: Web Scraping in 2016

#173

Post by headmelted »

minimaxir wrote: post #2

Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…

One thing I'm not quite clear on here.

I understand the use of ToS clauses to prevent scraping but I do kind of wonder to what extent they have authority here.

IANAL, but surely this would fall under copyright law? While re-publishing copyright-protected data without consent is probably unlawful in your region (like scraping an art site and re-posting the images), I wouldn't think just scraping data points for a different purpose (like scraping amazon for the purposes of price comparison) is nearly so clear cut (or enforceable), but maybe I'm just naive.

Re: Web Scraping in 2016

#174

Post by fareesh »

hk__2 wrote: post #135
fareesh wrote: post #112

Earlier quoted context omitted.

Is it common for developers in the eCommerce space to use scrapers as a means to aggressively push automated price-match algorithms? I've been asked to do this a number of times, was just curious as to how prevalent it is.

Yes, everybody scrapes the prices of the others.

With stuff like Facebook opengraph (e.g. og:price) and other meta tags meant to help search engines and social networks get this sort of data to display inline, do you think it's inevitable that complex scraping will no longer be needed in a practical sense since everyone will be inadvertently optimizing their markup in a way that you could write a really simple parser to grab the data?

Re: Web Scraping in 2016

#175

Post by cookiecaper »

mthoms wrote: post #167
cookiecaper wrote: post #119

Earlier quoted context omitted.

Yeah, Google violates the CFAA and infringes on copyright as a matter of course. Their service would be impossible if they weren't doing so. The main difference when Google was small was that Google was not dependent on any data source in particular, so even if someone denied their robot or sued them, they could cease and desist without affecting the overall value of their offering. This is different if you are getti…

Google's crawling and caching has been largely found to be fair use and thus is not considered to be infringing copyrights. https://en.wikipedia.org/wiki/Field_v._Google,_Inc . There are similar rulings for thumbnail images: https://en.wikipedia.org/wiki/Perfect_10,_Inc._v._Amazon.com... . And of course books: https://en.wikipedia.org/wiki/Authors_Guild,_Inc._v._Google,... .

Incidentally, this only further proves my point. If you're a big company that's retained massive law firms, you can successfully raise a fair use and implied license defense. If you're not, you can neither mount a strong offense against that defense nor mount a strong defense against Google's hypocritical offense if you find yourself on the other side.

Google's primary out here is its reputation (not guarantee) for obeying robots.txt. If Google indexed a page that disallowed it in robots.txt, the case would be much stronger. There's also the unofficial out, which is that judges think Google is a cool large company, so they rule in their favor based on their personal biases.

Fair use is a case-by-case basis, so you can't say that Google's infringing conduct is generally accepted to be fair use. The EFF had to take on Universal in Lenz v. Universal Music Group, and that went up to the Supreme Court. That's how individuals are left to assert their fair use rights.

Re: Web Scraping in 2016

#176

Post by headmelted »

I actually love Selenium for this purpose, for much the same reasons the author mentions here.

It's almost impossible for a website to reliably detect that a client web browser is being automated, and I find I can make Selenium scripts much more adaptable to breaking changes in websites when they occur than I can when hooking up my code directly.

I actually disagree with the contention that Selenium is slower than directly scraping though. The Firefox driver has always been lightning fast for me and the bottleneck is almost always server requests that would have been necessary either way.

Re: Web Scraping in 2016

#177

Post by buro9 »

twa927 wrote: post #73
minimaxir wrote: post #2

Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…

How can TOS have legal power for the case scraping? A website is a public property. If I'm visiting it without logging in, I don't have a chance to accept TOS. Imagine a hotel that makes guests sign a document saying they will not make photographs of the building. If I'm not a guest, I can take photographs of it and I can't even know that would be illegal.

The UK has a database law:

https://en.wikibooks.org/wiki/UK_Database_Law#Database_Right

If you scrape, and effectively reconstitute a database, then so long as the database originally had a "substantial investment" in it's "obtaining, verifying or presenting the contents" then yup... you have breached the database right, which is a modified form of copyright.

You may access said database (via the web), but as soon as you start reconstituting the database from scraping... you're in breach.

It's a law, it is illegal in the UK, I'm sure most countries have some equivalent law on their books, all of the EU does. The law looks recent, but UK copyright and patent used to cover it, the 1997 date is just a separate statute to clarify the position.

Re: Web Scraping in 2016

#178

Post by cookiecaper »

muglug wrote: post #166
cookiecaper wrote: post #121

Earlier quoted context omitted.

There are already 6-8 major scrapers that do this constantly, across the whole internet, called search engines. You can't handle that? What if you get a normal user who says "Hey, I wanna see some of the lesser known authors on this platform" and opens up a hundred tabs with rarely-read blogs? What if you get 10 users who decide to do that on the same day? Is it reasonable to sue them? Should there be a legal protect…

Search engines respect robots.txt – not sure many scrapers do.

First, search engines are scrapers. No need to make a distinction.

Second, search engines don't always respect robots.txt. They sometimes do. Even Google itself says it may still contact a page that has disallowed it. [0]

Third, robots.txt is just a convention. There's no reason to assume it has any binding authority. Users should be able to access public HTTP resources with any non-disruptive HTTP client, regardless of the end server's opinion.

[0] "You should not use robots.txt as a means to hide your web pages from Google Search results. This is because other pages might point to your page, and your page could get indexed that way, avoiding the robots.txt file." / http://archive.is/A5zh8

Re: Web Scraping in 2016

#179

Post by Greg-J »

fake-name wrote: post #49

I do a significant amount of scraping for hobby projects, albeit mostly open websites. As a result, I've gotten pretty good a circumventing rate-limiting and most other controls. I suspect I'm one of those bad people your parents tell you to avoid - by that I mean I completely ignore robots.txt. At this point, my architecture has settled on a distributed RPC system with a rotating swarm of clients. I use RabbitMQ for…

How can I get ahold of you directly?

Re: Web Scraping in 2016

#180

Post by cookiecaper »

lossolo wrote: post #160
cookiecaper wrote: post #149

Earlier quoted context omitted.

They aren't, or at least, they won't be if Google decides it doesn't like them anymore and decides to bring the matter to court. The CFAA says it's a crime to exceed "authorized access". Authorized access is whatever the server's owner says it is. If they change their mind, you must cease and desist or risk both civil and criminal penalties. A contract defining the length and nature of your authorization from the ser…

You forgot to add: In USA.

Unfortunately this is true in almost all of the developed world. While developing countries may not have specific legal prohibitions, we know that that doesn't stop big companies from having their way. Heck, even a first-world country like Sweden couldn't resist the pressure from Hollywood to prosecute and jail the operators of the Pirate Bay, which had long been recognized as totally legal in Sweden.

Another issue is that on the internet, jurisdiction is a very messy affair. An American judge will likely determine that California and/or the federal government has jurisdiction over such a case because Google is based in California. Most developed countries have treaties with one another that allow them to enforce foreign civil judgments on behalf of the jurisdiction that entered them. Most developed countries also have mutual extradition treaties. The countries that don't can easily be paid off by an interested party.

Post reply on HN