Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in 2016

franciskim.co

Post reply on HN
Print view View original

341–350 of 402 posts

Re: Web Scraping in 2016

#341

Post by croon »

MarkMc wrote: post #332
Bootvis wrote: post #319

Earlier quoted context omitted.

Which TOS? I might have accepted terms when I created a Google Account but in no way do I agree to a TOS by visiting a URL.

To see the terms that Google thinks you have agreed to, click 'Terms' at the bottom of www.google.com If that doesn't hold up in court, in future on your first visit to Google it will simply display some text and require that you click 'I agree' to continue. Either way, it seems reasonable to me that you should agree to their terms in order to use their service.

So if instead I scrape their site (like they are scraping others) I don't have any opportunity to agree to their terms? Much like their scrapers on other sites?

I'm honestly wondering about the double standard. There is a rational way to discuss morality/ethics and subsequent laws regarding most technical aspects, that often mirrors real world (read: offline/analog) scenarios. It's unfortunate that the legal system has instead been appropriated by lawyers.

Re: Web Scraping in 2016

#342

Post by kuschku »

DashRattlesnake wrote: post #162
cookiecaper wrote: post #131

Earlier quoted context omitted.

It's true that there's no way to know that for sure, but it doesn't make sense that a scraper, by virtue of its being a scraper, is incurring additional load. A scraper is only making requests that a person with a desktop browser or any other appliance that speaks HTTP could make. What's the difference between a user clicking the same button on the page 50 times or holding down F5 and a scraper that pings a page once…

> but it doesn't make sense that a scraper, by virtue of its being a scraper, is incurring additional load It does, because scrapers don't have normal usage patterns. They're robots and behave like robots. > What's the difference between a user clicking the same button on the page 50 times or holding down F5 and a scraper that pings a page once a minute? Typical users aren't usually in the habit of mashing F5, especi…

> Typical users aren't usually in the habit of mashing F5, especially not for robotically long periods of time. It's basically the difference between a theoretical activity and an actual activity.

Just like every university site ever is completely down during signup days because everyone is mashing F5.

Link me your site, I’ll treat it like a college student waiting to be able to sign up for their classes.

Re: Web Scraping in 2016

#343

Post by Bootvis »

jacquesm wrote: post #330
Bootvis wrote: post #319

Earlier quoted context omitted.

Which TOS? I might have accepted terms when I created a Google Account but in no way do I agree to a TOS by visiting a URL.

Try using google from a fresh install, they´ll force you to accept their TOS.

Are they A/B testing this or is acceptance IP-based? I reinstalled recently and I didn't see it. Firefox in private navigation mode also lets me use it without forcing me to agree with anything.

Re: Web Scraping in 2016

#344

Post by kuschku »

greglindahl wrote: post #209
buro9 wrote: post #177

Earlier quoted context omitted.

The UK has a database law: https://en.wikibooks.org/wiki/UK_Database_Law#Database_Right If you scrape, and effectively reconstitute a database, then so long as the database originally had a "substantial investment" in it's "obtaining, verifying or presenting the contents" then yup... you have breached the database right, which is a modified form of copyright. You may access said database (via the web), but as soon as…

Actually, such database laws are rare. The US and Canada don't have one. See Feist v. Rural Telephone for an example of databases getting scraped & the scraper winning in court.

Actually, the US does have one.

The World Copyright Treaty of the WIPO, which the US also signed, enforces in Article 5 that every member country has to have a database law of this kind.

    ________________________
> Article 5: Compilations of Data (Databases)

> Compilations of data or other material, in any form, which by reason of the selection or arrangement of their contents constitute intellectual creations, are protected as such. This protection does not extend to the data or the material itself and is without prejudice to any copyright subsisting in the data or material contained in the compilation.

http://www.wipo.int/wipolex/en/treaties/text.jsp?file_id=295...

> United States of America

> Signature: April 12, 1997

> Ratification: September 14, 1999

> In Force: March 6, 2002

    ________________________
In fact, this fucking treaty is the only reason so many countries even have that at all – the EU didn’t have any Database Law before it was created, and the US threatened (as always) to boycott any country not signing.

Re: Web Scraping in 2016

#345

Post by lugg »

comex wrote: post #109
cookiecaper wrote: post #83

Earlier quoted context omitted.

>How can TOS have legal power for the case scraping? A website is a public property. If I'm visiting it without logging in, I don't have a chance to accept TOS. This is called "clickwrap". There is usually a notice in the footer of each page that says something like "By using this site, you agree to our Terms of Service." Typically, this kind of notice has been held enforceable. More recently, judges have been demand…

"Clickwrap" refers to situations where you have to click through before using the service, hence the name. Agreements which are simply a passive notice in a footer somewhere are called "browse-wrap", and are much less likely to be considered enforceable: https://en.m.wikipedia.org/wiki/Browse_wrap

Heh, fwiw, there is another one, shrink-wrap agreements, where you can't read the agreement until you've removed the shrink-wrap but doing so means you've agreed.

Re: Web Scraping in 2016

#346

Post by Chris2048 »

hchasestevens wrote: post #61
cookiecaper wrote: post #50

Earlier quoted context omitted.

Eh, this is pretty weak. Scrapers are no different from other browsing devices. The web speaks HTTP. There's no reason that using another HTTP browser would cause any disparate impact just by virtue of not being a conventional desktop browser -- you've thrown out a pretty absurd hypothetical. In fact, scrapers usually cause less impact because they usually don't download images or execute JavaScript. I did an analysi…

A small counterpoint to this -- in the airline industry, it's relatively commonplace for seat reservations to be made for a user _before_ payment has occurred. In this case, if you're mirroring normal browser activity, you can (temporarily) reduce availability on a flight, potentially even bumping up the price for other, legitimate users, and almost certainly causing the airline to incur costs beyond normal bandwidth…

Cinemas and some online shops do the same. I've always wondered if it's possible to block any tickets sales for entire flights/screenings/products this way.

And if airline tickets are based on supply v demand, it might even be possible to drive down ticket prices by suddenly dropping a load of blocks near to the flight date.

Re: Web Scraping in 2016

#347

Post by [deleted] »

sjwright wrote: post #283
tangue wrote: post #200

Earlier quoted context omitted.

It would be ok if it wasn't "You can't scrap my site. Unless of course you're Google" this double standard drives me mad.

As the owner of a large website, I don't care what you think. I block by default and whitelist when I decide it's in my interest. If you don't think this is reasonable, chances are you've never run a large website, or analyzed the logs of a large website. You'd be astonished how much robotic activity you'll receive. If left unchecked it can easily swamp legitimate traffic. Unless you have a way for me to automaticall…

[deleted]

Re: Web Scraping in 2016

#348

Post by madshiva »

hobs wrote: post #13
minimaxir wrote: post #2

Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…

This post is kind of crazy, aggrandizing bad behavior and misuse of other's resources against their will. Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread.

So you think that people should waste their time solving captcha are the solution? people are paid to solve captcha, there's always something like that and then the users suffer more. It's not a solution at all.

Re: Web Scraping in 2016

#350

Post by keithb- »

croon wrote: post #341
MarkMc wrote: post #332

Earlier quoted context omitted.

To see the terms that Google thinks you have agreed to, click 'Terms' at the bottom of www.google.com If that doesn't hold up in court, in future on your first visit to Google it will simply display some text and require that you click 'I agree' to continue. Either way, it seems reasonable to me that you should agree to their terms in order to use their service.

So if instead I scrape their site (like they are scraping others) I don't have any opportunity to agree to their terms? Much like their scrapers on other sites? I'm honestly wondering about the double standard. There is a rational way to discuss morality/ethics and subsequent laws regarding most technical aspects, that often mirrors real world (read: offline/analog) scenarios. It's unfortunate that the legal system h…

>> It's unfortunate that the legal system has instead been appropriated by lawyers.

omg, really?

It's unfortunate that the internet has instead been appropriated by hackers. It's unfortunate that the stock market has instead been appropriated by traders. It's unfortunate that the asylum has instead been appropriated by inmates.

Post reply on HN