Web Scraping: Bypassing “403 Forbidden,” captchas, and more

Post by brilliantcode » Thu, Mar 16, 2017, 6:06 PM UTC

cookiecaper wrote: post #82

brilliantcode wrote: post #65

Earlier quoted context omitted.

ToS is not a legally binding mutual agreement between the website and the scraper. Only when it's materialized in the form of C&D and it caused damage to the website. In the QVC vs Resultly, they caused a website crashed. In the Craigslist vs 3Taps, the damages were non-existant but they still won because of the C&D letters. So if a website sends you a letter to stop, do not continue scraping them. Until that happens…

>In the Craigslist vs 3Taps, the damages were non-existant but they still won because of the C&D letters. (Technically they settled.) Read up on that case and you'll see that even absent a C&D, the judge reasoned that Craiglist's IP ban against 3Taps was a separate incident of affirmatively communicating its intention that 3Taps refrain from accessing the site: > The calculus is different where a user is altogether b…

Post by cookiecaper » Thu, Mar 16, 2017, 6:14 PM UTC

codedokode wrote: post #47

cookiecaper wrote: post #42

Earlier quoted context omitted.

>Doesn't publishing information on a public web server equals to granting authorization to download it? Why publish it otherwise? This is the argument that there is an implied license. The counterargument is that the user agreed to the Terms of Use which explicitly defined automated access as violative. In addition, these cases generally begin with a cease and desist demand letter, which explicitly informs the allege…

I think that is wrong too. I think that copyright laws should make a distinction between making and distriubuting a copy by a person (for example uploading copyrighted file to a website) and technical processes that happen inside a computer. Copying something from NIC buffers to memory should not be "copying" under copyright law.

    > The copies of webpages stored automatically in a computer's cache or 
    > random access memory ("RAM") upon a viewing of the webpage fall within 
    > the Copyright Act's definition of "copy." See, e.g., MAI Systems Corp. 
    > v. Peak Computer, Inc., 991 F.2d 511, 519 (9th Cir. 1993) ("We recognize 
    > that these authorities are somewhat troubling since they do not specify 
    > that a copy is created regardless of whether the software is loaded into 
    > the RAM, the hard disk or the read only memory (`ROM'). However, since 
    > we find that the copy created in the RAM can be `perceived, reproduced, 
    > or otherwise communicated,' we hold that the loading of software into 
    > the RAM creates a copy under the Copyright Act.") See also Twentieth 
    > Century Fox Film Corp. v. Cablevision Systems Corp., 478 F.Supp. 2d 607, 
    > 621 (S.D.N.Y. 2007) (agreeing with the "numerous courts [that] have held 
    > that the transmission of information through a computer's random access 
    > memory or RAM . . . creates a `copy' for purposes of the Copyright Act," 
    > and citing cases.) Thus, copies of ticketmaster.com webpages 
    > automatically stored on a viewer's computer are "copies" within the 
    > meaning of the Copyright Act.

Post by graphememes » Thu, Mar 16, 2017, 6:20 PM UTC

pjc50 wrote: post #71

thefifthsetpin wrote: post #62

Better solution: pay target-site.com to start building an API for you. Pros: * You'll be working with them rather than against them. * Your solution will be far more robust. * It'll be way cheaper, supposing you account for the ongoing maintenance costs of your fragile scraper. * You're eliminating the possibility that you'll have to deal with legal antagonism * Good anti-scraper defenses are far more sophisticated t…

> pay target-site.com to start building an API for you. When has that ever worked?

Post by heipei » Thu, Mar 16, 2017, 6:24 PM UTC

foxylion wrote: post #51

I'm curious what others use to scrape modern (javascript based) web applications. The old web (html and links) work fine with tools like Scrapy, but for modern applications which rely on javascript this does no longer work. For my last project I used a chrome plugin which controlled the browsers url locations and clicks. Results where transmitted to a backend server. New jobs (clicks, change urls) where retrieved fro…

Post by brilliantcode » Thu, Mar 16, 2017, 6:33 PM UTC

cookiecaper wrote: post #94

brilliantcode wrote: post #91

Earlier quoted context omitted.

So the answer is don't get IP banned. That's easy to solve.

That's potentially an answer if the judge decides that the browsewrap notice was not sufficiently conspicuous to constitute a binding agreement, etc. I would guess that most judges would not be charitable to someone pretending that they've circumvented this by rotating through proxies pre-emptively. In fact, this would likely work against the defendant as it'd be evidence of willful infringement, which is typically 3…