Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping: Bypassing “403 Forbidden,” captchas, and more

sangaline.com

Post reply on HN
Print view View original

131–140 of 232 posts

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#131

Post by brilliantcode »

cookiecaper wrote: post #101
brilliantcode wrote: post #99

Earlier quoted context omitted.

Well, looks like all the HN user's of Scrapy better lawyer up because Scrapy Cloud offers exactly that, as do rest of the web scraping vendors like Mozenda out on the market. They've all been around for 10+ years, doesn't seem like this is an issue for them.

Scrapinghub has several proactive/preventative restrictions on the sites they'll allow users to access because they're trying to avoid such liability. They've been successful up to this point and that's great. That doesn't mean that what they're doing is not a legal grey area. For scraping-related activities, Scrapinghub would probably be the party sued, as was the case in 3Taps , though the clients could probably al…

The more I read your comment, the less I'm worried. It's clear that you are not a lawyer but someone just overly reacting to perceived legal liabilities by simply generalizing court cases and attempting to reach a conclusion that tries to fit everyone.

Businesses that utilize web scraping to achieve business goals at a direct expense of another business will get you in trouble not because of web scraping but simply trying to create competition. Businesses with a large cash use litigation to snuff out competition because their businesses are largely undefensible without such forceful litigation ex. craigslist would not exist if they let anyone scrape them.

Businesses that build and sell web scraping sevices and tools are less likely to be impacted for the same reasons if they comply with formal requests to stop scraping. 3Taps received notices beyond just IP ban (this alone does not set enough of a context) but they chose to ignore it and continue on. 3Taps had enough of a financial motivation on the line to put out their neck for their customer, PadMapper. Pretty fucking stupid if you ask me, no one customer is worth risking the entirety of your business operation.

It's far more likely that the law exists to serve those who exploit it to protect their business interests. Generalizing and extrapolating based on a few court cases with their own dynamic set of variables and exceptions as fact is dangerous advice.

I just want to warn people reading your comments not to take it word for word as the reality is far far less legally hostile-you are too small for people to go after and not an existential threat to the target website.

The argument that web scraping puts strain on web servers is a pretty laughable defense. Craigslist alone gets millions of hits every day but can't serve pages requested by a python script? 3taps fucked themselves because they took money AND they put their neck out for their customer.

That's the lesson here, don't risk your entire business for one customer. It's not fair to the rest of your customer base.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#132

Post by [deleted] »

elmigranto wrote: post #122
hsod wrote: post #88

Earlier quoted context omitted.

I disagree. The person who owns the server should get to decide who has access and under what circumstances. Joe Scraper, having invested nothing, has no claim or rights to it whatsoever. Furthermore, the fact that the server gives 200 responses is not sufficient implied permission IF a no-scraping policy has been communicated in some other way such as robots.txt or (clearly communicated) TOS. The techno-nihilist arg…

Is there a law requiring me to read and accept TOS or robots.txt before opening a website? What is the legal status of those? What if TOS says "you owe us $100 for every byte included in this TOS"? What if I have my own TOS about opening their links where they owe me $100 per byte sent?

[deleted]

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#133

Post by brilliantcode »

cookiecaper wrote: post #100
brilliantcode wrote: post #83

Earlier quoted context omitted.

Well the difference is pretty clear between your case and the rest. Developers scraping a website isn't going anywhere. A business reliant on scraped data is making money off of it. That will lend you in precarious situation more. My criticism was that you mixed in service providers and tool providers that enable businesses to make money off scraped data-the vendor cannot be held responsible for misbehaving clients,…

>Well the difference is pretty clear between your case and the rest. Developers scraping a website isn't going anywhere. A business reliant on scraped data is making money off of it. That will lend you in precarious situation more. Developers presumably scrape websites because the data is of some value to them, frequently commercial value. Google's entire value proposition is based on scraped data, and it's one of th…

3Taps became liable because they put their neck out for PadMapper even after they received written letters.

It was a poorly executed business strategy because they were up against powerful legal team.

Best thing to do if you receive C&D or requests to stop scraping, best to not continue and just let that customer go.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#134

Post by brilliantcode »

IanCal wrote: post #103
brilliantcode wrote: post #67

Earlier quoted context omitted.

A few things turn me off about Scrapy is that it feels over engineered for what it does. Why do I need an entire framework? I'm taking on technical debt to access data I don't have programmatic access to. CSS/Xpath are very fragile. You most likely will be changing them in the future.

I've used scrapy a few times and it's never felt like a big over-arching framework. I've been able to change what I need, and what it does for me are all the things I'd have to have done myself (caching, parallel requests but throttling per domain, dropping into debug mode, scheduled runs, etc). Really, it feels more like a skeleton + lots of sensible defaults. The meat of the code will be in the parsing, and so if f…

Interesting. So the web crawling/page fetching component is a major value add to you as a developer?

Whereas the parsing is less of a value add because you prefer to code the parser yourself so that you have more control?

What about changing the parser and crawler as the websites changes?

What other pain points about scrapy do you have?

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#135

Post by piker »

jazoom wrote: post #125
piker wrote: post #123

Proposition: 99% of scraping use cases are eliminated if the scraper agrees to subsequently abide by the target's terms of service.

Googlebot doesn't abide by 99% of websites' terms of use.

I agree with the implication here, but (1) it doesn't actually rebut the proposition and (2) some significant fraction of websites crawled by Googlebot have signed up on Google's webmaster tools, which includes its own ToS that likely governs.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#136

Post by camus2 »

jazoom wrote: post #130
simplyluke wrote: post #128

Earlier quoted context omitted.

But that's "different" because they've built a $600bn company off it.

More that the websites actually want to be found by someone.

The over-reliance on google search is both a blessing and a curse for the web. Today, google IS basically the web, a centralized version of it.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#137

Post by brilliantcode »

jakubbalada wrote: post #79
foxylion wrote: post #51

I'm curious what others use to scrape modern (javascript based) web applications. The old web (html and links) work fine with tools like Scrapy, but for modern applications which rely on javascript this does no longer work. For my last project I used a chrome plugin which controlled the browsers url locations and clicks. Results where transmitted to a backend server. New jobs (clicks, change urls) where retrieved fro…

Disclaimer: I'm a co-founder of Apifier [1]. It's not an open source, but free up to 10k pages per month. And it can handle modern JS web applications (your code runs in a context of crawled page). You can for example scrape API key at first and then use internal AJAX calls. There's also a community page [2] where you can find and use crawlers made by other users. [1] https://www.apifier.com [2] https://www.apifier.c…

interesting. are you seeing any product/market fit for this?

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#138

Post by cookiecaper »

brilliantcode wrote: post #133
cookiecaper wrote: post #100

Earlier quoted context omitted.

>Well the difference is pretty clear between your case and the rest. Developers scraping a website isn't going anywhere. A business reliant on scraped data is making money off of it. That will lend you in precarious situation more. Developers presumably scrape websites because the data is of some value to them, frequently commercial value. Google's entire value proposition is based on scraped data, and it's one of th…

3Taps became liable because they put their neck out for PadMapper even after they received written letters. It was a poorly executed business strategy because they were up against powerful legal team. Best thing to do if you receive C&D or requests to stop scraping, best to not continue and just let that customer go.

You can be sued (and lose) for damages incurred by illegal activity whether the aggrieved party sends a notice or not. It's not the plaintiff's job to let you know you're breaking the law, and they're entitled to damages whether you know you're breaking the law or not.

In fact, it's assumed that defendants weren't intentionally breaking the law, which is why when it's clear that they were, courts triple the actual damages for willful violations. [0]

If a reasonable person wouldn't realize that they were "exceeding authorized access", that probably limits a potential CFAA claim, but that's it, and that's not only the potentially perilous statute when you're a scraper. In the QVC case, Resultly got lucky that QVC did not have an up-to-date robots.txt; otherwise, they very well may have been on the hook for multiple days of lost online revenue, despite their immediate cessation upon receipt of a C&D.

Again, you are more than welcome to take your perspective and run with it, and it's plausible that no one will get mad enough at you to sue over it. That doesn't change the law.

I would assume that 3Taps pursued this litigation not because they had special love for PadMapper, but because they felt it was important for their business to be allowed to scrape major data sources and thought they'd be able to win. Pretty sure Skadden was their law firm so they gave it an earnest try, but ultimately lost.

[0] https://en.wikipedia.org/wiki/Treble_damages

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#139

Post by brilliantcode »

hsod wrote: post #88
codedokode wrote: post #38

Earlier quoted context omitted.

I think such laws are wrong. Scraping is what Google and Web Archive does and it serves good purposes. For example, one can make an application that compares prices for the same item at different internet shops and helps to find the cheapest offer. I don't understand what's wrong with downloading the information that is published on a public web server. That is what that server was made for in the first place. Of cou…

I disagree. The person who owns the server should get to decide who has access and under what circumstances. Joe Scraper, having invested nothing, has no claim or rights to it whatsoever. Furthermore, the fact that the server gives 200 responses is not sufficient implied permission IF a no-scraping policy has been communicated in some other way such as robots.txt or (clearly communicated) TOS. The techno-nihilist arg…

This feels a lot like the "downloading mp3 is stealing".

If you don't want people stealing your music, don't put it other people's hands. The minute you release it to the world, it can't be reversed. See streisand effect.

Likewise, you cannot place burden on your visitors to read and analyze ToS with their lawyers and submit an official request via fax.

If you don't want people access your server, put it behind a paywall or a login screen at least so that you can easily ban people who don't play by your rules.

Otherwise, you have no excuse. If it's publicly accessible, then you cannot enforce any legally binding agreement as you've left the front door wide open and expect people to read tiny letters you put in front of your mailbox.

ToS is a one way street.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#140

Post by brilliantcode »

elmigranto wrote: post #122
hsod wrote: post #88

Earlier quoted context omitted.

I disagree. The person who owns the server should get to decide who has access and under what circumstances. Joe Scraper, having invested nothing, has no claim or rights to it whatsoever. Furthermore, the fact that the server gives 200 responses is not sufficient implied permission IF a no-scraping policy has been communicated in some other way such as robots.txt or (clearly communicated) TOS. The techno-nihilist arg…

Is there a law requiring me to read and accept TOS or robots.txt before opening a website? What is the legal status of those? What if TOS says "you owe us $100 for every byte included in this TOS"? What if I have my own TOS about opening their links where they owe me $100 per byte sent?

No. TOS is not mutually binding agreement. You can't invite people to your garage sales and tell people that by looking at the items they've made a final sale automatically without their consent.
Post reply on HN