Earlier quoted context omitted.
Has anybody ever been sued for scraping a public website?
Weev spent about a year in jail for doing it
And the conviction was vacated on a technicality, so standing precedent on the issue is a little thin.
Web Scraping: Bypassing “403 Forbidden,” captchas, and more
211–220 of 232 posts
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#212Here's an idea (although probably an unpopular one around here): if a site is responding to your scraping attempts with 403s -- a.k.a. "Forbidden" -- stop what you're doing and go away.
This is a very obvious thing to say. Perhaps it's needed to be said, I don't know -- It's just a very obvious counter.
Yeah, it seemed obvious... but judging by all of the comments here on how to "bypass" 403s, it actually wasn't obvious at all.
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#213Earlier quoted context omitted.
No. TOS is not mutually binding agreement. You can't invite people to your garage sales and tell people that by looking at the items they've made a final sale automatically without their consent.
Not true (in the U.S.). Just ask Aaron Swartz. The fact that it does (or, at least, has been interpreted to) criminalize TOS violations is one of the must important flaws in the Computer Fraud and Abuse Act. https://www.eff.org/deeplinks/2013/01/rebooting-computer-cri...
Aaron Swartz was a weird one. Some zealous IT guy exaggerated it as if national security data were being scraped when in fact most were just shoddy research papers written at 1 AM before the due date.
All people that landed a hand in this man's death should be charged with manslaughter.
Truly a shame and how draconian laws end up killing the innocent in the United States.
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#214Earlier quoted context omitted.
Seems like your argument is based on the idea that it's unreasonable to expect the end user to understand the terms under which a service is offered. And that's fair in some cases, but it's also kind of a cop out. "How was I supposed to know they didn't want me to scrape it? I'm just an innocent passerby dropping bits on a wire" is bullshit in many, many cases. You do know, or at least could easily find out if you wa…
Scraping a website can be compared to only downloading mp3 file, not redistributing it to other people. And I don't see how electronic ToS have any legal power. It is not a contract one has signed before visiting a site. I think ToS should only bind the website owner, not its visitors.
It doesn't. It would be like stepping in to a store because people were welcoming you and realizing that by sitting down you've automatically agreed and consented to tiny ToS written behind every napkin.
It's unenforceable and has never stood up in court.
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#215Earlier quoted context omitted.
Are these developers? Business people? I'm curious because we've been searching for a tool like this for a while but ultimately management thought it was a bad idea to rely on scraping, there's simply no replacement for a REST api.
Both - developers on a free plan using own RSS for sites without one and business people (mainly startups) building their products on top of Apifier. Typical use is an aggregator that needs common API for all partners who are not able to provide it. So they have running API on Apifier in an hour. It might break once in a while - than you have to update your crawler (not that often if you use internal AJAX calls).
I see, so there's not much value beyond startups and bootstrappers.
I feel like it's a hard sell to enterprises. Scraping is viewed inferior to an API so it makes sense for enterprises to just pay the target website for access to the data.
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#216Note that 99% of time, if a web page is worth scraping, it probably has an accompanying mobile app. It's worth downloading the app and running mitmproxy/burp/charles on the traffic to see if it uses a private API. In my experience, it's much easier to scrape the private mobile API than a public website. This way you get nicely formatted JSON and often bypass rate limits.
Another fun aspect of that - most banks don't release public APIs giving safety as a reason. On the other hand, each one of them have mobile apps with APIs inside.
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#217Note that 99% of time, if a web page is worth scraping, it probably has an accompanying mobile app. It's worth downloading the app and running mitmproxy/burp/charles on the traffic to see if it uses a private API. In my experience, it's much easier to scrape the private mobile API than a public website. This way you get nicely formatted JSON and often bypass rate limits.
you realize if you piss off anyone doing this, they can have you sent to jail right? Automated scraping is unlikely to qualify as fair use in court. As far as I know, only Japan has laws.that allow scraping without explicit permission.
How does Google function in Japan then?
Western world allows for scraping as long as you follow robots.txt. If you don't, it's still not a jailable offence unless you DOS the system.
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#218Good article! I been doing scraping for the last 10 years and I've seen a lots of differents things to try to avoid us. Also, I'm in the other side protecting websites to ban scrapers, so funny!
I'm in the same position for the first time (protecting against scraping) and honestly I'm kind of blind right now. Which is weird because of how much scraping I've done (okay not that much). Any tips or tricks or blogs you know of off the top of your head for protecting your site?
After the years, I've arrived at the conclusion that everything can be scrapped. What you have to do is try to put as many walls as you can. But if someone really wants to crawl your site, with the right knowledgement he will able to do it despite of all your walls.
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#219Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#220Earlier quoted context omitted.
I disagree. The person who owns the server should get to decide who has access and under what circumstances. Joe Scraper, having invested nothing, has no claim or rights to it whatsoever. Furthermore, the fact that the server gives 200 responses is not sufficient implied permission IF a no-scraping policy has been communicated in some other way such as robots.txt or (clearly communicated) TOS. The techno-nihilist arg…
The person who owns a server can restrict access by adding a login form and creating user accounts. Or taking other measures (for example, banning IP networks or countries or implementing a captcha). I think that the only thing that has to be regulated is the load (number of request per unit of time) on the server. So that it doesn't prevent server from serving pages to other visitors. Regarding robots.txt, I am not…
[deleted]