Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping: Bypassing “403 Forbidden,” captchas, and more

sangaline.com

Post reply on HN
Print view View original

201–210 of 232 posts

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#201

Post by j_s »

jedberg wrote: post #63
chatmasta wrote: post #52

Note that 99% of time, if a web page is worth scraping, it probably has an accompanying mobile app. It's worth downloading the app and running mitmproxy/burp/charles on the traffic to see if it uses a private API. In my experience, it's much easier to scrape the private mobile API than a public website. This way you get nicely formatted JSON and often bypass rate limits.

How do you deal with the issue that most mobile apps have a baked in security key for their private API? Or am I being naive to think that most apps have that?

If you haven't heard of Frida yet, check it out!

https://www.frida.re/

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#202

Post by cookiecaper »

brilliantcode wrote: post #161
cookiecaper wrote: post #138

Earlier quoted context omitted.

You can be sued (and lose) for damages incurred by illegal activity whether the aggrieved party sends a notice or not. It's not the plaintiff's job to let you know you're breaking the law, and they're entitled to damages whether you know you're breaking the law or not. In fact, it's assumed that defendants weren't intentionally breaking the law, which is why when it's clear that they were , courts triple the actual d…

You can be sued for crossing the street. You can be sued for flipping the bird and someone happens to get aneurism from it. You can be sued for writing what you just wrote!

Yes, but if you fight it adequately, you won't lose. If you get sued for scraping, it's quite likely you'll lose, as the law has numerous pitfalls for scrapers, including things as basic as regarding RAM copies as infringing.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#203

Post by Crespyl »

sjwright wrote: post #200
reacweb wrote: post #196

Earlier quoted context omitted.

When an application you have legally installed on your own computer is communicating with the outside world, it seems a fundamental right to inspect the exchanged data to check that it is not leaking personal information. If the data is encrypted or obfuscated, this could make us suspicious (why hiding if there is nothing to hide ?) and gives additional motivation to audit the security. Once the api is reverse engine…

And I suppose you personally test the physical security measures of every retail store you shop at?

No, but I do personally test the physical security measures of every car or computer I purchase and bring into my home.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#204

Post by pdabbadabba »

codedokode wrote: post #175
hsod wrote: post #88

Earlier quoted context omitted.

I disagree. The person who owns the server should get to decide who has access and under what circumstances. Joe Scraper, having invested nothing, has no claim or rights to it whatsoever. Furthermore, the fact that the server gives 200 responses is not sufficient implied permission IF a no-scraping policy has been communicated in some other way such as robots.txt or (clearly communicated) TOS. The techno-nihilist arg…

The person who owns a server can restrict access by adding a login form and creating user accounts. Or taking other measures (for example, banning IP networks or countries or implementing a captcha). I think that the only thing that has to be regulated is the load (number of request per unit of time) on the server. So that it doesn't prevent server from serving pages to other visitors. Regarding robots.txt, I am not…

> The person who owns a server can restrict access by adding a login form and creating user accounts.

Or, say, returning "403 Forbidden" or using a CAPTCHA?

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#205

Post by libertymcateer »

sfifs wrote: post #190
chatmasta wrote: post #52

Note that 99% of time, if a web page is worth scraping, it probably has an accompanying mobile app. It's worth downloading the app and running mitmproxy/burp/charles on the traffic to see if it uses a private API. In my experience, it's much easier to scrape the private mobile API than a public website. This way you get nicely formatted JSON and often bypass rate limits.

you realize if you piss off anyone doing this, they can have you sent to jail right? Automated scraping is unlikely to qualify as fair use in court. As far as I know, only Japan has laws.that allow scraping without explicit permission.

> you realize if you piss off anyone doing this, they can have you sent to jail right?

Are you talking about Aaron Swartz? There were complicating factors. Typically, no, scraping is not a criminal matter. If you need a lawyer, however, you should talk to one.

> Automated scraping is unlikely to qualify as fair use in court.

Scraping is not [usually] the problem. It is what you do with the scraped content that is the problem. Additionally, the nature of the scraped content itself is at issue: facts are not copyrightable, for instance, just particular expressions.

Fair use is a hugely, hugely complex topic. If you ever ask someone if something is fair use and they give you a straight "yes" or "no" answer - don't trust it unless they are also showing you a court decision that covers your exact scenario and is valid in your jurisdiction. Any good answer will include a healthy does of "it depends."

The only way to conclusively determine if something is fair use is going through the fair use "run-time" - which is a court decision on the topic. There is no other way - none - to determine if something is fair use and claims to the contrary are false. To be clear: fair use is a judicial determination and it is highly fact specific.

Disclaimer: I am a lawyer, but I am not your lawyer. If you need one, you should get one.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#206

Post by pdabbadabba »

brilliantcode wrote: post #140
elmigranto wrote: post #122

Earlier quoted context omitted.

Is there a law requiring me to read and accept TOS or robots.txt before opening a website? What is the legal status of those? What if TOS says "you owe us $100 for every byte included in this TOS"? What if I have my own TOS about opening their links where they owe me $100 per byte sent?

No. TOS is not mutually binding agreement. You can't invite people to your garage sales and tell people that by looking at the items they've made a final sale automatically without their consent.

Not true (in the U.S.). Just ask Aaron Swartz.

The fact that it does (or, at least, has been interpreted to) criminalize TOS violations is one of the must important flaws in the Computer Fraud and Abuse Act.

https://www.eff.org/deeplinks/2013/01/rebooting-computer-cri...

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#207

Post by [deleted] »

chatmasta wrote: post #52

Note that 99% of time, if a web page is worth scraping, it probably has an accompanying mobile app. It's worth downloading the app and running mitmproxy/burp/charles on the traffic to see if it uses a private API. In my experience, it's much easier to scrape the private mobile API than a public website. This way you get nicely formatted JSON and often bypass rate limits.

[deleted]

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#208

Post by driverdan »

tehlike wrote: post #4
herbst wrote: post #2

I've used antigate for captchas and ether Tor or proxies for 403s before. Usually the browser header alone does not help for long.

Anticaptcha and deatbycaptcha are some others. But it mames me feel sad to use them, as it exploits cheap labor overseas.

It's not exploitation.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#209

Post by officelineback »

dsacco wrote: post #68
jedberg wrote: post #63

Earlier quoted context omitted.

How do you deal with the issue that most mobile apps have a baked in security key for their private API? Or am I being naive to think that most apps have that?

You reverse engineer the application, or you run it in a debugger. If the app features certificate pinning to block MITM eavesdropping through your own proxy, you either use one of the XPosed Framework libraries that removes it on the fly in a process hook, or you decompile the app, return-void the GetTrustedClient, GetTrustedServer, AcceptedIssuers, etc. functions. If it features HMAC signing, you decompile the app,…

Reminds me of the battle between hackers and Niantic to create scanners for Pokemon Go.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#210

Post by 15155 »

amelius wrote: post #191
sfifs wrote: post #190

Earlier quoted context omitted.

you realize if you piss off anyone doing this, they can have you sent to jail right? Automated scraping is unlikely to qualify as fair use in court. As far as I know, only Japan has laws.that allow scraping without explicit permission.

If there's no robots.txt, then you can scrape all you want. That's how Google got big.

Is robots.txt compliance written into law anywhere?
Post reply on HN