Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping: Bypassing “403 Forbidden,” captchas, and more

sangaline.com

Post reply on HN
Print view View original

161–170 of 232 posts

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#161

Post by brilliantcode »

cookiecaper wrote: post #138
brilliantcode wrote: post #133

Earlier quoted context omitted.

3Taps became liable because they put their neck out for PadMapper even after they received written letters. It was a poorly executed business strategy because they were up against powerful legal team. Best thing to do if you receive C&D or requests to stop scraping, best to not continue and just let that customer go.

You can be sued (and lose) for damages incurred by illegal activity whether the aggrieved party sends a notice or not. It's not the plaintiff's job to let you know you're breaking the law, and they're entitled to damages whether you know you're breaking the law or not. In fact, it's assumed that defendants weren't intentionally breaking the law, which is why when it's clear that they were , courts triple the actual d…

You can be sued for crossing the street. You can be sued for flipping the bird and someone happens to get aneurism from it. You can be sued for writing what you just wrote!

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#162

Post by brilliantcode »

MichaelGG wrote: post #156
brilliantcode wrote: post #146

Earlier quoted context omitted.

Story doesn't add up. First, it's very hard to pull off a DDOS attack using Tor. The most you could get would be less than someone repeatedly pressing refresh every second. This is because if you hit the same domain repeatedly the network will flag and throttle you. How bad was your server configuration that it would choke if somebody tried to scrape it? Was this running on a dreamhost $10/year server or something? T…

> This is weird. First, it's very hard to pull off a DDOS attack using Tor. The most you could get would be less than someone repeatedly pressing refresh every second. Please explain. Why do you think Tor can't provide a user with many RPS?

The network as I understand will automatically throttle and flag you if you are firing too many RPS. If you are hitting a particular domain over and over especially. So it's not possible to take down websites with TOR unless it's running on Dreamhost's shared hosting plan with a PHP solution.

This is why I find OP's story hard to believe, it doesn't add up.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#163

Post by brilliantcode »

jakubbalada wrote: post #157
brilliantcode wrote: post #137

Earlier quoted context omitted.

interesting. are you seeing any product/market fit for this?

We see a lot of users who needs data from the web or APIs for sites which doesn't have one. Just not all of them can code and we have to scale custom development.

Are these developers? Business people? I'm curious because we've been searching for a tool like this for a while but ultimately management thought it was a bad idea to rely on scraping, there's simply no replacement for a REST api.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#164

Post by skinnymuch »

brilliantcode wrote: post #84
eli wrote: post #78

Earlier quoted context omitted.

> CSS/Xpath are very fragile. You most likely will be changing them in the future. Genuinely curious what the alternative is

I've been doing research on this but it's not clear whether this problem is a pain for enough number of businesses to justify further investments. I often feel like web scraping is a commodity without understanding any of the inherent technological complexities and challenges. Very discouraging field to be in, especially when people claim to have pain but are unwilling to pay very much for it or show appreciation for…

How many downvotes did you get? A few may have just been because your response is vague and doesn't say much for the question. But that doesn't mean a downvote should occur, of course.

Otherwise complaining about downvotes is no good either. Some will downvote because of that.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#165

Post by skinnymuch »

jlgaddis wrote: post #153

Here's an idea (although probably an unpopular one around here): if a site is responding to your scraping attempts with 403s -- a.k.a. "Forbidden" -- stop what you're doing and go away.

This is a very obvious thing to say. Perhaps it's needed to be said, I don't know -- It's just a very obvious counter.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#166

Post by skinnymuch »

jordif wrote: post #151

Good article! I been doing scraping for the last 10 years and I've seen a lots of differents things to try to avoid us. Also, I'm in the other side protecting websites to ban scrapers, so funny!

I'm in the same position for the first time (protecting against scraping) and honestly I'm kind of blind right now. Which is weird because of how much scraping I've done (okay not that much). Any tips or tricks or blogs you know of off the top of your head for protecting your site?

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#167

Post by skinnymuch »

elchief wrote: post #86
foxylion wrote: post #51

I'm curious what others use to scrape modern (javascript based) web applications. The old web (html and links) work fine with tools like Scrapy, but for modern applications which rely on javascript this does no longer work. For my last project I used a chrome plugin which controlled the browsers url locations and clicks. Results where transmitted to a backend server. New jobs (clicks, change urls) where retrieved fro…

We use HTMLUnit. Works pretty well. Not super fast, but you want to scrape individual sites at a moderate rate anyway

Have you run into issues? I'd think HTMLUnit isn't robust enough and it's "browser" Ian limiting?

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#168

Post by hackits »

robryan wrote: post #155
thefifthsetpin wrote: post #62

Better solution: pay target-site.com to start building an API for you. Pros: * You'll be working with them rather than against them. * Your solution will be far more robust. * It'll be way cheaper, supposing you account for the ongoing maintenance costs of your fragile scraper. * You're eliminating the possibility that you'll have to deal with legal antagonism * Good anti-scraper defenses are far more sophisticated t…

Many instances of us building scrapers are cases where a partner has data or has tools which are only built into the UI or the UI ones are much more capable. Rather than waiting potentially years for their IT team to make the required changes we can build a scraper in a matter of days.

I can attest to this. From personal experience I found websites that would ignore scrappers and just allow me to access their data on their public web-site easier to deal with code wise and time wise. I make the request, you give me the data I need and then I can piss off.

Web-sites that make it a cluster &&*& to get access to the data do two things. They setup a challenge to break their idiotic `are you a bot?` and secondly it is trivial in most situation just to spin up a vm, and run chrome with selenium and a python script.

Granted I don't use AJAX API or anything like that. Instead I've found developer who nativly have a JSON string along side the data within the HTML to easiest to parse.

Reasons why I've setup bots/scrappers 1) My local city rental market is very competitive and I hate wasting time email landlords who have all-ready signed up a lease. 2) House prices 3) Car prices 4) Stock prices 5) Banking 6) Water Rates 7) Health insurance comparison 8) Automated billing and payment systems.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#169

Post by userbinator »

dsacco wrote: post #68
jedberg wrote: post #63

Earlier quoted context omitted.

How do you deal with the issue that most mobile apps have a baked in security key for their private API? Or am I being naive to think that most apps have that?

You reverse engineer the application, or you run it in a debugger. If the app features certificate pinning to block MITM eavesdropping through your own proxy, you either use one of the XPosed Framework libraries that removes it on the fly in a process hook, or you decompile the app, return-void the GetTrustedClient, GetTrustedServer, AcceptedIssuers, etc. functions. If it features HMAC signing, you decompile the app,…

In many cases, if all you're after is the data, you don't even need to reverse-engineer much; I'm not familiar with the mobile world but in Windows you can just send input events to the process, essentially simulating a user, and read its output in a similar fashion. You can still treat the app like a black box and regardless of how much obfuscation they put in the code, you still get the output.

(This technique is useful not just for scraping data, but for UI testing of your own apps.)

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#170

Post by lhnz »

phs318u wrote: post #159
chatmasta wrote: post #52

Note that 99% of time, if a web page is worth scraping, it probably has an accompanying mobile app. It's worth downloading the app and running mitmproxy/burp/charles on the traffic to see if it uses a private API. In my experience, it's much easier to scrape the private mobile API than a public website. This way you get nicely formatted JSON and often bypass rate limits.

Perfectly feasible technically. However aren't you stepping into a legal minefield? After all, it's a "private" API. Furthermore, since you're explicitly going around the hurdles they've laid out for the public (captchas, T&C), you're surely painting a target on your head? Note, I'm not debating whether this is a good or bad thing, just that in the current environment this is surely a legally dodgy manoeuvre.

Has anybody ever been sued for scraping a public website?
Post reply on HN