Good article! I been doing scraping for the last 10 years and I've seen a lots of differents things to try to avoid us.
Also, I'm in the other side protecting websites to ban scrapers, so funny!
Web Scraping: Bypassing “403 Forbidden,” captchas, and more
151–160 of 232 posts
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#152Note that 99% of time, if a web page is worth scraping, it probably has an accompanying mobile app. It's worth downloading the app and running mitmproxy/burp/charles on the traffic to see if it uses a private API. In my experience, it's much easier to scrape the private mobile API than a public website. This way you get nicely formatted JSON and often bypass rate limits.
Also get the Android APK and decompile it.
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#153Here's an idea (although probably an unpopular one around here): if a site is responding to your scraping attempts with 403s -- a.k.a. "Forbidden" -- stop what you're doing and go away.
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#154Earlier quoted context omitted.
You reverse engineer the application, or you run it in a debugger. If the app features certificate pinning to block MITM eavesdropping through your own proxy, you either use one of the XPosed Framework libraries that removes it on the fly in a process hook, or you decompile the app, return-void the GetTrustedClient, GetTrustedServer, AcceptedIssuers, etc. functions. If it features HMAC signing, you decompile the app,…
The initial idea was to make your life simpler by parsing JSON instead of HTML. Now we are decompiling binaries. Somewhere on the way, we got lost.
The story of every IT project ever...
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#155Better solution: pay target-site.com to start building an API for you. Pros: * You'll be working with them rather than against them. * Your solution will be far more robust. * It'll be way cheaper, supposing you account for the ongoing maintenance costs of your fragile scraper. * You're eliminating the possibility that you'll have to deal with legal antagonism * Good anti-scraper defenses are far more sophisticated t…
Many instances of us building scrapers are cases where a partner has data or has tools which are only built into the UI or the UI ones are much more capable.
Rather than waiting potentially years for their IT team to make the required changes we can build a scraper in a matter of days.
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#156Earlier quoted context omitted.
Having been the victim of a VERY badly behaved scraper, I'm willing to listen to this. When that "attack" was going on, we talked about that very thing, if the scraper would only identify himself. (we were able to identify the actual culprit, and circumstantial evidence suggested they were going after our complete price list for a client) The cost of the bad scraper was pretty significant. They were hitting us as har…
Story doesn't add up. First, it's very hard to pull off a DDOS attack using Tor. The most you could get would be less than someone repeatedly pressing refresh every second. This is because if you hit the same domain repeatedly the network will flag and throttle you. How bad was your server configuration that it would choke if somebody tried to scrape it? Was this running on a dreamhost $10/year server or something? T…
> This is weird. First, it's very hard to pull off a DDOS attack using Tor. The most you could get would be less than someone repeatedly pressing refresh every second.
Please explain. Why do you think Tor can't provide a user with many RPS?
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#157Earlier quoted context omitted.
Disclaimer: I'm a co-founder of Apifier [1]. It's not an open source, but free up to 10k pages per month. And it can handle modern JS web applications (your code runs in a context of crawled page). You can for example scrape API key at first and then use internal AJAX calls. There's also a community page [2] where you can find and use crawlers made by other users. [1] https://www.apifier.com [2] https://www.apifier.c…
interesting. are you seeing any product/market fit for this?
We see a lot of users who needs data from the web or APIs for sites which doesn't have one. Just not all of them can code and we have to scale custom development.
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#158Earlier quoted context omitted.
> CSS/Xpath are very fragile. You most likely will be changing them in the future. Genuinely curious what the alternative is
I've been doing research on this but it's not clear whether this problem is a pain for enough number of businesses to justify further investments. I often feel like web scraping is a commodity without understanding any of the inherent technological complexities and challenges. Very discouraging field to be in, especially when people claim to have pain but are unwilling to pay very much for it or show appreciation for…
FYI, I only downvoted you after you complained about downvotes.
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#159Note that 99% of time, if a web page is worth scraping, it probably has an accompanying mobile app. It's worth downloading the app and running mitmproxy/burp/charles on the traffic to see if it uses a private API. In my experience, it's much easier to scrape the private mobile API than a public website. This way you get nicely formatted JSON and often bypass rate limits.
Perfectly feasible technically. However aren't you stepping into a legal minefield? After all, it's a "private" API. Furthermore, since you're explicitly going around the hurdles they've laid out for the public (captchas, T&C), you're surely painting a target on your head?
Note, I'm not debating whether this is a good or bad thing, just that in the current environment this is surely a legally dodgy manoeuvre.
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#160What if the target page is blocking by IP address and if even with 20 different IP addresses you wouldn't be able to fetch all the data you need in a month?