What if they use that before:after thing where the content takes say a couple of seconds to appear so when you try to scrape the site it appears that nothing is there. I have only used HTMLSimpleDom scraper with PHP at this point.
Web Scraping: Bypassing “403 Forbidden,” captchas, and more
181–190 of 232 posts
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#182Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#183Earlier quoted context omitted.
I've been doing research on this but it's not clear whether this problem is a pain for enough number of businesses to justify further investments. I often feel like web scraping is a commodity without understanding any of the inherent technological complexities and challenges. Very discouraging field to be in, especially when people claim to have pain but are unwilling to pay very much for it or show appreciation for…
FYI, I only downvoted you after you complained about downvotes.
I only downvoted because there was no alternative offered, just complaining about how underappreciated scraper creators are.
> perfect illustration of how innovation is punished
I see no innovation in your post.
The complaining about downvotes was just the icing on the cake, cementing the downvote.
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#184I use greasemonkey on firefox. Recently, I have written a crawler for a major accomondation listing website in Copenhagen. Guess what? I got a place to live right in the center in 2 weeks. I love SCRAPERS I love CRAWLERS.
Similarly I wrote a scraper for a local used item marketplace and whenever I need to purchase something that isn't urgent and I'm OK with it being second hand, I plug in the relevant stats and load it in to cron. Near instant notification with contact information and details in my email when a match is found.
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#185Try lynx
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#186Earlier quoted context omitted.
We see a lot of users who needs data from the web or APIs for sites which doesn't have one. Just not all of them can code and we have to scale custom development.
Are these developers? Business people? I'm curious because we've been searching for a tool like this for a while but ultimately management thought it was a bad idea to rely on scraping, there's simply no replacement for a REST api.
Both - developers on a free plan using own RSS for sites without one and business people (mainly startups) building their products on top of Apifier.
Typical use is an aggregator that needs common API for all partners who are not able to provide it. So they have running API on Apifier in an hour. It might break once in a while - than you have to update your crawler (not that often if you use internal AJAX calls).
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#187Earlier quoted context omitted.
I've used scrapy a few times and it's never felt like a big over-arching framework. I've been able to change what I need, and what it does for me are all the things I'd have to have done myself (caching, parallel requests but throttling per domain, dropping into debug mode, scheduled runs, etc). Really, it feels more like a skeleton + lots of sensible defaults. The meat of the code will be in the parsing, and so if f…
Interesting. So the web crawling/page fetching component is a major value add to you as a developer? Whereas the parsing is less of a value add because you prefer to code the parser yourself so that you have more control? What about changing the parser and crawler as the websites changes? What other pain points about scrapy do you have?
> Interesting. So the web crawling/page fetching component is a major value add to you as a developer?
When I have a scraping task, yes. It's a set of things that are required each time but also a bit fiddly to get right, and scrapy has solved them well.
> Whereas the parsing is less of a value add because you prefer to code the parser yourself so that you have more control?
These I see as fundamentally having to be things I code as they're the parts that are different each time.
> What about changing the parser and crawler as the websites changes?
Really just a cost of doing scraping. Anything fancier has generally taken more time and created more problems than just assuming an ongoing maintenance cost. The debugging shell in scrapy is very useful for this.
Last big job I did I also built a cache that you could query by time, so all versions of the page seen were stored which was very useful for debugging intermittent problems, and finding page changes. I don't know if scrapy has this in its cache, I don't think so but wouldn't conflict with it.
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#188What if the target page is blocking by IP address and if even with 20 different IP addresses you wouldn't be able to fetch all the data you need in a month?
Professional proxy services. Price, IP pool size and quality vary hugely but if you're not trying to scrape an aggressively defended target and don't need to make more than a handful of requests per second, 100K IPs will usually be more than enough to circumvent most rate limits and a pool that size can be rented for under $100/month.
Interested to know where to get 100k proxies for $100/mo. Can you give some options?
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#189Earlier quoted context omitted.
How do you deal with the issue that most mobile apps have a baked in security key for their private API? Or am I being naive to think that most apps have that?
You reverse engineer the application, or you run it in a debugger. If the app features certificate pinning to block MITM eavesdropping through your own proxy, you either use one of the XPosed Framework libraries that removes it on the fly in a process hook, or you decompile the app, return-void the GetTrustedClient, GetTrustedServer, AcceptedIssuers, etc. functions. If it features HMAC signing, you decompile the app,…
I feel like at a certain point this crosses the line from unintended use of a private API to unethical hacking.
If the data owner went through the trouble of encrypting the traffic between it and it's app they have a certain expectation of private communications that you'd better have a damn good reason for violating.
Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more
#190Note that 99% of time, if a web page is worth scraping, it probably has an accompanying mobile app. It's worth downloading the app and running mitmproxy/burp/charles on the traffic to see if it uses a private API. In my experience, it's much easier to scrape the private mobile API than a public website. This way you get nicely formatted JSON and often bypass rate limits.
you realize if you piss off anyone doing this, they can have you sent to jail right? Automated scraping is unlikely to qualify as fair use in court.
As far as I know, only Japan has laws.that allow scraping without explicit permission.