Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping: Bypassing “403 Forbidden,” captchas, and more

sangaline.com

Post reply on HN
Print view View original

61–70 of 232 posts

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#61

Post by jmkni »

foxylion wrote: post #51

I'm curious what others use to scrape modern (javascript based) web applications. The old web (html and links) work fine with tools like Scrapy, but for modern applications which rely on javascript this does no longer work. For my last project I used a chrome plugin which controlled the browsers url locations and clicks. Results where transmitted to a backend server. New jobs (clicks, change urls) where retrieved fro…

I have used Selenium for this with quite a bit of success, or as others have mentioned, just figure out where the API endpoints are with fiddler and pull the data directly from the source.

Sometimes this can be a PITA though, for example Tableau obfuscates the JSON they send back, so it's easier to use Selenium to wait ten seconds and then scrape the resulting HTML.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#62

Post by thefifthsetpin »

Better solution: pay target-site.com to start building an API for you.

Pros:

* You'll be working with them rather than against them.

* Your solution will be far more robust.

* It'll be way cheaper, supposing you account for the ongoing maintenance costs of your fragile scraper.

* You're eliminating the possibility that you'll have to deal with legal antagonism

* Good anti-scraper defenses are far more sophisticated than what the author dealt with. As a trivial example: he didn't even verify that he was following links that would be visible!

Cons:

* Possible that target-site.com's owners will tell you to get lost, or they are simply unreachable.

* Your competitors will gain access to the API methods you funded, perhaps giving them insight into why that data is valuable.

Alternative better solution for small one-off data collection needs: contract a low-income person to just manually download the data you need with a normal web browser. Provide a JS bookmarklet to speed their process if the data set is a bit too big for that.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#63

Post by jedberg »

chatmasta wrote: post #52

Note that 99% of time, if a web page is worth scraping, it probably has an accompanying mobile app. It's worth downloading the app and running mitmproxy/burp/charles on the traffic to see if it uses a private API. In my experience, it's much easier to scrape the private mobile API than a public website. This way you get nicely formatted JSON and often bypass rate limits.

How do you deal with the issue that most mobile apps have a baked in security key for their private API? Or am I being naive to think that most apps have that?

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#64

Post by tokenizerrr »

janci wrote: post #45
janci wrote: post #41

I use Java with simple task queue and multiple worker threads (scrapy is only singlethreaded, although uses async I/O). Failed tasks are collected into second queue and restarted when needed. Used Jsoup[1] for parsing, proxychains and HAproxy + tor [2] for distributing across multiple IPs. [1] https://jsoup.org/ [2] https://github.com/mattes/rotating-proxy

Hardest part was synchronization - to end the main thread only if all tasks are done - when every running task can produce multiple new tasks - with limiting the maximum number of running threads - always running the maximum nubmer of threads if possible semaphores to the rescue

Doesn't ThreadPoolExecutor take care of all of that if you store the returned Future from the submit method? Then you just have the main thread wait for those.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#65

Post by brilliantcode »

cookiecaper wrote: post #42
codedokode wrote: post #39

Earlier quoted context omitted.

Doesn't publishing information on a public web server equals to granting authorization to download it? Why publish it otherwise? And copyright laws are supposed to protect only creative works, not every page on the web.

>Doesn't publishing information on a public web server equals to granting authorization to download it? Why publish it otherwise? This is the argument that there is an implied license. The counterargument is that the user agreed to the Terms of Use which explicitly defined automated access as violative. In addition, these cases generally begin with a cease and desist demand letter, which explicitly informs the allege…

ToS is not a legally binding mutual agreement between the website and the scraper. Only when it's materialized in the form of C&D and it caused damage to the website. In the QVC vs Resultly, they caused a website crashed. In the Craigslist vs 3Taps, the damages were non-existant but they still won because of the C&D letters.

So if a website sends you a letter to stop, do not continue scraping them. Until that happens, it's fair game. ToS is not legally binding. Those two cases were the result of damage caused to the website.

STOP. SPREADING. FUD. cookiecaper. All of your comments are the same unsound legal advice yet Mozenda, Import.io, and a whole bunch of tools & service providers are humming along just fine.

Disclaimer: I'm not a lawyer, this is not a legal advice, consult a real lawyer and not random HN comments.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#66

Post by RandomBookmarks »

foxylion wrote: post #51

I'm curious what others use to scrape modern (javascript based) web applications. The old web (html and links) work fine with tools like Scrapy, but for modern applications which rely on javascript this does no longer work. For my last project I used a chrome plugin which controlled the browsers url locations and clicks. Results where transmitted to a backend server. New jobs (clicks, change urls) where retrieved fro…

Not open-source but free: Kantu (https://kantu.io) uses OCR to support web scraping. You mark an anchor image/text with a green frame and mark the area of data that needs to be extracted with pink frames. The image inside the pink frames is then sent to https://ocr.space for processing and Kantu api returns the extracted text. This works very well as long as you do not need a lot of data. It is certainly not a "high-speed" solution for scraping terabytes of data.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#67

Post by brilliantcode »

nip wrote: post #35

Scrapy is indeed excellent. One feature that I really like is Scrapy Shell [1]. It allows to run and debug the scraping code without running the spider, right from the CLI. I use it extensively to test that my selectors (both CSS and XPATH) are returning the proper data on a test URL. [1] https://doc.scrapy.org/en/latest/topics/shell.html

A few things turn me off about Scrapy is that it feels over engineered for what it does. Why do I need an entire framework?

I'm taking on technical debt to access data I don't have programmatic access to.

CSS/Xpath are very fragile. You most likely will be changing them in the future.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#68

Post by dsacco »

jedberg wrote: post #63
chatmasta wrote: post #52

Note that 99% of time, if a web page is worth scraping, it probably has an accompanying mobile app. It's worth downloading the app and running mitmproxy/burp/charles on the traffic to see if it uses a private API. In my experience, it's much easier to scrape the private mobile API than a public website. This way you get nicely formatted JSON and often bypass rate limits.

How do you deal with the issue that most mobile apps have a baked in security key for their private API? Or am I being naive to think that most apps have that?

You reverse engineer the application, or you run it in a debugger.

If the app features certificate pinning to block MITM eavesdropping through your own proxy, you either use one of the XPosed Framework libraries that removes it on the fly in a process hook, or you decompile the app, return-void the GetTrustedClient, GetTrustedServer, AcceptedIssuers, etc. functions.

If it features HMAC signing, you decompile the app, find the key, reverse engineer the algorithm that chooses and sorts the paramaters for the HMAC function, and rewrite it outside the app. If the key is generated dynamically you reverse engineer that too, and if it's retrieved from native .so files you're going to have a fun time, but it's still quite doable.

All they can do is pile on layers and layers of abstraction to make it painful. They can't make the private API truly private if it requires something shipped with the client.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#69

Post by dsacco »

chatmasta wrote: post #52

Note that 99% of time, if a web page is worth scraping, it probably has an accompanying mobile app. It's worth downloading the app and running mitmproxy/burp/charles on the traffic to see if it uses a private API. In my experience, it's much easier to scrape the private mobile API than a public website. This way you get nicely formatted JSON and often bypass rate limits.

Most of the "worthwhile" apps have certificate pinning and HMAC-signing in place these days, so it's a bit more of an onerous process than simply booting up Charles and playing with the API. But yeah, it's definitely better to deal with JSON outputs than have to parse HTML.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#70

Post by brilliantcode »

RandomBookmarks wrote: post #66
foxylion wrote: post #51

I'm curious what others use to scrape modern (javascript based) web applications. The old web (html and links) work fine with tools like Scrapy, but for modern applications which rely on javascript this does no longer work. For my last project I used a chrome plugin which controlled the browsers url locations and clicks. Results where transmitted to a backend server. New jobs (clicks, change urls) where retrieved fro…

Not open-source but free: Kantu ( https://kantu.io ) uses OCR to support web scraping. You mark an anchor image/text with a green frame and mark the area of data that needs to be extracted with pink frames. The image inside the pink frames is then sent to https://ocr.space for processing and Kantu api returns the extracted text. This works very well as long as you do not need a lot of data. It is certainly not a "hig…

Tried the OCR for scraping and gave up because it was too slow and inaccurate.

OCR works well for certain scenarios where UI is fixed like on desktop applications but it's still fragile very much like CSS and Xpath selectors.

In fact, often OCR performs far slower and less accurate than CSS/Xpath selectors.

It has it's niches but I think it's sub optimal for web automation/scraping.

Post reply on HN