Previously i had some success with this https://httptoolkit.tech/ and running the app on android emulator
Scraping data from an app: real world example
11–20 of 26 posts
Re: Scraping data from an app: real world example
#12No post body was provided.
Re: Scraping data from an app: real world example
#13Most app have some form of SSL pinning system in place which means that you have to perform additional work to allow the proxy to decrypt the HTTPS traffic.
"Most app have some form of SSL pinning system in place..." I would like to see the data, if any, supporting this statement. I would expect some apps would use pinning, but most would not. Google recommends against it. https://developer.android.com/training/articles/security-ssl...
I would say based on personal observation that the more scrape-worthy an app is, the more likely it has cert pinning. Rather obvious if you think about it, really. High value targets especially from big shops tend to have other measures like complex MACs that make scraping hell.
I’m sure most largely-worthless-to-scrape apps don’t employ cert pinning.
Re: Scraping data from an app: real world example
#14I use Burp for same purposes. Very convenient and solves the problem of MITM certificates.
Re: Scraping data from an app: real world example
#15Earlier quoted context omitted.
"Most app have some form of SSL pinning system in place..." I would like to see the data, if any, supporting this statement. I would expect some apps would use pinning, but most would not. Google recommends against it. https://developer.android.com/training/articles/security-ssl...
I would say based on personal observation that the more scrape-worthy an app is, the more likely it has cert pinning. Rather obvious if you think about it, really. High value targets especially from big shops tend to have other measures like complex MACs that make scraping hell. I’m sure most largely-worthless-to-scrape apps don’t employ cert pinning.
> other measures like complex MACs that make scraping hell.
Do you have examples of these techniques?
Re: Scraping data from an app: real world example
#16Earlier quoted context omitted.
>Caution: Certificate Pinning is not recommended for Android applications due to the high risk of future server configuration changes, such as changing to another Certificate Authority, rendering the application unable to connect to the server without receiving a client software update. This actually applies to websites and browsers as well.
Why isn't there a site-controlled fallback setting for this? Does this not make sense? Abu given website's beet interest is to continue to be reachable.
Every escape hatch in the certificate validation is also an additional avenue for attack. For example, using a DNS record to override certificate pins makes DNS cache poisoning much more valuable to the attacker.
Re: Scraping data from an app: real world example
#17Most app have some form of SSL pinning system in place which means that you have to perform additional work to allow the proxy to decrypt the HTTPS traffic.
Can someone suggest some resources to understand the additional work needed to decrypt the pinned https traffic?
Re: Scraping data from an app: real world example
#18That is really nice. The last time I attempted scraping an app was using an android emulator (bluestacks), then using maybe Wireshark or Charles for getting the API endpoint. It didn't work for some reason though. I don't remember the exact error and I am kinda skeptical about app scraping being this easy.
I recently did something similar with good results (I found the api endpoints I was interested in) using the official Android emulator and https://github.com/mitmproxy/mitmproxy
I did have to jump through some hoops with the emulator and pushing my own ssl cert to it's RO system partition. But it was a few commands and easy enough.
Re: Scraping data from an app: real world example
#19Earlier quoted context omitted.
I would say based on personal observation that the more scrape-worthy an app is, the more likely it has cert pinning. Rather obvious if you think about it, really. High value targets especially from big shops tend to have other measures like complex MACs that make scraping hell. I’m sure most largely-worthless-to-scrape apps don’t employ cert pinning.
> other measures like complex MACs that make scraping hell. Do you have examples of these techniques?
Recent example I encountered: TikTok web API has dynamically generated parameters X-Bogus, msToken and _signature (could be slightly wrong, it’s been a while) that are verified server-side. I haven’t reversed their mobile app so not sure if they also employ MACs there, but I’ve seen these from other apps in the past. And it’s harder when employed in an app; on the web you’ll be reversing (obfuscated) JavaScript in a readily available debugger, whereas for an app you’ll likely be reversing from disassembly.
Re: Scraping data from an app: real world example
#20Most app have some form of SSL pinning system in place which means that you have to perform additional work to allow the proxy to decrypt the HTTPS traffic.
"Most app have some form of SSL pinning system in place..." I would like to see the data, if any, supporting this statement. I would expect some apps would use pinning, but most would not. Google recommends against it. https://developer.android.com/training/articles/security-ssl...
Most games I've tried to examine have had cert pinning enabled.