Scraping data from an app: real world example

#1

Post by DantesTravel » Sun, Sep 04, 2022, 8:20 PM UTC

Scraping data from an app: real world example
substack.thewebscraping.club

Re: Scraping data from an app: real world example

#2

Post by IMTDb » Sun, Sep 04, 2022, 10:37 PM UTC

Most app have some form of SSL pinning system in place which means that you have to perform additional work to allow the proxy to decrypt the HTTPS traffic.

Re: Scraping data from an app: real world example

#3

Post by 1vuio0pswjnm7 » Sun, Sep 04, 2022, 11:25 PM UTC

IMTDb wrote: post #2

Most app have some form of SSL pinning system in place which means that you have to perform additional work to allow the proxy to decrypt the HTTPS traffic.

"Most app have some form of SSL pinning system in place..."

I would like to see the data, if any, supporting this statement. I would expect some apps would use pinning, but most would not.

Google recommends against it.

https://developer.android.com/training/articles/security-ssl...

Re: Scraping data from an app: real world example

#4

Post by forgotmypw17 » Mon, Sep 05, 2022, 12:40 AM UTC

1vuio0pswjnm7 wrote: post #3

IMTDb wrote: post #2

Most app have some form of SSL pinning system in place which means that you have to perform additional work to allow the proxy to decrypt the HTTPS traffic.

"Most app have some form of SSL pinning system in place..." I would like to see the data, if any, supporting this statement. I would expect some apps would use pinning, but most would not. Google recommends against it. https://developer.android.com/training/articles/security-ssl...

>Caution: Certificate Pinning is not recommended for Android applications due to the high risk of future server configuration changes, such as changing to another Certificate Authority, rendering the application unable to connect to the server without receiving a client software update.

This actually applies to websites and browsers as well.

Re: Scraping data from an app: real world example

#5

Post by metadat » Mon, Sep 05, 2022, 12:58 AM UTC

forgotmypw17 wrote: post #4

1vuio0pswjnm7 wrote: post #3

Earlier quoted context omitted.

"Most app have some form of SSL pinning system in place..." I would like to see the data, if any, supporting this statement. I would expect some apps would use pinning, but most would not. Google recommends against it. https://developer.android.com/training/articles/security-ssl...

>Caution: Certificate Pinning is not recommended for Android applications due to the high risk of future server configuration changes, such as changing to another Certificate Authority, rendering the application unable to connect to the server without receiving a client software update. This actually applies to websites and browsers as well.

Why isn't there a site-controlled fallback setting for this?

Does this not make sense? Abu given website's beet interest is to continue to be reachable.

Re: Scraping data from an app: real world example

#6

Post by matheusmoreira » Mon, Sep 05, 2022, 2:46 AM UTC

This is a great method. It's essentially creating a custom client for their server. Fiddler makes interception of encrypted traffic from apps a lot easier than the last time I tried this. Really nice.

Re: Scraping data from an app: real world example

#7

Post by nothasan » Mon, Sep 05, 2022, 2:47 AM UTC

I have worked with the site mentioned in the article’s API previously. I am not sure why they used the overhead of a "scraping framework" when it was just JSON they needed to parse.

Re: Scraping data from an app: real world example

#8

Post by matheusmoreira » Mon, Sep 05, 2022, 2:51 AM UTC

nothasan wrote: post #7

I have worked with the site mentioned in the article’s API previously. I am not sure why they used the overhead of a "scraping framework" when it was just JSON they needed to parse.

Perhaps they're used to the Scrapy APIs and tools? I agree that an HTTP requests library would have been sufficient but maybe it was just easier for them to use the framework they're used to.

Re: Scraping data from an app: real world example

#9

Post by ratg13 » Mon, Sep 05, 2022, 2:56 AM UTC

IMTDb wrote: post #2

Most app have some form of SSL pinning system in place which means that you have to perform additional work to allow the proxy to decrypt the HTTPS traffic.

Can someone suggest some resources to understand the additional work needed to decrypt the pinned https traffic?

Re: Scraping data from an app: real world example

#10

Post by anyfactor » Mon, Sep 05, 2022, 3:03 AM UTC

That is really nice. The last time I attempted scraping an app was using an android emulator (bluestacks), then using maybe Wireshark or Charles for getting the API endpoint. It didn't work for some reason though. I don't remember the exact error and I am kinda skeptical about app scraping being this easy.