This! Mobile IPs are far more lucrative. Many services will drop captchas and other anti-bot stuff for consumer mobile IPs. I recall Plaid at some point would run their bank scraping through mobile IPs. This sketchy company lets mobile app developers monetize user base by letting other people pay $$ to route requests through random people’s mobile IPs: https://brightdata.com/
Brightdata is formerly known as as Luminati who is owned by same company as Hola VPN.
Similarly NordVPN owns Oxylabs (who mostly hack routers and cameras and sell those as residential IP’s).
This is really bad. Imagine if someone plants these proxy inside app how user are even going to know? I think every OS should come with firewall so if app tries to make connection it should prompt with Accept | Accept Forever | Deny | Deny Forever. I think these companies used to go for extension developer now it seems they have found new idea to implant malware on apps which is not easy to detect.
This is EXACTLY how mobile proxy companies like Luminati and OxyLabs acquire their ip address pool. They pay devs to embed a lib inside their app.
This list is pretty interesting. If you don't mind me asking, what do you work on that you requires such sophisticated stuff? Also, does this work only for browsers or also for mobile apps? I have always assumed that it is always theoretically possible to get data from browsers (very extreme resort is save the browser page / (screenshot + computer vision)); but it can be impossible to get data from apps (especially i…
Mobile IPs will be a problem until the entire Internet is IPv6. The issue is that there are not enough IPv4 addresses for everyone to get their own IP every time their phone connects to the Internet. So the mobile networks use one IP for many handsets. Block the IP, block dozens of different (innocent) people. Once we're all on IPv6 we can go back to blocking IPs. But then IPv6 creates its own problems.
Doesn't the same idea apply for normal IPs (non-mobile IPs)? Our computers also don't get their separate IPv4 address. We use NATs. So how can someone block our computer specifically without blocking dozens of innocent people using the same public IPv4 address?
Basically what's different about mobile and laptop IPs?
Yeah, I used to work for one of the major anti-bot vendors. Customers weren't clueless. Nobody buys these solutions because they're so much fun, it's a cost center and they monitor their ROI quite closely. Credit card charge backs, impact to infrastructure, extra incurred cost due to underlying api's (like in the Airline industry in particular) etc are all reasons why bot mitigation is a better option than nothing fo…
Do you know much about airline api pricing more? I’ve noticed most sites won’t let you search business fares efficiently, so I made my own for Google Flights which only worked for like 6months until they added bunch of changes that made it near impossible to scrape.
Yeah, there’s a central service that all Flight search is connected to, irregardless of airline. The airlines are charged per search to that api, so they monitor their ”look to book” rationvery closely. That ratio remains quite stable im the absence of bots, but skyrockets with any bot activity. Hence, they know from that metric how big of a bot problem they have and how much money they are losing. Major flight search software vendors have dedicated teams for this.
Bots aren't just trying credential stuffing. They are: - committing clickfraud to game ad and referral revenue systems - posting fake or spam reviews and comments - generating fake behavioral signals to help bypass CAPTCHAs to help create accounts on other sites that can post spam comments - validating stolen credit card details - screwing with your metrics collection if you can't identify them as bots All of that is…
> -validating stolen credit card details To be clear, "validating" is an industry euphemism for stealing, just for a different purpose. How do you validate the card is live? Run a real transaction through it and mark it based on the result. But what do you run for this real transaction? Well, whatever you want. Typically it'll be something to avoid suspicion as much as possible, but the thief gets to pick what they t…
If you're trying to fight financial crime it's important to not simplify behavior seen in the wild. Like the other commented noted, there is a very clear difference in behaviour between those validating cards and those using cards in order to steal.
Mobile IPs will be a problem until the entire Internet is IPv6. The issue is that there are not enough IPv4 addresses for everyone to get their own IP every time their phone connects to the Internet. So the mobile networks use one IP for many handsets. Block the IP, block dozens of different (innocent) people. Once we're all on IPv6 we can go back to blocking IPs. But then IPv6 creates its own problems.
Doesn't the same idea apply for normal IPs (non-mobile IPs)? Our computers also don't get their separate IPv4 address. We use NATs. So how can someone block our computer specifically without blocking dozens of innocent people using the same public IPv4 address? Basically what's different about mobile and laptop IPs?
CGNAT is done at a ridiculously large scale in mobile networks.
Your home or local coffee shop might have 10-20 users max behind a single IP. A mobile network on the other hand might put most a small-medium city behind 3 or 4 IPs.
One of the reasons for this is that the vast majority of the time, mobile LTE data users are behind cgnat for ipv4. You can't block one ip without possibly blocking hundreds of innocent IPs using the same exit point. As a scraper operator on a mobile data connection all you need is a new useragent and browser fingerprint, there's no easy way for a scraper-blocker-operator to tell that you're not a totally new person.
This is reason why most of services uses App instead of Browser. When there is App it can use lot of thing like phone fingerprint derived from various sources.
Apple provides a framework or some sort of unique id
2FA should be a requirement on everything now. And if your site can't for some reason or you don't want to deal with it, then limit your site to external login providers only. 2FA, especially app based, has been proven to work really really well.
thats great till you're in a foreign country and your phone suddnely decides to die leaving you stranded and unable to access bank accounts or prove your identity. (happened to me)
That sounds like a bad planning problem in which you shouldn't have left yourself vulnerable to loss of electronic services. Not a tech issue that justifies intrusive spyware.
Do you know much about airline api pricing more? I’ve noticed most sites won’t let you search business fares efficiently, so I made my own for Google Flights which only worked for like 6months until they added bunch of changes that made it near impossible to scrape.
Yeah, there’s a central service that all Flight search is connected to, irregardless of airline. The airlines are charged per search to that api, so they monitor their ”look to book” rationvery closely. That ratio remains quite stable im the absence of bots, but skyrockets with any bot activity. Hence, they know from that metric how big of a bot problem they have and how much money they are losing. Major flight searc…
In fact the airlines are charged per book, but if and only if the look to book stays within reasonable bounds. If it rockets up, they’re on the hook for the penalties