Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Web Is Broken – Botnet Part 2

jan.wildeboer.net

Post reply on HN
Print view View original

231–240 of 301 posts

Re: The Web Is Broken – Botnet Part 2

#231

Post by jeroenhd »

greesil wrote: post #166
wyck wrote: post #130

Earlier quoted context omitted.

Install a network monitor or go even deeper and sniff packets.

I feel like this could be automated. Spin up a virtual device on a monitored network. Install one app, click on some stuff for awhile, uninstall and move onto the next. If the app reaches out to a lot of random sites then flag it Google could do this. I'm sure Apple could as well. Third parties could for a small set of apps

This is being done by a couple of SDKs, it'd be much easier to just find and flag those SDK files. Finding apps becomes a matter of a single pass scan over the application contents rather than attempting to bypass the VM detection methods malware is packed full of.

Re: The Web Is Broken – Botnet Part 2

#232

Post by rollcat »

aorth wrote: post #223

In the last week I've had to deal with two large-scale influxes of traffic on one particular web server in our organization. The first involved requests from 300,000 unique IPs in a span of a few hours. I analyzed them and found that ~250,000 were from Brazil. I'm used to using ASNs to block network ranges sending this kind of traffic, but in this case they were spread thinly over 6,000+ ASNs! I ended up blocking all…

Try Anubis: https://anubis.techaro.lol>

It's a reverse proxy that presents a PoC challenge to every new visitor. It shifts the initial cost of accessing your server's resources back at the client. Assuming your uplink can handle 300k clients requesting a single 70kb web page, it should solve most of your problems.

For science, can you estimate your peak QPS?

Re: The Web Is Broken – Botnet Part 2

#233

Post by JimDabell »

reconnecting wrote: post #172
sroussey wrote: post #74

Earlier quoted context omitted.

My phone will be on the home network until I walk out of the house and then it will change networks. This should not be a red flag.

Effective fraud prevention relies on both the full user context and the behavioral patterns of known online fraudsters. The key idea is that an IP address cannot be used as a red flag on its own without considering the broader context of the account. However, if we know that the fraudsters we're dealing with are using mobile networks proxies and are randomly switching between two mobile operators, that is certainly a…

An awful lot of free Wi-Fi networks you find in malls are operated by different providers. Walking from one side of a mall to the other while my phone connects to all the Wi-Fi networks I’ve used previously would have you flag me as a fraudster if I understand your approach correctly.

Re: The Web Is Broken – Botnet Part 2

#234

Post by luckylion »

aorth wrote: post #223

In the last week I've had to deal with two large-scale influxes of traffic on one particular web server in our organization. The first involved requests from 300,000 unique IPs in a span of a few hours. I analyzed them and found that ~250,000 were from Brazil. I'm used to using ASNs to block network ranges sending this kind of traffic, but in this case they were spread thinly over 6,000+ ASNs! I ended up blocking all…

I've seen a few attacks where the operators placed malicious code on high-traffic sites (e.g. some government thing, larger newspapers), and then just let browsers load your site as an img. Did you see images, css, js being loaded from these IPs? If they were expecting images, they wouldn't parse the HTML and not load other resources.

It's a pretty effective attack because you get large numbers of individual browsers to contribute. Hosters don't care, so unless the site owners are technical enough, they can stay online quite a bit.

If they work with Referrer Policy, they should be able to mask themselves fairly well - the ones I saw back then did not.

Re: The Web Is Broken – Botnet Part 2

#235

Post by 0manrho »

hubraumhugo wrote: post #178

We all agree that AI crawlers are a big issue as they don't respect any established best practices, but we rarely talk about the path forward. Scraping has been around for as long as the internet, and it was mostly fine. There are many very legitimate use cases for browser automation and data extraction (I work in this space). So what are potential solutions? We're somehow still stuck with CAPTCHAS, a 25 years old co…

> So what are potential solutions?

It won't fully solve the problem, but with the problem relatively identified, you must then ask why people are engaging in this behavior. Answer: money, for the most part. Therefore, follow the money and identify the financial incentives driving this behavior. This leads you pretty quickly to a solution most people would reject out-of-hand: turn off the financial incentive that is driving the enshittification of the web. Which is to say, kill the ad-economy.

Or at least better regulate it while also levying punitive damages that are significant enough to both disuade bad-actors and encourage entities to view data-breaches (or the potential therein) and "leakage[0]" as something that should actually be effectively secured against. Afterall, there are some upsides to the ad-economy that, without it, would present some hard challenges (eg, how many people are willing to pay for search? what happens to the vibrant sphere of creators of all stripes that are incentivized by the ad-economy? etc).

Personally, I can't imagine this would actually happen. Pushback from monied interests aside, most people have given up on the idea of data-privacy or personal-ownership of their data, if they ever even cared in the first place. So, in the absence of willing to do do something about the incentive for this maligned behavior, we're left with few good options.

0: https://news.ycombinator.com/item?id=43716704 (see comments on all the various ways people's data is being leaked/leached/tracked/etc)

Re: The Web Is Broken – Botnet Part 2

#236

Post by nottorp »

udev4096 wrote: post #182
Centigonal wrote: post #176

Earlier quoted context omitted.

yeah, but you can't, that's the problem. Plenty of service operators would like to block every scraper that doesn't obey their robots.txt, but there's no good way to do that without blocking human traffic too (Anubis et al are okay, but they are half-measures). On a separate note, I believe open web scraping has been a massive benefit to the internet on net, and almost entirely positive pre-2021. Web scraping & crawl…

We should have a way to verify the user-agents of the valid and useful scrapers such as Internet Archive by having some kind of cryptographic signature of their user-agents and being able to validate it with any reverse proxy seems like a good start

Self signed, I hope.

Or do you want a central authority that decides who can do new search engines?

Re: The Web Is Broken – Botnet Part 2

#237

Post by nottorp »

gruez wrote: post #111
abaymado wrote: post #86

Earlier quoted context omitted.

> iOS have app privacy report where one can check what connections are made by app, how often, last one, etc. How often is the average calculator app user checking there Privacy Report? My guess, not many!

All it takes is one person to find out and raise the alarm. The average user doesn't read the source code behind openssl or whatever either, that doesn't mean there's no gains in open sourcing it.

The real solution is to add a permission for network access, with the default set to deny.

Re: The Web Is Broken – Botnet Part 2

#238

Post by nottorp »

klabb3 wrote: post #122
philippta wrote: post #79

Earlier quoted context omitted.

Apps should be required to ask for permission to access specific domains. Similar to the tracking protection, Apple introduced a while ago. Not sure how this could work for browsers, but the other 99% of apps I have on my phone should work fine with just a single permitted domain.

On the one hand, yes this could work for many cases. On the other hand, good bye p2p. Not every app is a passive client-server request-response. One needs to be really careful with designing permission systems. Apple has already killed many markets before they had a chance to even exist, such as companion apps for watches and other peripherals.

> On the other hand, good bye p2p.

You mean, good bye using my bandwidth without my permission? That's good. And if I install a bittorrent client on my phone, I'll know to give it permission.

> such as companion apps for watches and other peripherals

That's just apple abusing their market position in phones to push their watch. What does it have to do with p2p?

Re: The Web Is Broken – Botnet Part 2

#239

Post by nottorp »

hinkley wrote: post #183
akoboldfrying wrote: post #174

Earlier quoted context omitted.

I think it should be possible to build something that generalises the idea of Web of Trust so that it's more flexible, and less prone to catastrophic breakdown past some scaling limit. Binary "X trusts Y" statements, plus transitive closure, can lead to long trust paths that we probably shouldn't actually trust the endpoints of. Could we not instead assign probabilities like "X trusts Y 95%", multiply probabilities a…

True. Perhaps a collective vote past 2 degrees of freedom out where multiple parties need to vouch for the same person before you believe they aren't a bot. Then you're using the exponential number of people to provide diminishing weight instead of increasing likelihood of malfeasance.

But do we need an infinite and global web of trust?

How about restricting them to everyone-knows-everyone sized groups, of like a couple hundred people?

One can be a member of multiple groups so you're not actually limited. But the groups will be small enough to self regulate.

Re: The Web Is Broken – Botnet Part 2

#240

Post by nottorp »

wyck wrote: post #132
amiga-workbench wrote: post #7

What is the point of app stores holding up releases for review if they don't even catch obvious malware like this?

This isn't obvious, 99% of apps make multiple calls to multiple services, and these SDK's are embedded into the app. How can you tell whats legit outbound/inbound? Doing a fingerprint search for the worst culprits might help catch some, but it would likely be a game of cat and mouse.

> How can you tell whats legit outbound/inbound?

If the app isn't a web browser, none are legit?

Post reply on HN