Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hijack of Amazon’s domain service used to reroute web traffic for two hours

doublepulsar.com

Post reply on HN
Print view View original

211–220 of 291 posts

Re: Hijack of Amazon’s domain service used to reroute web traffic for two hours

#211

Post by fooblitzky »

925dk wrote: post #209

Why do AWS http://status.aws.amazon.com say that only Google resolvers were affected? Between 4:05 AM PDT and 5:56 AM PDT, some customers may have experienced elevated errors resolving DNS records hosted on Route 53 using DNS resolvers 8.8.8.8 / 8.8.4.4. This issue was caused by a problem with a third-party Internet provider. The issue has been resolved and the service is operating normally.

It seems like either the title is misleading, or the article does a poor job of explaining the situation. From my reading of the article text, it seems DNS traffic was rerouted to Route53, which the attackers then used to serve false DNS records. That does not sound like Route53 was hijacked at all, just that the attackers happened to use the service it provides.

Re: Hijack of Amazon’s domain service used to reroute web traffic for two hours

#212

Post by tialaramex »

Manozco wrote: post #100
philjohn wrote: post #33

Earlier quoted context omitted.

Except, it would have been trivial for them to get Lets Encrypt to issue them a valid cert.

it depends, let's encrypt might be performing request from different sources (and then it would need a bigger BGP hijack to fake them)

Yes, Let's Encrypt acknowledges that they perform their validations from multiple observation points on the Internet. They (intentionally) don't provide a list.

This occasionally annoys people who have an idea like "I will firewall everything but whitelist Let's Encrypt" and then find out that while you're welcome to try to puzzle out some way to make this work, Let's Encrypt won't help you and when it breaks you get to keep both halves.

Re: Hijack of Amazon’s domain service used to reroute web traffic for two hours

#213

Post by xraystyle »

chatmasta wrote: post #171

What separates BGP hijacking from BGP anycast? Amazon is already advertising a bunch of routes to its route53 IP's for BGP anycast. How can an upstream peer tell if a new advertisement is another anycast route vs. a hijacked prefix? Is the only way to tell by validating the advertisement came from Amazon?

The issue here is that the attackers were advertising more specific prefixes, and when doing route selection, routers pick routes to these over less specific prefixes that cover the same ranges.

e.g. if Amazon was announcing 205.51.192.0/21, that would cover all the /24s that were announced in the hijack. Not sure what Amazon actually announces, this is all just examples.

Say I'm trying to get to 205.51.192.12, for example. If I have two routes, one advertising 205.51.192.0/21 and another advertising 205.51.192.0/24, the router will forward the traffic to the next-hop advertising the /24, all else being equal.

So if you have the ability to advertise routes over BGP to the internet, you advertise a more specific route than what someone else is already advertising, and your peer has no filtering in place with regard to what they're accepting from you, you can potentially hijack IP space like this.

ETA: Anycast DNS generally advertises the exact same prefix to multiple peers. That way, the best path to that IP will be dependent on where your traffic is coming from, and will hopefully come in over the closest peering to you.

Re: Hijack of Amazon’s domain service used to reroute web traffic for two hours

#214

Post by 925dk »

fooblitzky wrote: post #211
925dk wrote: post #209

Why do AWS http://status.aws.amazon.com say that only Google resolvers were affected? Between 4:05 AM PDT and 5:56 AM PDT, some customers may have experienced elevated errors resolving DNS records hosted on Route 53 using DNS resolvers 8.8.8.8 / 8.8.4.4. This issue was caused by a problem with a third-party Internet provider. The issue has been resolved and the service is operating normally.

It seems like either the title is misleading, or the article does a poor job of explaining the situation. From my reading of the article text, it seems DNS traffic was rerouted to Route53, which the attackers then used to serve false DNS records. That does not sound like Route53 was hijacked at all, just that the attackers happened to use the service it provides.

Traffic to Route 53 was rerouted to an alternative DNS server. But that has nothing to do with my question re. AWS calling out Google in particular in their status update.

Re: Hijack of Amazon’s domain service used to reroute web traffic for two hours

#215

Post by mlosapio »

Wonder if services like Let’s Encrypt were affected. I imagine a scenario where a small hijack of DNS could allow for properly signed certificates for domains that are not owned. If I operated a CA service, I would carefully examine the requests received during this time frame. Maybe someone can audit the Transparency Logs during this period for anomalous activity.

Re: Hijack of Amazon’s domain service used to reroute web traffic for two hours

#216

Post by acdha »

peterwwillis wrote: post #77
acdha wrote: post #62

Earlier quoted context omitted.

> hijack the target's DNS, generate a valid cert for the domain you want. That’s hardly “anyone” – it requires non-trivial resources and, thanks to certificate transparency, is going to be noticed and revoked relatively quickly. That further lowers the likelihood that someone capable of the attack will burn it for a limited window of access.

You consider two hours a limited window? To automate capturing traffic for any domain on the internet? Just as one example, you could initiate bank transfers for everyone doing online banking at any bank. Malware sitting on desktops have been doing this instantly, automatically, using valid sessions from the client side, for like a decade. No reason they couldn't do this sitting in the middle. On top of this, many hi…

> You consider two hours a limited window? To automate capturing traffic for any domain on the internet? Just as one example, you could initiate bank transfers for everyone doing online banking at any bank. Malware sitting on desktops have been doing this instantly, automatically, using valid sessions from the client side, for like a decade. No reason they couldn't do this sitting in the middle.

Again, I'm not saying that this isn't a big deal but that you're really downplaying the level of effort required for the worst-case scenarios. Besides the average script kiddie not being likely to compromise an ISP to send the malicious announcements, your online banking scenario would only work for the people actively logging in during those two hours to banks which don't use IP fixing, out of band confirmation or other precautions to actually transfer money (every single financial institution I use has a period measured in days with human-on-phone verification to setup a new outbound transfer target); whose security staff don't check certificate transparency logs; and you'd have to develop the code to collect credentials and run the transfer for each bank, taking care to avoid triggering key-pinning and other high visibility errors.

Malware is similar: yes, it'd suck if you could get a valid HTTPS cert for someone's automatic update service but these days those certificates are pinned and most system use signed updates which would require a separate exploit to avoid.

One thing to remember is that this is coming after a decade of everyone responsible in the tech industry trying to protect against malicious state actors. This attack is what happens all the time to people in countries with repressive governments.

Re: Hijack of Amazon’s domain service used to reroute web traffic for two hours

#217

Post by tialaramex »

peterwwillis wrote: post #39
vertex-four wrote: post #32

Earlier quoted context omitted.

The solution to this is the CAA issue record combined with DNSSEC. DNSSEC ensures that an attacker can't spoof your DNS even if they have your DNS traffic directed to them globally, and then CAA ensures that only the issuers you want can create certificates for your domain. This is a solved problem, it just requires caring about it.

CAA depends on CAs (all 650 of them) to respect it, and for all of them to implement everything perfectly (it was shown in 2017 that they don't). This also depends on DNS again, which again not everyone will do properly, so CAA requests from CAs can be hijacked. So CAA is stupid. The client should be getting the list of authorized CAs for the domain from DNS on first connection, and/or from inside the first cert, wit…

How do you get 650?

I count significantly less than 100 organisations, I started with the CCADB (the database shared by Mozilla and Microsoft) for all intermediates, I cut those that aren't part of the Mozilla root programme, and then I asked for the "CA Owner" field, which avoids thinking a single owner is really several distinct entities when it isn't.

Re: Hijack of Amazon’s domain service used to reroute web traffic for two hours

#218

Post by erulabs »

mlosapio wrote: post #215

Wonder if services like Let’s Encrypt were affected. I imagine a scenario where a small hijack of DNS could allow for properly signed certificates for domains that are not owned. If I operated a CA service, I would carefully examine the requests received during this time frame. Maybe someone can audit the Transparency Logs during this period for anomalous activity.

I would assume they go far out of their way to resolve the address from a large number of distributed DNS resolvers - the question is if they're just round robin instead of multiple simultaneous queries, but I would hope that's part of why the DNS test usualy takes >60 seconds. If they're not treating DNS as a consensus protocol instead of as an immutable database, they'll always be open to poisoning attacks.

If any certs were issued for hijacked domains (which as far as I've ready was only one, not using LetsEncrypt), it's a pretty glaring failure on the issuer, assuming they used "DNS Validation"

Re: Hijack of Amazon’s domain service used to reroute web traffic for two hours

#219

Post by adamb_ »

7ewis wrote: post #133

How would you see this in ThousandEyes? I see he has a screenshot of it in the post, and doesn't appear to be logged in. Was troubleshooting this for the two hours, until I started reading reports of the BGP leak. Not in networking, so don't fully understand how it works. I'd just like to know how to troubleshoot this in the future, and check for leaks, if any. Are there any other online tools I could use? Obviously…

> How would you see this in ThousandEyes?

Checkout this video analysis from the team on the hijack: https://youtu.be/YXm4GJMUlP0

Re: Hijack of Amazon’s domain service used to reroute web traffic for two hours

#220

Post by tptacek »

tialaramex wrote: post #207
rocqua wrote: post #165

Earlier quoted context omitted.

> There is an alternate universe in which DANE helps this problem. In that universe, there are no CAs anymore There is another, even less likely, universe where DANE works. Here, every cert needs to be validated by both DANE and a CA. Security wise, this would be an improvement over just CAs. However, the large governments would still be able to MitM TLDs in their control (just patriot act some CA). This is where I p…

The TLS WG is currently haggling about, effectively, DANE's future. The thing up in the air was, should this technology for saying "Hi, I can do DANE" be pinned? e.g. a client sees "I can do DANE" and it says to itself "OK, I will now expect DANE certificates from this TLS service for... 24 hours? 10 minutes? 10 years? An amount of time defined in some new parameter?" Or maybe it never does this. Last I looked at the…

The TLS WG can haggle about DANE all it wants, but DANE is a dead letter: it was tried in two of the four major browsers and failed, and a third (Safari) briefly had nascent framework support for it and jettisoned it. Without support from the browser vendors, DANE is irrelevant.

The big problem Chrome had with DANE was that they couldn't (and presumably still can't) assume reliable DNSSEC/DANE record lookups. If you can't fail closed with DANE, then all DANE really is is another CA; attackers can "strip" DANE from connections, trivially, and be exactly where they are today. Meanwhile, end-users pay the price in latency and reduced performance for a cosmetic security improvement that most people don't care about.

Post reply on HN