Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Cloudflare responded to the “Copy Fail” Linux vulnerability

blog.cloudflare.com

Post reply on HN
Print view View original

61–70 of 86 posts

Re: How Cloudflare responded to the “Copy Fail” Linux vulnerability

#61

Post by aduwah »

tptacek wrote: post #59

This is an interesting post from Cloudflare, as usual, but it's not clear to me why they would have been vulnerable to CopyFail. Did I miss the point in this blog where that's addressed? What triggered the threat hunting and mitigation exploit? At what points in their architecture were they reliant on Linux user-based access control?

The whole IT industry is reliant on Linux user-based access controls, it is not a Cloudflare thing.

Also leaving a massive gap like this behind would be a mistake on multiple levels. For example, it might get combined with another exploit that can achieve unprivileged access to some piece of metal, or you can have a disgruntled employee without admin access escalating their permissions on a box they aren't supposed to see all the secrets.

Re: How Cloudflare responded to the “Copy Fail” Linux vulnerability

#62

Post by TacticalCoder »

aduwah wrote: post #61
tptacek wrote: post #59

This is an interesting post from Cloudflare, as usual, but it's not clear to me why they would have been vulnerable to CopyFail. Did I miss the point in this blog where that's addressed? What triggered the threat hunting and mitigation exploit? At what points in their architecture were they reliant on Linux user-based access control?

The whole IT industry is reliant on Linux user-based access controls, it is not a Cloudflare thing. Also leaving a massive gap like this behind would be a mistake on multiple levels. For example, it might get combined with another exploit that can achieve unprivileged access to some piece of metal, or you can have a disgruntled employee without admin access escalating their permissions on a box they aren't supposed t…

> For example, it might get combined with another exploit that can achieve unprivileged access ...

Yeah. TFA mentions datacenters in 330 cities. That's a lot of Linux boxen. And many of those have, by definition, ports opened to the big bad Internet. These Linux servers are running services. They answer to ping, for a start. I even heard some are running DNS servers. Remote local exploits are a thing.

What does CloudFlare prefer: that when the next remote local exploit surface all their fleet is one copy.fail away from privilege escalation to root or that they get the time (seen that they obviously have quite advanced detection measures in place) to detect the intruder before it gains root everywhere?

It's Linux. It's datacenters in 330 cities. Linux powers the world and that's how things works.

I, for one, I'm glad to own CloudFlare stocks since right after the 2022 crash and, for two, I'm happy they don't let their huge fleet of Linux servers with a non-patched exploit.

Re: How Cloudflare responded to the “Copy Fail” Linux vulnerability

#63

Post by tptacek »

TacticalCoder wrote: post #62
aduwah wrote: post #61

Earlier quoted context omitted.

The whole IT industry is reliant on Linux user-based access controls, it is not a Cloudflare thing. Also leaving a massive gap like this behind would be a mistake on multiple levels. For example, it might get combined with another exploit that can achieve unprivileged access to some piece of metal, or you can have a disgruntled employee without admin access escalating their permissions on a box they aren't supposed t…

> For example, it might get combined with another exploit that can achieve unprivileged access ... Yeah. TFA mentions datacenters in 330 cities. That's a lot of Linux boxen. And many of those have, by definition, ports opened to the big bad Internet. These Linux servers are running services. They answer to ping, for a start. I even heard some are running DNS servers. Remote local exploits are a thing. What does Cloud…

I'm not asking why they'd need to go threat-hunting if there was an ICMP kernel RCE in Linux. CopyFail requires someone untrusted running shell commands somewhere. Where is that exposure in their architecture?

I'm asking because I don't think they have such an exposure.

Re: How Cloudflare responded to the “Copy Fail” Linux vulnerability

#64

Post by tptacek »

robotbikes wrote: post #60
tptacek wrote: post #59

This is an interesting post from Cloudflare, as usual, but it's not clear to me why they would have been vulnerable to CopyFail. Did I miss the point in this blog where that's addressed? What triggered the threat hunting and mitigation exploit? At what points in their architecture were they reliant on Linux user-based access control?

I would assume it was about protecting their servers from internal sources escalating privileges vs. them providing publicly accessible Linux shells.

I mean, that's a real project, but Linux LPEs kind of grow on trees, so you can't literally rely on threat intelligence for this problem; presumably you handle it by drastically scoping down and surveilling what people do on prod hosts.

Re: How Cloudflare responded to the “Copy Fail” Linux vulnerability

#65

Post by Yoric »

tptacek wrote: post #63
TacticalCoder wrote: post #62

Earlier quoted context omitted.

> For example, it might get combined with another exploit that can achieve unprivileged access ... Yeah. TFA mentions datacenters in 330 cities. That's a lot of Linux boxen. And many of those have, by definition, ports opened to the big bad Internet. These Linux servers are running services. They answer to ping, for a start. I even heard some are running DNS servers. Remote local exploits are a thing. What does Cloud…

I'm not asking why they'd need to go threat-hunting if there was an ICMP kernel RCE in Linux. CopyFail requires someone untrusted running shell commands somewhere. Where is that exposure in their architecture? I'm asking because I don't think they have such an exposure.

At the very least, Cloudflare hosts web workers, which let a customer execute more-or-less arbitrary wasm code on their servers. If there's an exploit that lets you escape the wasm sandbox, copy.fail can be chained into (afaiu) an exploit against the Linux host. That's a pretty big risk.

Also, Cloudflare hosts some AI services, so it's possible that some consumers are running Python code in their containers, without the wasm sandbox.

Re: How Cloudflare responded to the “Copy Fail” Linux vulnerability

#66

Post by tptacek »

Yoric wrote: post #65
tptacek wrote: post #63

Earlier quoted context omitted.

I'm not asking why they'd need to go threat-hunting if there was an ICMP kernel RCE in Linux. CopyFail requires someone untrusted running shell commands somewhere. Where is that exposure in their architecture? I'm asking because I don't think they have such an exposure.

At the very least, Cloudflare hosts web workers, which let a customer execute more-or-less arbitrary wasm code on their servers. If there's an exploit that lets you escape the wasm sandbox, copy.fail can be chained into (afaiu) an exploit against the Linux host. That's a pretty big risk. Also, Cloudflare hosts some AI services, so it's possible that some consumers are running Python code in their containers, without…

If there's a direct link from Cloudflare workers / WASM to uid=nobody execve or arbitrary syscalls on their hosts, they're already fucked, so I don't think that's true.

Re: How Cloudflare responded to the “Copy Fail” Linux vulnerability

#67

Post by HDBaseT »

tptacek wrote: post #66
Yoric wrote: post #65

Earlier quoted context omitted.

At the very least, Cloudflare hosts web workers, which let a customer execute more-or-less arbitrary wasm code on their servers. If there's an exploit that lets you escape the wasm sandbox, copy.fail can be chained into (afaiu) an exploit against the Linux host. That's a pretty big risk. Also, Cloudflare hosts some AI services, so it's possible that some consumers are running Python code in their containers, without…

If there's a direct link from Cloudflare workers / WASM to uid=nobody execve or arbitrary syscalls on their hosts, they're already fucked, so I don't think that's true.

I don't understand your point.

You seem so pressed on the fact "why would they even patch this!!!", maybe because its best practice to patch things? You never known what things could be chained together, so you might as well patch this, given its so obviously bad.

Re: How Cloudflare responded to the “Copy Fail” Linux vulnerability

#68

Post by js2 »

HDBaseT wrote: post #67
tptacek wrote: post #66

Earlier quoted context omitted.

If there's a direct link from Cloudflare workers / WASM to uid=nobody execve or arbitrary syscalls on their hosts, they're already fucked, so I don't think that's true.

I don't understand your point. You seem so pressed on the fact "why would they even patch this!!!", maybe because its best practice to patch things? You never known what things could be chained together, so you might as well patch this, given its so obviously bad.

That's a straw man and not what he asked. Literally, he asked: "why they would have been vulnerable to CopyFail?"

I've been a sysadmin/programmer since the mid-90s. Local root exploits are a dime a dozen. If your infrastructure relies upon the tenuous difference between root and non-root accounts, you've already lost. Cloudflare isn't an ISP handing out shell accounts on Unix machines.

So again, yes, of course you should patch your Linux machines. Defense in depth and all that. But the question remains: "why Cloudflare would have been vulnerable to CopyFail?" (in anything but an academic sense). Because I do not believe that they can possibly be relying on the difference between root and non-root account.

Re: How Cloudflare responded to the “Copy Fail” Linux vulnerability

#69

Post by js2 »

tptacek wrote: post #59

This is an interesting post from Cloudflare, as usual, but it's not clear to me why they would have been vulnerable to CopyFail. Did I miss the point in this blog where that's addressed? What triggered the threat hunting and mitigation exploit? At what points in their architecture were they reliant on Linux user-based access control?

They weren't vulnerable to it in anything but an academic sense. They call that out up front: "There was no impact to the Cloudflare environment, no customer data was at risk, and no services were disrupted at any point."

This was probably written by their security team. Security teams are paranoid. They want everything patched everywhere all at once at a severity level zeo. Also, PR. Also, also, if through some lack of imagination, this was somehow involved in an exploit of their services, it would look really really bad. So, CYA.

Re: How Cloudflare responded to the “Copy Fail” Linux vulnerability

#70

Post by HDBaseT »

js2 wrote: post #68
HDBaseT wrote: post #67

Earlier quoted context omitted.

I don't understand your point. You seem so pressed on the fact "why would they even patch this!!!", maybe because its best practice to patch things? You never known what things could be chained together, so you might as well patch this, given its so obviously bad.

That's a straw man and not what he asked. Literally, he asked: "why they would have been vulnerable to CopyFail?" I've been a sysadmin/programmer since the mid-90s. Local root exploits are a dime a dozen. If your infrastructure relies upon the tenuous difference between root and non-root accounts, you've already lost. Cloudflare isn't an ISP handing out shell accounts on Unix machines. So again, yes, of course you sh…

I don't care about your credentials. It doesn't take a genius to realize that having known major security holes is not ideal.

It is pretty clear they aren't too concerned about this being a issue for this business, after the first paragraph in bold on the blog:

"There was no impact to the Cloudflare environment, no customer data was at risk, and no services were disrupted at any point. Read on to learn how our preparedness paid off."

As mentioned, you never want to give options to a potential attacker/exploit by keeping known vulnerabilities present in your system. You cannot always predict every single avenue an attack could leverage.

Imagine having a data center with barbed wire fences, guard posts, security and cameras covering every square meter of the facility. You wouldn't just leave a door right open because in theory, people shouldn't be able to walk right in. But why would you willingly leave a door open? Even if the possibility is 0.000001%?

People like you would be the first to turn and say "Cloudflare are morons for not patching this!!! Me and my 1 billion years experience and goat status would of prevented this' when some major Cloudflare hack occurs and it was found that phishing 30 different people and using 9 different exploits (including Copyfail) allowed the attacker to bring down Cloudfare

Post reply on HN