Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the actual fallout been from the Cloudflare bug?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 80 posts

Re: Ask HN: What's the actual fallout been from the Cloudflare bug?

#21

Post by tgragnato »

jsmeaton wrote: post #13
tgragnato wrote: post #7

An information leak is an information leak : we still fail to realise that it's something that's happening daily. There's no drama in it. Criminals are taking advantage of opportunities like this every day, still no one cares too much about it (HN bubble & friends excluded). Things like this may have a strong impact or not in the press/popularity circus, but in this particular case it seems they promptly monitored th…

> What most surprises me is that their highly competent staff is thoughtlessly violating one of the security principles in sw : SECURITY BY ISOLATION . I don't think this is really true, but I'm open to hearing your thoughts on this. There was a bug in their HTML parser which caused unrelated memory to be dumped to the process. Their SSL termination servers were isolated elsewhere which is why SSL keys weren't dumped…

I read their post "incident report" too . Isolation is claimed, but if I understand correctly, HTTP handling is shared between customers. Am I wrong ?

Suppose you're a "bad actor", knowing this is a shared service, wouldn't you look for 0 days in it ? A carefully crafted exploit has the potential to leak specific content from unaware customers again.

The attack surface is nginx (http://nginx.org/en/security_advisories.html) plus each component of each loaded module ...

It would be saner to apply isolation to each element of the cartesian product between customers and services.

The performance (and cost) impact can be mitigated by scheduling resources over a pool of disposable virtual machines (obviously in xen and with iommu protection), but I bet they can develop even better solutions.

Re: Ask HN: What's the actual fallout been from the Cloudflare bug?

#23

Post by jsmeaton »

tgragnato wrote: post #21
jsmeaton wrote: post #13

Earlier quoted context omitted.

> What most surprises me is that their highly competent staff is thoughtlessly violating one of the security principles in sw : SECURITY BY ISOLATION . I don't think this is really true, but I'm open to hearing your thoughts on this. There was a bug in their HTML parser which caused unrelated memory to be dumped to the process. Their SSL termination servers were isolated elsewhere which is why SSL keys weren't dumped…

I read their post "incident report" too . Isolation is claimed, but if I understand correctly, HTTP handling is shared between customers. Am I wrong ? Suppose you're a "bad actor", knowing this is a shared service, wouldn't you look for 0 days in it ? A carefully crafted exploit has the potential to leak specific content from unaware customers again. The attack surface is nginx ( http://nginx.org/en/security_advisori…

Even if there were a set of VMs per customer (and all the scaling per customer overhead that goes along with that), a carefully crafted exploit would still reveal details for that customer. Then it'd be a matter of enumerating all of the customers you were interested in exploiting, which would make it easier to get data for a specific target.

The operational overhead of VM/Container isolation for the cartesian product of customer + service sounds like it'd be extremely prohibitive. It's certainly a tradeoff, but to claim it's saner is missing all of the other costs associated with such a system.

Re: Ask HN: What's the actual fallout been from the Cloudflare bug?

#24

Post by jsmeaton »

icebraining wrote: post #15
jsmeaton wrote: post #13

Earlier quoted context omitted.

> What most surprises me is that their highly competent staff is thoughtlessly violating one of the security principles in sw : SECURITY BY ISOLATION . I don't think this is really true, but I'm open to hearing your thoughts on this. There was a bug in their HTML parser which caused unrelated memory to be dumped to the process. Their SSL termination servers were isolated elsewhere which is why SSL keys weren't dumped…

I think the implication is that the isolation should be per customer , each being allocated their own parsing process, isolated from the other customers. That's roughly what we do, though we run an hosted version of an open source webapp, not a CDN. It's more expensive resource-wise (particularly RAM), but it has meant that we were immune to 90%+ of the security bugs discovered in the platform.

Sure, that's a valid question to ask. But imagine you have 1,000,000 customers. Now you have to calculate and manage scaling groups for 1,000,000 customers * number of services. The resourcing costs alone would be outlandish, not to mention trying to independently scale each customer. Perhaps container systems would make this easier, but do they have better memory isolation? Is it possible for a container process to overrun into another containers memory without an exploit in the container system?

Re: Ask HN: What's the actual fallout been from the Cloudflare bug?

#25

Post by JshWright »

ScottBurson wrote: post #5

As I understand it -- somebody please correct me if I have this wrong -- the thing about Cloudbleed is that there isn't necessarily any relationship between the site whose page is cached and the site whose credentials appear in that cached page. So the only way to know that a particular site didn't have credentials leaked is to search all the caches of all the search engines on the Internet. So, as perlgeek says, we'…

Plus all the non-search engine caches, plus the computers of the zero or more people that made requests knowing these sorts of leaks were happening but didn't say anything.

The stuff that got cached was just the persistent vulnerability, there's no way to know how many people noticed the issue taking place in the direct requests they were making.

Re: Ask HN: What's the actual fallout been from the Cloudflare bug?

#26

Post by saurik »

dedalus wrote: post #18

The way you know it's real is when you call up cloudflare's top customers and ask if they would switch to the competition the answer was a resounding yes. That's how I know it's not based on a HN bubble

I can't parse the tense here to determine if you or others did this already?

Re: Ask HN: What's the actual fallout been from the Cloudflare bug?

#27

Post by icebraining »

jsmeaton wrote: post #24
icebraining wrote: post #15

Earlier quoted context omitted.

I think the implication is that the isolation should be per customer , each being allocated their own parsing process, isolated from the other customers. That's roughly what we do, though we run an hosted version of an open source webapp, not a CDN. It's more expensive resource-wise (particularly RAM), but it has meant that we were immune to 90%+ of the security bugs discovered in the platform.

Sure, that's a valid question to ask. But imagine you have 1,000,000 customers. Now you have to calculate and manage scaling groups for 1,000,000 customers * number of services. The resourcing costs alone would be outlandish, not to mention trying to independently scale each customer. Perhaps container systems would make this easier, but do they have better memory isolation? Is it possible for a container process to…

A container is a process with some extra isolation (namespaces), they certainly can't overrun into each other without an exploit.

Why would the costs be outlandish? We offer that and we're fairly cheap. Since the cost is mostly fixed per customer, it should scale linearly.

As for scaling, they already have to do that, by pointing different requests at different servers depending on their load, etc.

Re: Ask HN: What's the actual fallout been from the Cloudflare bug?

#28

Post by tgragnato »

jsmeaton wrote: post #23
tgragnato wrote: post #21

Earlier quoted context omitted.

I read their post "incident report" too . Isolation is claimed, but if I understand correctly, HTTP handling is shared between customers. Am I wrong ? Suppose you're a "bad actor", knowing this is a shared service, wouldn't you look for 0 days in it ? A carefully crafted exploit has the potential to leak specific content from unaware customers again. The attack surface is nginx ( http://nginx.org/en/security_advisori…

Even if there were a set of VMs per customer (and all the scaling per customer overhead that goes along with that), a carefully crafted exploit would still reveal details for that customer . Then it'd be a matter of enumerating all of the customers you were interested in exploiting, which would make it easier to get data for a specific target. The operational overhead of VM/Container isolation for the cartesian produ…

Yeah it makes a targeted attack easier. But it prevents attacks across different customers. Tradeoffs ...

Maintaining a running pool of VMs per service in a sufficient number to serve the load of requests grouped by customer, and assigning the VM to a specific customer only at needs is different than running permanently a pool of (n).customers x (m).services VMs.

This is why an efficient scheduler and the usage of disposable VMs is a need. Still depending on the load and the variety of the traffic it may not be feasible, you are absolutely right !

Another approach to ensure isolation is the usage of a MAC framework. As I wrote "I bet they can develop even better solutions" ;)

Post reply on HN