CPU Usage Differences After Applying Meltdown Patch at Epic Games • Hacker News Board

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#161

Post by mwarkentin » Sat, Jan 06, 2018, 4:27 PM UTC

bonzini wrote: post #87

rocqua wrote: post #35

Earlier quoted context omitted.

The replies are missing something from the original article. They are running on the cloud, and Meltdown / spectre means that exploits can escape a VM. This means you don't just need to trust your VM, but also any VMs you are sharing the hardware with.

Only Spectre CVE-2017-5715 can escape a VM, not Meltdown. Even then, you don't need to trust your co-tenants. Updating the host (which your provider has already done, or which is anyway out of your control) is both necessary and sufficient to protect from VM escapes.

I think it’s the other way around.. and the host updates are what have been introducing these performance hits as well.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#162

Post by Waterluvian » Sat, Jan 06, 2018, 4:29 PM UTC

forgotpw2018 wrote: post #132

Huge real world performance impact, they didn't say how much but it looks like close to 100%. I smell a class action lawsuit coming.

I've been too focused on the tech that I forgot about the legal part of this. Is this why they've all given wishy-washy responses that take no responsibility and seemingly don't even admit there's a problem? Sounds like they expect massive legal recourse and have no real choice but to listen to their lawyers who are telling them to admit no guilt.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#163

Post by merb » Sat, Jan 06, 2018, 4:29 PM UTC

drawnwren wrote: post #76

Darthy wrote: post #71

The Meltdown attack requires an attacker to have a piece of code executed on your server. Epic's servers are used for login, where people send you data, and for game logic, where people also just send you data like "player x moved his avatar here, player y shoots etc". If all the server does is execute the code which Epic wrote themselves and already trust, why would it need to apply the Meltdown patch?

This is a horrible approach to security. If you only secure against attacks you expect, you're gonna have a bad time.

if you do not apply spectre/meltdown migtation on non virtualized hardware which only you control you are safe. of course a malicious process could do harm without them, but a malicious process probably can be as harmful, even if you have the patches applied. basically we need these patches since on the internet a ton of stuff lives on virtualized hardware and there you can actually be harmful to other vms or even the host itself. basically spectre/meltdown needs to be applied in hosted environments and on clients. on everything else it's way less scary than on those two environments.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#164

Post by QAPereo » Sat, Jan 06, 2018, 4:33 PM UTC

Waterluvian wrote: post #162

forgotpw2018 wrote: post #132

Huge real world performance impact, they didn't say how much but it looks like close to 100%. I smell a class action lawsuit coming.

I've been too focused on the tech that I forgot about the legal part of this. Is this why they've all given wishy-washy responses that take no responsibility and seemingly don't even admit there's a problem? Sounds like they expect massive legal recourse and have no real choice but to listen to their lawyers who are telling them to admit no guilt.

Yes. They’re facing potentially ruinous class actions from individuals, and the legal fury of uncounted corporations which are going to suffer. It’s hard to imagine Intel dying, but if the impact on performance is so enormous, it could happen. So yes, they are in ass-covering-survival-mode. Having said that, I wouldn’t expect Intel to go down in legal flames, but the endless PR dripping is going to have to become a flood for a long time to recover.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#165

Post by Abishek_Muthian » Sat, Jan 06, 2018, 4:33 PM UTC

bartread wrote: post #147

Abishek_Muthian wrote: post #133

Earlier quoted context omitted.

Actually, there is not any measurable difference. Our architecture is completely state-less and when compared with equivalent load there's no difference at all. I guess, default network throughput bottle-neck itself is higher than the random memory cache bottle-neck in my case. There was no impact on the latency either. Fearing automatic update by AWS resulting in performance issue, we rushed to update all our serves…

Thanks - this is what's worrying me: the result of patching seems to be all over the map, and it's somewhat hard to predict. My biggest worry is SQL Server, so our plan is to fix a performance issue we isolated recently before we patch - I'd like that system running optimally before we take the hit.

Judging from our MongoDB instance, I'm optimistic that your SQL server wouldn't face much impact either. Even Redhat tests put the risk at modest.

But for something like Redis, the case could be different.

If you are on a cloud provider, do check that automatic update is not scheduled; if so it's better to update manually to follow up with mitigation if necessary.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#166

Post by Waterluvian » Sat, Jan 06, 2018, 4:35 PM UTC

forgotpw2018 wrote: post #153

Filligree wrote: post #142

Earlier quoted context omitted.

More than a doubling of CPU use, but for consistency's sake: The other percentages we've been offered have been in terms of performance reduction. We were told to expect a range of 5-30%; this is about 60%. Somewhere a sysadmin's hair has started smoldering.

Intel's PR dept is in overdrive, but the truth about this vulnerability is that it's essentially worst-case. It really only affects workloads where high performance is important. The average user might not see an impact but if you need fast IO God help you. The solution is to 'make less syscalls' but the problem is that syscalls have always been slow and the people making a lot of them are only doing so because they…

Isn't that like saying that it turns out this car we sold can only do 150, not 200. But it's okay because most of you never drive above 80 anyways.

You're right. Most people who just surf the Netflix and download the YouTubes will not notice. But it's still a form of fraud, even to those who never max out CPU. I think fraud is a strong word knowing that this wasn't intentional. But they sold a lesser product than they advertised and need to make customers whole. Otherwise it pretty much is fraud.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#167

Post by stefantalpalaru » Sat, Jan 06, 2018, 4:36 PM UTC

It's time to move to userspace network drivers like https://github.com/snabbco/snabb/

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#168

Post by sillysaurus3 » Sat, Jan 06, 2018, 4:38 PM UTC

forgotpw2018 wrote: post #132

Huge real world performance impact, they didn't say how much but it looks like close to 100%. I smell a class action lawsuit coming.

Hopefully not. Lawsuits are a fine way to stifle innovation. Imagine how hard it would be to push through any idea at Intel.

No one saw this coming. Things happen. It's impossible to predict every contingency.

They acted in good faith.

Also, heh, users are funny: "all other games i have work fin by the way so there must be a problem whit fortnite."

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#169

Post by Tuna-Fish » Sat, Jan 06, 2018, 4:41 PM UTC

arien wrote: post #74

imjasonmiller wrote: post #32

Earlier quoted context omitted.

If, as you say, AMD is not affected by Meltdown unlike Intel, will this significantly change the server market? Excuse the pun, but would this make e.g. AMD EPYC a lot more attractive for such data centers?

Aren't they still vulnerable to Spectre?

Yes to the bounds check violation version, so far no to the BTB poisoning version. The bounds check version only works inside a single process, so is only relevant when you run untrusted code within the same process as some private data (such as JS in a web browser).

AMD claims that they believe that the way their branch predictor works effectively makes the BTB poisoning unusable, but there is no actual proof, and their statement regarding it was much more wishy-washy than they were with meltdown. (Which they specifically state they are completely immune to.)

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#170

Post by nudgeee » Sat, Jan 06, 2018, 4:42 PM UTC

forgotpw2018 wrote: post #132

Huge real world performance impact, they didn't say how much but it looks like close to 100%. I smell a class action lawsuit coming.

I think CPU designers are getting too much flak for this. The side effects of speculative execution has been around ever since their introduction. A compounding problem is how abstract software has become, it has made assumptions on the underlying execution model without a deeper understanding of its sublteties (sandboxing, containers, etc trying to guarantee process isolation). It’s the Law of Leaky Abstractions [0] at work. What is alarming is how long it has taken for us to collectively realize this is a problem. Hats off to the folks for uncovering this blind spot to us.

[0] https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...