CPU Usage Differences After Applying Meltdown Patch at Epic Games • Hacker News Board

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#151

Post by nine_k » Sat, Jan 06, 2018, 4:10 PM UTC

metaobject wrote: post #148

Filligree wrote: post #142

Earlier quoted context omitted.

More than a doubling of CPU use, but for consistency's sake: The other percentages we've been offered have been in terms of performance reduction. We were told to expect a range of 5-30%; this is about 60%. Somewhere a sysadmin's hair has started smoldering.

I’d go farther, anybody who is concerned with application performance may soon see/smell their hair smoldering. We write a lot of code that needs to run as fast as possible (processing, post-processing, generating real-time weather/satellite data) and I’m concerned about time windows and whether we’ll be able to meet requirements.

I suppose these guys just don't apply the patch. They are not running in a public cloud with potentially hostile neighboring VMs; they run their own trusted code. They might choose to tighten perimeter security instead, in the short term.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#152

Post by jameshart » Sat, Jan 06, 2018, 4:11 PM UTC

lsd5you wrote: post #27

dagss wrote: post #10

Earlier quoted context omitted.

...at which point that untrusted code gains full access to the only userspace process that matters on that node. If you gain access to run code on a server process why escalate further, you already have access to everything that matters?... Assuming bare metal. In shared hosting / cloud / VMs it is different.

This is a great point. Probably someone will argue properly configured apache will not have access to data ... etc. I think the practical reality is in many setups it already does, and what these hardware bugs mean is that the lazy imperfect - effectively single layer - security that probably exists on the majority of servers is now essentially equivalent to the security of systems where the security minded have been…

I think this is a fantastic insight. There is a particular mindset of security thinking which compartmentalizes breach impacts on the basis of how much of the security infrastructure itself is compromised. 'well, they get remote code execution, but at least they can't recover passwords', or 'this vulnerability is bad because it allows recovery of temporary TLS keys'. And Spectre/Meltdown seems to turn every vulnerability into one of these 'world shattering' security breaches that mean your secret keys are all exposed.

But for servers, the application-level vulnerabilities that are needed in order to get meltdown or spectre attacks to run are already devastating. Take over a game server process and you own the in game currency and the scores and the ability to ban users, and probably user level login as well. And you have your pick of privilege escalation mechanisms already, probably.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#153

Post by forgotpw2018 » Sat, Jan 06, 2018, 4:11 PM UTC

Filligree wrote: post #142

forgotpw2018 wrote: post #132

Huge real world performance impact, they didn't say how much but it looks like close to 100%. I smell a class action lawsuit coming.

More than a doubling of CPU use, but for consistency's sake: The other percentages we've been offered have been in terms of performance reduction. We were told to expect a range of 5-30%; this is about 60%. Somewhere a sysadmin's hair has started smoldering.

Intel's PR dept is in overdrive, but the truth about this vulnerability is that it's essentially worst-case.

It really only affects workloads where high performance is important. The average user might not see an impact but if you need fast IO God help you. The solution is to 'make less syscalls' but the problem is that syscalls have always been slow and the people making a lot of them are only doing so because they absolutely have to

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#154

Post by lazyjones » Sat, Jan 06, 2018, 4:12 PM UTC

Those who have looked at the patches in detail: does it treat older generation CPUs differently than 7th gen? The Paper authors who wrote the KAISER patch expected much worse performance on older CPUs due to implementation issues...

Unfortunately, Epic Games don't provide any details about their CPUs AFAICT.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#155

Post by Thaxll » Sat, Jan 06, 2018, 4:13 PM UTC

contrarian_ wrote: post #21

Pretty much what I predicted here: https://news.ycombinator.com/item?id=16054674 > Sounds like servers handling lots of small UDP packets would be hit pretty hard.

Gameservers don't handle that many packets because they're limited in the number of player they host. A game a 60hz with 64 players will only receive 3840 packets/sec.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#156

Post by merb » Sat, Jan 06, 2018, 4:20 PM UTC

if people would still use good old owned hardware without any virtualizsation, spectre and meltdown would not be as scary as it really is on server hardware. which means that only clients would be affected. but since everything runs on the cloud, we basically need to update the whole world.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#157

Post by chrisper » Sat, Jan 06, 2018, 4:20 PM UTC

pitaj wrote: post #137

forgotpw2018 wrote: post #132

Huge real world performance impact, they didn't say how much but it looks like close to 100%. I smell a class action lawsuit coming.

Why? This wasn't gross incompetence in processor design, this kind of attack is completely new. I don't see how a class action could apply here, by IANAL

If you know that your product is flawed, do you keep selling it or will you pull it from the shelves?

To this date I can still buy broken Intel CPUs....

They knew about the flaw in June. Yet they still kept selling Coffee Lake CPUs.

If my 8700k wouldn't still be significantly faster than Amd Ryzen (or I wouldn't have to also return the MB), I would have switched to an Amd in a heart beat.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#158

Post by Demiurge » Sat, Jan 06, 2018, 4:21 PM UTC

dogma1138 wrote: post #100

contrarian_ wrote: post #57

Earlier quoted context omitted.

Yep, I was actually debating whether I'd mention sendmmsg/recvmmsg in my original post but I left it out. Definitely an option for UDP, but you're out of luck if your game server uses TCP (surprisingly many do) because you have to recv from each socket separately.

The majority of online games use TCP these days with good prediction it’s more reliable than UDP.

Latency is more important than reliability for online gaming because the world state instantly gets stale. Instead of retransmit you want latest snapshot. I'd be curious to find out in what online games that is not the case.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#159

Post by ebbv » Sat, Jan 06, 2018, 4:24 PM UTC

merb wrote: post #156

if people would still use good old owned hardware without any virtualizsation, spectre and meltdown would not be as scary as it really is on server hardware. which means that only clients would be affected. but since everything runs on the cloud, we basically need to update the whole world.

First off, having dedicated hardware doesn't make these a non-issue, it just makes them less of an issue. If you have dedicated hardware they are concerns. Applications still need to be isolated from each other, and it's a path for someone who gets in as a user to potentially leverage it to root access.

Second, saying everyone should have dedicated hardware is just nonsense. It's like saying everyone should run their own datacenter. For most people neither of those makes financial sense.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#160

Post by falcolas » Sat, Jan 06, 2018, 4:25 PM UTC

Abishek_Muthian wrote: post #90

bonzini wrote: post #85

Earlier quoted context omitted.

Are you using PV or HVM instances?

All of them on HVM.

IIRC, XEN said that 32bit HVM VMs are not affected by Meltdown, and so probably don't get impacted by AWS' patches. They still require Linux kernel updates to protect the kernel space, so changes might still be seen there.