Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CPU Usage Differences After Applying Meltdown Patch at Epic Games

epicgames.com

Post reply on HN
Print view View original

271–280 of 296 posts

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#271

Post by merb »

msbarnett wrote: post #259
merb wrote: post #163

Earlier quoted context omitted.

if you do not apply spectre/meltdown migtation on non virtualized hardware which only you control you are safe. of course a malicious process could do harm without them, but a malicious process probably can be as harmful, even if you have the patches applied. basically we need these patches since on the internet a ton of stuff lives on virtualized hardware and there you can actually be harmful to other vms or even th…

Do you also run all processes on your server as root? If you’re not going to apply patches that close privilege escalation bugs, you might as well. You’re putting all your security eggs in one basket.

Well a lot of processes run as Root anyway. Also if possible i run as few processes as possible

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#272

Post by krylon »

djsumdog wrote: post #210
serf wrote: post #185

Earlier quoted context omitted.

>They acted in good faith. That's new for a company, and it's not indicated by their PR spin right now. What I think happened : a company produced a product with a problem. Probably not out of malice, but ignorance. One of two things happened after, which can kill the 'good faith' argument ; the problem was found internally and hushed, or the problem was found externally and minimized to reduce financial burden arisi…

Well this type of attack has been theoretical for years. The Project Zero referenced some papers from the mid-2000s that talked about it. But the implementation, even today, isn't exactly trivial. Modern processors are insanely complex systems. Branch prediction, out of order execution, hardware virtual memory management, hardware virtualization, etc. Not to mention that these are side-channel attacks. It's not a dir…

> The Project Zero referenced some papers from the mid-2000s that talked about it.

http://www.daemonology.net/papers/htt.pdf

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#273

Post by krylon »

Tempest1981 wrote: post #244
forgotpw2018 wrote: post #132

Huge real world performance impact, they didn't say how much but it looks like close to 100%. I smell a class action lawsuit coming.

With the Pentium floating point bug (1994), Intel offered to replace the defective CPUs. https://en.m.wikipedia.org/wiki/Pentium_FDIV_bug IIRC almost nobody took them up on the offer. But today there is far more cloud computing, so datacenters might.

But back then, Intel could not deny that the FDIV bug was an actual bug in the CPU. Intel's press release on Meltdown is carefully worded to avoid admitting it's a bug in the CPU.

But since the USA have the highest lawyer-per-citizen ratio in the world, it would be very interesting to see what happened if, say, Google sued Intel because from one day to the next, Google's electricity bills rise steeply, their servers crash, because the AC cannot handle the extra heat, ... , and Google demanded Intel pay for their damages.

We live in very interesting times, indeed.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#274

Post by burnte »

yborg wrote: post #195
Abishek_Muthian wrote: post #60

If it's helpful, Our Node.js, MongoDB, Python servers all with significant network traffic didn't have any measurable impact after KPTI patches on Amazon Linux on T2.medium(burst), M4.large, T2.large(burst) respectively. Our impact is lesser than the figures suggested by redhat's advisory - https://access.redhat.com/articles/3307751

"Significant network traffic" tells me almost nothing, that's a totally subjective description and you don't describe your workload at all. A back-end MMO game infrastructure server I have some understanding of in terms of the kind of load it is handling.

It tells us his workload is more network bound than computationally intensive.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#275

Post by krylon »

sillysaurus3 wrote: post #168
forgotpw2018 wrote: post #132

Huge real world performance impact, they didn't say how much but it looks like close to 100%. I smell a class action lawsuit coming.

Hopefully not. Lawsuits are a fine way to stifle innovation. Imagine how hard it would be to push through any idea at Intel. No one saw this coming. Things happen. It's impossible to predict every contingency. They acted in good faith. Also, heh, users are funny: "all other games i have work fin by the way so there must be a problem whit fortnite."

> They acted in good faith.

They were aware of the problem for a couple of years: https://twitter.com/TheSimha/status/949361495468642304

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#276

Post by krylon »

pitaj wrote: post #137
forgotpw2018 wrote: post #132

Huge real world performance impact, they didn't say how much but it looks like close to 100%. I smell a class action lawsuit coming.

Why? This wasn't gross incompetence in processor design, this kind of attack is completely new. I don't see how a class action could apply here, by IANAL

> this kind of attack is completely new.

Not really: http://www.daemonology.net/hyperthreading-considered-harmful...

That was in 2005. It might be an over-generalization, but one could say that Intel made that same mistake (at least) twice.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#277

Post by lucb1e »

simik wrote: post #269
jitbit wrote: post #263

Thought I'd share our graph too: https://www.jitbit.com/alexblog/270-cpu-usage-stats-after-pa... Seeing almost 40% increase in CPU load

Do you mean 40 percentage points? In other words, tripling?

Looking at the graph: probably. It was already increasing (from 16-19%, one outlier to 27%, first 8 datapoints) to around 25-30% (the last 6 before the big jump). Then a big jump to 62-85% (N=20) with most around 75% (N≃13). The last bit is around 60-72% (N=12) and the very last datapoint is below 40% again but that is half outside the graph (cherry-picking datapoints?).

So to summarize, from 25% to 65%, or 2.6× the original.

Note that this is all from eyeballing that graph and estimating what percentage a datapoint is at, since the scale has an interval of 20 percent points and no minor grid or anything.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#278

Post by Groxx »

fermienrico wrote: post #174
erdbeerkuchen wrote: post #111

The image didn't show up for me, here is a direct link for anyone with the same problem: https://lh6.googleusercontent.com/MwzsHRXQLVbmJ3pusNuGwn0ZQV...

Can someone explain what 1, 2 and 3 series in the plot means? What are we looking at? Why did one of them all of a sudden spiked?

They patched a single host.

----------

Text of the forum post:

Attention Fortnite community,

We wanted to provide a bit more context for the most recent login issues and service instability. All of our cloud services are affected by updates required to mitigate the Meltdown vulnerability. We heavily rely on cloud services to run our back-end and we may experience further service issues due to ongoing updates.

Here is a link to an article[1] which describes the issue in depth.

The following chart shows the significant impact on CPU usage of one of our back-end services after a host was patched to address the Meltdown vulnerability.

[the image]

Unexpected issues may occur with our services over the next week as the cloud services we use are updated. We are working with our cloud service providers to prevent further issues and will do everything we can to mitigate and resolve any issues that arise as quickly as possible. Thank you all for understanding. Follow our twitter @FortniteGame for any future updates regarding this issue.

Epic suggests following security best practices by always staying up to date with latest patches. General Recommendations for Computer Security[2]

We will continue to update this thread with similar information as it comes to us.

[1]: https://spectreattack.com/

[2]: https://www.howtogeek.com/173478/10-important-computer-secur...

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#279

Post by Groxx »

nudgeee wrote: post #170
forgotpw2018 wrote: post #132

Huge real world performance impact, they didn't say how much but it looks like close to 100%. I smell a class action lawsuit coming.

I think CPU designers are getting too much flak for this. The side effects of speculative execution has been around ever since their introduction. A compounding problem is how abstract software has become, it has made assumptions on the underlying execution model without a deeper understanding of its sublteties (sandboxing, containers, etc trying to guarantee process isolation). It’s the Law of Leaky Abstractions [0]…

How does software abstraction impact this? (aside from the on-chip software/"software" doing the branch prediction)

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#280

Post by pixl97 »

merb wrote: post #271
msbarnett wrote: post #259

Earlier quoted context omitted.

Do you also run all processes on your server as root? If you’re not going to apply patches that close privilege escalation bugs, you might as well. You’re putting all your security eggs in one basket.

Well a lot of processes run as Root anyway. Also if possible i run as few processes as possible

>Well a lot of processes run as Root anyway.

Then you didn't listen to DJB like 15 years ago. Moreso, your processes shouldn't be running under the same username. Principle of least user authorization people.

Post reply on HN