Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CPU Usage Differences After Applying Meltdown Patch at Epic Games

epicgames.com

Post reply on HN
Print view View original

131–140 of 296 posts

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#131

Post by forgotpw2018 »

_wmd wrote: post #47
contrarian_ wrote: post #21

Pretty much what I predicted here: https://news.ycombinator.com/item?id=16054674 > Sounds like servers handling lots of small UDP packets would be hit pretty hard.

There's potential for a little rearchitecting to help, at least in the case of UDP: NAME sendmmsg - send multiple messages on a socket SYNOPSIS #define _GNU_SOURCE /* See feature_test_macros(7) */ #include int sendmmsg(int sockfd, struct mmsghdr *msgvec, unsigned int vlen, unsigned int flags);

You could use it, but you would probably have to reachitect the server to use green threads to avoid the overhead. Frequent syscall sends are done with games to keep the latency as low as possible. Any batching would increase delay

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#133

Post by Abishek_Muthian »

bartread wrote: post #127
Abishek_Muthian wrote: post #60

If it's helpful, Our Node.js, MongoDB, Python servers all with significant network traffic didn't have any measurable impact after KPTI patches on Amazon Linux on T2.medium(burst), M4.large, T2.large(burst) respectively. Our impact is lesser than the figures suggested by redhat's advisory - https://access.redhat.com/articles/3307751

Would you be able to post some ballpark figures, please?

Actually, there is not any measurable difference. Our architecture is completely state-less and when compared with equivalent load there's no difference at all. I guess, default network throughput bottle-neck itself is higher than the random memory cache bottle-neck in my case. There was no impact on the latency either.

Fearing automatic update by AWS resulting in performance issue, we rushed to update all our serves and gladly there weren't any performance impact.

I don't think not every one has been fortunate, there's still lot of variables reg performance impact of KPTI. This user here using PHP server mentioning 50% performance - https://twitter.com/timgostony/status/948682862844248065 and of-course OP issue is being covered here.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#134

Post by mrweasel »

Darthy wrote: post #71

The Meltdown attack requires an attacker to have a piece of code executed on your server. Epic's servers are used for login, where people send you data, and for game logic, where people also just send you data like "player x moved his avatar here, player y shoots etc". If all the server does is execute the code which Epic wrote themselves and already trust, why would it need to apply the Meltdown patch?

Perhaps they don't, but eventually the patches become part of the mainline kernels, and then they will run with the patch applied.

Regardless of what some of my customers believe patches are rarely hand picked and then applied. All security patches are applied, always, kernel patches even more so. They are rolled out with the tools provided by the operating system, be it Windows, Redhat, Ubuntu, OpenBSD, Solaris or whatever, and they don't make assumptions about your use case, they just apply all security patches.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#135

Post by LV-426 »

hungerstrike wrote: post #116
LV-426 wrote: post #107

Earlier quoted context omitted.

> which unfortunately seems to be getting rarer and rarer on this forum. If this isn't unnecessarily "snarky" itself, then what is it? snarky : (of a person, words, or a mood) sharply critical; cutting; snide.

That wasn't snarky because it was only mildly critical. Since it was mild, it also wasn't cutting. Lastly, it was not derogatory or mocking in an indirect way and so it was not snide either.

Replying here, since the other post I replied to disappeared:

Despite the avalanche of downvotes burying my posts, not even one person can actually answer the question I asked:

The poster could have simply stopped typing after "Thank you for the explanation". He chose to unnecessarily add something more.

So, for everyone who disagrees with me: if what he said wasn't itself "snarky", what was it?

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#136

Post by toomuchtodo »

drawnwren wrote: post #76
Darthy wrote: post #71

The Meltdown attack requires an attacker to have a piece of code executed on your server. Epic's servers are used for login, where people send you data, and for game logic, where people also just send you data like "player x moved his avatar here, player y shoots etc". If all the server does is execute the code which Epic wrote themselves and already trust, why would it need to apply the Meltdown patch?

This is a horrible approach to security. If you only secure against attacks you expect, you're gonna have a bad time.

Security is about risk mitigation. You cannot derisk entirely, so you make tradeoffs. Without knowing all of the parameters, it’s disingenuous to say it’s a horrible approach to security.

The most secure computer is powered off, enclosed in concrete 6 feet below the surface of the earth. It is not very useful though.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#137

Post by pitaj »

forgotpw2018 wrote: post #132

Huge real world performance impact, they didn't say how much but it looks like close to 100%. I smell a class action lawsuit coming.

Why? This wasn't gross incompetence in processor design, this kind of attack is completely new.

I don't see how a class action could apply here, by IANAL

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#138

Post by jbg_ »

dogma1138 wrote: post #115
tgb wrote: post #62

Earlier quoted context omitted.

Since they describe this as log-in issues, should we expect that to be a server using lots of UDP? Or do you think that the log-in service is hemmed in by the load on the game servers?

No, login in Fortnite is over 443 and it uses normal HTTPS. It also uses HTTPs to load a lot of other data such as server data, friends list, chat etc. Unreal Engine comes with a version of chromium built in which is used for many in game things like social tabs, news, and in game purchases these all work over HTTP/S. Game data is sent over 5222 TCP.

Out of interest, do you know if the game data uses XMPP or if it just uses the same port?

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#139

Post by tetha »

bhouston wrote: post #40
aceoflolo wrote: post #25

Earlier quoted context omitted.

For a CPU-bound problem it mostly is as simple as that, yes

Only if the algorithm can be split across multiple machines easily. Sometimes people assume that you can use local shared memory or something between threads in order to synchronize state. You figure out how many individuals can be on a server at once and then ensure that you can handle that load on a specific machine. I've seen this type of stuff for game state before because they need to keep everyone in a specific…

Yup. When I was in a team working on distributed game servers, this forced us to shard games instead of distributing individual games.

Terminology wise, if a game session was fully distributed across multiple instances, each server could accept traffic for this game session. Think elasticsearch - each node can answer searches for any index in the cluster.

If you shard your games, you just put all games with an even ID on box 1 and all games with an odd ID on box 2. If box 1 dies, all games on that box disappear. And then you usually end up with a lobby server as an initial connection and redirect to the actual game server in the background.

This is a very simple architecture. It's easy to develop for this architecture, because you don't need to worry about complex clustering issues - exactly 1 client talks to exactly 1 server and it doesn't matter if there's 50 other servers answering other clients.

This is also very nice to scale. Most server side code for games tends to have very predictable resource consumption, because it's running a pretty predictable simulation loop on a pretty predictable and bounded data set. Especially from this perspective, I can see why the Epic guys are bugged. It's not pretty to put a factor of 2 into those calculations.

Re: CPU Usage Differences After Applying Meltdown Patch at Epic Games

#140

Post by jameshart »

progval wrote: post #82
Darthy wrote: post #71

The Meltdown attack requires an attacker to have a piece of code executed on your server. Epic's servers are used for login, where people send you data, and for game logic, where people also just send you data like "player x moved his avatar here, player y shoots etc". If all the server does is execute the code which Epic wrote themselves and already trust, why would it need to apply the Meltdown patch?

It makes sense if you don't want a remote code execution in one of your app to provide a full root shell.

If you're talking about a game server like this, though, where you don't have sandboxes running client-originated code, just processes running your own software, any remote code execution vulnerability is already going to be in a position to do incredible damage. There are almost certainly going to be other privilege escalations available to you once you've buffer-overflowed or whatever to dupe the server into running code you control in process - and honestly, once you own the main process on a server like this you can probably do enough damage from in there without even needing to escalate.

I think the frightening thing about meltdown/spectre is the ability for code to escape otherwise solid sandboxes - including VMs, as well as javascript runtimes. It's a change in the threat landscape for systems which expect to run remote code but believe they can do so safely.

Post reply on HN