Live data from Hacker News

Fixing stutters in Papers Please on Linux

blog.jhm.dev

161–170 of 202 posts

Re: Fixing stutters in Papers Please on Linux

#161
post #26
post #9

Earlier quoted context omitted.

It is appropriate to use your main thread for your OS interaction - polling fds, talking to display server, whatever I/O you need, etc. An open/close call should never take this long, and you should never need to make a large amount of them in sequence after startup. What should not be on your main thread is any long blocking compute, which is why rendering/game logic often goes to another thread - although simple ga…

> An open/close call should never take this long > What should not be on your main thread is any long blocking compute Isn't that contradicting yourself? I'm pretty sure open() can block.

> Isn't that contradicting yourself? I'm pretty sure open() can block.

No, blocking compute would be you doing something for a long period of time.

"open/close can block" means very little. You only need a mitigation if it ends up blocking long enough to be a problem in reasonable setups.

You do that have to care about what happens if someone runs the code on an intentionally terrible/horribly slow but technically in spec toy filesystem. No need to prematurely optimize for this scenario.

And especially with devfs you cannot just listen to POSIX and must know what the kernel is providing you - knowing how long operations take on such fds is normal design input.

Re: Fixing stutters in Papers Please on Linux

#162

Earlier quoted context omitted.

Linux has absolutely come very far, don't get me wrong! I'm also mostly a casual gamer, and only have my "dedicated gaming PC" because it's 7 year old hardware I've replaced with a dedicated "workstation" I bought after getting a job and saving some money. On all my other hardware, I just run Linux, and I pretty much do the same as you -- most of my games work fine on Linux, a surprising number natively! Linus brough…

Where does this sentiment come from that Linux has come very far when it comes to gaming? When Doom 3 released in 2004 I had to use a hex editor to hand patch the executable to get sound working. Luckily someone did what OP did and posted the instructions on a forum. I've used Linux/Unix for 20+ years but I wouldn't recommend it for gaming unless you enjoy debugging Linux software and want to do more of it. Frankly,…

>Where does this sentiment come from that Linux has come very far when it comes to gaming?

Look at what you could run in Wine in 2004 and what you had to do to get it working and what you can do now in Proton. It might technically not be "native Linux" but I couldn't care less as long as it works.

>I've used Linux/Unix for 20+ years but I wouldn't recommend it for gaming unless you enjoy debugging Linux software and want to do more of it.

I game a little bit on Linux and have never touched a hex editor to do it. Right now I only have one game where I have to manually download a dll, all the rest I want to play works flawlessly. I only have to enable Proton in Steam and that's it. Yes, they're mostly older games and I still certainly wouldn't recommend Linux for gaming, but there is definitely progress.

Re: Fixing stutters in Papers Please on Linux

#163

Earlier quoted context omitted.

> “It’s a blocking syscall so if it takes 1s to close a file, that’s technically not a bug” is correct, but is any player of “Papers, Please” going to be sympathetic to that explanation? Probably not; they’ll think “Linux is slow,” “Linux is buggy,” “why can’t Linux run basic applications correctly that I have no problem running on Windows or OS X?,” etc. I don’t agree with this logic. Windows and macOS system calls…

> Linux already operates by this principle. We are examining a situation where best effort was not good enough to hide poor application design. Linux has this principle as a goal, but it's probably not checked often. I would say this code fails the principle, independent of particular application problems.

> I would say this code fails the principle, independent of particular application problems.

For every system call you determine satisfies that principle, I could come up with a application level algorithm that is broken because of it. The principle is aspirational, Linux does a best effort as all Unix systems do not because Linux is buggy but because it can never be 100% given the spec. The core issue here was not close() taking 100ms or whatever it took, the core issue was doing unbounded work on the main drawing thread, which has strict timing requirements.

Re: Fixing stutters in Papers Please on Linux

#165

This reminds me of the recent Linus Tech Tips series on gaming on Linux[1]. Their conclusion is that although many games work out of the box (although usually not at launch), Linux is not ready for mainstream gamers. Not many people would have the expertise or the interest to troubleshoot the problem as OP did. [1] https://www.youtube.com/watch?v=Rlg4K16ujFw

Then again I play Papers Please through Steam and have no problems, so maybe they're using the 32 bit version.

I agree that Linux use in general requires troubleshooting skill. We shouldn't assume there will never be any issues worth troubleshooting and recommend Linux to novices as a Microsoft killer. We should instead assume problems will happen and therefore a robust restore process is much more valuable to a novice Linux user.

This is why I believe in btrfs that Fedora uses. Imagine having the powerful restore options many Windows computers ship with. Just press a key, go into a menu, select a point in time recovery, restore.

But that said, what I really wanted to say was that I play exclusively on Linux now thanks to Proton and it's amazing. I can play big titles like Witcher 3, RDR2 and more, but I mostly play smaller titles like Oxygen not included, Rimworld and Ostriv.

Re: Fixing stutters in Papers Please on Linux

#166
post #7

The issue has been identified before, but seems like it stalled: https://gitlab.freedesktop.org/libinput/libinput/-/issues/50... , https://patchwork.kernel.org/project/linux-input/patch/20201... .

What the proposed patch does is delay a specific latent operation to an asynchronous context so that close() doesn’t block on that operation (which is freeing some memory). The proposed patch isn’t a comprehensive fix, it admits there are still other sources of relatively high close() latency. So that got me thinking, there is no way to fix this “bug” because there is no specification on how long close() should take…

You should use io_uring to open and close files asynchronously, instead of open/close.

Re: Fixing stutters in Papers Please on Linux

#167
post #7

The issue has been identified before, but seems like it stalled: https://gitlab.freedesktop.org/libinput/libinput/-/issues/50... , https://patchwork.kernel.org/project/linux-input/patch/20201... .

What the proposed patch does is delay a specific latent operation to an asynchronous context so that close() doesn’t block on that operation (which is freeing some memory). The proposed patch isn’t a comprehensive fix, it admits there are still other sources of relatively high close() latency. So that got me thinking, there is no way to fix this “bug” because there is no specification on how long close() should take…

Patch author here.

It is important to not conflate POSIX requirements with expected behavior, especially for device files which require very specific knowledge of their implementation to use (DRM ioctl's and resources anyone?).

You might think that as a well-behaved game should not be opening/closing evdev fds during gameplay at all, this is clearly just an application bug. However, games are not the main user of evdev devices, your display server is! This bug causes input device closure during session switching (e.g. VT switching) to take abnormally long - on the machine I discovered the bug on, it ends up adding over a second to the session switch time, significantly impacting responsiveness.

This is absolutely a kernel bug. I did not push the patch further as I had other priorities, and testing this kind of patch is quite time-consuming when it only reproduces in a measurable way on single physical machine. Other machines end up with a much shorter synchronize_rcu wait and often have many fewer input devices, explaining why the issue was not discovered/fixed earlier.

call_rcu is intended to be used wherever you do not want the writer to block, while alternative fixes involve synchronize_rcu_expedited (very fast but expensive), identifying if the long synchronize_rcu wait is itself a bug that could be fixed (might be correct), or possibly refactoring evdev (which is quite a simple device file).

As for putting things in threads, I would consider it a huge hack to move open/close. Threads are not and will never be mandatory to have great responsiveness.

Re: Fixing stutters in Papers Please on Linux

#168

Earlier quoted context omitted.

What the proposed patch does is delay a specific latent operation to an asynchronous context so that close() doesn’t block on that operation (which is freeing some memory). The proposed patch isn’t a comprehensive fix, it admits there are still other sources of relatively high close() latency. So that got me thinking, there is no way to fix this “bug” because there is no specification on how long close() should take…

You should use io_uring to open and close files asynchronously, instead of open/close.

You would not use io_uring for things like that. Not only will you still use regular file operations on device files for various reasons, should you chose to use io_uring you would want it to run your entire eventloop and all you I/O rather than single operations here and there. Otherwise it just adds complexity with no benefit.

Re: Fixing stutters in Papers Please on Linux

#169
post #26

Earlier quoted context omitted.

> An open/close call should never take this long > What should not be on your main thread is any long blocking compute Isn't that contradicting yourself? I'm pretty sure open() can block.

> Isn't that contradicting yourself? I'm pretty sure open() can block. No, blocking compute would be you doing something for a long period of time. "open/close can block" means very little. You only need a mitigation if it ends up blocking long enough to be a problem in reasonable setups. You do that have to care about what happens if someone runs the code on an intentionally terrible/horribly slow but technically in…

"You only need a mitigation if it ends up blocking long enough to be a problem in reasonable setups."

That criteria is established here - the OP is about an issue affecting paying end-users! Premature optimisation is not relevant - the software is failing.

It is unsafe to make fair-weather assumptions about customer systems.

Consider a common software failure: where the user is saving data to a SMB or NFS partition, and then there is a loss of connectivity to that file-server, and the developer has done whatever I/O they need on the main thread. This causes data loss.

You /should/ be able to assume (1) reliably fast return from core async-coordination syscalls (e.g. select, poll), and (2) that you will not suffer process or thread starvation caused by someone else. Respecting those constraints, it is good practice to isolate sync calls to a non-main thread in order to catch when they are not returning. This robustly covers both common scenarios (like the missing-filesystem) and obscure scenarios like the one in blog post.

Re: Fixing stutters in Papers Please on Linux

#170

Earlier quoted context omitted.

> Isn't that contradicting yourself? I'm pretty sure open() can block. No, blocking compute would be you doing something for a long period of time. "open/close can block" means very little. You only need a mitigation if it ends up blocking long enough to be a problem in reasonable setups. You do that have to care about what happens if someone runs the code on an intentionally terrible/horribly slow but technically in…

"You only need a mitigation if it ends up blocking long enough to be a problem in reasonable setups." That criteria is established here - the OP is about an issue affecting paying end-users! Premature optimisation is not relevant - the software is failing. It is unsafe to make fair-weather assumptions about customer systems. Consider a common software failure: where the user is saving data to a SMB or NFS partition,…

> That criteria is established here - the OP is about an issue affecting paying end-users! Premature optimisation is not relevant - the software is failing.

The software is not failing, it is experiencing performance degradation: a 500ms pause whenever excessive and entirely unnecessary work is done.

The solution is to not do the work. Moving open/close to a different thread is premature optimization, as no necessary call has been profiled to cause issues on any known system.

Performance 101, do not do things that you do not need done. Even if you want live hotplug and input reconfiguration during gameplay without touching any menus, you only open a device when it appears.

> It is unsafe to make fair-weather assumptions about customer systems.

It is more pointless to optimize for worst-case scenarios - experiencing performance degradation on a faulty system is fine.

All applications have a minimum performance requirement to remain responsive, which is equivalent to always making a certain degree of "fair-weather assumptions".

> Consider a common software failure: where the user is saving data to a SMB or NFS partition, and then there is a loss of connectivity to that file-server, and the developer has done whatever I/O they need on the main thread. This causes data loss.

This is non-sequitur - doing something on the main thread does not cause data-loss. Losing connectivity causes data-loss.

Heck, as main-thread I/O with an event loop implies non-blocking fds, you would not even be blocked by this unless you call fsync(2) to explicitly block until flush is complete, which a normal application does not need to do. The other (horrible) side-effects of network filesystems will cause problems for your application no matter how you interact with the fd.

Furthermore, you cannot use device files without reasoning about their exact implementation. They are not basic files.

> Respecting those constraints, it is good practice to isolate sync calls to a non-main thread in order to catch when they are not returning.

That's a hack, and is not even a solution. What are you going to do when they don't return? Accumulate dead threads and inconsistent shared application state?

Post reply on HN