Implement mechanism to wait on any of several futexes • Hacker News Board

Re: Implement mechanism to wait on any of several futexes

#141

Post by [deleted] » Thu, Aug 01, 2019, 6:37 AM UTC

dingu wrote: post #135

jdsully wrote: post #132

Earlier quoted context omitted.

You may be surprised to learn that some windows handles are actually just addresses with no kernel state too. A great example is an HMODULE. In fact windows actually reserves a block of user address space that will never be allocated to disambiguate memory vs non memory handles. Its really a pitty these two great systems refuse to learn from each other.

Ah you edited your response? An HMODULE.. That is weak dude. Calling and HMODULE a HANDLE is cheating... by that definition a UINT is a HANDLE since you can cast it. Can you use HMODULEs orthogonally to common APIs that use Handles? That's what really matters. Just because something is 4 or 8 bytes wide and you call it the same thing isn't interesting. Like can you pass HMODULE to WaitForSingleObject? Oh Ok, I guess…

[deleted]

Re: Implement mechanism to wait on any of several futexes

#142

Post by jdsully » Thu, Aug 01, 2019, 6:48 AM UTC

monocasa wrote: post #134

jdsully wrote: post #132

Earlier quoted context omitted.

You may be surprised to learn that some windows handles are actually just addresses with no kernel state too. A great example is an HMODULE. In fact windows actually reserves a block of user address space that will never be allocated to disambiguate memory vs non memory handles. Its really a pitty these two great systems refuse to learn from each other.

The handle is a _kernel_ address... backed by a validly constructed kernel object. EDIT: And since you dirty edited, can you point to an example of HMODULE or any other dataless HANDLE being used on the syscall interface?

Handles are not kernel addresses in NT and windows will not intentionally leak those to user space. Handle accesses are always through a layer of indirection in the object manager.

You will find that windows never makes guarantees of what handles actually are - in case it wants to change them.

In windows NT most GDI handles are user mode and not kernel objects. Other objects may or may not be kernel based depending on the version and whims of the implementor.

Re: Implement mechanism to wait on any of several futexes

#143

Post by monocasa » Thu, Aug 01, 2019, 7:14 AM UTC

jdsully wrote: post #142

monocasa wrote: post #134

Earlier quoted context omitted.

The handle is a _kernel_ address... backed by a validly constructed kernel object. EDIT: And since you dirty edited, can you point to an example of HMODULE or any other dataless HANDLE being used on the syscall interface?

Handles are not kernel addresses in NT and windows will not intentionally leak those to user space. Handle accesses are always through a layer of indirection in the object manager. You will find that windows never makes guarantees of what handles actually are - in case it wants to change them. In windows NT most GDI handles are user mode and not kernel objects. Other objects may or may not be kernel based depending o…

I mean, they're very clearly offsets into the HANDLE table kernel space.

And can you WFMO on user space only HANDLEs?

Re: Implement mechanism to wait on any of several futexes

#144

Post by ithkuil » Thu, Aug 01, 2019, 7:49 AM UTC

dingu wrote: post #122

jdsully wrote: post #120

Earlier quoted context omitted.

While the Unix philosophy is that everything is a file - that is not the case with Linux. Futexs being a good example. There was an attempt to integrate them but it was done incorrectly with inherent race conditions and abandoned. The VMS and NT philosophies of everything being an object are a bit more general and easier to follow in practice.

what are you on about? I just gave you the specific examples: timerfd, eventfd, signalfd, inotify... these are all epollable fds in Linux? Futex is a special case because futexes themselves are quite special. There is no userspace equivalent to them in Windows anyway as has been mentioned. Windows Events are similar to what is provided by eventfd, but not as featureful.

Probably the core of the issue here is that VMS and derivatives tried to hard to fit everything into a generic handle/fd interface while UNIX historically came with a smaller core with many important API surfaces (like signals or timers) only relatively recently getting absorbed under the unified handle/fd interface, giving the impression it's an afterthought.

It's also not something that works on all Unices (e.g. afaik macos has no timerfd). But in all fairness, that problem only exists because OS speciation, as it's biological counterpart, is not as clear cut a concept as one would desire.

Re: Implement mechanism to wait on any of several futexes

#145

Post by Sahhaese » Thu, Aug 01, 2019, 8:06 AM UTC

cesarb wrote: post #54

ori_b wrote: post #38

Earlier quoted context omitted.

The technical review wasn't technical. It was a human roleplaying as a code formatter. The technical content was entirely found in the comment about ABI compatiblity. I didn't see any discussion about tradeoffs, alternative approaches, or a survey of what other systems do for this kind of functionality, or detailed benchmark results. The tone was roughly what I'd want people to give me in a code review -- the only pr…

> The technical review wasn't technical. [...] The technical content was entirely found in the comment about ABI compatiblity. That review also had a comment about an implicit limit on the number of objects, which is caused by a limit on the amount of physically contiguous memory the kernel memory allocator can obtain at once, and a comment that the code being reviewed would allow for a large increase of the referenc…

This is my complaint about the styling issue, that the important technical notes get lost in the 'noise' of styling issues.

Re: Implement mechanism to wait on any of several futexes

#146

Post by rbanffy » Thu, Aug 01, 2019, 9:37 AM UTC

kbenson wrote: post #73

rbanffy wrote: post #23

Earlier quoted context omitted.

I got that. I intended to go for "actually useful".

Source 2 has a linux port I believe, and it's an engine that other games could (do?) use to get easy linux support for a lot of their codebase. I think that counts for actually useful. Depending on what they mean by "massively-threaded", that might cover some other popular network applications that thread to handle requests, but I'm not sure how much work they would put into a linux specific solution if it complicate…

It occurred to me that a message passing OO language that mapped every active object (that's responding to a message) to a thread could benefit from it.

Re: Implement mechanism to wait on any of several futexes

#147

Post by Crinus » Thu, Aug 01, 2019, 9:42 AM UTC

LukeShu wrote: post #102

danappelxx wrote: post #87

Earlier quoted context omitted.

Oblivious question: is there any notion of granulated permissions, ie a ‘mouse’ user group that the game could add itself to? Seems like something like this shouldn’t be a deal-breaker.

There is an 'input' user group, which gives you access to raw mouse, keyboard, joystick, et c. under `/dev/input/`. (I'm sure you could configure udev to use more more granular user groups for different types of input devices, if you really wanted). Normal human users of a single-user system should be members of the 'input' group, so this "should" be a non-issue.

I have added support for reading the input device directly on my 3D game engine exactly so i can provide unfiltered mouse input, but at least on Debian (and i think on derivatives) the user isn't on the 'input' group by default. The engine shows a message if raw input is enabled it cannot open the device handle (something like "cannot open handle check if you have permissions for accessing - maybe you need to be in the input group?") and it falls back to regular X11 events (the feeling of which depend greatly on the current configuration, which input driver is used, etc - on my PC where i use udev and 1:1 mapping it feels fine, but others use libinput, which i think is the default nowadays, and mappings can be all over the place).

Re: Implement mechanism to wait on any of several futexes

#148

Post by Crinus » Thu, Aug 01, 2019, 9:45 AM UTC

k_sze wrote: post #104

LukeShu wrote: post #99

Earlier quoted context omitted.

Normal human users will/should almost always be members of the 'input' group.

Maybe being a member of the input group allows you to spy on other users? Just maybe. I don’t know Linux well enough.

It does but this is only in an environment where the same computer is accessible by more than one people at the same time with different keyboards and monitors. In practice this is an extremely rare case and in that case (and assuming the people involved cannot just trust each other for some reason) the permissions can be set to only allow access to a single device per user.

Re: Implement mechanism to wait on any of several futexes

#149

Post by Crinus » Thu, Aug 01, 2019, 9:47 AM UTC

Shorel wrote: post #71

gnufied wrote: post #46

Personally for me - the problematic part of gaming on Linux has been input(i.e mouse) latency and acceleration profile. I am not sure if this is just my experience but when using libinput on Fedora for example - the cursor movement is not exactly precise. This is not obvious when working but while gaming this is a deal breaker.

Games in Windows can get raw mouse input just by writing the code for it, and Linux games usually don't because that would require root permissions, to add the user to the input group, etc. It is a security issue and an X-Window design issue.

There is nothing in its design that prevents X windows to be made to send an event like WM_INPUT that games can handle in a similar way to how they handle WM_INPUT on Windows today.

Re: Implement mechanism to wait on any of several futexes

#150

Post by ot » Thu, Aug 01, 2019, 10:30 AM UTC

I wonder if they have considered implementing futexes in userland so they don't have to wait for the kernel implementing this.

WebKit has its own implementation (see the ParkingLot in https://webkit.org/blog/6161/locking-in-webkit/) which inspired an implementation in folly (https://github.com/facebook/folly/blob/master/folly/synchron...).

Surprisingly, implementing futexes in userland can have performance benefits wrt kernel futexes (because of better control over the fast path, possibly avoiding syscalls) and a richer interface (for example the value doesn't need to be 32 bit, just any atomic).