Earlier quoted context omitted.
I don't think it's possible on Windows. It's trivial on Unix - just make the program setgid and change the folder permissions to only allow the group. This can be nested, though that requires that the relevant program be aware of the need to walk through several levels, though often a symlink can hide that. Note that when creating such a directory setup, `chown`ing away the user requires a privileged helper utility.…
Or use selinux/apparmor - those have supported app sandboxing without group tricks for a long time.
There's probably some mandatory mode but since it breaks all sorts of programs nobody can afford to use it.