It'd make a lot of sense to
sandbox AF_ALG, then, wouldn't it? At least for userspace-driven invocations. Let the kernel keep its current code-path for kernel-driven invocations, but have the same code unit files
also build some other sandboxed form, to be invoked by the crypto-accelerating syscalls.
If these syscalls are used by userspace as rarely as you say, the performance impact of this kind of sandboxing wouldn't matter much. And maybe there could be a KCONFIG/boot flag to switch back to using the un-sandboxed code path for userspace invocations too, for enterprises stuck with old software who really care.
---
My own thought process on how this could work below (but I'm not a kernel contributor, so you can probably immediately picture a design better than I can):
The naive way to do this, would be for the kernel build process to emit a separate AF_ALG userland IPC server as an additional build artifact; to get distros to package this IPC server as a component package of kernel packages; and to set up the sandboxed AF_ALG "kernel bridge" so that it proxies calls through to this IPC server if it exists, and errors out otherwise. (Basically like kfuse, except in this case the only "FUSE servers" are first-party.)
But that's a bit painful, organizationally. Puts a lot of work on the distro maintainers' shoulders, that they might just not bother doing. Prone to error. I think there are better alternatives.
1. Maybe the userland syscalls that rely on AF_ALG could instead ground out inside the kernel in a copy of AF_ALG that's been compiled to eBPF? Then that eBPF bytecode could just be embedded into the kernel.
2. Maybe the Linux kernel could consider a facility that would enable it to act as a hybrid microkernel (similar to macOS's XNU) — with arbitrary static sections of the kernel image/kernel modules [or perhaps standalone static ELF binaries embedded within kernel/kmod .data sections] being spawned not as supervisor-mode kthreads doing their own autonomous thing, but rather as unprivileged user-mode kernel threads, running as IPC-servers for the rest of the kernel to talk to?
- The rest of the kernel could talk to these "userspace kthreads" via some nonblocking IPC mechanism; but this mechanism wouldn't need to be exposed to userland the way macOS's XPC is; it could be kernel-to-kernel only (where these "userspace kthreads", despite being in userspace, are still fundamentally kernel threads, and so get to participate in it.)
- Also, these "userspace kthreads", when they're the active scheduled task, would have the kernel image's read-only sections [or their binary's sections, from within the kernel's .data section] mapped into their address space, since that's the binary they're executing against. But they wouldn't inherit [or the spawning mechanism would actively prune from their task struct] the rest of the kernel's mappings. So they'd have to either use the IPC mechanism, or use regular syscalls, to do anything with the kernel, just like any userspace task.)