Mainframe designers had this problem under control by 1970. Mainframes had, and have, "channels". A channel is part of the processor architecture. It takes commands, sends them to a peripheral, and manages the data transfer in both directions. Channels have some privileged functions through which the OS tells them where the data is supposed to go in memory. The architecture of channels is well defined, and peripherals are built to talk to channels. The CPU has I/O instructions to control channels in a well defined way.
The peripheral never has access to main memory. There is no peripheral-controlled "direct memory access" (DMA). So it's possible to give control of a channel to a userland program without a memory security risk.
Minicomputers of the 1970s had low transistor counts and slow CPUs. So peripherals were usually put directly on the memory bus, with full access to memory. I/O operations were performed by storing into memory addresses, which caused bus transactions detected by the peripheral device. There were no CPU I/O instructions.
Microprocessors copied the minicomputer model. IBM's people knew this was a bad idea, and in the IBM PS/2, they introduced the "microchannel". Peripheral vendors, facing a new architecture that required more transistors, screamed. IBM backed down and went back to bus-oriented peripherals.
That model persists today, even though the few thousand transistors required for a channel controller are nothing today. Even though most modern CPUs have I/O channel like machinery, it's exposed to the program as registers the program stores into and memory accesses by the peripheral device.
So there's no standardization on how to talk to devices at the hardware level. Some CPUs have protection systems, an "I/O MMU", and there have been various channel-like interfaces, especially from Intel, but they have never caught on.
Instead, we mostly have heavy kernel mediation between the hardware and the user program. And way too many "drivers". This has become a problem with "solid state disk", which is really a random access memory device that doesn't write very fast. Mostly, it's used to emulate rotating disks.
Samsung makes a key/value store device which uses SSD-type memory devices but manages the key/value store itself. But you need a kernel between the device and the user program. You can't just open a channel to it and let the user program access it.