My approach was to remove completely docker and start using podman as rootless daemonless alternative. It also has a good solution to user-namespace mapping when I mount my repositories inside the sandbox.
I let the ai agent make changes but then I review them and push them from my host. As an alternative, you can create a ssh key pair on the host and load it into a dedicated ssh-agent, so you expose that agent socket to the agent container.
For API Tokens and similar credentials, I solved it by putting a credential broker between the podman agent container and the LLM provider. To avoid doing any internal firewalling in the container I run a proxy as a podman container that enforces the egress policy (only https on allowed domains); so no network capabilities are required, the proxy itself handles the filtering.
Because I had the idea of using the sandboxing framework to allow specific communication inside the network (for example for pentesting), it has also (apart from the proxy mode) a "routed"-mode implemented where a podman gateway is started; this does the network filtering without holding any kernel capabilities. nftables is configured using an ephemeral container with net_admin in the same network space (one-shot network initializer).
The framework orchestrates the different required networks and Podman containers.
I haven't tried the Docker solution yet, so I'll avoid comparing them.