I believe rootless containers require Linux user namespaces which have historically been the source of many vulnerabilities: https://news.ycombinator.com/item?id=43517734 I'm conflicted about whether or not it's better to run a root daemon that can launch unprivileged non-root containers or run rootless containers launched by a non-root user. Anyone have thoughts or more definitive resources they could point to that…
Running podman with SELinux enforcing (the default) and with "--security-opt=no-new-privileges" combined with running applications as non-root inside their containers should further reduce the security risk. You could also disable unprivileged user namespaces inside the containers if you want, which would mean that exploiting unprivileged user namespaces would first require arbitrary code execution on the host.