It seems like using ssh-agent is quite insecure.

What do you use instead?