There are some tools that restrict access with varying levels of granularity, but if you just want to restrict access on a per-repo-per-sshkey basis, one of my projects is a simple shell script that does just that:
https://github.com/cbdevnet/fugit
It originally came to be because I've found gitolite too big to maintain for simply sharing some repositories with a few other people. It has since served me well and is used in some business applications, too.