I have seen many outsourcing projects with my coachees and with founders, I've managed my own and cancelled several when I was called in to fix them.
The key to this kind of work is to understand:
The agency is not your buddy, they have very different goals than you have. Too often do I seen people who have nice chats with the agency over a coffee. They are not your friends.
You need to write the contract to align the incentives of the agency as much with yours as possible. For example: I see hourly billing, and bug fixing counting as billed hours. How has the agency an incentive to keep bugs low if it makes them more money? Agency has low retention, new people are slower, slower means more money for the agency. How has the agency an incentive to keep people on the project?
[Edit] You might think this is obvious, but I have seen unaligned incentives in mostly every outsourcing project I've looked into and was asked to fix. Tip: Do not take the developers they give you/have on the project. Interview all of them and reject the bad ones. As a new customer, they will not give you the best but those available (currently not on project/rejected by other clients)