Maybe I'm old fashioned, but I've never put clients on retainer for more than a fraction of the typical work-hours per month... which I only really know after I'm well past their first project. The point of a retainer is to square off enough time and resources to be on call when the next project comes up and the one after that. I don't think it's at all the appropriate way to handle a resource crunch on a one-off project.
I do shy away from design-only work for one time clients, although ten years ago that was typical of my business. From an art director's perspective there are certain red flags on both sides of this. I would never present that many logo options to a client, or engage in constant back and forth over the options until we had internally narrowed it down to a maximum of 3. Clients are not designers and presenting them with too many choose-your-own questions tends to lead them to micromanaging - what I call client vanity logos - and inevitably (though paradoxically) they are less happy with the results than if they are presented with a few solid choices from the get-go and dissuaded from injecting too much of their own design aesthetic. The reason is that they come to believe they could have done it better themselves. Whereas if it is done for them professionally, they will comfort themselves knowing that this is what the professionals think and they got the pro opinions they paid for. This is something I learned very early on, when I started at an ad agency at 15. (We also learned that two of the three you present should be slightly flawed, to drive the customer to the design strategy we had already settled on but give them the illusion of choice. I don't really waste time with that anymore, but it's still a tool in the kit).
I'm not blaming the OP for any of this, or saying they're especially picky. In my experience it really comes down to the quality of work and quality of advice they're getting from an agency, and an agency should know how to deal with it.
Another red flag is that each portion of the job should have been estimated individually beforehand. That's really essential to preventing time overflows and also to dissuade micromanagement. Instead, it sounds to me like this entered a loop focused on the logo which sucked up more time than anyone expected, and they allowed that to be a driver. They probably no longer liked the project, and as a result, the final product lacked coherence and vision.