Agencies charge like a very good contractor but use midrange to lousy contractors/employees to do most of the work, and bring the good ones in only in need. It’s selling the difference between the two.
It took him awhile but the devs he fed work to were solid engineers, almost FAANG tier but he paid them s*t or good for their locations. I've kept their names handy because poaching them would be dirt easy.
Knowing this as well as what you said, I avoid agencies now and simply assemble teams of individuals for this reason alone. They're easier to evaluate and you can let them go easily as opposed to a full service agency.
However, what annoys me the most about Upwork are agencies larping as solo guys. Anytime I evaluate a guy who has a solo freelancer account and then mentions he has guys he works with that bill through him, I cut them lose. This is easy to spot. I wish uowork would ban these guys.