I'm not talking about T-shirts here, which are usually extremely price sensitive, but high-street brands that sell a shirt for $30 or a pair of trousers/pants for $50-100.
A change in cost for suppliers of these products doesn't move business from one country to another. And the margin a supplier makes varies greatly.
A lot of products sourced in high-street brands comes from a variety of the same global suppliers (some of which sub-contract, some do in-house, and some sub-sub contract). One month from one supplier, and another month from another (varying by continent, country, to in-country location), for a pretty much identical product. One supplier may make 10% margin, and another 25%. It is of concern to the purchaser that a supplier doesn't make much of a margin, as they're feared as being pushed under, but puzzlement that these suppliers stay in business for decades. What is more important is turn-around time. A high-street brand needs a design idea to go from cat-walk to hanger in weeks, not months. That means logistics have to be extremely tight - design sent (electronically), prototypes mocked and air-mailed back, BOM sourced and often imported (at least in part), and the super-price sensitive suppliers just can't deliver here, whatever the margin, let alone the Q+A. A day turnaround and assurance of delivery time can make a difference more important than 5-15% on the quote.
When it comes to cheap textile production, or production of the lowest quality product then yes, price sensitivity is important. But when it comes to delivering mid-level textile products logistics trump jumping to the lowest margin location (where lack of logistical facilities, bribes, un-predictable delays).
I agree things can often improve, but minimum wage has little to do with it. A functional port and road to a factory are far more important in being competitive than how much a worker costs (except for the cheapest of cheapest textile products).