You have a group of people who care deeply about performance, reliability, maintainability, and long-term health of the code base.Having a large, single, closed-source codebase is a mistake. You should have a service-oriented architecture for the necessarily private stuff. Anything infrastructural should eventually be planned for release into the open-source ecosystem.
That's because maintenance is utterly necessary, but maintenance requires good people, and talented people will maintain OSS, because it benefits their reputations and usually involves improvements to infrastructure they need, but closed-source maintenance always ends up being a ball of work thrown onto people with the least clout, who are either not very good or pissed off.
That's how you solve the SRE/SWE conflict you talked about.
Either that or you take the hedge fund strategy: pay people so goddamn well (hedonic adaptation says that's in relative/growth terms, so 25% over market in the first year, and 25% growth each year) that they'll be happy to do whatever you ask of them. But for an average $120k developer that has you over $1M/year after 10 years, and most tech companies aren't willing to pay that.
You have a brilliant developer who is terrible at working with other people and doesn't particularly want to deal with them. In an open-allocation world, he will end up shunned; nobody will want to work with him. In a managed org, it's often possible for his manager to pick off a certain really difficult technical challenge, have him go off and find a solution, and then interface with the rest of the group via some technical API.
It's actually the norm for the best team size, in software, to be one. There are exceptions, but not many.
In an open-allocation world, either he finds a project where he can contribute on his own, or he asks for suggestions (from someone like this manager). The difference is that he gets to pick (although if he doesn't, he might have to be let go). He's not reliant on one company-assigned manager (who might abuse the power) to be his interface.
In an open-allocation world, nobody will want to work with him, which means that he won't be a tech lead for very long, and eventually he'll probably get frustrated and leave.
I see no problem with that outcome.
If he's not showing leadership, and no one follows him, that's what should happen. If it bugs him, he can change his attitude or quit.
How many managers, in a closed allocation system, are actually going to know that a tech lead is taking the best work? This requires people on the team going to the manager (behind the tech lead's back) and most people would be too afraid to do that, knowing what will happen if the manager names them to the tech lead.
You have a project where consistency is the whole benefit of the project. For example, you're doing a company-wide visual redesign to make the company's product offerings more consistent for users, or you're introducing a language styleguide, or you're defining standard APIs for communication between components. These efforts typically never succeed unless they have top-down support from a strong leader.
Ok, this one is a really good point. You're right that this is something that requires top-down delivery of vision. It's hard to imagine Steve Jobs running an open-allocation Apple. I think the benefits of open allocation in terms of what people are allowed to work on an experiment with are very strong. The fact is, though, that this can't be taken so far as to allow people to impose complexity on the design or the ecosystem as a whole. You probably need centralized authority and vision to keep, for example, a simple product design. I don't think that that should be an excuse to prevent people from organically forming and changing teams, though. It requires conservatism about what actually gets into the product, but not as much about how people work and what on.