Speaking as a proponent of remote work, I think the reason is that remote workers are less productive in a _non-remote enabled culture._ And by default most companies are not remote enabled. They don't even understand what that means. This is totally reasonable, as we still have very few examples of large, successful teams that are fully remote. Many of the ones we do have aren't actually companies, Linux kernel deve…
The ones who are "allow remote work" are entirely office-focused and view remote workers with suspicion. A remote worker is very unlikely to get promoted, be able to take on interesting projects or large responsibilities. And they'll be out of the loop as far as what is going on in the company. (Even at a large company with probably 15% of the workers remote, it was frequent that people would just forget to open up a conference line for company-wide announcement meetings.)
On the other hand, "remote first" companies structure themselves around tools to enable remote working. So meetings happen online (via phone or web conference), project discussion or just water cooler chat happens in a messaging system like Slack or IRC, and projects are documented in a system accessible to everyone. In these environments remote workers are first class citizens and are as much a part of the company as any other.
I was at a "remote first" company that grew out of the open source development world for close to 5 years. I went to a company with the "allow remote work" model and lasted barely a year before leaving for another "remote first" company. It was a painful year, but at least now I know what to look for.