Working remotely pretty much destroyed my career for two reasons: (1) my networking opportunities plummeted to zero and (2) I was fired for reasons I believe could have been avoided if I had worked in an office and had stronger relationships with more co-workers.
My remote job was at a fully distributed company with good compensation.
I worked there for about five years, receiving only positive reviews.
Groups in the company worked in a very silo-ed manner — there was not a lot of work overlap or cross-team communication. But I loved my team and work so it was fine with me.
Once the company started struggling, they re-organized our group several times, and eventually a non-technical manager with more seniority than me was brought in.
Although I was highly effective, well-reviewed, and never had issues with other managers, the non-technical manager and I dealt with all of the classic issues that come up in this scenario: unrealistic deadlines and expectations, scope creep, etc. I tried to deal with this in an ego-less, professional manner, but he took our issues personally and lobbied HR to have me fired.
At that point, after all of the re-organizations, I was the only dev left on my team, and my other managers who I had good relationships with had moved on. I had no one to vouch for me. I believe that if I had worked in an office, in a less silo-ed manner, I would have had more opportunities to network with other developers within the company and my accomplishments would have been more visible, which would have at least allowed me to switch teams.
Instead, I was quietly led out the back door and disappeared.
In a remote company it's a lot easier to be fired. It's a lot more difficult to resolve interpersonal issues over IRC than face-to-face. There is a huge amount of value to passing people randomly in the hallway, by "the watercooler," or at a social event. These kinds of relationships are critical and are just not replicable via remote work.