I was quite comfortable with refactoring for a very long time before I started doing tests, and I think that colors my approach to the problem space. I write code until it misbehaves the first time, and then I start writing tests, comfortable in the knowledge that in another half hour this code will be unit testable and look better than what everyone else is turning in, and that I'll spend 20 minutes at the end cleaning it up even more.
But I am not everybody, and trying to set team policy on my comfort levels is quite likely a bit dangerous.
I've paired with a few smart people this month and was horrified by how they refactor. Most trust their hands more than the tools. They do a LOT of transcription instead of copy and paste, and don't use the IDE's refactoring tools at all. That's not just inviting human error to the table, that's giving it the seat of honor.
For most processes there are ways to do it that are worse than nothing at all, and unfortunately many of us seem drawn to them almost instinctively. I try not to think about it because it just makes me angry that the distance between what we collectively know and what we individually do is so wide.