My boss and I had written a distributed transaction manager. Of all the components of our product, this was considered the lynchpin of reliable enterprise software.
The day came when we had to present it at a very well attended product launch. He and I took turns making different points. Then he announced to the crowd that he had a surprise for me (and turns out, to the entire company), that he had changed the configuration to write the transaction log to a removable drive. Then, in the middle of the demo he proceeded to pop the drive. Our careful demo script had got Putin'd. I went numb, our marketing manager nearly had a coronary.
He then announced that if it were indeed enterprise quality, the system should recover once the disk is put back in again. It freakin' did; the system worked!! He got a laugh out of the crowd that if it survives a demo it would survive failures in production, esp. when one goes off-script. He said that we had spent months and months in careful design and implementation and we have no business writing the word 'reliable' in our marketing literature if we didn't believe it in our core. The crowd lapped it up.
It was a huge humbling and learning moment for me. Although I was the primary implementor of that system, I didn't nearly have enough faith in it as he did. In one instant, I realized all the practices he'd been drilling into our team from the first day, the insistence on creating elaborate mock frameworks to induce faults, embracing the idea that if something could go wrong, it will. I realized that if you are surprised that the system works, you haven't reflected on that doubt enough. He would light up like a child when we encountered a real bug, and he'd follow it up with a session of reflection on why it escaped our attention in the first place. That session would result in a raft of new tests.
Rest in peace, Ed Felt, my friend, mentor and guide. Taken too soon (United flight 91, alas).