Precisely the opposite: exceptions are a fail-fast mechanism that gives you an alternative to terminating the program.
Now, as slx26 mentioned, it's only half of the story. Most APIs (including .NET) document exceptions badly, they're not discoverable, and if you try to use them to _recover_ from a condition, you're in for a world of pain. The workflow is usually: attach the debugger, try to make the exceptional condition happen, inspect relevant info in the debugger and write the catch block.
I wish that programming languages supported some contract-like mechanism of declaring: "This method can throw only X, Y and Z". If the method throws anything else, a system-defined "UnexpectedException" would be thrown, encapsulating the invalid one. C++ used this model once upon a time, but they went away from it due to runtime costs and it being little-used. (Also, it terminated the program instead of rewrapping the exception.)
Exceptions are first-class values, but few programmers treat them as such, probably because the programming language allows them to do so. So we should start by fixing PLs.