Besides the usual quality considerations (correct, available, performant, etc.), well engineered software systems should be:
- observable: you need to be able to tell the state of the system to a sufficient degree (metrics, status reports, etc.)
- operable: it should be possible to influence the behavior of the system even in case of failure (feature flags, tools for retrying or pausing jobs, throttle processing speed, etc.)
Failure modes should be considered as early as you can, and provide sufficient tooling to deal with them.
Most of the effort on large scale or mission critical systems is dealing with failure modes, rather than the happy path and making sure it's extremely difficult to make it fail catastrophically without recourse.
Failure should be considered business as usual and dealt with accordingly, hopefully automatically but you need to leave tooling in place for those situations you didn't consider.