Earlier quoted context omitted.
> You need a high level design up-front but it should not be cast in stone. Yes, you need a design that precedes code. > Writing code and iterating is how you learn and get to a good, working design. You are confusing waterfall-y "big design upfront" with having a design. It isn't. This isn't even the case in hard engineering fields such as aerospace where prototypes are used to iterate over design. In software engin…
The difference I see is that in other fields, part of your design process is thinking through the logical details of the thing. Essentially, doing some math. In software, the logical details are the finished product. The math is what you're trying to make. If you've actually thought through all of the details, you have written the software (if only in your head). If you haven't thought through all of the details and…
The "some math" is used in engineering fields in things like preliminary design, sizing, verification&validation, etc. To a lesser degree, "some math" can be used in the design stages of software development projects. For example, estimating the impact of micro services tax in total response times to verify if doing synchronous calls can work vs doing polling/messaging. Another example is estimating max throughput per service based on what data features in a response and how infrastructure is scaled. This is the kind of things that you do way before touching code to determine if the expected impact of going with a particular architecture vs another that mitigates issues.
> In software, the logical details are the finished product. The math is what you're trying to make.
You're confused. The design stage precedes writing any code, let alone the finished product. Any remotely complex work, specially if it involves architecture changes, is preceded by a design stage where alternatives are weighed and validated, and tradeoffs are evaluated.
To further drive the point home, in professional settings you also have design reviews for things like security and data protection. Some companies even establish guidelines such as data classification processes and comparative design to facilitate these reviews.
> If you've actually thought through all of the details, you have written the software (if only in your head). If you haven't thought through all of the details and only figured out a high level design, you've still written some software (essentially, stubbing out some functionality, or leaving it as a dependency to be provided. However you want to think of it).
You're confusing having a design stage with having a big design upfront. This is wrong.
The purpose of the design stage is to get the necessary and sufficient aspects right from the start, before resources are invested (and wasted) in producing something that meets requirements. No one cares what classes or indentation style you use to implement something. The ultimate goal is to ensure the thing is possible to deliver, what it actually does and how it does it, and if it is safe enough to use. You start writing code to fill in the details.