I've been coding now for about 30 years. I've never moved up into management and, now at almost 50, it's pretty clear I never will, and will spend the remainder of my career coding (which I'm perfectly fine with). In the 30 years I've been coding, languages have come and gone, platforms have come and gone, frameworks have come and gone, methodologies have come and gone, but two things have never changed: 1) Software…
1. When doing estimates, estimate hierarchically and break down every task until the leaf nodes are 4 hours or less. If the leaf nodes are a week of effort, that means that you don't yet understand fully what the task entails. If you don't know what the task entails, it is very difficult to get an accurate estimate.
2. Maintain a database of various projects and features, and use them for comparison purposes. If you know how long something actually took in the past, you can much more accurately how long it will take to do something similar again.
3. Include specific time for various types of "overhead": meetings, code review, writing tests, debugging, making presentations, etc.
4. Derisk ahead of time things that are truly unpredictable. Doing something in a brand new programing language or framework, training an ML model on a new type of data, working on datasets of unknown quality, etc. For those you need to estimate out a project to determine feasibility and generate an estimate.
The main thing is that this is clearly not impossible. Years ago I got really interested in the Apollo program as an example of bringing in perhaps the most complex and risk-laden project ever attempted, on time. One of the things that clearly stands out about that project was that there was huge derisking happening first. At one level you had the Mercury and Gemini programs that were figuring out human spaceflight generally, and docking with Gemini, and also things like the first stage engines of the Saturn V were already in development in the mid fifties.