This is one of those discourses that disappoints me about our industry.
Estimation can be done. It's a skillset issue. Yet the broad consensus seems to be that it can't be done, that it's somehow inherently impossible.
Here are the fallacies I think underwrite this consensus:
1. "Software projects spend most of their time grappling with unknown problems."
False.
The majority of industry projects—and the time spent on them—are not novel for developers with significant experience. Whether it's building a low-latency transactional system, a frontend/UX, or a data processing platform, there is extensive precedent. The subsystems that deliver business value are well understood, and experienced devs have built versions of them before.
For example, if you're an experienced frontend dev who's worked in React and earlier MVC frameworks, moving to Svelte is not an "unknown problem." Building a user flow in Svelte should take roughly the same time as building it in React. Experience transfers.
2. "You can't estimate tasks until you know the specifics involved."
Also false.
Even tasks like "learn Svelte" or "design an Apache Beam job" (which may include learning Beam) are estimable based on history. The time it took you to learn one framework is almost always an upper bound for learning another similar one.
In practice, I've had repeatable success estimating properly scoped sub-deliverables as three basic items: (1) design, (2) implement, (3) test.
3. Estimation is divorced from execution.
When people talk about estimation, there's almost always an implicit model: (1) estimate the work, (2) "wait" for execution, (3) miss the estimate, and (4) conclude that estimation doesn't work.
Of course this fails. Estimates must be married to execution beat by beat. You should know after the first day whether you've missed your first target and by how much—and adjust immediately.
Some argue this is what padding is for (say, 20%). Well-meaning, but that's still a "wait and hope" mindset.
Padding time doesn't work. Padding scope does. Scope padding gives you real execution-time choices to actively manage delivery day by day.
At execution time, you have levers: unblock velocity, bring in temporary help, or remove scope. The key is that you're actively aiming at the delivery date. You will never hit estimates if you're not actively invested in hitting them, and you'll never improve at estimating if you don't operate this way. Which brings me to:
4. "Estimation is not a skillset."
This fallacy is woven into much of the discourse. Estimation is often treated as a naïve exercise—list tasks, guess durations, watch it fail. But estimation is a practicable skill that improves with repetition.
It's hard to practice in teams because everyone has to believe estimation can work, and often most of the room doesn't. That makes alignment difficult, and early failures get interpreted as proof of impossibility rather than part of skill development.
Any skill fails the first N times. Unfortunately, stakeholders are rarely tolerant of failure, even though failure is necessary for improvement. I was lucky early in my career to be on a team that repeatedly practiced active estimation and execution, and we got meaningfully better at it over time.