I've seen Agile™ estimation break down when the task is either
1. something nobody has done before (or has no close analog), or
2. is so interconnected that it can't be broken down.
The former is just a matter of hiring more experienced engineers _or_ allocating exploratory/prototyping time. Still high uncertainty but these kinds of tasks become rarer over a time.
For the latter, the common refrain is "break it down" but there certainly exists a relatively common type of work that must be completed all at once. And I find it increases as the complexity or popularity of the product increases, so with time. Therefore perhaps the metaphor of building becomes less appropriate, and surgery paints a more accurate picture.
Builders can construct a house, then add a garage, go work on another house, then return and add a guest bedroom, then remodel the kitchen, all with relatively minimal pausing or switching cost. But once a patient is put under and opened up, the surgeon really should work on finishing up that one patient before moving on to the next one. And for some weird reason we tend to prefer one big surgery to multiple small "atomic" ones.