I get that some PMs can undervalue refactoring code / minimizing technical debt, and that's super frustrating and shortsighted. But the basic premise of this criticism of agile process is deeply flawed:
1. The claim that features took "longer" to ship is directly contradictory to the admission that the project was months behind schedule.
2. If they were behind schedule, then "inflating" their estimates in response to a detailed tracking process was not, in fact, an inflation. It was a reality check.
Estimates are really hard. But the solution is not to just "give autonomy" with zero estimation, because that's how you get into that problem they had in the first place: a guy in a room that doesn't ship for months. That's why agile's approach of breaking things down into tasks with very rough sizing and measuring actual after-the-fact time and shipping small milestones frequently is so helpful. You don't have to have super accurate estimates. You just need to report your time and then it figures out a team "velocity". That helps teams be more predictable, which is what it means to be professional and deliver things to customers on time.