It's unfortunate. Especially in agile, a good project (err.. product...) manager is worth their weight in gold. I have to be honest, I've had my hand slapped here on HN for conflating project managers with product managers. I admit that I do this because from an agile software development perspective you need a variety of skills.
One of the biggest problems I've seen on agile projects is a lack of analysis. Often "agile" development processes are chosen because people want to avoid analysis. I think the thinking is that if you remove all the documentation and discussion around what you are building, the developers will just "do the right thing" and you'll save a whack of time.
In fact, on the XP teams I've run, I tend to have 2-3 month backlogs of stories. You want to think through what you are building and what the implications are. There are 2 main differences between this and a more analysis-up-front approach. First, your stories are changeable and prone to re-ordering. Second, you defer some decisions until later. There is no phase when the requirements are carved into stone. However, if you get to the beginning of the sprint and you don't know what the acceptance criteria for all of the stories in the sprint are, then you are in big trouble.
Also, I come from the ancient days. I've personally written a 1300 page requirements document. That took 6 months to write. That was largely ignored. When I tell people that I want a couple of months out of "this is what we're doing if nothing changes", these days people think I've got all top heavy on them. Times change...
This is where a good project/product manager comes in handy. You need someone who owns the overall vision of what you are building and has time to think through the implications. You need someone who is constantly grooming the backlog and updating the cards as the vision changes. You need someone who understands the business priorities and knows when and how to reprioritise the backlog. You need someone who is constantly asking the question, "It's fine to do X, but what does that mean for Y", or "What makes you think that this is the solution we need" or "Who is going to be affected when we do X".
Often product/project managers think, "Oh I don't have to know how the product works in detail". This is a huge red flag for me. They often think "Oh the programmers can decide what to do here". Yes, the programmers can, but they already have their plate full with how to do what needs to get done. If you ask them to think through all the details of what and why then they will have less time for how. Similarly, if you have a team of 6-8 developers, then you will have 6-8 opinions about what the best thing to do is. The vision will potentially be weakened and you will often find that the programmers spend half their time arguing about who has the best idea. At best you'll have 1 developer doing the product/project manager's job and the rest will be doing programming jobs.
I could go on forever here. I haven't even touched on dealing with internal politics, reporting, ensuring visibility of problems, coordination with other groups, gathering data, etc, etc, etc. If you don't have someone doing these things then they are often not getting done at all (often to your detriment) or you are wasting your precious programming horse power dealing with the realities of business.
As you can imagine, I over the years I've regularly butted heads with "I don't really want to do anything at all" product/project managers. But if I get someone willing and able, I will absolutely load them down with work that has the potential to make a massive difference to everyone.