Technical coach here. This is a passion of mine. I've seen far too many teams waste far too much time with horrible, horrible backlogs and tooling systems -- usually set up with the most sincere good intentions, by the way. I care so much about it I wrote a book on managing project information. The key example, repeated throughout the book, is a small team meeting folks and starting to make stuff people want. (Obligatory link:
https://leanpub.com/info-ops )
I believe if you can get the small team scenario working over-and-over again scaling will work itself out. So far I have no reason to believe this isn't the case -- and I've applied the principles in the book both to functional coding and program management.
mjul's comment is the key one: you can't know everything before you start. That doesn't mean you can't know anything. It means that there is a "progressive elaboration" that has to happen on an as-needed basis. Otherwise you're stuck either not knowing enough to get going -- or having created a monster or a tools/information system that ends up running the project instead of the team.
There are some sanity checks for whatever backlog/requirements system you are using. Instead of my continuing to pitch, I'll just list the things that your system should do no matter what kind of system you have.
- Handle whatever detail is needed before actual development happens.
- Be able to "flip-around" and start from scratch within an hour or so. (And "starting from scratch" means beginning with nothing and ending with the team starting coding) While keeping all that detail in test 1.
- Be reusable with other teams doing similar work. A backlog/requirements system can't make work fungible, but it can enable better conversations in other teams without being a burden.
- Limit meetings around organization to under an hour or so. Yep, you gotta have those "meta" meetings from time-to-time and talk about things like release plans. But they shouldn't take over your afternoon.
- Help the larger organization (if there is one) learn and grow over time. Orgs learn from the bottom-up. Good tools should facilitate this learning.
- Drive directly to acceptance tests. Good backlogs are testable. Things shouldn't be in there that don't drive tests.
- Have controls in place to prevent abuse. As soon as you create some tool for the requirements/scoping phase, somebody is going to go all architecture-astronaut and overuse it. There has to be controls to prevent this from happening. The tool should facilitate the work of understanding and scoping, not replace it. (I think even a lot of tools vendors get confused on this one).
I even wrote an analysis compiler that demonstrates all of this as part of writing the book. So if you think all of this is impossible -- happy to do a demo.
There are a few other testable criteria for whatever your requirements/backlog system is, but that should be enough to get you started deciding whether some system is better or worse than another system.