With more than two participants, real-time peer to peer video needs a lot more bandwidth on your local internet connection, compared with a hub. That's because you need to send your AV stream to all the other participants in separate streams, instead of one stream to the hub. Same for what you receive, as the hub sends you a composite of the current speaker and thumbnail videos of everyone else.
Most people don't have a network at home that can handle, say, sending 50 copies of your AV stream when you're speaking to 50 other participants. With that many, even audio-only is a challenge.
There are some clever p2p relay mechanisms you could use in theory to bring the bandwidth down to not grow as much with number of participants (but still a constant factor more than the hub method), but they would add significant latency. For example your AV is sent to one peer, they send it to the next peer, and so on in a chain or tree. This works well for non-real-time AV like p2p distribution of movies, but not real-time meetings.
If multicast IP worked over the internet at large that would help but it doesn't work. Even if it did it wouldn't be enough for the full-vs-thumbnails continuous renegotiation needed to keep download bandwidth steady, nor could it do per-user quality-vs-rate adaptation.