If anyone has tried playing a processing heavy software synth on a pc in real-time you will have experienced how unplayable it is as soon as latency goes beyond 10-20ms - you can't play music if there is noticeable delay between your fingers and ears, and we are much more sensitive to sound latency, it would be the same problem trying to play in time with each other.
I like the idea, but feel like the internet isn't there yet for the majority of users, and latency hasn't exactly been improving at a great pace. I expect it will still be useful in internet rich areas like city to city.