There have been some great answers from others in this thread already. Here's my take on things:
1. As others pointed out, Plan 9 was not initially free. If you go back and look at the very oldest messages from the 9fans mailing list (http://9fans.net/archive/, starts in 1993) you'll see lots of messages from people just trying to figure out how to get a license. At the same time, you could get Linux for free or very cheap.
2. Plan 9 is unfamiliar for new users. Face it: most people define "intuitive", "powerful", and "useful" as "that thing I'm already used to". Programming in Plan 9 is pretty similar to programming Unix, and the shell feels much the same, but you still need to learn new concepts and people don't like that.
3. Plan 9 is not Unix. People have, from the beginning, wanted to reshape Plan 9 in the image of the Unix/Linux environments they are familiar with, even when it doesn't make sense. They want Emacs or Vim, without having even tried Acme or sam. They want bash, ssh, X11, Firefox, and GNOME. Some of this software, like ssh, X11, and even Vim, have been ported or re-implemented because they're useful for interoperability with Unix or simply because somebody wanted it bad enough to do the port (Vim).
Those aren't the only reasons, but they're pretty substantial.
As for what the next kernel/OS should do, I'd say it's not enough to just exist and be good. You have to be used widely and have enough developers become familiar with it. If, for instance, some engineer at Facebook comes up with a nifty kernel and convinces them to start using it on all their servers, that kernel will probably be successful outside because there's suddenly a massive deployment of it.
Alternately, if you can get into a market that's less entrenched than the desktop, you may have a chance. We saw this to some extent with netbooks (some of the earliest ones shipped with Linux by default), we've definitely seen it with smartphones, and I have a feeling that we may see it in tablets. The problem with the desktop frequently comes down to: "Why can't it run Firefox and open my Word documents?"