Back at the 2001 Comdex, Bill Gates announced that by 2007, such devices would be the most popular form of PC in the U.S. Although his timing was off, I think Gates was definitely on to something.
Yet as brm notes, it's probably going to take more than just the open source community. Open source seems great for advancing the bleeding-edge; but honestly, unless money is on the line, how many devs are going to cut out their cool feature that they spent hours developing, just because it's hard to use or might make the device more complicated?
Perhaps, a team (but not necessarily a large team) of hackers devoted full-time to making a successful interface / kiosk for this might make it big. Perhaps, such a business might make a good fit for Y Combinator. ;-)