Java has gotten a bad rep lately due to some high-profile drive-by-malware bugs. But if the java codebase would have gotten the same intensive care that the webkit codebase got, this would no longer be an issue.
Many people remember java to be sloooow. When I first came into contact with it in school, that was certainly the case, but since a couple of years it has had a modern JIT that could easily rival native code.
Java applets are ugly, sure, but that is largely due to the decades old AWT, and the poor font support it used to have. With SWT, you can have native widgets (dunno if they work in Applets, but they are nice on the Desktop), and with antialiased drawing you can get the same results as with HTML5 canvas.
Java applets (and Flash, and Silverlight) died for marketing reasons, and political reasons. There were no unsurmountable technical issues. The outcome is that we are stuck with "worse is better" for the foreseeable future, only max. 50% to 1% of the possible native performance, and a bunch of restrictions we only slowly realize what they mean (no sockets, no signed applications, no anonymous/serverless mashups, less hardware access than we used to have, suboptimal caching, suboptimal tooling like languages, debuggers, content creation tools (I haven't seen anything that can replace Flash for simple vector animations yet) and so on.