I also did Java development back then. Applets died for many reasons...
First, the Java plugin required a JDK installation, which was a considerable download. At the time, many apps/tools included some JDK version, but as a developer, you can't count on that. It took many years for Java to have a modular JDK, but when it happened, applets were already gone. By comparison, the user experience of Flash was orders of magnitude better.
UI programming was problematic. AWT was ugly, and Swing was too slow for the machines then. In comparison, Flash had tools to create excellent interactive games and small UIs. Also, as HTML/CSS/JS supported more and more UI patterns, there wasn't a need to use applets for complex UIs.
The JVM is sandboxed, but there are so many moving pieces and outdated JDKs installed that they become a security problem and attackers started to poke into the applet plugin bugs to find holes.
Finally, another issue shared with frameworks like ActiveX or browser plugins was DOM integration. When you mix an applet with a page, the applet feels alien. You can't navigate inside the applet without breaking the browser navigation, and using apples for small UI pieces (Sun's home page used to have a news ticker applet) is an overkill. The best use case for applets or ActiveX was video playback, but browsers added native support.
That's why Java Web Start was more interesting than applets. But, at the time, Sun was spread too thin. Adobe did the same with the Rich Internet Apps and Flex but with a better execution (until the iPhone killed the RIA hype).