Sun was done in by the following mistakes from around 2002:
- Failure to license Solaris to Google (reportedly Sun wanted to have a license count, while Google considered the number of systems they ran a secret).
- The "termination" of Solaris on x86, which was really (or became) a pause in development for six months -- whichever the case, it was a disaster as it gave Linux a huge boost.
- The closing down of Sun Professional Services (presumably in order to preserve R&D). Compare to IBM, which did the opposite, and whose professional services division became hugely valuable to IBM. Professional services organizations sell their services and their parent companies other products and services. Losing the professional services organization meant losing sales.
That was all in 2002. Many more mistakes were made later (and before), but these are the ones that really killed Sun.
Other mistakes after 2002:
- Treating J2ME as a cash cow and thus failing to get it on competitors' handsets when iPhones came about. This failure being one of limited R&D and mispricing the product. Compare to Android essentially adopting Java not that much later.
- In general treating x86 as a second class citizen to protect SPARC. Now, this one is not very fair because after the cancellation of Solaris on x86 was reverse, internally x86 was a first class citizen, and Sun built a great systems group around x86, but Sun was quite the cash cow milker and put too much effort to keep SPARC from falling too far behind x86, and from the outside it probably looked like Sun just didn't really mean x86 support until around the time of the Oracle acquisition (at which time Sun had great x86 systems for its storage appliance). You have to know when the cash cow is dying and when it's beyond rebuilding, and Sun just didn't.
- The MySQL acquisition. $1bn out the window. There were others, like Cobalt.
Oracle too made mistakes, chiefly killing OpenSolaris in response to Greenbytes' shipping non-release OpenSolaris ZFS dedup code before Oracle. There were other ways to handle this. Oracle is a company built on mindshare, but it didn't understand that Solaris was now the underdog and needed to build mindshare, and the way to do it was to loss-lead. Friends of mine immediately told me they'd no longer run any new Solaris systems and would migrate from existing ones ASAP because of shades of the 2002 cancellation of Solaris on x86 -- they felt they could not trust Oracle.
This could be my bias because 2002 is when I joined Sun. Clearly there were other mistakes made before 2001. But the Solaris team at Sun did incredible things in the 00s with a smaller team than in the 90s.
The Sun culture that created ZFS, SMF, FMA, and other things in the 00s doesn't exist anywhere now. It was a culture of internal openness and skunkworks projects that paid off.