There's definitely room for improvement over currently used server and desktop operating systems; the security models used by desktop Linux, OS X, Windows, and BSD don't meet the needs of users in a world of malicious software. iOS and Android were able to improve on the situation a good deal by adopting higher-level security models that protect applications (and their data) from one another, but the desktop lags behind.
As far as I know there's no technological reason the same couldn't be done with the server or desktop, it would just be an enormous shift in momentum to get there from where we are now. But if we're willing to stomach shifts in momentum, we could even go further and consider other, more radically different approaches to security than what's commonly used now, such as Microsoft's Singularity.
I don't know enough to say whether an operating system startup could be successful, but I think if it were, it would be less about basic operating systems research than about getting what research we already have into the hands of the masses on the desktop or the server. Presumably by targeting sectors where customers care enough about security to make the difficult switch to a fundamentally incompatible system.