I don't have a fully baked design in mind, but I could probably come up with one if I had a long time to focus on it. I do have a general framework in mind for how one might approach the question.
First and foremost, security should be a primary OS design objective and it should go into the design deep. It should be something you think about before secondary concerns like a process model, binary format, driver framework, or memory management.
Every single executing piece of code should be signed (with self-signing by the user an option of course). In that respect the app store model has it right, but I'd like to see something that ultimately puts the keys in the hands of the user. That means that a keyring is something you put down with the task manager, memory manager, etc. It's a core OS function. The ability to administrate the keyring and control permissions is also a core OS function.
Every function call outside a context should do a permission check against the certificate of the executing code. The right way to design this would be to make it work first, then figure out how to make it fast without compromising security. I think fundamental innovation would be needed here. I'm not sure exactly how this should work.
We should get away from distributing compiled code that runs straight on bare metal for most things. It might still be available as a permission, but one that would come with a warning to the effect that this could allow something to pwn your machine. Honestly I'm not sure if it's necessary. I'd look into the idea of shipping binaries as LLVM byte code and AOT compiling everything, and possibly including a secure implementation of OpenCL for really high performance computing needs.
A concept of users should be baked in from the get-go too, and should be part of the permission set of an executing context. Each user should have a key and be able to authorize other keys by signing them, etc.
So yeah, I think that's sort of a starting point. Crypto and permissions should be baked in from the get-go.
It'd also be important to think about usability from the get-go, since if it doesn't "just work" nobody will use it. UI/UX would be a challenging part of the project.
Storage is another challenge -- how to allow execution contexts to hand-off and/or share data without compromising security and without being too inefficient. The fact that storage is getting so cheap means things like copy-on-write with versioning might be baked in from the get-go to permit almost any operation to be rewound for a good period of time. So if a piece of bad code borks your work, just undo. I wonder if the whole OS could be built around a command model where things just fall off the end when they're too old? Log-structured everything? Again, not fully baked but I think it's the right general direction.
I highly doubt I am unique in thinking these things. I'm not the sharpest tack in the world and these kinds of ideas strike me as obvious results of reasoning from first principles about current OS challenges and failures.
The app store and mobile sandboxing models are steps in the right general direction but they are very, very ham-fisted compared to what I'm imagining here. They're the right ideas applied as a band-aid to fundamentally obsolete systems. They also cut the user out of the picture. I think that's because their models are ultimately too shallow and coarse-grained (and also because the vendors want control). Develop something good enough and the user can be put in the driver's seat without the machine turning into a malware cesspit. If the user authorizes a piece of bad code, just de-authorize it and it dies.