What I'm worried about is that it will be hard to avoid these security patches when you don't need them. Say you have a non-virtualized, non-shared server that only runs your own trusted code. I don't want to be forced to pay the performance penalty but it might be unavoidable without resorting to maintaining your own linux fork.
Do you trust all of the userland? Is your machine internet connected? Do you have any open ports? Do you run everything as root? If your answers are yes, no, no, and yes than it likely will make no difference. Otherwise (and the last one is just for fun to attempt to show you this is probably not a wise decision) you probably would do better to take this serious.
For an average user that doesn't bother with security to the point that nothing on the server runs in sandboxes - Spectre and Meltdown are not of much concern. Running under a different user is not the same as running in a sandbox, certainly not on linux/bsd/windows. A user still has an enormous attack surface to play with. And due to the nature of multiuser software typically running on servers where all users live within the same trust boundaries any exploitable vulnerability is going to do as much damage, as if it was running under root, think Cloudbleed for example. You can't make things noticeably worse by turning off KPTI.