And what are those threads doing all the time?
On N-core, 2-way SMT hardware (which describes pretty much all consumer hardware), the maximum amount of threads that can be usefully doing work in parallel is between N and 2 * N. Any more threads than that, and it must be the case that you expect those threads to spend a lot of time doing nothing to justify the cost of their creation.
There's two main categories that can justify that. The first is threads that are spending a lot of time stuck in blocking I/O--i.e., they're calling things that exist as yield-points in practical kernels anyways. The second thing is some sort of event processing loop, which looks like kind of like this:
while (!shutdown) {
event_t *e = get_next_event(queue);
if (!e) { sleep_until_work(queue); }
else { execute(e); }
}
The call in sleep_until_work already today uses a syscall to indicate that the thread shouldn't be scheduled. But even get_next_event can likely be trivially modified to use a syscall to add an event loop. Since multiple threads are able to enqueue events (else who would fill in work while you're sleeping?), you need some sort of threading library support to implement get_next_event correctly. Change the equivalent of pthread_condvar_wait for your OS to introduce a yield point, and you'll have introduced yield points to the vast majority of applications.