We don't actually know the mortality rate. Or rather we don't know it defined in the way people care about ("what is the likelihood that I might die"). The closest data seems to be South Korea where they're doing mass testing, but even then, are they really doing scientifically valid random testing or are people still self selecting?
I've sort of swung back and forth on my concern levels about this, so I guess I understand the OP's concern but also the lack of concern in the people he talks to. I think it's a mix of things:
1. People learn about this via the media but a lot of people (more than half) deeply distrust the media. They think journalists lie all the time. Trust in media polls have been in decline for a long time, so when reporters turn up with "omg panic" type stories, people don't take it seriously anymore.
https://www.cjr.org/the_media_today/trust-in-media-down.php
2. People have been receiving very mixed messages. It's a killer plague but it's also kinda like having a bad cold. The death rate is way higher than normal but it's 0.5% in the only country doing mass testing, which doesn't sound high. It's incredibly dangerous but it's also harmless to children and healthy adults. Thousands are dying but most of them were about to die of old age anyway. It's going to crash the economy but that's because the government is shutting everything down, not because people are actually unable to work. It's so infectious a single touch can give it to you, but it also requires standing around someone infected for 20 minutes to get it. It will infect the entire world but China's cases have already peaked. Etc etc.
It's pretty hard for people to know what to make of all this. I'm not saying the information people are getting is wrong, but it sounds contradictory and confusing.
3. If you believe it's really so infectious then there isn't much you can do. Nobody can self-isolate forever, and many can't self-isolate at all.