There is a massive mismatch between what people think an average employee looks like versus actual reality.
My girlfriend and I both have reasonable higher education, but it's nothing fancy. Culturally, we come from a working class background. So we have a very strong sense of duty and a great work ethic. It doesn't translate into crazy hours, but we're reliable. We show up. Focus during work hours. Deliver on promises.
We've always considered this to be the absolute minimum one can do. We're basic workers, nothing more.
How wrong we were. It seems that in whichever company we work, or in whichever department of some large company, we almost immediately rise to the top 10-20%. Every single time. There's almost a gravitational pull towards us as peers and managers discover that we get things done.
We've said it many times to each other: "surely it can't be this easy?"
This isn't a compliment to ourselves. We're not brilliant. We don't work crazy hours. We're not super ambitious. Just plain workers. I don't know what everybody else is doing, but apparently not that much.
And this connects to your situation, the standard is very low.
Another aspect that works in your favor is obscurity. Even if they have the idea that you don't do much, they can't be sure, it could be that you're working on things they're not aware of.
Confronting a peer that seems to do little has a very high social cost, so people prefer to avoid it. It's not their money pissed away and people prefer to avoid drama.
And yet another aspect is our species' adaptability. We rapidly adjust to any situation. When we have a lazy co-worker, a few weeks down the road that's the new normal. It's no longer remarkable and we've adjusted to work with that reality.
By the way, I pass no judgement on people with a lower work ethic. A huge amount of work is incredibly dreadful, boring, non-rewarding, and we ask people to basically sacrifice their entire life to it, decades in a row, every waking moment. This idea that everybody should just switch to their dream job isn't as realistic as some claim it to be.