It's not strictly worse than LTE, but it's not quite just you. A lot of it will depend on a bunch of factors.
If you're in an area with mid-band 5G (5G UC on T-Mobile USA, 5G UW on Verizon, 5G+ on AT&T), 5G will provide a meaningfully better experience most of the time. TDD (time-divided rather than frequency divided) 5G on mid-band spectrum is seeing great results. T-Mobile's 5G network is averaging 217Mbps (including both their low-band and mid-band networks, but it shows how much mid-band 5G can do).
If you're in an area with low-band 5G, it's complicated. If you're on Verizon, they're using DSS (dynamic spectrum sharing) to use the same spectrum for both LTE and 5G. This means that the network is constantly context switching between LTE and 5G and it provides pretty crappy service. If you're on T-Mobile, their low-band 5G network is using new spectrum and does often offer some moderate advantages. Another complication is that you might be sharing that 5G with all the heavier users while an LTE channel is less used. You could also be connecting to a tower farther away to get a 5G signal.
It's probably not that it's higher frequencies since you're probably in an area where it's using similar frequencies. I'd say that this is more likely carrier-related than 5G related.