Semantic versioning solves one specific problem that's worth solving - whether you can (expect to) automatically upgrade. That is a problem people have with languages just as much as libraries, and it is a problem that affects both big changes and small just as much as it affects libraries. It is not the only way to solve this problem, but that problem very much needs to be addressed.
When a language adds any features, if your dependencies (whether real library dependencies or just things you're copying from Stack Overflow) start using the new features, you must upgrade to the new language version. That is an inherent usability constraint, and every time a language designer chooses to add a feature, they're making a tradeoff. But if upgrading to the new language version is trivial, then it's generally a worthwhile tradeoff.
For instance, suppose I find some code that uses Python's removeprefix() method on strings. I need to use Python 3.9 or newer to use that code. It doesn't matter that this is a very small feature.
However, I can generally expect to upgrade my Python 3.8 code to Python 3.9 without trouble. It's different from, say, code that uses Unicode strings. For that code, I need to upgrade from Python 2 to Python 3, which I can expect to cause me trouble. The version numbers communicate that. It's true that Python 3 was a "big" change - but "big" isn't really the point. The point is that I can't use Python 2 code directly with Python 3 code, but I can use Python 3.8 code directly with Python 3.9 code. There are plenty of "big" changes happening within the Python 3 series, such as async support, that were made available in a backwards-compatible manner.
As it happens, Python does not use semantic versioning. But they have a deprecation policy which requires issuing warnings for two minor releases: https://www.python.org/dev/peps/pep-0387/ It's technically possible, I think, that a change like Unicode strings could happen within the Python 3.x series, but that's okay, provided they follow the documented versioning policy. This policy addresses the same question that semantic versioning does, but it provides a different answer: you can always upgrade to one or two minor versions newer, but at that point you must stop and address deprecation warnings before upgrading further.
You are, of course, free to also have a marketing version of your project to communicate how big and exciting the changes are. Windows is a great example here: Windows 95 was 4.0 (communicating both backwards incompatibility with 3.1 and major changes) and Windows 7 was 6.1 (communicating backwards compatibility with Vista but still major changes).