Sec-Fetch-Site: It's more like `Origin`but without actually containing the Origin instead just delivering information about if it's the same site, same origin different origin or has not origin.
This makes it much more privacy friendly then both the `Origin` and `Refer` header, it also makes it easier to user for the intended use case and in turn IMHO a strict improvement over both `Origin` and `Referer`.
Sec-Fetch-Dest, Sec-Fetch-Mode, Sec-Fetch-User: Provide a bit more context about how the request was made, while this can leak slightly more information compared to `Origin` or worse `Referer` it's still much better.
So from a privacy POV I would say this is a strict improvements.
From a functionality POV it might look like it further limits 3rd party resource re-usage but CORS already does so. And like CORS it can be circumvented by using download apps which are not your browsers or servers republishing things or similar.
I could imagine there could be some web-extensions which "extend" a website by injecting code or similar which would become harder to do with this. Through I don't know of any where there isn't a reasonable workaround.
So from what I can tell the worst thing it might do is that using `curl` for sites where you need to set a `Origin` header now also need you to set some other headers which could be annoying.