For almost all the uses you've listed, I can readily think of examples where I've seen it done badly. Thinking of good examples is considerably harder. Can you provide some?
In my experience, deferred loading merely guarantees that there's visible latency multiple times when interacting with a page even after it has "finished" loading. And since lazy image loading can now be accomplished without JavaScript, I'm not sure what still needs to be lazily loaded via scripts watching scroll behavior.
Removing elements after you scroll past them sounds like the kind of optimization only necessary on infinite scroll pages, unless the page in question has a multitude of auto-playing videos.
Restoring scroll position between interactions: I'm not sure what kind of interactions make this necessary, so an example would help.
Scroll-snap: The examples on MDN don't even behave properly.
Offsetting jump in-page anchor/jump links: Stop obscuring 20+ % of the vertical space with "position:fixed" dickbars. Then you have a much smaller problem which I'm not sure requires JS scrollhacking to solve.
I'm not sure scroll hijacking is completely bad, but it's overwhelmingly more bad than good. Breaking fundamental interaction mechanisms should only be done where it's absolutely necessary, or in a context where all the normal expectations for interaction can be abandoned. These capabilities go beyond merely "ripe for abuse" and are perhaps better described as UX footguns that are mysteriously attractive to web designers.