Async is a huge wart and I try and avoid it wherever possible. It's simply not useful in a lot of cases that I encounter. However, if a library uses async, you have little choice but to make your whole project async. This adds to its horrible reputation.
Integrating 'async' into a language is not a good approach to concurrency. If you want to run a single function on a different thread it can work. If you want something to run after that, that can work out.
Once you go beyond that, you are building a graph with very crude tools. Then you have lots of problems, including how you handle packaging dependencies when something you want to run depends on data coming from multiple other async sources.
Treating this as a language issue and not a library and tools issue is a huge mistake. Another reason is exactly what you outlined - libraries getting infected with a languages half baked concurrency solution instead of doing what they are supposed to while the user can fit them in where they want.
What actually works is graphs that handle dependencies and data structures for synchronization, but ultimately those need to be done well too.