I was speaking to checked exceptions vs unchecked. "Exceptions at all" is a separate question.
If we're going to dig into it, first we need to call out the occasional need to interrupt running pure code (eg. "the user hit ctrl-c"). As it can happen at any point, I think it's uncontroversial to say that return values are inappropriate? These are Haskell's "asynchronous exceptions", I think they do their job fairly well, and I don't know of a compelling exception-free alternative (although I'd be very interested to learn).
So now we can restrict ourselves to "synchronous exceptions" - those raised because of the function we've called.
I agree that we don't need them. They bring to the table early termination and transparent error propagation. There are contexts where these are desirable - if you disagree on that point let me know and we can dig in. In sufficiently expressive languages, we can do it ourselves - so that's plenty to show we don't need them.
But we may want them.
When we do it ourselves, we usually lack a way of telling the compiler which paths to privilege, and we often find some overhead compared with native exceptions.
When exceptions are provided in the language, we get special syntax to distinguish them. You describe this as a drawback, but nonuniform syntax can help with legibility - if different things look different, I can more easily figure out what I'm looking at. Obviously there are tradeoffs here.
Putting exceptions in the language also provides a distinction to the compiler in type checking, allowing different rules to apply. For instance, Java methods throw some set of exceptions, which get unioned (&c) in ways you can't speak about sets of types otherwise. Simply applying the ordinary value-level rules would be problematic (I hit this in Haskell sometimes). Ideally you can broaden your type rules to support what you need for all of it in one system, but it's not always clear they combine well. Even if it all can be made to fit, I could easily see a situation where assumptions need to be made for inference to work, and the most ergonomic assumptions are different for most values vs. bags of errors.