It's further than that even. JSX has the semantics of (modulo a couple of optimisations there and there) a bunch of nested function calls returning normal JavaScript objects. That means you can, in your head, very easily convert between the JSX representation of an expression and the equivalent transpiled JavaScript code.
This is unlike a lot of other templating languages where, even if the expression part of the language is pure JavaScript (or PHP or Python or whatever), it's still interleaved with arbitrary text which will get printed out according to its own rules. This makes the whole thing much harder to reason about (leading to the philosophy that you should put as little logic as possible in your templates because it makes them harder to understand, _even when that logic is directly related to the templating process_.
A good example is for-loops. In a lot of templating languages, you start in text-land, then you enter expression-land to write the opening {% for (const X of ...) %} line, then you're back in text-land again. You sprinkle in a couple of expressions, and then at the end you go back to expression-land to close the loop. You're jumping backwards and forwards between the two worlds, but there's no real syntactical or structural support for mixing them.
Meanwhile, in JSX, you start in text-land, then you open up an expression between two curly braces, and in that expression you write the entirety of your loop. If you need to go back to text-land within that loop, you can create a _new_ set of text nodes, but you're not interleaving expressions and text in the same way.
The result of this is that, once you understand how your JSX will get compiled, it's very easy to read it as if it were the JavaScript that it will get compiled to, rather than as a separate templating language. Which in turn makes it very easy to think of it as "just JavaScript", even if it's technically a syntax extension.