The confusion is perhaps due to your usage focus and the security constraints browser compiler makers face to make something secure.
First off, remember that initially all we had was JS, then Asm.JS was forced down Apple throats by being "just" a JS compatible performance hack (remember that Google had tried to introduce NaCl beforehand but never got traction). You can still see the Asm.JS lineage in how Wasm branching opcodes work (you can always easily decompose them into while loops together with break and continue instructions).
The target market for NaCl, Asm.JS and Wasm seems to have been focused on enabling porting C/C++ games even if other usages was always of interest, so while interop times can be painful it's usually not a major factor.
Secondly, As a compiler maker (and to look at performance profiles), I usually place languages into 3 categories.
Category 1: Plain-memory-accessors, objects are usually a pointer number + offsets for members, more or less manually managed memory. Cache friendlyness is your own worry, CPU instructions are always simple.
C, C++, Rust, Zig, Wasm/Asm.JS, etc goes here.
Category 2: GC'd offset-languagses, while we still have pointers(now called references) they're usually restricted from being directly mutated, instead going through specialized access instructions, however as with category 1 the actual value can often be accessed with the pointer+offset and object layouts are _fixed_ so less freedom vs JS but higher perf.
Also there can often be GC-specific instructions like read/write-barriers associated with object accesses. Performance for actual instructions is still usually good but GC's can affect access patterns to increase costs and some GC collection unpredictability.
Java, C#, Lisps, high perf functional languages,etc usually belong here (with exceptions).
Category 3: GC'd free-prop languages, objects are no longer of fixed size (you can add properties after creation), runtimes like V8 tries their best to optimize this away to approach Category 2 languages but abuse things enough and you'll run out a performance cliff. Every runtime optimization requires _very careful_ design of fallbacks that can affect practically almost any other part of the runtime (these manifest as type-confusion vulnerabilities if you look at bug-reports) as well as how native-bindings are handled.
JS, Python, Lua, Ruby, etc goes here.
Naturally some languages/runtimes can straddle these lines (.NET/CIL has always been able to run C as well as later JS, Ruby and Python in addition to C# and today C# itself is gaining many category 1 features), I'm mostly putting the languages into the categories where the majority of user created code runs.
To get back to the "troubles" of WasmJS, as you noticed they are of category 1 and 3, since Wasm is "wrapped" by JS you can usually reach into Wasm memory from JS since it's "just an buffer", the end-user security implications are fairly low since the JS has well defined bounds checking (outside of performance costs).
The other direction is a pure clusterf from a compiler writers point of view, remember that most of these optimizations of Cat 3 languages have security implications? Allowing access would require every precondition check to be replicated on the Wasm side as well as in the main JS runtime (or build a unified runtime but optimization strategies are often different).
The new Wasm-GC (finally usable with Safari since late last year) allows GC'd Catgory 2 languages to be built directly to Wasm (and not ship their own GC via Cat 1 emulation like C#/Blazor) or be compiled to JS, and even here they punted any access to category 3 (JS) objects, basically marking them as opaque objects that can be referred and passed back to JS (improvement over previous WASM since there is no extra GC synching as one GC handles it all but still no direct access standardized iirc).
So, security has so far taken a center stage over usability. They fix things as people complain but it's not a fast process.