I should probably refer you to the encryption whitepaper [1], but the basics are that Chat uses the Noise Protocol rather than TLS. All things being equal, the security properties are about equivalent, however all things aren't equal. The Noise handshake is smaller than the TLS handshake, and Noise doesn't have extraneous features WhatsApp doesn't use. Additionally, at the time of Noise adoption, TLS lacked a means for 0-RTT data (now available with TLS 1.3 Early Data), which meant using TLS would have added at least one round trip; possibly two, depending on which TLS library used. [2] You
can use TLS without x.509, but it's not very common; avoiding x.509 was a definite plus.
I wasn't much involved in anything on the chat channel, and I didn't do any implementation work on Noise, but I did some later prototype work with it, and if I recall correctly, it had much simpler framing than TLS as well; although maybe that was mostly TLS options getting me down --- the SNI header has 9 bytes of overhead, 5 of which are lengths, Noise didn't have anything like that as I recall. Do you really two bytes of versioning on every application data packet, like TLS has? I'm not sure you really need a type indicator byte either, context says you're sending a handshake packet initially, and then application data after that, but I'm pretty rusty on this now, so maybe there's a justification.
For users paying for internet by the byte, every byte counts. For users on networks with large delays, every round trip counts. For attachments, it's less critical (if your data access costs were high, you could configure attachments not to load) and that infrastructure was always built around http(s), so while there would have been an efficiency improvement to move that off https, it would be hard to justify the engineering time; especially post the move to FB infrastructure with its CDN that was easily configured for our attachments. OTOH, chat never ran on TLS, so adopting Noise vs adopting TLS was a choice we could consider, and we picked the best solution for us. Unfortunately, it's pretty easy to identify Noise vs TLS --- OTOH, the service IPs are already identifiable, so a little more blending on the protocol level wouldn't help much.
[1] https://www.whatsapp.com/security/WhatsApp-Security-Whitepap...
[2] Also using system TLS libraries is fraught with peril. It's fine, but not super great, for http, but using it for a custom binary protocol is going to be terrible. You'll need to debug all of the edge cases that the system https library doesn't hit, and will then have to craft workarounds that just work, even if you can't reliably identify the underlying versions because Android OEMs do weird stuff.