Apple has their own custom conventions and ABI (
https://developer.apple.com/documentation/xcode/writing-arm6...). I believe you are right that they have their own custom instructions too. The asahi Linux project has good documentation on their wiki about the hardware.
Also important here is the bespoke ASIC functionality on their CPUs - as it's a system on chip, you'll likely need to emulate the full SoC, including all the peripherals the system expects to be there. Unlike x86_64, ARM isn't really designed to do device discovery on extensible buses in the same way. Your peripherals should all be in the same place, many being on the chip on internal interconnects.
Emulating anything would be possible with enough effort. But you'd likely need to emulate a whole host of extra bespoke silicon functionality on the SoC to get it working. For example, given every ARM Mac has a hardware neural engine, you might find unmodified Mac OS assumes it's there and usable for core functionality (or does so in future). It would probably be a lot of work to emulate all of the extra SoC stuff to the point you could boot the OS unmodified. But nothing is impossible - I think iOS has been run in third party emulators etc.