Take what I say with a grain of salt. There are many people that know more than me. But I'll try to answer to the best of my knowledge.
If a malicious app tries to alter the system in a bootloader relevant way, it would most likely fail. On those roms, apps don't have root rights, and users are even unable to activate a root account (part of why we need unlocked bootloaders in the first place to achieve user control over bought devices). But yes, as part of AVB system parts are hashed and a mismatch would be detected, see https://emteria.com/blog/android-verified-boot for a writeup.
For system apps, again two aspects. It's not that easy for an app to become a system app, it has to be moved to a specific place. Think about how the Gapps package is usually installed when you install a ROM, externally by the recovery system and not inside Android itself, that would be the reason. But yes, there are platform keys that the docs at https://source.android.com/docs/core/ota/sign_builds claim should be secret release keys.
About those release keys being also used for the system app verification, I think so. There are different keys on Android, like the release keys and the verity_key, but I think it follows from the docs that the release key is the one used to verify system apps (on modern Android versions).
It is debatable whether users not being able to exchange system apps then is a valid requirement for a FOSS Android distribution like /e/. But that position does exist, claiming users should build their ROM variants on their own with custom keys if they want to modify the system, to close this attack vector.