I've seen this kind of thing fail in so many ways just as a user with a custom domain. Someone adds my real email (the custom domain) to a Google invite and my primary Google Account uses its own email (a Gmail account) that is in no way my primary email address and Google gets confused and won't let me access the thing sent to my real email because I'm trying to access it from my Google Account. (Plus additional variations where my custom domain was briefly hosted on what today is called Google Workspace and some things still seem to be misconfigured to point to shutdown accounts for my real primary address still apparently ghosts in Google Workspace.)
That's eternally a frustrating experience. It ranks among the reasons I groan every time someone asks if we can meet in Google Meeting instead of other options.
That's just dealing with one large giant of a company who you would think if anyone would have the software resources to truly untangle email addresses from account assumptions, they would have more than enough resources available.
Lots of little papercuts like that exist everywhere else. I have multiple accounts with various "Pledge Managers" because I gave Kickstarter a specific filtered address but still tend to respond to surveys and other things sent more directly to me with my primary email address. "For my convenience" most of these "Pledge Managers" tie their accounts to whatever email address they get from a Kickstarter export spreadsheet and then assume that each email address is 1:1 useful for marking an account as mine. It's often easier to juggle a bunch of reminder notes in my Password Manager which campaign is with which account with which "Pledge Manager" than to unify accounts. It becomes the opposite of convenience.
I've seen the same thing happen with + addresses, not just custom addresses at custom domain addresses. Add a + to a signup to get autofiltering when you sign up for a service and most services see `accountname+filter@gmail.com` as a different email address than `accountname@gmail.com`. Which is the correct behavior because `+` filtering is a convention not a guarantee/standard. But it again adds complexity in these sorts of "auto-invites" when you are trying to auto-correlate based on specific email address.
It seems important to me to not assume that the address an invite is delivered to is the primary address for that person nor the intended address for their account with you. It may be useful to associate that delivered address automatically as a "secondary" address in some cases, but to insist that it be the primary on the account that results from the invite is a papercut friction for at least some of your potential users.