As someone who works specifically on user authentication stuff...
The problem is often that there are multiple sources of truth for who the user is. And if you have an impersonation feature, you by definition have two sources of truth: who the user actually is, and who the user is impersonating. It would just be a matter of a single mistake of using the wrong one.
Considering that "view as" requires your page view to render every control as the impersonated user but only when it comes to your profile, but renders all controls outside of your profile as the original user, I could see any engineering team dealing with some very carefully drawn and potentially confusing boundary cases.
Edit: just to elaborate, it's not just obvious impersonation contexts where this gets interesting. For example, linking your Humble Bundle account to your Steam account, or on Netflix which user you are vs. which email address is being billed. Many apps have a function to share some document using a one-time expiring token. If you're also logged in, then do you read permissions from the shared token or from your account? If you mix them, do you make sure anything that writes to this shared view can't touch your account itself on accident? We don't think about it much but I think you can see how these subtle distinctions are important when you are thinking about access control, and that makes it a breeding ground for subtle mistakes.