Using the signed exchange mechanism means you allow anyone to serve your content. You will no longer know when it has been served and by whom. Instead, Google will know more about what your users are consuming on your website than you - despite HTTPS! Also, there is no mechanism to limit who is allowed to serve your content for you. I see no technical reason why the content has to be prefetched from Google instead of…
The reason it has to be prefetched from not-you is to protect the users privacy. Until they click a link it is not considered acceptable to leak their search to the potential destination. Links have to be fetched from a third-party who the search engine trusts not to share the data, that at the moment is Google but will hopefully expand.
I think the real reason is that Google wants to build a walled garden, but doesn't want the walls to be noticeable. Even with AMP, they display a header that looks like a browser's address bar [1]
Also, on that page Google admits that it uses AMP Viewer to collect information about users:
> Data collection by Google is governed by Google’s privacy policy.
Which is probably their real motication for creating AMP.
[1] https://developers.google.com/search/docs/guides/about-amp