Earlier quoted context omitted.
It's the ads. It's always the ads. The basic stuff loads lightning fast. And then the Javascript ad garbage starts loading. 20 seconds later and 15 domain lookups later, your page finally becomes responsive. So, what is happening is that AMP is removing everything except Google served ads and that makes everything nice and fast.
Maybe the problem is javascript
* Top bar with a damn small company logo and navigation links. Top bar does not follow the user down the page. Your logo isn't that cool and nobody is using your navigation links half way though an article.
* No fancy or dynamic layouts. No scrolling effects. No bars that follow them down the page. No floating buttons. No animations. No 'expandable' content. No hamburger menus (they're lazy design anyway).
* Minimal CSS. Sane font settings focusing on readability. No web fonts. Use the system default sans serif for the article content.
* Email solicitation exists as an in-band static form at the end of the page (or the beginning if you really can't help yourself).
* Use text placeholders for images and use JS to load them at the request of the user. Or more generally, only use JS to reduce the traffic over the wire.
* (If you must) Use a tracking pixel rather than something like GA or Piwik.