> Could it not also be the problematic traffic getting routed to a Cloudfront node that's overloaded or otherwise misbehaving?
This is plausible. Nothing I saw in the packet capture specifically pointed in this direction, but I don't really have the information I'd want to rule this out totally.
I've seen issues like this when the TCP stack on the server effectively runs out of allocated memory. This can cause the server not to use all of the available tcp window.
The only reason I think this is less likely, is I'd expect a company like AWS to monitor and tune the memory on their CDN, and notice something like a kernel bug here or block an attack exploiting the kernel memory.
Some network providers, especially wireless providers also deploy TCP acceleration equipment, so that network speeds are faster. On a wireless carrier if you've got temporary bad signal and drop some packets, if the carrier can retransmit them instead of going to the internet, this makes for a faster connection. Crappier implementations of these TCP accelerators, do a sort of transparent redirect to the kernel TCP stack. So if the ATT proxy has run out of kernel memory, it could be providing a slow connection. It wouldn't surprise me for a telecom carrier to miss and not tune or monitor kernel memory.
If the issue was caused by a proxy, I'd expect more to be broken then a single server, and other customers to also be complaining.
If ATT is using these accelerators, it also means the proxy could be hiding packet loss and delays between the proxy and upstream server or network. Because the client TCP connection is only to the proxy, we only get to see what happened between the client and proxy. So this precluded drawing strong conclusions from the client side capture only.
There is one thing about the packet capture that suggests a shaper to me. And that's how even the traffic is from server to client. While I didn't spend a bunch of time timing out all the packets, it looked a bit like I'd expect to see for packets getting released by a token bucket algorithm. This isn't necessarily safe to conclude, because the client has an offload that is buffering and merging packets using a receive segment offload, which causes us to miss some timing information in the capture.
So based on the above, shaping looks a little bit more likely to me, but I don't have the right information to rule out a broken or miss-configured server. An overloaded or otherwise broken network link like errors on a link doesn't appear likely to me at all.