Funny coincidence, I was running into this exact issue earlier today. Had a customer complain about high response times from even our /time endpoint (which doesn't do anything except return server time) as measured by curl, and turns out it was just the TLS handshake:
$ curl -o /dev/null -s -w "@time-format.txt" http://rest.ably.io/time
time_namelookup: 0.012
time_connect: 0.031
time_appconnect: 0.000
time_pretransfer: 0.031
time_total: 0.053
$ curl -o /dev/null -s -w "@time-format.txt" https://rest.ably.io/time
time_namelookup: 0.012
time_connect: 0.031
time_appconnect: 0.216
time_pretransfer: 0.216
time_total: 0.237
Luckily not that much of an issue for us as when using an actual client library (unlike with curl) you get HTTP keep-alive, so at least the TCP connection doesn't need to be renewed for every request. And most customers who care about low latency are using a realtime library anyway, which just keeps a websocket, so sidesteps the whole issue. Certainly not enough to make us reconsider using TLS by default.
Still, a bit annoying when you get someone who thinks they've discovered with curl that latency from them to us is 4x slower than to Pubnub, just because the Pubnub docs show the http versions of their endpoints, wheras ours show https, even though we're basically both using the same set of AWS regions...