Docker operations slowing down on AWS

In other words, if you do not rate limit yourself then others will rate limit you.

tldr: AWS EC2 has the concept of I/O credits for storage. If your instance runs out of credits, bad things, which may seem completely unrelated, will happen.

I was having similar issues last week and did not consider I/O credits. I think AWS could do better at notifying you if your EC2 instance gets into this state (without having to set up a cloud watch alarm).

Rather EBS performance is non-deterministic. This is definitely not new, but is an informative deep dive.

The other big gotcha is that EBS volumes are lazy loaded. Not necessarily something that's going to bite you in a production environment regularly, but it's something that could very easily throw off your benchmarks and performance testing.

Why not use an ephemeral volumes for the docker data. This is a CI system, so the docker images are all transient anyways. Seems like an easy way to avoid the I/O credits.

And then people consider me a dinosaur when I say, no cloud, just rent a server or two (not colo! just dedicated servers). Your average web service does not need to scale near infinitely; for the same amount of money you pay to Amazon you can overprovision 3-5-10 times and that'll handle your spikes. No surprises. Same amount of work: EC2 and bare metal both gives you a root prompt, go from there. These days you can get things provisioned with 24 hours -- some providers will do it in minutes. Of course, Amazon provides a lot of services beyond basic EC2 instances but if you use them you have a very ugly vendor lock and heaven forbid you want to do something else in the future...

This is like a double trap many try to sell to startups: a) you need to scale across many machines and b) the way to scale is the cloud. My take: a single machine (or two for HA) will be enough, if you really want to go big separate the web server from the database but that's it. And yes, I am in the website performance business, I worked on the video purchase platform of one of the largest British television stations and even that didn't require more than a single database server and a single Redis server for caching layer. Harken to https://stackoverflow.com/questions/5131266/increase-postgre... this question from 2011 discussing speeding up from like a thousand inserts per second at the cost of data loss -- today you will write on an SSD and don't need to risk data loss. Does your web site / app really get a thousand writes every second? I thought not. Does it even get a thousand reads? If not, then why are you building a complex database cluster...?

The other day I saw quad E7-4870 (yeah won't win any single thread contest but has 40 cores and 80 threads) 512GB RAM servers for $299 a month, with 1TB RAM for $499. Had a low end 2TB SSD for boot and you could add 8x1TB HDD w/ HW RAID for $40...