Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Docker operations slowing down on AWS

jeremyeder.com

Post reply on HN
Print view View original

21–30 of 176 posts

Re: Docker operations slowing down on AWS

#21

Post by bdcravens »

vacri wrote: post #15
chx wrote: post #10

And then people consider me a dinosaur when I say, no cloud, just rent a server or two (not colo! just dedicated servers). Your average web service does not need to scale near infinitely; for the same amount of money you pay to Amazon you can overprovision 3-5-10 times and that'll handle your spikes. No surprises. Same amount of work: EC2 and bare metal both gives you a root prompt, go from there. These days you can…

Ah, that nasty AWS vendor lockin. Guess I'll just run my own CDN then, shouldn't be hard (can't use other vendors, because lock-in). L'il ol me running edge nodes all over the five continents where we have clients. I'll have it done by closing time today. AWS is more than just compute cores, and you're hand-waving away what it offers.

I think the comment does ring true in many cases however, where a minimal infrastructure is more than sufficient. We recently received notice that an EC2 instance was being retired and we needed to relaunch. It had no load balancing; it was just an m1.medium with a standard Java app on Ubuntu, with a static IP address, and it ran for like 5 years. No elasticity etc. Essentially 1:1 for a standard "old school" setup.

Re: Docker operations slowing down on AWS

#22

Post by Dunedan »

eikenberry wrote: post #8

Why not use an ephemeral volumes for the docker data. This is a CI system, so the docker images are all transient anyways. Seems like an easy way to avoid the I/O credits.

The most recent "general purpose" instance families (C4, G3, M4, P2, R4) don't offer ephemeral storage anymore. My guess is that AWS will only offer ephemeral storage for a few selected instance families in future.

So you have to decide: older instance types (which will become more expensive as they usually don't benefit from new price reductions) or no ephemeral storage

Re: Docker operations slowing down on AWS

#23

Post by not_kurt_godel »

bdcravens wrote: post #21
vacri wrote: post #15

Earlier quoted context omitted.

Ah, that nasty AWS vendor lockin. Guess I'll just run my own CDN then, shouldn't be hard (can't use other vendors, because lock-in). L'il ol me running edge nodes all over the five continents where we have clients. I'll have it done by closing time today. AWS is more than just compute cores, and you're hand-waving away what it offers.

I think the comment does ring true in many cases however, where a minimal infrastructure is more than sufficient. We recently received notice that an EC2 instance was being retired and we needed to relaunch. It had no load balancing; it was just an m1.medium with a standard Java app on Ubuntu, with a static IP address, and it ran for like 5 years. No elasticity etc. Essentially 1:1 for a standard "old school" setup.

In other words, not really an appropriate (or at least not cost-efficient) use of EC2. EC2 != VPSes

Re: Docker operations slowing down on AWS

#24

Post by morecoffee »

rixed wrote: post #2

In other words, if you do not rate limit yourself then others will rate limit you.

It's called out in the post: most applications are not written to deal with reads / writes suddenly dropping in throughput. How are you supposed to rate limit the IOPS of a program you don't control? Can you make `docker' do exponential backoff if it notices some operations are taking too long?

In a network scenario, the remote service could say something like Quota exceeded, please try again later. read and write syscalls don't really work like that. They instead remain in uninterruptible sleep, which means they can't wake up, or be killed or stop.

Re: Docker operations slowing down on AWS

#25

Post by chx »

maximz wrote: post #13
chx wrote: post #10

And then people consider me a dinosaur when I say, no cloud, just rent a server or two (not colo! just dedicated servers). Your average web service does not need to scale near infinitely; for the same amount of money you pay to Amazon you can overprovision 3-5-10 times and that'll handle your spikes. No surprises. Same amount of work: EC2 and bare metal both gives you a root prompt, go from there. These days you can…

Rent a server: do you mean dedicated hosting or something else?

Dedi, yes.

Re: Docker operations slowing down on AWS

#27

Post by f4rker »

chx wrote: post #10

And then people consider me a dinosaur when I say, no cloud, just rent a server or two (not colo! just dedicated servers). Your average web service does not need to scale near infinitely; for the same amount of money you pay to Amazon you can overprovision 3-5-10 times and that'll handle your spikes. No surprises. Same amount of work: EC2 and bare metal both gives you a root prompt, go from there. These days you can…

Your giving really general advice that fits YOUR needs and experience. Meanwhile, my company has been in business for 10+ years and we have 60+ severs streaming a shit to of data 24x7.

Running "a few" dedicated servers is not some magic catch all for everyone.

Re: Docker operations slowing down on AWS

#28

Post by m00x »

inferiorhuman wrote: post #6
shusson wrote: post #4

tldr: AWS EC2 has the concept of I/O credits for storage. If your instance runs out of credits, bad things, which may seem completely unrelated, will happen. I was having similar issues last week and did not consider I/O credits. I think AWS could do better at notifying you if your EC2 instance gets into this state (without having to set up a cloud watch alarm).

Rather EBS performance is non-deterministic. This is definitely not new, but is an informative deep dive. The other big gotcha is that EBS volumes are lazy loaded. Not necessarily something that's going to bite you in a production environment regularly, but it's something that could very easily throw off your benchmarks and performance testing.

Should you ever assume IO to be deterministic? Hard drives can crash, they can be slow(especially magnetic). Networks can disconnect, become congested or have random interference.

This is amplified on the cloud, but it should also be well known by anyone that works with the cloud. These things are well documented, although I could see why you don't really look for it until you have issues with it.

Re: Docker operations slowing down on AWS

#29

Post by flukus »

anonacct37 wrote: post #16
chx wrote: post #10

And then people consider me a dinosaur when I say, no cloud, just rent a server or two (not colo! just dedicated servers). Your average web service does not need to scale near infinitely; for the same amount of money you pay to Amazon you can overprovision 3-5-10 times and that'll handle your spikes. No surprises. Same amount of work: EC2 and bare metal both gives you a root prompt, go from there. These days you can…

Yes. Look at https://www.packet.net/bare-metal/ for bare metal boxes provisioned in less time than an ec2 instance. Or if you want to go old school, cheaper, and less sexy/api driven: https://www.delimiter.com/ If you shop, for around 30-40 a month you can get 16 cores, 32gb ram, and a 120GB ssd or 1-2TB spinny disk.

A raspberry pi would also be more than enough for many use cases and can be hosted for ~$40/year: https://raspberry-hosting.com/en/order.

I'd have one (or more) already if I didn't live on the opposite side of the world to the data center.

Re: Docker operations slowing down on AWS

#30

Post by x7467 »

We hit the IOPS limit on AWS many times, both on VMs and SQL instances. The solution was always to artificially inflate the size of the underlying storage, as you get 3 IOPS per GB on persistent disks (the other option was to buy IOPS, but this somehow always turned out to be way more expensive).

This issue was on a "pros" section when we decided to move our operations to GCP, when you get 30 IOPS per GB on persistent storage, so 10x more than on AWS. One way or another, if you really need _a lot_ of IOPS, you better stick with a local (ephemeral) SDD storage – just bear in mind it will vanish along with your VMs.

Post reply on HN