Docker operations slowing down on AWS • Hacker News Board

Re: Docker operations slowing down on AWS

#41

Post by Artemis2 » Wed, Jul 26, 2017, 7:34 AM UTC

BurritoAlPastor wrote: post #12

The author's takeaways include moving disks to io1. This is a bad bargain in most cases, and particularly bad in the ~500 IOPS range (which is what I'm seeing in the Grafana screenshot there). gp2 disks get 3 iops per gig "free", bursting up to 3k. (They don't burst after 1 tb, because your baseline performance is higher than the burst rate.) io1 is 25% more expensive per-gb, and you pay by the IOPS on top of that. A…

That's what we typically do to scale most EBS volumes (critical disk-intensive workloads run on io1 anyway). Instead of switching directly io1 we over-provision space which solves most IOPS issues.

We recently upgraded one of our etcd clusters from v2 to v3 and it turned out that etcd v3 uses significantly more IOPS than v2, using EBS burst credit during periods with normal load. Our solution was to increase the EBS volume sizes from 16GB to 50GB (100 IOPS -> 150 IOPS [1]). Our costs went from $1.6 per volume to $5, instead of $11.75 for io1, for a comparable performance for this particular application.

1: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolume...

Re: Docker operations slowing down on AWS

#42

Post by znep » Wed, Jul 26, 2017, 7:37 AM UTC

vacri wrote: post #14

How do you flip a volume on the fly? I thought you had to do the "snapshot > make new volume > reattach" route edit: thanks for the info, znep (hit my comment limit, hence the edit... )

Not any more, as of some number of months ago you can grow and change volume type on the fly especially if it isn't a boot volume.

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-expan...

Read the limitations very carefully, http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/considera... In particular if it was attached before Nov. 1, 2016 you need to do a one time stop and start of the instance or detach/reattach the volume and there are limits on instance types supported (but the definition of "current generation" is broader than you might assume at first).

This is quite an awesome enhancement, we were able to transparently convert a bunch of 15TB volumes from gp2 to st1 without downtime or impact to the app, and save a bunch of money.

Re: Docker operations slowing down on AWS

#43

Post by voltagex_ » Wed, Jul 26, 2017, 7:38 AM UTC

anonacct37 wrote: post #31

dman wrote: post #17

Earlier quoted context omitted.

That price for that config sounds amazing. Any pointers?

Right this second: https://cc.delimiter.com/cart/dedicated-servers/ $50/month 24GB ram, 16 threads, 2TB HD, gigabit uplink (20TB/month free). It's not hard to wait until they have a sale and/or coupons and/or pay upfront yearly to get a similar config for 30-40. (disclaimer: these servers are pretty "unmanaged") Or packet gives you a completely "cloud/api-driven" experience that's still on bare metal and reasonably p…

Where do you find the discounts/sales?

Re: Docker operations slowing down on AWS

#44

Post by clarkdave » Wed, Jul 26, 2017, 7:40 AM UTC

eikenberry wrote: post #8

Why not use an ephemeral volumes for the docker data. This is a CI system, so the docker images are all transient anyways. Seems like an easy way to avoid the I/O credits.

We do this for our CI and it works well; we use I3 (spot) instances, introduced earlier this year, which have large amounts of NVMe instance storage. It is essential to push everything you want to keep to S3, ECR, etc so it won't work for every case, but I'd recommend it if possible. And of course, if the instance does go down, your Docker cache goes with it so you'll suffer slow builds while it warms up.

The I3 instances are pricey, but if you can suffer possible CI outages and everything is self-healing, they're extremely cheap as spot instances.

Re: Docker operations slowing down on AWS

#45

Post by TurboHaskal » Wed, Jul 26, 2017, 7:42 AM UTC

chx wrote: post #10

And then people consider me a dinosaur when I say, no cloud, just rent a server or two (not colo! just dedicated servers). Your average web service does not need to scale near infinitely; for the same amount of money you pay to Amazon you can overprovision 3-5-10 times and that'll handle your spikes. No surprises. Same amount of work: EC2 and bare metal both gives you a root prompt, go from there. These days you can…

That might make sense for your (organisation's) business needs, but should you continue being that pragmatic you'll end up being an unemployable mess.

Just do the whole Docker thing, the more dependencies and lock-in the better, update your CV and move on.

Re: Docker operations slowing down on AWS

#46

Post by znep » Wed, Jul 26, 2017, 7:47 AM UTC

sudhirj wrote: post #32

While the article is factually correct, the tone strikes me as being disingenuous. The problem seems to be that the servers were running on gp2 disks, which offer a performance baseline with free short term bursts based on credits collected. The author has just realised that for consistent throughput, they would have to choose provisioned throughput and pay accordingly. This isn’t some conspiracy by AWS, though. It’s…

Exactly. I'd go further and say the burst capacity is actually a super useful and powerful feature that is very hard to get on your own hardware. Definitely can catch you unaware if you aren't on top of it and can get expensive for some needs, but as you say not hidden at all.

I have a hacky shell script I sometimes use for moderately sized environments that don't have better monitoring setup, it reports the minimum percent of burst balance remaining over the past day, requires the aws cli and parallel:

  #!/bin/bash
  while getopts "p:" opt; do
    case $opt in
      p)
        export AWS_PROFILE=$OPTARG
        ;;
      \?)
        echo "Invalid option: -$OPTARG" >&2
        ;;
    esac
  done

  if [ ! "$AWS_PROFILE" ]
  then
    echo "-p  or AWS_PROFILE env var required"
    exit 1
  fi

  start=$(date -v-1d +%Y-%m-%dT%H:%M:%S)
  end=$(date +%Y-%m-%dT%H:%M:%S)

  aws ec2 describe-volumes --output text --query 'join(`"\n"`, Volumes[*].VolumeId)' | parallel -j 20 "echo -ne {}\\\t && echo \$(aws cloudwatch get-metric-statistics --start-time $start --end-time $end --period 86400 --namespace AWS/EBS --statistics Minimum --metric-name BurstBalance --dimensions Name=VolumeId,Value={} --output text --query 'Datapoints[*].Minimum')" | sort -n -k 2

Re: Docker operations slowing down on AWS

#47

Post by cuu508 » Wed, Jul 26, 2017, 7:47 AM UTC

nodesocket wrote: post #36

anonacct37 wrote: post #16

Earlier quoted context omitted.

Yes. Look at https://www.packet.net/bare-metal/ for bare metal boxes provisioned in less time than an ec2 instance. Or if you want to go old school, cheaper, and less sexy/api driven: https://www.delimiter.com/ If you shop, for around 30-40 a month you can get 16 cores, 32gb ram, and a 120GB ssd or 1-2TB spinny disk.

Quick back of the napkin math, but Google Compute Engine (while it is shared) is significantly less expensive than Packet. This is just comparing purely on price though. Packet - $292/mo 4 cores 32 GB memory 128 GB SSD Google Compute Engine - $176/mo $156 - custom-4-32-extended 4 cores / 32 GB memory $20 - 120 GB SSD storage

"4 cores" on GCE means 4 hyperthreads, or 2 physical cores.

https://cloud.google.com/compute/docs/machine-types

> For the n1 series of machine types, a virtual CPU is implemented as a single hardware hyper-thread on a 2.6 GHz Intel Xeon E5 (Sandy Bridge), 2.5 GHz Intel Xeon E5 v2 (Ivy Bridge), 2.3 GHz Intel Xeon E5 v3 (Haswell), 2.2 GHz Intel Xeon E5 v4 (Broadwell), or 2.0 GHz Intel Skylake (Skylake).

Re: Docker operations slowing down on AWS

#48

Post by chris_overseas » Wed, Jul 26, 2017, 7:48 AM UTC

voltagex_ wrote: post #43

anonacct37 wrote: post #31

Earlier quoted context omitted.

Right this second: https://cc.delimiter.com/cart/dedicated-servers/ $50/month 24GB ram, 16 threads, 2TB HD, gigabit uplink (20TB/month free). It's not hard to wait until they have a sale and/or coupons and/or pay upfront yearly to get a similar config for 30-40. (disclaimer: these servers are pretty "unmanaged") Or packet gives you a completely "cloud/api-driven" experience that's still on bare metal and reasonably p…

Where do you find the discounts/sales?

Here I guess? https://cc.delimiter.com/cart/clearance-servers/&step=0

Re: Docker operations slowing down on AWS

#49

Post by Corrado » Wed, Jul 26, 2017, 7:57 AM UTC

Roritharr wrote: post #34

chx wrote: post #10

And then people consider me a dinosaur when I say, no cloud, just rent a server or two (not colo! just dedicated servers). Your average web service does not need to scale near infinitely; for the same amount of money you pay to Amazon you can overprovision 3-5-10 times and that'll handle your spikes. No surprises. Same amount of work: EC2 and bare metal both gives you a root prompt, go from there. These days you can…

The problem with renting boxes is the hidden costs if you want to do it right. First of all, if you have anything mission critical, you need to run it in a high availability config, this is easy for stateless microservices, but when it comes to running your DB, you start renting three boxes instead of one or two and configuring them accordingly. And then you setup your Backup Infrastructure for disaster recovery, Gla…

I completely agree with your line of reasoning and I have a few more thoughts on the matter. There are a ton of "soft" problems that arise when building out your stack. Licensing, warranties, and maintenance issues are things that never come up with a cloud provider. Then there are physical problems with building your own cabinet like cabling and power & heat management. The list just grows and grows.

Another big thing that rarely gets mentioned is research into exactly what hardware to purchase and how to configure it. Do you know what compute hardware you should purchase? There are 10s of vendors and thousands of options. What about network hardware? Do you know which switch is the best for your stack?

With a cloud vendor all of these questions disappear.

Even if you make a mistake in your cloud provisioning, it's easy to correct; just shut it down and start over. Make a mistake buying your own hardware and you have to live with it for 3 years or pay purchase more hardware.

I think people tend to forget or ignore all of these costs when evaluating cloud providers. You look at the total bill each month and are surprised that it cost that much. However, the costs for your custom built architecture are likely higher, but they are spread out over more time and more projects.

Re: Docker operations slowing down on AWS

#50

Post by jopsen » Wed, Jul 26, 2017, 7:58 AM UTC

anonacct37 wrote: post #16

chx wrote: post #10

And then people consider me a dinosaur when I say, no cloud, just rent a server or two (not colo! just dedicated servers). Your average web service does not need to scale near infinitely; for the same amount of money you pay to Amazon you can overprovision 3-5-10 times and that'll handle your spikes. No surprises. Same amount of work: EC2 and bare metal both gives you a root prompt, go from there. These days you can…

Yes. Look at https://www.packet.net/bare-metal/ for bare metal boxes provisioned in less time than an ec2 instance. Or if you want to go old school, cheaper, and less sexy/api driven: https://www.delimiter.com/ If you shop, for around 30-40 a month you can get 16 cores, 32gb ram, and a 120GB ssd or 1-2TB spinny disk.

also scaleway.com (they have API, but no user-data).

IMO, docker and container orchestration spells a bright future for bare-metal boxes like these, as you won't need cloudformation, etc..

But I still see few alternatives to S3, many vendors offers block devices, but only the big clouds offer blob storage. Backup and restore from blob storage makes recovery from crash pretty easy.