Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Docker operations slowing down on AWS

jeremyeder.com

Post reply on HN
Print view View original

61–70 of 176 posts

Re: Docker operations slowing down on AWS

#61

Post by viraptor »

inferiorhuman wrote: post #6
shusson wrote: post #4

tldr: AWS EC2 has the concept of I/O credits for storage. If your instance runs out of credits, bad things, which may seem completely unrelated, will happen. I was having similar issues last week and did not consider I/O credits. I think AWS could do better at notifying you if your EC2 instance gets into this state (without having to set up a cloud watch alarm).

Rather EBS performance is non-deterministic. This is definitely not new, but is an informative deep dive. The other big gotcha is that EBS volumes are lazy loaded. Not necessarily something that's going to bite you in a production environment regularly, but it's something that could very easily throw off your benchmarks and performance testing.

Can't find it in the docs now, but the certification course I'm doing now claims the lazy loading is not happening anymore. Apparently prewarming volumes is not necessary these days unless you created the volume from a snapshot.

Re: Docker operations slowing down on AWS

#62

Post by jopsen »

sandromehic wrote: post #54
jopsen wrote: post #50

Earlier quoted context omitted.

also scaleway.com (they have API, but no user-data). IMO, docker and container orchestration spells a bright future for bare-metal boxes like these, as you won't need cloudformation, etc.. But I still see few alternatives to S3, many vendors offers block devices, but only the big clouds offer blob storage. Backup and restore from blob storage makes recovery from crash pretty easy.

How do open source solutions like minio.io on bare metal compare to s3? I don't have much experience with any of those, but it seems like a compatible solution.

I haven't tried it either... but the nice thing with S3 is zero maintenance. A disk or machine dies it's not my problem.

The not-my-problem aspect is really powerful.

Re: Docker operations slowing down on AWS

#63

Post by pbarnes_1 »

vacri wrote: post #15
chx wrote: post #10

And then people consider me a dinosaur when I say, no cloud, just rent a server or two (not colo! just dedicated servers). Your average web service does not need to scale near infinitely; for the same amount of money you pay to Amazon you can overprovision 3-5-10 times and that'll handle your spikes. No surprises. Same amount of work: EC2 and bare metal both gives you a root prompt, go from there. These days you can…

Ah, that nasty AWS vendor lockin. Guess I'll just run my own CDN then, shouldn't be hard (can't use other vendors, because lock-in). L'il ol me running edge nodes all over the five continents where we have clients. I'll have it done by closing time today. AWS is more than just compute cores, and you're hand-waving away what it offers.

CloudFront is one of the worst CDNs out there, so this isn't really a good point.

Any CDN can be stood up in front of a small fleet of dedicated hosts, and you'd still be saving 80%.

Re: Docker operations slowing down on AWS

#64

Post by chx »

voltagex_ wrote: post #43
anonacct37 wrote: post #31

Earlier quoted context omitted.

Right this second: https://cc.delimiter.com/cart/dedicated-servers/ $50/month 24GB ram, 16 threads, 2TB HD, gigabit uplink (20TB/month free). It's not hard to wait until they have a sale and/or coupons and/or pay upfront yearly to get a similar config for 30-40. (disclaimer: these servers are pretty "unmanaged") Or packet gives you a completely "cloud/api-driven" experience that's still on bare metal and reasonably p…

Where do you find the discounts/sales?

webhostingtalk

Re: Docker operations slowing down on AWS

#65

Post by pizza234 »

_Codemonkeyism wrote: post #57
inopinatus wrote: post #56

Earlier quoted context omitted.

If I followed this advice, it would hurt my startup badly. We would be investing far more time into system operations than I do now thanks to AWS's automation of standard stuff. My hosting bill would increase - partly because we can run on t2.micro instances, but that awfully glib advice about overprovisioning is most definitely asking for trouble; and we'd lose curated services like RDS and OpsWorks - which, by the…

I think people need to make a conscious tradeoff between ~5x AWS cost compared to rented server vs. OPS costs. Some startups (A) I work with have basically no OPS costs beyond setup, integrating Docker deployements and getting automatic backup working. Most simple technology just works and devs easily can do operations. The largest pain point still is VPN. Machines today are very very fast and load of many startups i…

I managed a metal to AWS transition, and 5x definitely doesn't match the costs I experienced (in this specific case, it was around the lines of 1.2x).

I don't mean this case to be universal, in particular, I think cloud services force applications to have a particularly good/modular design (which is a cost in itself) - where, with metal, as you wrote, you can relatively cheaply overprovision.

I think the analysis you're making overlooks some important characteristics of the infrastructure engineering aspect.

Some typical network/infrastructure elements, in particular firewalling, load balancing, and network management don't necessarily belong to the "rocket science" type of application; they are easy to overlook in "type A" services, ending with a "kind-of-HA-but-not-really" infrastructure, which is ok, but it makes the comparison cloud metal not really meaningful, as in the cloud, those features are baked in ("almost" for free).

I'm very skeptical for example, that the 5x figure includes hardware for the above network equipment and management.

To summarize, it's perfectly fine not to have an "advanced" infrastructure but it must be highlighted that such conditions make a direct comparison incorrect.

Re: Docker operations slowing down on AWS

#66

Post by geff82 »

dman wrote: post #17
anonacct37 wrote: post #16

Earlier quoted context omitted.

Yes. Look at https://www.packet.net/bare-metal/ for bare metal boxes provisioned in less time than an ec2 instance. Or if you want to go old school, cheaper, and less sexy/api driven: https://www.delimiter.com/ If you shop, for around 30-40 a month you can get 16 cores, 32gb ram, and a 120GB ssd or 1-2TB spinny disk.

That price for that config sounds amazing. Any pointers?

32€ quadcore 32GB RAM, 2x3TB HDD: hetzner.de in their "second hand market"

Re: Docker operations slowing down on AWS

#67

Post by revelation »

laumars wrote: post #55
chx wrote: post #10

And then people consider me a dinosaur when I say, no cloud, just rent a server or two (not colo! just dedicated servers). Your average web service does not need to scale near infinitely; for the same amount of money you pay to Amazon you can overprovision 3-5-10 times and that'll handle your spikes. No surprises. Same amount of work: EC2 and bare metal both gives you a root prompt, go from there. These days you can…

I don't disagree with your point per se as I do love working with physical gear, but I do think you're grossly missing the point in places: > "My take: a single machine (or two for HA) will be enough" 2 bare metal instances isn't HA. Not even close. > "if you really want to go big separate the web server from the database but that's it." I would always recommend separating the web server from the database server on a…

Let's be real on high availability. If you are honest with yourself, on the cloud that doesn't mean 2.. AWS regions but 2.. cloud providers. It's a yearly occurrence now that 90% of SaaS stop working because AWS is broken, and it's not any of the actually redudant parts like power supplies that are broken, but because a human pushed software or configuration and the whole thing came crashing down.

Re: Docker operations slowing down on AWS

#68

Post by RX14 »

jopsen wrote: post #50
anonacct37 wrote: post #16

Earlier quoted context omitted.

Yes. Look at https://www.packet.net/bare-metal/ for bare metal boxes provisioned in less time than an ec2 instance. Or if you want to go old school, cheaper, and less sexy/api driven: https://www.delimiter.com/ If you shop, for around 30-40 a month you can get 16 cores, 32gb ram, and a 120GB ssd or 1-2TB spinny disk.

also scaleway.com (they have API, but no user-data). IMO, docker and container orchestration spells a bright future for bare-metal boxes like these, as you won't need cloudformation, etc.. But I still see few alternatives to S3, many vendors offers block devices, but only the big clouds offer blob storage. Backup and restore from blob storage makes recovery from crash pretty easy.

b2 is a really good alternative to s3.

Re: Docker operations slowing down on AWS

#69

Post by user5994461 »

Performance management 101 on AWS volumes:

1) The IO of a disks is proportional to the size of the volume. You need to get bigger volumes to get more performances. 3 io/GB

2) The high performance volumes (io1/PIOPS) are extortion. It's cheaper to pay for a bigger regular volume (gp2) that comes with a higher IO quota than to pay for the special high performance volume.

3) Each instances type has a disk performance cap. They are lower than you think.

4) Don't use t2 instances for anything that requires non negligible sustained IO.

P.S. Clearly the author is just discovering AWS.

Re: Docker operations slowing down on AWS

#70

Post by sideproject »

chx wrote: post #10

And then people consider me a dinosaur when I say, no cloud, just rent a server or two (not colo! just dedicated servers). Your average web service does not need to scale near infinitely; for the same amount of money you pay to Amazon you can overprovision 3-5-10 times and that'll handle your spikes. No surprises. Same amount of work: EC2 and bare metal both gives you a root prompt, go from there. These days you can…

I've been running a web application for the past 6 months and it just crossed 150,000 page views/month mark. Sure, for others it's not that great, but for me, this is the project that is showing the biggest potential.

Anyway, the funny thing is, I'm running it on $2.50/month Vultr VPS. I got so worried when it crossed 30,000 that my site will crash. But it didn't. Then when its views got higher, I optimized further. I learned more things about how to make things faster. I refactored my code to squeeze out that much more juice out of $2.50 - not because I'm cheap (well, perhaps), but because I wanted to see how far I can push it.

I'll probably cross 200,000 soon and I think by then I'll need to increase it to (yikes heaven forbid!) $5/month plan.

Recently, just because of all the hype about AWS (in general and also at my work) I wanted to get started with AWS. Then I looked at their ridiculous pricing calculation page and I just closed the browser. I thought my time was better spent working on my project than learning about how to deploy a simple PHP application to AWS.

Post reply on HN