Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Docker operations slowing down on AWS

jeremyeder.com

Post reply on HN
Print view View original

101–110 of 176 posts

Re: Docker operations slowing down on AWS

#101

Post by problems »

empath75 wrote: post #90
jazoom wrote: post #86

Earlier quoted context omitted.

I don't understand the point of your comment. The parent was simply explaining why they don't need extreme scalability with their scenario. No one is under the impression that multiplying their traffic by 75000 (so you work at Google then?) will not require a big server upgrade.

He's making a judgement on the value of aws using a use case that makes no sense to use aws, anyway. He could be doing something similar with lightsail on aws at similar cost if he really wanted to, though.

> He's making a judgement on the value of aws using a use case that makes no sense to use aws,

I think the problem here is that a lot of people use AWS when there's no value in using AWS. It's commonly reached for as a first choice when it's often not a good option if you're looking to run lean.

AWS is extremely popular, but probably only efficient for a small percent of companies who have wildly variable traffic patterns.

Re: Docker operations slowing down on AWS

#102

Post by mbrock »

vidarh wrote: post #91
inopinatus wrote: post #88

Earlier quoted context omitted.

All our duelling anecdotes aside, you've picked out the one category I would most definitely not host on AWS: high constant outbound traffic sites. "Don't build your CDN on AWS", I can agree about that. Using public clouds for the spiky part of an otherwise predictable steady-state workload is also a great strategy. Also this: > Their biggest benefit is to defer the initial setup of a self-managed cluster And that, I…

I tend to favour starting with rented servers too, rather than purchases (or leases, rather) - whether that's on AWS, or rented by the month then becomes a much simpler comparison. As for the CDN, we absolutely agree. I noted elsewhere that even if you host on AWS, if you have any kind of volume of outbound bandwidth use, you should probably get a CDN elsewhere whether or not you think you need a CDN. A good caching…

What about CloudFront?

Re: Docker operations slowing down on AWS

#103

Post by AYBABTME »

sideproject wrote: post #70
chx wrote: post #10

And then people consider me a dinosaur when I say, no cloud, just rent a server or two (not colo! just dedicated servers). Your average web service does not need to scale near infinitely; for the same amount of money you pay to Amazon you can overprovision 3-5-10 times and that'll handle your spikes. No surprises. Same amount of work: EC2 and bare metal both gives you a root prompt, go from there. These days you can…

I've been running a web application for the past 6 months and it just crossed 150,000 page views/month mark. Sure, for others it's not that great, but for me, this is the project that is showing the biggest potential. Anyway, the funny thing is, I'm running it on $2.50/month Vultr VPS. I got so worried when it crossed 30,000 that my site will crash. But it didn't. Then when its views got higher, I optimized further.…

Not sure what your website is doing, but here's a quick thought exercise:

If you serve a page in 10s, then you can serve 259'200 pages/month.

    render_time page_view/month
    10s         259'200
    9s          288,000
    8s          324,000
    7s          370,285
    6s          432,000
    5s          518,400
    4s          648,000
    3s          864,000
    2s          1,296,000
    1s          2,592,000
So like, think 1 million views per month with 2s render time.

This is obviously skipping over a ton of details, but it's a good rule of thumb.

Re: Docker operations slowing down on AWS

#104

Post by falcolas »

breerly wrote: post #51
chx wrote: post #10

And then people consider me a dinosaur when I say, no cloud, just rent a server or two (not colo! just dedicated servers). Your average web service does not need to scale near infinitely; for the same amount of money you pay to Amazon you can overprovision 3-5-10 times and that'll handle your spikes. No surprises. Same amount of work: EC2 and bare metal both gives you a root prompt, go from there. These days you can…

Cloud services don't save you money, they buy you focus.

Only slightly tongue in cheek: Imagine the focus you could achieve with dedicated personnel taking care of all of the operations concerns for your devs...

Re: Docker operations slowing down on AWS

#105

Post by draw_down »

chx wrote: post #10

And then people consider me a dinosaur when I say, no cloud, just rent a server or two (not colo! just dedicated servers). Your average web service does not need to scale near infinitely; for the same amount of money you pay to Amazon you can overprovision 3-5-10 times and that'll handle your spikes. No surprises. Same amount of work: EC2 and bare metal both gives you a root prompt, go from there. These days you can…

Knock yourself out. But, not everybody is doing the average.

Re: Docker operations slowing down on AWS

#106

Post by falcolas »

sudhirj wrote: post #32

While the article is factually correct, the tone strikes me as being disingenuous. The problem seems to be that the servers were running on gp2 disks, which offer a performance baseline with free short term bursts based on credits collected. The author has just realised that for consistent throughput, they would have to choose provisioned throughput and pay accordingly. This isn’t some conspiracy by AWS, though. It’s…

The addition of the burst mechanism to EBS volumes is relatively new. If you have been running on AWS for any reasonable amount of time, the sudden degradation of your disks will leave you scratching your head.

Dedicated IOPS wasn't originally pitched as the alternative to burst disks - it was simply that normal ELBs were not predictable based on peer load, and dedicated IOPS ELBs were.

Re: Docker operations slowing down on AWS

#107

Post by geofft »

empath75 wrote: post #84
sideproject wrote: post #70

Earlier quoted context omitted.

I've been running a web application for the past 6 months and it just crossed 150,000 page views/month mark. Sure, for others it's not that great, but for me, this is the project that is showing the biggest potential. Anyway, the funny thing is, I'm running it on $2.50/month Vultr VPS. I got so worried when it crossed 30,000 that my site will crash. But it didn't. Then when its views got higher, I optimized further.…

150,000 views a month is like a hit every two seconds. That's nothing. Talk to me when you're managing 150000 hits a second. Edit: sorry didn't mean to imply that your site isn't successful, only that in terms of traffic, it doesn't make aws worth it.

How many projects actually get 150,000 hits a second?

(And are they not better served by buying and operating their own datacenters?)

Re: Docker operations slowing down on AWS

#108

Post by falcolas »

BurritoAlPastor wrote: post #12

The author's takeaways include moving disks to io1. This is a bad bargain in most cases, and particularly bad in the ~500 IOPS range (which is what I'm seeing in the Grafana screenshot there). gp2 disks get 3 iops per gig "free", bursting up to 3k. (They don't burst after 1 tb, because your baseline performance is higher than the burst rate.) io1 is 25% more expensive per-gb, and you pay by the IOPS on top of that. A…

> Also, your burst balance is available as a Cloudwatch metric, as I recall! Set alarms on that shit!

That's my takeaway as well. If you can track and alert on it, you can provision whatever you want and only buy the expensive io1 when it makes sense.

Re: Docker operations slowing down on AWS

#109

Post by falcolas »

krallja wrote: post #92
shusson wrote: post #4

tldr: AWS EC2 has the concept of I/O credits for storage. If your instance runs out of credits, bad things, which may seem completely unrelated, will happen. I was having similar issues last week and did not consider I/O credits. I think AWS could do better at notifying you if your EC2 instance gets into this state (without having to set up a cloud watch alarm).

> I think AWS could do better at notifying you if your EC2 instance gets into this state (without having to set up a cloud watch alarm). How would you configure these notifications? Who do they get sent to? How often? Do they escalate to someone else? When? Once you answer these questions, you have come up with ... CloudWatch.

Surfacing the metric on the host itself, and not buried in the EBS volume, would be a good start.

WRT alerting - if you're at 0 credits; your performance on something obviously under load is now being degraded, why not surface an alert on it? AWS already has a configurable events system in place for EC2 - this seems like a good fit.

Re: Docker operations slowing down on AWS

#110

Post by falcolas »

m00x wrote: post #28
inferiorhuman wrote: post #6

Earlier quoted context omitted.

Rather EBS performance is non-deterministic. This is definitely not new, but is an informative deep dive. The other big gotcha is that EBS volumes are lazy loaded. Not necessarily something that's going to bite you in a production environment regularly, but it's something that could very easily throw off your benchmarks and performance testing.

Should you ever assume IO to be deterministic? Hard drives can crash, they can be slow(especially magnetic). Networks can disconnect, become congested or have random interference. This is amplified on the cloud, but it should also be well known by anyone that works with the cloud. These things are well documented, although I could see why you don't really look for it until you have issues with it.

I think the problem is that from your average application developer's point view - there's nothing that they can really do that they aren't already doing (unless they're blocking the main thread on disk IO, in which case they can do a little bit more).
Post reply on HN