Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Docker operations slowing down on AWS

jeremyeder.com

Post reply on HN
Print view View original

81–90 of 176 posts

Re: Docker operations slowing down on AWS

#81

Post by _Codemonkeyism »

pizza234 wrote: post #65
_Codemonkeyism wrote: post #57

Earlier quoted context omitted.

I think people need to make a conscious tradeoff between ~5x AWS cost compared to rented server vs. OPS costs. Some startups (A) I work with have basically no OPS costs beyond setup, integrating Docker deployements and getting automatic backup working. Most simple technology just works and devs easily can do operations. The largest pain point still is VPN. Machines today are very very fast and load of many startups i…

I managed a metal to AWS transition, and 5x definitely doesn't match the costs I experienced (in this specific case, it was around the lines of 1.2x). I don't mean this case to be universal, in particular, I think cloud services force applications to have a particularly good/modular design (which is a cost in itself) - where, with metal, as you wrote, you can relatively cheaply overprovision. I think the analysis you…

Disclaimer: My experience is with growing startups younger than 5 years. So everything I say needs to be seen in that context.

From what did you transition to what? 20% sounds like a very small markup, from my experience and from other people experiences documented on the web it's much much larger (at 10x?) than your experience.

So I would be rather interested in more details as many people ask me about Amazon transitioning, and 20% markup would be killer.

"Some typical network/infrastructure elements, in particular firewalling, load balancing, and network management don't necessarily belong to the "rocket science" type of application;"

I surely do not know your demands, but firewalling, load balanicng etc. looks rather easy to me today for everyone except Google, Amazon, LinkedIn, AirBnB and 99% of startups are not one of these.

"I'm very skeptical for example, that the 5x figure includes hardware for the above network equipment and management."

Not sure what you are using, the hardware would be around $40 per month for that kind of network architecture (FW,HAProxy,Nginx,...).

In my last job we had large NetScalers which where much more powerful then HAProxy/Nginx on a rented server, and I assume AWS is as powerful, but for most of my clients this would be huge overkill.

Re: Docker operations slowing down on AWS

#82

Post by notacoward »

The debugging story was interesting, but what really sticks out for me is that Amazon has pretty tight QoS working for distributed storage. That's actually a really hard problem - much harder than its better known networking equivalent. As much as I might curse the Amazon business folks for using it to screw customers, I also have to give kudos to the engineers for implementing it.

Re: Docker operations slowing down on AWS

#83

Post by vidarh »

pbarnes_1 wrote: post #63
vacri wrote: post #15

Earlier quoted context omitted.

Ah, that nasty AWS vendor lockin. Guess I'll just run my own CDN then, shouldn't be hard (can't use other vendors, because lock-in). L'il ol me running edge nodes all over the five continents where we have clients. I'll have it done by closing time today. AWS is more than just compute cores, and you're hand-waving away what it offers.

CloudFront is one of the worst CDNs out there, so this isn't really a good point. Any CDN can be stood up in front of a small fleet of dedicated hosts, and you'd still be saving 80%.

In fact, if you are using AWS you really ought to put a proper caching CDN in front, given the crazy high bandwidth prices at AWS.

The bandwidth prices are high enough that I at one point mulled over setting up an "S3 compatible" storage service using S3 as the backend for durability, but storing a single local copy to most-of-the-time avoid hitting S3.

S3 bandwidth prices are high enough that there are huge cost savings to be had if objects are retrieved reasonably regularly.

Re: Docker operations slowing down on AWS

#84

Post by empath75 »

sideproject wrote: post #70
chx wrote: post #10

And then people consider me a dinosaur when I say, no cloud, just rent a server or two (not colo! just dedicated servers). Your average web service does not need to scale near infinitely; for the same amount of money you pay to Amazon you can overprovision 3-5-10 times and that'll handle your spikes. No surprises. Same amount of work: EC2 and bare metal both gives you a root prompt, go from there. These days you can…

I've been running a web application for the past 6 months and it just crossed 150,000 page views/month mark. Sure, for others it's not that great, but for me, this is the project that is showing the biggest potential. Anyway, the funny thing is, I'm running it on $2.50/month Vultr VPS. I got so worried when it crossed 30,000 that my site will crash. But it didn't. Then when its views got higher, I optimized further.…

150,000 views a month is like a hit every two seconds. That's nothing. Talk to me when you're managing 150000 hits a second.

Edit: sorry didn't mean to imply that your site isn't successful, only that in terms of traffic, it doesn't make aws worth it.

Re: Docker operations slowing down on AWS

#85

Post by vidarh »

f4rker wrote: post #27
chx wrote: post #10

And then people consider me a dinosaur when I say, no cloud, just rent a server or two (not colo! just dedicated servers). Your average web service does not need to scale near infinitely; for the same amount of money you pay to Amazon you can overprovision 3-5-10 times and that'll handle your spikes. No surprises. Same amount of work: EC2 and bare metal both gives you a root prompt, go from there. These days you can…

Your giving really general advice that fits YOUR needs and experience. Meanwhile, my company has been in business for 10+ years and we have 60+ severs streaming a shit to of data 24x7. Running "a few" dedicated servers is not some magic catch all for everyone.

If you're streaming any kind of volume of data from AWS, you're pretty much burning money compared to other services, unless you've secured steep discounts on the published prices.

I too have run networks for companies that's been in business 10+ years, with as many servers, and I've done that on both AWS, colo'd servers, managed servers, and hybrid setups of all of that, and I've yet to see an instance where AWS was cost effective at published prices for base load.

I have seen AWS be cost effectively used to handle spikes or batch jobs, and I have recommended it for clients that care more about the brand name (to tell their customers for example) than cost, or have very specific needs. I have also seen it used cost effectively once you get big enough to secure steep discounts.

Re: Docker operations slowing down on AWS

#86

Post by jazoom »

empath75 wrote: post #84
sideproject wrote: post #70

Earlier quoted context omitted.

I've been running a web application for the past 6 months and it just crossed 150,000 page views/month mark. Sure, for others it's not that great, but for me, this is the project that is showing the biggest potential. Anyway, the funny thing is, I'm running it on $2.50/month Vultr VPS. I got so worried when it crossed 30,000 that my site will crash. But it didn't. Then when its views got higher, I optimized further.…

150,000 views a month is like a hit every two seconds. That's nothing. Talk to me when you're managing 150000 hits a second. Edit: sorry didn't mean to imply that your site isn't successful, only that in terms of traffic, it doesn't make aws worth it.

I don't understand the point of your comment. The parent was simply explaining why they don't need extreme scalability with their scenario. No one is under the impression that multiplying their traffic by 75000 (so you work at Google then?) will not require a big server upgrade.

Re: Docker operations slowing down on AWS

#87

Post by empath75 »

pizza234 wrote: post #65
_Codemonkeyism wrote: post #57

Earlier quoted context omitted.

I think people need to make a conscious tradeoff between ~5x AWS cost compared to rented server vs. OPS costs. Some startups (A) I work with have basically no OPS costs beyond setup, integrating Docker deployements and getting automatic backup working. Most simple technology just works and devs easily can do operations. The largest pain point still is VPN. Machines today are very very fast and load of many startups i…

I managed a metal to AWS transition, and 5x definitely doesn't match the costs I experienced (in this specific case, it was around the lines of 1.2x). I don't mean this case to be universal, in particular, I think cloud services force applications to have a particularly good/modular design (which is a cost in itself) - where, with metal, as you wrote, you can relatively cheaply overprovision. I think the analysis you…

I used to work at aol, and I think 5x was on the low end of what we experienced, if you just forklifted an app over. You can bring that cost down eventually by rearchitecting and laying off/reassigning some ops guys, but that takes time.

Re: Docker operations slowing down on AWS

#88

Post by inopinatus »

vidarh wrote: post #76
inopinatus wrote: post #56

Earlier quoted context omitted.

If I followed this advice, it would hurt my startup badly. We would be investing far more time into system operations than I do now thanks to AWS's automation of standard stuff. My hosting bill would increase - partly because we can run on t2.micro instances, but that awfully glib advice about overprovisioning is most definitely asking for trouble; and we'd lose curated services like RDS and OpsWorks - which, by the…

I do contract devops for both physical servers (that includes me occasionally travelling to a data center), managed servers in a full service colo, and AWS, and the overall cost per server consistently ends up higher including all devops time for the AWS instance in my experience. I love it when customers pick AWS (though I usually advice not to, unless they have very specific needs), as my billable hours are way hig…

All our duelling anecdotes aside, you've picked out the one category I would most definitely not host on AWS: high constant outbound traffic sites. "Don't build your CDN on AWS", I can agree about that. Using public clouds for the spiky part of an otherwise predictable steady-state workload is also a great strategy. Also this:

> Their biggest benefit is to defer the initial setup of a self-managed cluster

And that, I think, is why (nearly) every startup, or indeed every new project, should begin on a public cloud. The capital waste of an incorrect server purchase can ruin a project and makes "fast failure" hard to stomach.

I don't move services away until an alternative is clearly both cheaper in 3yr NPV, and will not constrain future business opportunities. Except in cash cow operations where innovation has ceased, I rate the second criteria more important than the first and a compelling reason to stay on a public cloud.

Re: Docker operations slowing down on AWS

#89

Post by chrisan »

jopsen wrote: post #50
anonacct37 wrote: post #16

Earlier quoted context omitted.

Yes. Look at https://www.packet.net/bare-metal/ for bare metal boxes provisioned in less time than an ec2 instance. Or if you want to go old school, cheaper, and less sexy/api driven: https://www.delimiter.com/ If you shop, for around 30-40 a month you can get 16 cores, 32gb ram, and a 120GB ssd or 1-2TB spinny disk.

also scaleway.com (they have API, but no user-data). IMO, docker and container orchestration spells a bright future for bare-metal boxes like these, as you won't need cloudformation, etc.. But I still see few alternatives to S3, many vendors offers block devices, but only the big clouds offer blob storage. Backup and restore from blob storage makes recovery from crash pretty easy.

How does docker replace cloudformation? Don't you still need something that says "hey you are running out of capacity soon, time to add more hardware for your software to run on"

There has to be something that adds more bare-metal for your docker containers to run on when the existing bare-metal reaches capacity, right?

Re: Docker operations slowing down on AWS

#90

Post by empath75 »

jazoom wrote: post #86
empath75 wrote: post #84

Earlier quoted context omitted.

150,000 views a month is like a hit every two seconds. That's nothing. Talk to me when you're managing 150000 hits a second. Edit: sorry didn't mean to imply that your site isn't successful, only that in terms of traffic, it doesn't make aws worth it.

I don't understand the point of your comment. The parent was simply explaining why they don't need extreme scalability with their scenario. No one is under the impression that multiplying their traffic by 75000 (so you work at Google then?) will not require a big server upgrade.

He's making a judgement on the value of aws using a use case that makes no sense to use aws, anyway. He could be doing something similar with lightsail on aws at similar cost if he really wanted to, though.
Post reply on HN