Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Docker operations slowing down on AWS

jeremyeder.com

Post reply on HN
Print view View original

131–140 of 176 posts

Re: Docker operations slowing down on AWS

#131

Post by problems »

eropple wrote: post #130
problems wrote: post #128

Earlier quoted context omitted.

AWS tooling doesn't save developer time in my experience. In fact, I'd argue that optimizing for it more often than not wastes it. I've watched someone spend days learning to configure a performant DB server on AWS due to their poor disk IO performance and spend an enormous amount to get a high ram instance when a simple SSD based server where more IOPS were trivially available would have had it working out of the bo…

So I'm not trying to big-time you, but I have experience across a wide range of environments and shop sizes (both in clouds and, unfortunately, people who bought the "VPSes are fine too" idea in like 2015) and, after being in these trenches for a while, "it doesn't improve developer productivity" reads more to me as "we don't know how to leverage AWS for developer productivity." Elasticity is nice; pervasive automati…

I think you're either focusing on a very small part of the market - very large websites - or you're overestimating the requirements most websites have. Either way, it seems like you're taking about a very different scenario than I am.

Endless inventory and minimized administration but higher upfront and ongoing server investment is not a benefit when you need to write a website used by under 10k people a day. Nor is it a benefit for internal services moved to the cloud for nonsense reasons. Both of which I've seen done far too frequently. Simple tooling and small virtual machines or singular dedicated servers like are easily enough for these applications.

I can definitely understand the need for a small team working on a very large, high hit count application though. I'm not saying there are no uses for it, there totally are and you make valid points, just that the uses more limited than one might expect by the hype.

Re: Docker operations slowing down on AWS

#132

Post by ivanhoe »

sly010 wrote: post #124
ivanhoe wrote: post #123

Earlier quoted context omitted.

It's not just about scaling up, but also down. In my experience most of the startups initially greatly over-estimate what they need, and end up with only small part of the resources really used. With cloud you can provision what you think you'll need for a few weeks and then, once you get actual usage data, you can scale and fine-tune the setup to fully utilize it (and optimize the costs). And it's done with a few cl…

> With cloud you can provision what you think you'll need for a few weeks and then, once you get actual usage data, you can scale and fine-tune the setup to fully utilize it (and optimize the costs). Or in a more realistic scenario you forget to dial back an are paying $200/month extra for unused provisioned IO for years before anyone notices it. Happened, even though I was looking for improvements after every bill.

True, but did you ever scale down an already setup & running dedicated machine? With dedicated hardware it's always like "if it works don't touch, and we might need it one day anyway" :)

Re: Docker operations slowing down on AWS

#133

Post by eropple »

problems wrote: post #131
eropple wrote: post #130

Earlier quoted context omitted.

So I'm not trying to big-time you, but I have experience across a wide range of environments and shop sizes (both in clouds and, unfortunately, people who bought the "VPSes are fine too" idea in like 2015) and, after being in these trenches for a while, "it doesn't improve developer productivity" reads more to me as "we don't know how to leverage AWS for developer productivity." Elasticity is nice; pervasive automati…

I think you're either focusing on a very small part of the market - very large websites - or you're overestimating the requirements most websites have. Either way, it seems like you're taking about a very different scenario than I am. Endless inventory and minimized administration but higher upfront and ongoing server investment is not a benefit when you need to write a website used by under 10k people a day. Nor is…

So...I run one service (gratis, as it's a friend) that gets about 8K uniques a day. It costs $26 a month in AWS (it was $35 but that's apparently gone down, cool!) and makes about $400/month. That price tag is despite me not being particularly cost-conscious when rolling it out. It does use free tier AWS resources, but EC2 is not in free tier right now; we're talking about keeping DynamoDB at free tier read levels and stuff like that.

And, unlike the still-really-weird-and-ad-hoc VPS world, stuff like DR is a solved problem when-not-if you need it. The vig for AWS is consistently between 20% and 25% and you are able to leverage implicitly all the incredibly useful tooling and systems around you. If 20% to 25% of $26 is going to materially damage your business, you do not have a business.

Re: Docker operations slowing down on AWS

#134

Post by pizza234 »

hinkley wrote: post #129
empath75 wrote: post #87

Earlier quoted context omitted.

I used to work at aol, and I think 5x was on the low end of what we experienced, if you just forklifted an app over. You can bring that cost down eventually by rearchitecting and laying off/reassigning some ops guys, but that takes time.

I really struggle with this color of money nonsense that western finance has invented. Tell me again why hiring a developer is cheaper than buying one $8k server that will save us 35 man hours a week? Drives me batty. Rearchitecting may be necessary anyway, but doing this kind of work involves hiring and training better senior devs and retaining them for a couple of years at least. That's not cheap. It's A lot more e…

It's not so black and white. The changes mentioned may be a necessary evil when scaling; in this perspective, moving to the cloud makes a set of solutions (rearchitecting) happen sooner than later.

Also, an 8k server is a big unit. Cloud services are much more granular. This is a problem of bare metal - it's easy to overprovision because the base unit is large, and one ends up being happy of having an "overprovisioned" system, when in reality it's money down the drain.

Also, you don't count the management of the 8k server. It may (but not necessarily) be at a click distance; if it is, the management hardware (eg. one from a very famous servers producer) may have a poor software.

There are reasons why, in some cases (of course, not in all or not in many), cloud may be more advantageous than an "8k" bare metal server.

All in all, I think without numbers, talking about metal vs. cloud in abstract, generic terms, makes a poor argument.

Re: Docker operations slowing down on AWS

#135

Post by qaq »

eropple wrote: post #130
problems wrote: post #128

Earlier quoted context omitted.

AWS tooling doesn't save developer time in my experience. In fact, I'd argue that optimizing for it more often than not wastes it. I've watched someone spend days learning to configure a performant DB server on AWS due to their poor disk IO performance and spend an enormous amount to get a high ram instance when a simple SSD based server where more IOPS were trivially available would have had it working out of the bo…

So I'm not trying to big-time you, but I have experience across a wide range of environments and shop sizes (both in clouds and, unfortunately, people who bought the "VPSes are fine too" idea in like 2015) and, after being in these trenches for a while, "it doesn't improve developer productivity" reads more to me as "we don't know how to leverage AWS for developer productivity." Elasticity is nice; pervasive automati…

That reads more like AWS marketing copy :) At scale you have to architect around AWS crapy network.

0 network transparency

abismal IOPS performance

very limited config. options

Horrible uptime (US East has worth uptime as a region than wast majority of quality DCs)

0 Access to people who can really help you (unless you are at several mil. per month spend)

Re: Docker operations slowing down on AWS

#136

Post by r1ch »

chx wrote: post #10

And then people consider me a dinosaur when I say, no cloud, just rent a server or two (not colo! just dedicated servers). Your average web service does not need to scale near infinitely; for the same amount of money you pay to Amazon you can overprovision 3-5-10 times and that'll handle your spikes. No surprises. Same amount of work: EC2 and bare metal both gives you a root prompt, go from there. These days you can…

I feel the exact same way. I run a 45M pv/mo network on a single dedicated box. The cost of our bandwidth alone at any cloud provider costs multiples of what we are paying for our current server. The cloud just doesn't make economical sense for a lot of workloads.

Re: Docker operations slowing down on AWS

#137

Post by sly010 »

ivanhoe wrote: post #132
sly010 wrote: post #124

Earlier quoted context omitted.

> With cloud you can provision what you think you'll need for a few weeks and then, once you get actual usage data, you can scale and fine-tune the setup to fully utilize it (and optimize the costs). Or in a more realistic scenario you forget to dial back an are paying $200/month extra for unused provisioned IO for years before anyone notices it. Happened, even though I was looking for improvements after every bill.

True, but did you ever scale down an already setup & running dedicated machine? With dedicated hardware it's always like "if it works don't touch, and we might need it one day anyway" :)

I guess the point is if you own the machine, there is no reason to scale down, because you already paid the costs upfront. In conclusion cloud providers solve some of the problems they create in the first place very well ;)

Re: Docker operations slowing down on AWS

#138

Post by jzelinskie »

chrisan wrote: post #112
Diederich wrote: post #98

Earlier quoted context omitted.

A fully containerized setup allows one to use something like Kubernetes to do such orchestration. K8s is getting pretty easy to setup these days.

I was asking about once your bare metal capacity reaches its limit. At some point you need to provision more bare metal and expand the total resources kubernetes can consume with docker. Cloudformation, to me at least, is the power to expand resources for large traffic events. Most of the time you can get by with a small amount of instances but its nice when it scales up to hundreds of instances in minutes. The bare…

>Cloudformation, to me at least, is the power to expand resources for large traffic events. Most of the time you can get by with a small amount of instances but its nice when it scales up to hundreds of instances in minutes.

This is not what CloudFormation does. CloudFormation allows a declarative way to express a group of AWS resources to be created and coupled together. There's nothing that's quite exactly the same as CloudFormation, but stock Kubernetes is quite close, since you effectively describe what resources you want in individual declarative YAML/JSON files. However, there's nothing standard in Kubernetes for coupling them together into one thing equivalent to a Stack in CloudFormation.

Re: Docker operations slowing down on AWS

#139

Post by StillBored »

jcurbo wrote: post #113
chx wrote: post #10

And then people consider me a dinosaur when I say, no cloud, just rent a server or two (not colo! just dedicated servers). Your average web service does not need to scale near infinitely; for the same amount of money you pay to Amazon you can overprovision 3-5-10 times and that'll handle your spikes. No surprises. Same amount of work: EC2 and bare metal both gives you a root prompt, go from there. These days you can…

I always think of this as the "Stack Overflow approach" since I was listening to their podcast when they were building it and that's what they elected to do: https://blog.codinghorror.com/building-servers-for-fun-and-p...

The last sentence pretty much sums it up..

"I know that hardware is cheap, and programmers are expensive."

The way people here are talking makes it sound like they are spending man years setting up a couple servers. From my experience the time consuming parts are messing with the software stack. A problem none of these vendors of machines/VMs/etc really solve (ignoring way overpriced shared databases/etc). Buying a server installing an OS/whatever on it, tossing it in the rack and providing an IP/VLAN/etc is less than 1% of the time/effort spent on spinning up an application. Plus if you buy used, its possible to get 3 year old machines for less money than AWS charges for a month... To which I always hear about "reliability" of old hardware despite the fact that a large part of this rented hardware is just as old, and outside of disks/SFPs and batteries none of it really dies of old age.

As others have said, the effort to "cloudify" your application is probably more than the effort to buy some massively over-provisioned machine. Sure all the big boys need all this fancy management, but your startup with a few hundred thousand hits a month can probably be run on a 5 year old machine if any effort at all were spent assuring its not doing something stupid that results in second long page responses.

Re: Docker operations slowing down on AWS

#140

Post by philipodonnell »

nvivo wrote: post #74

EBS optimized VMs looks nice on paper, you can choose the size and pay for your needs only. But the once you start to use the disk in production you see the problems. In short, if you want to use the disk a lot, you need to pay a lot. If your app is slow on aws and uses EBS as storage, increase the disk size, not the VM instance type. This is true mostly for database performance, which once the RAM is filled, relies…

This is the kind of advice I wish was easier to get. The AWS docs are very lacking in my opinion for "If you're exhibiting this specific performance problem the cheapest option is to increase [x]" where [x] is increase instance size, disk size, performance, add a read replica, etc...
Post reply on HN