Even though amazon.com is all on EC2 and capacity is demand driven, someone is still buying servers and has some capacity overhead, right? They've just shifted the spend from the amazon.com business unit to the AWS business unit (assuming that's how it's set up)?
Amazon deploys every 11.6 seconds
11–20 of 28 posts
Re: Amazon deploys every 11.6 seconds
#12Re: Amazon deploys every 11.6 seconds
#13The same panel also has stats like "max number of hosts simultaneously receiving a deployment (3000)". Depending on how you run the numbers those 3000 hosts could all take nine hours to receive a single deployment and average out to 11.6 deploys / second / host. Unless their systems are heavily modularized, I have a bit of a hard time believing that something new at Amazon goes live every 11.6 seconds. Maybe I'm wron…
Just to clarify, the "deployment every 11.6 seconds" refers to all prod deployments, including internal applications and services. This doesn't mean that a deployment to the retail website (and dependent services) is done every 11.6 seconds, just that some production deployment is done at Amazon every 11.6 seconds. disclaimer: I work for Amazon, and used to work in the presenter's org.
Re: Amazon deploys every 11.6 seconds
#14Even though amazon.com is all on EC2 and capacity is demand driven, someone is still buying servers and has some capacity overhead, right? They've just shifted the spend from the amazon.com business unit to the AWS business unit (assuming that's how it's set up)?
You can flatten the demand easily with EC2 by having the cost vary dynamically based on overall load. So (using his example) during the end of November Amazon themselves would want more servers and so the cost could go up slightly. A drug company doing discovery might decide then not to run their computations and to wait until the price drops back down. Likewise with someone cracking passwords, or mining bitcoins.
Re: Amazon deploys every 11.6 seconds
#15The same panel also has stats like "max number of hosts simultaneously receiving a deployment (3000)". Depending on how you run the numbers those 3000 hosts could all take nine hours to receive a single deployment and average out to 11.6 deploys / second / host. Unless their systems are heavily modularized, I have a bit of a hard time believing that something new at Amazon goes live every 11.6 seconds. Maybe I'm wron…
> "Unless their systems are heavily modularized" I used to work for Amazon. This is exactly how things are, to a scale that's hard to comprehend. Knowing how their stuff works internally, a prod deployment every 11.6 seconds is not hard to imagine at all.
Re: Amazon deploys every 11.6 seconds
#16Earlier quoted context omitted.
> "Unless their systems are heavily modularized" I used to work for Amazon. This is exactly how things are, to a scale that's hard to comprehend. Knowing how their stuff works internally, a prod deployment every 11.6 seconds is not hard to imagine at all.
This. I also used to work at Amazon until recently. Amazon understands decoupling, deeply. And it's applied everywhere. In the code. In the architecture. How teams are organized. How processes are designed.
Re: Amazon deploys every 11.6 seconds
#17Earlier quoted context omitted.
This. I also used to work at Amazon until recently. Amazon understands decoupling, deeply. And it's applied everywhere. In the code. In the architecture. How teams are organized. How processes are designed.
I'd love to see a pros-and-cons of this approach from a team organizational standpoint.
Clearly, a benefit is that you can move fast. You don't need permissions from someone half a building away to do something. You don't need to touch code that needs another team's approval. There are no committees that decides on global rules. Your team decides on your team's rules.
Like a shared nothing architecture, there's very little that is shared between teams. Teams are often connected only via their service interfaces. Not much else beyond common tooling.
But even their tooling reflects decoupling. Every tool follows the self-service model ("YOU do what you WANT to do with YOUR stuff"). Their deployment system (named Apollo, mentioned in the slides) and their build system, and their many other tooling, all reflect this model.
Cons. What happens is that you might be reinventing the wheel at Amazon. Often. Code reuse is very low across teams. So there's no shared cost of ownership at Amazon, more often than not. It's the complete opposite at Google w.r.t. code reuse. There are many very high-quality libraries at Google that are designed to be shared. Guava (the Java library) is a great example.
Another con. You may not know what you're doing. But as a team you will still build a rickety solution that gets you to a working solution. This is the result of giving a team complete ownership: they'll build what they know with what they have. Amazon is slowly correcting some of these problems by having teams own specific Hard Problems. A good example is storage systems.
And a lack of consistency is a common issue across Amazon. Code quality and conventions fluctuate wildly across teams.
Overall, Amazon has figured out how to decouple things very well.
Re: Amazon deploys every 11.6 seconds
#18The same panel also has stats like "max number of hosts simultaneously receiving a deployment (3000)". Depending on how you run the numbers those 3000 hosts could all take nine hours to receive a single deployment and average out to 11.6 deploys / second / host. Unless their systems are heavily modularized, I have a bit of a hard time believing that something new at Amazon goes live every 11.6 seconds. Maybe I'm wron…
Our systems are extremely modular. We've previously disclosed that in excess of a hundred discrete services may be called to generate a single page on our web site. You can find more info about that at the following link.
http://highscalability.com/amazon-architecture
When we refer to a deployment at Amazon it means a single code push to one or more servers. For example, if you deploy a new piece of code to a thousand hosts that counts as one deployment. In other words a distinct update is pushed every 11.6 seconds.
Hopefully that makes sense.
Re: Amazon deploys every 11.6 seconds
#19Earlier quoted context omitted.
> "Unless their systems are heavily modularized" I used to work for Amazon. This is exactly how things are, to a scale that's hard to comprehend. Knowing how their stuff works internally, a prod deployment every 11.6 seconds is not hard to imagine at all.
This. I also used to work at Amazon until recently. Amazon understands decoupling, deeply. And it's applied everywhere. In the code. In the architecture. How teams are organized. How processes are designed.
Re: Amazon deploys every 11.6 seconds
#20Earlier quoted context omitted.
You can flatten the demand easily with EC2 by having the cost vary dynamically based on overall load. So (using his example) during the end of November Amazon themselves would want more servers and so the cost could go up slightly. A drug company doing discovery might decide then not to run their computations and to wait until the price drops back down. Likewise with someone cracking passwords, or mining bitcoins.
They have consistent pricing year-round for their on-demand and reserve instances. The spot-instances are priced dynamically by auction and the supply of them would be reduced when Amazon is using more instances itself. The price of the spot instances will never exceed that of the on-demand rate, since no-one would bid greater than a fixed rate for the same service. At the peak usage, spot instances reach the same pr…
Checking the price history in the AWS console reveals that the prices for spot instances occasionally exceed the on-demand rate. In particular t1.micro instances reached $0.05/hr (vs the on-demand $0.02/hr). One possible explanation is that spot instances are more valuable because you can run more of them at a time than on-demand instances (100 total vs 20 total) without having to get an exemption for your use case. Another possible explanation is that people bid higher amounts to guarantee that their instances will run uninterrupted, knowing that even if the price briefly exceeds the on-demand price, the average should still be lower overall.