Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon deploys every 11.6 seconds

assets.en.oreilly.com

Post reply on HN
Print view View original

21–28 of 28 posts

Re: Amazon deploys every 11.6 seconds

#21

Post by jonjenk »

DigitalBison wrote: post #6
awj wrote: post #2

The same panel also has stats like "max number of hosts simultaneously receiving a deployment (3000)". Depending on how you run the numbers those 3000 hosts could all take nine hours to receive a single deployment and average out to 11.6 deploys / second / host. Unless their systems are heavily modularized, I have a bit of a hard time believing that something new at Amazon goes live every 11.6 seconds. Maybe I'm wron…

Minor correction: the slides had the max number of hosts receiving a deployment at 30,000, not 3,000.

I think the most impressive stat is a little later in the presentation. Only ~0.001% of deployments actually cause an outage.

Anyone can create a system that generates a lot of deployments, but what really matters is that you can complete all of those deployments safely. Of course, that is still ~0.001% too many outages due to deployments and we are working hard to make that number zero.

Re: Amazon deploys every 11.6 seconds

#22

Post by jonjenk »

r0dica wrote: post #9
andrewguenther wrote: post #3

Is there a recording of the talk associated with this?

per the first comment: http://www.youtube.com/watch?v=dxk8b9rSKOo :)

Here are two other great talks by one of my friends at Amazon.

http://www.youtube.com/watch?v=UL2WDcNu_3A

http://www.youtube.com/watch?v=coNDCIMH8bk

Re: Amazon deploys every 11.6 seconds

#23

Post by awj »

jonjenk wrote: post #18
awj wrote: post #2

The same panel also has stats like "max number of hosts simultaneously receiving a deployment (3000)". Depending on how you run the numbers those 3000 hosts could all take nine hours to receive a single deployment and average out to 11.6 deploys / second / host. Unless their systems are heavily modularized, I have a bit of a hard time believing that something new at Amazon goes live every 11.6 seconds. Maybe I'm wron…

Wow. Look at me on the front page of Hacker News! Our systems are extremely modular. We've previously disclosed that in excess of a hundred discrete services may be called to generate a single page on our web site. You can find more info about that at the following link. http://highscalability.com/amazon-architecture When we refer to a deployment at Amazon it means a single code push to one or more servers. For examp…

Yeah, it does, and thanks for taking the time to point me at something a little meatier than slides.

I guess it's just hard to imagine that kind of situation when I'm on a two man web dev team that pushes out into the testing server 10-20 times a day and to production once a week, if that.

Re: Amazon deploys every 11.6 seconds

#26

Post by tlack »

bonafidehan wrote: post #17
mncaudill wrote: post #16

Earlier quoted context omitted.

I'd love to see a pros-and-cons of this approach from a team organizational standpoint.

I think that decoupling teams boils down to giving teams complete ownership. And Amazon got parts of it right. It means that your team owns everything it builds. You own the code, you own the testing and you own the operations: you own the product. Various tools are laid at your feet, and you are asked to build. Clearly, a benefit is that you can move fast. You don't need permissions from someone half a building away…

How do these services communicate with each other? How can a single page make hundreds of requests to build a page and yet get it all together in a fraction of a second?

Re: Amazon deploys every 11.6 seconds

#27

Post by srdev »

tlack wrote: post #26
bonafidehan wrote: post #17

Earlier quoted context omitted.

I think that decoupling teams boils down to giving teams complete ownership. And Amazon got parts of it right. It means that your team owns everything it builds. You own the code, you own the testing and you own the operations: you own the product. Various tools are laid at your feet, and you are asked to build. Clearly, a benefit is that you can move fast. You don't need permissions from someone half a building away…

How do these services communicate with each other? How can a single page make hundreds of requests to build a page and yet get it all together in a fraction of a second?

There's several different communication methods between services, including REST, SOAP, message queues, and an internal service framework. Its a perfect example of bonafidehan's post.

As for the second question, a page generally doesn't have to make hundreds of requests. You're thinking of a flat architecture. Think of it more like a pipeline: data goes in at A, flows from A->B->C->D, page reads D. So you end up having to call a handful of services. That can be scaled by 1) caching, 2) careful selection of service calls (don't call ordering service unless you're placing an order), 3) asynchronous requests (you're typically going to be IO bound on the latency, so just spin up X service requests and then wait on them all). There are also other tricks that are fairly well known for reducing latency, such as displaying a limited set of information and loading the rest via AJAX.

As a disclaimer for the above, my work doesn't involve working with the Amazon.com website directly, so its based on my limited view in my domain space.

If you own a page or service that calls a bunch of other service, you typically collect metrics on latency of your downstream services. Amazon has libraries to facilitate this, and a good internal system for collecting and presenting this data. If one service is particularly troublesome, then you can reach out to that other team and get them to lower their latency. The other option is to pull in their data closer to you, in a format that you can consume quickly.

Re: Amazon deploys every 11.6 seconds

#28

Post by doki_pen »

onedognight wrote: post #11
mcclung wrote: post #10

Even though amazon.com is all on EC2 and capacity is demand driven, someone is still buying servers and has some capacity overhead, right? They've just shifted the spend from the amazon.com business unit to the AWS business unit (assuming that's how it's set up)?

You can flatten the demand easily with EC2 by having the cost vary dynamically based on overall load. So (using his example) during the end of November Amazon themselves would want more servers and so the cost could go up slightly. A drug company doing discovery might decide then not to run their computations and to wait until the price drops back down. Likewise with someone cracking passwords, or mining bitcoins.

I don't know about "easily". If you take a higher level view, like the entire internet, then you will see that most sites have a similar usage pattern. How many have the opposite problem?

I guess as we move towards a more global economy it will level out somewhat on a day to day basis, but I don't know if that is a realistic expectation. The season spikes probably won't change.

Post reply on HN