Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon’s distributed computing manifesto (1998)

allthingsdistributed.com

Post reply on HN
Print view View original

31–40 of 61 posts

Re: Amazon’s distributed computing manifesto (1998)

#31

Post by another_devy »

p_l wrote: post #29
manv1 wrote: post #20

This is one example of the CEO making something happen that essentially birthed AWS. Bezos, of all people, was like "make it happen." And it did. It was basically work for no reason except future proofing. Having someone up the food chain OK this much work for the future (and no hard dollar benefit) is highly unusual. And besides that they've done some incredible things with their infrastructure, like authorization.…

It's easy till you break IAM itself with your policy complexity and random services start dying because other AWS components few layers deep cnst get IAM to parse

Intriguing, can you share details or overview why it failed for you. Will be kind of gotchas for me

Re: Amazon’s distributed computing manifesto (1998)

#32

Post by thomastjeffery »

wging wrote: post #16
viburnum wrote: post #13

I don't think this is really from 1998.

The blog post is recent, but it describes much older work, so I think the “(1998)” tag is right. “Distributed Computing Manifesto Created: May 24, 1998”

The blog post (what the OP link takes you to) is from 2022, but the manifesto itself (the substance of the post) is from 1998; so both dates should be used:

> Amazon's distributed computing manifesto (1998) (2022)

Re: Amazon’s distributed computing manifesto (1998)

#33

Post by anonymousDan »

manv1 wrote: post #20

This is one example of the CEO making something happen that essentially birthed AWS. Bezos, of all people, was like "make it happen." And it did. It was basically work for no reason except future proofing. Having someone up the food chain OK this much work for the future (and no hard dollar benefit) is highly unusual. And besides that they've done some incredible things with their infrastructure, like authorization.…

Can you (or anyone) say a bit about how the auth service is implemented from a distributed systems perspective? For example is it some kind of custom KV store?

Re: Amazon’s distributed computing manifesto (1998)

#34

Post by bwestergard »

0xbadcafebee wrote: post #15
bwestergard wrote: post #14

Earlier quoted context omitted.

"But also imagine getting every employee to change how they deal with vacations" Interesting example. Why would changing distributed computing architecture have an impact on vacation policy?

I'm saying architecture is just one way of changing an organization. Other ways of changing an organization, separate from anything technical, might include changing people's schedules or vacation policy, or who you hire, or where, or how. Another would be how you store parts, make orders, assemble products. Or starting work in an entirely new industry. Maybe you work at a company that sometimes works with the govern…

Oh, I got it, you meant the "architecture" of the firm (i.e. the org chart).

Re: Amazon’s distributed computing manifesto (1998)

#35

Post by yedava »

0xbadcafebee wrote: post #10

"And with every few orders of magnitude of growth the current architecture would start to show cracks in reliability and performance, and engineers would start to spend more time with virtual duct tape and WD40 than building new innovative products. At each of these inflection points, engineers would invent their way into a new architectural structure to be ready for the next orders of magnitude growth." That last pa…

I wish there was a way to quantify the externalities of "success" of this kind. How many developers had to burnout? How many relationships had to suffer, or never even had a chance to bloom because "success" didn't leave time for anything else? And also to be considered are the downstream effects of a culture of such "success", like how Amazon's warehouse employees are treated.

Re: Amazon’s distributed computing manifesto (1998)

#36

Post by p_l »

another_devy wrote: post #31
p_l wrote: post #29

Earlier quoted context omitted.

It's easy till you break IAM itself with your policy complexity and random services start dying because other AWS components few layers deep cnst get IAM to parse

Intriguing, can you share details or overview why it failed for you. Will be kind of gotchas for me

Essentially there's a maximum size of IAM policy, which AFAIK is not documented properly anywhere - get close to it or exceed it and you start getting random failures everywhere.

Re: Amazon’s distributed computing manifesto (1998)

#37

Post by mannyv »

anonymousDan wrote: post #33
manv1 wrote: post #20

This is one example of the CEO making something happen that essentially birthed AWS. Bezos, of all people, was like "make it happen." And it did. It was basically work for no reason except future proofing. Having someone up the food chain OK this much work for the future (and no hard dollar benefit) is highly unusual. And besides that they've done some incredible things with their infrastructure, like authorization.…

Can you (or anyone) say a bit about how the auth service is implemented from a distributed systems perspective? For example is it some kind of custom KV store?

Well, it's really tough, because (1) every operation has to check if the calling entity is authorized, (2) changes need to propagate super quickly, and (3) performance needs to be pretty much realtime.

At some level every API call is authorized (and tracked).

To be honest, this is one of the secret sauces that makes AWS go. Someone once told me that they're not doing anything exciting, just caching, but I'm pretty sure they didn't really know what was going on.

Re: Amazon’s distributed computing manifesto (1998)

#38

Post by dastbe »

anonymousDan wrote: post #33
manv1 wrote: post #20

This is one example of the CEO making something happen that essentially birthed AWS. Bezos, of all people, was like "make it happen." And it did. It was basically work for no reason except future proofing. Having someone up the food chain OK this much work for the future (and no hard dollar benefit) is highly unusual. And besides that they've done some incredible things with their infrastructure, like authorization.…

Can you (or anyone) say a bit about how the auth service is implemented from a distributed systems perspective? For example is it some kind of custom KV store?

in AWS, authentication and authorization happens within the application.

For the purposes of authorization, services integrate with a library that handles retrieving and caching policies based on caller identity. services create a context that includes all of the relevant metadata (service, operation, resources, etc.) and the library evaluates the policy and says allow or deny.

Doing it all in application means that if the control/distribution systems for auth go down most things that are in motion will remain in motion, and that deployments of the authentication/authorization code deploy out at a per-service granularity which also scopes blast radius.

There's some pretty obvious pain points (doing anything as a library means update the world for new features) but it has nice degradation properties and is relatively straightforward to grok as a service owner.

Re: Amazon’s distributed computing manifesto (1998)

#39

Post by benhoyt »

I think this is a very interesting and clear manifesto, and almost certainly the right thing for Amazon at the right time (and presumably was part of what led to AWS). However, at one of the previous companies I worked at we got an ex-Amazon person as CTO and he grew the company from 50 engineers to 500 in 2 years, and pushed microservices everywhere. Very impressive, and I think all-microservices made sense at one level to handle that sort of growth, but it doesn't make sense technically in a lot of cases.

Essentially I think we've gone too far: service-oriented architectures turned into "micro" services, which come with a lot of complexity and distributed systems issues. I think for most small companies monoliths are right, for medium-sized companies (say 50+) it makes sense to carefully introduce a few separate services, and only for large companies (say 300+) does many services (which may or may not be "micro") start to pay off. I've heard it said that "microservices solve a people problem, not a technical one", and I think that's true.

Re: Amazon’s distributed computing manifesto (1998)

#40

Post by jerglingu »

benhoyt wrote: post #39

I think this is a very interesting and clear manifesto, and almost certainly the right thing for Amazon at the right time (and presumably was part of what led to AWS). However, at one of the previous companies I worked at we got an ex-Amazon person as CTO and he grew the company from 50 engineers to 500 in 2 years, and pushed microservices everywhere. Very impressive, and I think all-microservices made sense at one l…

This sounds similar to Flexport's CTO situation (he came from Amazon) and attempted microservice-ification of everything. Except it sounds like they weren't even able to get wheels up and are still floundering after years of planning and attempted execution.
Post reply on HN