Seeing that Werner Vogels has submitted this entry, I wonder if he can comment how long it took to actually build this out. When did a form of this service oriented architecture work in production at Amazon?
Amazon’s distributed computing manifesto (1998)
41–50 of 61 posts
Re: Amazon’s distributed computing manifesto (1998)
#42This is one example of the CEO making something happen that essentially birthed AWS. Bezos, of all people, was like "make it happen." And it did. It was basically work for no reason except future proofing. Having someone up the food chain OK this much work for the future (and no hard dollar benefit) is highly unusual. And besides that they've done some incredible things with their infrastructure, like authorization.…
[deleted]
Re: Amazon’s distributed computing manifesto (1998)
#43Earlier quoted context omitted.
The blog post is recent, but it describes much older work, so I think the “(1998)” tag is right. “Distributed Computing Manifesto Created: May 24, 1998”
The blog post (what the OP link takes you to) is from 2022, but the manifesto itself (the substance of the post) is from 1998; so both dates should be used: > Amazon's distributed computing manifesto (1998) (2022)
Or:
> Amazon's 1998 distributed computing manifesto
Title has already been changed from TFA to include 'Amazon', and it's still 2022 so no need for that.
Re: Amazon’s distributed computing manifesto (1998)
#44Earlier quoted context omitted.
Intriguing, can you share details or overview why it failed for you. Will be kind of gotchas for me
Essentially there's a maximum size of IAM policy, which AFAIK is not documented properly anywhere - get close to it or exceed it and you start getting random failures everywhere.
Character limits & the number of applied policies are all publicly documented https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_i.... Im not aware of any evaluation complexity limits and have never run in to that sort of problem in my ~10 years of dealing with IAM.
I expect you ran in to this sharp bit "You can add as many inline policies as you want to an IAM user, role, or group. But the total aggregate policy size (the sum size of all inline policies) per entity cannot exceed the following limits." Calculating the sum would be a pain as a user.
Re: Amazon’s distributed computing manifesto (1998)
#45> All of this was being done before terms like service-oriented architecture existed. I feel like the first time I heard the term was early 2000's, and wasn't it a mainframe thing first? Dunno, just wondering. Anyhow, it's nicely written, very concise, and worth noting how the original author focuses more on "What kind of realistic options do we have?" than winning the A vs. B vs. C argument in one fell swoop.
It was definitely pre-2000. First “SoA” firm I worked for, I started at in 99, and they had been doing it for 2 years already and most of the crew brought if from a prior gig.
Although not using the same buzzwords, we had the same architecture deep into the 1980s in Inmos/Transputer/Occam-land.
Re: Amazon’s distributed computing manifesto (1998)
#46This is one example of the CEO making something happen that essentially birthed AWS. Bezos, of all people, was like "make it happen." And it did. It was basically work for no reason except future proofing. Having someone up the food chain OK this much work for the future (and no hard dollar benefit) is highly unusual. And besides that they've done some incredible things with their infrastructure, like authorization.…
Distributed authorization is indeed hard! IAM is one of the few (maybe the only) AWS service that isn't regional and it's because permissions must propagate globally for correctness' sake. As a distributed systems junkie, I'm shocked that other folks aren't as interested in authorization systems because they really push the boundaries of what we can do with data consistency at scale.
It's unfortunate that only Amazon themselves can add new permissions to IAM to secure their services. Why can't our applications add new permissions to IAM and query those? This is going to be a shameless plug, but it was this very problem that caused my cofounders and I to quit our jobs and start a company. Together (and now with a community of hundreds of users and contributions from a few well-known companies) we built SpiceDB[0], which is the culmination of state of the art distributed systems and authorization technology developed open source instead of behind closed doors at a hyper-scaler. We were mostly inspired by the internal system at Google, which is actually more powerful than AWS or Google Cloud's IAM services, despite a fork of it actually powering GCP's IAM.
Re: Amazon’s distributed computing manifesto (1998)
#47I think this is a very interesting and clear manifesto, and almost certainly the right thing for Amazon at the right time (and presumably was part of what led to AWS). However, at one of the previous companies I worked at we got an ex-Amazon person as CTO and he grew the company from 50 engineers to 500 in 2 years, and pushed microservices everywhere. Very impressive, and I think all-microservices made sense at one l…
It is most likely true. If you take away all constraints related to planning, communication, prioritization, collaboration, development efficiency etc, most of the arguments against monoliths goes away. What remains are considerations for memory, bandwidth and similar that could motivate a breakdown.
Re: Amazon’s distributed computing manifesto (1998)
#48This is one example of the CEO making something happen that essentially birthed AWS. Bezos, of all people, was like "make it happen." And it did. It was basically work for no reason except future proofing. Having someone up the food chain OK this much work for the future (and no hard dollar benefit) is highly unusual. And besides that they've done some incredible things with their infrastructure, like authorization.…
I wonder if we are far too quick to bestow CEOs with credit for something other people effected. Sure the CEO is the one to sign off on everything, but the question to ask is could any other person in the CEO role at that time, have done any different? I don't know the details in this particular case, but I'll go out on a limb and say the CEO quite likely did not proclaim "make this happen". Business was growing at an unbelievable pace, their systems were probably stressed to the max, their development was likely choked and the technical team comes up and says this is what we need to do otherwise we can't handle more than this much traffic. What choice does the CEO have? He says, send me your proposal and broadcasts it.
As for AWS, as far as I remember, Bezos was initially against the idea. The idea was the brainchild of one Andy Jassy who along with Rick Salzell convinced a reluctant board into trying this out. They realized that they had been unintentionally building this cloud platform for some years now in order to provide sellers with computing resources. Opening up to public users was just a small sales move. Whether they do it or not, they were going to continue to invest in their cloud platform and nothing would change as far as their technical direction was concerned, so the board finally relented.
Re: Amazon’s distributed computing manifesto (1998)
#49As one of the main designers of the original system (but who had left by the time this architectural change was done), that is an interesting read. Always good to see the things that we missed in 1994/1995, even though we believed we were thinking far, far ahead.
I'm sure it would have been nice to have that tech in 94 and yet at the same time I get the feeling it had to play out the way it did for Amazon to succeed. Without the first part of the journey Amazon would not have gone on to build AWS.
The rest of us in 1994 were doing Sun RPC calls, while getting started with DCOM and CORBA, actually quite interesting Amazon's bet on distributed computing given the landscape back then.
Re: Amazon’s distributed computing manifesto (1998)
#50> All of this was being done before terms like service-oriented architecture existed. I feel like the first time I heard the term was early 2000's, and wasn't it a mainframe thing first? Dunno, just wondering. Anyhow, it's nicely written, very concise, and worth noting how the original author focuses more on "What kind of realistic options do we have?" than winning the A vs. B vs. C argument in one fell swoop.
SOA as buzzword started with DCOM and CORBA distributed computing, then it evolved into the XML spaghetti of XML-RPC and WebServices.
Ironically when .NET was launched, Microsoft's vision was web services everywhere, with orchestration servers like Bizztalk.
We got there eventually, only using REST (aka JSON-RPC) and gRPC instead.