Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The 2002 mandate for internal communication systems at Amazon

sametab.com

Post reply on HN
Print view View original

181–187 of 187 posts

Re: The 2002 mandate for internal communication systems at Amazon

#181

Post by jrochkind1 »

blandflakes wrote: post #120
jrochkind1 wrote: post #74

Earlier quoted context omitted.

"an internal communication system" does sound like something like an "RPC framework", but Yegge's paraphrase actually says "It doesn’t matter what technology they use. HTTP, Corba, Pubsub, custom protocols — doesn’t matter. Bezos doesn’t care." I read this as saying different teams/services don't have to use the same thing either. That doesn't sound like an "RPC framework" or "an internal communications system" at al…

This edict was before my time at Amazon, so I can't speak to whether there was an RPC framework in existence when this was mandated. By the time I arrived, however, there was a cross-language RPC framework that integrated with Amazon's monitoring, request tracing, and build infrastructure (for building and releasing client versions). It was very full-featured and the de-facto system for creating a service. Most of ou…

Interesting, people don't talk about this much, although the OP seems to be aware of it and think it was important.

Does anyone know if there's been much written on how this came to be and what it looked like? If not, it would be a useful thing to write about!

Cause it does seem like a really important thing, without it, the narrative seems to be that you make a decree like Bezos', and bing bang magic, you get what AWS got. Where in fact, succesfully pulling off that RPC framework seems to be really important, and undoubtedly took a lot of work, good succesful design, and social organizing to get everyone to use it (perhaps by making it the easy answer to Bezos' mandate). But none of that stuff just happens, some have failed where AWS succeeded, the mandate alone isn't enough.

Re: The 2002 mandate for internal communication systems at Amazon

#182

Post by syncsynchalt »

Waterluvian wrote: post #4

> 6) Anyone who doesn’t do this will be fired. So I've never worked at a company over 150 people. Is this... a normal thing for an email? Maybe I'm just one of those softies but an email with that line would throw me off my day and cause a serious hit to my morale and confidence of working there.

I've worked at several large companies via acquisition.

Yegge has punched things up a little, but I can translate #6 into what I believe was the original text: "This represents the new direction for {company}, and anyone who does not wish to align with this new direction should pursue other opportunities".

The day-to-day employee at a large company becomes numb to executive emails, as 90% of the time they will be countermanded a month later. A sentence like the above is inserted to wake people up and indicate "no, seriously, this means you".

Re: The 2002 mandate for internal communication systems at Amazon

#183

Post by blandflakes »

jrochkind1 wrote: post #181
blandflakes wrote: post #120

Earlier quoted context omitted.

This edict was before my time at Amazon, so I can't speak to whether there was an RPC framework in existence when this was mandated. By the time I arrived, however, there was a cross-language RPC framework that integrated with Amazon's monitoring, request tracing, and build infrastructure (for building and releasing client versions). It was very full-featured and the de-facto system for creating a service. Most of ou…

Interesting, people don't talk about this much, although the OP seems to be aware of it and think it was important. Does anyone know if there's been much written on how this came to be and what it looked like? If not, it would be a useful thing to write about! Cause it does seem like a really important thing, without it, the narrative seems to be that you make a decree like Bezos', and bing bang magic, you get what A…

I think a lot of Amazon's internal tooling is sort of "unpublished" - I've not found a great reference for a lot of the really excellent dev support they had.

The AWS story is particularly interesting because a lot of the internal setup I was doing at the time was on old fashioned metal. There was an internal project called Move to AWS (MAWS) that encouraged using newly-developed integrations with the AWS systems that the public was using.

In other words, AWS lived alongside old-fashioned provisioning practices up until even the early 2010s.

Re: The 2002 mandate for internal communication systems at Amazon

#184

Post by rwmj »

repolfx wrote: post #178
atombender wrote: post #142

Earlier quoted context omitted.

Protobufs were invented for Stubby, the RPC layer which is apparently used for absolutely everything inside Google. It's existed since at least 2001, and uses protobufs as the RPC serialization. gRPC is based on Stubby (though not the actual implementation).

Yeah. Google is actually a much better example of this mentality than Amazon is, if I'm reading the thread right. Google Cloud isn't behind AWS because of some service architecture nonsense. It's behind because Google started later, and it started later because for the longest time (I was there) the senior management had the following attitude: "Why would we sell our cloud platform? We can always make more money and…

I have to wonder if maybe Google's management weren't right. Why are Google in cloud, a low margin business?

Re: The 2002 mandate for internal communication systems at Amazon

#185

Post by ozymandias12 »

ozgune wrote: post #86
Twirrim wrote: post #2

Yegge's post was very interesting reading, and I took similar learnings away from it. I was at Amazon at the time, however, and there were things that certainly weren't true any more: >3) There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service inte…

(Former Amazonian, part of the team that drove the change to SOA at the time) > Now I think that this internal email is what has actually stuck with me the most. Bezos realized that he had to change the internal communication infrastructure [..]. > He understood that a radical organizational change was required to arrange the internal dynamics in a way that would allow the creation of something like AWS. This is quit…

I was under the impression that before AWS, Amazon did SOA'd almost the entirety of its departments, as in OP mail, every dept had to enable web services to other depts, creating an environment where

Between 2000/2003 Amazon started to learn how to properly make something that resembled AWS (launched in 2006), it was a necessity targeted towards scalability. What Yegge letter meant is that Amazon was doing SOA to all its depts, making these be useful both internally and externally

>"5) All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world."

So you see, it wasn't to enable AWS as the digital service as we know it, it was to build a scalable company that could use everything it had both to serve its needs, and also to sell it as a product (then, all services we all know EC2 etc). Bezos applied SOA as a company, not only to its digital philosophy. And this corporate move, with no precedent, enabled not only AWS, but the whole Amazon as it is today.

Re: The 2002 mandate for internal communication systems at Amazon

#186

Post by closeparen »

jhallenworld wrote: post #110
leoh wrote: post #5

Earlier quoted context omitted.

How do services talk to each other without an API? Is it something like "put a non-well-documented object into a queue?"

IBM has an appliance for you: https://www.ibm.com/products/datapower-gateway It's a middleware appliance that allows any API to talk to any other API and handles just about any data format. You can script it with javascript or XSLT. It can handle ad-hoc things like ftp polling. It has the added benefit that you can add security for outside facing clients. Disclaimer: I helped develop this appliance (but I no longer w…

This is the Enterprise Service Bus concept, right? I remember a pretty good conference talk about how we realized in the mid 2000s that these things are problematic and you probably want services communicating directly over dumb pipes.

In showed an evolution from a monolithic spaghetti codebase, to an SOA, to realizing there are now spaghetti connections between services in an SOA, to a very clean looking ESB architecture, to showing how all the chaos is still there, just inside the ESB where it’s even harder to reason about or change.

Re: The 2002 mandate for internal communication systems at Amazon

#187

Post by closeparen »

xyzzyz wrote: post #40

> While the third point makes all the difference in the world, what Amazon really did get right that Google didn’t was an internal communication system designed to make all the rest possible. > Having teams acting like individual APIs and interacting with one another through interfaces over the network was the catalyst of a series of consequent actions that eventually made possible the realization of AWS in a way tha…

Lots of places will use APIs between teams, but the APIs aren’t primary. An integration still starts with a document and a kickoff meeting. You might need to post config or even code changes to the platform, which the platform team will review and deploy at their own pace, before the endpoint is usable. Interfaces and schemas have use case specific fields in them. It’s not at all like using an AWS service.
Post reply on HN