Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The 2002 mandate for internal communication systems at Amazon

sametab.com

Post reply on HN
Print view View original

131–140 of 187 posts

Re: The 2002 mandate for internal communication systems at Amazon

#131

Post by username90 »

gowld wrote: post #107
xyzzyz wrote: post #40

> While the third point makes all the difference in the world, what Amazon really did get right that Google didn’t was an internal communication system designed to make all the rest possible. > Having teams acting like individual APIs and interacting with one another through interfaces over the network was the catalyst of a series of consequent actions that eventually made possible the realization of AWS in a way tha…

A protocol buffer is a serialized data object, not a service API. It doesn't (and didn't) prevent anyone from using shared memory, shared database, or shared flat files to communicate. Also, 2002 is time immemorial. Google was founded in 1998.Protocol buffers were invented in 2001.

Protocol buffers are both serialized data objects and service API's.

https://developers.google.com/protocol-buffers/docs/proto#se...

Re: The 2002 mandate for internal communication systems at Amazon

#132

Post by dang »

eigen-vector wrote: post #84

This was not exactly a Jeff Bezos mandate but the result of an engineering brainstorm. The mandate came more out of a "how to scale Amazon for the next decade" discussion. In large companies, one where distributed/independent teams are as important as distributed systems this ended up being the only way to operate. Initially, during the good old days of Amazon, there was what you'd call a single datawarehouse. It mad…

Ok, we'll take Bezos out of the title above.

Re: The 2002 mandate for internal communication systems at Amazon

#133

Post by ineedasername »

function_seven wrote: post #102
morelisp wrote: post #96

Earlier quoted context omitted.

At the time Amazon was building out their merchant portal as a white-label-ish service for other large retailers to sell products online. The 'customers' in the memo would be other merchants, and the early AWS offering (e.g. SQS) reflect this. "Elastic" clouds weren't really on the menu yet but obviously part of the point is that you can offer it to customers regardless of where the architectural fad goes.

I remember the early days of target.com and (I think?) toysrus.com being thinly skinned versions of Amazon.

Yes, if I remember correctly even Borders Books got in on the action there.

Re: The 2002 mandate for internal communication systems at Amazon

#134

Post by prepend »

boldslogan wrote: post #36
prepend wrote: post #23

I wish the actual body of the email was available and published. I’ve only read Yegge’s account of the note and didn’t see it in any of Bezos’ books. I suppose it’s nice that the email, or really any amazon emails, has not been leaked.

I only could find his autobiography. What other books does he have / wrote?

Brad Stone wrote a Bezos/Amazon bio called “Everything Store.”

Re: The 2002 mandate for internal communication systems at Amazon

#135

Post by rantwasp »

guidoism wrote: post #115
ozgune wrote: post #86

Earlier quoted context omitted.

(Former Amazonian, part of the team that drove the change to SOA at the time) > Now I think that this internal email is what has actually stuck with me the most. Bezos realized that he had to change the internal communication infrastructure [..]. > He understood that a radical organizational change was required to arrange the internal dynamics in a way that would allow the creation of something like AWS. This is quit…

(Another former Amazonian from the same team) ozgune is correct as far as I remember. The beginnings of AWS, though, were for the retail site. We exposed search and browse and item metadata via an API first for a deal with AOL to provide them with product search and then later we opened it up to everyone. People built stuff like Simple Amazon on top of it, which I thought was pretty nice. I don't think this part of A…

i though the first one was mechanical turk.

Re: The 2002 mandate for internal communication systems at Amazon

#136

Post by rantwasp »

nexuist wrote: post #114
ozgune wrote: post #86

Earlier quoted context omitted.

(Former Amazonian, part of the team that drove the change to SOA at the time) > Now I think that this internal email is what has actually stuck with me the most. Bezos realized that he had to change the internal communication infrastructure [..]. > He understood that a radical organizational change was required to arrange the internal dynamics in a way that would allow the creation of something like AWS. This is quit…

> Second, Kindle and SimpleDB (now Dynamo) Minor nitpick, these are two separate services. SDB is still independently accessible and probably will be for a while. That being said they definitely don't encourage any new use of SDB and push Dynamo instead. Kind of a shame because a modern NoSQL store with SQL support would be great for rapid prototyping before a concrete data schema is established.

dynamodb is the spiritual successor of sdb.

sdb is build on the same principles outlined in the dynamo paper: https://www.allthingsdistributed.com/files/amazon-dynamo-sos...

Re: The 2002 mandate for internal communication systems at Amazon

#137

Post by atombender »

mav3rick wrote: post #72
Twirrim wrote: post #2

Yegge's post was very interesting reading, and I took similar learnings away from it. I was at Amazon at the time, however, and there were things that certainly weren't true any more: >3) There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service inte…

I have to ask. Isn't this just microservices ? Also isn't this terrible for latency and debugging ?

"Microservice" is just a new name for something that has had many names throughout history. For as long as we've had networks, we've had small, loosely coupled networked apps communicating with each other. We've also had standards for communicating and for discovering nearby services — at one point it was DCE/RPC, later CORBA or DCOM, or Java RMI. Then people started with REST and Thrift, and these days it's more often gRPC and GraphQL.

I think the core of the Amazon invention was to mandate that the API be a strict boundary around a service. No cheating by making assumptions about the internals of the implementation. Force teams to work off the published API documentation, and make teams the guardians of each service. Microservices probably aren't developed exactly like that in many companies.

Re: The 2002 mandate for internal communication systems at Amazon

#138

Post by rickbad68 »

jrochkind1 wrote: post #57

> what Amazon really did get right that Google didn’t was an internal communication system designed to make all the rest possible. I'm not following what he means. What is the thing he is describing as "an internal communication system" here? That made all the rest possible? What is/was this internal communications system?

It seems the OP is interpreting Bezos mandate as if he were talking about the interaction/communication between actual teams (people) as opposed to between each teams software service(s). Whereas there are parallels between these, I'm pretty sure Bezos wasn't saying anything about inter-team communication. It sounds like they already had a good company organizational structure but there were no limits on how services used each other, which would result in tight, often-hidden coupling that would be too complex to manage.

My guess is this was the thinking behind the mandate, limiting all inter-service coupling to a single, well-documented interface prevents the whole system from devolving into a big ball of mud; primarily because it forces all dependencies to be explicitly declared and documented in the same way.

Re: The 2002 mandate for internal communication systems at Amazon

#140

Post by remus »

saalweachter wrote: post #124
morelisp wrote: post #15

Earlier quoted context omitted.

A queue if you're lucky! There's also: - Hire an intern / "Customer Service Representative" / "Technical Account Specialist" to manually copy data from one service into another - Dump some file in a directory and hope something is treating that directory like a queue - Read/write from the same database (/ same table) Or the classic Unix trajectory of increasingly bad service communication: - Read/write from the same…

Don't forget the most important step. "Think of the acronym CSV. Don't look up the definition of the format, just meditate on the idea of the format for a bit. Then write your data in the format you have just imagined is CSV, making whatever choices you feel personally best or most elegant regarding character escapes. Pass this file on to your downstream readers, assuring them it is a CSV file, without elaborating on…

"Comma separated values? But my data has commas in it! Ah, I know, I'll use tabs instead, I've never seen a user put a tab so that'll work perfectly forever and definitely won't cause a huge fucking mess for the poor bastard who has to try and decipher this steaming pile."
Post reply on HN