I'm sure it would have been nice to have that tech in 94 and yet at the same time I get the feeling it had to play out the way it did for Amazon to succeed. Without the first part of the journey Amazon would not have gone on to build AWS.
The rest of us in 1994 were doing Sun RPC calls, while getting started with DCOM and CORBA, actually quite interesting Amazon's bet on distributed computing given the landscape back then.
So interestingly, they made that bet internally, standardised their own platform and then released some sort of abstracted on demand cloud compute services. However the tools they developed for themselves and the SOA style of development would have been valuable to others too. Google did the same. Netflix did the same. None of this stuff really emerged as a product. I'd argue it still hasn't. IF it had, maybe we'd be doing things a bit differently now. But then I guess proprietary RPC based architectures sort of failed along the way when we look at the list you mentioned.
Essentially there's a maximum size of IAM policy, which AFAIK is not documented properly anywhere - get close to it or exceed it and you start getting random failures everywhere.
Character limits & the number of applied policies are all publicly documented https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_i... . Im not aware of any evaluation complexity limits and have never run in to that sort of problem in my ~10 years of dealing with IAM. I expect you ran in to this sharp bit "You can add as many inline policies as you want to an IAM user, role, or group. But the total aggregate p…
We didn't use inline policies much, but we had many policies linked across different objects, and the error message never pointed properly and we somehow didn't stumble upon the docs you mention (that's going into my notes :D).
I no longer work on that project, but it was considerable blocker when I was leaving as Sagemaker notebooks started randomly failing to start depending on role they were launched with.
The rest of us in 1994 were doing Sun RPC calls, while getting started with DCOM and CORBA, actually quite interesting Amazon's bet on distributed computing given the landscape back then.
So interestingly, they made that bet internally, standardised their own platform and then released some sort of abstracted on demand cloud compute services. However the tools they developed for themselves and the SOA style of development would have been valuable to others too. Google did the same. Netflix did the same. None of this stuff really emerged as a product. I'd argue it still hasn't. IF it had, maybe we'd be…
The didn't fail per se, hence why we now have gRPC, after WebServices, Jini, RMI, .NET Remoting, XML-RPC, JSON-RPC,...
I'm sure it would have been nice to have that tech in 94 and yet at the same time I get the feeling it had to play out the way it did for Amazon to succeed. Without the first part of the journey Amazon would not have gone on to build AWS.
The rest of us in 1994 were doing Sun RPC calls, while getting started with DCOM and CORBA, actually quite interesting Amazon's bet on distributed computing given the landscape back then.
Shel and I would not have touched DCOM nor CORBA with a 22m fishing rod.
The rest of us in 1994 were doing Sun RPC calls, while getting started with DCOM and CORBA, actually quite interesting Amazon's bet on distributed computing given the landscape back then.
Shel and I would not have touched DCOM nor CORBA with a 22m fishing rod.
Sure, but plenty of us did, Nokia Networks infrastructure had plenty of CORBA for several years, and so did many CERN research projects processing HLT data.
Intriguing, can you share details or overview why it failed for you. Will be kind of gotchas for me
Essentially there's a maximum size of IAM policy, which AFAIK is not documented properly anywhere - get close to it or exceed it and you start getting random failures everywhere.
Yeah, I can see that happening. There are combinations of roles etc that might hit the limit.
Do you remember what was failing? That would give some insight into how these get evaluated.
I know that S3 does evaluation differently than the other services, which gave me some insight into the process. Unfortunately I forgot what the insight was (doh).
The rest of us in 1994 were doing Sun RPC calls, while getting started with DCOM and CORBA, actually quite interesting Amazon's bet on distributed computing given the landscape back then.
Shel and I would not have touched DCOM nor CORBA with a 22m fishing rod.
CORBA was interesting, but the authorization side really strangled it. I worked with Tivoli back in the day, which was pretty much the largest production CORBA application in existence. CORBA allowed them to be super flexible when implementing methods, but the auth was brutal. In the end they had to cache all the auth information everywhere just to get decent performance out of it.
The rest of us in 1994 were doing Sun RPC calls, while getting started with DCOM and CORBA, actually quite interesting Amazon's bet on distributed computing given the landscape back then.
Shel and I would not have touched DCOM nor CORBA with a 22m fishing rod.
I've read the stories about what you guys built. It's pretty epic. And not like you were trying to do cool stuff, it was literally based on a need. That's what's amazing. Just manipulating software and infrastructure to do something it wasn't particularly made for just yet.
Nowadays you separate service by business capability and not by "layer". Layers just lead to a dependencies and dependencies lead to bad reliability and terrible development speed.
What Amazon were describing here is simply the division between a frontend web gateway service (or, in modernity, client-delivered SPAs); an API backend service to serve the XHRs of the web-gateway / SPA; and some kind of DBMS where user-visible query schema is separable from storage architecture via e.g. views. I don't think there's any modern system that doesn't have those things, no?
Certainly you can build a server-side rendered web application without a strict separation between frontend and backend and you absolutely should do so if you can. The common separation in frontend and backend microservice is only because JavaScript is so terrible that it's worth the effort to use a different language for backend, but at the same time you can't go all the way because frontend-tooling for backend languages (i.e. Java) is even worse. Introducing this technical separation generally only causes more complexity, inefficient network communication and bad developer experience. It is a historic wart that will hopefully go away over time. As for "a DB where query schema is separated from storage via views": The usual pattern nowadays is to not share data wherever possible (by building self-contained microservices that are aligned with business capabilities instead of layers), have a private database per microservice (in which case it is pointless to do this view indirection) and then provide a stream of business events to other microservices who build their own replicated data model from that, thus decoupling their own data model from external influences. I haven't seen any modern company use views to decouple the schemas but I suppose it is the obvious solution in a 1998 world where everyone shares the same database. If you add asynchronous replication to that, it is basically identical to the modern event-based replication.
Essentially there's a maximum size of IAM policy, which AFAIK is not documented properly anywhere - get close to it or exceed it and you start getting random failures everywhere.
Yeah, I can see that happening. There are combinations of roles etc that might hit the limit. Do you remember what was failing? That would give some insight into how these get evaluated. I know that S3 does evaluation differently than the other services, which gave me some insight into the process. Unfortunately I forgot what the insight was (doh).
The service that hit it was Sagemaker Notebooks, or specifically underlying EC2 instance (which you normally don't see as customer, afaik) - it failed trying to attach a network interface to the instance, because of IAM failure mentioning something rhyming with blown stack (been over a year since, so I don't recall details)