Live data from Hacker News

On Internal Engineering Practices at Amazon

jatins.gitlab.io

31–40 of 130 posts

Re: On Internal Engineering Practices at Amazon

#31
post #10

Based on my experience, the article contains a lot of misinformation. Some of the statements might have been true at one point in the past, but are now out of date by years, while others have never been true in the time I've been around. Without getting into a point-by-point rebuttal, my reaction to each section/Exhibit is "that's wrong/misleading".

Isn't that a bit disingenuous? Your role at Amazon is nothing like the typical engineer at Amazon. You live in the shiny new world while the majority of engineers are stuck on something that's not too far from the article.

I was an intern at AWS this past summer(and will be returning as a new grad in May) and this article is not accurate based on my experience. My code (which was part of our team's actual production system, not a toy project) ran on EC2 instances in an autoscaling group. There was no manually provisioning servers. I worked with a Kinesis Stream and several other resources and there was a way for us to programmatically retrieve the names of the resources. My impression was the system we used to find a resource name works for all AWS resources so I imagine you could get the name of a load balancer this way as well. Finally, I'll just say my team worked entirely in Python. Maybe that was an exception, but it is certainly sufficient to directly falsify the statement that "internally, at Amazon, you can't even use a language other than Java." (How would the author even know this? Was he in on the S-team meeting where it was declared that all non java users get a pip?)

Re: On Internal Engineering Practices at Amazon

#32

Well, this article gets something right, but gets a lot of stuff wrong as well. What can be confirmed is that his access to the Amazon tech scene is limited, and he takes a sweeping generalization that is how the whole Amazon works. Disclaimer: Ex-Amazonian, left like one year ago.

I agree. Disclaimer: Also ex-amazonian, left like one year ago.

What happened like one year ago?

Re: On Internal Engineering Practices at Amazon

#34
post #10

Based on my experience, the article contains a lot of misinformation. Some of the statements might have been true at one point in the past, but are now out of date by years, while others have never been true in the time I've been around. Without getting into a point-by-point rebuttal, my reaction to each section/Exhibit is "that's wrong/misleading".

Isn't that a bit disingenuous? Your role at Amazon is nothing like the typical engineer at Amazon. You live in the shiny new world while the majority of engineers are stuck on something that's not too far from the article.

The median tenure of engineers at Amazon is 1 year. That means that the new engineers need senior and principle engineers to guide them on what tools exist. If an engineering organization happens to have strong senior Amazon engineers then they can guide their teams/org to use the tools that exist, because they do exist.

However, everything (and I mean everything) at Amazon depends on the team (and organization) that you land in. Some organizations do not have senior technical leadership; service ownership is handed off to teams without long tenured Amazon engineers so they do not get exposed to the types of tools to use (nor do these teams get time to discover, learn, and on-board to the tools that do exist). This is how an engineer can have the experience written about in the article.

The article is anecdotal, and definitely not the norm for the "majority of engineers"

Re: On Internal Engineering Practices at Amazon

#35
Context: I spent five years as an engineer at Amazon, the last two as a tech lead on an internal developer tool (think SaaS for performance engineering).

This article is not untrue but it misses the fact that teams are empowered to own their solutions are not restricted in how they setup their environments and which tools they use. It's true that fixing these problems feels like wasted effort, it's by design: Amazon operates as many separate internal entities and I think replication of effort is an acknowledged downside of operating this way.

> 1. Deployments > Their internal deployment tool at Amazon is Apollo, and it doesn't support auto-scaling.

I had to manually scale up my service once in two years and we weren't over-provisioning wastefully. Before I left my product was supporting +40K internal applications with an infra+AWS cost We had good CI with deep integration with Apollo, you could track any change across the pipeline, we had reproducible builds and we had a comprehensive deployment log listing all changes.

Apollo is sloooooow though and the UI is very 90s.

> 2. Logs > Any self respecting company running software on distributed machines should have centralized, searchable logs for their services.

We were using Elastic Logstash Kibana powered by AWS ElasticSearch. I wrote a thin wrapper around logstash that was used in over 1K environments internally, so weren't the only ones doing this.

> 3. Service Discovery > What service discovery? We used to hard wire load balancer host names in config files.

Agree with this one. I will never forget the quality time I spent configuring those load balancers and ticketing people about DNS.

> 4. Containers

As other commenters mentioned, if you want to use containers, you're free to bypass all of this and run your service in AWS where you can use ECR, EKS etc if you want.

> (As far as I know, Node.js was used inside in a limited capacity, but they had an alternative for npm for security reasons, and you had to get an npm package approved to use it internally.)

I built my UI from scratch using create-react-app and yarn offline builds (no mystery meat) and I bypassed all the internal JS tooling, which I thought was very poor. This was changing though.

Finally, my personal anecdote: you could onboard our product in less than an hour (including reading docs), it required no further maintenance and gave you performance stats for free. So not all was bad :)

Re: On Internal Engineering Practices at Amazon

#36

This matches my experience using the internal tools pretty well. But hte team I was on was mostly using AWS, which meant the tooling was better. We also had a bunch of people who wanted python instead of Ruby, so they started using python and eventually we were just a team that used python. Cool. Apollo is probably the worst thing. Brazil, Pipelines, etc... those are mostly fine. Amazon has done a ton with the JVM. T…

Working on a side project is super easy. Just file a TT (or I guess SIM now?). As long as it isn't competing and not a game, it's generally quick approval.

Re: On Internal Engineering Practices at Amazon

#37
It's hardly surprising. The quality of their public products isn't much better than what's described here. It's fine for companies with plenty of engineers, money, and time to base their tools off of, but that's about it. Without building a ton of extra tooling and having specialized information only available through paid support, it's almost impossible to operate anything on aws. The documentation is plentiful, mostly out of date, wrong, incomplete, and difficult to browse, search, and use. Doing devops using aws is a nightmare that never ends. Not to mention the speed of deploying anything is beyond slow, so any work takes many times the amount of time it should. For large companies with plenty of resources, these are minor points. For small and medium sized ones, it's a loss of productivity and money that simply cannot be justified over other methods.

Re: On Internal Engineering Practices at Amazon

#38
post #36

This matches my experience using the internal tools pretty well. But hte team I was on was mostly using AWS, which meant the tooling was better. We also had a bunch of people who wanted python instead of Ruby, so they started using python and eventually we were just a team that used python. Cool. Apollo is probably the worst thing. Brazil, Pipelines, etc... those are mostly fine. Amazon has done a ton with the JVM. T…

Working on a side project is super easy. Just file a TT (or I guess SIM now?). As long as it isn't competing and not a game, it's generally quick approval.

What is the issue with games?

Re: On Internal Engineering Practices at Amazon

#39
post #36

Earlier quoted context omitted.

Working on a side project is super easy. Just file a TT (or I guess SIM now?). As long as it isn't competing and not a game, it's generally quick approval.

What is the issue with games?

Amazon owns game studios and might have many different game concepts being worked on, such that an employee might inadvertently compete with a similar game studio idea.

Re: On Internal Engineering Practices at Amazon

#40

Ex-Amazon SDE here. A lot of this has changed. First, there is a movement to build a lot of services in Native AWS instead of MAWS/Apollo. Apollo doesn't require copying configs anymore; you can have the config exist as part of the package you are deploying. Generally, that's a best practice. Pipelines can be configured as code too. There is a centralized log service which requires onboarding. It does require some co…

Any take on how this looks for someone who doesn't work at Amazon?

I know Apollo is their deployment tool. No idea how they plays into a Kubernetes or even container stack.

Post reply on HN