Live data from Hacker News

On Internal Engineering Practices at Amazon

jatins.gitlab.io

41–50 of 130 posts

Re: On Internal Engineering Practices at Amazon

#41

Ex-Amazon SDE here. A lot of this has changed. First, there is a movement to build a lot of services in Native AWS instead of MAWS/Apollo. Apollo doesn't require copying configs anymore; you can have the config exist as part of the package you are deploying. Generally, that's a best practice. Pipelines can be configured as code too. There is a centralized log service which requires onboarding. It does require some co…

> you would want to use bare metal for Amazon's scale

Containers are "bare metal".

Re: On Internal Engineering Practices at Amazon

#42

Ex-Amazon engineer of several years here. This is a pretty interesting article, but it's important to know that Amazon's internal tooling changes pretty fast, even if it's mostly several years behind state-of-the-art. Exhibit A: Apollo Apollo used to be insane . It was designed for the use case of deploying changes to thousands of C++ CGI servers on thousands of website hosts, worrying about compiling for different a…

Your comment is better than the original article. Can we push this one to the top, HN?

Re: On Internal Engineering Practices at Amazon

#43
post #41

Ex-Amazon SDE here. A lot of this has changed. First, there is a movement to build a lot of services in Native AWS instead of MAWS/Apollo. Apollo doesn't require copying configs anymore; you can have the config exist as part of the package you are deploying. Generally, that's a best practice. Pipelines can be configured as code too. There is a centralized log service which requires onboarding. It does require some co…

> you would want to use bare metal for Amazon's scale Containers are "bare metal".

Yeah i was confused by this too. Parent possibly thinking of virtual machines?

Re: On Internal Engineering Practices at Amazon

#44
post #41

Earlier quoted context omitted.

> you would want to use bare metal for Amazon's scale Containers are "bare metal".

Yeah i was confused by this too. Parent possibly thinking of virtual machines?

Fun fact: all of Google (including Google Cloud) is run in containers. IIRC even VMs are run in containers. The VMs, of course, could contain customer containers, too. It's literally containers "all the way down".

Re: On Internal Engineering Practices at Amazon

#45
What amuses me is that most of retractions come from ex-Amazonians not from the current staff. This is the only company i know dealing with that much criticism from engineering.

Even more to add is that the article is more or less fresh and at Amazon's scale i doubt any major changes had undergone in the last 10 months.

Re: On Internal Engineering Practices at Amazon

#46
post #37

It's hardly surprising. The quality of their public products isn't much better than what's described here. It's fine for companies with plenty of engineers, money, and time to base their tools off of, but that's about it. Without building a ton of extra tooling and having specialized information only available through paid support, it's almost impossible to operate anything on aws. The documentation is plentiful, mos…

Dang probably true for the products you were using but Redshift has been great for me, including the documentation which I have never found to be out of date.

I wouldn’t say its the economical option though.

Oh and its high time ‘count(distinct) over()’ partitions was supported.

Re: On Internal Engineering Practices at Amazon

#47

Ex-Amazon SDE here. A lot of this has changed. First, there is a movement to build a lot of services in Native AWS instead of MAWS/Apollo. Apollo doesn't require copying configs anymore; you can have the config exist as part of the package you are deploying. Generally, that's a best practice. Pipelines can be configured as code too. There is a centralized log service which requires onboarding. It does require some co…

Any take on how this looks for someone who doesn't work at Amazon? I know Apollo is their deployment tool. No idea how they plays into a Kubernetes or even container stack.

Conceptually, there is a lot of overlap. Apollo had a few layers to it but would, more or less, matched kubernetes plus additional tooling. Dare I say closer to openshift? Brazil, the build tool, infrastructure and repository was similar to using nixpkgs. Brazil had some features beyond nixpkgs that were absolutely essential for an business like Amazon, but otherwise the goals and even techniques overlapped.

I onboarded very, very quickly by already knowing nixpkgs and container management. The concepts translated well. A few aspects that didn't either were things Amazon handled that other tools have yet to encounter: Cool to learn about. Or historical bits that didn't matter: fine to ignore.

Re: On Internal Engineering Practices at Amazon

#48
One misconception people get wrong, from my perspective, about most FAANG is that you get to work with new and shiny things, especially new langauges. This is really more applicable to startups where risk taking is in the DNA, you will mostly get really strong pushback because there is just too much effort involved to support more than two or three languages at scale. There are normally niche languages, but they are essentially statistical anomalies and are usually borne out of a real business need (like swift for iOS). Mostly engineering for engineering sake is also going to be frowned upon unless it really helps the business.

I do agree that Amazon is the worst in regards to OSS. They really need to fix that, even if jut for PR, because they are consuming so much of it for AWS.

Re: On Internal Engineering Practices at Amazon

#49

I will be joining Amazon in about a month. Is there any chance I'll be able to work on OSS and/or "modern" tech (e.g. containers, Go, etc.) without a ton of push-back? It also seems Amazon is obsessed with reinventing wheels and keeping their stuff internal, which is worrying. Is there any chance to introduce solid OSS tools to the development process? (whatever they might be)

I worked there for two years (summer 2013-summer 2015), so long enough to get an idea of the culture. Take this post with the caveat that I have no idea whether it's changed since then and if so to what extent.

> Is there any chance I'll be able to work on OSS and/or "modern" tech (e.g. containers, Go, etc.) without a ton of push-back?

Sure there's a greater chance than zero, but not if only motivated by it being "modern". Amazon is a business. It exists to make money by providing a valuable service to customers. The only reason it's a "tech company" is because writing software serves that goal, but "cool tech" is not a goal in itself. If you have a serious, documentable reason why rewriting your team's service in Go would help you achieve business goals, then you might be able to get the attention of engineering decision makers, but that's a much higher bar than "it's modern".

Personally I don't see what is "modern" about Go or how it would help the business serve customers better and/or make more money than it does using Java. I suspect many of your coworkers would feel the same way and these are the terms that the decision would be framed in within Amazon's culture.

By the way there has been over the last several years a move away from C++ and Perl and towards Java. When I was there the majority of stuff was in Java but there was still plenty of important stuff in those other two languages. I suspect C++ and Perl are even rarer now. I guess that's the modernization you're talking about, but maybe not at the pace you want.

> It also seems Amazon is obsessed with reinventing wheels

In many cases, either these wheels were invented at Amazon before they became available in the OSS world, or the OSS tools do not fit Amazon's needs (especially w.r.t. scale).

There is certainly no "obsession" with reinventing wheels -- any sane manager at Amazon would definitely rather use something off-the-shelf than waste a bunch of money developing it from scratch, assuming it fit their needs well.

> and keeping stuff internal

Well, I guess this is true. Amazon contributes less to open source than a few other famous tech companies.

> Is there any chance to introduce solid OSS tools to the development process?

Amazon uses plenty of very solid OSS tools: for example, Linux, Java, gcc, perl, git (though they were on Perforce when I started), and Tomcat are all core parts of Amazon infrastructure. As well as the same grab bag of common tools and libraries you'd find in use anywhere else. In general things with permissive licenses (BSD/MIT/etc) are fair game, but getting approval for things with copyleft licenses (GPL/LGPL/etc) is an uphill battle. As for whether you could introduce more, it would depend on the value of the tool and your motivation for doing so.

Re: On Internal Engineering Practices at Amazon

#50
post #41

Ex-Amazon SDE here. A lot of this has changed. First, there is a movement to build a lot of services in Native AWS instead of MAWS/Apollo. Apollo doesn't require copying configs anymore; you can have the config exist as part of the package you are deploying. Generally, that's a best practice. Pipelines can be configured as code too. There is a centralized log service which requires onboarding. It does require some co…

> you would want to use bare metal for Amazon's scale Containers are "bare metal".

> Containers are "bare metal"

Please explain this use of jargon. Are you saying that a Docker container is "bare metal" in some culture? A VM is bare metal? What do you mean by container, and bare metal?

Post reply on HN