Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On Internal Engineering Practices at Amazon

jatins.gitlab.io

Post reply on HN
Print view View original

81–90 of 130 posts

Re: On Internal Engineering Practices at Amazon

#81

Post by C1sc0cat »

tmarthal wrote: post #34
throwaway504 wrote: post #12

Earlier quoted context omitted.

Isn't that a bit disingenuous? Your role at Amazon is nothing like the typical engineer at Amazon. You live in the shiny new world while the majority of engineers are stuck on something that's not too far from the article.

The median tenure of engineers at Amazon is 1 year. That means that the new engineers need senior and principle engineers to guide them on what tools exist. If an engineering organization happens to have strong senior Amazon engineers then they can guide their teams/org to use the tools that exist, because they do exist. However, everything (and I mean everything ) at Amazon depends on the team (and organization) tha…

1 Year median wow - they must lose lot of institutional knowledge that way.

Re: On Internal Engineering Practices at Amazon

#82

Post by vonseel »

As far as I know, Node.js was used inside in a limited capacity, but they had an alternative for npm for security reasons, and you had to get an npm package approved to use it internally.

FWIW, this is a very good thing. A company as large as Amazon should do this with all of their repositories; even a small start-up should be doing this to mitigate suspicious packages / third-party code vulnerabilities.

Re: On Internal Engineering Practices at Amazon

#83

Post by yomly »

vonseel wrote: post #82

As far as I know, Node.js was used inside in a limited capacity, but they had an alternative for npm for security reasons, and you had to get an npm package approved to use it internally. FWIW, this is a very good thing . A company as large as Amazon should do this with all of their repositories; even a small start-up should be doing this to mitigate suspicious packages / third-party code vulnerabilities.

I feel like the whole NPM/JS culture is around "there's a library for that" so I can see how the friction around having to get approval for npm packages would be an absolute velocity killer, but when you're as big as Amazon I really wouldn't want that particular JS flavour of "move fast and break things" powering the company...

Re: On Internal Engineering Practices at Amazon

#84

Post by cimi_ »

cimi_ wrote: post #35

Context: I spent five years as an engineer at Amazon, the last two as a tech lead on an internal developer tool (think SaaS for performance engineering). This article is not untrue but it misses the fact that teams are empowered to own their solutions are not restricted in how they setup their environments and which tools they use . It's true that fixing these problems feels like wasted effort, it's by design: Amazon…

> They have some amount of Rails, and JavaScript has to be there, but if you want to experiment with, say, Go, Kotlin, or anything else, you are going to get nothing but push back.

I missed this - starting 2018 we were writing all our backend logic in Kotlin and we got no push back from anyone.

Re: On Internal Engineering Practices at Amazon

#85

Post by jrockway »

vonseel wrote: post #82

As far as I know, Node.js was used inside in a limited capacity, but they had an alternative for npm for security reasons, and you had to get an npm package approved to use it internally. FWIW, this is a very good thing . A company as large as Amazon should do this with all of their repositories; even a small start-up should be doing this to mitigate suspicious packages / third-party code vulnerabilities.

I worked on third-party package approvals at Google. The reasoning behind reviews was largely due to license compliance. If the license said "you have to display this license to end-users" then we had to make sure that the license was machine-readable and would be automatically bundled into the build to be displayed in that "open source licenses" section of pretty much every app ever. If the license said "by linking this into your code, you have to opensource all code at your company", we had to deny it. That sort of thing.

We suggested that people get security reviews, but it was up to the user of the package to figure out whether or not that was necessary. Often security reviews would be blocking the project's launch and would be done at that time.

The final thing we enforced was a "one version" policy. If everyone was using foobar-1.0, and you wanted to use foobar-2.0, it was on you to update everyone to foobar-2.0. This was the policy that people hated the most, but basically mandatory at the time because none of the languages widely used at Google supported versioned symbols. Having library A depend on foobar-1.0 and library B depending on foobar-2.0 meant that application C could not depend transitively on library A and library B at the same time, which would cause many disasters.

Re: On Internal Engineering Practices at Amazon

#86

Post by lunchables »

jhayward wrote: post #68
m0zg wrote: post #57

Earlier quoted context omitted.

Containers are just namespaces for things within the Linux kernel. Unlike with VMs, you're not running separate instances of the OS, it's all run by just one kernel instance, and that kernel usually runs directly on the hardware, that is, on "bare metal". That "ubuntu" base image you can spin up does not actually run the Ubuntu kernel. As a result, bare metal containers incur none of the "virtualization penalty" that…

Ah, you're using it to mean "no virtualization". Got it, thanks. I tend to use the term to mean single-tenant, user-supplied OS, non-virtualized myself but I see where you're coming from.

My workstation runs multiple processes in multiple namespaces without containers. Is it bare metal? What if I install applications via flatpak? Still bare metal? Now what if I use docker?

Re: On Internal Engineering Practices at Amazon

#87

Post by nerdponx »

jrockway wrote: post #85
vonseel wrote: post #82

As far as I know, Node.js was used inside in a limited capacity, but they had an alternative for npm for security reasons, and you had to get an npm package approved to use it internally. FWIW, this is a very good thing . A company as large as Amazon should do this with all of their repositories; even a small start-up should be doing this to mitigate suspicious packages / third-party code vulnerabilities.

I worked on third-party package approvals at Google. The reasoning behind reviews was largely due to license compliance. If the license said "you have to display this license to end-users" then we had to make sure that the license was machine-readable and would be automatically bundled into the build to be displayed in that "open source licenses" section of pretty much every app ever. If the license said "by linking…

This all sounds like justified inconvenience in the name of safety at scale. As long as you have a good relationship with the package approval guys this is a fine workflow in my experience.

Re: On Internal Engineering Practices at Amazon

#88

Post by lewisjoe »

I worked at a startup and then switched to a bigger product company: Zoho.

If there's one thing I'd take away for the rest of my career from Zoho, it would be frugality in adopting the latest of tech.

When NoSQL was all the rage, the company stood firm that relation databases had rock-solid mathematical foundation and stayed away from the bandwagon. It paid off.

When every other company wrote blogs about rewriting their software in NodeJS, Ruby & Python, the company stood ground with statically typed languages. It paid off.

My own team, Zoho Writer has a strong policy against incorporating third party libraries without good reasons. This way, the product is nearly a decade old, but the JS size has remained surprisingly small, all through its evolution.

I believe staying frugal in adopting the latest hype, can only be reasoned about in hindsight.

Re: On Internal Engineering Practices at Amazon

#89

Post by mrep »

jcrites wrote: post #17
kerng wrote: post #11

I like how Amazon has an MAWS movement internally, meaning "Move to AWS". I think most people think that they use AWS mostly, but they dont. Its an interesting look behind the scenes at Amazon and how antiquated they appear to operate. Makes you wonder if Azure and Google have pretty good chances beating them down the road. Edit: Interesting, further down one person commented that Amazon doesn't use AWS broadly becau…

Based on my experience, that information and some of the comments about it in the thread are out of date or inaccurate. 'Move to AWS' was a program focused on accelerating AWS adoption that was primarily active something like 5-7 years ago. The program achieved its goals and concluded: virtually all infrastructure was running on AWS. I worked on the program for part of that time, in the last couple of years it was ac…

Until they move off of sable, their NoSQL backend for retail, calling them mostly on AWS is laughable especially considering their major prime day outage this last year was caused by sable not being able to dynamically scale up [0].

[0]: https://www.cnbc.com/2018/07/19/amazon-internal-documents-wh...

Re: On Internal Engineering Practices at Amazon

#90

Post by Pfhreak »

ex_amazon_sde wrote: post #74
coreyoconnor wrote: post #47

Earlier quoted context omitted.

Conceptually, there is a lot of overlap. Apollo had a few layers to it but would, more or less, matched kubernetes plus additional tooling. Dare I say closer to openshift? Brazil, the build tool, infrastructure and repository was similar to using nixpkgs. Brazil had some features beyond nixpkgs that were absolutely essential for an business like Amazon, but otherwise the goals and even techniques overlapped. I onboar…

> Apollo had a few layers to it but would, more or less, matched kubernetes plus additional tooling Have you looked at the codebase? Apollo and Brazil are tiny compared to Kubernetes & friends. You could reimplement them with a small script on top of traditional Linux package management (modulo the web UI).

As someone who worked on Apollo, I'd challenge this. Without divulging any details, Apollo is a non trivial system.
Post reply on HN