Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On Internal Engineering Practices at Amazon

jatins.gitlab.io

Post reply on HN
Print view View original

111–120 of 130 posts

Re: On Internal Engineering Practices at Amazon

#111

Post by andrewguenther »

throwaway1280 wrote: post #6
throwaway1280 wrote: post #4

Ex-Amazon engineer of several years here. This is a pretty interesting article, but it's important to know that Amazon's internal tooling changes pretty fast, even if it's mostly several years behind state-of-the-art. Exhibit A: Apollo Apollo used to be insane . It was designed for the use case of deploying changes to thousands of C++ CGI servers on thousands of website hosts, worrying about compiling for different a…

Speaking of twitchy lawyers and Move to AWS... one of the weirdest things we had to deal with inside Amazon was that, for many years after AWS launched, we weren't allowed to use it because it "wasn't secure enough". Given that we were actively shopping it around to major financial institutions at the time, doesn't that strike you as particularly hypocritical? :)

This is misleading. This had more to do with internal audit tools and guardrails available than it did the services themselves.

Re: On Internal Engineering Practices at Amazon

#112

Post by blub »

geggam wrote: post #100
mmartinson wrote: post #95

Earlier quoted context omitted.

Definitely, but, I'm sure that adopting new technologies at the right time can be a huge competitive edge for startups. Sure it can fail horribly in some places where hindsight will show that the trusted stack was the right choice, but it could also mean winning a race to market and continued relevance for a company if it works out.

Startups are complete risk and should jump on the new cool and shiny toy ship. When you start making money or wish to start making money you need to stabilize the insanity. I have done both.

Something that's "complete risk" should take even more risk? Doesn't seem logical.

Re: On Internal Engineering Practices at Amazon

#113

Post by m0zg »

scott_s wrote: post #92
m0zg wrote: post #57

Earlier quoted context omitted.

Containers are just namespaces for things within the Linux kernel. Unlike with VMs, you're not running separate instances of the OS, it's all run by just one kernel instance, and that kernel usually runs directly on the hardware, that is, on "bare metal". That "ubuntu" base image you can spin up does not actually run the Ubuntu kernel. As a result, bare metal containers incur none of the "virtualization penalty" that…

It's tempting to say that containers don't have any virtualization costs, but they often do for networking. For some applications, the addition of another networking driver doesn't matter. For others, it does.

There is no "another networking driver" in containers. It's the same networking driver.

Re: On Internal Engineering Practices at Amazon

#114

Post by alfalfasprout »

geggam wrote: post #91
lewisjoe wrote: post #88

I worked at a startup and then switched to a bigger product company: Zoho. If there's one thing I'd take away for the rest of my career from Zoho, it would be frugality in adopting the latest of tech. When NoSQL was all the rage, the company stood firm that relation databases had rock-solid mathematical foundation and stayed away from the bandwagon. It paid off. When every other company wrote blogs about rewriting th…

I have shared this blog posting more than I care to admit. https://mcfunley.com/choose-boring-technology It amazes me that people will put their companies and employees at risk by using the new cool stuff just because everyone else is ( looking at you k8s )

This is one position... albeit an extreme one.

Kubernetes has worked perfectly for me in multiple places now. Including where I am now at a large unicorn.

I agree, picking up all the shiny new stuff on hacker news is generally a bad idea. But foresaking new technologies altogether can backfire.

Part of it is how you handle failures. You could always have an aerospace-grade QA process... but for a consumer web product that's almost always going to vastly delay your progress. If you have the ability to fix issues quickly then it's sometimes better to "move fast and break things". If you're working on autonomous vehicle software then this is a terrible idea (I'm looking at you Uber ATG).

Re: On Internal Engineering Practices at Amazon

#115

Post by michaelper22 »

vonseel wrote: post #82

As far as I know, Node.js was used inside in a limited capacity, but they had an alternative for npm for security reasons, and you had to get an npm package approved to use it internally. FWIW, this is a very good thing . A company as large as Amazon should do this with all of their repositories; even a small start-up should be doing this to mitigate suspicious packages / third-party code vulnerabilities.

Morgan Stanley revealed at a recruiting event that they follow this practice. "You can definitely use Apache Spark! It's been reviewed by our Enterprise Architecture team; some APIs were removed, and we substituted 'Morganized' variants of others, which you won't find code/docs for on Github ;)"

Given the npm debacle, I could totally see even a small org running an internal Maven repo with approved versions of (popular, and especially obscure) libs.

Re: On Internal Engineering Practices at Amazon

#116

Post by ilovecaching »

Frost1x wrote: post #103
School-Cotton wrote: post #62

Earlier quoted context omitted.

Very good comment, +1. People don't really get the rabbit hole that is necessary in order to introduce a new technology at a large company. Just off the top of my head, a few of the issues: * Not everyone has followed the broader technology world outside the big company. It's totally possible to imagine a situation where nobody in your management chain or team has even heard of something relatively mainstream like Do…

"...or it has never been used with a codebase big enough to take >1 day to build..." I've not worked in this scale of enterprise environments so I find this tidbit fascinating. Why does the codebase take over a day to build? Does that time include rebuilding and deploying the underlying infrastructure? I'm genuinely interested here. I can only imagine a few highly niche cases myself so I'm curious what I'm missing he…

If you look at buck or hazel, they both use caching or artifacts to speed up builds. It still takes a long time, because everything depends on everything. But for most projects a fresh build is going to take tens of minutes normally for a medium sized project pulling in a big deep.

Re: On Internal Engineering Practices at Amazon

#117

Post by amzn-throw »

mrep wrote: post #89
jcrites wrote: post #17

Earlier quoted context omitted.

Based on my experience, that information and some of the comments about it in the thread are out of date or inaccurate. 'Move to AWS' was a program focused on accelerating AWS adoption that was primarily active something like 5-7 years ago. The program achieved its goals and concluded: virtually all infrastructure was running on AWS. I worked on the program for part of that time, in the last couple of years it was ac…

Until they move off of sable, their NoSQL backend for retail, calling them mostly on AWS is laughable especially considering their major prime day outage this last year was caused by sable not being able to dynamically scale up [0]. [0]: https://www.cnbc.com/2018/07/19/amazon-internal-documents-wh...

Depends on what you mean by mostly. Almost all code runs on EC2. All object storage is in S3. Almost all service asynchronous interactions are decoupled via SQS. Almost all notifications are shared via SNS.

Yes, most legacy systems use Sable but most new development for the last 2 years uses DynamoDB.

More and more event-driven applications with highly variable throughout are on Lambda. Even on AWS Service teams.

The Sable outage had a particularly laughable interpretation by the armchair quarterbacks from CNBC. The kind of scale involved makes Oracle-based approaches completely infeasible. (the internal Correction of Error document leaked and was wildly mis understood)

And the scaling timelines involved are so compressed that no company in their right mind "dynamically scales up as a result" - it's always projected scheduled scaling. There were other cascading effects.

I know it's fun to dunk on Amazon but their commitment to operational excellence is unparalleled as public AWS post mortems after major events should reveal.

If you read the entire Correction of Error document on tbe Sable outage you'd agree, of course CNBC would never publish that, one gets more clicks by getting a professor out of touch with the realities of production software engineering to blurb some juicy quotes.

Re: On Internal Engineering Practices at Amazon

#118

Post by jacobparker »

jrockway wrote: post #85
vonseel wrote: post #82

As far as I know, Node.js was used inside in a limited capacity, but they had an alternative for npm for security reasons, and you had to get an npm package approved to use it internally. FWIW, this is a very good thing . A company as large as Amazon should do this with all of their repositories; even a small start-up should be doing this to mitigate suspicious packages / third-party code vulnerabilities.

I worked on third-party package approvals at Google. The reasoning behind reviews was largely due to license compliance. If the license said "you have to display this license to end-users" then we had to make sure that the license was machine-readable and would be automatically bundled into the build to be displayed in that "open source licenses" section of pretty much every app ever. If the license said "by linking…

There's some public documentation on the "one version" policy here: https://opensource.google.com/docs/thirdparty/oneversion/

Re: On Internal Engineering Practices at Amazon

#119

Post by scott_s »

m0zg wrote: post #113
scott_s wrote: post #92

Earlier quoted context omitted.

It's tempting to say that containers don't have any virtualization costs, but they often do for networking. For some applications, the addition of another networking driver doesn't matter. For others, it does.

There is no "another networking driver" in containers. It's the same networking driver.

For standard Docker container deployments, there is. See: https://docs.docker.com/network/

Re: On Internal Engineering Practices at Amazon

#120

Post by tokyoHacker »

tanilama wrote: post #14

Well, this article gets something right, but gets a lot of stuff wrong as well. What can be confirmed is that his access to the Amazon tech scene is limited, and he takes a sweeping generalization that is how the whole Amazon works. Disclaimer: Ex-Amazonian, left like one year ago.

Ofcourse. What else can we expect from someone who has been there for just 2 years. It takes some audacity to speak about the tech scene of a company as large as Amazon with such limited time spent. Unless he has the overall view of the org ( like a VP of Engineering), I would take any assessment with a pinch of salt.
Post reply on HN