Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On Internal Engineering Practices at Amazon

jatins.gitlab.io

Post reply on HN
Print view View original

21–30 of 130 posts

Re: On Internal Engineering Practices at Amazon

#21

Post by anth_anm »

This matches my experience using the internal tools pretty well.

But hte team I was on was mostly using AWS, which meant the tooling was better.

We also had a bunch of people who wanted python instead of Ruby, so they started using python and eventually we were just a team that used python. Cool.

Apollo is probably the worst thing. Brazil, Pipelines, etc... those are mostly fine.

Amazon has done a ton with the JVM. They don't need something new, so they don't bother. That's fine.

They also do adopt other tech as needed to do things. I know people who worked in Go, because it was container stuff.

This was also one of things that gave me pause when considering interviewing with snap a while back. So many former Amazon people. Seemed like a lot of stuff was going with "just copy amazon, make it a bit better". I don't want to write java. bleh.

Anyway, this article is going to have a lot of people saying "oh no that's not right". It is. There are exceptions but overall it's pretty much bang on.

Oh and open source. My understanding is jeff doesn't like contributing back. The company doesn't like contributing back. They keep an iron grip on IP. They are ridiculous about letting employees do side projects. They fight back with every single FOSS contribution and even after some of our senior guys did a whole bunch of work it still required multiple layers of approvals and a whole bunch of hooops and an extra training course and blah blah blah blah. It's really really dumb. I find it crazy that Amazon doesn't get more flak for being probably the worst company for open source around right now.

But MS has tossed out their old CEOs, has tons of interest in making open source work well for them, and contributes loads... still get shit on.

Re: On Internal Engineering Practices at Amazon

#22

Post by anth_anm »

throwaway772643 wrote: post #19

I will be joining Amazon in about a month. Is there any chance I'll be able to work on OSS and/or "modern" tech (e.g. containers, Go, etc.) without a ton of push-back? It also seems Amazon is obsessed with reinventing wheels and keeping their stuff internal, which is worrying. Is there any chance to introduce solid OSS tools to the development process? (whatever they might be)

No, that's not accurate at all.

Amazon made choices on how to do things, years ago. These choices are being remade just because the new hotness is containers and not using uncool java. They have a lot of tooling that works for them, and a lot of it is quite good.

They aren't hostile to new stuff either. It's just that why would you waste time and money trying to shoehorn some new way of doing things when the old way works just fine.

You will still get to do lots. maybe, depends on the team.

I would say don't waste your energy. Learn when to pick the battles. Accept that you will get push back that doesn't make sense to you.

Re: On Internal Engineering Practices at Amazon

#23

Post by anth_anm »

kerng wrote: post #11

I like how Amazon has an MAWS movement internally, meaning "Move to AWS". I think most people think that they use AWS mostly, but they dont. Its an interesting look behind the scenes at Amazon and how antiquated they appear to operate. Makes you wonder if Azure and Google have pretty good chances beating them down the road. Edit: Interesting, further down one person commented that Amazon doesn't use AWS broadly becau…

MS Internal tooling fucking sucks compared to Amazon.

Just atrocious. Though I wasn't in Azure so that might be better.

Re: On Internal Engineering Practices at Amazon

#24

Post by anth_anm »

Hamuko wrote: post #8
throwaway1280 wrote: post #6

Earlier quoted context omitted.

Speaking of twitchy lawyers and Move to AWS... one of the weirdest things we had to deal with inside Amazon was that, for many years after AWS launched, we weren't allowed to use it because it "wasn't secure enough". Given that we were actively shopping it around to major financial institutions at the time, doesn't that strike you as particularly hypocritical? :)

So wait, when I need to convince customers why AWS is secure for their data, I can't say "It's good enough for Amazon!"?

No, you can say hey amazon might use this if they do a security evaluation first.

It's totally cool for your data though, don't worry about it.

Amazon fucking sucks at dogfooding.

Re: On Internal Engineering Practices at Amazon

#26

Post by discodave »

throwaway772643 wrote: post #19

I will be joining Amazon in about a month. Is there any chance I'll be able to work on OSS and/or "modern" tech (e.g. containers, Go, etc.) without a ton of push-back? It also seems Amazon is obsessed with reinventing wheels and keeping their stuff internal, which is worrying. Is there any chance to introduce solid OSS tools to the development process? (whatever they might be)

AWS SDE here.

The short answer is, in order to get your team to adopt something, you need to make the case that it's better for customers (including things like migration costs). If the modern thing is more efficient, is higher availability, increases velocity, and so on then the case can be made.

Some specific examples based on things you cite:

* For an example of something OSS or "modern" coming from AWS, checkout Firecracker (written in Rust): https://firecracker-microvm.github.io/

* With regards to "reinventing wheels" Apollo + EC2 solves a lot (not all) of the problems that containers solve, and existed for years before containers became the hotness.

* Docker, which brought containers to the masses launched in 2013.

* EC2 launched in 2006 (7 years before Docker).

* Apollo (and the build system Brazil) predated EC2 by many years.

* Amazon.com was migrating to EC2/AWS before 2012 (https://www.youtube.com/watch?v=f45Uo5rw6YY)

* Another example, Lambda, which launched in 2014 runs on EC2 (https://www.youtube.com/watch?v=QdzV04T_kec&t=1611s).

* New services get to build in AWS and use Lambda, ECS, DynamoDB etc based on their business needs.

Re: On Internal Engineering Practices at Amazon

#27

Post by thramp »

throwaway1280 wrote: post #5
wmf wrote: post #3

Earlier quoted context omitted.

Yes, but it probably makes Node.js useless in any such company since any non-trivial app will have 1,000 npm dependencies.

Yeah. I did this once a few years ago, and it was quite unpleasant. Did get it done in the end, but it definitely put my team off looking for any other useful NPM packages. I wonder if it's any more streamlined now?

There's some internal builds tools that I can vouch for. If you're still at Amazon, feel free to ping me at dbarsky@ and we can chat.

Re: On Internal Engineering Practices at Amazon

#28

Post by jtoberon »

Hamuko wrote: post #8
throwaway1280 wrote: post #6

Earlier quoted context omitted.

Speaking of twitchy lawyers and Move to AWS... one of the weirdest things we had to deal with inside Amazon was that, for many years after AWS launched, we weren't allowed to use it because it "wasn't secure enough". Given that we were actively shopping it around to major financial institutions at the time, doesn't that strike you as particularly hypocritical? :)

So wait, when I need to convince customers why AWS is secure for their data, I can't say "It's good enough for Amazon!"?

To clarify, an AWS customer has a shared responsibility to describe the security of their systems including how they use AWS tools, and in this respect Amazon is no different than other AWS customers.

Re: On Internal Engineering Practices at Amazon

#29

Post by raiflip »

Don't want to go into too much detail but this article is like taking the crappiest parts of the crappiest systems and declaring it representative of an entire product. There is a lot of really good internal tooling not mentioned here, and for the internal tooling mentioned here (like Apollo) absolutely none of its benefits are mentioned.

Re: On Internal Engineering Practices at Amazon

#30

Post by throwawayamz27 »

The main benefit for amazon's tools is once you've been there a while you know how they work, and all the complexity and bugs have been stripped out of them. And because they force engineers to go oncall everyone has a pretty good idea of how to fix things.

When you have SRE's spending all day creating the next new thing (generally after deprecating the previous one with no replacement), you end up in a situation where you forget how to say, rollback a bad deployment. Or scale a fleet.

The problem with fancy infrastructure as code, containers and logging services is when they break you have no idea how to get out of trouble. SSH and grep almost always work, as does symlinking a directory.

Post reply on HN