Do big companies like Google and Amazon fire developers for buggy releases? • Hacker News Board

Re: Do big companies like Google and Amazon fire developers for buggy releases?

#11

Post by notacoward » Sat, May 25, 2019, 11:49 AM UTC

At traditional product-oriented companies, buggy releases are recognized as a programmer-performance issue and dealt with eventually (though as others have pointed out immediate firing would be absurd). At modern internet-facing companies, developers who produce buggy releases are rewarded. It's not intentional. It's what happens when performance is evaluated 100% on releases regardless of bugginess. If you push lots of code, you get rewarded. If you slow down to ensure quality, or spend time fixing code that already exists, you get penalized.

BTW my impression is that, within that peer group, both Google and Amazon are on the better side. They don't have "move fast and break stuff" as a core engineering value.

Re: Do big companies like Google and Amazon fire developers for buggy releases?

#12

Post by msiyer » Sat, May 25, 2019, 12:32 PM UTC

planetjones wrote: post #8

I have never seen any case of a developer holding the project to ransom in order to fix a bug, as alluded to in the article. That said the key point that those who are always firefighting and ‘saving the day’ do often get the recognition as opposed to the people who squat the bugs early, follow sane processes and test their own stuff well is true. But the question is about tech companies (not outsourcing shops like i…

This is so funny. I once worked on a year long project. The aim was to reduce customer escalations for my client. Changed the architecture of an extremely complex software making sure no regressions occurred and all new features worked flawlessly. The software was released and no customer escalations. I was happy, but found no one else cared. A few months later, during a meeting with the client manager, I brought this point up. I was told that since I delivered an error-free software, people forgot about it and that is a good thing. I wad told that if I wanted to get noticed, I should firefight instead of writing bullet-proof software.

Re: Do big companies like Google and Amazon fire developers for buggy releases?

#13

Post by jdm2212 » Sat, May 25, 2019, 12:40 PM UTC

I've worked at Google and Amazon and they would never fire a developer for a buggy release. All engineers write buggy code. With mentorship, code reviews, and reliable test infrastructure, they write fewer bugs. With strong QA and monitoring they ship a lot fewer bugs.

Google and Amazon are not dumb enough to fire an engineer who would cost 10s to 100s of thousands of dollars to replace because that engineer happened to get unlucky and push a release that slipped through all those cracks. (Doing so maliciously would be another story, and would warrant firing.)

Re: Do big companies like Google and Amazon fire developers for buggy releases?

#14

Post by kevinventullo » Sat, May 25, 2019, 12:59 PM UTC

msiyer wrote: post #12

planetjones wrote: post #8

I have never seen any case of a developer holding the project to ransom in order to fix a bug, as alluded to in the article. That said the key point that those who are always firefighting and ‘saving the day’ do often get the recognition as opposed to the people who squat the bugs early, follow sane processes and test their own stuff well is true. But the question is about tech companies (not outsourcing shops like i…

This is so funny. I once worked on a year long project. The aim was to reduce customer escalations for my client. Changed the architecture of an extremely complex software making sure no regressions occurred and all new features worked flawlessly. The software was released and no customer escalations. I was happy, but found no one else cared. A few months later, during a meeting with the client manager, I brought thi…

My take is that you don't get rewarded for it directly, but rather once you're finished, you can move on to the next thing without having to constantly firefight and squash bugs on the previous project. So ideally you can get a lot more done in a given half than someone who pushed out a pile of buggy spaghetti that they're now constantly having to deal with.

On the other hand, if you can ship something a few weeks faster that might have a few bugs, but the customer never notices, I can understand why a business would reward that.

Re: Do big companies like Google and Amazon fire developers for buggy releases?

#15

Post by allengeorge » Sat, May 25, 2019, 1:32 PM UTC

The company I work at does not fire people for introducing bugs (everyone introduces bugs at some point and time). We try to find out what failures in our process (dev, test, release) contributed to the failure and try fix them. Some of the causes may be large, so it takes us time to get the organizational commitment and focus to work on them, but we do act.

I have (personally) been fortunate never to have worked with an “S”, so I think the author is being overbroad.

Re: Do big companies like Google and Amazon fire developers for buggy releases?

#16

Post by sys_64738 » Sat, May 25, 2019, 1:40 PM UTC

Everybody writes buggy code and those who say they don't are kidding themselves. The most important thing is how you react to bugs being found. The right answer is to acknowledge the problem and fix it. Write a unit test so it never occurs again in this code. If you get hit by a common type of bug that's preventable in code, often, then your unit testing is not up to scratch.

If you write software for production and don't do unit testing then you are in the wrong job.

Re: Do big companies like Google and Amazon fire developers for buggy releases?

#17

Post by sys_64738 » Sat, May 25, 2019, 1:44 PM UTC

msiyer wrote: post #12

planetjones wrote: post #8

I have never seen any case of a developer holding the project to ransom in order to fix a bug, as alluded to in the article. That said the key point that those who are always firefighting and ‘saving the day’ do often get the recognition as opposed to the people who squat the bugs early, follow sane processes and test their own stuff well is true. But the question is about tech companies (not outsourcing shops like i…

This is so funny. I once worked on a year long project. The aim was to reduce customer escalations for my client. Changed the architecture of an extremely complex software making sure no regressions occurred and all new features worked flawlessly. The software was released and no customer escalations. I was happy, but found no one else cared. A few months later, during a meeting with the client manager, I brought thi…

This is similar to all the fixed issues we solved for Y2K. When nothing happened then rather than saying 'well done to all who fixed the issues' it became 'that was a waste of time and overblown as nothing happened.' Sometimes I think they wished a plane fell from the sky due to a software bug so they could slam software developers for incompetence.

There's no middle ground :(

Re: Do big companies like Google and Amazon fire developers for buggy releases?

#18

Post by msiyer » Sat, May 25, 2019, 2:18 PM UTC

kevinventullo wrote: post #14

msiyer wrote: post #12

Earlier quoted context omitted.

This is so funny. I once worked on a year long project. The aim was to reduce customer escalations for my client. Changed the architecture of an extremely complex software making sure no regressions occurred and all new features worked flawlessly. The software was released and no customer escalations. I was happy, but found no one else cared. A few months later, during a meeting with the client manager, I brought thi…

My take is that you don't get rewarded for it directly, but rather once you're finished, you can move on to the next thing without having to constantly firefight and squash bugs on the previous project. So ideally you can get a lot more done in a given half than someone who pushed out a pile of buggy spaghetti that they're now constantly having to deal with. On the other hand, if you can ship something a few weeks fa…

I agree with you.

Re: Do big companies like Google and Amazon fire developers for buggy releases?

#19

Post by msiyer » Sat, May 25, 2019, 2:24 PM UTC

sys_64738 wrote: post #17

msiyer wrote: post #12

Earlier quoted context omitted.

This is so funny. I once worked on a year long project. The aim was to reduce customer escalations for my client. Changed the architecture of an extremely complex software making sure no regressions occurred and all new features worked flawlessly. The software was released and no customer escalations. I was happy, but found no one else cared. A few months later, during a meeting with the client manager, I brought thi…

This is similar to all the fixed issues we solved for Y2K. When nothing happened then rather than saying 'well done to all who fixed the issues' it became 'that was a waste of time and overblown as nothing happened.' Sometimes I think they wished a plane fell from the sky due to a software bug so they could slam software developers for incompetence. There's no middle ground :(

Planes fell and software was blamed. By extension, the developers were blamed too.

Re: Do big companies like Google and Amazon fire developers for buggy releases?

#20

Post by tehlike » Sat, May 25, 2019, 2:56 PM UTC

Yoric wrote: post #2

Not going to answer for Google, but my company, which I assume has a philosophy pretty similar to Google, will typically not fire developers for buggy releases, but not for the crappy reasons mentioned in the OP. The general idea is that: - everybody writes buggy code, once in a while – living under the pressure of being fired for an error that anybody could have made is not conductive to innovation and experiments;…

This.

We accepted that bugs are fact of life, and instead of burdening the engineers with fear, we reduce the scope of damage through data: we use canaries, experiments and monitoring to limit the damage and react fast.

Going against human nature as a result just slow things down. The other benefits to above setup is engineers are less afraid to make mistakes and are willing to try ideas more since it is now very cheap to try them

There are times when you need to be more thorough such as medical devices, space programs, and so on as they might risk human life, but for the most part you will go really far with above

Disclaimer: current google employee