Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The development pipeline is a production system

sundry.jerryorr.com

Post reply on HN
Print view View original

21–30 of 98 posts

Re: The development pipeline is a production system

#21

Post by tetha »

This is one of the unintuitive parts when you get into operations: If you go to lower layers in the stack, production expands towards dev:

To the product developers and operators, customer-facing systems are production.

To us in infra-operations, dev and testing are actually production as well. Maybe with a lower SLA and easier maintenance scheduling, but if we fry dev or testing, a hundred developers can't work and start screaming.

Within the infra-ops team, our config management tests and the deployment pipelines are production. If those don't work, infra operators cannot test or roll out changes to the infrastructure.

This was recently discovered by a dev team providing a cross-cutting service: Their testing environment can halt work for a lot of other teams, so they have to be really careful with their testing environment.

Re: The development pipeline is a production system

#22

Post by donatj »

> If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority.

Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?

FWW I think a good QA person is worth their weight in gold and this has been a horrible mistake. I'm just curious if there are any left.

Re: The development pipeline is a production system

#23

Post by creesch »

donatj wrote: post #22

> If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority. Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?…

Different country probably, here QA is still often part of the team. More often than not as part of a team with a title like "test automation engineer".

Re: The development pipeline is a production system

#24

Post by [deleted] »

donatj wrote: post #22

> If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority. Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?…

[deleted]

Re: The development pipeline is a production system

#25

Post by creesch »

malikolivier wrote: post #13

Once you enumerate what your development pipeline actually depends on, you will see it rely on a lot of stuff. A typical CI run pulls from npm, PyPI, Docker Hub, a distro mirror and a handful of third-party, none under your control, none with an SLA to you. Third-party outages, yanked versions, compromised packages are the things that can waste your time. Some packages may also drop support for whatever distro you're…

> CI run pulls from npm, PyPI, Docker Hub, a distro mirror and a handful of third-party, none under your control, none with an SLA to you.

To be honest, they shouldn't do so directly. For the obvious reason you stated, they don't have a SLA to you or your company. But also because it is a shitty thing to do as a company. Companies should have their own proxies and mirrors for any repository they pull from. Both to be in control of the dependencies you as a company rely on and to not unnecessarily put load on free third party repositories.

Had not heard of your solution. More often than not I come across Harbor, Nexus or things like GCPs internal stuff.

Re: The development pipeline is a production system

#26

Post by chii »

donatj wrote: post #22

> If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority. Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?…

The problem with QA is that from the POV of management, all they do is find issues and delay releases. If the QA team did their jobs well, then once all of the reported bugs are found and fixed (passes QA), the software runs fine and there are no future problems. This sounds good in theory, but a good release doesn't get the credit it deserves - it's "business as usual".

A buggy release is fine, if it came out faster, and costed less. The bugs in the release can be fixed in the future, and the "heroic" fixes are "creditworthy".

Therefore, over time, the QA team is politically relegated as a "useless" team in the department, and thus given how politics play out, they get sidelined until the entire team is gone.

Re: The development pipeline is a production system

#28

Post by hiAndrewQuinn »

donatj wrote: post #22

> If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority. Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?…

Here in Finland my company has a dedicated testing team that's almost at large at the actual engineering staff. We sell a combined hardware-software stack, though, and it's basically a contractual requirement in our industry. They are very helpful to be on good terms with when working on a given project, they often know ways to work with the gestalt I never would have guessed in a million years.

Re: The development pipeline is a production system

#29

Post by RossBencina »

creesch wrote: post #25
malikolivier wrote: post #13

Once you enumerate what your development pipeline actually depends on, you will see it rely on a lot of stuff. A typical CI run pulls from npm, PyPI, Docker Hub, a distro mirror and a handful of third-party, none under your control, none with an SLA to you. Third-party outages, yanked versions, compromised packages are the things that can waste your time. Some packages may also drop support for whatever distro you're…

> CI run pulls from npm, PyPI, Docker Hub, a distro mirror and a handful of third-party, none under your control, none with an SLA to you. To be honest, they shouldn't do so directly. For the obvious reason you stated, they don't have a SLA to you or your company. But also because it is a shitty thing to do as a company. Companies should have their own proxies and mirrors for any repository they pull from. Both to be…

> to be in control of the dependencies

Maybe I'm out of touch, but I think control over dependencies is underrated. It's not just about freezing and pinning. There should be better tooling to support the whole dependency evolution pipeline. For example updates should be reviewed/ingested/integrated/validated based on local policy not based on source release schedule.

Re: The development pipeline is a production system

#30

Post by SideburnsOfDoom »

wxw wrote: post #7

In my experience, most large companies do treat not being able to ship code (i.e. deploy to prod) as outages. Being on-call in a CI/CD infra team is fairly common. Agreed that many parts of the development pipeline can be hit-or-miss. At scale, it's great to have a dedicated "Developer Experience/Tools" org, though I've seen mixed results even with one.

> In my experience, most large companies do treat not being able to ship code (i.e. deploy to prod) as outages.

In my experience, we treat "not being able to deploy to prod" as an warning light, because our ability to fix an actual issue or outage in prod is impacted.

i.e. if there is a prod issue, then we can often ether fix it in code, or roll back to an earlier release. But we need the deploy pipeline to be working in order to do that. The longer a pipeline outage persists over time and across multiple services, the more likely it is that there will be some impact.

It's like driving without a seatbelt - you're fine so long as nothing else goes wrong, so it's not recommended as a state to stay in.

Post reply on HN