Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The development pipeline is a production system

sundry.jerryorr.com

Post reply on HN
Print view View original

41–50 of 98 posts

Re: The development pipeline is a production system

#41

Post by Leherenn »

rcbdev wrote: post #39
malikolivier wrote: post #13

Once you enumerate what your development pipeline actually depends on, you will see it rely on a lot of stuff. A typical CI run pulls from npm, PyPI, Docker Hub, a distro mirror and a handful of third-party, none under your control, none with an SLA to you. Third-party outages, yanked versions, compromised packages are the things that can waste your time. Some packages may also drop support for whatever distro you're…

The fix is easy and ubiquitous. You host your own artifact repository and docker registry. This also makes sure your supply chain remains easily auditable.

That works for artifacts, but there's no easy solutions for all the security tools that relies on pulling up to date external databases. Databases that are out of your control and often down in my experience.

Re: The development pipeline is a production system

#42

Post by ownagefool »

tetha wrote: post #21

This is one of the unintuitive parts when you get into operations: If you go to lower layers in the stack, production expands towards dev: To the product developers and operators, customer-facing systems are production. To us in infra-operations, dev and testing are actually production as well. Maybe with a lower SLA and easier maintenance scheduling, but if we fry dev or testing, a hundred developers can't work and…

This is correct, but there's additional nuance here, which is the security of your development pipeline plays a part in the production system too.

Way back when, you'd see a lot of people stuff their jenkins in their dev account. There's no way that the system that builds, publishes and deploys your application should be treated as anything but as sensitive as the production system itself.

( To be fair there are mitigations such as reproducible builds, but you're now doing a bunch of engineering to tie in your deploy system. )

Re: The development pipeline is a production system

#43

Post by nonameiguess »

dsjoerg wrote: post #11

There's more to making an argument than stating your conclusion forcefully and repeatedly. EDIT. In particular the author hasn't acknowledged the prioritization problem, hasn't explained what factors might go into the necessary cost benefit analysis. The author merely points out "developer productivity is important!" You might as well say "invest in your tools!"

Not seeing this get traction, but this is exactly what I thought while reading this. The problem with calling something "a" top priority is you can only have one top priority. That's how priority works. A customer facing outage is a higher priority than a company-wide developer tool outage, which is higher priority than a single developer's personal workstation failing, etc. etc. That doesn't mean the lower-priority things don't matter, but you can't make any and every failure of anything an all hands on deck situation. That's a certain path to burning out your entire company.

Re: The development pipeline is a production system

#44

Post by locknitpicker »

donatj wrote: post #22

> If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority. Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?…

> Genuine question, does anyone here ITT working in software still have dedicated QA?

A few years ago I worked at a company who had a team of QAs dedicated to do manual acceptance tests. Up until the day where a critical regression went unnoticed and it turned out QAs were going through the test suite without actually checking the behavior. Following that the company got rid of all QAs except the software engineer in testing who was responsible for the whole automated test infrastructure, and instead of QAs the team held test days to go through the same manual test suite while we progressively automated them away.

And the whole process improved all across the board.

Re: The development pipeline is a production system

#45

Post by hnlmorg »

This is the first thing you learn when you move into DevOps. And something management don’t appreciate when DevOps engineers ask for another environment just for themselves (ie somewhere they can dev infra without breaking stuff for developers who are currently building stuff in dev)

Re: The development pipeline is a production system

#46

Post by reactordev »

TeMPOraL wrote: post #32
reactordev wrote: post #3

So is the recruiting pipeline. Kill that and you kill the company. Agents and AI can only do so much, the rest is people power. Sadly too many companies have broken recruiting processes.

That sounds nice and pleasantly confirms our biases, but it's blatantly false. Existence of recruiters and recruitment companies as separate job entities is a proof positive that you do not, in fact, need a working recruitment pipeline. If anything, recruitment in our industry was always broken. And yet here we are.

You aren’t wrong. That’s why recruiters and placement agencies exist. The statement is still true though. It’s just mostly deferred now onto another entity.

Re: The development pipeline is a production system

#47

Post by firesteelrain »

hnlmorg wrote: post #45

This is the first thing you learn when you move into DevOps. And something management don’t appreciate when DevOps engineers ask for another environment just for themselves (ie somewhere they can dev infra without breaking stuff for developers who are currently building stuff in dev)

When we do updates, I mandate that our teams do blue/green deployments meaning they setup the exact VM side by side with the actual production data and test with the new version of software (eg GitLab) before doing the DNS flip. This is on prem. It’s an ephemeral sandbox.

Re: The development pipeline is a production system

#48

Post by firesteelrain »

nonameiguess wrote: post #43
dsjoerg wrote: post #11

There's more to making an argument than stating your conclusion forcefully and repeatedly. EDIT. In particular the author hasn't acknowledged the prioritization problem, hasn't explained what factors might go into the necessary cost benefit analysis. The author merely points out "developer productivity is important!" You might as well say "invest in your tools!"

Not seeing this get traction, but this is exactly what I thought while reading this. The problem with calling something "a" top priority is you can only have one top priority. That's how priority works. A customer facing outage is a higher priority than a company-wide developer tool outage, which is higher priority than a single developer's personal workstation failing, etc. etc. That doesn't mean the lower-priority…

You need more than one team working outages. It’s not one team doing it all.

Re: The development pipeline is a production system

#49

Post by hnlmorg »

firesteelrain wrote: post #47
hnlmorg wrote: post #45

This is the first thing you learn when you move into DevOps. And something management don’t appreciate when DevOps engineers ask for another environment just for themselves (ie somewhere they can dev infra without breaking stuff for developers who are currently building stuff in dev)

When we do updates, I mandate that our teams do blue/green deployments meaning they setup the exact VM side by side with the actual production data and test with the new version of software (eg GitLab) before doing the DNS flip. This is on prem. It’s an ephemeral sandbox.

I don’t think I’ve worked in an industry where it’s safe to use production data in a non-prod environment in probably 20 years.

I think for dev, it’s fine not to have blue/green. You want dev deployments to be fast. UAT / pre-production/ or whatever your final env before prod — and particularly if stakeholders are intended to test on that env — is a different matter.

Re: The development pipeline is a production system

#50

Post by prabhanjana_c »

Recently I heard a term "Software factory" - I am strongly rooting with this term, particularly while working with agents and AI. Heard from Vercel, Guillermo. "The (software) factory is the product. Your product is only as good as the agents you set up to autonomously maintain it." https://x.com/rauchg/status/2081123293340520642?s=20
Post reply on HN