The development pipeline is a production system • Hacker News Board

Re: The development pipeline is a production system

#71

Post by Maxion » Sat, Aug 01, 2026, 4:33 PM UTC

rcxdude wrote: post #37

esikich wrote: post #35

Earlier quoted context omitted.

I think the question there is how do you make test actually test? Allowing them to break their test env so they can actually fix something they broke, imo. It's enabling and it's educational. Too often I see it so abstracted away that devs don't even know what their environment is made out of, so how can you expect them to make performant decisions that align with the infra? Maybe the cost you pay is in the complexit…

IMO having a test environment is a bit of a failure: you should be able to spin up and stop a full test environment easily, ideally (and feasibly for most systems) on the developer's local machine.

When you have a lot of interdepended systems it is not always possible. In our case our test environment cannot be replicated locally on developer machines because it integrates to various 3rd parties to which we only have one set of test credentials.

Re: The development pipeline is a production system

#72

Post by SR2Z » Sat, Aug 01, 2026, 5:12 PM UTC

rswail wrote: post #63

rcxdude wrote: post #37

Earlier quoted context omitted.

IMO having a test environment is a bit of a failure: you should be able to spin up and stop a full test environment easily, ideally (and feasibly for most systems) on the developer's local machine.

Depends on the type of testing. If you're doing integration testing between systems, then it's not just a matter of spinning it up on a developer's local machine. You need a testing environment, especially if there are other physical elements that are being integrated (embedded and other equipment).

It's bizarre to me that we live in a world where whole machines and their networking stack can be spun up and virtualized and yet integration testing STILL actually has to go out and touch a real resource running on a different machine.

Yeah, obviously this isn't gonna work for embedded development, but not that many people have that specific problem. Is it really so hard to spin up ephemeral testing clusters on a single machine? Everywhere that I've seen it done, it's a horrible kludge of scripts and hacks that's almost as difficult to understand as the code under test.

Re: The development pipeline is a production system

#73

Post by SR2Z » Sat, Aug 01, 2026, 5:23 PM UTC

cyanregiment wrote: post #5

Being a production system doesn't make it a priority. > A team with a broken development pipeline can’t produce software, and must treat this as a production outage. While I sympathize, I can't go that far. An "outage" means user- or stakeholder-facing. To the extent that a developer is a stakeholder (surprisingly common) you might be right, and they would prioritize such processes. Increasingly though founders and e…

This is a great way to encourage your devs to jump ship as soon as anything even a little better comes up.

It's trendy to act like devs are fungible resources or that AI will replace them, but the truth is that hiring smart people, paying them lots of money, and then forcing them to spend half their time dealing with prod issues that they're not empowered to fix is a staggering waste of resources.

Since deployment or testing issues are mostly invisible to upper management (who never have to spend half of THEIR time fixing them) this doesn't come with any recognition or promotions, so the only thing left to absorb the impact is morale and goodwill towards the company.

Sadly, engineer-driven companies are increasingly rare.

Re: The development pipeline is a production system

#74

Post by tetha » Sat, Aug 01, 2026, 6:22 PM UTC

esikich wrote: post #35

tetha wrote: post #21

This is one of the unintuitive parts when you get into operations: If you go to lower layers in the stack, production expands towards dev: To the product developers and operators, customer-facing systems are production. To us in infra-operations, dev and testing are actually production as well. Maybe with a lower SLA and easier maintenance scheduling, but if we fry dev or testing, a hundred developers can't work and…

I think the question there is how do you make test actually test? Allowing them to break their test env so they can actually fix something they broke, imo. It's enabling and it's educational. Too often I see it so abstracted away that devs don't even know what their environment is made out of, so how can you expect them to make performant decisions that align with the infra? Maybe the cost you pay is in the complexit…

In our case, "testing" as an environment has the goal to be a fully integrated setup of the entire service landscape. Each team is responsible for the uptime of their service, just like in production.

The teams are running automated end-to-end tests against this environment. On top, pre-sales uses it to build demo and sample environments, sales shows demos on it, solution engineers use it to test/try out the important parts of projects.

That generates a good production-like experience for the team: You have customers using the system, and possibly screaming if you break it. And it has been pretty successful at catching problems missed or integration failures.

Re: The development pipeline is a production system

#75

Post by rcxdude » Sat, Aug 01, 2026, 6:49 PM UTC

Uvix wrote: post #62

rcxdude wrote: post #37

Earlier quoted context omitted.

IMO having a test environment is a bit of a failure: you should be able to spin up and stop a full test environment easily, ideally (and feasibly for most systems) on the developer's local machine.

We can do that on our developers' machines no problem, but there still needs to be a place for QA to do their testing.

Can QA not do the same thing?

Re: The development pipeline is a production system

#76

Post by alexzz11 » Sat, Aug 01, 2026, 6:50 PM UTC

The failure mode I hit today isn't CI running untrusted code — it's simpler and I suspect more common. The deploy target contained state that wasn't in the source tree.

Deploy was an rsync of a few directories to a server. That server directory also held .env.local, and a content/ dir that a scheduled job writes to directly. Neither is in git. Someone (me) added --delete so the destination would "match source", and rsync faithfully removed everything outside the source set: the env file, the five .bak copies sitting next to it, and a week of generated content.

Two accidents kept it from being worse. The rsync excluded the build output, so the already-running process kept serving; and because the app had read its secrets into memory at boot, checkout kept working off a file that no longer existed on disk. That bought a few hours to restore from git.

What I'd add to the article: treating the pipeline as production should include auditing the pipeline's write scope. Mine had unrestricted write+delete over a directory holding live credentials, and that was invisible until it mattered. Also — backups in the same directory as the original are not backups.

Re: The development pipeline is a production system

#77

Post by gjvc » Sat, Aug 01, 2026, 6:54 PM UTC

no shit. another insightless article.

Re: The development pipeline is a production system

#78

Post by gjvc » Sat, Aug 01, 2026, 6:55 PM UTC

throwatdem12311 wrote: post #69

donatj wrote: post #22

> If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority. Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?…

We have 2 QA and I’m grateful every day for them. The devs generate slop at light speed but nothing gets passed those guys, they’re fantastic.

"past"

Re: The development pipeline is a production system

#79

Post by rcxdude » Sat, Aug 01, 2026, 7:57 PM UTC

rswail wrote: post #63

rcxdude wrote: post #37

Earlier quoted context omitted.

IMO having a test environment is a bit of a failure: you should be able to spin up and stop a full test environment easily, ideally (and feasibly for most systems) on the developer's local machine.

Depends on the type of testing. If you're doing integration testing between systems, then it's not just a matter of spinning it up on a developer's local machine. You need a testing environment, especially if there are other physical elements that are being integrated (embedded and other equipment).

Sure, for integration with some specific hardware, there may be some limits (though I would ask if that could be simulated for most testing, or if multiple could be made available). But it's a goal that should be aimed for and is achievable for most software, if some effort is spent on it. I see a lot of places seem to cargo-cult a dev/staging/prod setup (each of which inevitably winds up slightly different and hard to reproduce for incidental reasons) and it seems like an approach with a lot of headaches that you probably don't want to use unless you're forced to.

Re: The development pipeline is a production system

#80

Post by Illniyar » Sat, Aug 01, 2026, 7:57 PM UTC

No it's not. If the development pipeline breaks on Saturday night there is absolutely no need to go fix it right away.

You should have a method to deploy hotfixes to code that bypasses your development pipeline though, in case the development pipeline breaks while you need to fix production- because you must be able to fix PRODUCTION even if your dev pipeline is broken ( because again your dev pipeline, while important, is not production)