Live data from Hacker News

The development pipeline is a production system

sundry.jerryorr.com

81–90 of 98 posts

Re: The development pipeline is a production system

#81
post #72
post #63

Earlier quoted context omitted.

Depends on the type of testing. If you're doing integration testing between systems, then it's not just a matter of spinning it up on a developer's local machine. You need a testing environment, especially if there are other physical elements that are being integrated (embedded and other equipment).

It's bizarre to me that we live in a world where whole machines and their networking stack can be spun up and virtualized and yet integration testing STILL actually has to go out and touch a real resource running on a different machine. Yeah, obviously this isn't gonna work for embedded development, but not that many people have that specific problem. Is it really so hard to spin up ephemeral testing clusters on a si…

Docker has made this much much easier than it used to be, and the ease of publishing images has led to many SaaS companies providing "dockerized" versions of their services.

I write a lot of Terraform to manage internal infrastructure with dependencies on other internal and external infrastructure that's outside of my control. I define a module parameter for each dependency (usually with a default value that points to the real service URL) and then create a second module that uses Docker to spin up and configure local instances of each dependency and overrides the corresponding parameter.

This makes local testing very straightforward: `terraform -chdir=tests apply`

Re: The development pipeline is a production system

#82

I guess the problem is visibility. In the same way that many companies don't take security seriously until it becomes a problem. If a problem never manifests, but was prevented, was it really a problem? Of course I know the importance of preventing problems, but how do you properly measure it? A data leak is immediately visible, but preventing them is less so.

> how do you properly measure it? > If a problem never manifests, but was prevented, was it really a problem?

For the scheduled half of it, i like inverting the alert. Instead of alerting when a job fails or wait for data leak, have the job check in on every successful run and alert when a check-in doesn't arrive. Dead man's switch. Ten minutes to set up, and it's the difference between "the cron is fine" and a statement that can actually turn out to be false.

Agree that it is difficult to implement for any process, but it is easier to create validation systems than "find the error" processes.

Re: The development pipeline is a production system

#84

No it's not. If the development pipeline breaks on Saturday night there is absolutely no need to go fix it right away. You should have a method to deploy hotfixes to code that bypasses your development pipeline though, in case the development pipeline breaks while you need to fix production- because you must be able to fix PRODUCTION even if your dev pipeline is broken ( because again your dev pipeline, while importa…

I feel like your example exactly describes why the development pipeline is production. Just because the critical hours of the system are different doesn't mean it's any less important.

Re: The development pipeline is a production system

#85
post #72
post #63

Earlier quoted context omitted.

Depends on the type of testing. If you're doing integration testing between systems, then it's not just a matter of spinning it up on a developer's local machine. You need a testing environment, especially if there are other physical elements that are being integrated (embedded and other equipment).

It's bizarre to me that we live in a world where whole machines and their networking stack can be spun up and virtualized and yet integration testing STILL actually has to go out and touch a real resource running on a different machine. Yeah, obviously this isn't gonna work for embedded development, but not that many people have that specific problem. Is it really so hard to spin up ephemeral testing clusters on a si…

At some point, it becomes hard to assert “we are NOT going to test the product the way a customer would actually use it”

Re: The development pipeline is a production system

#86
post #22

> If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority. Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?…

We have 2 QA and I’m grateful every day for them. The devs generate slop at light speed but nothing gets passed those guys, they’re fantastic.

Oh exactly the same here.

Small company in Germany, QA team with about 5 people. But not all the devs are bad, just some are too comfortable with AI and getting lazy.

Re: The development pipeline is a production system

#87
post #45

This is the first thing you learn when you move into DevOps. And something management don’t appreciate when DevOps engineers ask for another environment just for themselves (ie somewhere they can dev infra without breaking stuff for developers who are currently building stuff in dev)

Yeah at my place our DevOps team had been asking for resources on our CI/CD pipeline for a long time but wasn't prioritized. After awhile, after introducing something like teams wide "Retros', we noticed they didnt want the resources, we wanted them!

So we started being more vocal and management understood its for everyone and their wish got approved recently and its been a big change.

Re: The development pipeline is a production system

#88
post #66

Earlier quoted context omitted.

That’s not the same thing as what people generally mean when they say “production data”. And yes, your dev environments should mirror the production environments bar production data. That’s software development 101.

You are still mixing production and dev contexts. What is called software development environment is production for your software factory. It’s not development in terms of a sandbox for your deployment, Production environment. Within your software development environment (which should be treated as prod to maintain velocity), you have blue/green when doing deployments, established SLAs, backups and DR. This isn’t onl…

> You are still mixing production and dev contexts. What is called software development environment is production for your software factory. It’s not development in terms of a sandbox for your deployment, Production environment.

You’re the one drifting all over the place with the terminology.

I get what you’re trying to say. But you’re saying it badly.

You have your dev, test, UAT, production, etc environments. They’re the “software development” envs as you named them.

You then have your DevOps envs. You call them sandbox, but Personally I hate that term because the DevOps envs should be the same as the “software development” envs just not accessible to the software developers. A “pre-dev” env if you will. So it should be the same IaC and the same build pipelines.

Whereas a “sandbox” generally suggests that it’s “ClickOps” / manually provisioned envs with no rigour. A place to play and experiment before you’re ready to commit your infra to code. Which is not how the DevOps envs should exist. The DevOps pre-dev envs should still be a carbon copy of the “software developer” environments.

> ithin your software development environment (which should be treated as prod to maintain velocity), you have blue/green when doing deployments, established SLAs, backups and DR

No it shouldn’t. HA, DR, etc costs money. Blue/green is slow and you shouldn’t let DevOps be a blocker for development feedback loops. So you save that stuff for your pre-production envs, and keep your dev and test envs nimble and quick to deploy. The DR for your lowest environments should just be “destroy and recreate” because you’ll have pipelines to deploy that infra.

In short. You can still have the same approximate infra but you shouldn’t emulate production best practices precisely across all development environments.

If you find you cannot have a “production-like” dev but without the HA, DR, etc, then you have designed your pipelines wrong from the outset. Eg You’ve got “pets instead of cattle” (ie the lack of ephemeral infra), manually provisioned services, and other faux pas.

> This isn’t only for your (deployed) Production environment

It isn’t. But it’s also not for all environments indiscriminately either.

> It’s also not software development 101. This has roots in ITIL.

Of course it’s software development 101. If your developers don’t understand the basics of different software development environments then they’re either shit at their job or fresh out of uni. I don’t mean to be harsh here but I don’t know how else to emphasise that what you’re describing here is the absolute basics. 30 years ago, when I was starting out, I learned this before I learned about version control. That’s how basic a topic this is.

Re: The development pipeline is a production system

#90
post #88

Earlier quoted context omitted.

You are still mixing production and dev contexts. What is called software development environment is production for your software factory. It’s not development in terms of a sandbox for your deployment, Production environment. Within your software development environment (which should be treated as prod to maintain velocity), you have blue/green when doing deployments, established SLAs, backups and DR. This isn’t onl…

> You are still mixing production and dev contexts. What is called software development environment is production for your software factory. It’s not development in terms of a sandbox for your deployment, Production environment. You’re the one drifting all over the place with the terminology. I get what you’re trying to say. But you’re saying it badly. You have your dev, test, UAT, production, etc environments. They’…

[dead]
Post reply on HN