Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The development pipeline is a production system

sundry.jerryorr.com

Post reply on HN
Print view View original

61–70 of 98 posts

Re: The development pipeline is a production system

#61

Post by lifeisstillgood »

I think I disagree

There is a production eco-system - let’s call it The Hotel. It runs, people work in it and customers pay and it makes money.

But it constantly needs changing. New rooms need to be built, the kitchen rearranged as the chef changes his mind.

So they hire an architect and building firm. Who plan the chnages and then this awesome automated robot comes and builds the extra room.

The architect firm owns and is responsible for the robot and its changes.

Not being able to make a change really really matters, but I don’t call that production because the hotel is still able to operate

Most of the misunderstanding between business and tech is a misunderstanding of are you working on the architect firm or the hotel

Re: The development pipeline is a production system

#62

Post by Uvix »

rcxdude wrote: post #37
esikich wrote: post #35

Earlier quoted context omitted.

I think the question there is how do you make test actually test? Allowing them to break their test env so they can actually fix something they broke, imo. It's enabling and it's educational. Too often I see it so abstracted away that devs don't even know what their environment is made out of, so how can you expect them to make performant decisions that align with the infra? Maybe the cost you pay is in the complexit…

IMO having a test environment is a bit of a failure: you should be able to spin up and stop a full test environment easily, ideally (and feasibly for most systems) on the developer's local machine.

We can do that on our developers' machines no problem, but there still needs to be a place for QA to do their testing.

Re: The development pipeline is a production system

#63

Post by rswail »

rcxdude wrote: post #37
esikich wrote: post #35

Earlier quoted context omitted.

I think the question there is how do you make test actually test? Allowing them to break their test env so they can actually fix something they broke, imo. It's enabling and it's educational. Too often I see it so abstracted away that devs don't even know what their environment is made out of, so how can you expect them to make performant decisions that align with the infra? Maybe the cost you pay is in the complexit…

IMO having a test environment is a bit of a failure: you should be able to spin up and stop a full test environment easily, ideally (and feasibly for most systems) on the developer's local machine.

Depends on the type of testing. If you're doing integration testing between systems, then it's not just a matter of spinning it up on a developer's local machine.

You need a testing environment, especially if there are other physical elements that are being integrated (embedded and other equipment).

Re: The development pipeline is a production system

#64

Post by 28304283409234 »

tetha wrote: post #21

This is one of the unintuitive parts when you get into operations: If you go to lower layers in the stack, production expands towards dev: To the product developers and operators, customer-facing systems are production. To us in infra-operations, dev and testing are actually production as well. Maybe with a lower SLA and easier maintenance scheduling, but if we fry dev or testing, a hundred developers can't work and…

For 26 years I have tried and failed to explain this to my different managers at so many different jobs. I gave up.

Re: The development pipeline is a production system

#65

Post by 28304283409234 »

wxw wrote: post #7

In my experience, most large companies do treat not being able to ship code (i.e. deploy to prod) as outages. Being on-call in a CI/CD infra team is fairly common. Agreed that many parts of the development pipeline can be hit-or-miss. At scale, it's great to have a dedicated "Developer Experience/Tools" org, though I've seen mixed results even with one.

It's bigger than that even. Ensuring that the code you are shipping has not been tampered with, for example.

Re: The development pipeline is a production system

#66

Post by hnlmorg »

firesteelrain wrote: post #58
hnlmorg wrote: post #49

Earlier quoted context omitted.

I don’t think I’ve worked in an industry where it’s safe to use production data in a non-prod environment in probably 20 years. I think for dev, it’s fine not to have blue/green. You want dev deployments to be fast. UAT / pre-production/ or whatever your final env before prod — and particularly if stakeholders are intended to test on that env — is a different matter.

When I say Production, I mean Production Software Development data like Coverity or Fortify databases. GitLab source code and pipelines. The premise is that Software Development is akin to Production which it should be.

That’s not the same thing as what people generally mean when they say “production data”.

And yes, your dev environments should mirror the production environments bar production data. That’s software development 101.

Re: The development pipeline is a production system

#67

Post by firesteelrain »

hnlmorg wrote: post #66
firesteelrain wrote: post #58

Earlier quoted context omitted.

When I say Production, I mean Production Software Development data like Coverity or Fortify databases. GitLab source code and pipelines. The premise is that Software Development is akin to Production which it should be.

That’s not the same thing as what people generally mean when they say “production data”. And yes, your dev environments should mirror the production environments bar production data. That’s software development 101.

You are still mixing production and dev contexts. What is called software development environment is production for your software factory. It’s not development in terms of a sandbox for your deployment, Production environment.

Within your software development environment (which should be treated as prod to maintain velocity), you have blue/green when doing deployments, established SLAs, backups and DR.

This isn’t only for your (deployed) Production environment.

It’s also not software development 101. This has roots in ITIL.

Re: The development pipeline is a production system

#69

Post by throwatdem12311 »

donatj wrote: post #22

> If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority. Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?…

We have 2 QA and I’m grateful every day for them. The devs generate slop at light speed but nothing gets passed those guys, they’re fantastic.

Re: The development pipeline is a production system

#70

Post by jcims »

tetha wrote: post #21

This is one of the unintuitive parts when you get into operations: If you go to lower layers in the stack, production expands towards dev: To the product developers and operators, customer-facing systems are production. To us in infra-operations, dev and testing are actually production as well. Maybe with a lower SLA and easier maintenance scheduling, but if we fry dev or testing, a hundred developers can't work and…

Agreed. I've started calling our non-production infrastructure environments 'engineering' rather than dev, and nobody's invited. We'll pull in platform teams that are exposed to this shame environmental offset issue, but no customers.
Post reply on HN