Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The development pipeline is a production system

sundry.jerryorr.com

Post reply on HN
Print view View original

31–40 of 98 posts

Re: The development pipeline is a production system

#31

Post by pydry »

donatj wrote: post #22

> If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority. Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?…

My experience is 50/50 - half have dropped them, half still have them.

I also don't find most QA to be worthwhile compared to a good, quality focused dev.

There are several cultural issues with how QA are treated (low pay) and how they think they're expected to behave (e.g. hands off the code, hands off types, hands off unit tests) which make them less useful than a dev who is really good at engineering for reliability.

Where we had good engineering practices and that type of "black box" QA I noticed that product tended to find the majority of bugs, followed by devs, followed by customers, followed by QA.

Only when engineering practices sucked and they were used as a crutch by devs did they really catch a lot of bugs.

Of course, in these days of vibe coded madness they're probably getting more useful, not less.

Re: The development pipeline is a production system

#32

Post by TeMPOraL »

reactordev wrote: post #3

So is the recruiting pipeline. Kill that and you kill the company. Agents and AI can only do so much, the rest is people power. Sadly too many companies have broken recruiting processes.

That sounds nice and pleasantly confirms our biases, but it's blatantly false.

Existence of recruiters and recruitment companies as separate job entities is a proof positive that you do not, in fact, need a working recruitment pipeline.

If anything, recruitment in our industry was always broken. And yet here we are.

Re: The development pipeline is a production system

#33

Post by solatic »

RossBencina wrote: post #29
creesch wrote: post #25

Earlier quoted context omitted.

> CI run pulls from npm, PyPI, Docker Hub, a distro mirror and a handful of third-party, none under your control, none with an SLA to you. To be honest, they shouldn't do so directly. For the obvious reason you stated, they don't have a SLA to you or your company. But also because it is a shitty thing to do as a company. Companies should have their own proxies and mirrors for any repository they pull from. Both to be…

> to be in control of the dependencies Maybe I'm out of touch, but I think control over dependencies is underrated. It's not just about freezing and pinning. There should be better tooling to support the whole dependency evolution pipeline. For example updates should be reviewed/ingested/integrated/validated based on local policy not based on source release schedule.

This is why some projects will vendor their dependencies - changes within the dependency itself will also be reviewed, and it's easier to locally maintain patches to those dependencies until the patches can be upstreamed.

The problem is, there's a difference between doing that for a handful of C or Go libraries, versus trying to vendor thousands of NPM libraries and all of their interdependencies. So it's very ecosystem dependent.

Re: The development pipeline is a production system

#34

Post by lelanthran »

donatj wrote: post #22

> If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority. Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?…

Yes. We call them BAs and they have deep product knowledge, so they also get used for training, escalated support requests, estimations on effort required in new integrations, etc.

Re: The development pipeline is a production system

#35

Post by esikich »

tetha wrote: post #21

This is one of the unintuitive parts when you get into operations: If you go to lower layers in the stack, production expands towards dev: To the product developers and operators, customer-facing systems are production. To us in infra-operations, dev and testing are actually production as well. Maybe with a lower SLA and easier maintenance scheduling, but if we fry dev or testing, a hundred developers can't work and…

I think the question there is how do you make test actually test? Allowing them to break their test env so they can actually fix something they broke, imo. It's enabling and it's educational. Too often I see it so abstracted away that devs don't even know what their environment is made out of, so how can you expect them to make performant decisions that align with the infra?

Maybe the cost you pay is in the complexity of security. And then where does that land? Is it ops or security when some ephemeral rotating key or token that's controlled by some test service account doesn't work?

Re: The development pipeline is a production system

#36

Post by brabel »

donatj wrote: post #22

> If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority. Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?…

We have a QA team but it’s small in comparison to the dev team. We didn’t have one before as we thought automated tests and “good developers” who take testing seriously was enough. Unfortunately, even with extreme automation and huge test suites, our bug count per release was unacceptable. Having a QA team was a big help in fixing that. We have two weeks of testing before a release goes out. But we sell a security product so for us bugs must be far between. If you just deploy web apps and you can update every hour, I guess you don’t need it as much.

Re: The development pipeline is a production system

#37

Post by rcxdude »

esikich wrote: post #35
tetha wrote: post #21

This is one of the unintuitive parts when you get into operations: If you go to lower layers in the stack, production expands towards dev: To the product developers and operators, customer-facing systems are production. To us in infra-operations, dev and testing are actually production as well. Maybe with a lower SLA and easier maintenance scheduling, but if we fry dev or testing, a hundred developers can't work and…

I think the question there is how do you make test actually test? Allowing them to break their test env so they can actually fix something they broke, imo. It's enabling and it's educational. Too often I see it so abstracted away that devs don't even know what their environment is made out of, so how can you expect them to make performant decisions that align with the infra? Maybe the cost you pay is in the complexit…

IMO having a test environment is a bit of a failure: you should be able to spin up and stop a full test environment easily, ideally (and feasibly for most systems) on the developer's local machine.

Re: The development pipeline is a production system

#38

Post by philipp-gayret »

cadamsdotcom wrote: post #19

Worth mentioning CI is not part of production outages! You should never fix an incident by deploying another fix as it risks making things worse especially while everyone is panicking. Better to roll back - and spread non-backwards compatible work across multiple deploys that each are backwards compatible - always giving yourself a known good state to roll back to. Rollback should not done be via your CI either! Set…

Having spent too many years in CI/CD I'd agree but there is an even better solution; decouple deployments from releases. Make it possible to deploy without releasing, and release without deploying. (An implementation of this could be, for example: feature flags.)

Re: The development pipeline is a production system

#39

Post by rcbdev »

malikolivier wrote: post #13

Once you enumerate what your development pipeline actually depends on, you will see it rely on a lot of stuff. A typical CI run pulls from npm, PyPI, Docker Hub, a distro mirror and a handful of third-party, none under your control, none with an SLA to you. Third-party outages, yanked versions, compromised packages are the things that can waste your time. Some packages may also drop support for whatever distro you're…

The fix is easy and ubiquitous. You host your own artifact repository and docker registry. This also makes sure your supply chain remains easily auditable.

Re: The development pipeline is a production system

#40

Post by maccard »

donatj wrote: post #22

> If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority. Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?…

Yes. I work in games and we heavily rely on manual QA. Too heavily, really.
Post reply on HN