Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The development pipeline is a production system

sundry.jerryorr.com

Post reply on HN
Print view View original

91–98 of 98 posts

Re: The development pipeline is a production system

#91

Post by Illniyar »

dylanowen wrote: post #84
Illniyar wrote: post #80

No it's not. If the development pipeline breaks on Saturday night there is absolutely no need to go fix it right away. You should have a method to deploy hotfixes to code that bypasses your development pipeline though, in case the development pipeline breaks while you need to fix production- because you must be able to fix PRODUCTION even if your dev pipeline is broken ( because again your dev pipeline, while importa…

I feel like your example exactly describes why the development pipeline is production. Just because the critical hours of the system are different doesn't mean it's any less important.

I have never worked in a company where it is okay for production to be down at any time. Critical hours or not.

And if production is down even if there is just one user, it is all hands on deck.

Re: The development pipeline is a production system

#92

Post by ctkhn »

donatj wrote: post #22

> If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority. Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?…

I'm in legacy/finance right now and we have a QA team that we share with maybe 2 or 3 other dev teams. I think our integration testing systems could be a lot more efficient but I think if we had a proper pipeline that truly guaranteed integration testing (right now it feels like a shrug) we would lose the QA team like you did.

Re: The development pipeline is a production system

#93

Post by ctkhn »

locknitpicker wrote: post #44
donatj wrote: post #22

> If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority. Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?…

> Genuine question, does anyone here ITT working in software still have dedicated QA? A few years ago I worked at a company who had a team of QAs dedicated to do manual acceptance tests. Up until the day where a critical regression went unnoticed and it turned out QAs were going through the test suite without actually checking the behavior. Following that the company got rid of all QAs except the software engineer in…

I think our QAs do pay attention a bit more than that, but some of the things they do flag as issues show (to me) that they don't pay a lot of attention until after we've merged and released or we'd get some of their bugs as requirements on tickets instead of as bugs to resolve between QA deploy and before we go to prod.

Re: The development pipeline is a production system

#94

Post by ctkhn »

maciejgryka wrote: post #54

I’d bet there are as many stories of businesses failing because of inability to ship quickly as there are about focusing too much on your tooling instead of delivering value to customers. Both failures are dangerous! And depending on your market, product, team etc. a different spot on that spectrum is appropriate. In some places shipping quickly is part of the value. In others, the product works and delivers value an…

I worked at a startup that was briefly a unicorn and ended up burning cash because they did exactly what you said of focusing a lot on tooling and dev environment. It was great but way over the top for what we needed. I learned a lot of good practices for myself but they've been limping along for a while and most of my former team was laid off or saw what was coming and left before it hit them.

Re: The development pipeline is a production system

#95

Post by rswail »

SR2Z wrote: post #72
rswail wrote: post #63

Earlier quoted context omitted.

Depends on the type of testing. If you're doing integration testing between systems, then it's not just a matter of spinning it up on a developer's local machine. You need a testing environment, especially if there are other physical elements that are being integrated (embedded and other equipment).

It's bizarre to me that we live in a world where whole machines and their networking stack can be spun up and virtualized and yet integration testing STILL actually has to go out and touch a real resource running on a different machine. Yeah, obviously this isn't gonna work for embedded development, but not that many people have that specific problem. Is it really so hard to spin up ephemeral testing clusters on a si…

When you are integrating to other people's stacks, then you need a test facility where people can bring systems together, sometimes there are also physical elements that are required.

I work in Automatic Fare Collection (think railway passenger gates, ticket vending, contactless readers, bus driver consoles and readers etc as well as the backends that deal with the fare calculation and payments).

There are usually multiple vendors involved, and even with the best API and interface documentation and individual system testing against the specifications, you still need to bring the entire environment, including the hardware, together.

These ITFs (Integration Test Facilities) tend to be in existence for the life of the systems, including after hardware refreshes etc.

Yes you can test performance with jmeter etc, but sometimes you need 20 people walking in a circle between an entrance gate and an exit gate.

Re: The development pipeline is a production system

#96

Post by newsomix9xl »

cyanregiment wrote: post #8
newsomix9xl wrote: post #4

Sounds like someone wants to go on-call. You too can enjoy being paged at 2am for a flaky disk, a cron job related traffic spike, and AWS outage, or any of the wonderful things pagers get paged on. And you can re-read your essay after being awakened repeatedly for false alerts, knowing that non-prod alerts are now first class citizens and you are their Shepherd. Tldr? Go ahead and page yourself for non-prod alerts. N…

2:34 AM zZzzZ... 2:35 AM ALERT #42 - CRITICAL !!! ALERT #42 - CRITICAL - Joan forgot feir password to npm Help feir!

I wouldn't begrudge an actual human having an issue at 2am. But a wave of alerts for servers that were retired but their alerting wasn't, or hair trigger thresholds that pop when a new crown job pushes the load up 1% over WARN...

The issue is "production is defined by what humans are expected to maintain all night long" and dev ain't prod broskies.

Re: The development pipeline is a production system

#97

Post by RossBencina »

solatic wrote: post #33
RossBencina wrote: post #29

Earlier quoted context omitted.

> to be in control of the dependencies Maybe I'm out of touch, but I think control over dependencies is underrated. It's not just about freezing and pinning. There should be better tooling to support the whole dependency evolution pipeline. For example updates should be reviewed/ingested/integrated/validated based on local policy not based on source release schedule.

This is why some projects will vendor their dependencies - changes within the dependency itself will also be reviewed, and it's easier to locally maintain patches to those dependencies until the patches can be upstreamed. The problem is, there's a difference between doing that for a handful of C or Go libraries, versus trying to vendor thousands of NPM libraries and all of their interdependencies. So it's very ecosys…

Agreed. Your self-managed vendoring example is quite near one end of the continuum. I'm suggesting that there might be space for better tooling from there all the way to the NPM end. For example, automated and/or collaborative triage and review of change sets so that you don't see the changes you don't care about, but can easily review and integrate critical fixes.

Re: The development pipeline is a production system

#98

Post by SR2Z »

rswail wrote: post #95
SR2Z wrote: post #72

Earlier quoted context omitted.

It's bizarre to me that we live in a world where whole machines and their networking stack can be spun up and virtualized and yet integration testing STILL actually has to go out and touch a real resource running on a different machine. Yeah, obviously this isn't gonna work for embedded development, but not that many people have that specific problem. Is it really so hard to spin up ephemeral testing clusters on a si…

When you are integrating to other people's stacks, then you need a test facility where people can bring systems together, sometimes there are also physical elements that are required. I work in Automatic Fare Collection (think railway passenger gates, ticket vending, contactless readers, bus driver consoles and readers etc as well as the backends that deal with the fare calculation and payments). There are usually mu…

Fair enough, but IMO even a system like this one should have good mock implementations of the system hardware, down to and including a simple simulator for the physical layout of people near the machine.

In past years I wouldn't say that, but these days it's so easy to get an LLM to write something like this plus a quick and dirty visualization.

I'm not saying this can be a replacement for a physical test setup, but it's so much easier to make changes when the tests can just give a clear cut "yes, this acts like you expect" or "no, this is broken."

Post reply on HN