I wonder if the place is covered by something like ISO 9000?
It Takes 6 Days to Change 1 Line of Code
21–30 of 237 posts
Re: It Takes 6 Days to Change 1 Line of Code
#22Honestly, for code that controls millions of dollars worth of business and could seriously screw over a lot of people if a subtle flaw was introduced, I don't see a problem with doing it this way. Might be overkill for one line of code, but where do you set the threshold? If you say any change longer than four lines is subject to this process, then people will work on trickling in new stuff in increments of four line…
Reliability and quality are obviously important, but you have to question whether all these steps are actually helping you to achieve that. All of these items in the story add overhead and reduce agility, but potentially do very little to improve quality: - Mandatory paper trail via fields on the issue tracker - The need for documented internal sign offs - Standards and policies with no clear reason for the existence…
And more importantly: a complete lack of trust within the organisation.
Re: It Takes 6 Days to Change 1 Line of Code
#23Heh, not the same corporate snafu, but my most expensive work was done to change 1 char.
Spent about 3 weeks with a new codebase to find that one number had to be increased. Funny thing is, there was a comment there and somebody had increased it before when they had a similar problem.
Re: It Takes 6 Days to Change 1 Line of Code
#24People don't realize processes, reviews, standards, and audits are there because of painful (costly) incidents and breakdowns in the past. Changing one line of code in the production app is fast and easy. Crashing the system or screwing up the app logic can cause huge disruption to the operation.
After every single crisis, managers got together and vowed to never let this happen again, and thus processes and reviews are born. After couple years, things will get so stifled to the point of what OP was experiencing.
Re: It Takes 6 Days to Change 1 Line of Code
#25I understand the advice to quit because of the agonizing process involved, but worse than that for me is the idea that this parameter is hard coded, and even with this change in place that limitation still remains. NO WONDER the change process is so painful, the software design is clearly brain dead. I bet all kinds of unexpected shit happens when changes are rolled out. This is a code smell like a rotting corpse.
[deleted]
Re: It Takes 6 Days to Change 1 Line of Code
#26Honestly, for code that controls millions of dollars worth of business and could seriously screw over a lot of people if a subtle flaw was introduced, I don't see a problem with doing it this way. Might be overkill for one line of code, but where do you set the threshold? If you say any change longer than four lines is subject to this process, then people will work on trickling in new stuff in increments of four line…
Reliability and quality are obviously important, but you have to question whether all these steps are actually helping you to achieve that. All of these items in the story add overhead and reduce agility, but potentially do very little to improve quality: - Mandatory paper trail via fields on the issue tracker - The need for documented internal sign offs - Standards and policies with no clear reason for the existence…
Exactly. The point bengl3rt was making, I assume, is that cavalier avoidance of process is a bad thing because it allows mistakes to happen that would be caught. That much is true.
But the assumption that's wrong is that all processes avoid mistakes. Clearly they don't.
And some of the examples here are just plain cargo cult misapplications of good ideas. The point behind code review is to catch design flaws in new code. You never want to demand refactoring of existing code in order to patch features, that hurts, it doesn't help.
Re: It Takes 6 Days to Change 1 Line of Code
#27Honestly, for code that controls millions of dollars worth of business and could seriously screw over a lot of people if a subtle flaw was introduced, I don't see a problem with doing it this way. Might be overkill for one line of code, but where do you set the threshold? If you say any change longer than four lines is subject to this process, then people will work on trickling in new stuff in increments of four line…
I agree that it's worth being careful on mission-critical code, but this isn't being careful, it's just box-ticking. Ed said he'd tested the actual change, but most of the time is taken up by shifting it out to some parameters file, lengthening its name and waiting for permissions.
Re: It Takes 6 Days to Change 1 Line of Code
#28Honestly, for code that controls millions of dollars worth of business and could seriously screw over a lot of people if a subtle flaw was introduced, I don't see a problem with doing it this way. Might be overkill for one line of code, but where do you set the threshold? If you say any change longer than four lines is subject to this process, then people will work on trickling in new stuff in increments of four line…
Reliability and quality are obviously important, but you have to question whether all these steps are actually helping you to achieve that. All of these items in the story add overhead and reduce agility, but potentially do very little to improve quality: - Mandatory paper trail via fields on the issue tracker - The need for documented internal sign offs - Standards and policies with no clear reason for the existence…
A few notes on those points:
1) The fields are probably mandatory because so many times routine entries were not filled correctly, and even MORE overhead was required going back and forth between teams to find out all of the details. If there is a way to avoid this overhead, again, how do you stop people abusing it?
2 & 3) Yeah.. bad overhead.
4) A sanity check is ALWAYS a good idea, whether its a fresh intern or the most seasoned programming in the company making a change.
5) They security processes might seem excessive but once again are probably in place due to issues in the past that have required the review/signoff be there to stop major disruptions getting through.
6) As someone who worked in QA for a while, there is no way in a hell someone can say "push this change through" without a strong questioning of why. QA is there (in most cases) to be that last line of defence. Often letting people know just how many people are really going to be affected by this change.
Being on a testing team often gives you a broader oversight of how all of the components of the code work together, while people working on one project can sometimes get tunnel vision on getting their thing out the door.
Edit: Just like to point out, I'm agreeing with you
Re: It Takes 6 Days to Change 1 Line of Code
#29Honestly, for code that controls millions of dollars worth of business and could seriously screw over a lot of people if a subtle flaw was introduced, I don't see a problem with doing it this way. Might be overkill for one line of code, but where do you set the threshold? If you say any change longer than four lines is subject to this process, then people will work on trickling in new stuff in increments of four line…
"Changing one line of code took 6 days!" is a pretty common WTF story. It comes up every so often, and we all get a chuckle at the ponderous rate of change in large organizations.
However, "Changing one line of code broke everything for 6 days!" is also a common WTF story. As a rule of thumb, you can't avoid both of them, because avoiding one entails the other. I wouldn't fault a company that chooses the first story over the second.
The problem is that things that scale up don't always scale down. The processes implied aren't that bad, for handling large commits to a large process. They're obviously overkill for small changes. But just because the processes are a negative effect on this change in particular does not mean that the processes are a net negative overall. More flexible processes could also end up being more trouble than they're worth, due to allowing (encouraging) sloppy work.
Re: It Takes 6 Days to Change 1 Line of Code
#30Honestly, for code that controls millions of dollars worth of business and could seriously screw over a lot of people if a subtle flaw was introduced, I don't see a problem with doing it this way. Might be overkill for one line of code, but where do you set the threshold? If you say any change longer than four lines is subject to this process, then people will work on trickling in new stuff in increments of four line…
Did you miss this part:
> It [sic] we don't do this right away, we'll have to have a layoff.
As a company, it's great to have a process for incremental changes, but you also need to have a process for critical hot-fixes.
You're a phone company, and adding a new feature should take months. I get that.
You're a phone company, and because of an unforeseen Cinderella problem, 10% of your customers can't make phone calls. At all.
It is unacceptable to have a one-size-fits-all process, no matter what your business is.