That's the new Microsoft - less thinking about backwards compatibility and not breaking things, but faster development, reduced management overhead. Faster time to market HAS to come from somewhere.
Microsoft has been making quality assurance mistakes lately
41–50 of 62 posts
Re: Microsoft has been making quality assurance mistakes lately
#42This is exactly what I thought would happen when they gutted their QA staff and reassigned the surviving testers as developers. Your employees act in accordance with their incentives, and the incentives aren't there for QA. Microsoft made it clear that they only value feature development and shipping, and their employees now associate "doing QA" with "being fired en masse". To corrupt a phrase, they've crapped their…
This is something that deeply bothers me in the software industry, the segregation between Dev and QA, and the low value placed in QA. "Oh whatever, QA are low-paid techs that just run scripts!" No. QA is vital to the functionality of the product, and a proper QA person should be just as capable a programmer as a developer in order to 1) write and maintain a test suite and 2) understand the tested program and how best to test it.
My previous company in Europe understood this, and that seemed the norm in the surrounding industry. My current in America doesn't, and that seems the norm in the Silicon Valley.
Re: Microsoft has been making quality assurance mistakes lately
#43The Windows 10 Technical Preview is available to author as part of Microsoft's quality assurance program. Technical Previews are alpha software [1] by another name and Release Candidates are beta software in Microsoft speak. [1] see: http://www.hanselman.com/blog/IntroducingASPNETVNext.aspx and http://blogs.msdn.com/b/cesardelatorre/archive/2014/05/12/th... for examples
Are you making the point that "it's beta/alpha, deal with it"? Because the author adresses exactly that, and the headline incident is only a one (egregious) example of a series of recent problems out of Microsoft.
Re: Microsoft has been making quality assurance mistakes lately
#44Earlier quoted context omitted.
Seriously! You never hear about a car would accidentally accelerate when you try to brake, or that the tires would spontaneously blow out, or that they would just plain roll over. I mean, the phrase "automobile recall" is practically an oxymoron! [1] - https://en.wikipedia.org/wiki/Toyota_Prius_(XW30)#Recall [2] - http://www.forbes.com/2000/08/18/mu3.html [3] - http://www.cbsnews.com/news/toyota-will-recall-lexus-suv…
The fact that "automobile recall" is something that exists and is expected when defects occur, but "software recall" largely does not exist, was his point.
His point, as I read it, was that the software industry does not operate to the same quality standards as the auto industry. Which may have some validity, but is not as obvious or clear cut as he was implying.
"Software recall" does exist, they just call it "applying updates" instead.
Re: Microsoft has been making quality assurance mistakes lately
#45> Microsoft has been making quality assurance mistakes lately
Well, that's news to me!
I've always thought that everything that came out of MS is a quality assurance mistake. Heck, even MS itself is a quality assurance mistake.
Re: Microsoft has been making quality assurance mistakes lately
#46This is exactly what I thought would happen when they gutted their QA staff and reassigned the surviving testers as developers. Your employees act in accordance with their incentives, and the incentives aren't there for QA. Microsoft made it clear that they only value feature development and shipping, and their employees now associate "doing QA" with "being fired en masse". To corrupt a phrase, they've crapped their…
This is something that deeply bothers me in the software industry, the segregation between Dev and QA, and the low value placed in QA. "Oh whatever, QA are low-paid techs that just run scripts!" No. QA is vital to the functionality of the product, and a proper QA person should be just as capable a programmer as a developer in order to 1) write and maintain a test suite and 2) understand the tested program and how bes…
I'm a Quality Engineer, and have worked for a number of large and small companies in the Valley after moving into quality from software engineering 13 years ago. I've seen a lot of teams.
Problem is it's a feedback loop. A lot of companies maintain (or did maintain) armies of unskilled people who basically were script executors, with maybe a bit of barely-trained skill in finding equivalence classes and boundary testing so that basic test plans could be written.
I wouldn't even call this QA, even if it was misidentified as such--it's Quality Control to catch problems after the fact, with no front end assurance whatsoever. It's honestly really only good for pissing off people when you stop the shipment at the last minute. And that lack of value has been broadly recognized. When you combine the high cost associated with the brute-force documentation associated with verbose test scripts, result templates, and other prehistoric process artifacts aimed at this level of tester (and which are usually simply unnecessary and actually detrimental to agility and pace) it's no wonder they're being cut.
Problem is as classic brute force QA teams get gutted, the market has been flooded with these testers, and most of them kind of suck. Hiring a decent quality professional has gotten damned difficult. And when everyone you meet with "QA" in their resume is a non-technical, uninfluential yo-yo, you get a pretty bad impression of the discipline.
In the meantime, everyone as skilled, interested, or immersed as you describe has been moved into software engineering, because QA was treated as an "entry-level" position and nobody wants to be a non-technical, uninfluential yo-yo. The rise of the Software Engineer in Test is as much a collective gambit to keep skilled people in quality as it is identifying a new skill-set. And for people in my line of work, identifying yourself as a SET has about a 30% pay differential over identifying yourself as a QA Engineer, so there's plenty of incentive to move in that direction.
But even then, the rise of the SET reflects an over-reliance on automation, and an under-reliance on process-based quality improvement and the type of fuzzy testing and intelligent guessing only a human is going to reasonably provide. The net result is that edge and corner cases escape, because automation is pretty shitty at catching those unless a fuzzer or an automated monkey (which is essentially a UI fuzzer) can be used. Generally, you only find problems where you've programmed it to look, which will be necessarily be pretty narrow, and the automation cannot use discretion to vary off the script to where new problems actually might lie. People have conflated the value of QA with the value of bad QA, and have chucked it out the window without understanding the cost. I'd guess that many have never even seen good QA to understand what it can do.
Nevertheless, this particular story is egregious. However much you cut back on quality control or assurance, the ONE FUCKING THING you always guarantee is that the user can get out of any problems that arise with minimum damage to money, data, time, and customer goodwill, roughly in that order of priority. Bugs in updates that kill your code signing stack and block further updates are just plain unforgivable.
Re: Microsoft has been making quality assurance mistakes lately
#47One of the main points: > how in the world can a patch to the Web browser lock up the operating system when you have an office suite installed? In a perverse way, I enjoy stuff like this. Why? Because, someday, Raymond Chen will write about this and I will get to read about it on The Old New Thing, and while Chen will do a wonderful job of explaining the immediate causes, they will point to other technical oddities s…
Too bad there's only one Raymond Chen. I'm sure there is enough material within Microsoft to keep 10 of him busy.
Re: Microsoft has been making quality assurance mistakes lately
#48The Windows 10 Technical Preview is available to author as part of Microsoft's quality assurance program. Technical Previews are alpha software [1] by another name and Release Candidates are beta software in Microsoft speak. [1] see: http://www.hanselman.com/blog/IntroducingASPNETVNext.aspx and http://blogs.msdn.com/b/cesardelatorre/archive/2014/05/12/th... for examples
Are you making the point that "it's beta/alpha, deal with it"? Because the author adresses exactly that, and the headline incident is only a one (egregious) example of a series of recent problems out of Microsoft.
My point is that the software is equivalent to an alpha release.
People can build there own points based on the author's calling it beta software.
Re: Microsoft has been making quality assurance mistakes lately
#49The title seems to apply to Apple, too (see iOS and Yosemite WiFi bugs), and could probably be applied to Google, too (Hangouts bugs, Lollipop problems, etc). But mistakes happen and many bugs are less obvious to reproduce than others, so I wouldn't point fingers too hard, personally.
Having worked on and off in software testing for the last fifteen years or so, I'm becoming increasingly convinced that there's a correlation between the recent trend of calling software test teams "Quality Assurance" and general purpose software that seems to get less stable as time goes on. Bear with me on my admittedly whacky theory that I haven't completely fleshed out yet.
What is called "quality assurance" is really "quality control" in the places I've observed. The difference is subtle. For example, at my last position I carried the title "Director of QA". That was a made-up title that didn't mean anything. It didn't mean anything because if I were truly a manager of QA, then when dev throws something over the wall that has no code reviews and no unit tests, I'm empowered to say, "no, we're not shipping this". But as we all know, that's not how it turns out. Instead my team is "quality control" because we're just testing the output and have no empowerment over the creation process. In other words, we're a test team, so quit with the "QA" crap just because it has the word "quality" in it. As the saying goes, you can't test in quality.
With that, I see two trends. One, which has been going on for a while what with TDD and the like, is an increase of the testing burden on dev. This is, generally speaking, a good thing IMO, especially if dev can be backed by a dedicated test team. Where it goes off the rails is what I hear from my Microsoft friends: there is no more test, "devs" (many of whom might be converted testers that may or may not be qualified "devs") are responsible for testing, too. Don't ever rely on your own testing if you're a dev. You'll test it the way it was written, and you'll not only miss edge cases but likely scenarios as well.
Could this explain the steaming pile that was the Xbox One (the only MSFT product I have in the house, or use, these days) in its early days? Could it explain the unbelievably buggy, nearly unusable, _Halo: The Master Chief Collection_ we got from a Microsoft studio? Dunno, could be coincidence, but it conveniently fits my theory; might want to grab some tin foil.
Trend #2 is a regression bug: throw shit over the wall and hope test finds the bugs. Call 'em "QA" so that when the quality isn't there we have someone to blame. That's what we used to do 20 years ago, then we as an industry got a clue and put some quality onus on dev while test got brought into the process earlier. Maybe it's just my corner of the world, but many seem to be going back to just that (I left my last position mainly because of that reason, after they bait-and-switched me between the interview and the actual work.)
Could be that tools that help with quality haven't kept pace with the increasing complexity of software systems. Could be, but I'm not sure I buy that. We've got tools I could only dream of 20 years ago (Xcode's point-and-click profiling tools, code coverage tools, static analysis). Test modeling tools need some work, but at least they exist.
I think an overly simplistic answer is that the industry doesn't know what to do with software test. We know we need some, but damn does it cost money. So how can we minimize that cost without appearing to just throw up our hands? After all, most of us are not building the guidance control for space probes, so who cares if Angry Confectionary Squashing Avians crashes once in a while? Instead we throw an understaffed and unempowered team of victims at the problem, then scream "why didn't test (pardon me: "QA") find this?!"
That's my take on it, anyway. I'm astounded at the poor quality of most software I encounter, but I think the general user population is used to thinking that it's either something they're doing wrong, or that's just the way software is. There are no late night pitchfork parties along Microsoft Way or One Infinite Loop, and people keeping paying, so there's not always a huge incentive to do otherwise. Many of us have said we'd pay for an upgrade that included nothing but bug fixes. Companies think otherwise (and they're probably right). Apple had their Snow Leopard, which they had to give away, but I can't think of any other examples.
Solutions? Oh, I've got a few but this is getting long-winded and rambling, so I'll spare those of you that stuck with me this long.
Re: Microsoft has been making quality assurance mistakes lately
#50My own WTF moment came recently when we dropped XP support for our product. Suddenly the window frames started painting improperly. One of the system calls we use in NCPaint returns different values depending on whether the executable is marked as XP compatible or not, even with no other change to the program.
How are you supposed to create reliable software when you can't even count on the raw API that's been unchanged since forever?