Live data from Hacker News

Maybe getting rid of your QA team was bad

davidkcaudill.medium.com

261–269 of 269 posts

Re: Maybe getting rid of your QA team was bad

#261

Earlier quoted context omitted.

I can't count how many products I've seen with 100% code coverage that objectively, quantifiably doesn't work. That’s because code coverage doesn’t find the bugs that result from code you didn’t write, but should have. Code coverage is but one measure, and to treat it as the measure is folly. (But, yes, I have heard a test manager at a large software company we’ve all heard of declare that test team was done because…

It's more because code coverage isn't measured correctly. What you want is branch coverage. If you have something like this: if condition1: do_something1() if condition2: do_something2() if condition3: do_something3() There are 8 possible paths for the code to follow here, but you can cover 100% of lines of code in one test. If your code coverage measure tells you that testing the case where conditions 1, 2, and 3 ar…

No, it's more because when the bug was filed, you said to yourself, "oh, was I supposed to update the $HOOZIWIZIT in that transaction, too? Oops." Code coverage won't save you from that. Matching test cases to requirements, or summat, can be helpful, but good golly it's a bunch more paperwork.

Your point is a good one, but not where I was going.

Re: Maybe getting rid of your QA team was bad

#262
post #85

I am currently working with a startup that spends a lot of time on building tests that need to be refactored every sprint because it's early stage and the assumptions change. I am shocked at the amount of developer-hours spent on tests that need to be disabled / deleted instead of just hiring 1 - 2 super cheap manual testers that just go through the flows days in and out. For me it's a no brainer, if I were CEO / CTO…

When outsourced, you either A) rely on someone in your org to tell them what to test and what the workflows are, ie use them as a warm body/monkey to click on things for you - this is what most people see QA as, which is silly - or B) you rely on the outsourced QA to know your product and know what is important or what all of the edge cases are. If your product is non-trivial in size or scope, ie it is not a cookie-c…

> If your product is non-trivial in size or scope

Are there many products that reach such sizes without achieving Product Market Fit (PMF)? I feel like after this step is achieved, QA becomes pivotal and involves a great combination of manual and automated procedures. So I agree with you in this regard.

But going back to my initial assumption. I think starting a fresh company without PMF and spending a lot on QA until that is achieved, might not be the best approach.

Re: Maybe getting rid of your QA team was bad

#263
post #104
post #56

I worked at two companies 15-20 years ago that invested in top-tier QA teams. They were worth their weight in gold. The products were world class because the QA team were fantastic at finding bugs we developers didn't think of looking for because we were too close to the problem. We are too used to looking at the happy path. One key attribute to both companies is that it was dictated from on high that the QA team had…

How often did you release?

At the first company we released a few times a year. You have to remember that CI/CD wasn't a thing back then. While I worked on a web-based product at this company, it was a time when it was still very common to release software on physical media like CD's.

At the second company we integrated Agile so we released every two or weeks (I can't recall which). The first time I heard about CI/CD was at this company from co-workers who came back from a conference all excited about it.

Re: Maybe getting rid of your QA team was bad

#264

Earlier quoted context omitted.

I've always been impressed by hardware QA test teams I've worked with. On Google Fiber, they had an elaborate lab with every possible piece of consumer electronics equipment in there, and would evaluate every release against a (controlled) unfriendly RF environment. ("In version 1.2.3.4, the download from this MacBook Pro while the microwave was running was 123.4Mbps, but in version 1.2.4.5, it's 96.8Mbps." We actual…

> Going deeper into the training aspect, something I find very useful are fuzz tests. Could you share some details of fuzz tests that you've found useful? I tend to work with backend systems and am trying to figure out whether they will still be useful in addition to unit and integration tests.

Sure!

I wrote my first fuzz test with a go1.18 prerelease just to get a feel for the new framework. It found a crash instantly: https://github.com/jrockway/json-logs/commit/77c87854914d756.... This commit contains the fuzz test and the fix. (3 cases could be nil; I only ever thought about 2 of them. The fuzzer found the 3rd almost immediately.)

For my project at work, we have an (g)RPC server with a ton of methods. The protos have evolved since 2015 while trying to be backwards compatible with older clients. As a result, many request parameters are optional but not expressed as oneof, so it's hard to figure out what a valid message actually is, and different parts of the code do it differently. (As an aside, oneof is totally backwards compatible... but the protos evolved without knowing that.) I wrote a fuzz test that generates random protos and calls every single RPC with them: https://github.com/pachyderm/pachyderm/blob/master/src/serve...

This test is imperfect in many ways; it knows nothing about protos, and just generates random bytes and sends it to every method. Most of the time, our protoc-gen-validate validators reject the message as being nonsense, but that's totally fine. The test found about 50 crashes pretty immediately (almost null messages, basically a non-null message where the first tag is null), and one really interesting one after about 16 hours of runtime on my 32 core Threadripper workstation. Writing this and fixing the code took about a day of work. I just randomly did it one day after writing the call-GRPC-methods-with-introspection code for a generic CLI interface.

I did this concurrent with a professional security review as part of our company's acquisition and... they didn't find any of these crashes. So, I think I added some value.

It's worth mentioning that most people wrap gRPC calls with a defer func() { ... recover() } block so that bad RPCs don't crash the server. We didn't at the time, but I implemented the fuzz tests with such a wrapper. (So the test doesn't crash when it finds a panic, I just return a special response code when there is a panic, and the test is marked as failing. Then the next test can proceed immediately in the presence of --keep-fuzzing or whatever.) Either way, I prefer an error message like "field foo in message BarRequest is required" to "Aborted: invalid memory reference or nil pointer dereference." and now we have that for all the cases the fuzzer noticed. The user is sad either way, but at least they can maybe self-help in the first case.

All in all, I haven't spent a ton of time writing fuzz tests, but each one has caught some bugs, so I'm happy. A good tool to have in your toolkit.

Re: Maybe getting rid of your QA team was bad

#265

This might be my personal experience, but I've never encountered a QA team that actually writes the tests for engineering. I have only had QA teams that wrote "test plans" and executed them manually, and in rarer cases, via automated browser / device tests. I consider these types of tests to be valuable, but less so than "unit tests" or "integration tests". With this model, I have found that the engineering team ends…

in the classic model, most QA orgs were a useless appendage. partially by construction, but largely because QA gets squeezed out when dev is late (when does that happen?). they aren't folded in early, so they twiddle their thumbs doing 'test infrastructure' and 'test plans', until they finally get a code drop and a 48 hr schedule to sign off, which they are under extreme pressure to do. but every once and a while you…

I worked on a team where we structured things so we had working groups of 2-3 devs + 1 QA engineer for individual features, which were typically less than a week. Idea being the QA person was involved in the writing of the acceptance criterion and the testing at the end, etc, but also along-the-way testing.

It worked exceptionally well, in my opinion but the QA revolted; they didn't like it, said that it felt like we were trying to make them into software developers. And so it ended, and it was back to "throw it over the wall".

Re: Maybe getting rid of your QA team was bad

#266

Earlier quoted context omitted.

False dichotomy. Poor dev practice is not fixed by elimination of QA, but rather fixed by improving dev practice. The “five why’s” can help.

If you're working without a net, you're going to be more careful. And 5 whys is not a particularly great overall practice. https://qualitysafety.bmj.com/content/26/8/671

You're also more likely to die when mistakes inevitably happen. Dismissing reason is another strategy not likely to work well in your favor.

Re: Maybe getting rid of your QA team was bad

#267
post #54

> The most conscientious employees in your organization are the most bitter. They see the quality issues, they often address them, and they get no recognition for doing so. When they speak up about quality concerns, they get treated like mouthbreathers who want to slow down. They watch the “move fast and break things” crowd get rewarded time after time, while they run around angrily cleaning up their messes. To these…

This comment requires some context and nuance. I've worked with conscientious engineers. Sometimes they are right but their delivery mechanism is broken. Sometimes they are just in the wrong place. If we're building a POC SASS product, it really doesn't need the quality of a avianoics microcontroller. All these trade-offs come with risk and cost, good engineers need to know the difference.

I think it is a matter of perspective as who has "broken delivery mechanism". Would you consider expectations of fixing unique key constraints in 8 year old database tables, which are regularly cause of bugs, "avionics grade quality expectations"? And this is one of many such things I have seen first hand.

Re: Maybe getting rid of your QA team was bad

#268

QA Engineers are some of the best debuggers too. They have their hands in the pipeline, src and test directories, and often work with all aspects of developing and deploying the application. When I was a QA lead I often ran into software engineers that couldn’t be bothered to read a pipeline error message (and would complain daily in Slack) and when it came to optimizing the pipeline they would ignore base problems a…

[deleted]

Re: Maybe getting rid of your QA team was bad

#269

Earlier quoted context omitted.

> Good QA'ers know more about how the product actually works than anybody else. More than PM's, more than sales, and more than most dev teams. Not disagreeing with this, but there's one thing they won't always be aware of. They won't always know what code a dev touched underneath the hood and what they might need to recheck (short of a full regression test every single time) to verify everything is still working. I k…

>They won't always know what code a dev touched underneath the hood and what they might need to recheck (short of a full regression test every single time) to verify everything is still working. Not really. As QA I always reviewed the checkins since yesterday before opening up the daily build. Between the bug comments and the patch comments, even if the patch itself is a bit Greek to me, I can tell what was going on…

Let's say that there's some sort of list component that's used in like six other places (that's maybe not obvious because there's different configurations or something).

Six months later I work on a new page that incorporates that component that requires some sort of change in that component that could theoretically break it elsewhere, at least how it looks or something, like alters the padding or something in a way that shoves things onto the next line in one of the instances.

If I don't mention what other places that component is used in my comments, are you going to know where all to look for the other instances of that component? Even if you look at the source code, it will only show changes, it won't say 'oh this component is used elsewhere' unless you actively bring up the source code yourself and do a code search yourself.

Because it's a visible component, maybe you'll just have the memory that something that looks like this is elsewhere, but maybe you won't remember. Or maybe you're fairly new to the project and aren't aware of these other parts of the site using it.

What about something that's even less visible, like something that takes in data and then sorts, transforms, and adds metadata to it? Then I make a change to it to accomodate for some new requirement on a new page. Maybe it's used for four other locations, but the changes only shows the function being altered and the new page using it. That change I made broke it elsewhere, but I don't mention where else to check in my notes. Are you going to think to check each place where data is loaded to make sure it's working correctly, without me explicitly telling you to check those spots?

If you can, great, you've got a real intuition for this that I doubt many QA people have. I don't even know to check these other places sometimes, and I worked on the code myself and can do a quick text search to see where else it's being used.

This was why I said what I did. It's not just about 'that area', the change could potentially affect another application in different repository entirely sometimes, especially if a shared library is involved. I've literally had like four or five of these situations happen in the past six months of my current project, where my change broke something elsewhere.

Usually I caught it, but I also told QA to check it explicitly anyway since I don't usually test things 100% (since I'm not QA). When I was a more junior developer, I missed these things more often and QA would sometimes miss it also since they didn't know to retest something.

Post reply on HN