Interesting reading replies here. Way more formalities than I've ever engaged in.
Requirements are codified into test cases. After signoff of the spec/design/test plan is complete, there's no going back and checking.
Ask HN: How do you keep track of software requirements and test them?
111–120 of 134 posts
Re: Ask HN: How do you keep track of software requirements and test them?
#112Earlier quoted context omitted.
AFAIK the bugs were caught, known about, and deliberately ignored. In fact even when the bug caused a fatal error that brought an instance crashing (to the ground, literally!), it was ignored both by Boeing and the US government.
Saying they 'ignored' it is quite generous, considering the former CEO essentially blamed the pilots (source: https://www.bloomberg.com/news/features/2021-11-16/are-boein... ). Here's an excerpt from the article... --- “No, again, we provide all the information that’s needed to safely fly our airplanes,” he answered. Bartiromo pressed: But was that information available to the pilots? “Yeah, that’s part of the traini…
Typical to blame it on PEBKAC. "The pilots should have turned the plane off and on again." - Muillenberg, probably.
Re: Ask HN: How do you keep track of software requirements and test them?
#113Earlier quoted context omitted.
Actually most functional safety projects use the v-model (or similar, topography can vary a little as to needs), which is waterfall laid out a slightly different way to more clearly show how verification and validation closes out all the way back to requirements with high degrees of traceabilty. I've always wanted to break that approach for something a little more nimble, probably by use of tools - but I can't see ag…
what does functional safety mean in the context you are talking about? like fighter jets? or what?
He listed standards. They're for industrial processes and machines - think factories where the processes and machines have life-safety hazards.
Re: Ask HN: How do you keep track of software requirements and test them?
#114With a small team, I'm using an open source tool called reqflow (https://goeb.github.io/reqflow/) to track requirement to source code with a Doxygen keyword //! \ref RQ-xxxx. It's generating a traceability matrix and is quite simple to use (perfect for a small team). In my case, I'm using grep on the source code to create the traceability matrix.
For tracking requirement to test, I'm using testlink (testlink.org) where you can enter your requirements from existing documents and linked them to test cases. The documentation is not perfect, better start here:
https://www.guru99.com/testlink-tutorial-complete-guide.html
You can go to Bitnami to get a docker image.
Re: Ask HN: How do you keep track of software requirements and test them?
#115Tests with the Jira issue id in them. Simple, easy, scriptable.
Bonus point: you can run the code coverage with all the tests for a certain feature and see which code is responsible for supporting this feature.
Re: Ask HN: How do you keep track of software requirements and test them?
#116We're an FDA regulated medical device startup, with a pretty low budget for the moment. Our current setup is two pronged, in-house, and automated. The first piece is the specification documents, which are simple word docs with a predictable format. These cover how the software SHOULD be implemented. From these documents, we automatically generate the mission critical code, which ensures it matches what we say it does…
Thanks for sharing. I just realized your use case could fit quite neatly into Inflex (my app, not open for signups, but has a “try” sandbox), a use case I never considered. I have an unexposed WIP type of cell that is a rich text document. The rich document you edit has corresponding code, which you can also edit in the other direction. (screenshot https://mobile.twitter.com/InflexHQ/status/14923564133263360... ) The…
The data in the spreadsheet is really the most valuable part! I like the idea of using a small DSL to manage this, we actually already have some (very simple) DSL-like things in the spreadsheet to make it easier on editors of the spreadsheet.
Of course, with systems like JAMA, that automatically ingest Word docs and extract requirements, even to the point of generating verification tests, in turn kicking off dependency updates, the development loop becomes quite tight.
We definitely have to improve on the process that exists right now, as it's quite cumbersome to get all set up, and won't really generalize to new devices, but it's been a great learning experience, and I think it's really improved our development process overall!
Definitely a very complex problem to automate in the general case, makes sense why the JAMAs and Greenlight Gurus of the world can charge ungodly sums per seat.
Re: Ask HN: How do you keep track of software requirements and test them?
#117I think it's important to keep requirements in Git along with the source code. That way when you implement a new feature you can update the requirements and commit it along with the code changes. When the PR is merged, code and requirements both get merged (no chance to forget to update e.g. a Confluence document). Each branch you check out is going to have the requirements that the code in that branch is supposed to…
What happens when your requirements span multiple projects and repositories and microservices including frontend and backend work etc.? That doesn't all fit into a git PR.
Re: Ask HN: How do you keep track of software requirements and test them?
#118Gitlab. Just use Issues you can do everything with the free tier. (It's called "Issues workflow" - gitlab goes a little overboard though, but I'd look at pictures of peoples issues list to get examples). My opinion would be to not use all the fancy features that automatically tie issues to merge requests, releases, epics, pipelines etc... it's way to much for a small team that is not doing any type of management. Jus…
If possible, could I get your opinion on a specific example? In my current situation, I was asked to add a feature which required a few (java) classes. So - * It seems like this would have been a milestone? * So then maybe a few issues for the different classes or requirements? * For each issue, after/during development I would note what tests are needed, maybe in the comments section of the issue? Maybe in the descr…
Each issue can be from a 1 line fix to a week or even 2 week thing. But by that point, probably a metaissue that corresponds to other issues or issue with inline subtasks (GitHub has check boxes).
Features cut across code, so no 1-1 mapping with classes. Tests are generally self-documenting and land alongside the feature they are for. You can document them, but likely either a comment in the issue/PR if technically interesting, or in a separate ~wiki doc as part of a broader specification.
Ideally each commit is valid & passes tests (see "conventional commits") and each issue/PR has accompanying tests whether around a new feature or bugfix. Particular test frameworks change every year.
Re: Ask HN: How do you keep track of software requirements and test them?
#119Earlier quoted context omitted.
For safety system software most people I know would be very nervous (as in, I'm outta here) about testing software components and then not testing the end result as a whole, just too many possible side effects could come into play, including system wide things that only reveal themselves when the entire program is complete and loaded/running. What you describe already occurs to some extent in the process and machiner…
to reiterate on parents endorsement for agile and the point that you seem to be taking issue with: nothing in Agile says you can't run final acceptance tests or integration tests before shipping. we have done this in quite a couple of companies where things like functional safety or other requirements had to be met. agile sadly gets a bad rep (as does devops) for the way it is rolled out in its grotesque perverted st…
to reiterate on parents endorsement for agile and the point that you seem to be taking issue with: nothing in Agile says you can't run final acceptance tests or integration tests before shipping.
This is exactly right. I work in a highly regulated space, and we have been working in an Agile framework for awhile now. There are two iterations baked into every release cycle (at the end) for final regression testing. That cycle will re-run every test case generated during the program increment, plus additional test cases chosen based on areas of the application that were touched during development.
On top of final validation, we also have an acceptance validation team that runs full integration tests after final validation is complete.