Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you keep track of software requirements and test them?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 134 posts

Re: Ask HN: How do you keep track of software requirements and test them?

#11

Post by oddeyed »

In a small team, I have found that a simple spreadsheet of tests can go a long way. Give it a fancy name like "Subcomponent X Functional Test Specification" and have one row per requirement. Give them IDs (e.g. FNTEST0001).

What sort of tests you want depends a lot on your system. If you're working on some data processing system where you can easily generate many examples of test input then you'll probably get lots of ROI from setting up lots of regression tests that cover loads of behaviour. If it's a complex system involving hardware or lots of clicking in the UI then it can be very good to invest in that setup but it can be expensive in time and cost. In that case, focus on edge or corner cases.

Then in terms of how you use it, you have a few options depending of the types of test:

- you can run through the tests manually every time you do a release (i.e. manual QA) - just make a copy of the spreadsheet and record the results as you go and BAM you have a test report

- if you have some automated tests like pytest going on, then you could use the mark decorator and tag your tests with the functional test ID(s) that they correspond to, and even generate a HTML report at the end with a pass/fail/skip for your requirements

Re: Ask HN: How do you keep track of software requirements and test them?

#12

Post by lotyrin »

When it's technically feasible, I like every repo having along side it tests for the requirements from an external business user's point of view. If it's an API then the requirements/tests should be specified in terms of API, for instance. If it's a UI then the requirements should be specified in terms of UI. You can either have documentation blocks next to tests that describe things in human terms or use one of the DSLs that make the terms and the code the same thing if you find that ergonomic for your team.

I like issue tracking that is central to code browsing/change request flows (e.g. Github Issues). These issues can then become code change requests to the requirements testing code, and then to the implementation code, then accepted and become part of the project. As products mature, product ownership folks must periodically review and prune existing requirements they no longer care about, and devs can then refactor as desired.

I don't like over-wrought methodologies built around external issue trackers. I don't like tests that are overly-concerned with implementation detail or don't have any clear connection to a requirement that product ownership actually cares about. "Can we remove this?" "Who knows, here's a test from 2012 that needs that, but no idea who uses it." "How's the sprint board looking?" "Everything is slipping like usual."

Re: Ask HN: How do you keep track of software requirements and test them?

#14

Post by onion2k »

oumua_don17 wrote: post #8

Gitlab does have requirements management integrated but it’s not part of the free tier. [1] https://docs.gitlab.com/ee/user/project/requirements/

I just tried that feature. I added a requirement. I could only add a title and description, which wasn't great. The requirement appeared in the Issues list, which was a bit odd, and when I closed the issue the requirement disappeared from the requirements list.

Whatever that feature is meant to be, it definitely isn't requirements management. Requirements don't stop being requirements after you've written the code.

Re: Ask HN: How do you keep track of software requirements and test them?

#15

Post by Cerium »

We used to use MKS and switched to Siemens Polarion a few years ago. I like Polarion. It has a very slick document editor with a decent process for working on links between risks, specifications, and tests. Bonus points for its ability to refresh your login and not loose data if you forget to save and leave a tab for a long time.

For a small team you can probably build a workable process in Microsoft Access. I use access to track my own requirements during the drafting stage.

Re: Ask HN: How do you keep track of software requirements and test them?

#16

Post by adrianmsmith »

I think it's important to keep requirements in Git along with the source code. That way when you implement a new feature you can update the requirements and commit it along with the code changes. When the PR is merged, code and requirements both get merged (no chance to forget to update e.g. a Confluence document). Each branch you check out is going to have the requirements that the code in that branch is supposed to implement.

For simple microservice-type projects I've found a .md file, or even mentioning the requirements in the main README.md to be sufficient.

I think it's important to track requirements over the lifetime of the project. Otherwise you'll find devs flip-flopping between different solutions. E.g. in a recent project we were using an open-source messaging system but it wasn't working for us so we moved to a cloud solution. I noted in the requirements that we wanted a reliable system, and cost and cloud-independence wasn't an important requirement. Otherwise, in two years if I'm gone and a new dev comes on board, they might ask "why are we using proprietary tools for this, why don't we use open source" and spend time refactoring it. Then two years later when they're gone a new dev comes along "this isn't working well, why aren't we using cloud native tools here"....

Also important to add things that aren't requirements, so that you can understand the tradeoffs made in the software. (In the above case, for example, cost wasn't a big factor, which will help future devs understand "why didn't they go for a cheaper solution?")

Also, if there's a bug, is it even a bug? How do you know if you don't know what the system is supposed to do in the first place?

Jira tickets describe individual changes to the system. That's fine for a brand new system. But after the system is 10 years old, you don't want to have to go through all the tickets to work out what the current desired state is.

Re: Ask HN: How do you keep track of software requirements and test them?

#17

Post by mytailorisrich »

It's very useful to keep track of changes and to be able to have text to describe and explain, so for me the simplest tool would not be to use a spreadsheet but to create a git repo and to have one file per requirement, which can be grouped into categories through simple folders. You can still have a spreadsheet as top level to summarise as long as you remember to keep it up to date.

Top-level requirements are system requirements and each of them should be tested through system tests. This usually then drips through the implementation layers from system tests to integration tests, to unit tests.

Regression testing really is just running your test suite every time something changes in order to check that everything still works fine.

Re: Ask HN: How do you keep track of software requirements and test them?

#18

Post by stefanoco »

Zooming into "requirements management" (and out of "developing test cases") there's a couple of Open Source projects that address specifically this important branch of software development. I like both approaches and I think they might be used in different situations. By the way, the creators of these two projects are having useful conversations on aspects of their solutions so you might want to try both and see what's leading from your point of view.

* https://github.com/doorstop-dev/doorstop * https://github.com/strictdoc-project/strictdoc

Of course requirements can be linked to test cases and test execution reports, based on a defined and described process.

How to build test cases is another story.

Re: Ask HN: How do you keep track of software requirements and test them?

#19

Post by jcon321 »

Gitlab. Just use Issues you can do everything with the free tier. (It's called "Issues workflow" - gitlab goes a little overboard though, but I'd look at pictures of peoples issues list to get examples).

My opinion would be to not use all the fancy features that automatically tie issues to merge requests, releases, epics, pipelines etc... it's way to much for a small team that is not doing any type of management.

Just use some basic labels, like "bug" or "feature" and then use labels to denote where they are in the cycle such as "sprinted", "needs testing" etc. Can use the Boards feature if you want something nice to look at. Can even assign weights and estimates.

You can tie all the issues of a current sprint to a milestone, call the milestone a version or w/e and set a date. Now you have history of features/bugs worked on for a version.

In terms of testing, obviously automated tests are best and should just be a requirement built into every requirement. Some times though tests must be done manually, and in that case attach a word doc or use the comments feature on an issue for the "test plan".

Re: Ask HN: How do you keep track of software requirements and test them?

#20

Post by asabla »

adrianmsmith wrote: post #16

I think it's important to keep requirements in Git along with the source code. That way when you implement a new feature you can update the requirements and commit it along with the code changes. When the PR is merged, code and requirements both get merged (no chance to forget to update e.g. a Confluence document). Each branch you check out is going to have the requirements that the code in that branch is supposed to…

I really like this idea.

However, what would be missing from this is discussions for each requirement specified. Or would you want to include that as well?

It would be nice having a dedicated directory for requirements, src, infra, tests and docs. Which would make things easier to track over long period of time I think

Post reply on HN