My automated tests produce two reports:
- History report.
- Changes report.
In my test logs, each verdict is one of: passed, failed, missed (the verification point was not reached).
Now what the managers want to know is: How many of each there were. That’s what’s in the history report: today’s results, along with historical results. I’ll write about the history report in my next post.
What I want to know is: What’s different from the previous test run. That’s what’s in the changes report: all the differences between the current test run and the previous one.
The changes report groups verdicts as follows:
- New failed.
- New missed.
- New passed.
- Changed failed.
- Changed passed.
- Old failed.
- Old missed.
- Old passed.
The last three — old failed, old missed, and old passed — are of no immediate interest to me. The current result is exactly the same as the previous result. There’s no action I need to take, because all these were dealt with after some previous test run: defect reports opened, closed, updated, etc.
The first three — new failed, new missed, and new passed — obviously need my attention. Defect reports will need to be opened, closed, updated, etc.
The middle two — changed failed and changed passed — also need my attention:
Changed failed: A changed failed verdict is one that failed in the previous test run, then failed in the current test run, but in a different way. This occurs when the actual value changes from one wrong value to another. Investigation is required.
Changed passed: A changed passed verdict is one that passed in the previous test run, then passed in the current test run, but in in a different way. This occurs when both the expected value in the test and the actual value delivered by the application have changed, but also agree. Usually this would be because the developer gave advance notice of a change, which the tester accommodated by pre-adapting the test.
So what of the changes report itself? Well, it has nine sections: a summary, plus a section for each item in the first list above.
The summary lists the other sections, linking to each, and showing me the count of verdicts in each. The links allow me to navigate quickly to whichever section I want.
Each of the other sections begins with a list of the verdict ids for the verdicts it contains; each verdict id in that list links to the data for the verdict. Again, the links facilitate navigation.
At the links, each verdict’s data is presented in a small table that gives the verdict id, along with the expected and actual values for both the previous test run and the current one. The table is “over-and-under,” showing the corresponding values one above the other; this makes it easy for me to spot differences, even between similar values. The values in the table are displayed in a monofont, which also makes spotting differences easier.
And of course, my reports are kinder and gentler than some others.