For a long time now, I’ve been working on automated testing for legacy software (or legacy features of evolving software).
It’s been an education.
The thing is, a new feature should be developed with a test-first strategy that uses assertions. And a failed assertion should be repaired immediately.
But for a legacy feature, automated testing will reveal a number of low-severity bugs that will not be fixed anytime soon. That’s why detailed logging and a detailed changes report are paramount.
So now I’ve founded a new GitHub project, RubyTest, that will make available, as working software, much of what I’ve learned.
The beginning is modest — just a logger, a few helpers, and their tests. (Yes, the framework itself gets thorough testing.)
Next, I think, will be a simple REST client example that uses the endpoints at JSONPlaceholder.com. This will be the opportunity to show how the framework will handle endpoints, data objects, and verdicts.
Much more to come.
Here are some principles I try to keep in mind.
|Readability||Make code easy to read: code is read more often than
|DRYness: Don’t Repeat Yourself||Avoid redundant code and data.|
|YAGNIty: You Ain’t Gonna Need It||Don’t write code before it’s needed.|
|Sloth||Maximize use of existing packages and libraries. The line of code you don’t write is the line of code you never have to debug. — Steve Jobs|
|Explicitness||Explicate everything, even when “unnecessary.” If it goes without saying, it would go better by saying it. — Tallyrand|
|Cleanliness||Keep everything clean and consistent: run static code
analysis; resolve all issues before committing.
|Failed Verdict Diagnostics||Log data sufficient to diagnose a failed verdict.|
|Error Diagnostics||Detect test errors early, fail early, log useful information.|
|Monitoring||Trust, but verify. Monitor documentation for changes (programatically, of course).|
From the documentation for Ruby gem
Contracts let you clearly – even beautifully – express how your code behaves, and free you from writing tons of boilerplate, defensive code. You can think of contracts as
assert on steroids.
I wrote here about getting better by reviewing the literature from time to time. The literature I specifically mentioned was one of the programming language cookbooks.
Well, recently I reviewed the Ruby Cookbook (while my car was being serviced) and saw Recipe 10.16, about Contracts. The recipe had code for a Contracts module; I was delighted to find later that since the book’s publication date [2005 — but a new edition is in work!], the Contracts idea has been made into a Ruby gem: Contracts.
You’re gonna love this!
I. Don’t. Bargain.
— Marshall Sam Gerard, The Fugitive
I wrote yesterday about the warnings I’ve been getting from RubyMine, and my eliminating them.
RubyMine warns when it cannot find the definition of a method. I got many of these warnings for methods that are defined dynamically, at runtime, instead of statically.
At first I thought I might leave the code as-is, because if a method really is not defined, there will be a runtime error when it’s time to call it. But then I realized that there are two important consquences: RubyMine will not be able to perform either code completion or code navigation for an ‘unfound’ method.
This is important to me, because I want someone writing test code to get as much help as possible from the tools. If RubyMine can’t do code completion for a method, the programmer will have to know outright the name (and correct spelling) of the method he wants.
That will not do. So I’ve changed my code so that RubyMine now finds the definitions, and can do both code completion and code navigation.
After 90 days in my new project, it seems certain that we’re going to continue testing with Ruby, so I’ve bought an IDE, RubyMine. That gives me a debugger, which is often convenient and sometimes critical.
It also gives me static code analysis (RubyMine calls it code inspection), which is the equivalent of compiler warnings. In a few hours, I’ve reduced the number of warnings to zero, which is where I want it to remain. If there are a lot of unexamined warnings, something important may be hiding therein. (Actually, there was a warning about a case statement that had no else clause. I’m usually scrupulous about that, because is absence can cause downstream symptoms that are difficult to diagnose later on.)
Now my code is warning-free!
Okay, my recent post was about my Changes Report. In this post I’m writing about my History Report, which is a spreadsheet.
(If your history report is a spreadsheet, too, you may want to skip the first three paragraphs below, and resume reading at Each verdict cell.)
The left headers are in the first few columns at the left; their job is to identify each row as belonging to a single test verdict. I’m using the Ruby gem
MiniTest::Unit, so the identifying information is: suite name, test name, method name, verdict identifier.
The top headers are in the first few rows at the top; their job is to identify the build and summarize the results. Each column’s headers include the date and time, the build identifier, and the count of each possible outcome: passed, missed, failed. The leftmost of these build columns is for the most recent build. Older builds are represented in columns to the right.
Each column (other than those I’ve just mentioned) show the verdicts for a single test run; the most recent run is just to the right of the identifying information, and the older runs are to the right of that.
Each verdict cell shows the outcome for the verdict in that row: passed, missed, or failed. These outcome cells are colored according to their values. (See my post on colors.)
Beyond that, there’s one other important bit of data involved: if the verdict or its underlying data changed since the previous test run, the verdict is rendered in bold and italic, and is in fact a link. The link takes me to the very verdict in the Changes Report, and there I find the full information about the verdict: its expected and actual values for the current and previous test runs.
The bold italic link is present only when there was a change in the verdict. That means that for an old (unchanged) verdict, I can look to the right to find the most recent bold italic link, and that tells me when the most recent change occurred.
The remaining item I’ll be adding (soon) is a column for defects. Each cell will be a link to the defect item (in Rally), if there is one.
Oh, and did I say? Both my Changes Report and my History Report are generated automatically from the test logs (the only exception being the defect information, which must be updated manually).