Monthly Archives: August 2014

Test Pages, Not Workflows

There can be a strong impulse to build automated testing (or even manual testing) around workflows. Makes sense, right? A workflow embodies the flow of actual work done by actual persons, so it’s good to test it.

The thing is, workflows are like chess games: many begin with similar opening moves. So when we execute a large number of workflow tests, many of the opening steps are repeated over and over again.

This is extremely wasteful of wall-clock time.

An alternative is to design testing that minimizes repetition.

What would that look like? Well, let’s take advantage of the page object pattern.

The page objects cover the application, right? So there’s a page object for each page in the application. Let’s organize the testing around that.

For each page, we can create a page test that:

  • Navigates to the page the most efficient way.
  • Performs and verifies all actions that can be initiated on the page.
  • Verifies all links on the page.

Much more efficient than a workflow test, is it not?

Like any lengthy automated test, the page test needs to guard against premature termination caused by an unhandled exception. The strategy I’ve used in the past is for the test to be divided into major parts, each of which is a function. If an exception is raised, the function catches and logs it, then returns cleanly so that the next function can continue.

(In my C# code, I’ve done this not by putting a try block into each function, but instead passing each function name to a special “catcher” function, wherein the named function is then invoked inside a try block.)

I’ll be trying out the page test concept in the sprint that starts today (yes, I’m agile!), and will report back.

Tiered Testing [Socratic Dialog]

Socrates: Let’s begin at the beginning. Now, tell me, Tester, what is the purpose of the build verification test?

Tester: Its purpose, Socrates, is to determine whether full regression testing should proceed.

Socrates: I see. And what is the alternative, if the full regression testing should not proceed?

Tester: The alternative is that the build is considered failed, and repairs must be made before the build is tried again.

Socrates: What sorts of test failures, bugs, would fail the build?

Tester: Well, bugs that block important parts of the testing, certainly.

Socrates: The full regression testing should not proceed unless the tests can actually do their work?

Tester: That’s correct, I think.

Socrates: Are there other failures that would fail the build?

Tester: Yes. I think so: failures of important functionality.

Socrates: Regression testing should not proceed unless the major functionality works?

Tester: That’s right.

Socrates: Any others besides blocking failures and major functionality failures?

Tester: No, Socrates, I think that’s it.

Socrates: Very well. Then let’s think about just those two types of failures.

Tester: As you say.

Socrates: Of the two, is each type of failure sufficient, by itself, to fail the build?

Tester: Yes, Socrates, certainly.

Socrates: All right, then. Suppose that there are failures in major functionality, but there are not any blocking bugs. In that case, the regression testing should not proceed?

Tester: I think that’s right.

Socrates: That must mean, then, that the information gathered by the regression testing would not help in diagnosing the failures in major functionality, and therefore is not needed.

Tester: Well, the information might be helpful. Let me think. Yes, it would be helpful. Very much so, now that I think about it.

Socrates: So a failure in major functionality should not, by itself, be sufficient to fail the build. The regression testing should begin, and would gather helpful information.

Tester: Yes, I do now think that’s so.

Socrates: And a blocking bug alone would be sufficient, regardless of whether there are major functionality failures.

Tester: Yes, it would be sufficient.

Socrates: I see. Therefore the major functionality testing on the one hand does increase the duration of the build verification test, but on the other hand does not contribute to determining whether to fail the build.

Tester: Again, true.

Socrates: Why, then, is major functionality testing included in the build verification?

Tester: I’m not sure, Socrates. Perhaps it should be included because we need to identify important failures sooner rather than later.

Socrates: Indeed, that is important.

Tester: Well, Socrates, at least I get some agreement from you today.

Socrates: I’m glad for that. But, according to what we’ve said, would it not be better to separate the testing into three tiers: build verification, major functionality, and full regression testing? That way, the build verification can complete sooner; ideally, the major functionality testing would be started at the same time, but if not, then immediately after the build verification test.

Tester: Yes, Socrates, you’re right.

Socrates: Thanks for that.

Tester: Therefore I see, finally, that it would be good to have three-tiered testing:

  1. Build verification test: Find disqualifying bugs first.
  2. Major functionality test: Find important bugs fast.
  3. Full regression test: Find as many bugs as possible.

Socrates: As you say, Tester.

Tester: And if possible, all three should begin at the same time, to get the results soonest. In case the build is failed, diagnosis and repair can begin immediately.

Socrates: Again, true.

Tester: Thanks, Socrates. I’ll begin working on this.

Socrates: You’re very welcome, Tester.

[Ed: Modern thinking is that the BVT should fail the build for a single failed verification. Note, however, that a single verification may be, under the hood, compound and complex. For example, if there are two ways to register a user on a website, the verification might be that at least one of those ways succeeds. The verification would fail only if there’s no effective way to register a user, because that would block testing.]