Tag Archives: verification

Report Verifications, Not ‘Tests’

In many testing shops, counts of passed/failed ‘tests’ are the main part of summary test reporting. But a ‘test’ result is just a collection of verification results, and its reporting is just a way to obscure the actual situation.

Suppose, just for a very simple example, that we have 100 tests, each with 10 verification points. Suppose further that the report says that 5 tests failed.

What, exactly, does that mean? Well, actually, it doesn’t mean anything very exact.

At one extreme, it could mean that in the 5 tests, all 50 verifications failed. At the other it could mean that just one verification failed in each of the 5 tests — 5 failures in all.

So we know that for failures we have somewhere in the range 5% (50/1000) down to 0.5% (5/1000). Pretty fuzzy, no?

That’s why I don’t report passed/failed ‘tests’; I report passed/failed verifications.

[Actually, I report passed/missed/failed verifications, where ‘missed’ means that the verification point in the test was not reached.]

Clean and DRY Verifiers

In a Coded UI Test (CUIT), a test method is a method that has attribute TestMethod. A test method is what many might call a test script. It’s the outermost method in the test, and directs the test steps.

Some say that only the test method itself should perform verifications, that a method in a page object (or other supporting object) should not perform verifications automatically:

The usual reason given is that such automatically performed verifications definitely will affect performance, and may not even be wanted or needed in a particular test context.

I agree, but with one addition: a method can appropriately perform verification at the request (direct or indirect) of the test method. So the request for verification should originate in the test method.

No matter where the actual verification is performed, the verifier method must log the expected value, the actual value, the verdict (pass or fail), and a descriptive message.

Question: Where is the best place to perform the actual verification?

Answer: Wherever it will be clean and DRY (Don’t Repeat Yourself).

A that will be where the verification method has the fewest and simplest parameters passed to it: in a page object!

A page object encapsulates its entire page, so it already has access to the HTML control that has the actual value for the verification. That means that a verification method in the page object need not pass the actual value. And that means, in turn, that a call to the verification method has fewer parameters: at most, just the expected value and a message string. That’s pretty DRY.

But wait, there’s more!

When the expected value is a constant (a table column header, for example), that value can also be stored in the page object. So in that case, the verification method would have no parameters at all. That’s really DRY.

Examples:

  • Home page verifies logged-in user’s name:
    public Boolean VerifyUserName(String expectedValue, String message = "User name")
  • User page verifies user data:
    public Boolean VerifyUser(User expectedValue, String message = "User")
  • Users page verifies that user does not exist:
    public Boolean VerifyUserNotExist(User expectedValue, String message = "User does not exist")
  • Page object knows its own column headers: public VerifyColumnHeaders()
  • Page object knows its own URL: public VerifyUrl()

Finally, I have a special-purpose verifier:

  • Verify that the locators in a page object correspond to actual controls in the UI: public Boolean VerifyLocators()

So performing verification in a page object, under the supervision of the test method, is easy. And doing so improves both cleanliness and DRYness.