Monthly Archives: June 2014

Capture/Replay Fails

Well, I was just getting ready to write about how capture/replay does not work.

Instead, I'll link to this devastating assessment.

And, by the way, read the whole book.

Talking Pictures

Don’t forget that an important tool for showing the steps to reproduce a bug is a movie.

You can walk through the steps (and talk through at the same time), recording as you go. The movie can be an attachment to the bug report.

If necessary the movie can be speeded up or slowed down. I had an incorrect popover that disappeared in less than a second. I made a movie for the bug report, and included instructions for navigating to the right spot and running the movie slowly enough to see the popover, and stop the movie if desired.

The tool I’m using (Windows 7) is MS Screen Recorder, which is available for download.

This Is Gonna Be Fun!

In my two recent posts, I wrote about getting better by learning PowerShell, and about provisioning a test machine to run CUITs without VS.

Two things since then:

  • I’ve been able to determine that all the utilities involved in the provisioning have CLIs (that’s command-line interface, to the youthful among you) and can run in quiet (unattended) mode. That means that the provisioning can be automated.
  • I’ve looked a bit at PowerShell, and I think I’m going to like it!

Getting Better

One way to get better is to learn a new language.

I’m beginning Windows PowerShell. I have the Windows PowerShell Cookbook from Amazon. (I always like a cookbook.)

I’m going to love never again using cmd.exe, which I’ve always hated. I’ve sometimes used Cygwin instead, but this PowerShell is positively wired into Windows.

(Yes, yes, I know I’m late to the PowerShell party, but I haven’t really needed it up to now.)

Now Playing: CUITs Without VS!

I’ve spent time this week setting up a virtual machine. It now has MSTest, the Selenium binaries, and the Test Agent and Test Controller.

Together, they can run my Coded UI Tests without Visual Studio. This is essential if we’re going to have multiple test machines, because we don’t want to have to have a VS license for every test machine.

Wasn’t easy, though. I’ve captured all the steps in a doc. I need to write a PowerShell script to do this work!

Going Through Some Changes

Quite a while ago I ran across this post (now a PDF periodical article). I’d been doing what I call “change testing” for some time, and I thought this was a good introduction to the concept. Please see the section “Regression Testing” that begins on the second page. (But it’s okay — informative, even — to read the whole article.) I especially like the metaphor that sees change testing as a “quality ratchet.”

But I think it’s unfortunate that the authors characterize change testing as regression testing, which I think invites confusion.

I’ve done change testing on a large scale, and will have much more to say about it in forthcoming posts.

Concurrent Test Execution: Partitioning the Tests

When you execute your test suite across multiple machines, the tests should run concurrently. You want each test to run exactly once, so it’s important to partition the work correctly and efficiently.

You also want the total durations of the machines’ tests to be similar, so that you don’t have one machine still running long after the others have finished.

Here’s one way to do that.

The test machines will need:

  • A list of the names of the tests to execute.
  • A list of the names of the machines participating.

The names of the tests should be arranged by their expected durations: longest to shortest.

The names of the machines should be arranged by the expected “power” (speed of execution): fastest to slowest.

Each test machine makes some initial computations:

  • The count n of test machines participating.
  • The 0-based index i of its own machine name among the listed machine names.

In order for the testing to be distributed more or less evenly (by duration) among the test machines, each machine allocates to itself some of the tests.

The first, and fastest, machine (i == 0) is assigned:

  • The first (longest) test from the first n tests.
  • The last (shortest) test from the next n tests.
  • The first (longest) test from the next n tests.
  • The last (shortest) test from the next n tests.
  • And so forth.

More generally, for machine i, the tests are those among the listed tests that are at the following 0-based indexes:

  • n
  • 2ni – 1
  • 2n + i
  • 4ni – 1
  • 4n + i

You can think of it this in a simple way. If there are four test machines, then the tests — listed from longest to shortest — would be allocated to the machines as follows:   0,1,2,3;   3,2,1,0;   0,1,2,3;   3,2,1,0;   ….

This constitutes a partition (each test is allocated exactly once), and tends to distribute execution time evenly across the test machines.