Concurrent Test Execution: Partitioning the Tests

When you execute your test suite across multiple machines, the tests should run concurrently. You want each test to run exactly once, so it’s important to partition the work correctly and efficiently.

You also want the total durations of the machines’ tests to be similar, so that you don’t have one machine still running long after the others have finished.

Here’s one way to do that.

The test machines will need:

  • A list of the names of the tests to execute.
  • A list of the names of the machines participating.

The names of the tests should be arranged by their expected durations: longest to shortest.

The names of the machines should be arranged by the expected “power” (speed of execution): fastest to slowest.

Each test machine makes some initial computations:

  • The count n of test machines participating.
  • The 0-based index i of its own machine name among the listed machine names.

In order for the testing to be distributed more or less evenly (by duration) among the test machines, each machine allocates to itself some of the tests.

The first, and fastest, machine (i == 0) is assigned:

  • The first (longest) test from the first n tests.
  • The last (shortest) test from the next n tests.
  • The first (longest) test from the next n tests.
  • The last (shortest) test from the next n tests.
  • And so forth.

More generally, for machine i, the tests are those among the listed tests that are at the following 0-based indexes:

  • n
  • 2ni – 1
  • 2n + i
  • 4ni – 1
  • 4n + i

You can think of it this in a simple way. If there are four test machines, then the tests — listed from longest to shortest — would be allocated to the machines as follows:   0,1,2,3;   3,2,1,0;   0,1,2,3;   3,2,1,0;   ….

This constitutes a partition (each test is allocated exactly once), and tends to distribute execution time evenly across the test machines.

Advertisements

2 thoughts on “Concurrent Test Execution: Partitioning the Tests

  1. You forgot one very important thing. Dependencies of the tests; be it dependency on data to run a test (either already available or generated), dependency of one test upon another to run (run order) or a combination of both. If you have truely isolated (independent) tests then you can split them up along the lines of what you describe. The other ‘dependent’ tests can be grouped together and run on their own machines so that you do not tie up computing power. Also, you need to think about dividing them up according to purpose of the tests as well.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s