Conductor

A system for testing distributed systems across a network

Many test frameworks exist to test code on a single host or, across a network, on a single server. Conductor is a distributed system test framework, written in Python, that can be used to coordinate a set of tests among a set of clients. The Conductor system allows a single machine to control several systems, orchestrating tests that require the cooperation of several networked devices.

A common use for Conductor is to test a network devices, such as a router or firewall, that is connected to multiple senders and receivers. Each of the senders, receivers, and the device under test (DUT) are a Player, and another system is designated as the Conductor.

The players, read commands over a network channel, execute them and return results to the conductor.

The conductor reads test descriptions from a configuration file, written using Python's config parser, and executes the tests on the players.

The tests are executed in Phases. A Phase contains a set of internal or external (shell/program) commands that are executed in order, per client. The four Phases currently defined are:

• Startup

The startup phase is where commands that are required to set up each device are execute. Examples include setting up network interfaces, routing tables, as well as creating directories to hold result files on the players.

• Run

The Run phase contains the commands that are the core of the test. An example might be starting a number of transmitting and receiving programs to generate and sink traffic through the DUT.

• Collect

In the Collect phase the Conductor sends commands to the Players to retrieve any data that was generated during the test, and places that data into a results directory on the Conductor, for later analysis.

• Reset

The last phase is the Reset phase, where the Conductor instructs the Players to reset any configuration that might have been set in the Startup phase and which might influence later test runs. It is the goal when writing Conductor tests that all the systems used in the test return to the state they were in prior to the Startup phase.

Each Phase has three parts. The Conductor first downloads the Phase to the Player, but does not tell it to execute. Downloading the Phase to each client allows the Conductor to start each Phase nearly simultaneously, although the fact that the Conductor itself serializes its communication among the clients means that there is a small amount of skew in when each Player is told to execute its steps.

This work supported by: Rubicon Communications, LLC (Netgate)