The Python Experiment Suite

The Python Experiment Suite is an open source software framework written in Python to execute and organize large scale automated software experiments with many useful features:

Download

The Python Experiment Suite is available for download on the github repository.

Usage

Running experiments in the Python Experiment Suite only takes a few easy steps.

  1. Create a class and derive it from the PyExperimentSuite class
  2. Add object creation and a call to the suite's start() method at the bottom of the script
  3. Implement the reset() and iterate() methods
  4. Define your experiments and the parameters you want to use in the configuration file
  5. Run the script from the command line
Below is a very simple working example on how to implement an experiment. Create a file called "suite.py" with the code below:
from expsuite import PyExperimentSuite

class MySuite(PyExperimentSuite):

  def reset(self, params, rep):
    """ for this simple example, nothing needs to be loaded or initialized."""
    pass

  def iterate(self, params, rep, n):
    # access config file parameters through the params dictionary
    alpha = params['alpha']
    beta = params['beta']

    # your experiment code goes here...

    # return any results you want to store in log files
    ret = {'rep':rep, 'iter':n, 'alpha':alpha, 'beta':beta}
    return ret        

if __name__ == '__main__':
  # create object and call start method
  mysuite = MySuite()
  mysuite.start()
  
The configuration file "experiments.cfg" needs to be in the same directory and contain the following lines:
[DEFAULT]
repetitions = 5
iterations = 10
path = results

[myfirstexp]
alpha = 1
beta = 0.5
  
Now you can run the script from the command line: python suite.py. After execution, the script will have created a directory called "results" with a sub-directory called "myfirstexp" (the experiment name from the config file) and 5 log files, each with 10 lines.

Another example is available in the Snippets section of this website. It will show a few more features, like the optional restore support. For a full description of all features, check out the documentation pdf.