This post was updated on 2016-03-26 and is revision 0.2.0.
Testing a module you've written in python can easily spiral out of control, particularly one which simulates a physical phenomenon. You want to be sure that your tests are comprehensive, but its quite easy to fall down a rabbit hole of writing completely inane tests for edge cases that will never occur in practice. After writing and refactoring my
tec module several times, I've gotten a sense of the various types of tests that need to be written and how to organize them. I intend this strategy to be along the lines of a "style guidelines" type document like pep8 or Strunk and White.
These guidelines assume the reader has an understanding of testing python code and the
unittest framework. Resources for testing code with Python are given at the end of this essay.
Tests tend to fall into two categories: API tests and numerical tests. API tests test the interface of the module: ensuring proper return types of methods, ensuring exceptions are raised under appropriate conditions, methods that return units actually return the correct unit, etc. Numerical tests verify the numerical accuracy of the methods that return the results of the physical simulation.
Test style guidelines
Organization of test files in the filesystem
Files containing tests for a python module should be located in a
test directory in the root of the repo for the sake of separation of concerns. Each file in the
test directory should contain tests for one and only one class/function defined in the module. Files containing tests should be named according to the rubric
At the top of the file containing the tests, I have the typical python import statements. Below that, I usually define some common initialization parameters needed for the tests. These initialization parameters aren't used directly in the tests as I will explain momentarily.
Next, I usually define a base class (called
Base) which is a subclass of
unittest.TestCase. In this class I define a
setUp method (and possibly
tearDown) because I've found that most of the tests I write require a common environment in which to be run. This base class allows me to split up tests according to functionality by writing subclasses of
Base while avoiding copying/pasting the same
setUp method between these classes. Therefore I avoid introducing the category of bugs associated with copying and pasting code for reuse. In this
Base class I
copy the common initialization parameters I've defined at the top of the test file into attributes of the class so that a uniform environment is created for each test via
Base.setUp. I've found there's a category of bugs that appear if I directly use the initialization parameters defined at the top of the test file: some tests require the initialization parameters to be changed slightly. Its possible to define a parameter and have it change in memory as a result of a test. Subsequent tests will therefore throw errors.
I organize tests according to functionality by subclassing
Base (and thus
unittest.TestCase). For classes that implement a physical phenomenon, tests tend to fall into the following categories:
Instantiation- test all aspects of instantiating an object. Includes input of wrong type, input outside of a bound, etc.
Set- test all aspects of instantiating an object. Includes similar tests to the instantiation tests.
MethodsInput- test methods that take input. Includes passing data of the wrong type, data that's outside of a constraint, etc.
MethodsReturnType- test that methods return the proper type of data.
MethodsReturnUnits- test that methods return the proper units, assuming the method returns data of type
MethodsReturnValues- tests that the methods return appropriate values. These tests are particularly important for methods which actually implement the physical model; I call them "calculator methods."
Each class contains methods that implement a test. These methods are named according to the rubric
Where "name" refers to the name of the attribute or method being tested and "condition" refers to the condition being tested. The name of the class which contains the attribute or method under test should not appear in the test method name, nor should the functionality of the test. For example, the following test names should not be used:
The first example contains the name of the class (
ClassName) in which the attribute
temp is found. The class name is superfluous in this case because this test is presumably located in a file named
The second example contains the name of the functionality being tested, i.e. "input." The functionality in this case is superfluous because the test is presumably a method of the class
Docstrings should include the name of the class under test, the functionality being tested, the name of the attribute/method being tested, and the condition being tested. In this way,
nosetests will display all the information required to locate the failed/erronious test within this classification scheme. The order is not strictly required; sometimes it is easier to write something like "Setting ClassName.strictly_positive_attribute < 0 is invalid." The docstrings should describe logically what the test method does. In other words, it should be clear from the docstring how the test is passed or failed.
Additional style guidelines for numerical tests
Each calculator method requires the following items in order to be considered to be fully tested.
- A unit listed in its docstring. No mention of unit indicates the quantity is unitless.
- A value of uncertainty listed in its docstring.
- An analysis of the method's uncertainty documented in that method's test's docstring.
- A unit test that tests the method against standard data.
- Additional unit tests that test any other special, edge, or corner cases.
If a class has several methods that do similar things, many of the tests in a class will be very similar. I wrote a really dumb command line tool I call
testwriter to automatically generate code for these tests.
Example/skeleton test file
# -*- coding: utf-8 -*- # Import statements # ================= # Common input parameters # ======================= # Base classes # ============ class Base(unittest.TestCase): """ Base class for tests This class defines a common `setUp` method that defines attributes which are used in the various tests. """ def setUp(self): pass # Test classes # ============ class Instantiation(Base): """ Tests all aspects of instantiation Tests include: instantiation with args of wrong type, instantiation with input values outside constraints, etc. """ pass # Input arguments wrong type # ========================== # Input arguments outside constraints # =================================== class Set(Base): """ Tests all aspects of setting attributes Tests include: setting attributes of wrong type, setting attributes outside their constraints, etc. """ pass # Set attribute wrong type # ======================== # Set attribute outside constraint # ================================ class MethodsInput(Base): """ Tests methods which take input parameters Tests include: passing invalid input, etc. """ pass class MethodsReturnType(Base): """ Tests methods' output types """ pass class MethodsReturnUnits(Base): """ Tests methods' output units where applicable """ pass class MethodsReturnValues(Base): """ Tests values of methods against known values """ pass