Python Nose: making your python tests smell better

A little about Python testing.

This article covers some basic aspects for running a Python test suite using nosetests. Is focused on some practical aspects to get a quick working set of tests and review the results.

When i started writing Python code the usual way to write and run a series of unit tests was using the standard Unittest module for declare the assertions and write your own runner that would have to keep track on asserts counting and write a sort of human readeable report, nothing too standard. For this reason most of the old open source python projects still mantains this kind of runners, pretty harder to mantain, because they have been adapted over the time to generate nicer reports, add coverage, inspect other kind of files , etc.

Since some years until now, so many things have changed on the Python testing scene, the standard Python module has become strong; many assertions cases has been added and a big utils collection is getting bigger everyday.

In the meanwhile Nose emerged to be the de-facto standard way to run unit tests, also they included a great assertion/util base libraries to extend your unit tests classes, all the report generation has become now an standard and Xunit is the overlord of all the output formats.

Usually i run my tests on a pre-commit base, so before push a change into the repo my test base is runned and i can check if my changes will break something in the code. Now i usually use a develop branch , and before move them to master branch i used a Jenkins or Travis server to run continually my tests suites and check if they will pass.

Using nose

As i mentioned, nosetests is the default standard on every project i am working on. So, let me share with you my standard configuration and how i tie these components.

A pretty simple example of a testeable class could be the following:

class Math:

def sum(self, a, b):  
    return a + b

    def subs(self, a, b):
    return a - b

def mul(self, a, b):  
    return a * b

On every test suite i wrote i use the nose.tools collection because it provides a set of convenience functions to make writing tests easier and don't depends on the unittest module.

So the basic test suite for the previous class could be something like:

import code  
import operator

m = code.Math()

def test_sum():  
    """Check is sum method is equivalent to operator"""
    eq_(m.sum(1, 1), operator.add(1, 1))


def test_sub():  
    """Check is sub method is equivalent to operator"""
    eq_(m.sub(2, 1), operator.sub(2, 1))


def test_mul():  
    """Check is mul method is equivalent to operator"""
    eq_(m.mul(1, 1), operator.mul(1, 1))

For run this test suite using nose you will need to execute the nosetests command from command line, my usual sentence is:

$ nosetests --verbosity=3 -x . 

That will turn on a verbose output and will interrupt the tests runner if some of the tests fails. You can review other command line options here

In the example case, the output of nose will be something like:

Check is sum method is equivalent to operator ... ok  
Check is sub method is equivalent to operator ... ok  
Check is mul method is equivalent to operator ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.001s

OK  

Some of the most important aspects of running nose instead of any other runner are the output format and coverage options. By default you can specifify the --with-coverage option and nose will output the missing and covered statements on your code.

Name    Stmts   Miss  Cover   Missing  
-------------------------------------
code        7      0   100%  

For compatibility between CI environments like Jenkins or Travis the nose runner is able to produce a standard output file format called XUnit. This format is produced by a plugin that is now included in the base nose distribution!. In order to get the XUnit output you have to pass the following sentence:

$ nosetests --verbosity=3 -x --with-xunit 

This command will generate a nosetests.xml in the current working directory, the output format isn't really human readeable:

<?xml version="1.0" encoding="UTF-8"?>

<testsuite name="nosetests" tests="3" errors="0" failures="0" skip="0">  
<testcase classname="test" name="test_sum" time="0.000" />  
<testcase classname="test" name="test_sub" time="0.000" />  
<testcase classname="test" name="test_mul" time="0.000" />

</testsuite>  

In order to produce a human readeable report from this XUnit output i wrote a pretty nice XSLT template that includes filters and other nice eye candy. My project is available on Github Nosetests-XUnit-XSLT.

In order to get the report formatted you need to run the following commands sequence:

$ sudo apt-get install xlstproc
$ xsltproc nosetests.xslt nosetests.xml > tests.html

The output file is more human readeable , so you can explore what happened in your last nose run.

By default nosetests will try to load a configuration file located in the current working directory called nose.cfg , my default configuration file looks like:

[nosetests]
verbosity=3  
with-xunit=1  
xunit-file=xunit.xml  
with-xcoverage=1  
xcoverage-file=coverage.xml  

If do you want to integrate nosetests inside a Jenkins installation , you will need to add support for XUnit this can be accomplished by using the Jenkins XUnit plugin.

Auto-generating tests

Another cool feature of nose for improve your tests base is the test generator support. Test generators allows you to improve the amount/quality of the testing functions yielding test cases with different parameters.

A good example for our base example class using generators:

def tests_sum():  
    """Generate multiple tests for sum method"""
    for i in range(1, 10):
        yield check_sum, i, i ** 2

def check_sum(n, nn):  
    """Check is sum method is equivalent to operator"""
    eq_(m.sum(n, nn), operator.add(n, nn))

This will generate ten new test cases, which directly contributes to making more robust code base. The output of nosetests for this new tests will look like:

``language-bash

Generate multiple tests for sum method ... ok
Generate multiple tests for sum method ... ok
Generate multiple tests for sum method ... ok
Generate multiple tests for sum method ... ok
Generate multiple tests for sum method ... ok
Generate multiple tests for sum method ... ok
Generate multiple tests for sum method ... ok
Generate multiple tests for sum method ... ok
Generate multiple tests for sum method ... ok
Check is sub method is equivalent to operator ... ok
Check is mul method is equivalent to operator ... ok


XML: nosetests.xml

Name Stmts Miss Cover Missing

mate 7 0 100%

Ran 11 tests in 0.005s

OK
```

What's next

During this article we covered some basic aspects of the standard Python test suite. I hope you can start exploring nose in deep and start writing your own enhancements.

In a next article on this series i am going to explain you *how to extend nose to create your own test runner , how to run tests in parallel to accelerate the running process and also how to write useful nose plugins to get the maximum advantage of the awesome Python testing tool.


Jorge Niedbalski

Dev and Ops , and might be the opposite.


View or Post Comments