I recently gave a workshop on "TDD in Python with pytest", where I developed a very simple Python project together with the attendees following a TDD approach. It's a good way to introduce TDD, I think. I wrote each test together with the attendees, and then I left them the task of writing the Python code that passes the test. This way I could show TDD in action, introducing pytest features like the pytest.raises context manager or the use of assert while they become useful for the actual tests.

This is the approach that I follow in some of my posts on TDD here on the blog, for example A simple example of Python OOP development (with TDD) and A game of tokens: write an interpreter in Python with TDD - Part 1.

Part of the workshop was dedicated to pytest command line options and in general to what pytest can do as a testing framework. Unfortunately there was no time to go through this part, so I promised some of the attendees to give them a written version of it. This post is the fulfilment of that promise.

Please remember to import pytest before using functions, decorators or attributes prefixed by pytest., as I will not repeat it in each example.

Run single tests

If you want to run only a specific test you can provide its name on the pytest command line

$ pytest -svv tests/test_calc.py::test_addition

which for example runs the tests_addition test inside the tests/test_calc.py file. You can also specify the file name only to run all the tests contained there

$ pytest -svv tests/test_calc.py

Skipping tests

Sometimes it is useful to skip tests. The reason might be that some new code broke too many tests, and we want to face them one at a time, or that a specific feature had to be temporarily disabled. In all those cases the pytest.mark.skip decorator is your friend. Remember that a decorator is something that changes the way the decorated function works (for the skilled reader: it's a function wrapper). Assuming we are working on a tests/test_calc.py file the code might be

@pytest.mark.skip
def test_test_addition():
    [...]

The result on the command line will be (after running py.test -svv)

tests/test_calc.py::test_addition SKIPPED

Skipping with a reason

The previous solution is good for a temporary skip, but if the test has to remain deactivated for a long time it's better to annotate a specific reason for the exclusion. In my experience 1 day is enough to forget small details like this, so my advice is to always put a well-written reason on skipped tests. To add it you can use the reason attribute of the skip decorator

@pytest.mark.skip(reason="Addition has been deactivated because of issue #123")
def test_test_addition():
    [...]

Remember to add the -rs option to your command line to see the reason behind skipped tests. So after running py.test -svv -rs we will get something like

tests/test_calc.py::test_addition SKIPPED
[...]
============================= short test summary info =============================
SKIP [1] tests/test_calc.py:5: Addition has been deactivated because of issue #123

====================== 12 passed, 1 skipped in 0.02 seconds =======================

Skipping tests conditionally

Well, most of the time we will skip tests not for a stable reason, but according to some other condition that we can retrieve from the system, like the Python version, or maybe the region in which a server is running. The decorator that we need to use in that case is skipif, which accepts a condition (a boolean value) and a reason

import os


@pytest.mark.skipif(
    os.environ['AWS_REGION'] == 'us-west-2',
    reason="Addition has been deactivated in us-west-2 because of issue #234"
)
def test_addition():
    [...]

With this code running AWS_REGION=eu-west-1 py.test -svv -rs will run the test_addition test, while running AWS_REGION=us-west-2 py.test -svv -rs will skip it. The environment variable AWS_REGION set in the previous command lines is an example that simulates the presence of the variable in the system.

Run tests by name

You can selectively run tests by name using -k. This option accepts Python expressions that try to match the name of the test with the provided values. So

$ pytest -svv -k "test_addition"

will run all those tests which name contains the substring 'addition', like test_addiiton, test_addition_multiple_inputs, and test_complex_addition. A more complex expression could be for example

$ pytest -svv -k "test_addition and not complex"

which will run both test_addition and test_addition_multiple_inputs but not test_complex_addition.

Tagging tests

Tests can be tagged or labelled using pytest.mark, and the tag can be used to run or skip sets of tests. Let's say that we identify a set of very slow tests that we don't want to run continuously.

@pytest.mark.slow
def test_addition():
    [...]


def test_subtraction():
    [...]


@pytest.mark.slow
def test_multiplication():
    [...]

In the above example test_addition and test_multiplication have been decorated with pytest.mark.slow which tells pytest to label them with the slow identifier. At this point we can run all the tests that are tagged with the -m option

$ pytest -svv -m slow

Tests can be tagged multiple times

@pytest.mark.complex
@pytest.mark.slow
def test_addition():
    [...]

In this case the test will be run both by pytest -svv -m slow and by pytest -svv -m complex.

The -m option supports complex expressions like

$ pytest -svv -m 'not slow'

which runs all the tests that are not tagged with slow, or

$ pytest -svv -m 'mac or linux'

which runs all the tests tagged with mac and all the tests tagged with linux. Pay attention that -m expressions refer to the tags of each single test, so slow and complex will run only those tests that are tagged both with slow and with complex, and not all the tests marked with the first and all the tests marked with the second.

Adding a command line option

You can add custom command line options to pytest with the pytest_addoption and pytest_runtest_setup hooks that allows you to manage the command line parser and the setup for each test.

Let's say, for example, that we want to add a --runslow option that runs all the tests marked with slow. First, create the file tests/conftest.py, which is a file that pytest imports before running your tests, and use the pytest_addoption hook

def pytest_addoption(parser):
    parser.addoption("--runslow", action="store_true",
                     help="run slow tests")

The command line parser configuration will be stored into the config attribute of the setup of each test. Thus we can use the pytest_runtest_setup hook that runs before each test

def pytest_runtest_setup(item):
    if 'slow' in item.keywords and not item.config.getvalue("runslow"):
        pytest.skip("need --runslow option to run")

Here item is the single test, so item.keywords is the set of tags attached to the test, and item.config is the configuration after the parser run on the command line. This makes the previous code match all the tests that are decorated with @pytest.mark.slow and only when the --runslow option has been specified on the command line. If both those conditions are satisfied the pytest.skip function is run, which skips the current test adding the specified string as a reason.

Coverage

Coverage is a measure of the percentage of code lines are "hit" when running tests. Basically the idea is to discover if there are parts of the code that are not run during the tests, thus being untested.

If you follow a strict TDD methodology your coverage will be 100% always, because the only code you will write is the one that you need to pass the tests. But please, please, please keep this in mind: not everything is easily tested and in front of some complex parts of the code you always have to ask yourself "Is it worth?".

Is it worth spending 3 days to write a test for a feature? Well, if a failure in the new code means a huge financial loss for your company, yes. If you are writing a tool for yourself, and the code you are writing is not dangerous at all, maybe not. With all the shades of grey between these two black and white extreme cases.

So, don't become a slave of the coverage index. A coverage of more than 90% is heaven, and being over 80% is perfectly fine. I would say that, except for specific corner cases being under 80% means that you are not really following a TDD methodology. So, maybe go and review your work flow.

Anyway, pytest gives you a nice way to report the coverage using the coverage program. Just install pytest-cov with

$ pip install pytest-cov

and run pytest with

$ pytest -svv --cov=<name> --cov-report=term

where <name> is the name of the Python module you are testing (actually the path where the code you are testing is). This gives you a nice report with the percentage of covered code file by file.

$ py.test -svv --cov=mypymodule --cov-report=term

----------- coverage: platform linux, python 3.6.5-final-0 -----------
Name                          Stmts   Miss  Cover
-------------------------------------------------
mypymodule/__init__.py       3      0   100%
mypymodule/calc.py          23      0   100%
-------------------------------------------------
TOTAL                            26      0   100%

You may also use the term-missing report instad of just term, that lists the code blocks that are not covered

$ py.test -svv --cov=mypymodule --cov-report=term-missing

----------- coverage: platform linux, python 3.6.5-final-0 -----------
Name                          Stmts   Miss  Cover   Missing
-----------------------------------------------------------
mypymodule/__init__.py       3      0   100%
mypymodule/calc.py          23      2    91%   6, 11
-----------------------------------------------------------
TOTAL                            26      2    92%

Here I commented some of the tests to force the coverage percentage to drop. As you can see the report tells us that lines 6 and 11 of the mypymodule/calc.py file are not covered by any test.

Updates

2017-12-24: pytest.org (such an honour!) spotted a misspelled pytest.mark.skip. Thanks!

Feedback

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.