On testing#
When we write code, most of the time, we make mistakes. These mistakes can be hard to see.
Most untrained programmers write code, try it a few times at the interactive prompt, get the answers they expect, and then assume the code is OK.
Long experience shows that this is rarely true:
If it’s not tested, it’s broken
The code may give the right answer for some inputs and the wrong answer for others that you did not test;
The code may not work on another system or configuration.
The main way to reduce these problems is to write tests.
Writing tests#
For example, let’s say we had a module called rdmodule, like this:
%%file rdmodule.py
def rem_div(arg1, arg2):
""" Take `arg1` modulo 2, divide by `arg2`
"""
arg1 == arg1 % 2 # Remainder of dividing by 2.
return arg1 / arg2
Writing rdmodule.py
We call it rdmodule because it contains the rem_div function.
Interactively, we might try a few numbers:
import rdmodule
# Expecting (1 % 2) / 4 = 0.25
rdmodule.rem_div(1, 4)
0.25
# Expecting (0 % 2) / 3 = 0
rdmodule.rem_div(0, 3)
0.0
That looks right so far. But, if we had explored further, we would have found there’s a problem:
# Expecting (3 % 2) / 3 = 0.3333
rdmodule.rem_div(3, 2)
1.5
Oops, that was not what we wanted. Can you see the problem?
Keep looking for problems#
What we should have done, was write a range of tests for this function, to check it was working as we expect it too. We could make a test using assert. One way of doing that is to put some tests into a function called — say — test_rem_div, like this:
%%file rdmodule.py
def rem_div(arg1, arg2):
""" Take `arg1` modulo 2, divide by `arg2`
"""
arg1 == arg1 % 2 # Remainder of dividing by 2.
return arg1 / arg2
def test_rem_div():
# Expecting (1 % 2) / 4 = 0.25
assert rem_div(1, 4) == 1 / 4
# Expecting (0 % 2) / 3 = 0
assert rem_div(0, 3) == 0
# Expecting (3 % 2 / 3 = 0.3333
assert rem_div(3, 3) == 1 / 3
Overwriting rdmodule.py
Of course we will have to Changing the module, reloading to get the new version of the module:
import importlib
importlib.reload(rdmodule)
<module 'rdmodule' from '/home/runner/work/dipy-textbook/dipy-textbook/rdmodule.py'>
Then we can run the tests like this:
rdmodule.test_rem_div()
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
Cell In[7], line 1
----> 1 rdmodule.test_rem_div()
File ~/work/dipy-textbook/dipy-textbook/rdmodule.py:14, in test_rem_div()
12 assert rem_div(0, 3) == 0
13 # Expecting (3 % 2 / 3 = 0.3333
---> 14 assert rem_div(3, 3) == 1 / 3
AssertionError:
Indeed this reveals we have a problem we need to fix. We will do that soon.
Before we fix the problem, let us save ourselves the reload step, and the step of running the test_rem_div function by hand, by using Pytest.
Pytest has a command line script, pytest which will look for functions that start with the name test_ in .py files, and then run them.
Here we are using the Bash shell terminal available in Linux and macOS to run the command as if from the command line:
%%bash
python3 -m pytest rdmodule.py
============================= test session starts ==============================
platform linux -- Python 3.10.11, pytest-7.3.1, pluggy-1.0.0
rootdir: /home/runner/work/dipy-textbook/dipy-textbook
collected 1 item
rdmodule.py F [100%]
=================================== FAILURES ===================================
_________________________________ test_rem_div _________________________________
def test_rem_div():
# Expecting (1 % 2) / 4 = 0.25
assert rem_div(1, 4) == 1 / 4
# Expecting (0 % 2) / 3 = 0
assert rem_div(0, 3) == 0
# Expecting (3 % 2 / 3 = 0.3333
> assert rem_div(3, 3) == 1 / 3
E assert 1.0 == (1 / 3)
E + where 1.0 = rem_div(3, 3)
rdmodule.py:14: AssertionError
=========================== short test summary info ============================
FAILED rdmodule.py::test_rem_div - assert 1.0 == (1 / 3)
+ where 1.0 = rem_div(3, 3)
============================== 1 failed in 0.07s ===============================
---------------------------------------------------------------------------
CalledProcessError Traceback (most recent call last)
Cell In[8], line 1
----> 1 get_ipython().run_cell_magic('bash', '', 'python3 -m pytest rdmodule.py\n')
File /opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/IPython/core/interactiveshell.py:2475, in InteractiveShell.run_cell_magic(self, magic_name, line, cell)
2473 with self.builtin_trap:
2474 args = (magic_arg_s, cell)
-> 2475 result = fn(*args, **kwargs)
2477 # The code below prevents the output from being displayed
2478 # when using magics with decodator @output_can_be_silenced
2479 # when the last Python token in the expression is a ';'.
2480 if getattr(fn, magic.MAGIC_OUTPUT_CAN_BE_SILENCED, False):
File /opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/IPython/core/magics/script.py:153, in ScriptMagics._make_script_magic.<locals>.named_script_magic(line, cell)
151 else:
152 line = script
--> 153 return self.shebang(line, cell)
File /opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/IPython/core/magics/script.py:305, in ScriptMagics.shebang(self, line, cell)
300 if args.raise_error and p.returncode != 0:
301 # If we get here and p.returncode is still None, we must have
302 # killed it but not yet seen its return code. We don't wait for it,
303 # in case it's stuck in uninterruptible sleep. -9 = SIGKILL
304 rc = p.returncode or -9
--> 305 raise CalledProcessError(rc, cell)
CalledProcessError: Command 'b'python3 -m pytest rdmodule.py\n'' returned non-zero exit status 1.
If you get No module named pytest, you may need to install it. Check the Pytest web pages for instructions.
Notice that Pytest has found the test_rem_div function and run it,
finding our error. Notice too that Pytest gives us lots of information about
the test that failed, and the tests that it has run.
Finally, we fix the function:
%%file rdmodule.py
def rem_div(arg1, arg2):
""" Take `arg1` modulo 2, divide by `arg2`
"""
# Notice the single =
arg1 = arg1 % 2 # Remainder of dividing by 2.
return arg1 / arg2
def test_rem_div():
# Expecting (1 % 2) / 4 = 0.25
assert rem_div(1, 4) == 1 / 4
# Expecting (0 % 2) / 3 = 0
assert rem_div(0, 3) == 0
# Expecting (3 % 2) / 3 = 0.3333
assert rem_div(3, 3) == 1 / 3
Overwriting rdmodule.py
We confirm that the tests pass.
%%bash
python3 -m pytest rdmodule.py
============================= test session starts ==============================
platform linux -- Python 3.10.11, pytest-7.3.1, pluggy-1.0.0
rootdir: /home/runner/work/dipy-textbook/dipy-textbook
collected 1 item
rdmodule.py . [100%]
============================== 1 passed in 0.01s ===============================
Test modules#
It can get cluttered to have the test_ functions in the same module as the
code. To reduce clutter, we often write the tests out as a separate file
module, named after the module it is testing. In this case the file would be
test_rdmodule.py, like this:
%%file rdmodule.py
def rem_div(arg1, arg2):
""" Take `arg1` modulo 2, divide by `arg2`
"""
# Notice the single =
arg1 = arg1 % 2 # Remainder of dividing by 2.
return arg1 / arg2
Overwriting rdmodule.py
%%file test_rdmodule.py
# Import the function we are testing.
from rdmodule import rem_div
def test_rem_div():
# Expecting (1 % 2) / 4 = 0.25
assert rem_div(1, 4) == 1 / 4
# Expecting (0 % 2) / 3 = 0
assert rem_div(0, 3) == 0
# Expecting (3 % 2) / 3 = 0.3333
assert rem_div(3, 3) == 1 / 3
Writing test_rdmodule.py
%%bash
python3 -m pytest test_rdmodule.py
============================= test session starts ==============================
platform linux -- Python 3.10.11, pytest-7.3.1, pluggy-1.0.0
rootdir: /home/runner/work/dipy-textbook/dipy-textbook
collected 1 item
test_rdmodule.py . [100%]
============================== 1 passed in 0.01s ===============================
Luckily we thought to test this case. Now we have tested it, we have fixed it. We can keep testing it every time we edit the code, to make sure we haven’t broken anything. This turns out to be very important in assuring yourself that your code still does what you think it does.
The testing habit#
Testing is a habit. Once you have got into that habit, you will find it hard to break, because you will find lots of problems in your code that you did not suspect. With time, you will start to feel uncomfortable if you are using code without tests, because you know that there’s a big risk that it is wrong. Once that discomfort sets in, you are well on your way to become a programmer who can keep learning.