Python testing breadcrumbs

What is Python

Python is a very simple and powerful language that allows even newbies to quickly get interesting results without significant issues.
What’s not so well known, especially for those new to the topic, is that a project’s long term success, despite the selected language, depends not just on its good design but also on the tests’ architecture!
To beginners, tests can feel like a rough path they want to avoid, but trust me when I say that, when correctly approached, they’re easily tamed.

Let’s find out together what unit/integration and system tests are.

• Unit tests: those whose execution doesn’t rely on external components dependence;
• Integration tests: they aim at verifying the interfaces’ integrity between the application’s components;
• System tests: their goal is to test the entire product and verify if it meets the requirements.

Method is the keyword!

I suggest you create some tests early in the development stage, especially the unitary ones, and starting from the simplest classes.
As the project takes shape and its classes evolve, you can add more tests or edit the existing ones to reflect the changes you plan to make, as per the TTD paradigm.
The Test Driven Development (TDD) approach aims at implementing a test before the component to be tested. In other words, you test a single permutation and then write what little code you need to pass such test. Then you extend the tested permutations and extend the component’s code to keep on passing those tests.

Another useful piece of advice is: if the implementation of a test case is too complex of difficult, then probably the thing you’re testing in the first place is as well, so you’d better focus on making it simpler (keep an eye out for low cohesion). A good option might be split it into simpler parts (divide et impera, anyone?).

Test case: back to the basics

To set up the examples you’ll find in this article we used Phyton test standard library, unittest. Each test case is written by implementing a class usually called test case. A test case is a set of single tests with the same goal or object implemented as class methods. Tests within the same test case sometimes share instruction or configurations known as test fixtures.

The success of each test depends on the occurrence (or lack of thereof) of errors during the test case method’s execution, and therefore the success of a test case depends on the success of all the single tests it contains.

Let’s start out with an example that will help us better understand the topics we treated so far: simple as it is, this example will give us a good starting point to understand more complex projects.
I set up a small project, a command line interface for the management of an address list. Too easy?
The following screenshot shows the project’s source tree.

.
|____tests
| |____integration
| | |______init__.py
| | |____test_contact.py
| |____system
| | |______init__.py
| | |____test_app.py
| |____unit
| | |______init__.py
| | |____test_address.py
| |______init__.py
|____address.py
|____app.py
|____contact.py
|____main.py

The app.py file manages the interaction with the users. In the contact.py file is the class responsible for the contact’s management, while in the address.py file is the class for the address’ management. The main.py file is the app’s simple entrypoint, while the test package will host the test suite for this example: inside it are the three sub-packages that will contain the three types of tests we’ll use for this example. Such division is not mandatory, but recommended. The file with the test case classes, on the other hand, will need to match the test_*.py pattern to be identified by the test runner when it’s launched on default.
There’s a second rule about the test case method’s names: the test runner will run as single tests those that start with “test”. There are no other rules, except that the classes containing the test cases are, of course, an extension of the TestCase class from the unittest package, about which I strongly recommend you check on the official documents.

Unit Tests: what they are and how they work

Let’s start from the basics: unit tests. As I mentioned before, these are tests for single features and have no dependences. They’re usually simpler, but you must not ignore or underestimate them.
We start by testing the Address class; inside it, other the builder, is a method that shows the address’ parts as a JSON object. The first test case aims at verifying the integrity of an instance from the Address class and the correctess of the JSON object obtained from the eponymous method.
The file with our test case is in the tests/unit folder: to test the Address class in the light of the previous recommendations, the file was named test_address.py.

class AddressTest(TestCase):

   def setUp(self) -> None:
       self.a = Address('via Test', '1', '00000', 'Test City', 'Testing State')

   def tearDown(self) -> None:
       self.a = None

   def test_create_post(self):
       self.assertEqual('via Test', self.a.street)
       self.assertEqual('1', self.a.number)
       self.assertEqual('00000', self.a.postcode)
       self.assertEqual('Test City', self.a.locality)
       self.assertEqual('Testing State', self.a.state)
       self.assertIsNone(self.a.recipient)

   def test_json(self):
       self.assertDictEqual({
           'street': 'via Test',
           'number': '1',
           'postcode': '00000',
           'locality': 'Test City',
           'state': 'Testing State',
           'recipient': None,
       }, self.a.json())

The setUp and tearDown methods allow to define the instructions carried out before and after each class test respectively. In this case, they’re used to instantiate and then delete the Address item used in the tests. Obviously, the object could’ve been instantiated from the test methods, but we chose this path to show an example of test fixture.

The assert* invocations used in the test_* methods come from TestCase and check that the invocation of the tested object’s methods is in accordance with the expected. In this case, we used a small subset, but you can find all the available ones on the official documents.

This test may look trivial and pedantic, and it somehow is, but if the class builder’s signature should change (due to refactoring or change of requisites), the tests’ failure will be a good reminder to check on all the code that uses that class! Besides, one must start testing something, right? 😉

Now let’s move to another exercise to create a unitary test case for the Contact class, too.

Integration tests: what they are and how they work

The Contact class features the builder, a method that returns contact information including the associated addresses as a JSON object, and a method to add an address to the contact. Contrarily to the Address class, that has no dependencies, Contact depends on the latter since it uses it in the add_address and json methods. Here come the integration tests.

class ContactTest(TestCase):

   def setUp(self) -> None:
       self.c = Contact('Name', 'Surname', 'Nickname')
       self.c.add_address('via Test', '1', '00000', 'Test City', 'Testing State')

   def test_add_address(self):
       self.assertEqual(1, len(self.c.addresses))
       self.assertEqual('via Test', self.c.addresses[0].street)
       self.assertEqual('1', self.c.addresses[0].number)
       self.assertEqual('00000', self.c.addresses[0].postcode)
       self.assertEqual('Test City', self.c.addresses[0].locality)
       self.assertEqual('Testing State', self.c.addresses[0].state)

   def test_json(self):
       expected = {
           'name': 'Name',
           'surname': 'Surname',
           'nickname': 'Nickname',
           'addresses': [
               {
                   'street': 'via Test',
                   'number': '1',
                   'postcode': '00000',
                   'locality': 'Test City',
                   'state': 'Testing State',
                   'recipient': None,
               },
           ],
       }
       self.assertDictEqual(expected, self.c.json())

The ContactTest class, as seen in the picture, only tests two methods that use Address: the goal is to ensure that the two classes are well integrated, as in the add_address method correctly creates an Address instance, and that json correctly manages it. The two test_* methods body is easy to understand: there’s nothing new compared to what previously described. The other methods’ tests are responsibility of unitary tests instead.
 

System Tests: what they are and how they work

class AppTest(TestCase):

   def setUp(self) -> None:
       # Mock contacts variable in app.py
       contact = Contact('Test', 'Testing', 'test')
       app.contacts = {
           'test': contact
       }

   def test_menu_prints_prompt(self):
       with patch('builtins.input', return_value='q') as mocked_input:
           app.menu()
           mocked_input.assert_called_with(app.MENU_PROMPT)

   def test_menu_calls_print_contacts(self):
       # patch does the magic!
       with patch('builtins.print') as mocked_print:
           with patch('builtins.input', return_value='q'):
               app.menu()
               mocked_print.assert_called_with('- Test Testing, aka test')

   def test_menu_calls_print_contacts_twice(self):
       with patch('builtins.input') as mocked_input:
           with patch('app.print_contacts') as mocked_print_contacts:
               mocked_input.side_effect = ('l', 'q')
               app.menu()
               self.assertEqual(mocked_print_contacts.call_count, 2)

At a first glance, the AppTest class may look intimidating, but let’s approach it gradually. This system test’s goal is to check that the command line interface correctly responds to user’s interactions. The simplest case can’t be but the proof of the correct start of the app, which is what we tested in the test_menu_prints_prompt method.
Yet, there’s some important news: the unittest.mock library (better check its documents) patch function. patch is actually a decorator/context manager used in the with block and momentarily replaces the input function or method with another object that imitates it (unittest.mock.Mock).

Why this imitation? Well, we could set it up to return a specific output, or interrogate it to know what parameters were used to invoke it during tests, or even to know how many times it was invoked.
Let’s now analyze the test_menu_prints_prompt method: first we patch the input function and configure so that, when invoked, it returns the “q” string. We then call the app.menu function and verify that the patched input version is invoked inside it with the command line prompt.
The second test, the test_menu_calls_print_contacts method, looks a lot like the first one, but in this case, other than input, the print function is patched too. Its goal is to check that all the contact are correctly printed when the app is launched, in fact the setUp method creates one to ensure the set isn’t empty and something actually passes to the patched version of print.
The last method we’re analyzing is test_menu_calls_print_contacts_twice: this test will show how the menu correctly responds to the received input. Here the input feature is patched to mimic the user’s choice to see the contact list (“l”): since the print happens at the launch of the app.menu feature, the print_contacts feature, if correctly patched, is invoked twice.
A detail on the side_effect propriety of the Mock item: it can be a feature to invoke anytime the instance of that object is invoked, or a list (or tupla) of values that will be returned in the specific order they’re passed at every invocation of the instance.

Test execution

Now that we’re done with the examples, we can reveal the last piece of this guide: the test execution!
If we use an IDE such as PyCharm the job’s basically done, since this tool gives us the chance to execute with one click even the single test case method.
For those who can’t resist the terminal’s charm, and staying within the borders of our example, the command to launch is really simple:

python -m unittest discover -s tests

The command will execute all the tests it finds in the tests folder, giving the result in output. In this case, too, I suggest you check on the command’s documents for further examples and other options.

Grand Finale (...?)

This guide only showed a small subset of the possible tests, even if just in some easy example. In fact, to clearly prove that all requirements are met, it’s better to set two test cases for each requirement: one to ensure the expected result (positive) and another to ensure the launch of an exception or an error message (negative). So, if you want to do things right, roll up your sleeve, but it’s only a matter of time/resources, because there’s nothing too complicated!

Lastly, I strongly suggest you adopt a dedicated tool to automate the test execution, such as Jenkins or Travis CI, since these tools offer you a chance to set up an environment to manage the app’s building and/or release. Alternatively, you can properly set up a GIT Hook in development environment to launch the tests’ execution with a GIT event: the possible events are many, and in my opinion it’s better to use a pre-commit, but even better a pre-push if the tests take a long time to run.

See you next time!