diff options
author | 2019-10-11 23:05:01 +0200 | |
---|---|---|
committer | 2019-10-11 23:05:01 +0200 | |
commit | d6bedd16eb8fc81b2f0a17992ba000ed27fd7d72 (patch) | |
tree | a63f9d0924c251539d52edd25628cd504ef8516b | |
parent | Change pipeline testrunner to xmlrunner (diff) |
Make textual changes to testing guide
I've made some textual changes to the testing guidelines defined
in README.md.
-rw-r--r-- | tests/README.md | 54 |
1 files changed, 34 insertions, 20 deletions
diff --git a/tests/README.md b/tests/README.md index 471a00923..4ed32c29b 100644 --- a/tests/README.md +++ b/tests/README.md @@ -1,6 +1,6 @@ # Testing our Bot -Our bot is one of the most important tools we have to help us run our community. To make sure that tool doesn't break, we've decided to start writing unit tests for it. It is our goal to provide it with 100% test coverage in the future. This guide will help you get started with writing the tests needed to achieve that. +Our bot is one of the most important tools we have for running our community. As we don't want that tool tool break, we decided that we wanted to write unit tests for it. We hope that in the future, we'll have a 100% test coverage for the bot. This guide will help you get started with writing the tests needed to achieve that. _**Note:** This is a practical guide to getting started with writing tests for our bot, not a general introduction to writing unit tests in Python. If you're looking for a more general introduction, you may like Corey Schafer's [Python Tutorial: Unit Testing Your Code with the unittest Module](https://www.youtube.com/watch?v=6tNS--WetLI) or Ned Batchelder's PyCon talk [Getting Started Testing](https://www.youtube.com/watch?v=FxSsnHeWQBY)._ @@ -12,38 +12,45 @@ We are using the following modules and packages for our unit tests: - [unittest.mock](https://docs.python.org/3/library/unittest.mock.html) (standard library) - [coverage.py](https://coverage.readthedocs.io/en/stable/) -To ensure the results you obtain on your personal machine are comparable to those in the Azure pipeline, please make sure to run your tests with the virtual environment defined by our [Pipfile](/Pipfile). To run your tests with `pipenv`, we've provided two "scripts" shortcuts: +To ensure the results you obtain on your personal machine are comparable to those generated in the Azure pipeline, please make sure to run your tests with the virtual environment defined by our [Pipfile](/Pipfile). To run your tests with `pipenv`, we've provided two "scripts" shortcuts: -1. `pipenv run test` will run `unittest` with `coverage.py` -2. `pipenv run report` will generate a coverage report of the tests you've run with `pipenv run test`. If you append the `-m` flag to this command, the report will include the lines and branches not covered by tests in addition to the test coverage report. +- `pipenv run test` will run `unittest` with `coverage.py` +- `pipenv run report` will generate a coverage report of the tests you've run with `pipenv run test`. If you append the `-m` flag to this command, the report will include the lines and branches not covered by tests in addition to the test coverage report. -**Note:** If you want a coverage report, make sure to run the tests with `pipenv run test` *first*. +If you want a coverage report, make sure to run the tests with `pipenv run test` *first*. ## Writing tests -Since our bot is a collaborative, community project, it's important to take a couple of things into when writing tests. This ensures that our test suite is consistent and that everyone understands the tests someone else has written. +Since consistency is an important consideration for collaborative projects, we have written some guidelines on writing tests for the bot. In addition to these guidelines, it's a good idea to look at the existing code base for examples (e.g., [`test_converters.py`](/tests/bot/test_converters.py)). ### File and directory structure -To organize our test suite, we have chosen to mirror the directory structure of [`bot`](/bot/) in the [`tests`](/tests/) subdirectory. This makes it easy to find the relevant tests by providing a natural grouping of files. More general files, such as [`helpers.py`](/tests/helpers.py) are located directly in the `tests` subdirectory. +To organize our test suite, we have chosen to mirror the directory structure of [`bot`](/bot/) in the [`tests`](/tests/) subdirectory. This makes it easy to find the relevant tests by providing a natural grouping of files. More general testing files, such as [`helpers.py`](/tests/helpers.py) are located directly in the `tests` subdirectory. All files containing tests should have a filename starting with `test_` to make sure `unittest` will discover them. This prefix is typically followed by the name of the file the tests are written for. If needed, a test file can contain multiple test classes, both to provide structure and to be able to provide different fixtures/set-up methods for different groups of tests. -### Writing individual and independent tests +### Writing independent tests -When writing individual test methods, it is important to make sure that each test tests its own *independent* unit. In general, this means that you don't write one large test method that tests every possible branch/part/condition relevant to your function, but rather write separate test methods for each one. The reason is that this will make sure that if one test fails, the rest of the tests will still run and we feedback on exactly which parts are and which are not working correctly. (However, the [DRY-principle](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) also applies to tests, see the `Using self.subTest for independent subtests` section.) +When writing unit tests, it's really important to make sure each test that you write runs independently from all of the other tests. This both means that the code you write for one test shouldn't influence the result of another test and that if one tests fails, the other tests should still run. + +The basis for this is that when you write a test method, it should really only test a single aspect of the thing you're testing. This often means that you do not write one large test that tests "everything" that can be tested for a function, but rather that you write multiple smaller tests that each test a specific branch/path/condition of the function under scrutiny. + +To make sure you're not repeating the same set-up steps in all these smaller tests, `unittest` provides fixtures that are executed before and after each test is run. In addition to test fixtures, it also provides special set-up and clean-up methods that are run before the first test in a test class or after the last test of that class has been run. For more information, see the documentation for [`unittest.TestCase`](https://docs.python.org/3/library/unittest.html#unittest.TestCase). #### Method names and docstrings -It's very important that the output of failing tests is easy to understand. That's why it is important to give your test methods a descriptive name that tells us what the failing test was testing. In addition, since **single-line** docstrings will be printed in the output along with the method name, it's also important to add a good single-line docstring that summarizes the purpose of the test to your test method. +As you can probably imagine, writing smaller, independent tests also means that the number of tests will be large. This means that it is incredibly import to use good method names to identify what each test is doing. A general guideline is that the name should capture the goal of your test: What is this test method trying to assert? + +In addition to good method names, it's also really important to write a good *single-line* docstring. The `unittest` module will print such a single-line docstring along with the method name in the output it gives when a test fails. This means that a good docstring that really captures the purpose of the test makes it much easier to quickly make sense of output. #### Using self.subTest for independent subtests -Since it's important to make sure all of our tests are independent from each other, you may be tempted to copy-paste your test methods to test a function against a number of different input and output parameters. Luckily, `unittest` provides a better a better of doing that: [`subtests`](https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests). +Another thing that you will probably encounter is that you want to test a function against a list of input and output values. Given the section on writing independent tests, you may now be tempted to copy-paste the same test method over and over again, once for each unique value that you want to test. However, that would result in a lot of duplicate code that is hard to maintain. + -By using the `subTest` context manager, we can perform multiple independent subtests within one test method (e.g., by having a loop for the various inputs we want to test). Using this context manager ensures the test will be run independently and that, if one of them fails, the rest of the subtests are still executed. (Normally, a test function stops once the first exception, including `AssertionError`, is raised.) +Luckily, `unittest` provides a good alternative to that: the [`subTest`](https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests) context manager. It is often used in conjunction with a `for`-loop iterating of a collection of values you want to test your function against and it provides two important features. First, it will make sure that if an assertion statements fails on of the iterations, the other iterations are still run. The other important feature it provides is that it will distinguish your iterations from each other in the output. -An example (taken from [`test_converters.py`](/tests/bot/test_converters.py)): +This is an example of `TestCase.subTest` in action (taken from [`test_converters.py`](/tests/bot/test_converters.py)): ```py def test_tag_content_converter_for_valid(self): @@ -68,14 +75,19 @@ FAIL: test_tag_content_converter_for_valid (tests.bot.test_converters.ConverterT TagContentConverter should return correct values for valid input. ---------------------------------------------------------------------- -# Snipped to save vertical space +# ... ``` ## Mocking -Since we are testing independent "units" of code, we sometimes need to provide "fake" versions of objects generated by code external to the unit we are trying to test to make sure we're truly testing independently from other units of code. We call these "fake objects" mocks. We mainly use the [`unittest.mock`](https://docs.python.org/3/library/unittest.mock.html) module to create these mock objects, but we also have a couple special mock types defined in [`helpers.py`](/tests/helpers.py). +As we are trying to test our "units" of code independently, we want to make sure that we do not rely objects and data generated by "external" code. If we we did, then we wouldn't know if the failure we're observing was caused by the code we are actually trying to test or an external piece of code. -An example of mocking is when we provide a command with a mocked version of `discord.ext.commands.Context` object instead of a real `Context` object and then assert if the `send` method of the mocked version was called with the right message content by the command function: + +However, the features that we are trying to test often depend on those objects generated by other pieces of code. It would be difficult to test a bot command, without having access to a `Context` instance. Fortunately, there's a solution for that: we use fake objects that act like the true object. We call these fake objects "mocks". + +To create these mock object, we mainly use the [`unittest.mock`](https://docs.python.org/3/library/unittest.mock.html) module. In addition, we also defined a couple of specialized mock objects that mock specific `discord.py` types (see the section on the below.). + +An example of mocking is when we provide a command with a mocked version of `discord.ext.commands.Context` object instead of a real `Context` object. This makes sure we can then check (_assert_) if the `send` method of the mocked Context object was called with the correct message content (without having to send a real message to the Discord API!): ```py import asyncio @@ -102,13 +114,13 @@ class BotCogTests(unittest.TestCase): ### Mocking coroutines -By default, `unittest.mock.Mock` and `unittest.mock.MagicMock` can't mock coroutines, since the `__call__` method they provide is synchronous. In anticipation of the `AsyncMock` that will be [introduced in Python 3.8](https://docs.python.org/3.9/whatsnew/3.8.html#unittest), we have added an `AsyncMock` helper to [`helpers.py`](/tests/helpers.py). Do note that this drop-in replacement only implements an asynchronous `__call__` method, not the additional assertions that will come with the new `AsyncMock` type in Python 3.8. +By default, the `unittest.mock.Mock` and `unittest.mock.MagicMock` classes cannot mock coroutines, since the `__call__` method they provide is synchronous. In anticipation of the `AsyncMock` that will be [introduced in Python 3.8](https://docs.python.org/3.9/whatsnew/3.8.html#unittest), we have added an `AsyncMock` helper to [`helpers.py`](/tests/helpers.py). Do note that this drop-in replacement only implements an asynchronous `__call__` method, not the additional assertions that will come with the new `AsyncMock` type in Python 3.8. ### Special mocks for some `discord.py` types To quote Ned Batchelder, Mock objects are "automatic chameleons". This means that they will happily allow the access to any attribute or method and provide a mocked value in return. One downside to this is that if the code you are testing gets the name of the attribute wrong, your mock object will not complain and the test may still pass. -In order to avoid that, we have defined a number of Mock types in [`helpers.py`](/tests/helpers.py) that follow the specifications of the actual Discord types they are mocking. This means that trying to access an attribute or method on a mocked object that does not exist by default on the equivalent `discord.py` object will result in an `AttributeError`. In addition, these mocks have some sensible defaults and **pass `isinstance` checks for the types they are mocking**. +In order to avoid that, we have defined a number of Mock types in [`helpers.py`](/tests/helpers.py) that follow the specifications of the actual Discord types they are mocking. This means that trying to access an attribute or method on a mocked object that does not exist on the equivalent `discord.py` object will result in an `AttributeError`. In addition, these mocks have some sensible defaults and **pass `isinstance` checks for the types they are mocking**. These special mocks are added when they are needed, so if you think it would be sensible to add another one, feel free to propose one in your PR. @@ -193,8 +205,10 @@ FAILED (errors=1) What's more, even if the spelling mistake would not have been there, the first test did not test if the `member_information` function formatted the `member.join` according to the output we actually want to see. +All in all, it's not only important to consider if all statements or branches were touched at least once with a test, but also if they are extensively tested in all situations that may happen in production. + ### Unit Testing vs Integration Testing -Another restriction of unit testing is that it tests in, well, units. Even if we can guarantee that the units work as they should independently, we have no guarantee that they will actually work well together. Even more, while the mocking described above gives us a lot of flexibility in factoring out external code, we work under the implicit assumption of understanding and utilizing those external objects correctly. So, in addition to testing the parts separately, we also need to test if the parts integrate correctly into a single application. +Another restriction of unit testing is that it tests, well, in units. Even if we can guarantee that the units work as they should independently, we have no guarantee that they will actually work well together. Even more, while the mocking described above gives us a lot of flexibility in factoring out external code, we are work under the implicit assumption that we fully understand those external parts and utilize it correctly. What if our mocked `Context` object works with a `send` method, but `discord.py` has changed it to a `send_message` method in a recent update? It could mean our tests are passing, but the code it's testing still doesn't work in production. -We currently have no automated integration tests or functional tests, so **it's still very important to fire up the bot and test the code you've written manually** in addition to the unit tests you've written. +The answer to this is that we also need to make sure that the individual parts come together into a working application. In addition, we will also need to make sure that the application communicates correctly with external applications. Since we currently have no automated integration tests or functional tests, that means **it's still very important to fire up the bot and test the code you've written manually** in addition to the unit tests you've written. |