Testing

Testing is an integral part of developing software and validating the code. It is also builds trust in users of the software that the software will work as expected. When adding new code to the library itself a test must also be added. Ideally the tests test every single line of code so that we have 100% code coverage. When we have good coverage of tests over the library we inherently get regression testing. We don’t want to have new code changes breaking code that is already considered to be working.

For the testing of the Fortran code the pFUnit testing framework has been chosen. The pFUnit testing framework uses Python to manage some of the test generation, without Python we cannot build the tests.

How to add a test

All tests live under the tests tree and mirror what is in the source tree. In the following example we are going to add a new testing module for the diagnostics module in the lib directory from the src tree.

Write test

To start we are first going to make sure we have the correct structure that matches the src tree. Starting from the root directory of the lungsim repository we need to make sure that the directory:

tests/lib

exists and if not create it, from the command line on UNIX based oses this can be done with the mkdir command:

mkdir tests/lib

Once the directory structure is correct we then create the testing module. Because we want to test the diagnostics module from the library we will create a test file named test_diagnostics.pf in the tests/lib directory. The pf extension indicates that this file is a hybrid Python fortran file, this file is a preprocessor input file which is Fortran free format file with preprocessor directives added. To create the test a Python script will generate a valid Fortran file from directives written into this file. With your favourite text editor create a file named test_diagnostics.pf. We could choose vi for this task as shown below but any text editor will work:

vi tests/lib/test_diagnostics.pf

Into this file we will write our first test for the module. This test will check that the diagnositcs flag has been set when using the set_diagnostics_on subroutine:

@test
subroutine testSetDiagnostics()
   use pfunit_mod
   use diagnostics, only: get_diagnostics_on, set_diagnostics_on
   implicit none

   logical :: state

   call get_diagnostics_on(state)
   @assertFalse(state)
   call set_diagnostics_on(.true.)
   call get_diagnostics_on(state)
   @assertTrue(state)

end subroutine testSetDiagnostics

With our test written we now need to add this into the CMake build generation system.

Add test to CMake

The first task to do when adding a test to the CMake files is to check that a CMake file exists. When adding a test to a new directory, as we are doing here, there won’t be a CMake file for us to use. To fix this we first need to tell CMake that a new subdirectory is available. We do this by adding a sub_directory command into an existing CMakeLists.txt file in a parent directory of the directory we have just added a test to. In our example we would edit the file (any text editor will do, don’t feel you need to use vi):

vi tests/CMakeLists.txt

and add the line at the bottom of the file:

add_subdirectory(lib)

Then we need to create a new CMakeLists.txt (the capitalisation of this file is important) file in the tests/lib directory (any text editor will do, don’t feel you need to use vi):

vi tests/lib/CMakeLists.txt

and add the following to create an executable test that will work with CTest (we will also be able to execute this test directly):

# Add all the files that make a single test, we could have multiple files testing
# the same module.  Don't add test files into the same test that test different modules.
# These are all .pf files.
set(DIAGNOSTICS_TEST_SRCS
    test_diagnostics.pf)

# Make use of the pFUnit helper function to create a test.
# Arguments    : - test_package_name: Name of the test package
#                - test_sources     : List of pf-files to be compiled
#                - extra_sources    : List of extra Fortran source code used for testing (if none, input empty string "")
#                - extra_sources_c  : List of extra C/C++ source code used for testing (if none, input empty string "")
add_pfunit_test(diagnostics_test ${DIAGNOSTICS_TEST_SRCS} "" "")
# Link the test to the aether library target.
target_link_libraries (diagnostics_test aether)
target_include_directories(diagnostics_test PRIVATE $<TARGET_PROPERTY:aether,Fortran_MODULE_DIRECTORY>)

With our test added to the test framework we can now build and run our test.

Build and run test

The test we have just completed will be built when we build the configuration from the build directory by default. That is if we execute the BUILD_ALL build target for IDEs like Visual Studio or on Makefile generation builds we would simple issue the command make in the build directory. We can also build our test directly be building the target diagnostics_test, for Makefile generation builds we would issue the command:

make diagnostics_test

To run the test we can execute the ctest command from the command line in the build directory with the following arguments:

ctest -R diagnostics_test

we will also execute all tests if we execute the command:

ctest

A handy flag to add to both of these commands is the –verbose flag. This gives us the details output from each test and not just the summary statement.