vtkModuleTesting#

VTK uses the ExternalData CMake module to handle the data management for its test suite. Test data is only downloaded when a test which requires it is enabled and it is cached so that every build does not need to redownload the same data.

To facilitate this workflow, there are a number of CMake functions available in order to indicate that test data is required.

Loading data#

vtk_module_test_data#

Download test data. module

Data may be downloaded manually using this function:

vtk_module_test_data(<PATHSPEC>...)

This will download data inside of the input data directory for the modules being built at that time (see the TEST_INPUT_DATA_DIRECTORY argument of vtk_module_build).

For supported PATHSPEC syntax, see the associated documentation in ref:ExternalData. These arguments are already wrapped in the DATA{} syntax and are assumed to be relative paths from the input data directory.

Creating test executables#

vtk_module_test_executable#

This function creates an executable from the list of sources passed to it. It is automatically linked to the module the tests are intended for as well as any declared test dependencies of the module.

vtk_module_test_executable(<NAME> <SOURCE>...)

This function is not usually used directly, but instead through the other convenience functions.

Test name parsing#

Test names default to using the basename of the filename which contains the test. Two tests may share the same file by prefixing with a custom name for the test and a comma.

The two parsed syntaxes are: - CustomTestName,TestFile - TestFile

Note that TestFile should already have had its extension stripped (usually done by _vtk_test_parse_args).

In general, the name of a test will be <EXENAME>-<TESTNAME>, however, by setting vtk_test_prefix, the test name will instead be <EXENAME>-<PREFIX><TESTNAME>.

Test function arguments#

Each test is specified using one of the two following syntaxes

  • <NAME>.<SOURCE_EXT>

  • <NAME>.<SOURCE_EXT>,<OPTIONS>

Where NAME is a valid test name. If present, the specified OPTIONS are only for the associated test. The expected extension is specified by the associated test function.

_vtk_test_parse_args#

module-internal Given a list of valid “options”, this function will parse out a the following variables:

  • args: Unrecognized arguments. These should be interpreted as arguments that should be passed on the command line to all tests in this parse group.

  • options: Options specified globally (for all tests in this group).

  • names: A list containing all named tests. These should be parsed by _vtk_test_parse_name.

  • _<NAME>_options: Options specific to a certain test.

_vtk_test_parse_args(<OPTIONS> <SOURCE_EXT> <ARG>...)

In order to be recognized as a source file, the SOURCE_EXT must be used. Without it, all non-option arguments are placed into args. Each test is parsed out matching these:

_vtk_test_set_options#

For handling global option settings module-internal, this function sets variables in the calling scoped named <PREFIX><OPTION> to either 0 or 1 if the option is present in the remaining argument list.

_vtk_test_set_options(<OPTIONS> <PREFIX> <ARG>...)

Additionally, a non-0 default for a given option may be specified by a variable with the same name as the option and specifying a prefix for the output variables.

C++ tests#

vtk_add_test_cxx#

This function declares C++ tests module. Source files are required to use the cxx extension.

vtk_add_test_cxx(<EXENAME> <VARNAME> <ARG>...)

Each argument should be either an option, a test specification, or it is passed as flags to all tests declared in the group. The list of tests is set in the <VARNAME> variable in the calling scope.

Options:

  • NO_DATA: The test does not need to know the test input data directory. If it does, it is passed on the command line via the -D flag.

  • NO_VALID: The test does not have a valid baseline image. If it does, the baseline is assumed to be in ../Data/Baseline/<NAME>.png relative to the current source directory. If alternate baseline images are required, <NAME> may be suffixed by _1, _2, etc. The valid image is passed via the -V flag. - TIGHT_VALID: Uses euclidian type metrics to compare baselines. Baseline comparison is sensitive to outliers in this setting. - LOOSE_VALID: Uses L1 type metrics to compare baselines. Baseline comparison is somewhat more forgiving. Typical use cases involve rendering that is highly GPU dependent, and baselines with text. - LEGACY_VALID: Uses legacy image compare. This metric generates a lot of false negatives. It is recommended not to use it.

  • NO_OUTPUT: The test does not need to write out any data to the filesystem. If it does, a directory which may be written to is passed via the -T flag.

Additional flags may be passed to tests using the ${_vtk_build_test}_ARGS variable or the <NAME>_ARGS variable.

MPI tests#

vtk_add_test_mpi#

This function declares C++ tests which should be run under an MPI environment. module Source files are required to use the cxx extension.

vtk_add_test_mpi(<EXENAME> <VARNAME> <ARG>...)

Each argument should be either an option, a test specification, or it is passed as flags to all tests declared in the group. The list of tests is set in the <VARNAME> variable in the calling scope.

Options:

  • TESTING_DATA

  • NO_VALID: The test does not have a valid baseline image. If it does, the baseline is assumed to be in ../Data/Baseline/<NAME>.png relative to the current source directory. If alternate baseline images are required, <NAME> may be suffixed by _1, _2, etc. The valid image is passed via the -V flag.

Each test is run using the number of processors specified by the following variables (using the first one which is set):

  • <NAME>_NUMPROCS

  • <EXENAME>_NUMPROCS

  • VTK_MPI_NUMPROCS (defaults to 2)

Additional flags may be passed to tests using the ${_vtk_build_test}_ARGS variable or the <NAME>_ARGS variable.

C++ test executable#

vtk_test_cxx_executable#
vtk_test_cxx_executable(<EXENAME> <VARNAME> [RENDERING_FACTORY] [<SRC>...])

Creates an executable named EXENAME which contains the tests listed in the variable named in the VARNAME argument. The EXENAME must match the EXENAME passed to the test declarations when building the list of tests.

If RENDERING_FACTORY is provided, VTK’s rendering factories are initialized during the test.

By default, VTK’s rendering tests enable FP exceptions to find floating point errors in debug builds. If DISABLE_FLOATING_POINT_EXCEPTIONS is provided, FP exceptions are not enabled for the test. This is useful when testing against external libraries to ignore exceptions in third-party code.

Any additional arguments are added as additional sources for the executable.

vtk_test_mpi_executable#

MPI executables used to have their own test executable function.|module-internal| This is no longer necessary and is deprecated. Instead, vtk_test_cxx_executable should be used instead.

Python tests#

vtk_add_test_python#

This function declares Python tests.|module| Test files are required to use the py extension.

vtk_add_test_python(<EXENAME> <VARNAME> <ARG>...)

If the _vtk_testing_python_exe variable is not set, the vtkpython binary is used by default. Additional arguments may be passed in this variable as well.

If the given Python executable supports VTK’s --enable-bt flag, setting _vtk_testing_python_exe_supports_bt to 1 is required to enable it.

JavaScript tests#

vtk_add_test_module_javascript_node#

This function declares JavaScript tests run with NodeJS. Test files are required to use the mjs extension. Additional arguments to node can be passed via _vtk_node_args variable.

vtk_add_test_module_javascript_node(<VARNAME> <ARG>...)

The _vtk_testing_nodejs_exe variable must point to the path of a node interpreter.

MPI tests#

vtk_add_test_python_mpi#

A small wrapper around vtk_add_test_python which adds support for running MPI-aware tests written in Python.

The $<module library name>_NUMPROCS variable may be used to use a non-default number of processors for a test.

This forces running with the pvtkpython executable.

ABI Mangling tests#

vtk_add_test_mangling#

This function declares a test to verify that all of the exported symbols in the VTK module library contain the correct ABI mangling prefix. This test requires setting the option VTK_ABI_NAMESPACE_NAME to a value that is not “<DEFAULT>”.

Current limitations of this test are: - Does not run on non-UNIX platforms - Is not compatible with the option “VTK_ENABLE_KITS” - May not work outside of VTK itself

vtk_add_test_mangling(module_name [EXEMPTIONS ...])

Options: - EXEMPTIONS: List of symbol patterns to excluded from the ABI mangling test

where it is known that the symbols do not support the ABI mangling but are still exported. This option should be extremely rare to use, see the documentation on ABI mangling for how the handle C and C++ symbols before adding an EXEMPTION.