[cmake-developers] Unit testing CMake modules

[Wouter Klouwen]

Hi all,

We have a rather large amount of CMake code (4k+ lines) in various
modules/functions. They contain the common logic for many of our
projects. This makes it quite an important base for everything our team
does.

Rather shamefully at present these modules are rather poorly tested. I'd
like to improve this, but the current way of testing CMake code
is typically to run a trimmed project, and to verify whether certain
invocations produce a certain output or file hierarchy.
This involves a bit of infrastructure and can be a bit cumbersome
maintain, to diagnose when tests fail, and it requires a separate run of
the tests.
The overall cumbersomeness of the setup in turn discourages in our team,
including myself, from adding tests.

I'd like a more integrated approach that makes running at least some
basic tests part of the build progress, and a more direct way of
reporting failures.

In other programming environments, testing often involves some kind of
mocking environment, and CMake helpfully allows the overriding of CMake
built in functions, though this is typically discouraged.

In my ideal world it would be possible to save the state of the current
function set up, then call a function with a certain given number of
parameters, and expect a certain sequence of events, such as functions
to be called, environment variables to be set, etc.
Something akin to CMakePushCheckState, except for built in functions.

Then a module could provide for some functions that would set up
expectations and verify them, within a run of cmake, or possibly some
other commands could be added, to give some syntactic glossy coat to it.

As it wouldn't actually trigger any of the expensive generating
functions, it would be lightweight, quick to run and give pretty direct
errors in terms of failed expectations, reducing debug time.

If it was done with CMake commands, I might imagine it to look something
like:

function(foobar)
# function to test, does something non trivial
if ("FOO" IN_LIST ARGV)
install(FILES foo DESTINATION foo_dir)
else("BAR" IN_LIST ARGV)
message(FATAL_ERROR "Some error")
else("BAZ" IN_LIST ARGV)
set(BAZ True PARENT_SCOPE)
endif()
endfunction()

test(foobar)
expect(WITH "FOO" CALL install FILES foo DESTINATION foo_dir)
expect(WITH "BAR" CALL message FATAL_ERROR "Some error")
expect(WITH "BAZ" ENVIRONMENT BAZ True)
endtest(foobar)

What do people think? Is this crazy? Is there a quicker way to get
somewhere close? Should I put some effort into making this into an
actual proposal/working code?

Thanks in advance,
W


This transmission contains information that may be confidential and contain personal views which are not necessarily those of YouView TV Ltd. YouView TV Ltd (Co No:7308805) is a limited liability company registered in England and Wales with its registered address at YouView TV Ltd, 3rd Floor, 10 Lower Thames Street, London, EC3R 6YT. For details see our web site at http://www.youview.com
--
Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

[Stuermer, Michael SP/HZA-ZSEP]

Hello Wouter,

testing CMake code is indeed very important. This is why kitware does it as well. Did you check the CMake code testing infrastructure in "Tests/RunCMake" in the sources? It is a very flexible concept which makes adding tests easy enough for everyone contribute (IMO). At the beginning it might be a bit confusing how expected results etc. are handled, but once you understand how it works it's really nice.

Another (maybe the main) advantage: you have tons of examples and the RunCMake infrastructure is maintained so you don't have to it all on your own.

best regards,
Michael

-----Ursprüngliche Nachricht-----
Auftrag von Wouter Klouwen
Gesendet: Dienstag, 29. Mai 2018 18:36
An: CMake Developers
Betreff: [cmake-developers] Unit testing CMake modules
Hi all,
We have a rather large amount of CMake code (4k+ lines) in various
modules/functions. They contain the common logic for many of our projects.
This makes it quite an important base for everything our team does.
Rather shamefully at present these modules are rather poorly tested. I'd like
to improve this, but the current way of testing CMake code is typically to run
a trimmed project, and to verify whether certain invocations produce a
certain output or file hierarchy.
This involves a bit of infrastructure and can be a bit cumbersome maintain, to
diagnose when tests fail, and it requires a separate run of the tests.
The overall cumbersomeness of the setup in turn discourages in our team,
including myself, from adding tests.
I'd like a more integrated approach that makes running at least some basic
tests part of the build progress, and a more direct way of reporting failures.
In other programming environments, testing often involves some kind of
mocking environment, and CMake helpfully allows the overriding of CMake
built in functions, though this is typically discouraged.
In my ideal world it would be possible to save the state of the current
function set up, then call a function with a certain given number of
parameters, and expect a certain sequence of events, such as functions to
be called, environment variables to be set, etc.
Something akin to CMakePushCheckState, except for built in functions.
Then a module could provide for some functions that would set up
expectations and verify them, within a run of cmake, or possibly some other
commands could be added, to give some syntactic glossy coat to it.
As it wouldn't actually trigger any of the expensive generating functions, it
would be lightweight, quick to run and give pretty direct errors in terms of
failed expectations, reducing debug time.
If it was done with CMake commands, I might imagine it to look something
function(foobar)
# function to test, does something non trivial
if ("FOO" IN_LIST ARGV)
install(FILES foo DESTINATION foo_dir)
else("BAR" IN_LIST ARGV)
message(FATAL_ERROR "Some error")
else("BAZ" IN_LIST ARGV)
set(BAZ True PARENT_SCOPE)
endif()
endfunction()
test(foobar)
expect(WITH "FOO" CALL install FILES foo DESTINATION foo_dir)
expect(WITH "BAR" CALL message FATAL_ERROR "Some error")
expect(WITH "BAZ" ENVIRONMENT BAZ True)
endtest(foobar)
What do people think? Is this crazy? Is there a quicker way to get somewhere
close? Should I put some effort into making this into an actual
proposal/working code?
Thanks in advance,
W
This transmission contains information that may be confidential and contain
personal views which are not necessarily those of YouView TV Ltd. YouView
TV Ltd (Co No:7308805) is a limited liability company registered in England and
Wales with its registered address at YouView TV Ltd, 3rd Floor, 10 Lower
Thames Street, London, EC3R 6YT. For details see our web site at
http://www.youview.com
--
Powered by www.kitware.com
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
https://cmake.org/mailman/listinfo/cmake-developers

--
Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

[Sam Spilsbury]

<self-promotion>

A little while ago I wrote a framework to do just that:
https://github.com/polysquare/cmake-unit

I haven't maintained it in a few years, but it works exactly the way
you would expect it to (the syntax is a bit different though).

</self-promotion>

On Wed, May 30, 2018 at 12:36 AM, Wouter Klouwen

Post by Wouter Klouwen
Hi all,
We have a rather large amount of CMake code (4k+ lines) in various
modules/functions. They contain the common logic for many of our
projects. This makes it quite an important base for everything our team
does.
Rather shamefully at present these modules are rather poorly tested. I'd
like to improve this, but the current way of testing CMake code
is typically to run a trimmed project, and to verify whether certain
invocations produce a certain output or file hierarchy.
This involves a bit of infrastructure and can be a bit cumbersome
maintain, to diagnose when tests fail, and it requires a separate run of
the tests.
The overall cumbersomeness of the setup in turn discourages in our team,
including myself, from adding tests.
I'd like a more integrated approach that makes running at least some
basic tests part of the build progress, and a more direct way of
reporting failures.
In other programming environments, testing often involves some kind of
mocking environment, and CMake helpfully allows the overriding of CMake
built in functions, though this is typically discouraged.
In my ideal world it would be possible to save the state of the current
function set up, then call a function with a certain given number of
parameters, and expect a certain sequence of events, such as functions
to be called, environment variables to be set, etc.
Something akin to CMakePushCheckState, except for built in functions.
Then a module could provide for some functions that would set up
expectations and verify them, within a run of cmake, or possibly some
other commands could be added, to give some syntactic glossy coat to it.
As it wouldn't actually trigger any of the expensive generating
functions, it would be lightweight, quick to run and give pretty direct
errors in terms of failed expectations, reducing debug time.
If it was done with CMake commands, I might imagine it to look something
function(foobar)
# function to test, does something non trivial
if ("FOO" IN_LIST ARGV)
install(FILES foo DESTINATION foo_dir)
else("BAR" IN_LIST ARGV)
message(FATAL_ERROR "Some error")
else("BAZ" IN_LIST ARGV)
set(BAZ True PARENT_SCOPE)
endif()
endfunction()
test(foobar)
expect(WITH "FOO" CALL install FILES foo DESTINATION foo_dir)
expect(WITH "BAR" CALL message FATAL_ERROR "Some error")
expect(WITH "BAZ" ENVIRONMENT BAZ True)
endtest(foobar)
What do people think? Is this crazy? Is there a quicker way to get
somewhere close? Should I put some effort into making this into an
actual proposal/working code?
Thanks in advance,
W
This transmission contains information that may be confidential and contain personal views which are not necessarily those of YouView TV Ltd. YouView TV Ltd (Co No:7308805) is a limited liability company registered in England and Wales with its registered address at YouView TV Ltd, 3rd Floor, 10 Lower Thames Street, London, EC3R 6YT. For details see our web site at http://www.youview.com
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
https://cmake.org/mailman/listinfo/cmake-developers

--
Sam Spilsbury
pgp.mit.edu: 0xb8a90fb599bae9c2
--
Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers