Getting Test Coverage Information on SystemTap Will Cohen Aug 16, 2005 When developing software it is very useful to determine which sections of the program the tests actually exercise. Having this coverage information can show which sections of the code are being missed by the tests. The GCC compiler has options to compile code to produce this coverage information when the code is executed. The gcov and lcov tools can analyze this information and produce either text files or html that the developers can analyze to determine whether the current tests are sufficient or whether additional tests are needed to coverage areas of the program that haven't run. To enable collection of coverage information the following GCC options are used to compile code: -g -fprofile-arcs -ftest-coverage Code Coverage Tests of SystemTap First need to build a version of SystemTap with coverage turned on. Assuming you are building the SystemTap translator in /tmp/obj and the source is in /tmp/src to get a executable that generates coverage data run the following commands: cd /tmp/obj ../src/configure make clean CXXFLAGS="-g -fprofile-arcs -ftest-coverage" all To clear out coverage information use the following command in the build (/tmp/obj) directory: lcov --directory . -z To run the pass 1-4 tests (test translator and compile code generated by translator, but not actually load instrumentation into kernel): make check To run all tests (pass 1-5) include loading instrumentation into kernel: make installcheck The "make check" and "make installcheck" will produce a bunch of .gcda files containing the coverage information. For example after the "make check" the following files were produced: -rw-r--r-- 1 wcohen wcohen 7952 Aug 16 17:37 stap-buildrun.gcda -rw-r--r-- 1 wcohen wcohen 84728 Aug 16 17:37 stap-elaborate.gcda -rw-r--r-- 1 wcohen wcohen 21680 Aug 16 17:37 stap-main.gcda -rw-r--r-- 1 wcohen wcohen 72256 Aug 16 17:37 stap-parse.gcda -rw-r--r-- 1 wcohen wcohen 71296 Aug 16 17:37 stap-staptree.gcda -rw-r--r-- 1 wcohen wcohen 114968 Aug 16 17:37 stap-tapsets.gcda -rw-r--r-- 1 wcohen wcohen 87440 Aug 16 17:37 stap-translate.gcda The Linux Test Project (LTP) developed some scripts to convert the gcov data into something viewable on web pages. The scripts can either be obtained from the Fedora Extras or built and installed from a RPM listed in the References section. The lcov uses the same data as the gcov, but renders it into HTML files . A web browser such as Firefox can be use navigate it and examine the data. The main advantages of the LCOV information over GCOV are: -LCOV gives summaries listing the amount of code covered in each file. -LCOV makes it easy to scan for unexecuted code (highlighted in red). -LCOV results can be posted on webpage and navigated by web browser. The following list of commands will take the .gcda files in /tmp/obj and create a web browserable directory /tmp/stap-combined-tests-${DATE_TIME} and a matching compressed tarball with the coverage information generated by LCOV: export STAP_OBJ=/tmp/obj export DATE_TIME=`date -u +%Y%m%d%H%M` export COVERAGE_DIR=/tmp/stap-combined-tests-${DATE_TIME} export COVERAGE_PARENT=`dirname ${COVERAGE_DIR}` export COVERAGE_BASENAME=`basename ${COVERAGE_DIR}` pushd ${COVERAGE_PARENT} mkdir -p ${COVERAGE_DIR} lcov --directory $STAP_OBJ --capture --output-file \ ${COVERAGE_DIR}/stap.info genhtml -o ${COVERAGE_DIR} ${COVERAGE_DIR}/stap.info tar cfz ${COVERAGE_BASENAME}.tar.gz ${COVERAGE_BASENAME} popd The resulting directory can be navigated with a web browser to look at the test coverage information. The src subdirectory will contain the coverage information for the SystemTap translator. Note that there is a script stap_testing in the SystemTap CVS tests module's tools subdirectory that automates the checkout, build, and testing of the currect SystemTap CVS sources. One can look at this script as an additional reference. References The LTP GCOV extension (lcov), Jan 2006. http://ltp.sourceforge.net/coverage/lcov.php LCOV RPM for Fedora, http://download.fedora.redhat.com/pub/fedora/linux/extras/development/SRPMS/lcov-1.4-2.fc5.src.rpm