Test coverage in Tcl

Introduction

The Tcl script presented here enables the user to register the test coverage of Tcl scripts that have been properly prepared.

Knowing the test coverage is important because:

It enables you to see what parts of the application have not been fully exercised.
It enables you to define in a systematic manner new test that can take care of the blank spots in your current test suite.

To be precise: Testcov, the application described here, is limited to statement and branch coverage, as these are the easiest to detect. They are also the easiest to satisfy, but they do represent a very important class of testing strategies. In fact, unless most or (preferrably all) of the code is exercised, chances remain great that some horrific bug is lurking somewhere.

For more information: B. Beizer, Software Testing Techniques, 1992 gives an excellent overview.

The remainder of this document covers the following subjects:

VERSION AND COPYRIGHT
USING TESTCOV TO OBTAIN TEST COVERAGE REPORTS
IMPLEMENTATION NOTES
LIMITATIONS OF THE PRESENT VERSION

Version and copyright

This document describes Testcov, version 0.3, july 2001.

Usage of Testcov and AwkProc is free, as long as you acknowledge the author, Arjen Markus (e-mail: arjen.markus@wldelft.nl).

There is no guarantee nor claim that the results are accurate.

Using Testcov to obtain test coverage reports

Using Testcov is simple:

Copy the files "testcov.tcl" and "awkproc.tcl" to the directory containing your Tcl sources.
Run "testcov.tcl" with the option "-inst" on all relevant sources. This will create versions with an extension like ".tcl_inst", these are the instrumented versions.
Run the application. At the end (when executing "exit") the file "testcov.sav" is written to the then current directory.
Run "testcov.tcl" again, now with the option "-rep". This will produce a report file, "testcov.report" with the annotated source.

Consider the following example (in example.tcl):


proc SumOf { args } {
   set sum 0.0

   foreach value $args {
      set sum [expr $sum+$value]
   }

   return $sum
}

set sum [SumOf 1 2 3 4]

if { $sum < 0.0 } {
   puts "Sum is negative!"
}
exit 0

Running Testcov means, executing the following commands:


# Instrument the script
tclsh testcov.tcl -inst example.tcl
# Run the instrumented version
tclsh example.tcl_inst
# Report the coverage
tclsh testcov.tcl -rep example.tcl

The result is this:


Test coverage report
--------------------

File: example.tcl

   1: proc SumOf { args } {
Note: Procedure never called with zero extra arguments
Note: Procedure never called with one extra argument

         set sum 0.0

   4:    foreach value $args {
Note: No zero-iterations

            set sum [expr $sum+$value]
         }

         return $sum
      }

      set sum [SumOf 1 2 3 4]

   1: if { $sum < 0.0 } {
Note: Primary branch not executed

         puts "Sum is negative!"
      }
      exit 0

You should note the following:

The procedure has the special argument "args". This means that the number of arguments can be variable. Special cases include: no extra arguments, one extra argument, more than one.
A loop has an implicit alternative case, that is, it may be run zero times (in which case, variables may not be set, files may not opened etc.)
The same goes for if-statements without an else.
The numbers are the number of times the statement is executed or the procedure called.

Implementation notes

The instrumentation of Tcl source files works by inserting a number of statements into the existing code. Notably, the file "testcov.tcl" will be sourced at the start, so that the auxiliary routines (__cov_*) are defined. Also two ordinary commands, exit and source are redefined:

Calling the redefined exit command results in the file "testcov.sav" being written. So it is vital that the script exits via exit and not simply runs out of things to do.
Calling the redefined source command brings in an instrumented version of the file that is sourced, if one exists and is newer tghan the original.

Limitations of the present version

Testcov has a number of limitations that you should be aware of:

In many scripting languages you can produce code "on the fly". Such code can not be instrumented with a simple tool like Testcov. The facility may even present fundamental difficulties. (Another construct that defies the static analysis is of course "uplevel".)
The patterns that Testcov uses are somewhat tuned to the style of programming that I prefer. They may or may not suit your style. If not, then it is possible that the patterns simply do not match.
For instance:
```
       if { $value > 0 } { puts $value }
       
```
is not recognised because the line does not end in an open-brace. (There is also a banal reason for this: writing the instrumented code would be more complicated!)
Testcov does not recognise the keywords "elseif" and "switch". The reason for this is purely practical: the subset it does recognise allows a much simpler analysis. Recognising "elseif" and "case" means that the reported line numbers have to be related to the correct if-statement and "switch".
It does report this, however, with question marks at the line containing "elseif" or "switch", so that you are warned about this limitation.
I have not created a fully self-contained application yet that can be installed without any fuss.
There is no way to obtain the combined test coverage of several runs.
Testcov relies for its implementation on another package, AwkProc, that simulates the well-known AWK scripting facility. This package is not documented yet, other than by the comments in the source code.

Despite its severe limitations, Testcov can be quite useful:

It is an all-Tcl solution, hence as portable as any script.
It is fairly fast, as it monitors at the script level and only those lines that need monitoring.