Regression Test Harness and Hudson Continuous Integration System

Hudson the continuous integration system, has an inherent capability of displaying trends and other historical information of unit tests, provided the xml conforms to the ant's junit task output schema.

jtreg with the option -xml does just that, and thus makes it easy for the Hudson job to generate and render reports of jtreg tests.

Here is how with 2 simple steps

Step 1: In your Hudson job configuration, add a shell or ant script

An example shell script

( java -jar jtreg/lib/jtreg.jar \
  -automatic \
  -xml:verify \
  -agentvm -concurrency:8 \
  -v:err,fail \
  -r:build/jtreg \
  -w:build/jtreg/work \
  -jdk $SDK \
  test/tools/launcher \
  closed/java/awt/SplashScreen \
  test/tools/pack200 || exit 0 )

Note:

-xml:verify will have jtreg generate xml reports, as well as verify the xml conformance of the output files. It is far easier to debug any errors now than later when the same validation is done by Hudson.

jtreg upon encountering a failure will return a non-zero exit value, this will terminate the job, and will prevent data collection by Hudson. Therefore the test is run within a sub-shell guaranteeing a successful exit value. Not to worry, test failures will cause the job to be flagged as "unstable".

Step 2: Data collection

As shown below add the entry build/jtreg/work/**/*.jtr.xml, in the Hudson Job configurations page, this would be the jtreg work directory specified with the -w option in the above script.

job config

Thats it!, after a few builds you should be able to see similar results as shown below, this makes it easy to scan for test failures and drill down to get more information, such as test duration, historical failures, test output etc.

Click on the images below to enlarge