Running tests using jtreg
There are various ways to run the unit and regression tests, varying in convenience and flexibility.
In general, you should use a recent build of jtreg to run the tests in OpenJDK repositories. It is recommended that you use a build corresponding to the most recent "jtreg" tag in the source code repository, as seen here. See the main jtreg page for information on how to build or download the latest version.
- Using the 'test' make target
-
This method is recommended for running tests in any CI system, and for running any tests locally that have special requirements, such as native code libraries.
For detailed instructions on the support for running tests using the JDK build infrastructure, see Testing the JDK, or
doc/testing.[md|html]
in the JDK repo containing the tests you want to run.See Also: OpenJDK Developer's Guide
- Use convenience scripts to run jtreg
-
This method provides the most flexibility for specifying which tests are to be run, and how to run them. It is particularly well suited to running individual tests to help debug an issue in the platform being tested.
There is very little magic in the script, and anything described in this section can be equivalently achieved by running jtreg.jar directly.
In the following list, recommended options are shown with a pale green background. Other options which may be worth consideration are shown with a pale yellow background.
-
To specify which version of jtreg to use, add the appropriate jtreg-dir
/bin
directory to your execution path. -
If you are using jtreg to run tests on a new version of JDK, it is recommended to use a different, stable version of JDK to run jtreg itself. However, unless your new version of JDK is badly broken, it is usually good enough to use the new version of JDK to run jtreg as well.
If you want to explicitly specify the JDK used to run jtreg, set the
JT_JAVA
environment variable before running jtreg. Otherwise, leave it unset, and jtreg will default to using the same version of JDK to run jtreg as is used to execute the tests. -
To execute one or more tests, you must minimally specify the version of JDK to use, and which tests to be run:
% jtreg -jdk:test-jdk test-or-folder...
test-jdk will be used to run the tests in the specified files and/or folders. It will also be used to run jtreg unless you have set the
JT_JAVA
environment variable, as described above. -
jtreg uses two directories in which to store results, creating them if necessary. The work directory is used to store the compiled classes for tests, temporary files while tests are running, and the detailed results of each test. The report directory is used to write HTML and plain text reports at the end of the test run.
By default, these directories are named
JTwork
andJTreport
, and are placed in the current directory used when jtreg is run.It is common to execute jtreg in top level directory of the repository in which you are working. It is also common to place the work and report directories in a subdirectory of the main build directory, using the
-w
and-r
options. For example,% jtreg -w build-dir/jtreg/work -r build-dir/jtreg/report ...
-
To vary the amount of output written to the console for each test, use the
-verbose:arg
option. -
To enable assertions while running tests, use the standard JVM options
-ea:arg
and-esa
, or their equivalents. -
Some tests in the test suite may have problems such that the tests are currently deemed dysfunctional or invalid, and have therefore been disabled. This may be done in two ways.
A test may be directly labelled with
@ignore
. By default, these tests will cause an error if you try to run them. To avoid running them, use the-ignore:quiet
option. (You can also ignore the@ignore
directives by using the-ignore:run
option, but this is not common.)Tests may also be listed in a problem list or exclude list file. This is a more flexible mechanism than
@ignore
and allows the test to be listed as problematic on specific platforms. To avoid running these tests, use the-exclude:file
option. -
By default, jtreg runs each test in its own JVM. While this provides maximum isolation between tests, it also incurs maximum overhead. jtreg provides different modes for running tests, trading off isolation for performance. Many groups of tests can be run in the faster modes, and work is ongoing to upgrade additional tests. In general, it is recommended to use
-agentvm
to run the tests in "agentvm" mode. See jtreg: old and new for a blog entry giving details of the different execution modes. -
Some tests, typically GUI tests, may require manual interaction in order to determine whether they pass or fail. To run just those tests, use the
-m
or-manual
option. To avoid running such tests, use the-a
or-automatic
option. -
The tests in some parts of the test suite may be able to run concurrently. This feature is only available in the "othervm" and "agentvm" modes. To have jtreg run tests concurrently, use the
-conc:value
option, wherevalue
is either the maximum number of tests that should be allowed to run concurrently or the word "auto
" to set the value to the number of processors on your system. Make sure you have enough memory and/or swap space when you use this option. For more information, see Running tests concurrently with jtreg. -
If you have built a component of JDK such as the compiler (javac), you may be able to test it in conjunction with a complete JDK by putting the classes or jar file for new component on the system bootstrap classpath:
On JDK 8 and earlier versions of JDK, you can use:
% jtreg -jdk:jdk -Xbootclasspath/p:component.jar test-or-folder...
On JDK 9 and later versions of JDK, you can use the
--system
and--patch-module
options instead of-Xbootclasspath
. -
You can rerun any selection of tests. Use the
-status:arg
option to filter the set of tests to be run, based on their prior result status. For example, to rerun tests which previously failed or could not be run:% jtreg -jdk:jdk -status:fail,error test-or-folder...
-
For info on all the available options, use the
-help
option. For info on a specific option or topic, use-help option-or-word...
To read the online help, use the-onlineHelp
command line option, or start jtreg with the-gui
option, and use theHelp
menu. -
Here is an example that puts many of the preceding ideas together. JDK 17 is used to run jtreg, while jtreg uses JDK 18 to run the
java/lang/Class
tests in thejdk
repository located at/w/jjg/work/tl/jdk
. The tests are executed using the "agentvm" execution mode; a one line summary for each test is written as each test completes, and the overall results are stored in the local build directory.export JT_JAVA=/opt/jdk/17 cd /w/jjg/work/jdk /opt/jtreg/6.1/linux/bin/jtreg \ -jdk:/opt/jdk/18 -agentvm -verbose:summary \ -w build/jtreg/work -r build/jtreg/report \ test/java/lang/Class
-
- Run jtreg.jar directly
-
Instead of using the startup scripts provided, you can also execute the
jtreg.jar
file directly, or you can run jtreg from Ant. See the online help for details.
Test Results
Summary reports are generated in the report directory, in
plain text and HTML formats. To generate a detailed report in XML
as well, use the -xml
option.
- The plain text report in
report-dir/text/summary.txt
contains one test result per line and is well suited for processing with tools likegrep
. - The XML report can be used with CI systems like Hudson to generate trend graphs. See Regression Test Harness and Hudson Continuous Integration System on how to do this.
Detailed results for each test are stored in the work
directory, in .jtr
files. These are plain text
files that are easy enough to read directly if needed.
Individual tests can be in one of four states.
- Not run: the test has not yet been run.
- Passed: the test has been run successfully.
- Failed: the test has been run, but did not succeed.
- Error: the test could not be run for some reason.