Running tests using jtreg

There are various ways to run the unit and regression tests, varying in convenience and flexibility.

In general, you should use a recent build of jtreg to run the tests in OpenJDK repositories. It is recommended that you use a build corresponding to the most recent "jtreg" tag in the source code repository, as seen here. See the main jtreg page for information on how to build or download the latest version.

Using the 'test' make target

This method is recommended for running tests in any CI system, and for running any tests locally that have special requirements, such as native code libraries.

For detailed instructions on the support for running tests using the JDK build infrastructure, see Testing the JDK, or doc/testing.[md|html] in the JDK repo containing the tests you want to run.

See Also: OpenJDK Developer's Guide

Use convenience scripts to run jtreg

This method provides the most flexibility for specifying which tests are to be run, and how to run them. It is particularly well suited to running individual tests to help debug an issue in the platform being tested.

There is very little magic in the script, and anything described in this section can be equivalently achieved by running jtreg.jar directly.

In the following list, recommended options are shown with a pale green background. Other options which may be worth consideration are shown with a pale yellow background.

  • To specify which version of jtreg to use, add the appropriate jtreg-dir/bin directory to your execution path.

  • If you are using jtreg to run tests on a new version of JDK, it is recommended to use a different, stable version of JDK to run jtreg itself. However, unless your new version of JDK is badly broken, it is usually good enough to use the new version of JDK to run jtreg as well.

    If you want to explicitly specify the JDK used to run jtreg, set the JT_JAVA environment variable before running jtreg. Otherwise, leave it unset, and jtreg will default to using the same version of JDK to run jtreg as is used to execute the tests.

  • To execute one or more tests, you must minimally specify the version of JDK to use, and which tests to be run:

    % jtreg -jdk:test-jdk test-or-folder...

    test-jdk will be used to run the tests in the specified files and/or folders. It will also be used to run jtreg unless you have set the JT_JAVA environment variable, as described above.

  • jtreg uses two directories in which to store results, creating them if necessary. The work directory is used to store the compiled classes for tests, temporary files while tests are running, and the detailed results of each test. The report directory is used to write HTML and plain text reports at the end of the test run.

    By default, these directories are named JTwork and JTreport, and are placed in the current directory used when jtreg is run.

    It is common to execute jtreg in top level directory of the repository in which you are working. It is also common to place the work and report directories in a subdirectory of the main build directory, using the -w and -r options. For example,

    % jtreg -w build-dir/jtreg/work -r build-dir/jtreg/report ...

  • To vary the amount of output written to the console for each test, use the -verbose:arg option.

  • To enable assertions while running tests, use the standard JVM options -ea:arg and -esa, or their equivalents.

  • Some tests in the test suite may have problems such that the tests are currently deemed dysfunctional or invalid, and have therefore been disabled. This may be done in two ways.

    A test may be directly labelled with @ignore. By default, these tests will cause an error if you try to run them. To avoid running them, use the -ignore:quiet option. (You can also ignore the @ignore directives by using the -ignore:run option, but this is not common.)

    Tests may also be listed in a problem list or exclude list file. This is a more flexible mechanism than @ignore and allows the test to be listed as problematic on specific platforms. To avoid running these tests, use the -exclude:file option.

  • By default, jtreg runs each test in its own JVM. While this provides maximum isolation between tests, it also incurs maximum overhead. jtreg provides different modes for running tests, trading off isolation for performance. Many groups of tests can be run in the faster modes, and work is ongoing to upgrade additional tests. In general, it is recommended to use -agentvm to run the tests in "agentvm" mode. See jtreg: old and new for a blog entry giving details of the different execution modes.

  • Some tests, typically GUI tests, may require manual interaction in order to determine whether they pass or fail. To run just those tests, use the -m or -manual option. To avoid running such tests, use the -a or -automatic option.

  • The tests in some parts of the test suite may be able to run concurrently. This feature is only available in the "othervm" and "agentvm" modes. To have jtreg run tests concurrently, use the -conc:value option, where value is either the maximum number of tests that should be allowed to run concurrently or the word "auto" to set the value to the number of processors on your system. Make sure you have enough memory and/or swap space when you use this option. For more information, see Running tests concurrently with jtreg.

  • If you have built a component of JDK such as the compiler (javac), you may be able to test it in conjunction with a complete JDK by putting the classes or jar file for new component on the system bootstrap classpath:

    On JDK 8 and earlier versions of JDK, you can use:

    % jtreg -jdk:jdk -Xbootclasspath/p:component.jar test-or-folder...

    On JDK 9 and later versions of JDK, you can use the --system and --patch-module options instead of -Xbootclasspath.

  • You can rerun any selection of tests. Use the -status:arg option to filter the set of tests to be run, based on their prior result status. For example, to rerun tests which previously failed or could not be run:

    % jtreg -jdk:jdk -status:fail,error test-or-folder...

  • For info on all the available options, use the -help option. For info on a specific option or topic, use -help option-or-word... To read the online help, use the -onlineHelp command line option, or start jtreg with the -gui option, and use the Help menu.

  • Here is an example that puts many of the preceding ideas together. JDK 17 is used to run jtreg, while jtreg uses JDK 18 to run the java/lang/Class tests in the jdk repository located at /w/jjg/work/tl/jdk. The tests are executed using the "agentvm" execution mode; a one line summary for each test is written as each test completes, and the overall results are stored in the local build directory.

    export JT_JAVA=/opt/jdk/17
cd /w/jjg/work/jdk
/opt/jtreg/6.1/linux/bin/jtreg \
    -jdk:/opt/jdk/18 -agentvm -verbose:summary \
    -w build/jtreg/work -r build/jtreg/report \
    test/java/lang/Class
Run jtreg.jar directly

Instead of using the startup scripts provided, you can also execute the jtreg.jar file directly, or you can run jtreg from Ant. See the online help for details.

Test Results

Summary reports are generated in the report directory, in plain text and HTML formats. To generate a detailed report in XML as well, use the -xml option.

Detailed results for each test are stored in the work directory, in .jtr files. These are plain text files that are easy enough to read directly if needed.

Individual tests can be in one of four states.