Running tests concurrently with jtreg

With larger systems becoming more commonplace, it is possible to speed up test runs by running tests in parallel1,2.

The basic option to enable tests to be run in parallel is

-conc:value
-concurrency:value

value should be set either to an integer representing the number of tests that can be run at once, or the word "auto" which evaluates to the number of processors available on the host system.

Execution Mode

The -concurrency option can only be used when running tests in the agentvm or othervm test execution modes. Of these two, agentvm is the generally recommended choice, since it allows a greater throughput by reusing JVMs across tests. But, it does mean that more JVMs may be present at any one time, especially if you specify different VM options for compiling and running tests.

Execution mode Can use -concurrency
agentvm Yes, recommended
othervm Yes
samevm No

Memory size

The Hotspot JVM uses a feature called JVM Ergonomics3 to determine the default maximum size for each JVM that is created. This can be as much as 25% of the system's memory, up to a maximum of 1G. This means that you can easily over-allocate your system's memory, and ultimately, degrade the overall performance of the test run because of excessive swapping. You may want to limit the memory given to each machine by using the -Xmx512m option.

Test Suite

The characteristics of the tests to be run should also be taken into account when running tests in parallel. Depending on the JVM ergonomics, the JVM may be able to leverage multiple threads/processors even if the test itself is otherwise single threaded.

The following recommendations are for test suites in the jdk7u, jdk8 and related forests.

hotspot/test/
jdk/test/

Empirical evidence suggests that you can run the tests in these regression test suites with the concurrency set to the number of processors available on the system.

langtools/test/

The tests in the langtools regression test suite have been tuned to be fairly CPU intensive, with many tests creating and compiling thousands or tens of thousands of test cases. For the langtools/ tests the recommendation is to set the concurrency to half the number of available processors.

Note: tests that can use or need to use a significant proportion of a system resources should be declared as "exclusiveAccess" tests, so that jtreg avoid running other tests at the same time.

Summary

Run tests concurrently, using a command line such as this:

$ jtreg ... -agentvm -Xmx512m -conc:auto ... tests

Take care not to overload your system: it is better to run it at just under 100% utilization than just over, and then incur swapping. Use your system resource monitoring tools to help you tune the parameters for your test run.

1. Speeding up tests again: another milestone
2. jtreg update, 2011
3. Ergonomics in the 5.0 Java Virtual Machine