Writing jtreg tests

The following is a list of guidelines, or points for consideration, when writing tests to be run by jtreg.

Tests should expect and deal with unexpected failures, even if that simply means throwing an exception to be caught by the test harness. Don't assume that a test will behave as expected, don't ignore return codes that might indicate a failure, and don't ignore exceptions. These guidelines apply to all parts of the test code including, where appropriate, code to setup the conditions for a test, and code to cleanup afterwards.

Repeatable

Successive runs of the test should yield the same results. In particular, this means that care should be taken when using random numbers to generate data for a test. If a test fails, it is important to be able to run the test again and get the same results while trying to debug the issue. Likewise, if a test passes, it is important that the test should pass for subsequent test runs, such as during integration and promotion testing.

Reliable

When modifying the product, it is really important to be able to run all appropriate tests, and to have reasonable confidence that if a test fails, it is likely a side effect of the modifications. Any time a test fails, the failure needs to be analyzed, to see if it is a "known failure", or if it has failed for some other reason, and that takes both time and resources.

Maintainable

Tests are as important as the main source code for the product, and need to be well-written, with the expectation that other folk may need to read and understand the code, in the event that the test uncovers an issue at some point in the future.

Tests often push API and other components into the dark corners of the spec, and may sometimes resort to strange coding tricks to do so. Use comments to document any seriously off-brand usage, to help readers differentiate between accidental typos and intentional cleverness.

Execution mode

Do not write tests to assume that any particular execution mode will be used to run tests. Although othervm mode is still the default, for historical reasons, agentvm mode is now generally recommended.

You can use /othervm to force the use of a new JVM for specific actions within a test. Use this when you need to force particular settings for a JVM, such as specific VM options.

Cleaning up after a test

JVM state: In othervm mode, there is obviously no need to clean up any JVM state. In the other two modes, jtreg will attempt to clean up JVM state that it knows about and which can easily be reset. This includes system properties, system streams, the security manager, and the default locale. Any other global JVM state that is modified by a test should be reset by the test. If the global state cannot be reset, consider running the test in othervm mode.
Files: jtreg will automatically clean up any files written in the scratch directory. In general, you should not clean these files up within the test because they may provide infomation useful to diagnose a test failure, should that be necessary.
Other resources: Any other system resources (windows, network connections, etc) that are used by a test should be released by the test, unless the test is explicitly using othervm mode.

Concurrency

It is becoming increasing common to run tests on multi-core systems, and to want to run tests concurrently. This requires some degree of cooperation between jtreg and the tests it is running. In general, it should be acceptable for a test to start a couple of extra JVMs while it is running, or to create some number of threads, but if it needs to use a significant amount of system resources, it should declare itself as needing exclusive access, so that jtreg will not run other tests at the same time.

Make sure tests run

This is another way of saying, check your test description: that is, the comment block near the beginning of the file, which must begin @test. Do not put a space between @ and test. Do not put @ignore or any other tag before @test.

Make sure tests run as expected

There is a difference between providing that doesn't fail, and a test that does what is expected. The interface between jtreg and each test is deliberately simple: a test succeeds if main(argv) return normally, and fails if an exception is thrown. But, there can be a risk of false positives. Here are some real world examples:

A common pattern is for a test to call an error method, to record an error, and to carry on with additional testing. When all the testing is complete, the test checks (or should check) if any errors have been recorded, and should at that point throw an exception to have jtreg take note. If it omits to do that, jtreg will presume the test to have pased, even though errors might have been reported internally, within the test. Oops.
Another common pattern is to use reflection to determine test cases within a test, perhaps by detecting methods maked with an @Test annotation. If you forget the annotation, the test will not execute the test case. Oops again.

Verbosity

Provide enough information to diagnose the issue if the test should fail. Especially in the case of tests that contain many test cases, identify the test case and all pertinent information about the test case. If a test can detect different error conditions, use a unique error message for each condition. Don't combine checks and lose information: It may be more concise to write code like:
if (condition-A || condition-B) throw new Exception("failed");
but if the exception is thrown, you won't know the specific reason why.

Conversely, don't provide too much unnecessary information, especially in the case of tests that contain many test cases. jtreg does limit the amount of output that is recorded for a test, and while the limit can be overriden, you can only do so on the command line, and you can't rely on everyone to do so. You might still want to give some information for each test cases that is run, just to give a warm fuzzy feeling that the test is actually executing the appropriate test cases, but maybe you don't need every little detail to be written out for those test cases that pass.

Golden files

Be careful when using "golden files" to check the output of a test. As well as the obvious problems, such as filenames, timestamps and other host-specific information, you also need to be careful handling line separators in text files. Files checked into the repository will have \n line separators; files written when a test is run on Windows will typically have \r\n line separators.

When comparing files using Java code, it is easy enough to read files line by line for the comparison. If you are writing a shell test, you can use diff --strip-trailing-cr; however, the option is not available on all platforms and so needs to be conditional on the platform being used to run the test.

Shell tests

In general, if you are considering to write a shell test, consider writing the test in Java instead. These days, it is generally easy enough to use Java for anything that you might want to do in a shell script, and the result is more likely to be more robust and more portable.

If you must write a shell script, consider the following:

Watch out for platform differences: file and path separators, line separators, and so on.
Watch out for commands having different options, or different names for options, on different platforms.
Watch out for falling off the end of the script and not returning an appropriate exit status. Unless you use set -e, the last command in the script should either be a command that will return an appropriate status, or exit value where value is the exit status from some earlier command.

For more information, see Shell Tests in jtreg.

Respect command line compiler and VM options

If you need to fork processes to run commands like java, javac, and so on, in general, you should use values specified on the command line for the JDK to use, and you should propagate any relevant compiler and VM options. See the table in Using JVM and javac options with jtreg to see how command line options are mapped to environment valiables (for use by shell scripts) and system properties (for use by Java code.)

Note: in general, you should not need to fork javac from Java code: you can invoke it directly via API.

Test frameworks

TestNG: jtreg supports the use of the standard, open source TestNG testing framework. Individual tests can specify the use of TestNG, using @run testng classname args, or entire directories can be declared to contain TestNG tests.
Custom libraries: Often, many related tests may share a common programming pattern or idiom. In this case, you can write a library to provide the common code required by the group of tests. One common idiom is for the tests to share a common superclass, often called something like somethingTester. Look in the various OpenJDK forests for examples of this idiom.

And finally...

Test your test!

Quis custodiet ipsos custodes? Especially in the case of a regression test, verify that your test demonstates the issue before the fix is applied, and demonstates that the issue is fixed after the fix is applied.

If you have coverage tools available, verify that your test exercises all the corresponding parts of the product code.

Writing jtreg tests

And finally...

See Also