Guidelines for "langtools" tests

The "langtools" tests in the test/langtools directory of the main jdk/jdk repository are the most important set of freely available tests for the products in the langtools family. And the test base is growing fast: in OpenJDK 6, there were 1408 tests; in OpenJDK 7, that number rose 36% to 1924 tests. In addition, many of the newer tests include hundreds or even thousands of individual test cases.

With that in mind, it is important to have some guidelines for the tests that appear here. The following reflect the current practices for the langtools tests in the main OpenJDK repository. Other teams may choose to have different guidelines.

1. The Golden Rule
All the tests1,2 should pass, all the time3, on all platforms4, in all modes5,6.
  1. except tests with @ignore, which itself is strongly discouraged
  2. not just javac tests
  3. at the time of checkin, and after
  4. including Windows and Mac OS/X
  5. jtreg modes: agentvm and othervm
  6. including with assertions enabled for langtools packages
2. Write tests
Most code changes require corresponding tests to exercise the new or updated functionality. The exceptions are listed in The OpenJDK Developers' Guide, under "Change Planning and Guidelines: Fixing a Bug", step 6.
3. Legal stuff
All the tests are Open Source, using GPL2. (Note: not GPL2 with the "Classpath" exception.) Most test files should include the appropriate header. The primary exception is "golden file tests".
4. No binary files
Some binary files currently exist: they need to be removed eventually. To provide a test with a class file that has particular characteristics, consider providing a source file, compiling it, and then mutating the resulting class file as necessary. In isolated cases, it may be appropriate to store an extended series of bytes as data within a Java source file, provided that such data is well commented.
5. No new shell scripts
Shell scripts are hard to maintain across all platforms, and are slower to execute. Write Java code instead. In many cases, in older versions of the langtools tests, shell scripts were used to write golden file tests. jtreg now provides improved support for such tests, so shell tests should no longer be necessary for that reason. Also, see the following for utilities to simplify writing shell-like functionality in Java:
6. No unnecessary use of Runtime.exec
Creating child processes is (relatively) slow, so avoid use of Runtime.exec unless it is important to the execution of the test. Tests that exec java should normally honor/propogate test VM options, so that options like -Xbootclasspath/p: and options to run jcov continue to work as expected in the child JVM.
7. No manual tests
i.e. no use of /manual in test descriptions. Some tests may have a mode in which they can be run directly: that is different, and is OK.
8. No external dependencies
No external dependencies on anything except the version of JDK being tested, and jtreg. Some tests refer to the langtools source code, and that is OK. jtreg itself may provide access to other facilities, like JUnit.
9. Tests should be fast
Generally, tests should take under a minute or so to run. Tests that take a long time need to be justified in terms of need and test cases covered. The current standard deviation of test execution time is under 3 seconds. That means that tests taking longer than 30 seconds are already 10 standard deviations from the mean (i.e. very atypical.) Slow tests that need to override the default timeout should generally specify a value that is double the expected execution time on a typical developer platform.
10. Golden file tests
Golden file tests (tests whose success depends on generating output to be matched a reference, or golden, file) are occasionally necessary. If the reference file contains references to line numbers in a source file, then that source file does not need a copyright block. Instead, it should contain the marker string "/nodynamiccopyright/" as an explicit indication that no copyright or other legal header should be added, because to do so would break the test. If the source file contains a jtreg test description, the marker string should normally appear after "@test", on the same line.
11. Negative compiler tests
Negative compiler tests (tests using @compile/fail) should always use a golden file. This is to protect against a false-positive test result caused by javac crashing. The preferred directive for negative compiler tests is: 
@compile/fail/ref=FILE.out -XDrawDiagnostics ...
The use of -XDrawDiagnostics isolates the test from changes in the resource file, and removes other elements, like directory names, that are specific to the instance of the test run.
12. JDK, and -Xbootclasspath/p:
Tests should all work as expected when testing a full JDK build. They should also work when testing an incremental build, using options such as -Xbootclasspath/p:.
13. Organization
The tests in langtools/test/tools/javac need some attention and TLC, when it comes to their organization. While that happens, no new tests should be added directly into the javac directory: instead, they should be added to a suitable sub-directory. By default, the hierarchy under the javac directory should mirror the package hierarchy under com.sun.tools.javac. For example, tests for com.sun.tools.javac.file.JavacFileManager should be placed in langtools/test/tools/javac/file. Additional sub-directories may be created as needed to group the files of a test or to group related tests. Try and come up with helpful names for tests before you fall back on using the bug number.
14. Whitespace and style
Tests should follow the standard white space rules and coding conventions as is the case for any OpenJDK source code. Code should not contain tabs, and indentation should be in multiples of 4 spaces. Lines should be terminated with Unix-style newlines ('\n'), and not have extraneous spaces before the newline character. It may be appropriate for some tests to test a tool's ability to handle source code that does not conform to these guidelines: in such cases, it is suggested and recommended that such code should be generated on the fly, as part of the test. New tests should use the standard Java coding conventions generally encouraged by modern IDEs. Modifications to existing tests should use the prevailing style of the code being modified. If a test warrants breaking these guidelines, the reason should be clearly documented in the test.
15. Disabling/ignoring tests
Sometimes a test may fail, and it may be desirable to avoid running the test until the underlying cause can be addressed. Such tests should be marked with @ignore followed by the bug number and description of the reason why the test has been so marked. If a test contains multiple test cases within it, a test may provide a way of identifying test cases that should not be run, such as an @ignore annotation on a test method. In such situations, the line should end with a bug number and description of why the test case has been so marked. The general intent is that it should be possible to write scripts that search for files containing lines that match the regular expression (?i)@ignore.*([0-9]{7}): +(.*) in order to report on tests or test cases that have been disabled.