The JDK Test Framework: Tag Language Specification
Comments and questions to: jtreg-use@openjdk.java.net.
1.47, 7 July, 2020
This is a specification document, not a tutorial. For more basic information please consult the jtreg FAQ at http://openjdk.java.net/jtreg.
Contents: Leading comments and defining files · Tag syntax · Informational tags · Declarative tags · Action tags (action options, class names as action arguments, class name wildcards, action types) · Order of tags · TestNG · Directory context · Test groups · Source-directory structure and test suite configuration files · Defaults · Shorthands · Examples · Supported names for @requires · Test-specific system properties and environment variables · System environment variables
LEADING COMMENTS AND DEFINING FILES
A particular test may involve several files of various types, but the test is specified in just one of them. Let this be called the defining file of the test.
The defining file of a test may be one of several types of files. Test tags must be enclosed in a comment at the head of the file, optionally preceded by comments which do not contain test tags. The comment in the defining file which includes the test tags is called the leading comment. The following types of defining files are recognized:
- A Java source file, with file extension ".java". Only the
"
/*
" to "*/
" comment syntax is recognized. On each comment line leading asterisks, if any, are ignored. - A Bourne shell file, with file extension ".sh". The leading comment is the first contiguous set of lines each of whose first character is '#'. The comment is terminated by the first line whose first character is not '#'. Note that shell tests should not contain "#!/bin/sh" on the first line; they are intended to be run only via explicit invocation of a shell.
- An HTML file, with file extension ".html". The leading comment
is enclosed within the usual "
<!--
" to "-->
" HTML tag.
The contents of a leading comment are parsed into a sequence of tokens. A token is a maximal contiguous sequence of non-whitespace characters. Whitespace characters, which include newline and carriage-return characters, are significant only insofar as they separate tokens.
Some files within the source directory structure may be
configured to implicitly be defining files. For all other files,
the first token in the leading comment in the defining file of a
test must be "@test
". The harness discovers defining
files by looking for such tokens: a file without comments is not a
defining file; a file whose leading comment does not start with
"@test
" is not a defining file.
TAG SYNTAX
Within a leading comment, a tag consists of a tag token followed by some number of arguments. A tag token is any token whose first character is '@'. There is at present no syntax for quoting a leading '@' character. Case is significant when comparing tag tokens; all tag tokens defined here are in lower case. It is an error for the leading comment of a defining file to contain any tag token not defined here.
As a special case, the token "@(#)
" is not treated
as a tag token. For historical reasons, this token may appear in
the arguments to an @test
tag as part of an SCCS
identification string.
The arguments of a tag are simply the tokens that follow it. The argument tokens of a tag extend from the first token after the tag token to the end of the comment, the end of the file, or the next tag token, whichever comes first. Argument tokens are separated by whitespace; if commas are used, they will be considered to be part of the tokens to which they are adjacent.
The arguments of a tag may contain references to various
properties, using the syntax
${<name>}
. The set of names is the
same as the set that may be used in an @requires
expression, and is defined in Appendix
1. Note: For compatibility reasons, this
feature is only enabled by default when the test suite declares a
required version of jtreg 4.2
b14
and higher. The feature can be explicitly enabled or
disabled by setting allowSmartActionArgs
in the
configuration files.
INFORMATIONAL TAGS
Informational tags document the test. They do not affect the operation of the harness, but may be used by the harness in generating reports.
Any particular informational tag, except the
@comment
tag, may occur at most once in a given test.
The @comment
tag may be used multiple times.
DECLARATIVE TAGS
Declarative tags govern the execution of the tags that follow them.
ACTION TAGS
Action tags tell the harness how to perform the test. They are executed in the order in which they are given. Each action either passes or fails. If an action fails, no following actions, if any, are performed. A test passes if, and only if, all of its actions pass.
Action tags begin with the tag token "@run
", and
have the following syntax:
@run <type><option>* <arg>*
The <type> describes the basic type of the action; the <option>s further describe how the action is to be performed. The <arg>s are passed to the test in a manner appropriate to the <type>.
The first token after the "@run
" token contains
both the action type and options, if any. To parse the type and the
options, this token is broken down into subtokens. A subtoken is
either the character '=', the character '/', or a maximal
contiguous sequence of characters not containing either of these
characters.
The type of an action tag is named by the first subtoken of the tag's first token. The remaining subtokens of the first token form the options. Options have the syntax
/<name>
or
/<name>=<value>
where <name> and <value> are a single subtokens. The <value> may be enclosed in double quotes to prevent the usual interpretation of '=', '/', and whitespace.
ACTION OPTIONS
Not all action types support all options.
CLASS NAMES AS ACTION ARGUMENTS
Some actions take one or more class names as arguments. Each such argument is the name of a class, not the name of a class file, so the ".class" suffix should not be given.
A class name may be specified in one of the following ways:
- a simple class name: e.g., "Foo"
- a package name followed by a simple name: e.g., "bar.Foo", "baz.bar.Foo"
- a module name followed by either
module-info
or a fully qualified class name: e.g., "myModule/module-info", "myModule/baz.bar.Foo"
CLASS-NAME WILDCARDS
Some actions take either class names or class-name wildcards as arguments. A class-name wildcard can be specified in one of the following ways:
- "*", meaning all classes in the default package
- "<package-name>.*", meaning all classes in the named package
- "<module-name>/<package-name>.*", meaning all classes in the named package in the named module
- "<module-name>/*", meaning all classes in the named module
ACTION TYPES
ORDER OF TAGS
The following is the recommended order for the tags in a test description:
@test
@bug
@summary
@key
@requires
@modules
@library
- action tags, in the order to be executed
TESTNG
A directory within the source directory structure may be configured to be the root of a set of TestNG tests. When so configured, all Java source files in or under the directory are considered to be defining files, and thus considered to be a test to be executed. If a file has a leading comment, the comment may only contain informative tags and declarative tags, and may not contain any action tags.
Each test within the set is deemed to have any implicit
@build
action specifying all the Java source files in
the set, followed by an implicit action to execute the test as a
TestNG test.
DIRECTORY CONTEXT
Each test is run in a context that defines the following directories.
- Working directory
- The directory in which the harness is running.
- Source directory
- The directory containing the defining file of the test, as well as any associated files (e.g., input data files). Generally not the same as the working directory.
- Class directory
- The directory into which class files compiled from source files in the source directory are compiled. It may be the source directory, or it may be some other directory.
- Java home directory
- The directory containing the JDK build or release being tested.
A test may create temporary files in the working directory as needed. Test harnesses will typically delete such files before or after each test is run. In order to work properly when run standalone, however, tests should not rely upon this behavior. This can be done having each test's initialization code delete any files that the test may have created in a previous invocation.
For each source directory in the test hierarchy there is a corresponding class directory, which may or may not the same as the source directory. Classes in different source directories are, therefore, in different name spaces and their names will not collide. Simple test harnesses may place class files in source directories, while more sophisticated harnesses will generally place class files in a parallel directory hierarchy.
The names of the source, class, and Java home directories of a
test are made available to shell-action scripts via the environment
variables TESTSRC
, TESTCLASSES
, and
TESTJAVA
, respectively. The directory names do not
have trailing separators.
The names of the source and class directories of a test are made available to main and applet actions via the system properties "test.src" and "test.classes", respectively. The directory names do not have trailing separators. Main and applet actions can read data files defined in the test's source directory using this idiom:
File f = new File(System.getProperty("test.src", "."), "foo"); InputStream in = new FileInputStream(f);
By defaulting the directory to ".", this technique allows the test to be run standalone in its source directory.
TEST GROUPS
Groups are defined in a test suite using one or more Java
properties files. The names of these files must be listed in the
"groups
" entry in TEST.ROOT.
If the filename is enclosed in square brackets, no error message
will be given if the file cannot be found. Within the property
files, each entry specifies items to be included or excluded from
the group.
- To include a test or directory of tests, simply specify the name of the test or directory.
- To exclude a test or directory of tests, use '-' followed by the name of the test or directory. There must be no spaces between the "-" and the name that follows.
- To include the contents of another group, use ':' followed by the name of the group. There must be no spaces between the ":" and the name that follows.
SOURCE-DIRECTORY STRUCTURE AND TEST-SUITE CONFIGURATION FILES
This specification places few constraints upon the structure of a tree of source directories. Some test harnesses may be able to provide more robust behavior or more succinct reports when they can identify the root of such a tree. The root of a test-source tree must therefore be identified by placing a file named TEST.ROOT in the root directory. Exactly one such file must exist in every test-source tree.
The TEST.ROOT file contains
test-suite configuration information. It is in the standard Java
property-file format as described in the specification of the
java.util.Properties
class. Properties defined in this
file apply to the entire test suite. Additional properties files
named TEST.properties may appear in subdirectories of the test
suite, and may be used to provide additional configuration values
that apply to files in that directory and its subdirectories.
The following properties may only appear in TEST.ROOT:
checkBugId true|false
-
Whether or not to check the format of strings in @bug tags. Checking is disabled if this property is set to
false
. defaultExecMode agentvm|othervm|samevm
-
The default execution mode for tests in the test suite. A harness that supports multiple test execution modes may use this value to determine the default execution mode for tests in this test-suite.
groups <files>
-
Where to find group definition files. A space separated list of files. Optional files should be enclosed in square brackets.
requiredVersion <version>
-
Minimal version of the harness required for this test-suite. A version string in the format "[0-9.]+ ?b[0-9]+" (without the quotes). For example a value of "4.1 b03" declares that at least that version of the harness is required to run all the tests in the test suite.
requires.extraPropDefns <source-files>
-
This option is used to provide source files for classes that will be used at the beginning of each test suite run, to determine additional characteristics of the system for use with the
@requires
tag. Each class must implementjava.util.concurrent.Callable<java.util.Map<String, String>>
. When invoked, thecall()
method should return properties that can be referenced in an expression in a@requires
tag. Note: the names of the new properties that may be returned by calling these classes must also be declared in arequires.properties
entry. If an error occurs while computing a value, the value may be set to the special value__ERROR__
, optionally followed by a short explanation of the error that occurred.If this option is specified, the following additional options may also be specified:
requires.extraPropDefns.libs <source-files>
— source files for classes that will be put on the classpath when the primary classes are run.requires.extraPropDefns.bootlibs <source-files>
— source files for classes that will be put on the bootclasspath when the primary classes are run.requires.extraPropDefns.javacOpts <options>
— options that will be passed to javac when the source files are compiled.requires.extraPropDefns.vmOpts <options>
— options that will be passed to VM when the classes are executed.
In this family of options, if a source file is enclosed in square brackets, no error message will be given if the file is not available.
The following properties may appear in either TEST.ROOT or any TEST.properties file:
keys <keywords>
-
The set of valid keywords that can be used in @key tags. A harness that supports keyword-based test selection should reject any test containing keywords that are not present in this list. If this property is not present then no keywords should be accepted.
bootclasspath.dirs <directories>
-
Directories that when used should be placed on the bootclasspath.
othervm.dirs <directories>
-
Directories containing tests which should all be run in othervm mode. A harness that supports multiple test execution modes should ensure that all tests in the specified directories are run in
othervm
mode. exclusiveAccess.dirs <directories>
-
Directories containing tests which should not be run concurrently at the same time as other tests. A harness that supports concurrent test execution should ensure that all tests in the specified directories are run when no other tests in the same directory is being run.
TestNG.dirs <directories>
-
Directories containing TestNG-style tests.
lib.dirs <directories>
-
Default library directories used by all tests in or under the directory containing this property declaration.
lib.build <build-arguments>
-
Default library components that will be built by tests in TestNG directories in or under the directory containing this property declaration. The values should be of a form that can be accepted by an
@build
tag. If not specified, the entire contents of any currently specified libraries will be built. external.lib.roots <directories>
-
A search path for libraries specified in an
@library
tag. Relative paths will be evaluated relative to the directory containing the configuration file. modules <module[/package[:flag]]>+
-
Express a default dependence, for tests in this directory and its subdirectories, on modules in the system being tested, and optionally, on selected internal packages in some or all of those modules.
requires.properties <property-name>+
- Declare property names that may be used in a
@requires
expression. The names may identify either system properties or additional test-suite specific properties return by calling the classes defined by the@requires.extraPropDefns
family of configuration values. maxOutputSize <int-value>
- Override the default limit for the amount of output that will
be saved on any stream being recorded by the test harness. For
jtreg, the default limit for all tests is 100,000 characters, or
can be set on the command line by setting the system property
javatest.maxOutputSize
.Note:To set a system property for jtreg on the command line, use
-J-D<name>=<value>
. allowSmartActionArgs <true|false>
- Specify whether to permit the use of
${<name>
in the arguments for an action. The set of names is the same as the set that may be used in an@requires
expression. See Appendix 1. enablePreview <true|false>
- Specify whether the tests in this directory and any
subdirectories use preview features and that the necessary
compile-time and run-time options should be provided automatically.
The default value can be overridden in individual tests using the
@enablePreview
declarative tag.
DEFAULTS
If no @run
tags are present in a defining file, a
default is assumed depending upon the file's filename
extension.
File type | Default | Notes |
---|---|---|
.java | @run main <name> |
<name> is the name of the file without the ".java" suffix |
.sh | @run shell <file> |
|
.html | @run applet <file> |
SHORTHANDS
Shorthand | Equivalent |
---|---|
@build <classname>+ | @run build <classname>+ |
@clean <classname>+ | @run clean <classname>+ |
@compile<option>* <arg>+ | @run compile<option>* <arg>+ |
@ignore <word>* | @run ignore <word>* |
EXAMPLES
Run Foo
in a separate VM, with the heap limited to
2MB and the verifier turned on:
/* @test * @run main/othervm -mx2m -verify Foo arg1 arg2 */
Run Foo
in the same VM, with a two-second
timeout:
/* @test * @run main/timeout=2 Foo */
Compile Foo
with debugging, expecting failure, and
check the error message against a reference file:
/* @test @compile/fail/ref=Foo.ref -debug Foo.java */
Remove Bar
's class file, if it exists, then compile
Foo
, then compile Bar
, and finally run
Bar
, which is expected to fail:
/* @test @clean Bar @compile Foo.java @compile Bar.java @run main/fail Bar */
Run the applet described in the file Foo.html, waiting for the user to indicate success or failure.
/* @test * @run applet/manual=yesno Foo.html */
Run the class Snidely
with the security manager
Pinkerton
, using the paranoid.sp policy file:
/* @test @build Pinkerton @run main/othervm/secure=Pinkerton/policy=paranoid.sp Snidely */
Run the class SnowWhite
using classes in the
library directory dwarfs:
/* @test @library ../dwarfs @build Bashful Doc Dopey Grumpy Happy Sleepy Sneezy @run main SnowWhite */
APPENDIX 1: SUPPORTED NAMES FOR @requires AND ACTION ARGUMENTS
Note: See the details for the test-suite configuration files for information on how to extend the set of supported names for a particular test suite.
Name | Description | Value if set | Default value if not null |
---|---|---|---|
jdk.version |
The JDK version, as given by
java.specification.version system property of the test
JDK. |
||
jdk.version.major |
The major component of the version string, i.e. ignoring any leading "1." for versions prior to JDK 9. | ||
os.name |
The operating system name, as given by the corresponding system property. | ||
os.family |
The operating system family, derived from the operating system name. | One of linux mac solaris
windows |
first word of os.name |
os.arch |
The operating system architecture, as given by the corresponding system property. | ||
os.simpleArch |
A simplified representation of the operating system architecture. | One of x64 i586 ppc |
same as os.arch |
os.version |
The operating system version, as given by the corresponding system property. | ||
os.simpleVersion |
A simplified representation of the operating system version. | N.0 or N.M ,
where N and M are first two
numbers in os.version |
99.99 |
os.processors |
The number of processors on this system | The value as determined by
Runtime::availableProcessors |
|
os.maxMemory |
The maximum amount of memory available on this system. | The value as determined by
OperatingSystemMXBean::getTotalPhysicalMemorySize |
|
os.maxSwap |
The maximum amount of swap space available on this system. | The value as determined by
OperatingSystemMXBean::getTotalSwapSpaceSize |
|
vm.gc |
The garbage collection strategy, derived from the option
-XX:+UsenameGC |
name |
|
vm.opt.switch |
A boolean VM option, derived from option
-XX:+switch or
-XX:-switch |
true false |
|
vm.opt.name |
A VM option, derived from option
-XX:name=value |
value |
|
profile |
The name of the highest profile supported by the test JDK. | One of compact1 compact2
compact3 |
|
module:name |
Whether or not a module is available in the test JDK. | One of true false |
APPENDIX 2: TEST-SPECIFIC SYSTEM PROPERTIES and ENVIRONMENT VARIABLES
The system properties in the following table will be set while executing any class specified in an action tag; the environment variables will be set while executing a shell action tag.
System property | Environment variable | Notes |
---|---|---|
test.file |
TESTFILE |
The defining file of the test |
test.src |
TESTSRC |
The directory containing the defining file of the test |
test.src.path |
TESTSRCPATH |
The series of directories, including library directories, for the source of the test. |
test.classes |
TESTCLASSES |
The directory containing the compiled classes for source files in the directory containing the defining file of the test. |
test.class.path |
TESTCLASSPATH |
The series of directories for all the classes for the test. |
test.vm.opts |
TESTVMOPTS |
Additional VM options for all JVMs used in the test. |
test.tool.vm.opts |
TESTTOOLVMOPTS |
Additional VM options for all JVMs used in the test, each
preceded by '-J ', suitable for use in tool commands
like "javac". |
test.compiler.opts |
TESTJAVACOPTS |
Additional Java compiler options. |
test.java.opts |
TESTJAVAOPTS |
Additional VM options for all invocations of the main Java launcher used in the test. (i.e.excluding JDK tools like "javac".) |
test.jdk |
TESTJAVA |
The location of the JDK being used to run the tests. |
compile.jdk |
COMPILEJAVA |
The location of the JDK being used to compile the tests. By
default, this is the same as test.jdk and
TESTJAVA . |
test.timeout.factor |
TESTTIMEOUTFACTOR |
The timeout factor to be applied to the default timeout for the test. |
test.nativepath |
TESTNATIVEPATH |
The location of native executables used in the tests. |
test.modules |
TESTMODULES |
The module dependencies of the test, if any, defined in the
@modules tag. The dependencies will be ignored, and
the system property and environment variable will be unset or
empty, if the version of JDK used to run the tests does not support
modules. |
test.root |
TESTROOT |
The root directory of the test suite (the directory containing the TEST.ROOT file.) |
test.enable.preview |
TESTENABLEPREVIEW |
Set (to true code>) if the test is using
"preview features" as indicated by @enablePreview in
the test description or an enablePreview entry in a
TEST.properties file. |
Not Applicable | FS |
The file separator character to use. (\ for
Windows tools; / otherwise.) |
Not Applicable | PS |
The path separator character to use. (; for
Windows tools; : otherwise.) |
Not Applicable | NULL |
The "null sink" to use. (NUL in a Windows
environment; /dev/null otherwise.) |
APPENDIX 3: SYSTEM ENVIRONMENT VARIABLES
When any action is run, environment variables that may be required by various system components will be set.On Unix systems, the PATH
will be set to
/bin:/usr/bin:/usr/sbin
and the following variables
will be propagated if they are set:
DISPLAY
,GNOME_DESKTOP_SESSION_ID
,HOME
,LANG
,LC_ALL
,LC_CTYPE
,LPDEST
,PRINTER
,TZ
andXMODIFIERS
.
On Windows systems, the following variables will be propagated if they are set:
PATH
,SystemDrive
,SystemRoot
,TEMP
,TMP
,TZ
andwindir
.
WSLENV
will be set appropriately; and
EXE_SUFFIX
will be set to .exe
, to
facilitate creating paths for tools to be executed by the shell
script.