Shell Tests in jtreg

When writing a shell test, it is important to consider the take into account the differences in the way that scripts are executed on the various supported platforms.

Platform differences

The primary differences that may need to be taken into account when writing a test that can be run on a variety of platforms reflect the different ways to run tests on a Windows system, as compared to POSIX systems like Linux, macOS, and Solaris. On Windows, MKS Toolkit was originally used to run JDK shell tests; for OpenJDK shell tests, Cygwin is currently the standard, and support is coming for using Windows Subsystem for Linux (WSL) on Windows 10.

Note: While it is necessary to update existing shell tests if they are to support being run with WSL, there is no need to update any tests if such support is not required.

These are the differences that typically affect the way to write a shell test.

File separator
On POSIX systems, / is the standard file separator character; Windows originally required the use of \ but now accepts either.
Path separator
On POSIX systems, : is the standard character to separate file and directory names within a single string. On Windows, the character depends on the system being used to run shell tests: the standard Windows path separator character is ; but some systems allow or require the use of : instead.
Null device
On POSIX systems, the null device is /dev/null. On Windows, the name depends on the system being used to run tests, and is either NUL (MKS Toolkit) or /dev/null (other systems).
Suffix for executable binaries
On POSIX systems, executable binaries typically do not use any file name extension. On Windows, executable binaries use the .exe extension, but depending on the environment used to run tests, it may not be necessary to specify the extension when invoking the command from a shell. When invoking a Windows binary from WSL, the file extension must be specified.
Absolute file system paths
On native Windows, an absolute filename begins with letter:. On Cygwin, such paths are normally represented by /cygdrive/letter although it allows the use of "mixed-style" paths, beginning with letter: and using / as the file separator character. On WSL, such paths are represented by /mnt/letter. Unlike Cygwin, it does not allow the use of mixed-style paths, and paths must be in the correct form for the context in which it will be used. A utility (wslpath, similar to Cygwin's cygpath) is provided to convert between the different forms.

It is also worth noting that these differences only apply in the shell system itself. When values are passed in to a native Windows binary, via the command line or environment variables, the strict Windows-native forms must be used. This primarily applies to the use of the path separator character and the form of any absolute filesystem paths.

Most of these differences can be handled by determining the platform and setting environment variables to be used later in the script. That is not so for the issues related to absolute file system paths, and there the solution is to avoid the use of absolute file system paths as much as possible, relying on tools like jtreg to initialize any paths as needed, and to completely avoid the use of conversion utilities like cygpath and wslpath. (If your script is so complicated that it needs such utilities, maybe it shouldn't be written as a script. See Convert shell tests to Java.)

Determining the platform

Most platforms can be determined by examining the result of executing uname -s, which typically yields the name of the operating system. The following are exceptions to that rule:
Cygwin
uname -s begins with CYGWIN.
Windows Subsystem for Linux
uname -s reports Linux; /proc/version contains the word Microsoft.
MKS Toolkit
uname -s begins with Windows.
macOS
uname -s reports Darwin.

Handling platform-specific differences

1: Convert shell tests to Java

Convert the test to Java. In general, this continues to be the recommended alternative. Shell tests are fragile, and difficult to test in all likely environments. Converting a test to Java code is the best and most effective way to ensure correct operation on all platforms.

2: Use an inline case statement

You can use an inline case statement to set environment variables according to the characteristics of the platform being used to run the shell tests. These environment variables can then be used later in the script to accommodate the platform differences. Sometimes it will be necessary to use quotes around the reference to the environment variable, to prevent additional expansion by the shell. This is particularly true if the value may contain spaces. For example, "$TESTJAVA".

The following is an example script to set EXE_SUFFIX, FS and PS on different platforms.

# set platform-dependent variables
OS=`uname -s`
case "$OS" in
  SunOS )
    FS="/"
    PS=":"
    ;;
  Linux )
    FS="/"
    if grep -q Microsoft /proc/version \
            && test -x "$TESTJAVA"/bin/javac.exe; then
        PS=";"
        EXE_SUFFIX=".exe" ;
    else
        PS=":" ;
    fi
    ;;
  Darwin )
    FS="/"
    PS=":"
    ;;
  AIX )
    FS="/"
    PS=":"
    ;;
  CYGWIN* )
    FS="/"
    PS=";"
    ;;
  Windows* )
    FS="\\"
    PS=";"
    ;;
  * )
    echo "Unrecognized system!"
    exit 1;
    ;;
esac

3: Use a shared library file

Instead of using an inline case statement, you can put the statement in a shared library file, and execute it with the source command. You can either reference the file with a path that is relative to the TESTSRC directory, which will probably mean the path is different for different tests, or you can reference the file with a path relative to the test-suite root directory, using the TESTROOT environment variable set by jtreg.

source "$TESTROOT"/testlib/env.sh

Note that there is a small chicken-and-egg problem here: unless the shared script is in the same directory as the tests that will refer to it, there will be file separators in the relative path, and those file separators may vary between platforms. To reference the shared file, we potentially need to access the variables before they are set. However, / works as a file separator on all platforms, and so can be used instead.

Do not use the sh command to execute the script, because that will execute it in a child shell, which will not affect the instance of the shell running the test.

4: Use environment variables set by jtreg

When running a shell test, jtreg will set the following environment variables, intended to help avoid the need for a case statement in the test itself, or in shared library code. The following variables will be set:

Variable Description
EXE_SUFFIX Set to .exe when it is necessary to invoke a program such as java or javac.
FS Set to the file separator: / or \.
PS Set to the path separator: : or ;.
NULL Set to the name of the null device: /dev/null or NUL.

Running tests standalone

Amongst the reasons in the past to advocate the use of an inline case statement to handle platform differences was a desire to be able to develop and debug a test without the encumbrance of the jtreg harness infrastructure, and because of the prevailing practice at the time of doing "partial bringovers" (using an earlier source-code management system), meaning there was no guarantee that any shared library files would be available.

Although those reasons have mostly gone away, it can still occasionally be useful to run a shell test standalone. For example, one reason for running the test outside of the test harness is to insert a wrapper around the test process, such as to point LD_LIBRARY_PATH at a debugging malloc or run under strace -c to measure system calls.

These days, jtreg helps make it easy to run a test standalone: once a test has been run by jtreg, the test result (.jtr) file contains "rerun" sections with details on how to run each action of the test. You can either use the jtreg -show:rerun option to output the information to the standard output stream, which you can then save to a file, or you can copy/paste/edit from the .jtr file directly. If you copy from the .jtr, note that the text contains occasional escape sequences, which you will have to fix up before you can use the text in a script; if you use the -show:rerun option, those escape sequences are interpreted before the section is written to the output stream.

Recommendations

In all cases, whether modifying or removing an inline case statement, it will also be necessary to ensure that the rest of the script uses the .exe file extension, when necessary, when invoking Windows binaries.