JEP 281: HotSpot C++ Unit-Test Framework

Summary

Enable and encourage the development of C++ unit tests for HotSpot.

Goals

• Support writing and executing unit tests for methods, classes and subsystems

• Support unit tests, where only a unit is tested and nothing else is run

• Support tests which require VM initialization

• Support fast tests, where the execution time is in the order of milliseconds

• Support both positive and negative testing

• Allow test isolation

• Support tests co-located with the product source code

• Support integration with the current infrastructure

• Produce an individual result for each test

• Be able to easily run an individual tests (from the command line)

• Be able to provide minimal reproducers for test failures

• Provide IDE support

• Allow the framework to evolve, including making fast fixes to the framework

• Support test selection and test grouping, with granularity similar to jtreg

• Allow testing of any compile target, product as well as debug

• Allow tests of platform-dependent code

• Provide minimal documentation: A how-to wiki and examples in the repository

• Allow the exclusion of tests from execution by modifying the source of the tests or other files, such as exclude lists

• Support for all internal tests to be converted

• Support the JDK 9 build platforms supported by Oracle

Non-Goals

• Replace Java tests. Unit testing in C++ is a complement to testing for different use cases.

Motivation

In a well-tested codebase it is easier to make changes. The test suite supports the engineer who makes the changes by verifying that nothing unexpected breaks.

Today HotSpot has many tests, but not many tests of the most direct type, and it is not easy enough to write such tests.

Introducing a test framework for C++ is the first step towards a better test suite. A test framework for C++ supports test writing in the same language as the JVM and then the internal structures are directly exposed to the test code, which gives another level of possibilities to easily write small sharp tests, compared to doing functional testing from java using jtreg.

A possibility to develop unit tests for existing functionality will make it possible to test C++ code for new features in isolation and make it easier to write regression tests for some of the more esoteric issues.

Description

The Google Test framework (GTest) is the C++ unit test framework which most closely meets our goals, and it's an xUnit test framework, with a lot of traction in the community. The GTest framework:

• Is developed and supported by others
• Provides IDE integration with Eclipse IDE
• Is a battle-proven, complete API
• Has a feature-rich execution model
• Has existing documentation and examples
• Supports JUnit-style test results, and integration with Hudson and Jenkins

Several tasks are needed to allow using GTest for writing tests for HotSpot, and some additional tasks are needed to enhance it. In the current state of GTest:

• C++ constructs are used which HotSpot does not use and which are disabled in HotSpot, e.g., exceptions, templates, and STL
• Solaris/Oracle Solaris Studio isn't a supported OS/compiler

GTest is, admittedly, a third party tool, thus adding another dependency to the existing build and test process. GTest is also fairly big (71K LOC) and may change incompatibly in the future. To avoid the risk of changes in the test framework itself leading to problems, we need to control which version of GTest is used, and be able to specify that as part of the build (though it should be possible to override). It would be beneficial to have a dependency system to automate the download and install of the correct version of GTest.

HotSpot test directory layout

The new tests need a place to live in the source tree. The root directory for the tests should be placed close to, but not in, the product source itself, much like the existing test-directory structure. For clarity the tests should not be mixed with existing jtreg tests; instead, they should be split in two directories. We propose to split the current jdk9/hotspot/test directory into two subdirectories:

• jdk9/hotspot/test/java
• jdk9/hotspot/test/native

Existing jtreg tests will move down into the java directory (including JNI code and and shell scripts). The TEST.ROOT file will remain at the top level.

Build targets and binaries

The product binaries must not be affected by the test code in any visible way. For example, there should not be additional symbols exported and the product bundles should not include any tests. The compiled tests will be put in separate test bundles, one per configuration. The tests will link to symbols exported from a non-stripped JVM library, which is created from the same object files as regular library.

Invoking tests

It must be easy to run the tests from the command line using make. In order for the test results to be compatible with results from other tests the invocation may potentially run using a jtreg test wrapper, which in turn invokes GTest. GTest can itself produce JUnit-style results, which integrates nicely with Hudson/Jenkins and similar tools.

Alternatives

Alternative 1: HUTT. A prototype framework called "HotSpot Unit Test Tool" (HUTT) was created earlier, and is an xUnit framework. It is a significantly smaller framework than the GTest is (2K LOC) and is not an external dependency. It is a viable but more expensive solution. It also lacks IDE support.

Alternative 2: Keep implementing tests in Java. It is possible to access JVM internals using a Whitebox API. Adding Whitebox APIs is cumbersome in comparison and slow in execution. It is suitable for some introspection, but far from all testing. Java tests are more costly to write and execute, since in order to get high quality tests which target specific functionality the tests become very complex and it is often hard to guarantee determinism.

Alternative 3: Keep using internal tests. This solution would not meet many of the stated goals.

Risks and Assumptions

Risk: GTest could evolve in a direction which makes it unsuitable as a HotSpot unit test framework. The risk is estimated to be low.

Mitigation plan: Fork the GTest framework, or use HUTT.

Risk: Backporting GTest fixes will prove to be very costly.

Mitigation plan: Fork the GTest framework, or use HUTT.