Project Jigsaw: Goals & Requirements

DRAFT 3

Mark Reinhold

The primary goals of Project Jigsaw are to:

• Make the Java SE Platform, and the JDK, more easily scalable down to small computing devices;

• Improve the security and maintainability of Java SE Platform Implementations in general, and the JDK in particular;

• Enable improved application performance; and

• Make it easier for developers to construct and maintain libraries and large applications, for both the Java SE and EE Platforms.

To achieve these goals we propose to design and implement a standard module system for the Java SE Platform and to apply that system to the Platform itself, and to the JDK. The module system should be powerful enough to modularize the JDK and other large legacy code bases, yet still be approachable by all developers.

What follows is an expansion of the above goals into a set of sub-goals, or requirements. We also list some global constraints upon any design as well as some non-requirements, in order to make it clear what this project is not trying to achieve. The non-requirements are motivated by our general experience with the JDK over many years, and also by specific experience gained from a series of prototype designs and implementations.

Each (non-)requirement is shown as

• Indented, bulleted text with an italic title — Followed by a statement of the requirement.

Motivational or explanatory comments sometimes follow a requirement, in non-indented text.

This document supersedes the Java Module-System Requirements document last published in draft form in April 2011. In this document we have attempted to tie each requirement more tightly to the high-level goals of the project, which are somewhat different now. We have also endeavored to omit any requirement that is actually a design decision in disguise.

The terms “must,” “must not,” “required,” “shall,” “shall not,”“should,” “should not,” “recommended,” “may,” and “optional” in this document are to be interpreted as described in RFC 2119.

Please send comments on this draft to jigsaw dash dev at openjdk dot java dot net. You’ll need first to subscribe, if you haven’t done so already.

Contents

Constraints

In order to protect investments in existing libraries and applications, any approach to the goal-driven requirements listed in later sections is constrained by the following global, product-level requirements.

• Preserve existing deliverables — The JDK build process must continue to produce binary run-time images that are essentially equivalent to the existing Compact Profiles, the client and server JREs, and the JDK.

• Preserve compatibility — An application that uses only standard Java SE APIs, and possibly also JDK-specific APIs, must work the same way on modular Compact Profile, JRE, and JDK images as it does today.

A standard Java SE API is one that is part of the Java SE Platform, and whose specification is therefore governed by the Java Community Process. A JDK-specific API is one that is not part of the Java SE Platform but is nonetheless documented and intended for use outside the JDK. (As of JDK 8, every JDK-specific API is identified as such via the @jdk.Exported(true) annotation.) All other APIs of the JDK are JDK-internal, and are not intended for external use.

• Preserve performance — The static footprint, startup time, memory footprint, and run-time performance of an application running on a modular Compact Profile, JRE, or JDK image must be no worse than it is today, except that footprint metrics may increase in order to accommodate new features.

• Preserve workflows — Existing compilation, build, packaging, and deployment techniques must continue to work as they do today.

The Modular JDK

• Modular structure — Divide the APIs of the Java SE Platform and the JDK, and hence their implementations, into a set of modules.

A module, here, is a self-describing collection of code and data. Its code can include Java class files as well as native code, in the form of dynamically-loadable libraries; its data can include static resource files and user-editable configuration files.

A module specifies the modules upon which it depends, and it may define APIs for use by modules which depend upon it. Like a package, class, or interface it has both a specification and one or more implementations. It is a large-grained unit of compilation, packaging, release, transport, and re-use.

We do not propose to modularize the Java Language Specification, the Java Virtual Machine Specification, or their implementations in the JDK. The Java programming language is, itself, indivisible. A particular Java Virtual Machine implementation may have its own internal modular structure, but we do not see a compelling need to expose such low-level details in the form of modules; doing so would, moreover, be very difficult.

• Medium-grained modular structure — The modular structure of the JDK must support significantly more configuration flexibility than is presently provided by the very coarse-grained Compact Profiles, but it should not be so fine-grained that it is difficult to comprehend or costly to implement.

• Standard, JDK-specific, and JDK-internal modules — There must be a clear distinction between standard Java SE modules, JDK-specific modules, and JDK-internal modules.

The APIs that are part of the Java SE Platform will be divided into a set of standard modules that will be proposed for inclusion in the Platform Specification. The JDK-specific APIs will be placed into a set of JDK-specific modules. The entire set of JDK modules will include the standard modules, the JDK-specific modules, and possibly some JDK-internal modules.

• Refactoring — It must be possible to refactor the set of JDK modules over time, by splitting a module into smaller modules or merging an existing set of modules into a single module, without having to modify any external component that makes use of the original modules.

We expect the Java SE Platform and the JDK to continue to evolve for many years to come, so we must be able to evolve its modular structure in a compatible way that does not force existing applications to change.

• Build-time configuration — Provide a tool which a developer can use to combine modules into a custom binary run-time image that contains a specific configuration of modules, i.e., a set of modules and the modules transitively required by those modules.

The result of this process may be the full JDK itself, with all of its APIs and related tools, or it may be a minimal run-time image that contains only the java command-line launcher and one or a few modules.

• Launch-time configuration — Enhance the java launcher so that a run-time image can be invoked so as to appear to contain only a specific configuration of modules, so long as all of the necessary modules are already present in the image.

This will make it possible to develop, debug, and test an application intended to run on a small run-time image without having to build that image.

• Compile-time configuration — Enhance javac so that, for a given configuration of modules, the types available to the code being compiled are the same as those available in that configuration at build time and run time.

Fidelity across all phases of development is critical to developer productivity.

• Upgradeable modules — If a JDK module implements an endorsed standard or standalone technology then it must be possible to use a version of that module from a later release in any phase, i.e., at compile time, build time, or run time.

Applications that currently require newer versions of the endorsed or standalone technologies included in the JRE can satisfy such needs by using the endorsed-standards mechanism. To do that, however, entails either modifying the JRE itself or setting a system property, and in some scenarios neither of those actions is possible. The mechanism is also brittle and subject to abuse, since all it really does is prepend the overriding jar files to the bootstrap class path.

• Reflection, debugging, and tools — Reflective and diagnostic APIs and tools such the java.lang.reflect package, the javax.lang.model package, stack traces, JVMTI, and Javadoc must be upgraded to convey information about the available modules and, when appropriate, provide the ability to manipulate modules, in much the same manner as they already do for classes.

• OS-specific module packaging — It must be possible for an operating-system provider to use existing tools to package a set of JDK modules into a set of corresponding operating-system-specific packages, in formats such as RPM, Debian, and Solaris IPS, such that the installation of those packages creates a run-time image essentially equivalent to the image that would have resulted from combining those modules at build time.

This requirement does not apply to Windows or Mac OS because those operating systems do not have built-in, general-purpose packaging systems.

• Preserve testability — Modular JDK code must be at least as testable as it is today; the white-box testing of JDK-internal APIs, in particular, must continue to be supported.

• Preserve ease of JDK development — It must not be more difficult to maintain and evolve the JDK code base than it is today; build times, in particular, must not change significantly, and incremental builds must continue to be supported.

We expect that it will, in fact, be substantially easier to maintain and evolve the JDK code base once it has been modularized, but at the very least it must not get any more difficult.

Non-requirements

• Post-build augmentation — It is not necessary to be able to integrate additional JDK modules into an existing run-time image.

• Launch-time replacement — It is not necessary to be able to invoke a run-time image so that it appears to contain a version of a JDK module different from one already present, except as necessary to support upgradeable modules.

• Run-time replacement — It is not necessary to be able to replace a JDK module with a different version of that module after a run-time image has been invoked.

• Mix-and-match — It is not necessary to be able to configure the compiler or the run-time system to contain modules from different releases, or even different builds, of the JDK, except as necessary to support upgradeable modules.

• Preserve run-time image internals — It is not necessary to preserve the current structure and content of JDK run-time images.

The layout and content of the Compact Profile, JRE, or JDK binary images can be changed if there are good reasons to do so. In particular, if rt.jar needs to be replaced in order to meet the requirements laid out here then that is perfectly acceptable.

• Provisioning — It is not necessary to provide a new mechanism by which modules can be published to and downloaded from a server, either by specific request or as they are needed.

Providing a JDK-specific module-publication, download, and install mechanism was a requirement of earlier iterations of Project Jigsaw. This functionality is already provided well enough during development by the dependency-management features of build tools such as Maven, Ivy, and Gradle, while OS-specific packaging systems suffice for deployment and installation.

Security & maintainability

• Strongly encapsulate JDK-internal APIs — The internal APIs of the JDK must be available for use, in any phase, only within the JDK itself, even when a security manager is not present.

Broad use of the JDK’s internal APIs by libraries and applications continues to be both a security risk and a significant maintenance burden. At the moment the only boundary we have around these APIs is the compile-time ct.sym mechanism, which is easy to bypass.

• Limit the use of JDK-internal APIs — A JDK-internal API must be available for use, in any phase, only by those JDK components that actually need to use it, even when a security manager is not present.

Limiting the use of internal APIs even within the JDK itself helps to improve security, by limiting the APIs available to an adversary who cracks into a particular module, and also maintainability, by making it clear exactly which JDK components make use of which internal APIs.

• Obviate the SecurityManager.checkPackageAccess method – The constraints currently enforced via this method must be enforced by other, less brittle, means.

Painful experience has shown that a single omitted invocation of the checkPackageAccess method can amount to a serious, exploitable security vulnerability. (A solution to the previous two requirements will likely satisfy this requirement.)

• Create proper public APIs for commonly-used JDK-internal APIs — Survey existing bodies of source code to learn which JDK-internal APIs are commonly used and, where feasible, create JDK-specific or possibly standard APIs to replace them.

Performance

• Enable ahead-of-time, whole-program optimization techniques — Provide a tool which a developer can use to collect compiled code from a variety of sources, including modules and the application class path, and then analyze and optimize the code as a whole and produce a custom binary run-time image containing the result.

The optimization techniques envisioned here include, but are not limited to: Fast lookup of both JDK and application classes; early bytecode verification; aggressive inlining of, e.g., lambda expressions, and other standard compiler optimizations; construction of JVM-specific memory images that can be loaded more efficiently than class files; ahead-of-time compilation of method bodies to native code; and the removal of unused fields, methods, and classes. These kinds of techniques tend to work best when JDK and application code is analyzed together, prior to run time.

A modular JDK is, strictly speaking, not a prerequisite for such techniques. The build-time configuration of run-time images is, however, a very natural point in the development cycle at which to apply them. The program analyses required by many of these techniques, moreover, can be more effective when it is known that a class can refer only to classes in a few other specific modules rather than to any arbitrary class in the run-time environment.

• Optimize existing code as-is — It must be possible to apply the optimizer tool to existing code already packaged in traditional jar files, without changing those files.

• Actual optimizations — As a proof of concept, provide at least two optimizations that deliver nontrivial performance benefits.

Non-requirements

• Launch-time configuration of optimized images — It is not necessary to be able to invoke an optimized image so that it appears to contain only a specific subset of the modules that it contains.

More-sophisticated whole-program optimization techniques may merge code and even entire API elements across module boundaries, thereby making it impossible to separate the content of one module from another. The launch-time configuration of less-aggressively optimized images will, most likely, still be possible.

Modular libraries & applications

• Developer modules — Provide a means for developers to structure their own code into modular components, each of which specifies the developer modules and the JDK modules upon which it depends.

• Developer modules akin to JDK modules — Developer modules must be able to leverage the facilities available to JDK modules for refactoring, reflection, debugging, testing, optimization, encapsulation, and limitation.

An application structured solely as a set of modules need not make any use, in any phase, of the existing slow and error-prone class-path mechanism.

• Configuration of developer modules — Developer modules must be able to participate equally with JDK modules in the compile-time, build-time, and launch-time configuration processes described above.

In the case of build-time configuration, the result of combining developer modules and JDK modules together at build time will be a custom run-time image that contains the developer modules and just the JDK modules that they require, either directly or indirectly, together with a launcher to invoke the application.

• Launch-time augmentation of developer modules — It must be possible to specify developer modules to the java launcher so that they can participate in the launch-time configuration process, thereby augmenting the set of developer modules and JDK modules already present.

This will provide a modular analogue to the current class-path mechanism.

• Native code in developer modules — It must be possible for a developer module to contain native code which will be loaded and linked at run time, though this may be limited due to native-level restrictions imposed by the underlying operating system.

Such limitations may include, e.g., not creating name clashes with previously-loaded bodies of native code.

• Integrate smoothly with existing tools — A developer who currently uses tools such as Maven, Ivy, or Gradle to build a library or application should be able to use that tool to build the modular form of that library or application with a minimum of fuss.

A dead-simple adoption story around existing build tools is essential. To achieve this may require that we create and contribute custom plugins for at least some of these build tools.

• Multi-mode artifacts — The existing jar-file format must be enhanced so that a module that does not contain native code can be delivered in a single artifact that can be used both on the class path and as a module.

This will allow library developers to continue to produce a single artifact for both class-path and module-based applications.

• Gradual migration of applications — It should be possible for a developer to convert an existing application into modular form in a series of incremental steps, which gradually move code from the class path into modules, rather than one single step, which immediately moves all code into modules.

• OS-specific packaging of developer modules — It must be possible for a developer to use existing tools to package a set of developer modules into one or more OS-specific packages, in formats such as RPM, Debian, and Solaris IPS, such that the modules can, once the package is installed, make use of developer modules and JDK modules installed on the target system in like manner.

As with the earlier requirement to support OS-specific packaging of JDK modules, this requirement does not apply to Windows or Mac OS because those operating systems do not have built-in, general-purpose packaging systems.

• OS-specific packaging of applications — Provide a tool which a developer can use to package a set of developer modules comprising an application, together with the JDK modules that they require, if necessary, into an OS-specific package which can be installed and invoked by an end user in the manner that is customary for the target system.

If the target system is expected already to include the necessary JDK modules then they need not be included in the application package.

This requirement is intended to apply to all target operating systems, including Windows and Mac OS.

Non-requirements

• Provisioning of developer modules — It is not necessary to provide a new mechanism by which developer modules can be published to and downloaded from a server, either by specific request or as they are needed.

As with the provisioning of JDK modules, we see no need to replicate functionality already provided well enough by existing, widely-used systems.

Versioning

If all modules were JDK modules then the requirements for versioning would be very simple. If developers are going to create their own modules, however, then they will reasonably expect to be able to add version information to them and, moreover, use that information to constrain how modules are configured.

• Version strings — A module must be able to indicate a version string.

• Non-prescriptive version strings — Version strings must have a precisely-defined syntax and be totally ordered, but otherwise their format should be as general as possible in order to accommodate existing popular version-string schemes.

• Version strings in reflective APIs — The various reflective and diagnostic APIs must be upgraded to convey the version strings of modules, when present.

• Version constraints — A module must be able to express constraints upon the acceptable versions of the modules upon which it depends.

• Overrideable version information — It must be possible, in all phases, to override the version strings and version constraints of an existing developer module.

This capability allows a developer to make use of modules that may have been published with incorrect version information, yet without having to modify those modules.

• Enforced version constraints — The process of configuring a set of modules must, in all phases, satisfy every version constraint of every module except for constraints that are overridden, in which case the overriding constraint must be satisfied.

Non-requirements

• Multiple versions — It is not necessary to support more than one version of a module within a single configuration.

• Version selection — The process of configuring a set of developer modules and JDK modules, in any phase, need not consider more than one version of any particular module.

In other words, we see no need to build yet another dependency-resolution mechanism. Maven, Ivy, and Gradle have all tackled this difficult problem. We should leave it to these and other build tools to discover and select a set of candidate modules for a given library or application; the module system need only validate that the set of modules selected by the build system satisfies each module’s dependences and version constraints.

Dynamic configuration

The requirements for dynamic configuration are motivated by applications with plug-in or container architectures such as IDEs, test harnesses, and application servers. For the Java EE Platform, in particular, the goal is to enable a future modular war-file standard in which the components in a war file can be developer modules.

• Basic dynamic configuration — It must be possible for a running application to load an additional set of developer modules, relate them in a new configuration to each other and to modules in the application’s existing configuration, invoke them as necessary, and then release them for eventual garbage collection.

• Run-time augmentation of JDK modules — It must be possible to load and configure additional JDK modules after a run-time image has been invoked.

This will allow a container application to load additional JDK modules in order to satisfy the needs of the applications it hosts.

• Multiple dynamic configurations — An application must be able to create and manipulate multiple independent dynamic configurations.

• Isolated dynamic configurations — An application must be able to isolate the code in different dynamic configurations at least as well as is possible today, where this is typically done by using multiple class loaders.

• Nested dynamic configurations — An application must be able to create a dynamic configuration that relates to an existing dynamic configuration rather than to the application’s existing configuration.

• Alternate module versions in dynamic configurations — An application must be able to arrange for a dynamic configuration to include versions of developer modules and upgradeable JDK modules that are different from those already available in the enclosing configuration, and it must be able to control the manner in which such alternate versions are selected.

A canonical example of using multiple versions of an upgradeable API in a Java EE environment is that of a web application requiring a different version of the JAX-WS stack than is already available to the web container.

Acknowledgements

This document includes contributions from Alan Bateman, Alex Buckley, Mandy Chung, Jonathan Gibbons, Chris Hegarty, Karen Kinnear, and Paul Sandoz. It has been improved with the help of comments and suggestions from Brian Goetz, Bill Shannon, and Henrik Ståhl.