JEP 299: Reorganize Documentation

Summary

Update the organization of the documents in the JDK, in both the source repositories and the generated docs.

Goals

• Formally define an organization for the generated "docs" image, to include API specifications, "man pages" (which can be considered to be specifications for tools), and other JDK specifications.
• Consolidate the current 20+ sets of documentation generated by the javadoc tool into a single collection of API specification for a JDK image.
• Define an organization for non-API specifications to be present in the source repositories, such that they can be updated as needed along with the source code, and can easily be included in the generated "docs" image.

Non-Goals

• It is not a goal (it is an anti-goal) to change any of the processes or procedures by which it is decided to update any specifications. This includes JCP specifications, such as the API specifications, and related standards.
• It is not a goal to include all specifications in this work. For example, JLS and JVMS are not included in this proposal.
• It is not a goal to support documentation that is not a specification.
• Although it is a goal to define an organization that can accommodate man pages, it is not a goal to provide man pages for JDK 9.

Motivation

Currently, a standard build of the JDK "docs" target runs the javadoc tool over 20 times (22 on Linux, 25 on Solaris), to generate a corresponding number of different sets of documentation, organized in no obviously discernible manner. Even though javadoc documentation now provides a "Search" feature, having multiple sets of documentation implies that you cannot easily search all the provided documentation at once.

Here is a listing of the current collection of documentation sets; each entry corresponds to a separate invocation of the javadoc tool. The first entry (api) is the one that most people are familiar with: it is the Java SE Platform Specification.

api
jdk/api/attach/spec
jdk/api/dynalink
jdk/api/javac/tree
jdk/api/javadoc/doclet
jdk/api/javadoc/old/doclet
jdk/api/javadoc/old/taglet
jdk/api/jconsole/spec
jdk/api/jlink
jdk/api/jpda/jdi
jdk/api/jshell
jdk/api/nashorn
jre/api/accessibility/jaccess/spec
jre/api/management/extension
jre/api/net/httpserver/spec
jre/api/net/socketoptions/spec
jre/api/nio/sctp/spec
jre/api/plugin/dom
jre/api/plugin/jsobject
jre/api/security/jaas/spec
jre/api/security/jgss/spec
jre/api/security/smartcardio/spec

Although there is a superficial overall organization, the varying depths, the varying position of the api component, the inconsistent use of spec, all make it difficult to determine what documentation sets exist, and what should be the relative paths between related specifications. Currently, the only "index" into these pages is implicit in the so-called "brick wall" picture.

It would be good to significantly reduce the number of different sets of documentation, and to organize them in a well-defined manner, such that it is reasonable to create links between and into the sets of documentation.

These various runs of the javadoc tool are currently controlled by logic in the files make/common/CORE_PKGS.gmk and make/common/NON_CORE_PKGS.gmk. Updating these files is error-prone and sometimes overlooked. For example, there are currently four supported packages that are not included in any public documentation. It would be better to automatically derive the list of packages to be documented from the list of packages exported by the modules being documented. This would mean that any time a new package is listed as being exported from a module, it would automatically included in any documentation that includes the documentation for the module.

In addition, OpenJDK does not provide "man pages" for the tools it provides, even though they can be considered the specification of the tools' command lines. It would be good to incorporate such specifications into the repositories, such that they can be updated alongside the corresponding tools when necessary, and corresponding pages generated and places in a well-defined manner in the overall documentation bundle. The same can also be said for various additional specifications, such as the JNI Specification, or the Javadoc Tag Specification, which currently have no well-defined home.

Finally, the current documentation bundle is effectively "all or nothing". If we organized any documentation in the repositories such that it can be associated with the module to which it applies, we would be able to build images containing specific subsets of images, together with corresponding documentation. For example, if we build images for the different Compact Profiles, we could easily build and publish the corresponding documentation.

Description

Consolidated API documentation

The consolidation of the generated API documentation requires changes to the makefiles such that we identify the modules containing the API to be documented and generate a bundle containing all those modules for which all of their exported API should be documented. If that does not cover all the API documentation that is currently generated, we should define the organization of the additional sets of documentation. For example, define that the various sets exist as siblings within a single top-level "api" directory.

docs/api/<doc-set>/

<doc-set> should be a general name to describe the corresponding contents, such as "jdk" or "java.se".

'Man pages'

Source code within a JDK repository is typically organized as follows:

src/<module-name>/{share,<os-name>}/{classes,native,...}

It is proposed to extend the last such component with a new variant:

src/<module-name>/{share,<os-name>}/man

The man directory should contain the source for the man pages for tools in the enclosing module. Such files should use Markdown syntax, and be named for the corresponding tool. For example, the source for the man page for the javac tool should be in src/jdk.compiler/share/man/javac.md.

The build system should generate corresponding files in a new top-level directory in the generated docs directory. The generated files could either be in HTML, or could be in "man" format for those systems that support that. For example, the generated man page for the javac tool could be in

docs/man/javac.html
docs/man/man1/javac.1

Additional specifications

Some modules may have some additional associated specifications, that is not otherwise included in API specifications or man pages. For example, documentation comments are generally written according to the "Javadoc Tag Specification", which should be updated in conjunction with updates to the javadoc tool. (It is currently available within the Javac Tool Guide at the somewhat unmemorable URL [https://docs.oracle.com/javase/8/docs/technotes/tools/unix/javadoc.html#CHDFCBAD])

Here are some additional specifications that may be considered for inclusion:

• JavaBeans spec
• Input Methods Framework Specification
• Jar File Specification
• Java Native Interface Specification
• Java Remote Method Invocation

Such specifications should be placed in another new module-specific directory:

src/<module-name>/{share,<os-name>}/specs

Files in such a directory should be copied to a specs directory in the generated docs directory, with the exception of Markdown (.md) files, which should be translated into HTML files. If a specification consists of more than a single file, it is recommended that all the files should be grouped into an appropriate subdirectory.

Index

The build should generate a simple minimal top-level index.html file that links to each item in the overall generated documentation. This should be enough for simple navigation in a basic "docs" build, but may be overwritten as needed when providing a richer documentation bundle.

Formats

HTML files should be in either HTML 5 or HTML 4.01 format, and should pass validators such as tidy, link checkers, and accessibility checkers. Over time, we should migrate all older HTML files to HTML 5, because of the support for accessibility-related features.

Markdown files should be in one of the following formats: Markdown; CommonMark; or Github Flavored Markdown, which provides useful extensions for definition lists, tables, and syntax highlighting of code blocks.

Tools

It is proposed that we use the open source pandoc tool to convert Markdown files into HTML or man page (groff) format. This would mean some new tool dependencies for the build. pandoc supports all the Markdown variants listed in the previous section.

Summary

New source subdirectories:

src/<module-name/{share,<os-name>}/
    man
    specs

New generated docs directory:

docs/
    api/
        <doc-set-1>
        <doc-set-2>
        etc
    man/
        <HTML man pages>
        man1/
            <man pages in man format>
    specs/
        <HTML spec pages>
        <directories for multi-file specifications>