JEP 222: jshell: The Java Shell (Read-Eval-Print Loop)

Summary

Provide an interactive tool to evaluate declarations, statements, and expressions of the Java programming language, together with an API so that other applications can leverage this functionality.

Goals

The JShell API and tool will provide a way to interactively evaluate declarations, statements, and expressions of the Java programming language within the JShell state. The JShell state includes an evolving code and execution state. To facilitate rapid investigation and coding, statements and expressions need not occur within a method, and variables and method need not occur within a class.

The jshell tool will be a command-line tool with features to ease interaction including: a history with editing, tab-completion, automatic addition of needed terminal semicolons, and configurable predefined imports and definitions.

Non-Goals

A new interactive language is not the goal: All accepted input must match grammar productions in the Java Language Specification (JLS). Further, within an appropriate surrounding context, all accepted input must be valid Java code (JShell will automatically provide that surrounding context -- the "wrapping"). That is, if X is an input that JShell accepts (as opposed to rejects with error) then there is an A and B such that AXB is a valid program in the Java programming language.

Out of scope are graphical interfaces and debugger support. The JShell API is intended to allow JShell functionality in IDEs and other tools, but the jshell tool is not intended to be an IDE.

Motivation

Immediate feedback is important when learning a programming language and its APIs. The number one reason schools cite for moving away from Java as a teaching language is that other languages have a "REPL" and have far lower bars to an initial "Hello, world!" program. A Read-Eval-Print Loop (REPL) is an interactive programming tool which loops, continually reading user input, evaluating the input, and printing the value of the input or a description of the state change the input caused. Scala, Ruby, JavaScript, Haskell, Clojure, and Python all have REPLs and all allow small initial programs. JShell adds REPL functionality to the Java platform.

Exploration of coding options is also important for developers prototyping code or investigating a new API. Interactive evaluation is vastly more efficient in this regard than edit/compile/execute and System.out.println.

Without the ceremony of class Foo { public static void main(String[] args) { ... } }, learning and exploration is streamlined.

Description

Functionality

The JShell API will provide all of JShell's evaluation functionality. The code fragments that are input to the API are referred to as "snippets". The jshell tool will also use the JShell completion API to determine when input is incomplete (and the user must be prompted for more), when it would be complete if a semicolon were added (in which case the tool will append the semicolon) and also how to complete input when completion is requested with a tab. The tool will have a set of commands for query, saving and restoring work, and configuration. Commands will be distinguished from snippets by a leading slash.

Documentation

The JShell module API Specifications can found here:

• http://download.java.net/java/jdk9/docs/api/jdk.jshell-summary.html

Which includes the primary JShell API (jdk.jshell package) Specification:

• http://download.java.net/java/jdk9/docs/api/jdk/jshell/package-summary.html

The jshell tool reference:

• https://docs.oracle.com/javase/9/tools/jshell.htm

is part of the Java Platform, Standard Edition Tools Reference:

• https://docs.oracle.com/javase/9/tools/tools-and-command-reference.htm

Terms

In this document, the term “class” is meant in the sense used in the Java Virtual Machine Specification (JVMS), which includes Java Language Specification (JLS) classes, interfaces, enums, and annotation types. The text makes it clear if a different meaning is intended.

Snippets

A snippet must correspond to one of the following JLS syntax productions:

• Expression
• Statement
• ClassDeclaration
• InterfaceDeclaration
• MethodDeclaration
• FieldDeclaration
• ImportDeclaration

In JShell, a "variable" is a storage location and has an associated type. A variable is created explicitly with a FieldDeclaration snippet:

int a = 42;

or implicitly by an expression (see below). Variables have a small amount of field semantics/syntax (for example, the volatile modifier is allowed). However, variables have no user-visible class enclosing them and will be generally viewed and used like local variables.

All expressions are accepted as snippets. This includes expressions without side effects, such as constants, variable accesses, and lambda expressions:

1
a
2+2
Math.PI
x -> x+1
(String s) -> s.length()

as well as expressions with side effects, such as assignments and method invocations:

a = 1
System.out.println("Hello world");
new BufferedReader(new InputStreamReader(System.in))

Some forms of expression snippet implicitly create a variable to store the expression's value so it can be referred to later by other snippets. By default, an implicitly created variable has the name $X where X is the snippet identifier. A variable is not implicitly created if the expression is void (the println example), or if the value of the expression can already be referred to by a simple name (as in the case of 'a' and 'a=1' above; all other examples have variables implicitly created for them).

All statements are accepted as snippets, except for 'break', 'continue', and 'return'. However, a snippet may contain 'break', 'continue', or 'return' statements where they meet the usual rules of the Java programming language for enclosing context. For example, the return statement in this snippet is valid because it is enclosed in a lambda expression:

() -> { return 42; }

A declaration snippet (ClassDeclaration, InterfaceDeclaration, MethodDeclaration, or FieldDeclaration) is a snippet that explicitly introduces a name that can be referred to by other snippets. A declaration snippet is subject to the following rules:

• the access modifiers (public, protected, and private) are ignored (all declaration snippets are accessible to all other snippets)
• the modifier final is ignored (future changes/inheritance are permitted)
• the modifier static is ignored (there is no user-visible containing class)
• the modifiers default and synchronized are not allowed
• the modifier abstract is allowed only on classes.

All snippets, except for those of the form ImportDeclaration, may contain nested declarations. For example, a snippet that is a class instance creation expression may specify an anonymous class body with nested method declarations. The usual rules of the Java programming language apply to modifiers on nested declarations, rather than the rules above. For example, the class snippet below is accepted, and the private modifier on the nested method declaration is respected, so the snippet "new C().secret()" would not be accepted:

class C {
  int answer() { return 2 * secret(); }
  private int secret() { return 21; }
}

A snippet may not declare a package or a module. All JShell code is placed in a single package in an unnamed module. The name of the package is controlled by JShell.

Within the jshell tool, the terminal semicolon of a snippet can be omitted if that semicolon would be the last character of the input (excluding whitespace and comments).

State

The JShell state is held in an instance of JShell. A snippet is evaluated in a JShell with the eval(...) method, producing an error, declaring code, or executing a statement or expression. In the case of a variable with an initializer, both declaration and execution occur. An instance of JShell contains previously-defined and modified variables, methods, and classes, previously-defined import declarations, the side-effects of previously-entered statements and expressions (including variable initializers), and external code bases.

Modification

Since the desired use is exploration, the declarations (variables, methods, and classes) must be able to evolve over time while, at the same time, preserving evaluated data. One choice would be to make a changed declaration a new additional entity in some or all cases, but that is certain to be confusing and does not play well with exploring the interaction between declarations. In JShell, each unique declaration key has exactly one declaration at any given time. For variables and classes the unique declaration key is the name, and, the unique declaration key for methods is the name and the parameter types (to allow for overloading). As this is Java, variable, methods, and classes each have their own name spaces.

Forward reference

In the Java programming language, within the body of a class, references to members which will appear later can appear; this is a forward reference. As code is entered and evaluated sequentially in JShell, these references will be temporarily unresolved. In some cases, for example mutual recursion, forward reference is required. This can also occur in exploratory programming while entering code, for example, realizing that another (so far unwritten) method should be called. JShell supports forward references in method bodies, return type, and parameter types, in variable type, and, within a class. Since the semantics requires them to be immediately executed, forward references in variable initializers is not supported.

Snippet dependencies

The code state is kept up-to-date and consistent; that is, when a snippet is evaluated, any changes to dependent snippets are immediately propagated.

When a snippet is successfully declared, the declaration will be one of three kinds: Added, Modified, or Replaced. A snippet is Added if it is the first declaration with that key. A snippet is Replaced if its key matches a previous snippet, but their signatures differ. A snippet is Modified if its key matches a previous snippet and their signatures match; in this case, no dependent snippets are impacted. In both the Modified and Replaced cases the previous snippet is no longer part of the code state.

When a snippet is Added it may be providing an unresolved reference. When a snippet is Replaced it may update an existing snippet. For example, if a method's return type is declared to be of class C and then class C is Replaced then the method's signature has changed and the method must be Replaced. Note: This can cause previously-valid methods or classes to become invalid.

The desire is that user data persist whenever possible. This is attained except in the case of variable Replace. When a variable is replaced, either directly by the user or indirectly via a dependency update, the variable is set to its default value (null since this can only occur with reference variables).

When a declaration is invalid, either because of a forward-reference or becoming invalid through an update, the declaration is "corralled". A corralled declaration can be used in other declarations and code, however, if an attempt is made to execute it a runtime exception will occur which will explain the unresolved references or other issues.

Wrapping

In the Java programming language, variables, methods, statements, and expressions must be nested within other constructs, ultimately a class. When the implementation of JShell compiles a variable, method, statement, and expression snippet as Java code an artificial context is needed, as follows:

• Variables, Methods, and Classes
  • As static members of a synthetic class
• Expressions and Statements
  • As expressions and statements within a synthetic static method within a synthetic class

This wrapping also enables snippet update, so, note that a snippet class is also wrapped in a synthetic class.

Modular Environment Configuration

The jshell tool has the following options for controlling the modular environment:

• --module-path
• --add-modules
• --add-exports

The modular environment can also be configured by direct addition to the compiler and runtime options. Compiler flags may be added with the -C option. Runtime flags may be added with the -R option.

All jshell tool options are documented in the Tool Reference (see above).

The modular environment can be configured at the API level with the compilerOptions and remoteVMOptions methods on JShell.Builder.

The set of modules read by JShell's unnamed module is the same as the default set of root modules for the unnamed module, as established by JEP 261 "Root modules":

• http://openjdk.java.net/jeps/261

Naming

• Module
  • jdk.jshell
• Tool launcher
  • jshell
• API package
  • jdk.jshell
• SPI package
  • jdk.jshell.spi
• Execution engine "library" package
  • jdk.jshell.execution
• Tool launching API package
  • jdk.jshell.tool
• Tool implementation package
  • jdk.internal.jshell.tool
• OpenJDK Project
  • Kulla

Alternatives

A simpler alternative is just to provide a batch scripting wrapper without interactive/update support.

Another alternative is to maintain the status quo: Use another language or use a third-party REPL such as BeanShell, though that particular REPL has been dormant for many years, is based on JDK 1.3, and makes arbitrary changes to the language.

Many IDEs, for example the NetBeans debugger and BlueJ's CodePad, provide mechanisms to interactively evaluate expressions. Preserved context and code remains class-based, and method granularity is not supported. They use specially crafted parsers/interpreters.

Testing

The API facilitates detailed point testing. A test framework makes writing tests straight-forward.

Because the evaluation and query functionality of the tool is built on the API, most testing is of the API. Command testing and sanity testing of the tool is, however, also needed. The tool is built with hooks for a testing harness, which is used for tool testing.

Tests are comprised of three parts:

1. Tests for the API. These tests cover both positive and negative cases. Each public method must be covered by tests which include adding variables, methods, and class, redefining them, etc.

2. Testing of the jshell tool. These tests check that jshell commands and compilation, and execution of Java code, have correct behavior.

3. Stress testing. To ensure that JShell can compile all allowed Java snippets, correct Java code from the JDK itself will be used. These tests parse the sources, feed code chunks to the API, and test the behavior of the API.

Dependences

The implementation will make every effort to leverage the accuracy and engineering effort of the existing language support in the JDK. The JShell state is modeled as a JVM instance. Code analysis and the production of executable code (jdk.jshell API) will be performed by the Java Compiler (javac) through the Compiler API. Code replacement (jdk.jshell.execution) will use the Java Debug Interface (JDI).

Parsing of raw snippets (i.e., snippets that have not been wrapped) will be done using the Compiler API with a small subclassing of the parser to allow raw snippets. The resulting information will be used to wrap the snippet into a valid compilation unit including a class declaration with imports for previously evaluated code. Further analysis and generation of the class file will be done with unmodified instances of the Java compiler. Generated class files will be kept in memory and never written to storage. jdk.jshell.spi SPI exists to configure the execution engine. The default execution engine behaves as follows. Class files will be sent over a socket to the remote process. A remote agent will handle loading and execution. Replacement will be done via the JDI VirtualMachine.redefineClasses() facility.

Tab-completion analysis (jdk.jshell API) will also use the Compiler API. Completion detection will use the javac lexer, custom and table-driven code.

The jshell tool (jdk.internal.jshell.tool) will use 'jline2' for console input, editing, and history. jline2 has been, privately, rolled into the JDK.