JEP 126: Lambda Expressions & Virtual Extension Methods

Summary

Add lambda expressions (closures) and supporting features, including method references, enhanced type inference, and virtual extension methods, to the Java programming language and platform.

Goals

The primary features of lambda expressions and virtual extension methods, along with their set of secondary supporting features, further several platform goals:

• Simplifying the creation and consumption of more abstract, higher-performance libraries
• Supporting smoother library evolution with migration compatibility

Besides adding a now-common feature to the Java programming language, lambda expressions open up possibilities for improved multicore support by enabling internal iteration idioms.

The supporting language features around lambda include virtual extension methods, which will allow interfaces to be evolved in a source and binary compatible fashion.

In addition to language changes, coordinated libraries and JVM changes will occur as well.

Note that the active and ongoing Project Lambda OpenJDK project pre-dates the JEP process, as does the corresponding JSR, JSR 335, which is targeted for Java SE 8 (JSR 336).

Non-Goals

The language features of function types and general control abstraction are not goals of adding lambda expressions to Java. However, the intention is to not preclude the addition of such features in the future.

Motivation

Many other commonly used object-oriented programming languages, including those hosted on the JVM (for example Groovy, Ruby, and Scala) and hosted on other virtual machines (C# on the CLR), include support for closures. Therefore, Java programmers are increasingly familiar with the language feature and the programming models it enables.

Of particular interest is enabling the idiom of internal iteration. Arrays and collections currently support external iteration, where the control logic of an iteration lives outside of the data structure being traversed. For example, a for-each loop over an array or a collection is an example of external iteration. The semantics of the for loop in Java mandates strict serial iteration, which means that the only means the programmer has of iterating over a standard collection will not allow the use of all the available cores.

With internal iteration, the data structure is passed a piece of code to execute, as a lambda expression, and the data structure is responsible for partitioning the computation and reporting the results. Since the data structure is familiar with its own internal details it can potentially choose a better scheduling of the computation by tuning options such as

• Alternate execution order
• Concurrent execution using thread pools
• Parallel execution using partitioning and work stealing. The fork/join framework, added in Java SE 7, is one such candidate parallel execution framework and offers performance-robust partitioning of work over a wide range of core counts.

A prototypical example of the internal iteration style is a sequence of filter-map-reduce operations such as:

int maxFooWeight =
    collection.filter( /* isFoo Predicate as a lambda */)
              .map(    /* Map a Foo to its weight with a lambda */)
              .max();  /* Reduction step */

The lambda expressions are expressions with concise syntax to represent the desired operation. This style is in lieu of one or more explicit for loops which would unnecessary constrain the iteration order over the collection. In addition, a well-designed algorithm can not only execute these sets of operations in parallel, but can also aggregate the three operations into a single parallel pass.

Project Lambda also includes virtual extension methods, which will address the long-standing limitation of not being able to add methods to widely-used interfaces because of source compatibility concerns.

By adding extension methods to the existing collection interfaces, such as java.util.Collection and java.util.List, existing implementations of those types can participate in the new programming idiom. Implementations of those types in the JDK (and elsewhere) can override the default implementations of the extension methods from the superinterfaces to provide higher-performance or otherwise specialized implementations.

Description

The in-progress OpenJDK project page for Project Lambda and corresponding JSR have the most up-to-date information about the details of the effort.

The platform components impacted by the overall Project Lambda effort include:

• The specification of the Java programming language
• The specification of the Java virtual machine
• The reference implementation of the language changes in javac
• Class file changes to support compilation of lambda expressions and virtual extension methods
• (Possible) JVM enhancements to improve execution of lambdas
• JVM changes to support virtual extension methods
• Core API changes to add virtual extension methods
• Core API changes to support use of lambda expressions
• Updates to the JDK libraries to use the new extension methods
• Updates to the reflective APIs, such as core reflection and javax.lang.model, to expose lambda and extension method related information
• Updates to class file tools like javap and pack200/unpack200 to understand new JVM attributes
• (Possible) serialization updates, including IIOP serialization
• Enhancements to javadoc to indicate which interfaces can be used for lambdas
• Language runtime in java.lang.* to support the translation of lambda expressions and virtual extension methods

Alternatives

Project Lambda started with a straw-man proposal and has gone through several "State of the Lambda" iterations which have evolved both the syntax and feature set of the project.

The work on lambda expression portion of Project Lambda is informed by numerous previous efforts, including but not limited to:

• BGGA,
• CICE
• FCM
• Pizza

The feature set of Project Lambda is judged to better address the needs of evolving the Java platform than the earlier proposals.

The design of virtual extension methods is likewise informed by a large body of prior work in the programming language community including:

• multiple inheritance in C++
• static extension methods in C#
• traits in Fortress
• traits in Scala
• mixins in Strongtalk

Compared to other language environments, the Java language has always had much more predictable semantics, built-in primitive types are of a known size, exceptions are thrown at precise locations, the order of execution of expressions is precisely defined, and so on. Loosening the semantics of the built-in for and while loops to allow auto-parallelization attempts was judged to be neither desirable nor sufficient to meet the goals of supporting multi-core.

Testing

In addition to unit/regression tests, as a new language feature JCK tests will also be developed.

The utility of the feature will also be validated through usage in the JDK platform libraries.

Risks and Assumptions

As a large effort touching many aspects of the platform, there is the possibility for unforeseen complications and interactions to arise.

By starting early and planning for several iterations of, say, the supporting libraries work, the impact of those complications should be mitigated.

Dependences

The currently preferred implementation approach for lambda expressions relies on invokedynamic and method handles introduced by JSR 292. Therefore, acceptable performance of lambda depends on acceptable performance of method handles, amongst other factors.

There is expected to be feedback between the libraries work (JEPs 107 and 109) and the language features.

Impact

• Other JDK components: The impact of other JDK components is outlined in the description section.
• Compatibility: Virtual extension methods are intended to allow source compatible evolution of interfaces.add virtual extension methods
• Security: Security review will be needed on aspect of the runtime implementation of lambdas.
• Performance/scalability: Performance of lambda-friendly idioms versus traditional idioms should be tracked.
• Documentation: User guides and supporting documentation needs to be written.
• TCK: As a large platform feature, language and library TCKs need to be developed.