JEP 109: Enhance Core Libraries with Lambda

AuthorsStuart Marks, Mike Duigou
OwnerStuart Marks
TypeFeature
ScopeSE
StatusClosed / Delivered
Release8
Componentcore-libs
JSR335
Discussioncore dash libs dash dev at openjdk dot java dot net
EffortM
DurationL
BlocksJEP 107: Bulk Data Operations for Collections
JEP 155: Concurrency Updates
DependsJEP 126: Lambda Expressions & Virtual Extension Methods
Endorsed byBrian Goetz
Created2011/09/28 20:00
Updated2015/02/13 19:41
Issue8046099

Summary

Enhance the Java core library APIs using the new lambda language feature to improve the usability and convenience of the library.

Goals

The primary goal is to modernize the general library APIs by adding the use of Lambda in suitable locations. Most implementations will be provided as extension methods upon existing classes. We will target high-traffic areas of the library and add Lambda APIs where we think it will have the most benefit. Ideally, Lambda would appear in the API wherever a programmer familiar with Lambda would expect it to be. Alternatively, we'd like mainstream programmers to stumble over Lambda APIs in the library and think, "Oh cool, they added a Lambda here, and that lets me solve my problem more easily."

A secondary goal is to inform the design of Lambda language feature by using Lambda in library APIs, calling them from real code, evaluating the results, and providing feedback to the Lambda language/compiler team.

Goals can be summarized as follows:

Non-Goals

It is not a goal to use Lambda in every possible place it could be used, nor will the scope extend beyond core libraries. For example, client, XML, and CORBA are not covered by this effort.

Very little new functionality will be added to the core classes with this enhancement -- only new ways of doing familiar tasks.

There are no specific goals for the proportion of APIs that are upgraded to use Lambda, that is, nothing like "we will add Lambda to xx% of the library."

Success Metrics

Success will be evaluated based upon the degree to which the new APIs and features are adopted by developers. For complete success use of the Lambda features will become the default idiom for using the core libraries by Java developers.

Motivation

"Language design is library design.
Library design is language design." -- Andrew R. Koenig

Java 8 will include a new language feature called Lambda. Having this feature in the language is useful, but with just language changes for Lambda, the platform is incomplete. It will make the platform much more valuable to have Lambda support added to appropriate areas of the library API.

In the past several years, a variety of new programming languages have emerged and are gaining popularity. Most of these languages have some kind of block, closure, or first-class function construct. While Java is still the #1 programming language, the common view is that it hasn't kept up with recent developments in programming languages. This is well recognized as a motivation for the Lambda language feature itself. However, it's also necessary to consider library APIs that support the use of Lambda. The alternative languages all have libraries, and their APIs are tuned to work well -- smoothly and idiomatically -- with the closures or function objects provided by the language.

Similarly, with Java, we expect that as time progresses the use of Lambda will become widespread and various coding idioms will develop around this feature. The Java library APIs will need to be enhanced to support idiomatic usage with Lambda alongside their current, conventional usage.

Description

This project has two phases:

  1. A survey phase to identify candidates for Lambda API enhancements; and

  2. A scoping and implementation phase to prioritize and select a subset of the candidates and to implement them.

Candidate Survey

There are several approaches to discovering candidate sites for adding Lambda-based APIs. One approach is to examine other systems with similar language features and to look at their libraries and see how they use closures and functions.

Consider Ruby for instance. The Ruby language supports a variety of first class function-like constructs. These are useful in and of themselves, in that they enable programmers to use a functional style of programming, to create higher order functions, to compose functions, etc. In addition, since the Ruby class library was developed along with a language that supports first-class functions, many APIs in the class library make use of them. We can look through the Ruby class libraries to find examples that we can use as inspiration for potential enhancements to the Java libraries. Some of these examples from Ruby's class library include the following:

It's clear that there are many opportunities for using Lambda outside the collections classes. There is no requirement to implement Lambda-based APIs everywhere that Ruby does, but given Ruby's popularity and mindshare it seems reasonable to look to Ruby's class library for initial ideas. It might also be reasonable to look examine other systems, such as Groovy, Scala, Python, Clojure, and Smalltalk to find similar inspiration for Lambda use in library APIs.

Another approach is to look through existing Java code -- both within the library and outside, e.g. from the Qualitas Corpus -- and do pattern matching or synthesis to discover candidates. A set of techniques might be as follows:

Scoping and Implementation

The resulting set of candidates will probably be too long to implement in the JDK 8 time frame. They will need to be prioritized and then truncated to fit into the available schedule, with a given set of resources (people) allocated to the project. Prioritization can be according to subjective importance of the library, but it would be good to have some usage data to back it up. For example, it might be useful to do a survey over the Qualitas Corpus to determine which areas of the library are the most heavily used.

Language Design Issues

During API development using Lambda, the following issues should be considered and relevant information should be fed back to the Lambda language design team.

  1. Exception transparency: do you find a need to write lambda expressions that can throw checked exceptions? do you need extra sets of SAM types that have 'throws' clauses, maybe generic 'throws E'? is the result extremely tedious?

  2. Variance: Utility methods for functional interfaces will probably make heaver use of wildcards than anything else in the API; is this extremely tedious? Would it be useful to have better/less verbose variance support?

  3. Unboxed overriding: Would it be useful in the design of functional interfaces to allow primitive/void return types to override reference return types, such as Predicate<T> <: Function<T, Boolean> for example?

  4. Abstract class functional interfaces: In the previous item about abstract classes for, is it the case that there are so many useful candidates that it would be nice to directly support abstract classes as functional interfaces, rather than having to manually define a subclass?

  5. Generic-method functional interfaces: Do you find it useful to have a functional interface that has a generic method, like: interface MapFactory { <K,V> Map<K,V> make(); }. Note the significant point here is that it is the method that is generic, not the interface. This is not currently supported; the benefit is that new type arguments can be inferred for every invocation, which may be useful in some applications.

  6. Chaining inference: In expressions like 'foo().bar(23)', the type arguments for 'foo' are inferred independently of the 'bar(23)' part of the expression. We plan to improve the use of context in inference for many cases, but we're still looking for some experience that would justify taking it this far.

  7. Method reference disambiguation: Do you find a need to explicitly disambiguate method references (when the method is overloaded or there's a static/instance clash), either by writing a full signature or giving up and using a lambda expression instead?

Testing

Tests will need to be developed for each new API that is added to the system. New APIs are largely independent of each other, so it should be simple to test them individually. Furthermore, the API enhancements are likely to consist largely of adding new methods that do not impact other APIs in the same class. So, testing of these new APIs should be fairly straightforward and should not impact existing tests that use current APIs.

Risks and Assumptions

Adding an additional programming idiom increases the complexity for new users. If Lambda APIs are added piecemeal all over the system, they may tend to diverge in style. It would probably be beneficial to do a survey first and to develop a consistent style that is applied across the library.

One point that mitigates risk is that since the API enhancements are largely independent of each other, it is possible to change scope without too much project impact. That is, consider the scope to be a prioritized list of API enhancements. A line will be drawn at a certain point depending upon project schedule and staffing. If this line needs to move for any reason, this should have no impact on work already done, and little replanning should be necessary because the items are largely independent of each other. For the same reason, it should also be possible to reprioritize the list without too much effort, should new information become available while the project is underway.

Using these new APIs intermixed with the existing APIs has the prospect of making user code bases more complicated to maintain.

Dependences

This work depends only on the actual lambda implementation.

Even this is a fairly soft dependency, as lambdas are expressed as SAMs in the API. It's possible to add functional-interface-based APIs even before lambda has been integrated, though it won't be possible for callers to use lambda at that point.

Impact