Governance Board Minutes: 2007/8/23

The third meeting of the OpenJDK Governance Board took place via conference call on Thursday, 23 August 2007 at 15:00 UTC. The topic was the OpenJDK Community TCK License, which had been announced two weeks prior.

All GB members were present: Doug Lea, Fabiane Nardon, Dalibor Topic, Simon Phipps, and Mark Reinhold. Carla Schroer of Sun, who was deeply involved in the crafting of the new license, joined the call in order to answer questions and provide context.

The meeting was essentially a long question-and-answer session, with no agreed resolutions, so these minutes are structured in a Q&A format in the following sections:

1	License-application process
2	License eligibility
3	Running the TCK
4	Certifying a platform implementation
5	Conﬁdentiality
6	Miscellaneous

On the whole the GB membership was pleased with the new TCK license. Most members would like to see the TCK itself open-sourced in the long term, and also made available to certify any platform implementation rather than only those substantially derived from the OpenJDK code base. The license was, nonetheless, generally praised as a big step in the right direction and something with which most Linux distributors, in particular, will be able to work.

1 License-application process

Q1	When will the online TCK license-application process be available? There are developers who want to start working with the TCK as soon as possible.

Sun’s TCK team and the JCP Program Management Ofﬁce are working on that. It’s been delayed a bit by other responsibilities, but it should be done in the next few weeks. [Note: As of early December the online application page is not yet available, but interested parties have been able to execute the license agreement and acquire the TCK by contacting Sun directly.]

Q2	How long should it take for someone to receive the TCK once they’ve submitted an online application?

It depends partly on how many people apply; a large number would slow things down. With a moderate number it should be possible to approve an application in under a month. The TCK will be delivered to licensees using the same mechanism aleady in place for Sun’s commercial partners.

Q3	Is the OpenJDK TCK license meant to be executed by individuals, by organizations, or either?

It’s written to accommodate both cases. Which one is best depends upon your speciﬁc situation.

Q4	Suppose that a student at a university is working with the OpenJDK code and wants to get the TCK. Should he sign his own license, or should the university sign for him? What’s the best way to handle that kind of situation?

Students at universities are different than employees of corporations because they aren’t employees of the university and often do not have the ability to control access to university computers. That said, the recommendation is that students sign the TCK license as individuals if they want to put the TCK on computers that they can control. If a group of students is working on a project together then they can sign a single license and ﬁll in the names of all the students working on the project. This could include a professor too, if that’s appropriate. In other words, students should act as individuals or as groups of individuals, and install the software on computers they control.

2 License eligibility

Q5	What about hybrid implementations, for example what the Cacao folks are working on, where they’re using the OpenJDK class libraries on top of their own VM to run on PowerPC or S/390 machines? Would they be eligible for the TCK license?

Absolutely! As long as they’re GPL-licensed and contain substantial amounts of OpenJDK code then it’s in everybody’s interest to help them build a compatible implementation.

Q6	What does it mean for an implementation to be “substantially derived,” as the license says, from the OpenJDK code? For example, would the Classpath library combined with the HotSpot VM count as “substantially derived?”

There’s no precise deﬁnition of “substantial.” It can’t be expressed in terms of, e.g., a speciﬁc number of lines of code. It’s more of a principle which says that you can’t just take a few small things from OpenJDK into your own code base and claim that your code is “substantially derived.”

The Classpath library running with the HotSpot VM would certainly count as substantially derived, since the HotSpot VM is a signiﬁcant chunk of code.

Another obvious kind of case is where someone has replaced a few packages with alternative implementations, e.g., to customize for a particular operating system or windowing environment. That would also be considered to be substantially derived.

Q7	Suppose it’s 2008. The Classpath library has moved to GPLv3, and someone ships it with the VM from Apache Harmony. Does that count as “substantially derived?”

No. Aside from the likely absence of signiﬁcant amounts of OpenJDK code, such a combination would not be considered derived because the OpenJDK TCK license requires that a tested implementation be released under the same license as the OpenJDK code itself, which at the moment is GPL version 2 only.

Q8	Would the Harmony class libraries combined with the HotSpot VM be considered “substantially derived?”

In terms of the code, yes, this combination would be considered to be substantially derived.

To be eligible for the OpenJDK TCK License, however, you’d have to be planning to release the whole combination under GPLv2, and that’s likely a problem. Neither Apache, the FSF, nor Sun view the Apache license as compatible with GPLv2, at least not if you’re incorporating Apache code as-is. You’d have to decide for yourself whether such a combination is legitimate.

3 Running the TCK

Q9	So I get the license application approved, I download the TCK from Sun, and then I can run it locally against my implementation, right?

Right.

Q10	Are the TCK tests fully automated, or are they interactive?

The selection of tests and logging of results is fully automated. Most of the tests themselves are automated, but some of the GUI tests are interactive. When you run an interactive test a dialog box pops up, tells you what to do (e.g., “click the mouse here”) and what to look for, and gives you a button to click if the right thing happens. The test harness then records the result in its log ﬁle. You can run the automated tests separately from the interactive tests.

Q11	How long does it take to run the whole TCK suite?

There are tens of thousands of tests in the suite. It obviously depends on the speed of your machine, but running the automated tests on reasonably modern hardware is a matter of hours, not days. Many of Sun’s engineering teams have machines set up to run the automated tests every night, so they can review the results each morning.

It’s also easy to conﬁgure the harness to run just a subset of the tests, e.g., those for a speciﬁc package or feature, and that’s how most developers tend to use it when they’re working in a speciﬁc area. The test harness, which has been open-sourced in the “JT Harness” project on java.net, has fairly sophisticated test-selection facilities using keywords and other criteria. You can also easily re-run just the tests that failed in your most recent run.

4 Certifying a platform implementation

Q12	How complete is the TCK?

The primary quality metric for the TCK suite is speciﬁcation coverage rather code coverage, as for most other kinds of tests. The breadth and depth of coverage varies across the platform. The VM, language, and core libraries are pretty well covered by now; some of the newer areas, especially those further away from the core, are not as well covered.

Q13	Does an implementation have to pass every single test?

Yes, every single valid test in the suite must pass. You can challenge the validity of speciﬁc tests, however, and sometimes tests are thrown out as a result of such challenges.

Q14	What’s the difference between passing the test suite and actually certifying an implementation as compatible?

That’s an important distinction. A compatible implementation must meet all the requirements that are spelled out in the TCK User’s Guide, speciﬁcally those in Chapter 2. It must not only pass the test suite but also satisfy a set of rules that are, inherently, untestable. In other words, there are parts of the certiﬁcation process that amount to you asserting that, to the best of your knowledge, you’re not breaking compatibility. It doesn’t mean that you don’t have any bugs, it just means that you’re not knowingly violating any of the relevant speciﬁcations and other requirements.

The TCK rules also cover things like how you run the tests; e.g., you must use the provided test harness, unmodiﬁed, since the harness itself determines what passes and fails. That’s especially important for negative tests, e.g., compiler tests that are not intended to compile and should be considered to fail if they do. The harness contains all that pass/fail logic, so obviously if you could modify the harness then that could change the results.

Perhaps the most important rule is the famous “all modes” rule, which requires that an implementation be able to pass in all modes, i.e., in all possible combinations of command-line ﬂags, environment variables, and whatever other conﬁguration data may govern its behavior. It’s not sufﬁcient to say that the test suite passes in some conﬁguration, it must in principle be able to pass in all conﬁgurations. An implementation that has a ﬂag to disable array-bounds checking, e.g., would fail this rule.

Now the practical reality, of course, is that there will be modes that don’t pass, e.g., modes that just print a version string and exit, or that provide certain kinds of debugging information. The rules cover all these sorts of special cases and exceptions.

Q15	How hard is it to certify a new implementation?

A lot of it depends on how that implementation is constructed. Historically Sun has found that when people have something that’s almost completely based on Sun’s source code, and built in pretty much the way that Sun builds it, then it doesn’t take that much effort to certify.

When people have written their own VMs or other signiﬁcant components then the tests will ﬁnd bugs, and they’ll have to be ﬁxed. The time to certify such an implementation tends to be dominated by the time required to ﬁnd and ﬁx the bugs rather than the time required to run the tests. This can take weeks or even months, if the code in question has a lot of issues.

People who replace a lot of Sun’s code often ﬁnd that the hardest part of certifying is in handling error conditions correctly. They’ll implement all of the required positive behaviors and then discover that the TCK contains pretty thorough tests for negative behaviors too, e.g., throwing the correct exception for particular invalid inputs.

Q16	Have any of Sun’s commercial licensees replaced signiﬁcant portions of the Sun codebase?

Yes. Several Sun partners have written their very own VMs, e.g., and the ﬁrst couple of times that happened some issues in the tests were identiﬁed. Some of the testing techniques even had to be changed to eliminate assumptions that’d been true only of Sun’s VMs. At this point the suite has been pretty well vetted, especially in the VM and compiler areas, by third parties testing implementations that are only partly derived from Sun’s code.

Q17	What about source-based distributions such as Gentoo, in which binaries don’t exist until the source code is compiled on the end-user’s machine? Will the TCK be of interest to people working on such distributions?

Claims of compatibility in the Java world apply only to binaries, not to source code. If people working on a source-based distribution make modiﬁcations to the OpenJDK code they’ll still probably want to get their hands on the TCK, however, to help ﬂush out bugs and to verify that they can build a compatible implementation—even if that isn’t what they’ll actually ship in the end.

The trademark license, by the way, requires compatibility, so a source-based distribution containing OpenJDK code wouldn’t be able to carry the Java brand.

Q18	Suppose that the maintainer of an OpenJDK-derived implementation needs to issue an emergency release to ﬁx a security hole. Is it okay to do that without actually running the TCK all over again?

Yes. In general Sun tries to handle situations like this in a reasonable, common-sense sort of way. (Sun is in the business of shipping software too, so they understand.)

Claims of compatibility rest upon what is essentially a self-certiﬁcation process. You get the test suite, you get the rules, you look at them. When you think you’re ready, you write back to Sun and say, “I certiﬁed that this product meets your requirements using this version of the TCK.”

It’s understood that implementors sometimes have to do things quickly, like ship security ﬁxes. Whether to retest is a judgement call that you, as an implementor, would have to make for yourself. If the ﬁx involves a low-level change in the VM that could have broad consequences then you’d obviously want to retest before you continue to claim compatibility. If it’s an isolated ﬁx further up the stack, however, and you judge the risk to be low, then you might reasonably move ahead. There’s certainly not the expectation that you’ll always test every combination on every build; that’s just not practical.

The expectation is that people are acting in good faith, trying to meet the compatibility requirements, and that they aren’t knowingly doing something that would violate them. They’re expected to make their own choices around risk, with the understanding that if a mistake is made and an implementation isn’t compatible then Sun could ask them to stop claiming it’s compatible and stop using the Java logo with it.

This is pretty much how Sun has worked with its commercial licensees for many years now. This should scale to the relatively modest expected number of new OpenJDK TCK licensees that will want to self-certify and carry the brand.

Q19	If an OpenJDK-derived implementation is still in beta, or is declared ﬁnal but isn’t compatible, can it still be shipped?

Yes. This is a key difference between the OpenJDK TCK license and the traditional TCK license available under commercial terms from Sun.

Under the commercial license an implementor can ship early-access or beta releases and not worry about compatibility, so long as they don’t claim compatibility and don’t allow production use. Once they’re ready to ship the ﬁnal release to customers and support it in production, however, then it must be compatible—otherwise they can’t ship it.

The GPL, of course, doesn’t permit such constraints. An incompatible OpenJDK-derived implementation can go ﬁnal and be shipped—it just can’t be claimed to be compatible, nor can it carry the Java brand or logo.

Q20	Suppose that an implementation depends upon some libraries provided by the underlying system, e.g., the X11 library. When that library is updated then does the implementation have to be re-certiﬁed?

Claims of compatibility are always made against a speciﬁc set of software conﬁgurations. An implementation might be compatible on some other, possibly related conﬁguration, or it might not; there’s no guarantee. When Microsoft shipped Windows Vista, e.g., Sun had to re-certify its implementations on Vista in order to be able to claim compatibility on Vista.

5 Conﬁdentiality

Q21	What about the conﬁdentiality clause in the OpenJDK TCK license? Does it allow for the publication of test results? That is, can one talk about what passes and fails in public?

Yes. Sharing the results of test runs is ﬁne, per section 5.1 of the license. You can say things like “I passed all the tests,” or “I passed all the VM tests, but failed these seven compiler tests,” or you could even, if you really wanted to, publish the entire log of a TCK run.

Sun doesn’t, however, want people to make comparative claims of compatibility, e.g., “I’m just as compatible as that other guy over there, but not really actually compatible,” since that will tend to confuse people about what compatibility means.

The conﬁdentiality clause is intended to protect the tests themselves, not the results of running the tests. You can’t disclose the details of the tests publicly, though you can discuss them with other parties who’ve also signed the OpenJDK TCK license.

6 Miscellaneous

Q22	How does one go about writing TCK tests?

Since speciﬁcation coverage is the goal, the process is best started by marking up the speciﬁcation to be tested in order to identify all the assertions it makes, and then ﬁguring out which assertions are testable.

Q23	Will Sun accept contributions of new TCK tests?

Yes, but since TCK tests are so different from other kinds of tests any contributions will have to meet the standards of the rest of the suite. Given the overhead involved it’s probably not worthwhile for people to contribute just a few tests here and there. Sun does, however, very much welcome large sets of new tests, especially for new features. Doug Lea and the JSR-166 Expert Group, e.g., have made signiﬁcant contributions to the TCK, as have some of Sun’s commercial partners over the years.

Q24	What other kinds of tests does Sun have, besides the TCK?

Sun has several different kinds of tests.

The TCK tests are fairly special given their role in deﬁning what it means to be a compatible implementation, being able to run in a wide variety of environments, having to be robust enough to stand up to challenges of validity, and being the basis of various commercial contracts. These qualities tend to make compatibility tests harder to write than most other kinds of tests.

Aside from the TCK, Sun also has a large collection of regression and unit tests, written and maintained by the development team. Many of these tests have already been published under the GPL. The long-term goal is to publish as many of them as possible, subject to legal constraints.

Finally there are functional, performance, and reliability test suites. Some of these will likely be open-sourced over time although some cannot, again due to legal constraints.

Q25	When will the regression/unit-test harness, jtreg, be open-sourced?

The jtreg harness is actually just a small plugin for JT Harness, which has already been open-sourced. JT Harness is really more of a harness framework than a simple harness; it has some special features for compatibility testing but can actually be used for a wide variety of different kinds of tests.

It’s understood that people are interested in an open-source version of jtreg, and Sun does intend to release it, but doing so hasn’t been among the highest priorities.