JEP draft: PEM API (Preview)

OwnerAnthony Scarpino
TypeFeature
ScopeSE
StatusSubmitted
Componentsecurity-libs / java.security
Discussionsecurity dash dev at openjdk dot org
EffortM
DurationM
Reviewed byAlan Bateman, Sean Mullan
Created2023/01/23 18:28
Updated2024/09/17 20:11
Issue8300911

Summary

Introduce an API for encoding and decoding the Privacy-Enhanced Mail (PEM) format. The PEM format is used for storing and sending cryptographic keys, certificates, and certificate revocations lists. This is a preview API.

Goals

Non-Goals

Motivation

The Java API has a rich support for standard cryptographic objects, including Public Keys, Private Keys, Certificates, and Certificate Revocation Lists. Developers use these objects with security APIs to sign and verify signatures, verify TLS connections, and perform other cryptographic operations.

Some applications may only load cryptographic objects at startup or from a local keystore, but other applications may import objects from a user, the network, or a device. This requires support for a common way to store and communicate cryptographic objects from diverse environments. Per (RFC 7468)[https://www.rfc-editor.org/rfc/rfc7468], PEM defines a frequently used textual format for cryptographic objects.

This textual format was originally designed to send cryptographic objects via e-mail, but over time have been used and extended to different services. Certificate Authorities issue certificate chains in PEM. Microservices may use PEM for key and/or certificate stores when replicating multiple server instances that require pre-configured cryptographic objects. Cryptographic libraries, like OpenSSL, support cryptographic object generation and format conversion. Key Management applications may initialize and update cryptographic objects with PEM.

As the following example shows, PEM consists of a Base64 representation of the cryptographic object’s binary encoding surrounded by a header and footer containing "BEGIN" and "END", respectively. The header and footer identify the cryptographic object, "PUBLIC KEY", and are book-ended with five dashes. Details about the key, such as the algorithm, can be obtained by parsing the binary encoding. Below is an example of an Elliptic Curve public key:

-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDrYzJwsz8uI
wnmWznTr5r1N23e/nLzxYCAB
 -----END PUBLIC KEY-----

The Java API does not provide easy-to-use implementations of this textual encodings. While each cryptographic object provides a method to return the binary-encoded representation and there is a Base64 API to textualize that encoding, the Java API leaves rest of the work to the developer. This was confirmed by a survey the Java Cryptographic Extensions Survey in April 2022.

Decoding

The Java API provides no way to decode the above PEM. We must piece together security and non-security classes to convert the PEM text into a PublicKey object (Example 1):

String pemData = <PEM public key>
String base64Data = pemData.replace("-----BEGIN PUBLIC KEY-----", "") (1)
    .replaceAll(System.lineSeparator(), "")
    .replace("-----END PUBLIC KEY-----", "");

byte[] encoded = Base64.getDecoder().decode(base64Data);              (2)
X509EncodedKeySpec keySpec = new X509EncodedKeySpec(encoded);

KeyFactory keyFactory = KeyFactory.getInstance("EC");                (3)
var key = keyFactory.generatePublic(keySpec);

This code is very fragile, prone to parsing errors, and uses System.lineSeparator() which is platform-specific. We must manually strip the header, footer, and line separators before passing the encoding into the Base64 API API (1). We must know which EncodedKeySpec is associated with a public key to create the correct intermediate object from the decoded Base64 result (2). Finally, a EC KeyFactory is used to generate a PublicKey object from the intermediate X509EncodedKeySpec object (3).

To make matters worse, this example makes two assumptions about the PEM text. First, it only supports public keys by assuming a constant header and footer, and a particular intermediate object class. To address this, the application would have to parse the PEM header to determine what kind of Factory can generate the data (Example 2):

int start = pemData.indexOf(‘ ‘);
int end = pemData.indexOf(‘-‘, start);
String type = pemData.substring(start + 1, end  - 1);
switch (type) {
     case “PUBLIC KEY” -> {
         X509EncodedKeySpec keySpec = new X509EncodedKeySpec(encoded);
         KeyFactory keyFactory = KeyFactory.getInstance("EC");
         var key = keyFactory.generatePublic(keySpec);
    }
    case “PRIVATE KEY” -> {
         PKCS8EncodedKeySpec keySpec = new PKCS8EncodedKeySpec(encoded);
         KeyFactory keyFactory = KeyFactory.getInstance("EC");
         var key = keyFactory.generatePrivate(keySpec);
    }
    …

Secondly, it assumes the key is EC. If the key algorithm is not known by either prior knowledge or user-provided parsing of the binary encoding, an iteration over asymmetric KeyFactory instances would replace "(3)" in Example 1 with code like the following (Example 3):

String[] algorithms = { "RSA", "DSA", "EC", "EdDSA", "RSASSA-PSS" };  (1)

int i = 0;
PublicKey pubKey = null;                                             

while (algorithms.length > i) {                                       (2)
    try {
        KeyFactory kf = KeyFactory.getInstance(algorithms[i++]);
        pubKey = kf.generatePublic(keySpec);
        break;

    } catch (InvalidKeySpecException e) {                             (3)
        // continue loop
    }
}
if (pubKey == null) {
    throw new InvalidKeySpecException("unable to generate key");
}

The sample code contains a hardcoded algorithm list because there is no API that returns a supported list of asymmetric algorithms or KeyFactory instances (1). We must instantiate a KeyFactory for each supported algorithm until a public key is generated (2). The code must catch any unsuccessful public key generation and throw an exception if none were successful (3). The loop is an inefficient way find the key's algorithm and the complexity grows as the application supports more security objects. Other security objects are encoded with different PEM headers/footers and require different security APIs to decode the data into cryptographic objects.

Encoding

For some cryptographic objects, encoding a public key is easier than decoding because the key object contains the needed information (Example 4):

StringBuilder sb = new StringBuilder();
sb.append("-----BEGIN PUBLIC KEY-----");                            (1)
sb.append(Base64.getEncoder().encodeToString(pubKey.getEncoded())); (2)
sb.append("-----END PUBLIC KEY-----");
String s = sb.toString();

The example is straightforward and does not require a KeyFactory for encoding the key. We write the proper key type header and footer (1) and passes the key’s binary encoding through the Base64 encoder (2). Nevertheless, this task should be easier for the developer to implement. Encoding other cryptographic objects can be more tedious. An encrypted private key is the most complicated as first the PrivateKey must encrypted with a secret key before the PEM operations can be done. The code may look like this (Example 5):

char[] password;
PBEKeySpec pbeKeySpec = new PBEKeySpec(password);                    (1)
SecretKeyFactory keyFactory = SecretKeyFactory.getInstance(pbeAlgo);
SecretKey pbeKey = keyFactory.generateSecret(pbeKeySpec);

Cipher pbeCipher = Cipher.getInstance(pbeAlgo);                      (2)
pbeCipher.init(Cipher.ENCRYPT_MODE, pbeKey);
byte[] ciphertext = pbeCipher.doFinal(privateKey.getEncoded());

AlgorithmParameters aps = pbeCipher.getParameters();                 (3)
EncryptedPrivateKeyInfo epki = new EncryptedPrivateKeyInfo(aps, ciphertext);

StringBuilder sb = new StringBuilder();                              (4)
Base64.Encoder e = Base64.getEncoder();
sb.append("-----BEGIN ENCRYPTED PRIVATE KEY-----");
sb.append(e.encodeToString(epki.getEncoded()));
sb.append("-----END ENCRYPTED PRIVATE KEY-----");
String s = sb.toString();

As the long example shows, PBEKeySpec is first used with the password to generate a secret key from the SecretKeyFactory(1). That SecretKey is then used to encrypt the private key's binary encoded data (2). Then, the encrypted data and encryption parameters are used to create an EncryptedPrivateKeyInfo object, which contains the correct binary encoding format (3). Finally, we finish by Base64 encoding the encoded bytes of the EncryptedPrivateKeyInfo and surrounding it with the proper PEM header and footer (4). This use case requires detailed knowledge of the format of the encrypted private key PEM structure beyond what should be expected, as well as requiring six security classes and two provider service instances to encrypt and encode a PrivateKey to PEM. Surely, we can do better.

Description

This is a Preview API, disabled by default

To use this new API in JDK 24, you must enable preview features:

A new interface and two new classes will provide an easier to use experience with PEM. These additions include:

DEREncodable

This new sealed interface marks the permitted classes and interfaces that return security data in a binary encoded representation. Distinguished Encoding Rules (DER) are used with binary encoding formats, X.509 and PKCS#8, and supported by the PEM API. PEM uses DEREncodable to identify which objects can be used with Base64 encoding and decoding. This generic interface type simplifies the API and minimizes defined methods. The permitted classes are: AsymmetricKey, X509Certificate, X509CRL, KeyPair, EncryptedPrivateKeyInfo, PKCS8EncodedKeySpec, and X509EncodedKeySpec.

package java.security;

public sealed interface DEREncodable permits AsymmetricKey, KeyPair, X509CRL,
    X509Certificate, PKCS8EncodedKeySpec, X509EncodedKeySpec,
    EncryptedPrivateKeyInfo {}

With this new sealed class, some of the permitted classes will change slightly:

public non-sealed interface AsymmetricKey ...
public non-sealed abstract class X509Certificate ...
public non-sealed abstract class X509CRL ...
public final class EncryptedPrivateKeyInfo ...
public final class PKCS8EncodedKeySpec ...
public final class X509EncodedKeySpec ...

Encoding

The PEMEncoder class defines methods for encoding DEREncodable types to PEM:

package java.security;

public final class PEMEncoder {
    public static PEMEncoder of();
    public byte[] encode(DEREncodable so) throws IOException;
    public String encodeToString(DEREncodable so) throws IOException;
    public PEMEncoder withEncryption(char[] password);
}

To encode a DEREncodable, get a PEMEncoder instance by calling of(). The instance is reusable and thread-safe, allowing encode methods to be used repeatedly. There are two methods to complete the encoding process. The first is encode(DEREncodable) that returns PEM data in a byte[]. The other is encodeToString(DEREncodable) that returns PEM data as a String. Return values use StandardCharsets.ISO-8859-1.

If the DEREncodable used is a PrivateKey, there is an option to encrypt. The withEncryption(char[] password) method returns a new immutable PEMEncoder instance, configured with encryption using a default algorithm. Only PrivateKey objects can be encoded by an encrypted PEMEncoder. To use non-default encryption parameters or encrypt with a different encryption Provider, encode with an EncryptedPrivateKeyInfo object. See the EncryptedPrivateKeyInfo section below for more details.

Here are examples using the PEMEncoder class:

Encoding a PrivateKey into PEM. This replaces the code from Example 3 in the Motivation:

PEMEncoder pe = PEMEncoder.of();
byte[] pemData = pe.encode(privKey);

Encoding a PrivateKey into PEM with encryption. This replaces the code from Example 4 in the Motivation:

String pemString = pe.withEncryption(password).encodeToString(privKey);

Encoding both a public and private key into the same PEM:

byte[] pemData = pe.encode(new KeyPair(publicKey, privateKey));

Decoding

The PEMDecoder subclass defines methods for decoding PEM to a DEREncodable:

package java.security;

 public final class PEMDecoder {
     public static PEMDecoder of();
     public PEMDecoder withDecryption(char[] password);
     public PEMDecoder withFactory(Provider provider);
     public DEREncodable decode(String str) throws IOException;
     public DEREncodable decode(InputStream is) throws IOException;
     public <S extends DEREncodable> S decode(String string,
         Class<S> sClass) throws IOException;
     public <S extends DEREncodable> S decode(InputStream is,
         Class<S> sClass) throws IOException;
 }

To decode PEM data, get a PEMDecoder instance by calling of(). The instance is reusable and thread-safe, allowing decode methods to be used repeatedly. There are four methods to complete the decoding process. They each return a DEREncodable for which the caller can use pattern matching when processing the result. If the developer knows the class type being decoded, the two decode methods that take a Class<S> argument can be used to specify the returned object's class. If the class does not match the PEM type, an ClassCastExeption is thrown. When passing input data, the application is responsible for processing data ahead of the PEM header as it will be ignored by decode. All input data into these methods will use StandardCharsets.ISO-8859-1. Any IOExceptions from the InputStream will be thrown, all other errors will throw an IllegalArgumentException.

withDecryption(char[] password) is a helper method for decrypting encrypted private key PEM data. It returns a new immutable PEMDecoder configured with the given password. When this configured instance is used on encrypted private key PEM data, the decode methods will return a PrivateKey object. The other PEM data types can be decoded by this configured instance, as decryption is not relevant. On non-decryption configured instances, the decode methods return an EncryptedPrivateKeyInfo object for encrypted private keys, which can then be used to decrypt and generate the PrivateKey object (see EncryptedPrivateKeyInfo below for more details).

Some DEREncodable types may need to be generated from a particular JCE provider. The withFactory(Provider) method returns a new PEMDecoder instance that uses the specified JCE provider to generate the security object with a particular factory (for example, a KeyFactory). If the provider does not support the data being decoded, an IOException is thrown.

Here are some examples:

Decoding a PublicKey from PEM. This replaces the code from Example 1 & 2 in the Motivation:

PEMDecoder pd = PEMDecoder.of();
PublicKey key = pd.decode(pemData, PublicKey.class);

Using pattern matching when decoding PEM data of an unknown type simplifies Example 2:

switch (pd.decode(pemData)) {
    case PublicKey pubkey -> ...
    case PrivateKey privkey -> ...
    ...
    }

Decoding an encrypted ECPrivateKey from PEM:

ECPrivateKey eckey = pd.withDecryption(password).
    decode(pemData, ECPrivateKey.class);

Decoding a Certificate with a specific Factory provider:

PEMDecoder d = pd.withFactory(providerF);
Certificate c = d.decode(pemData, Certificate.class);

EncryptedPrivateKeyInfo

A common use case with private keys is to encrypt the key before encoding to PEM or decrypt the key after decoding from PEM. However, this operation may require additional encryption parameters than just a password. To keep the PEM APIs simple, the EncryptedPrivateKeyInfo class has been enhanced with additional methods that support these operations:

EncryptedPrivateKeyInfo {
     ...
     public static EncryptedPrivateKeyInfo encryptKey(PrivateKey key,
         char[] password) throws IOException;
     public static EncryptedPrivateKeyInfo encryptKey(PrivateKey key,
         char[] password, String algorithm, AlgorithmParameterSpec params,
         Provider p) throws IOException;
     public PrivateKey getKey(char[] password) throws IOException;
     public PrivateKey getKey(char[] password, Provider provider)
         throws IOException;
 }

The two new encryptKey() methods encrypt the given PrivateKey with the given password. For advanced usage, the second method allows all the cryptographic parameters to be specified if the defaults are not sufficient. The returned EncryptedPrivateKeyInfo object can then be passed to PEMEncoder.encode() to encode to PEM:

ekpi = EncryptedPrivateKeyInfo.encryptKey(privkey, password);
byte[] pemData = PEMEncoder.of().encode(epki);

The getKey() methods are used for decrypting an initialized EncryptedPrivateKeyInfo, such as when PEMDecoder.decode() has returned an EncryptedPrivateKeyInfo. The methods will return a PrivateKey with a given password and an optional JCE provider:

EncryptedPrivateKeyInfo epki = PEMDecoder.of().decode(pemData);
PrivateKey key = epki.getKey(password);

The default Password-based Encryption (PBE) algorithm used when encrypting a PrivateKey with PEMEncoder and EncryptedPrivateKeyInfo is stored in the jdk/conf/security/java.security file. The jdk.epkcs8.defaultAlgorithm security property defines the default algorithm to be "PBEWithHmacSHA256AndAES_128". There is no compatibility issue if the default algorithm is changed as the encrypted PKCS#8 encoding contains the algorithm and parameters necessary for decrypting.

Alternatives

A PEM API is a bridge between Base64 and cryptographic objects. Many of the API designs were rejected over how they fit with the security APIs. Retrofitting existing APIs can be awkward when all the pieces don't fit in the right classes. A few of the rejects could have sufficiently delivered the feature, but this API was chosen for its similarity to HexFormat and Base64's Encoder/Decoder classes. There were strong requirements to have a distinct encode and decode track, immutable, thread-safety, and delivering between PEM and DEREncodable in an easy-to-use way. Some of those design alternatives are documented below:

Extending the EncodedKeySpec API

The EncodedKeySpec API is used to encapsulate binary encoded key data for KeyFactory instances and other security classes which require the data type identification. Java provides two sub-classes, PKCS8EncodedKeySpec for PrivateKey encodings and X509EncodedKeySpec for PublicKey encodings. A new PEMEncodedKeySpec sub-class was considered which would type identify encapsulated PEM data, while providing encoding and decoding operations between PEM and the appropriate private or public key EncodedKeySpec. This new EncodedKeySpec sub-class could also be an input for KeyFactory instances, simplifying the user experience.

This design had a few deficiencies. This new PEMEncodedKeySpec sub-class would be a misuse of a translation class and would be Key-centric. It cannot support encoding certificates or CRLs to PEM. Also, the existing EncodedKeySpec sub-classes mentioned above are already used by KeyFactory instances for public and private key generation. A new EncodedKeySpec sub-class would add compatibility risks and ease of use issues with existing third-party providers.

Enhancing the CertificateFactory and KeyFactory APIs

Key and certificate factory classes are used to convert and generate their respective objects. The CertificateFactory API already supports decoding of PEM certificate and CRL data. Adding encoding to CertificateFactory and KeyFactory would be consistent with the existing design, but it complicates JCE providers. CertificateFactory makes the design look easy as there is one industry standard encoding for certificates. KeyFactory is much more complicated as key encoding formats differ between algorithm and key type and different JCE providers support different asymmetric algorithms. PEM adds an extra layer of complexity if each provider is responsible for its conversion; as well as, handling encrypted private keys.

To avoid this provider complexity, another design idea was to add static PEM methods to KeyFactory, but this deviates from the pluggable design of CertificateFactory and creates different solutions between the two factories.

Static method usage

Static methods are good for immutability and thread-safety, but for PEM, encrypted private keys (EPK) present a usability issue. EPK is a two-stage operation, while the other PEM types are single stage. An unencrypted private key can be decoded by a static method using Base64 decoding, parsing the binary encoding, and returning a PrivateKey object. Decoding an EPK requires Base64 decoding, parsing of encryption parameters, decryption, then parsing of the binary encoding and returning a PrivateKey object. The second and third steps complicate usability. There are solutions that could still use only static methods, but they require another object to contain the data, such as EncryptedPrivateKeyInfo (EPKI). EPKI could decrypted the object separately, but this produces an inconsistency in the PEM API operational flow. Decoding non-EPK objects returns usable objects for crypto operations, while EPK returns an EPKI that requires further operations to generate usable PrivateKey objects. This complicates usability when reading from a file or stream that has mixed PEM types, by making the application call other methods. EKPI can be the solution for certain configurations, but it hurts the ease-of-use as the default. Having encoder and decoder keep some configuration state, like a password, allows the application to know it is always getting a usable object for a crypto operation.

Intermediate PEM Object API

An intermediate object is a wrapper class for a Key, Certificate, CRL, or PEM data. This object could either contain methods to encode/decode or use a separate API performs operations on it. While it does provide an independent representation of PEM data, too much flexibility is a negative. Encoding operations could be called on an object initialized with PEM data or decoding from a PrivateKey. Having distinct encoding and decoding paths from given data gives clear operational usability.

Single Class API

This API design is defined as encoding and decoding from the same class. The initial prototype used static methods which complicated encrypted private keys usage, as described in Static Method API section. While switch to non-static with immutable configurations addresses this, this design also does not have distinct encoding and decoding paths, like the Intermediate PEM Object API. The Base64 API could be looked at as a comparable to the single class design, but it is not. Base64 has encoder and decoder classes under a single class. The PEM API differs by avoiding the umbrella single class and just providing the underlying operational classes.

Creating a new JCE API/SPI for Encodings

JCE Providers provide many services through the Java API, like Crypto or MessageDigest. Adding an Encoding service would make logical sense. Binary encodings have not been part of that infrastructure but used by individual provider for importing or exporting cryptographic objects. Adding services for textual and binary encodings could be useful beyond PEM.

Unfortunately, this alternative provides nothing beyond a lot of infrastructure to deliver what most do not need. Getting an instance of a binary encoding has not been a part of the Java API and too late introduce. Binary encodings are typically a part of the provider internals. It would make no sense for one provider’s encoding service to import into another provider. For keys, any encodings beyond X.509 and PKCS#8 can be supported via implementations of KeySpec.

To supporting PEM from a provider is more complicated API than java.util.Base64. Additional Java API methods would be needed for Providers to use encrypted private keys, extracting the key algorithm, and other details to use with a factory to generate a Key or Certificate object. These PEM-specific changes would eliminate any benefit of a provider-based solution. With third-party APIs existing for many years, a pluggable infrastructure offers nothing but making a simple task hard. And very unlikely to be extensible as other encodings have different features that would need individual API support.

General Purpose Encoder & Decoding API

Instead of a PEM-specific API, we looked at a generic API that could be used for many encodings. This was rejected because encodings do not all have to same features. PEM’s a single method call for all types except encrypted private keys. However other encodings can support both asymmetric and asymmetric keys, certificate chains, compression and other options. Having methods that only supported certain encodings would be confusing.

The most recent common API design was an Encoder and Decoder interface that required basic methods encode() and decode() with PEM{Encoder,Decoder} implementing the interfaces. However, those interfaces were removed to focus on PEM and not try to create a general API structure that was currently unknown. But in the future, those interfaces could be revived if a future encodings was implemented. At this time there are no plans to add another encoder/decoder API..

Testing

Tests should include: