JEP draft: PEM API (Preview)
Owner | Anthony Scarpino |
Type | Feature |
Scope | SE |
Status | Submitted |
Component | security-libs / java.security |
Discussion | security dash dev at openjdk dot org |
Effort | M |
Duration | M |
Reviewed by | Alan Bateman, Sean Mullan |
Created | 2023/01/23 18:28 |
Updated | 2024/09/17 20:11 |
Issue | 8300911 |
Summary
Introduce an API for encoding and decoding the Privacy-Enhanced Mail (PEM) format. The PEM format is used for storing and sending cryptographic keys, certificates, and certificate revocations lists. This is a preview API.
Goals
- Ease of use - Define a concise API that converts between PEM data and asymmetric key, certificate, and certificate revocation lists objects.
- Standards - Support conversions between PEM data and cryptographic objects that are represented in the following binary formats: PKCS#8 (private keys), X.509 (certificates, CRLs, and public keys), PKCS#8 v2.0 (OneAsymmetricKey and Encrypted private keys).
Non-Goals
- It is not a goal to support types in RFC 7468 that cannot be represented as a cryptographic Java object.
Motivation
The Java API has a rich support for standard cryptographic objects, including Public Keys, Private Keys, Certificates, and Certificate Revocation Lists. Developers use these objects with security APIs to sign and verify signatures, verify TLS connections, and perform other cryptographic operations.
Some applications may only load cryptographic objects at startup or from a local keystore, but other applications may import objects from a user, the network, or a device. This requires support for a common way to store and communicate cryptographic objects from diverse environments. Per (RFC 7468)[https://www.rfc-editor.org/rfc/rfc7468], PEM defines a frequently used textual format for cryptographic objects.
This textual format was originally designed to send cryptographic objects via e-mail, but over time have been used and extended to different services. Certificate Authorities issue certificate chains in PEM. Microservices may use PEM for key and/or certificate stores when replicating multiple server instances that require pre-configured cryptographic objects. Cryptographic libraries, like OpenSSL, support cryptographic object generation and format conversion. Key Management applications may initialize and update cryptographic objects with PEM.
As the following example shows, PEM consists of a Base64 representation of the cryptographic object’s binary encoding surrounded by a header and footer containing "BEGIN" and "END", respectively. The header and footer identify the cryptographic object, "PUBLIC KEY", and are book-ended with five dashes. Details about the key, such as the algorithm, can be obtained by parsing the binary encoding. Below is an example of an Elliptic Curve public key:
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDrYzJwsz8uI
wnmWznTr5r1N23e/nLzxYCAB
-----END PUBLIC KEY-----
The Java API does not provide easy-to-use implementations of this textual encodings. While each cryptographic object provides a method to return the binary-encoded representation and there is a Base64 API to textualize that encoding, the Java API leaves rest of the work to the developer. This was confirmed by a survey the Java Cryptographic Extensions Survey in April 2022.
Decoding
The Java API provides no way to decode the above PEM. We must piece together security and non-security classes to convert the PEM text into a PublicKey object (Example 1):
String pemData = <PEM public key>
String base64Data = pemData.replace("-----BEGIN PUBLIC KEY-----", "") (1)
.replaceAll(System.lineSeparator(), "")
.replace("-----END PUBLIC KEY-----", "");
byte[] encoded = Base64.getDecoder().decode(base64Data); (2)
X509EncodedKeySpec keySpec = new X509EncodedKeySpec(encoded);
KeyFactory keyFactory = KeyFactory.getInstance("EC"); (3)
var key = keyFactory.generatePublic(keySpec);
This code is very fragile, prone to parsing errors, and uses System.lineSeparator() which is platform-specific. We must manually strip the header, footer, and line separators before passing the encoding into the Base64 API API (1). We must know which EncodedKeySpec is associated with a public key to create the correct intermediate object from the decoded Base64 result (2). Finally, a EC KeyFactory is used to generate a PublicKey object from the intermediate X509EncodedKeySpec object (3).
To make matters worse, this example makes two assumptions about the PEM text. First, it only supports public keys by assuming a constant header and footer, and a particular intermediate object class. To address this, the application would have to parse the PEM header to determine what kind of Factory can generate the data (Example 2):
int start = pemData.indexOf(‘ ‘);
int end = pemData.indexOf(‘-‘, start);
String type = pemData.substring(start + 1, end - 1);
switch (type) {
case “PUBLIC KEY” -> {
X509EncodedKeySpec keySpec = new X509EncodedKeySpec(encoded);
KeyFactory keyFactory = KeyFactory.getInstance("EC");
var key = keyFactory.generatePublic(keySpec);
}
case “PRIVATE KEY” -> {
PKCS8EncodedKeySpec keySpec = new PKCS8EncodedKeySpec(encoded);
KeyFactory keyFactory = KeyFactory.getInstance("EC");
var key = keyFactory.generatePrivate(keySpec);
}
…
Secondly, it assumes the key is EC. If the key algorithm is not known by either prior knowledge or user-provided parsing of the binary encoding, an iteration over asymmetric KeyFactory instances would replace "(3)" in Example 1 with code like the following (Example 3):
String[] algorithms = { "RSA", "DSA", "EC", "EdDSA", "RSASSA-PSS" }; (1)
int i = 0;
PublicKey pubKey = null;
while (algorithms.length > i) { (2)
try {
KeyFactory kf = KeyFactory.getInstance(algorithms[i++]);
pubKey = kf.generatePublic(keySpec);
break;
} catch (InvalidKeySpecException e) { (3)
// continue loop
}
}
if (pubKey == null) {
throw new InvalidKeySpecException("unable to generate key");
}
The sample code contains a hardcoded algorithm list because there is no API that returns a supported list of asymmetric algorithms or KeyFactory instances (1). We must instantiate a KeyFactory for each supported algorithm until a public key is generated (2). The code must catch any unsuccessful public key generation and throw an exception if none were successful (3). The loop is an inefficient way find the key's algorithm and the complexity grows as the application supports more security objects. Other security objects are encoded with different PEM headers/footers and require different security APIs to decode the data into cryptographic objects.
Encoding
For some cryptographic objects, encoding a public key is easier than decoding because the key object contains the needed information (Example 4):
StringBuilder sb = new StringBuilder();
sb.append("-----BEGIN PUBLIC KEY-----"); (1)
sb.append(Base64.getEncoder().encodeToString(pubKey.getEncoded())); (2)
sb.append("-----END PUBLIC KEY-----");
String s = sb.toString();
The example is straightforward and does not require a KeyFactory for encoding the key. We write the proper key type header and footer (1) and passes the key’s binary encoding through the Base64 encoder (2). Nevertheless, this task should be easier for the developer to implement. Encoding other cryptographic objects can be more tedious. An encrypted private key is the most complicated as first the PrivateKey must encrypted with a secret key before the PEM operations can be done. The code may look like this (Example 5):
char[] password;
PBEKeySpec pbeKeySpec = new PBEKeySpec(password); (1)
SecretKeyFactory keyFactory = SecretKeyFactory.getInstance(pbeAlgo);
SecretKey pbeKey = keyFactory.generateSecret(pbeKeySpec);
Cipher pbeCipher = Cipher.getInstance(pbeAlgo); (2)
pbeCipher.init(Cipher.ENCRYPT_MODE, pbeKey);
byte[] ciphertext = pbeCipher.doFinal(privateKey.getEncoded());
AlgorithmParameters aps = pbeCipher.getParameters(); (3)
EncryptedPrivateKeyInfo epki = new EncryptedPrivateKeyInfo(aps, ciphertext);
StringBuilder sb = new StringBuilder(); (4)
Base64.Encoder e = Base64.getEncoder();
sb.append("-----BEGIN ENCRYPTED PRIVATE KEY-----");
sb.append(e.encodeToString(epki.getEncoded()));
sb.append("-----END ENCRYPTED PRIVATE KEY-----");
String s = sb.toString();
As the long example shows, PBEKeySpec is first used with the password to generate a secret key from the SecretKeyFactory(1). That SecretKey is then used to encrypt the private key's binary encoded data (2). Then, the encrypted data and encryption parameters are used to create an EncryptedPrivateKeyInfo object, which contains the correct binary encoding format (3). Finally, we finish by Base64 encoding the encoded bytes of the EncryptedPrivateKeyInfo and surrounding it with the proper PEM header and footer (4). This use case requires detailed knowledge of the format of the encrypted private key PEM structure beyond what should be expected, as well as requiring six security classes and two provider service instances to encrypt and encode a PrivateKey to PEM. Surely, we can do better.
Description
This is a Preview API, disabled by default
To use this new API in JDK 24, you must enable preview features:
- Compile the program with javac --release 24 --enable-preview Main.java and run it with java --enable-preview Main; or,
- When using the source code launcher, run the program with java --enable-preview Main.java; or,
- When using jshell, start it with jshell --enable-preview.
A new interface and two new classes will provide an easier to use experience with PEM. These additions include:
- A DEREncodeable interface that will be extended by Java classes that contain binary-encodable Key or Certificate material.
- PEMEncoder and PEMDecoder classes to provide support for PEM. These APIs are immutable and reusable and do not keep state from the previously used security object.
DEREncodable
This new sealed interface marks the permitted classes and interfaces that return security data in a binary encoded representation. Distinguished Encoding Rules (DER) are used with binary encoding formats, X.509 and PKCS#8, and supported by the PEM API. PEM uses DEREncodable to identify which objects can be used with Base64 encoding and decoding. This generic interface type simplifies the API and minimizes defined methods. The permitted classes are: AsymmetricKey, X509Certificate, X509CRL, KeyPair, EncryptedPrivateKeyInfo, PKCS8EncodedKeySpec, and X509EncodedKeySpec.
package java.security;
public sealed interface DEREncodable permits AsymmetricKey, KeyPair, X509CRL,
X509Certificate, PKCS8EncodedKeySpec, X509EncodedKeySpec,
EncryptedPrivateKeyInfo {}
With this new sealed class, some of the permitted classes will change slightly:
public non-sealed interface AsymmetricKey ...
public non-sealed abstract class X509Certificate ...
public non-sealed abstract class X509CRL ...
public final class EncryptedPrivateKeyInfo ...
public final class PKCS8EncodedKeySpec ...
public final class X509EncodedKeySpec ...
Encoding
The PEMEncoder
class defines methods for encoding DEREncodable
types to PEM:
package java.security;
public final class PEMEncoder {
public static PEMEncoder of();
public byte[] encode(DEREncodable so) throws IOException;
public String encodeToString(DEREncodable so) throws IOException;
public PEMEncoder withEncryption(char[] password);
}
To encode a DEREncodable
, get a PEMEncoder
instance by calling of()
. The instance is reusable and thread-safe, allowing encode methods to be used repeatedly. There are two methods to complete the encoding process. The first is encode(DEREncodable)
that returns PEM data in a byte[]
. The other is encodeToString(DEREncodable)
that returns PEM data as a String
. Return values use StandardCharsets.ISO-8859-1.
If the DEREncodable
used is a PrivateKey
, there is an option to encrypt. The withEncryption(char[] password)
method returns a new immutable PEMEncoder
instance, configured with encryption using a default algorithm. Only PrivateKey
objects can be encoded by an encrypted PEMEncoder
. To use non-default encryption parameters or encrypt with a different encryption Provider, encode with an EncryptedPrivateKeyInfo
object. See the EncryptedPrivateKeyInfo section below for more details.
Here are examples using the PEMEncoder
class:
Encoding a PrivateKey
into PEM. This replaces the code from Example 3 in the Motivation:
PEMEncoder pe = PEMEncoder.of();
byte[] pemData = pe.encode(privKey);
Encoding a PrivateKey
into PEM with encryption. This replaces the code from Example 4 in the Motivation:
String pemString = pe.withEncryption(password).encodeToString(privKey);
Encoding both a public and private key into the same PEM:
byte[] pemData = pe.encode(new KeyPair(publicKey, privateKey));
Decoding
The PEMDecoder
subclass defines methods for decoding PEM to a DEREncodable
:
package java.security;
public final class PEMDecoder {
public static PEMDecoder of();
public PEMDecoder withDecryption(char[] password);
public PEMDecoder withFactory(Provider provider);
public DEREncodable decode(String str) throws IOException;
public DEREncodable decode(InputStream is) throws IOException;
public <S extends DEREncodable> S decode(String string,
Class<S> sClass) throws IOException;
public <S extends DEREncodable> S decode(InputStream is,
Class<S> sClass) throws IOException;
}
To decode PEM data, get a PEMDecoder
instance by calling of()
. The instance is reusable and thread-safe, allowing decode methods to be used repeatedly. There are four methods to complete the decoding process. They each return a DEREncodable
for which the caller can use pattern matching when processing the result. If the developer knows the class type being decoded, the two decode methods that take a Class<S>
argument can be used to specify the returned object's class. If the class does not match the PEM type, an ClassCastExeption
is thrown. When passing input data, the application is responsible for processing data ahead of the PEM header as it will be ignored by decode. All input data into these methods will use StandardCharsets.ISO-8859-1. Any IOExceptions
from the InputStream
will be thrown, all other errors will throw an IllegalArgumentException
.
withDecryption(char[] password)
is a helper method for decrypting encrypted private key PEM data. It returns a new immutable PEMDecoder
configured with the given password. When this configured instance is used on encrypted private key PEM data, the decode methods will return a PrivateKey
object. The other PEM data types can be decoded by this configured instance, as decryption is not relevant. On non-decryption configured instances, the decode methods return an EncryptedPrivateKeyInfo
object for encrypted private keys, which can then be used to decrypt and generate the PrivateKey
object (see EncryptedPrivateKeyInfo below for more details).
Some DEREncodable
types may need to be generated from a particular JCE provider. The withFactory(Provider)
method returns a new PEMDecoder
instance that uses the specified JCE provider to generate the security object with a particular factory (for example, a KeyFactory
). If the provider does not support the data being decoded, an IOException
is thrown.
Here are some examples:
Decoding a PublicKey
from PEM. This replaces the code from Example 1 & 2 in the Motivation:
PEMDecoder pd = PEMDecoder.of();
PublicKey key = pd.decode(pemData, PublicKey.class);
Using pattern matching when decoding PEM data of an unknown type simplifies Example 2:
switch (pd.decode(pemData)) {
case PublicKey pubkey -> ...
case PrivateKey privkey -> ...
...
}
Decoding an encrypted ECPrivateKey
from PEM:
ECPrivateKey eckey = pd.withDecryption(password).
decode(pemData, ECPrivateKey.class);
Decoding a Certificate
with a specific Factory provider:
PEMDecoder d = pd.withFactory(providerF);
Certificate c = d.decode(pemData, Certificate.class);
EncryptedPrivateKeyInfo
A common use case with private keys is to encrypt the key before encoding to PEM or decrypt the key after decoding from PEM. However, this operation may require additional encryption parameters than just a password. To keep the PEM APIs simple, the EncryptedPrivateKeyInfo
class has been enhanced with additional methods that support these operations:
EncryptedPrivateKeyInfo {
...
public static EncryptedPrivateKeyInfo encryptKey(PrivateKey key,
char[] password) throws IOException;
public static EncryptedPrivateKeyInfo encryptKey(PrivateKey key,
char[] password, String algorithm, AlgorithmParameterSpec params,
Provider p) throws IOException;
public PrivateKey getKey(char[] password) throws IOException;
public PrivateKey getKey(char[] password, Provider provider)
throws IOException;
}
The two new encryptKey()
methods encrypt the given PrivateKey
with the given password. For advanced usage, the second method allows all the cryptographic parameters to be specified if the defaults are not sufficient. The returned EncryptedPrivateKeyInfo
object can then be passed to PEMEncoder.encode()
to encode to PEM:
ekpi = EncryptedPrivateKeyInfo.encryptKey(privkey, password);
byte[] pemData = PEMEncoder.of().encode(epki);
The getKey()
methods are used for decrypting an initialized EncryptedPrivateKeyInfo
, such as when PEMDecoder.decode()
has returned an EncryptedPrivateKeyInfo
. The methods will return a PrivateKey
with a given password and an optional JCE provider:
EncryptedPrivateKeyInfo epki = PEMDecoder.of().decode(pemData);
PrivateKey key = epki.getKey(password);
The default Password-based Encryption (PBE) algorithm used when encrypting a PrivateKey
with PEMEncoder
and EncryptedPrivateKeyInfo
is stored in the jdk/conf/security/java.security
file. The jdk.epkcs8.defaultAlgorithm
security property defines the default algorithm to be "PBEWithHmacSHA256AndAES_128". There is no compatibility issue if the default algorithm is changed as the encrypted PKCS#8 encoding contains the algorithm and parameters necessary for decrypting.
Alternatives
A PEM API is a bridge between Base64 and cryptographic objects. Many of the API designs were rejected over how they fit with the security APIs. Retrofitting existing APIs can be awkward when all the pieces don't fit in the right classes. A few of the rejects could have sufficiently delivered the feature, but this API was chosen for its similarity to HexFormat and Base64's Encoder/Decoder classes. There were strong requirements to have a distinct encode and decode track, immutable, thread-safety, and delivering between PEM and DEREncodable in an easy-to-use way. Some of those design alternatives are documented below:
Extending the EncodedKeySpec API
The EncodedKeySpec
API is used to encapsulate binary encoded key data for KeyFactory
instances and other security classes which require the data type identification. Java provides two sub-classes, PKCS8EncodedKeySpec for PrivateKey
encodings and X509EncodedKeySpec
for PublicKey
encodings. A new PEMEncodedKeySpec
sub-class was considered which would type identify encapsulated PEM data, while providing encoding and decoding operations between PEM and the appropriate private or public key EncodedKeySpec
. This new EncodedKeySpec
sub-class could also be an input for KeyFactory
instances, simplifying the user experience.
This design had a few deficiencies. This new PEMEncodedKeySpec
sub-class would be a misuse of a translation class and would be Key-centric. It cannot support encoding certificates or CRLs to PEM. Also, the existing EncodedKeySpec
sub-classes mentioned above are already used by KeyFactory
instances for public and private key generation. A new EncodedKeySpec
sub-class would add compatibility risks and ease of use issues with existing third-party providers.
Enhancing the CertificateFactory and KeyFactory APIs
Key and certificate factory classes are used to convert and generate their respective objects. The CertificateFactory
API already supports decoding of PEM certificate and CRL data. Adding encoding to CertificateFactory
and KeyFactory
would be consistent with the existing design, but it complicates JCE providers. CertificateFactory
makes the design look easy as there is one industry standard encoding for certificates. KeyFactory
is much more complicated as key encoding formats differ between algorithm and key type and different JCE providers support different asymmetric algorithms. PEM adds an extra layer of complexity if each provider is responsible for its conversion; as well as, handling encrypted private keys.
To avoid this provider complexity, another design idea was to add static PEM methods to KeyFactory
, but this deviates from the pluggable design of CertificateFactory
and creates different solutions between the two factories.
Static method usage
Static methods are good for immutability and thread-safety, but for PEM, encrypted private keys (EPK) present a usability issue. EPK is a two-stage operation, while the other PEM types are single stage. An unencrypted private key can be decoded by a static method using Base64 decoding, parsing the binary encoding, and returning a PrivateKey object. Decoding an EPK requires Base64 decoding, parsing of encryption parameters, decryption, then parsing of the binary encoding and returning a PrivateKey object. The second and third steps complicate usability. There are solutions that could still use only static methods, but they require another object to contain the data, such as EncryptedPrivateKeyInfo (EPKI). EPKI could decrypted the object separately, but this produces an inconsistency in the PEM API operational flow. Decoding non-EPK objects returns usable objects for crypto operations, while EPK returns an EPKI that requires further operations to generate usable PrivateKey objects. This complicates usability when reading from a file or stream that has mixed PEM types, by making the application call other methods. EKPI can be the solution for certain configurations, but it hurts the ease-of-use as the default. Having encoder and decoder keep some configuration state, like a password, allows the application to know it is always getting a usable object for a crypto operation.
Intermediate PEM Object API
An intermediate object is a wrapper class for a Key, Certificate, CRL, or PEM data. This object could either contain methods to encode/decode or use a separate API performs operations on it. While it does provide an independent representation of PEM data, too much flexibility is a negative. Encoding operations could be called on an object initialized with PEM data or decoding from a PrivateKey. Having distinct encoding and decoding paths from given data gives clear operational usability.
Single Class API
This API design is defined as encoding and decoding from the same class. The initial prototype used static methods which complicated encrypted private keys usage, as described in Static Method API section. While switch to non-static with immutable configurations addresses this, this design also does not have distinct encoding and decoding paths, like the Intermediate PEM Object API. The Base64 API could be looked at as a comparable to the single class design, but it is not. Base64 has encoder and decoder classes under a single class. The PEM API differs by avoiding the umbrella single class and just providing the underlying operational classes.
Creating a new JCE API/SPI for Encodings
JCE Providers provide many services through the Java API, like Crypto or MessageDigest. Adding an Encoding service would make logical sense. Binary encodings have not been part of that infrastructure but used by individual provider for importing or exporting cryptographic objects. Adding services for textual and binary encodings could be useful beyond PEM.
Unfortunately, this alternative provides nothing beyond a lot of infrastructure to deliver what most do not need. Getting an instance of a binary encoding has not been a part of the Java API and too late introduce. Binary encodings are typically a part of the provider internals. It would make no sense for one provider’s encoding service to import into another provider. For keys, any encodings beyond X.509 and PKCS#8 can be supported via implementations of KeySpec.
To supporting PEM from a provider is more complicated API than java.util.Base64. Additional Java API methods would be needed for Providers to use encrypted private keys, extracting the key algorithm, and other details to use with a factory to generate a Key or Certificate object. These PEM-specific changes would eliminate any benefit of a provider-based solution. With third-party APIs existing for many years, a pluggable infrastructure offers nothing but making a simple task hard. And very unlikely to be extensible as other encodings have different features that would need individual API support.
General Purpose Encoder & Decoding API
Instead of a PEM-specific API, we looked at a generic API that could be used for many encodings. This was rejected because encodings do not all have to same features. PEM’s a single method call for all types except encrypted private keys. However other encodings can support both asymmetric and asymmetric keys, certificate chains, compression and other options. Having methods that only supported certain encodings would be confusing.
The most recent common API design was an Encoder
and Decoder
interface that required basic methods encode()
and decode()
with PEM{Encoder,Decoder} implementing the interfaces. However, those interfaces were removed to focus on PEM and not try to create a general API structure that was currently unknown. But in the future, those interfaces could be revived if a future encodings was implemented. At this time there are no plans to add another encoder/decoder API..
Testing
Tests should include:
- Verifying all supported DEREncodable classes can encode and decode PEM.
- Verifying RSA, EC, and EdDSA security objects can be encoded and decoded.
- Reading PEM generated from third-party applications, and vice-versa.
- Negative testing with bad PEM data.