In this example, we’ll be using a wildcard certificate from Let’s Encrypt, obtained through their recently released wildcard certificate offering.
What we’re doing
The use case is that we want, for one reason or another, to use this certificate in an environment that does not have unrestricted internet access, such as a health institution or an office dealing with sensitive data. As we’ll soon discover, this presents some challenges for clients trying to verify the authenticity of the certificates the internal servers present to them.
So what’s the problem?
When a client, such as a web browser, connects to a web server using SSL, that server presents a certificate to the client, including any intermediate certificates between the server certificate and the root certification authority. The client is expected to have, and trust, the root certificate. On Debian Linux and derivatives, the root certificates are provided in the ca-certificates package. On Windows they’re provided through Windows Update.
Contained within the properties of the provided certificate, and any intermediates, there’s most likely going to be one or more URL’s to Certificate Revocation List (CRL) distribution points and/or Online Certificate Status Protocol (OCSP) endpoints. Before it will trust the certificate provided by the server, any well implemented client will want to visit one of these services to ensure that none of the certificates in the chain have been revoked by their respective authorities.
How can I determinne the CRL and OCSP URL’s (Linux)?
We’ll do this initially on Linux, using OpenSSL on our PEM encoded certificate. For an example on Windows using a .pfx/.p12 encoded certificate, see below. See the Wikipedia article on X.509 certificates for a reference on commonly used certificate formats.
Our example certificate, provided by Let’s Encrypt and retrieved using certbot, is stored in four base64 encoded files:
| cert.pem | The public part of the certificate, which is passed on to clients by SSL/TLS servers on authentication |
| chain.pem | The public part of the certificates for any intermediate certificate authorities in the chain |
| fullchain.pem | This is simply the public certificate, followed by the chain. cert.pem and chain.pem in one file. Some configurations warrants this input. |
| privkey.pem | The private part of the certificate, not to be given to anyone, ever. This is your key to the public part of the certificate. |
To extract the CRL and OCSP URL’s we need to access for verification, we must investigate the contents of the cert.pem and chain.pem files. The OpenSSL tool, probably available in your package manager on Linux, is appropriate for the job.
$ openssl x509 -text -in cert.pem
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
03:e9:e2:ae:d1:91:4d:bc:1e:c0:b6:65:c6:61:70:72:a2:cc
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
Validity
Not Before: Apr 25 10:30:02 2018 GMT
Not After : Jul 24 10:30:02 2018 GMT
Subject: CN = *.example.com
------ snip -----
Authority Information Access:
OCSP - URI:http://ocsp.int-x3.letsencrypt.org
CA Issuers - URI:http://cert.int-x3.letsencrypt.org/
------ snip -----
Here we see the OCSP address for our certificate. An old client that doesn’t support OCSP will not be able to check this and should always assume the certificate is not invalidated by the authority. Clients that are up to date will use OCSP here. Since CLR requires distribution of a complete list of invalidated certificates, it is not a practical solution for Let’s Encrypt due to the sheer volume of certificates and the relatively short lifetime (1 month in the example) of the issued certificates.
If we take a look at the intermediate certificate instead, we’ll see both a CRL URL, as well as one for OCSP:
$ openssl x509 -text -in chain.pem
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
0a:01:41:42:00:00:01:53:85:73:6a:0b:85:ec:a7:08
Signature Algorithm: sha256WithRSAEncryption
Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
Validity
Not Before: Mar 17 16:40:46 2016 GMT
Not After : Mar 17 16:40:46 2021 GMT
Subject: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
------ snip -----
Authority Information Access:
OCSP - URI:http://isrg.trustid.ocsp.identrust.com
CA Issuers - URI:http://apps.identrust.com/roots/dstrootcax3.p7c
X509v3 Authority Key Identifier:
keyid:C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10
X509v3 Certificate Policies:
Policy: 2.23.140.1.2.1
Policy: 1.3.6.1.4.1.44947.1.1.1
CPS: http://cps.root-x1.letsencrypt.org
X509v3 CRL Distribution Points:
Full Name:
URI:http://crl.identrust.com/DSTROOTCAX3CRL.crl
------ snip -----
So, to validate the intermediate certificate, the client will access either of the following CRL’s:
- http://isrg.trustid.ocsp.identrust.com
- http://crl.identrust.com/DSTROOTCAX3CRL.crl
Let’s do the same using common Windows tools.
How can I determinne the CRL and OCSP URL’s (Windows)?
On Windows, we’ll use certutil.exe to dump information about our .pfx file:
PS C:\howto> certutil -v -dump .\cert.pfx
Enter PFX password:
================ Certificate 0 ================
================ Begin Nesting Level 1 ================
Element 0:
X509 Certificate:
Version: 3
Serial Number: 0a0141420000015385736a0b85eca708
------ snip -----
1.3.6.1.5.5.7.1.1: Flags = 0, Length = 73
Authority Information Access
[1]Authority Info Access
Access Method=On-line Certificate Status Protocol (1.3.6.1.5.5.7.48.1)
Alternative Name:
URL=http://isrg.trustid.ocsp.identrust.com
------ snip -----
2.5.29.31: Flags = 0, Length = 35
CRL Distribution Points
[1]CRL Distribution Point
Distribution Point Name:
Full Name:
URL=http://crl.identrust.com/DSTROOTCAX3CRL.crl
------ snip -----
================ Certificate 1 ================
================ Begin Nesting Level 1 ================
------ snip -----
1.3.6.1.5.5.7.1.1: Flags = 0, Length = 63
Authority Information Access
[1]Authority Info Access
Access Method=On-line Certificate Status Protocol (1.3.6.1.5.5.7.48.1)
Alternative Name:
URL=http://ocsp.int-x3.letsencrypt.org
------ snip -----
We see we get the exact same URL’s from the certutil command, that’s run on a .pfx with both the server certificate (certificate 1) and the intermediate certificate (certificate 0) in it.
We have the URL’s – Now what?
So, from the above, we know our clients (and probably our servers, too) will attempt to access the following URL’s:
- http://crl.identrust.com/DSTROOTCAX3CRL.crl
- http://isrg.trustid.ocsp.identrust.com
- http://ocsp.int-x3.letsencrypt.org
Now we need to make some holes. Unless you’re planning on grabbing the external CRL’s on a regular basis, overriding your internal DNS, in this case for crl.identrust.com, and then hosting them on an internal webserver, your machines are simply going to have to access the outside through small pinholes in order to perform certificate verification.
Since all the URL’s use “http” and don’t specify a port number, we’ll be allowing traffic to port 80, the default port for http.
If you have a egress filtering proxy, transparent or opaque, and your closed environment definitely should, it should be trivial to allow access to .identrust.com and .letsencrypt.org, which happens to be the squid notation for anything ending with those two domain names. This notation does not limit wildcard matching to the first subdomain, unlike what *.example.com does for certificates.
The less attractive alternative is to regularly resolve crl.identrust.com, isrg.trustid.ocsp.identrust.com, and ocsp.int-x3.letsencrypt.org and then allowing connections to those.
How do I know it’s working?
Your first, and perhaps easiest, clue is a web browser, if applicable. Enter the URL of your internal server in a browser and look for the usual green padlock and lack of scary warning messages, indicating that an encrypted connection was successfully established. Then consider that your browser (and Windows itself) will cache OCSP and CRL results. There are some instructions on how to clear your OCSP and CRL cache here, and I’ll include them for reference:
PS C:\howto> certutil -urlcache * delete ------ lots of spam here ------ WinHttp Cache entries deleted: 175
Now for some debugging commands. The provided ones use PowerShell, but equivalents do of course exist for Linux.
All your devices need to be able to resolve the CRL/OCSP domains:
PS C:\howto> Resolve-DnsName crl.identrust.com Name Type TTL Section NameHost ---- ---- --- ------- -------- crl.identrust.com CNAME 60 Answer apps.digsigtrust.com Name : apps.digsigtrust.com QueryType : A TTL : 60 Section : Answer IP4Address : 192.35.177.64 Name : digsigtrust.com QueryType : SOA TTL : 60 Section : Authority NameAdministrator : sys-ops.identrust.com SerialNumber : 2016062800 TimeToZoneRefresh : 300 TimeToZoneFailureRetry : 3600 TimeToExpiration : 604800 DefaultTTL : 60
So far, so good. We also need to be able to connect to it:
PS C:\howto> Test-NetConnection -ComputerName crl.identrust.com -Port 80 ComputerName : crl.identrust.com RemoteAddress : 192.35.177.64 RemotePort : 80 InterfaceAlias : Wi-Fi SourceAddress : 10.1.2.3 TcpTestSucceeded : True
The important part above is that TcpTestSucceeded returns “True”.
Finally, let’s verify that we can actually run Verify() on our certificate, since we have it.
PS C:\howto> $cert = Get-PfxCertificate .\cert.pfx Enter password: **** PS C:\howto> $cert.Verify() True
If this returns “False”, there’s trouble somewhere along the line.
Conclusion
In short, you are going to need to figure out which servers your chosen certificate authority requires that you connect to, and allow all of the involved computers, phones, and other devices to connect to these servers. If you want to avoid this, you’ll have to deal with an internal, homebrew certificate authority, and getting that installed on all your devices opens a whole new can of worms. Let’s not go there today.