In this example, we’ll be using a wildcard certificate from Let’s Encrypt, obtained through their recently released wildcard certificate offering.

What we’re doing

The use case is that we want, for one reason or another, to use this certificate in an environment that does not have unrestricted internet access, such as a health institution or an office dealing with sensitive data. As we’ll soon discover, this presents some challenges for clients trying to verify the authenticity of the certificates the internal servers present to them.

So what’s the problem?

When a client, such as a web browser, connects to a web server using SSL, that server presents a certificate to the client, including any intermediate certificates between the server certificate and the root certification authority. The client is expected to have, and trust, the root certificate. On Debian Linux and derivatives, the root certificates are provided in the ca-certificates package. On Windows they’re provided through Windows Update.

Contained within the properties of the provided certificate, and any intermediates, there’s most likely going to be one or more URL’s to Certificate Revocation List (CRL) distribution points and/or Online Certificate Status Protocol (OCSP) endpoints. Before it will trust the certificate provided by the server, any well implemented client will want to visit one of these services to ensure that none of the certificates in the chain have been revoked by their respective authorities.

How can I determinne the CRL and OCSP URL’s (Linux)?

We’ll do this initially on Linux, using OpenSSL on our PEM encoded certificate. For an example on Windows using a .pfx/.p12 encoded certificate, see below. See the Wikipedia article on X.509 certificates for a reference on commonly used certificate formats.

Our example certificate, provided by Let’s Encrypt and retrieved using certbot, is stored in four base64 encoded files:

To extract the CRL and OCSP URL’s we need to access for verification, we must investigate the contents of the cert.pem and chain.pem files. The OpenSSL tool, probably available in your package manager on Linux, is appropriate for the job.

$ openssl x509 -text -in cert.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:e9:e2:ae:d1:91:4d:bc:1e:c0:b6:65:c6:61:70:72:a2:cc
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
        Validity
            Not Before: Apr 25 10:30:02 2018 GMT
            Not After : Jul 24 10:30:02 2018 GMT
        Subject: CN = *.example.com
------ snip -----
            Authority Information Access: 
                OCSP - URI:http://ocsp.int-x3.letsencrypt.org
                CA Issuers - URI:http://cert.int-x3.letsencrypt.org/
------ snip -----

Here we see the OCSP address for our certificate. An old client that doesn’t support OCSP will not be able to check this and should always assume the certificate is not invalidated by the authority. Clients that are up to date will use OCSP here. Since CLR requires distribution of a complete list of invalidated certificates, it is not a practical solution for Let’s Encrypt due to the sheer volume of certificates and the relatively short lifetime (1 month in the example) of the issued certificates.

If we take a look at the intermediate certificate instead, we’ll see both a CRL URL, as well as one for OCSP:

$ openssl x509 -text -in chain.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0a:01:41:42:00:00:01:53:85:73:6a:0b:85:ec:a7:08
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
        Validity
            Not Before: Mar 17 16:40:46 2016 GMT
            Not After : Mar 17 16:40:46 2021 GMT
        Subject: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
------ snip -----
            Authority Information Access: 
                OCSP - URI:http://isrg.trustid.ocsp.identrust.com
                CA Issuers - URI:http://apps.identrust.com/roots/dstrootcax3.p7c

            X509v3 Authority Key Identifier: 
                keyid:C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10

            X509v3 Certificate Policies: 
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.root-x1.letsencrypt.org

            X509v3 CRL Distribution Points: 

                Full Name:
                  URI:http://crl.identrust.com/DSTROOTCAX3CRL.crl
------ snip -----

So, to validate the intermediate certificate, the client will access either of the following CRL’s:

• http://isrg.trustid.ocsp.identrust.com
• http://crl.identrust.com/DSTROOTCAX3CRL.crl

Let’s do the same using common Windows tools.

How can I determinne the CRL and OCSP URL’s (Windows)?

On Windows, we’ll use certutil.exe to dump information about our .pfx file:

PS C:\howto> certutil -v -dump .\cert.pfx
Enter PFX password:
================ Certificate 0 ================
================ Begin Nesting Level 1 ================
Element 0:
X509 Certificate:
Version: 3
Serial Number: 0a0141420000015385736a0b85eca708
------ snip -----
    1.3.6.1.5.5.7.1.1: Flags = 0, Length = 73
    Authority Information Access
        [1]Authority Info Access
             Access Method=On-line Certificate Status Protocol (1.3.6.1.5.5.7.48.1)
             Alternative Name:
                  URL=http://isrg.trustid.ocsp.identrust.com
------ snip -----
    2.5.29.31: Flags = 0, Length = 35
    CRL Distribution Points
        [1]CRL Distribution Point
             Distribution Point Name:
                  Full Name:
                       URL=http://crl.identrust.com/DSTROOTCAX3CRL.crl
------ snip -----
================ Certificate 1 ================
================ Begin Nesting Level 1 ================
------ snip -----
    1.3.6.1.5.5.7.1.1: Flags = 0, Length = 63
    Authority Information Access
        [1]Authority Info Access
             Access Method=On-line Certificate Status Protocol (1.3.6.1.5.5.7.48.1)
             Alternative Name:
                  URL=http://ocsp.int-x3.letsencrypt.org
------ snip -----

We see we get the exact same URL’s from the certutil command, that’s run on a .pfx with both the server certificate (certificate 1) and the intermediate certificate (certificate 0) in it.

We have the URL’s – Now what?

So, from the above, we know our clients (and probably our servers, too) will attempt to access the following URL’s:

• http://crl.identrust.com/DSTROOTCAX3CRL.crl
• http://isrg.trustid.ocsp.identrust.com
• http://ocsp.int-x3.letsencrypt.org

Now we need to make some holes. Unless you’re planning on grabbing the external CRL’s on a regular basis, overriding your internal DNS, in this case for crl.identrust.com, and then hosting them on an internal webserver, your machines are simply going to have to access the outside through small pinholes in order to perform certificate verification.

Since all the URL’s use “http” and don’t specify a port number, we’ll be allowing traffic to port 80, the default port for http.

If you have a egress filtering proxy, transparent or opaque, and your closed environment definitely should, it should be trivial to allow access to .identrust.com and .letsencrypt.org, which happens to be the squid notation for anything ending with those two domain names. This notation does not limit wildcard matching to the first subdomain, unlike what *.example.com does for certificates.

The less attractive alternative is to regularly resolve crl.identrust.com, isrg.trustid.ocsp.identrust.com, and ocsp.int-x3.letsencrypt.org and then allowing connections to those.

How do I know it’s working?

Your first, and perhaps easiest, clue is a web browser, if applicable. Enter the URL of your internal server in a browser and look for the usual green padlock and lack of scary warning messages, indicating that an encrypted connection was successfully established. Then consider that your browser (and Windows itself) will cache OCSP and CRL results. There are some instructions on how to clear your OCSP and CRL cache here, and I’ll include them for reference:

PS C:\howto> certutil -urlcache * delete
------ lots of spam here ------
WinHttp Cache entries deleted: 175

Now for some debugging commands. The provided ones use PowerShell, but equivalents do of course exist for Linux.

All your devices need to be able to resolve the CRL/OCSP domains:

PS C:\howto> Resolve-DnsName crl.identrust.com

Name                           Type   TTL   Section    NameHost
----                           ----   ---   -------    --------
crl.identrust.com              CNAME  60    Answer     apps.digsigtrust.com

Name       : apps.digsigtrust.com
QueryType  : A
TTL        : 60
Section    : Answer
IP4Address : 192.35.177.64

Name                   : digsigtrust.com
QueryType              : SOA
TTL                    : 60
Section                : Authority
NameAdministrator      : sys-ops.identrust.com
SerialNumber           : 2016062800
TimeToZoneRefresh      : 300
TimeToZoneFailureRetry : 3600
TimeToExpiration       : 604800
DefaultTTL             : 60

So far, so good. We also need to be able to connect to it:

PS C:\howto> Test-NetConnection -ComputerName crl.identrust.com -Port 80

ComputerName     : crl.identrust.com
RemoteAddress    : 192.35.177.64
RemotePort       : 80
InterfaceAlias   : Wi-Fi
SourceAddress    : 10.1.2.3
TcpTestSucceeded : True

The important part above is that TcpTestSucceeded returns “True”.

Finally, let’s verify that we can actually run Verify() on our certificate, since we have it.

PS C:\howto> $cert = Get-PfxCertificate .\cert.pfx
Enter password: ****

PS C:\howto> $cert.Verify()
True

If this returns “False”, there’s trouble somewhere along the line.

Conclusion

In short, you are going to need to figure out which servers your chosen certificate authority requires that you connect to, and allow all of the involved computers, phones, and other devices to connect to these servers. If you want to avoid this, you’ll have to deal with an internal, homebrew certificate authority, and getting that installed on all your devices opens a whole new can of worms. Let’s not go there today.