Script for creating a compressed image file from a Raspbian SD card

I’ve previously presented a manual process for doing this, but lately I’ve had to do it more often, and I figured it was about time for automation.

This script will:

  • Run e2fsck on the root file system
  • Erase logfiles, such as bash_history and stuff in /var/log
  • Erase the test.h264 video file
  • Wipe resolv.conf
  • Defragment the root file system
  • Resize the root file system to the minimum size, according to resize2fs
  • Resize the root partition to match the file system
  • Zero-fill remaining free space on both partitions
  • Create an image file of the exact length of the partitions
  • Compress the image file to .zip, using the “ultra” setting for compression

Required runtime tools include:

  • dd
  • 7z
  • resize2fs
  • dumpe2fs
  • e4defrag

This script is intended for Raspbian SD card images only, and may not work as intended with other distributions.

The script is currently hosted on github, and of course, there’s a local copy.

Using external Certificate Authority certificates in a restricted or closed environment

In this example, we’ll be using a wildcard certificate from Let’s Encrypt, obtained through their recently released wildcard certificate offering.

What we’re doing

The use case is that we want, for one reason or another, to use this certificate in an environment that does not have unrestricted internet access, such as a health institution or an office dealing with sensitive data. As we’ll soon discover, this presents some challenges for clients trying to verify the authenticity of the certificates the internal servers present to them.

So what’s the problem?

When a client, such as a web browser, connects to a web server using SSL, that server presents a certificate to the client, including any intermediate certificates between the server certificate and the root certification authority. The client is expected to have, and trust, the root certificate. On Debian Linux and derivatives, the root certificates are provided in the ca-certificates package. On Windows they’re provided through Windows Update.

Contained within the properties of the provided certificate, and any intermediates, there’s most likely going to be one or more URL’s to Certificate Revocation List (CRL) distribution points and/or Online Certificate Status Protocol (OCSP) endpoints. Before it will trust the certificate provided by the server, any well implemented client will want to visit one of these services to ensure that none of the certificates in the chain have been revoked by their respective authorities.

How can I determinne the CRL and OCSP URL’s (Linux)?

We’ll do this initially on Linux, using OpenSSL on our PEM encoded certificate. For an example on Windows using a .pfx/.p12 encoded certificate, see below. See the Wikipedia article on X.509 certificates for a reference on commonly used certificate formats.

Our example certificate, provided by Let’s Encrypt and retrieved using certbot, is stored in four base64 encoded files:

cert.pem The public part of the certificate, which is passed on to clients by SSL/TLS servers on authentication
chain.pem The public part of the certificates for any intermediate certificate authorities in the chain
fullchain.pem This is simply the public certificate, followed by the chain. cert.pem and chain.pem in one file. Some configurations warrants this input.
privkey.pem The private part of the certificate, not to be given to anyone, ever. This is your key to the public part of the certificate.

 

To extract the CRL and OCSP URL’s we need to access for verification, we must investigate the contents of the cert.pem and chain.pem files. The OpenSSL tool, probably available in your package manager on Linux, is appropriate for the job.

$ openssl x509 -text -in cert.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:e9:e2:ae:d1:91:4d:bc:1e:c0:b6:65:c6:61:70:72:a2:cc
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
        Validity
            Not Before: Apr 25 10:30:02 2018 GMT
            Not After : Jul 24 10:30:02 2018 GMT
        Subject: CN = *.example.com
------ snip -----
            Authority Information Access: 
                OCSP - URI:http://ocsp.int-x3.letsencrypt.org
                CA Issuers - URI:http://cert.int-x3.letsencrypt.org/
------ snip -----

Here we see the OCSP address for our certificate. An old client that doesn’t support OCSP will not be able to check this and should always assume the certificate is not invalidated by the authority. Clients that are up to date will use OCSP here. Since CLR requires distribution of a complete list of invalidated certificates, it is not a practical solution for Let’s Encrypt due to the sheer volume of certificates and the relatively short lifetime (1 month in the example) of the issued certificates.

If we take a look at the intermediate certificate instead, we’ll see both a CRL URL, as well as one for OCSP:

$ openssl x509 -text -in chain.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0a:01:41:42:00:00:01:53:85:73:6a:0b:85:ec:a7:08
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
        Validity
            Not Before: Mar 17 16:40:46 2016 GMT
            Not After : Mar 17 16:40:46 2021 GMT
        Subject: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
------ snip -----
            Authority Information Access: 
                OCSP - URI:http://isrg.trustid.ocsp.identrust.com
                CA Issuers - URI:http://apps.identrust.com/roots/dstrootcax3.p7c

            X509v3 Authority Key Identifier: 
                keyid:C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10

            X509v3 Certificate Policies: 
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.root-x1.letsencrypt.org

            X509v3 CRL Distribution Points: 

                Full Name:
                  URI:http://crl.identrust.com/DSTROOTCAX3CRL.crl
------ snip -----

So, to validate the intermediate certificate, the client will access either of the following CRL’s:

  • http://isrg.trustid.ocsp.identrust.com
  • http://crl.identrust.com/DSTROOTCAX3CRL.crl

Let’s do the same using common Windows tools.

How can I determinne the CRL and OCSP URL’s (Windows)?

On Windows, we’ll use certutil.exe to dump information about our .pfx file:

PS C:\howto> certutil -v -dump .\cert.pfx
Enter PFX password:
================ Certificate 0 ================
================ Begin Nesting Level 1 ================
Element 0:
X509 Certificate:
Version: 3
Serial Number: 0a0141420000015385736a0b85eca708
------ snip -----
    1.3.6.1.5.5.7.1.1: Flags = 0, Length = 73
    Authority Information Access
        [1]Authority Info Access
             Access Method=On-line Certificate Status Protocol (1.3.6.1.5.5.7.48.1)
             Alternative Name:
                  URL=http://isrg.trustid.ocsp.identrust.com
------ snip -----
    2.5.29.31: Flags = 0, Length = 35
    CRL Distribution Points
        [1]CRL Distribution Point
             Distribution Point Name:
                  Full Name:
                       URL=http://crl.identrust.com/DSTROOTCAX3CRL.crl
------ snip -----
================ Certificate 1 ================
================ Begin Nesting Level 1 ================
------ snip -----
    1.3.6.1.5.5.7.1.1: Flags = 0, Length = 63
    Authority Information Access
        [1]Authority Info Access
             Access Method=On-line Certificate Status Protocol (1.3.6.1.5.5.7.48.1)
             Alternative Name:
                  URL=http://ocsp.int-x3.letsencrypt.org
------ snip -----

We see we get the exact same URL’s from the certutil command, that’s run on a .pfx with both the server certificate (certificate 1) and the intermediate certificate (certificate 0) in it.

We have the URL’s – Now what?

So, from the above, we know our clients (and probably our servers, too) will attempt to access the following URL’s:

  • http://crl.identrust.com/DSTROOTCAX3CRL.crl
  • http://isrg.trustid.ocsp.identrust.com
  • http://ocsp.int-x3.letsencrypt.org

Now we need to make some holes. Unless you’re planning on grabbing the external CRL’s on a regular basis, overriding your internal DNS, in this case for crl.identrust.com, and then hosting them on an internal webserver, your machines are simply going to have to access the outside through small pinholes in order to perform certificate verification.

Since all the URL’s use “http” and don’t specify a port number, we’ll be allowing traffic to port 80, the default port for http.

If you have a egress filtering proxy, transparent or opaque, and your closed environment definitely should, it should be trivial to allow access to .identrust.com and .letsencrypt.org, which happens to be the squid notation for anything ending with those two domain names. This notation does not limit wildcard matching to the first subdomain, unlike what *.example.com does for certificates.

The less attractive alternative is to regularly resolve crl.identrust.com, isrg.trustid.ocsp.identrust.com, and ocsp.int-x3.letsencrypt.org and then allowing connections to those.

How do I know it’s working?

Your first, and perhaps easiest, clue is a web browser, if applicable. Enter the URL of your internal server in a browser and look for the usual green padlock and lack of scary warning messages, indicating that an encrypted connection was successfully established. Then consider that your browser (and Windows itself) will cache OCSP and CRL results. There are some instructions on how to clear your OCSP and CRL cache here, and I’ll include them for reference:

PS C:\howto> certutil -urlcache * delete
------ lots of spam here ------
WinHttp Cache entries deleted: 175

Now for some debugging commands. The provided ones use PowerShell, but equivalents do of course exist for Linux.

All your devices need to be able to resolve the CRL/OCSP domains:

PS C:\howto> Resolve-DnsName crl.identrust.com

Name                           Type   TTL   Section    NameHost
----                           ----   ---   -------    --------
crl.identrust.com              CNAME  60    Answer     apps.digsigtrust.com

Name       : apps.digsigtrust.com
QueryType  : A
TTL        : 60
Section    : Answer
IP4Address : 192.35.177.64

Name                   : digsigtrust.com
QueryType              : SOA
TTL                    : 60
Section                : Authority
NameAdministrator      : sys-ops.identrust.com
SerialNumber           : 2016062800
TimeToZoneRefresh      : 300
TimeToZoneFailureRetry : 3600
TimeToExpiration       : 604800
DefaultTTL             : 60

So far, so good. We also need to be able to connect to it:

PS C:\howto> Test-NetConnection -ComputerName crl.identrust.com -Port 80

ComputerName     : crl.identrust.com
RemoteAddress    : 192.35.177.64
RemotePort       : 80
InterfaceAlias   : Wi-Fi
SourceAddress    : 10.1.2.3
TcpTestSucceeded : True

The important part above is that TcpTestSucceeded returns “True”.

Finally, let’s verify that we can actually run Verify() on our certificate, since we have it.

PS C:\howto> $cert = Get-PfxCertificate .\cert.pfx
Enter password: ****

PS C:\howto> $cert.Verify()
True

If this returns “False”, there’s trouble somewhere along the line.

Conclusion

In short, you are going to need to figure out which servers your chosen certificate authority requires that you connect to, and allow all of the involved computers, phones, and other devices to connect to these servers. If you want to avoid this, you’ll have to deal with an internal, homebrew certificate authority, and getting that installed on all your devices opens a whole new can of worms. Let’s not go there today.

My tweaks to get Kali Linux running well on the GPD Pocket

Mostly notes to myself, but hey, maybe it helps you too!

Get Kali for the GPD Pocket

I installed using re4son’s modified image from here.

Removing the GRUB splash screen (it’s sideways anyway, which looks horrible, and I’m not multibooting)

/etc/default/grub

GRUB_TIMEOUT=0
GRUB_BACKGROUND=""

Speeding up WiFi so it doesn’t lag if you SSH into the machine, and other power management tweaks

By default, Kali on the GPD Pocket will have some weird WiFi power saving mode enabled, which means that an incoming SSH session will feel very laggy unless the GPD is constantly sending data. The result is that an SSH session feels much smoother if you’re transferring a huge file at the same time, which is rather silly. A tool called “tlp” can disable the WiFi power saving.

apt-get install -y tlp
systemctl enable tlp
vim /etc/default/tlp

Firstly, disable WiFi power saving:
--------------
WIFI_PWR_ON_AC=off
WIFI_PWR_ON_BAT=off
--------------

Speed up the disk:
--------------
DISK_DEVICES="mmcblk0"
DISK_IOSCHED="deadline"
--------------

Less problems with USB devices that don't take well to being suspended:
--------------
USB_AUTOSUSPEND=0
--------------

I like to have the machine run cool even while on AC power:
--------------
CPU_HWP_ON_AC=balance_power
SCHED_POWERSAVE_ON_AC=1
SCHED_POWERSAVE_ON_BAT=1
ENERGY_PERF_POLICY_ON_AC=power
--------------

Making sure the screen turns properly off and back on again when closing and opening the lid

If the laptop is not set to sleep, the screen on mine didn’t shut off when closing the lid, and if I shut it off with a script, it would randomly turn back on due to magnetic interference from the magnets on the case, or on my bag, or on whatever else. I wrote a script to take care of this, with an autostart file to go along with it:

/root/bin/gpd-screen-watcher
/root/.config/autostart/gpd-screen-watcher.desktop

This is public domain.
gpd-screen-watcher needs to be executable 🙂

Some additional xorg config

This is from the page about running Arch Linux on the GPD Pocket:

/etc/X11/xorg.conf.d/30-monitor.conf

Section "Monitor"
    Identifier "DSI-1"
    Option "Rotate" "right"
EndSection

/etc/X11/xorg.conf.d/50-trackpoint.conf

Section "InputClass"
    Identifier "GPD trackpoint"
    MatchProduct "SINO WEALTH Gaming Keyboard"
    MatchIsPointer "on"
    Driver "libinput"
    Option "Emulate3Buttons" "True"
    Option "MiddleEmulation" "True"
EndSection

EDIT: The last two lines of InputClass, Emulate3Buttons and MiddleEmulation, were added here on 2018-08-13, and allow clicking both mouse buttons to simulate a middle click, for pasting and such.

Done!

Have fun with Kali 😀

Outlook removes “extra line breaks” from plain text emails: How to stop it


“We removed extra line breaks from this message.”

Well, how helpful of you. Now the email from my crontab script is all in one line.

You can disable this “helpful functionality” permanently in Outlook by doing this:

  1. Open Outlook
  2. On the File tab, select Options
  3. In the Options window, select Mail
  4. In the Message format section, clear the “Remove extra line breaks in plain text messages” check box
  5. Click OK

That’s all fine and dandy for you, but what about your colleagues? Getting them all to do this would be a pain. Well, Outlook has this curious idea that lines ending with three or more spaces should not have this helpful behaviour applied to them. Thus, the easy solution to the problem is to add three spaces at the end of every. single. line.

It’s not as hard as it may sound. awk will do it for us.

awk '{ print $0"   " }'

This reads every single line, and re-prints it with three spaces at the end. Outlook is now happy.

You can easily pipe stuff into it for your crontab entries. The below is a useless example:

find /etc -name "*.conf" | awk '{ print $0"   " }'

You can also make your script print like this internally, which in Bash is done by redirecting stdout and stderr through an awk process, as such:

#!/usr/bin/env bash
exec > >(awk '{ print $0"   " }') 2>&1
echo "This goes to stdout"
echo "This goes to stderr" >&2
echo "More stdout"

After the exec line, anything printed by your script will be sent through awk and on through stdout. Outlook will no longer try to help.

How to deal with error 0xc03f6506 when upgrading stubborn Windows 10 Home Edition machines to Pro using a VLK

So, the machine auto-activates Windows 10 Home Edition at install time. No prompts, no nothing. You insert the generic upgrade key, and it throws an unhelpful error message back at you:

This probably surprises nobody, but trying again, as the above message suggests, will not work for you.

The solution? Stuff the install media (in my case a USB stick made with the Media Creation Tool), back into the machine, then run the following command:

setup.exe /auto upgrade /pkey VK7JG-NPHTM-C97JM-9MPGT-3V66T

There’s that generic upgrade key again.

After it’s done updating, proceed as usual.

Shrinking a Windows 10 installation for small harddrives

Cheap Windows 10 capable devices often come with very limited internal storage space. To make use of this, it is of course crucial that Windows itself takes up as little space as possible. To combat the bloat, there’s a little known feature in compact.exe, the built-in disk compression tool, that compresses the operating system itself, often saving several of those valuable gigabytes.

Microsoft does have some documentation on this feature, right here.

Basically, all you do is open cmd.exe as an administrator and run:

cd %windir%\system32
Compact.exe /CompactOS:always

You’ll see something like this:

To know whether or not CompactOS is enabled on your Windows installation, issue the following commands:

cd %windir%\system32
Compact.exe /CompactOS:query

Whoa! Look at all that space, mom!

Checking for required packages on a Debian (or derivative) system

Tasked with cleaning up an installation script, I noticed it was calling dpkg dozens of times to verify that the required packages were in place. On a Raspberry Pi, or similar low powered device, this takes ages. I replaced it with the following:

c=0
while read -r pkg; do
    printf 'Missing package: %s\n' "$pkg" >&2
    (( ++c ))
done < <(dpkg-query -W -f='${binary:Package}\n' | cut -d ':' -f 1 | sort | comm -13 - <(sort <<-EOF
    libc-ares2
    libssl1.0.0
    zliblg
    mosquitto
    gdb
    tcpdump
    vim
EOF
))
printf 'Total missing packages: %d\n' "$c" >&2

In short:

  • We query dpkg once, listing every installed package, cut off any version numbers and sort the list
  • We sort a list of manually specified required packages (libc-ares2 … vim)
  • We compare the sorted lists (sorting is a requirement for comm), specifying that we want to not display columns 1 and 3, which are lines unique to the left side (installed ones), and lines that appear on both sides (installed and required), respectively. This leaves colums 2, the ones unique to the right side, which are packages we require but that are missing from dpkg’s list.

Sample output:

$ ./check
Missing package: gdb
Missing package: mosquitto
Missing package: zliblg
Total missing packages: 3

Archived here for future reference, and in case it’s useful to anyone 🙂

Avoid having Visual Studio put all its floating windows on top

Working with multiple screens, I often use several windows in Visual Studio. For a while now, I’ve been annoyed by all the miscellaneous windows forcing themselves on top so I couldn’t open a browser on one screen and switch to the code on another screen without the window on the browser’s screen popping up to obscure the browser.

I finally took the time to figure this one out, and the answer is right there in the options menu. Deactivate both of the highlighted options and your windows are free to move around as you wish. They even get their own icon on the task bar, if you have a task bar on all your screens!