Docker starts slowly on Raspberry Pi / Raspbian

After installing the latest stable version of docker-ce (5:18.09.0~3-0~raspbian-stretch as of the time of writing), Docker would take ages to start after a reboot. Commands would even time out waiting for it.

/var/log/daemon.log shows a gap of almost 4 minutes between the attempt to start the Docker Application Container Engine and the first log line from dockerd.

Jul  5 10:10:37 RPi systemd[1]: Starting Docker Application Container Engine...
Jul 5 10:14:07 RPi dockerd[683]: time="2019-07-05T10:14:07.808215720+02:00" level=info msg="parsed scheme: \"unix\"" module=grpc

After installing a later version of docker-ce from the test branch in attempt to remedy this, I got an interesting message while the issue persisted:

Jul  5 10:42:18 RPi systemd[1]: Starting Docker Application Container Engine...
Jul 5 10:43:30 RPi dockerd[720]: crypto/rand: blocked for 60 seconds waiting to read random data from the kernel
Jul 5 10:47:27 RPi dockerd[720]: time="2019-07-05T10:47:27.245479521+02:00" level=info msg="Starting up"

Turns out it doesn’t have enough random entropy to start. We can help the Pi generate this by running haveged. A quick apt-get install haveged and a reboot later, the logs look much better, and Docker starts immediately.

# apt-get install haveged
...
# reboot

The result, after downgrading to the stable version of Docker again:

Jul  5 11:08:02 RPi systemd[1]: Starting Docker Application Container Engine...
Jul 5 11:08:06 RPi dockerd[707]: time="2019-07-05T11:08:06.793516408+02:00" level=info msg="parsed scheme: \"unix\"" module=grpc

From 4 minutes to 4 seconds is a rather significant improvement, I think 🙂

Reinstalling a used iMac (20-inch, Early 2008)

So, during spring cleaning, a used iMac was discovered at work. I figured it could be used for something, so I set it up on a table. It only booted to recovery mode, so I had to reinstall it. Simple, right? No, of course not! This is an Apple product.

Apple ID needed

So, booting into the recovery mode, there’s a built-in option to reinstall the operating system. Nice. However, it immediately asks for an Apple ID. So I go make one, in a browser on another machine. You can’t use the one you’re installing to. Then I connect the iMac to the WiFi and log in.

DENIED:

This Apple ID has not yet been used with the App Store

Apple wants all your information

So, I can’t reinstall without “using the Apple ID with the App Store”. How does one do that? A bit of Google action informed me that I needed to give them a credit card. What the heck?!

I’m not aware of any other manufacturer who requires you to provide your credit card details in order to reinstall their operating system.

So I grab my VISA card, put in the details on appleid.apple.com, and try to log in on the iMac again. No luck.

An unexpected error occurred while signing in. Failure.

Turns out I have to restart the recovery installer to log in again. Great job, Apple. So I restart, sign in again, and get the same message as before, saying “This Apple ID has not yet been used with the App Store”.

You need iTunes

Apparently, I don’t have to review my account information as the message says, I have to accept the license agreement. How do you do that? The forums told me to use iTunes. What the heck, again?!

So, I fire up a disposable virtual machine, install Apple’s bloated music/store software, log in, accept the license agreement, restart the iMac again and log in with the account once more.

DENIED AGAIN:

This item is temporarily unavailable. Please try again later.

Who’s the previous owner?

According to the Apple forums, the reason for the above error message is that my freshly created Apple ID is not associated with the version of Mac OS (El Capitan) I’m installing on this machine. I haven’t bought it, which is obviously true, as it’s simply what was on this machine that was in storage.

Various people have solved this by using the Apple ID that was originally used to install the machine. Not an option. I have no idea who originally installed this machine.

Full reinstall from a USB stick

Some more time on Google told me that a clean install from a USB stick was the way around this account issue. So now I need the El Capitan installation image. This used to be available in the App Store, but of course Apple removed that.

So, more Google, and I found the .dmg image on allpcworld.com. I tried various processes to get this image onto a USB stick, but in the end the only method that produced a bootable stick was to mount the .dmg on another mac, and then run an adaptation of the following commands:

$ cd /Volumes/Install\ OS\ X\ El\ Capitan/Install\ OS\ X\ El\ Capitan.app/Contents/Resources
$ sudo ./createinstallmedia --volume /Volumes/My\ USB\ Stick/ --applicationpath /Volumes/Install\ OS\ X\ El\ Capitan/Install\ OS\ X\ El\ Capitan.app/

This created a USB stick the iMac would boot. Not in any given USB port, mind you. For the USB stick to be detected, I needed to put it in the rightmost USB port when the iMac is viewed from the back.

Holding the “alt” (“option”) key on the keyboard when turning the iMac on made the USB stick show up as a bootable drive.

This copy can’t be verified

That’s right. Cue more hurdles.

This copy of the Install OS X El Capitan application can't be verified. It may have been corrupted or tampered with during downloading.

So the iMac boots from the USB stick, but it can’t verify the image. Turns out this is because the date and time on the machine is wrong. Obviously.

Luckily, there’s a terminal available on the utilities menu at the top of the screen. I set the date using the “date” command. The solution was found in this rather shaky video.

# date 0202020216
Tue 2 Feb 2016 02:02:00 CET

Then you can close the terminal and run the installer again.

Finally installed

The rest of the installation took ages, probably due to the rather slow USB stick I selected for the job. Regardless, the rest of the procedure was fairly uneventful.

The iMac is up and running, and I like Apple even less than I did previously.

The trust relationship between this workstation and the primary domain failed

So, you resurrected a VM from the cold, damp crypts of the deepest backup dungeons, and now you can’t log in because the VM doesn’t trust the domain server anymore?

That’s alright. The Windows Domain will rotate its keys from time to time, and an offline VM snapshot will not know about it.

You don’t have to re-join the domain to fix this. Simply log in with a local admin account, or a cached domain login that’s somehow an administrator of the machine. Open a PowerShell session as said administrator, and run the following command:

Reset-ComputerMachinePassword

That’s it!

This answer was found at implbits.com (local archive).

Finding the first (default) IPv4 gateway in a Windows batch script

I recently had to whip out my Windows batch scripting skills to grab the default gateway for a routing script. Here’s what I ended up with:

route print | findstr /R /C:"^[ ][ ]*0.0.0.0[ ]" | for /f "tokens=3" %%i in ('more') do (echo %%i & exit) >gw.tmp
set /P gw=<gw.tmp
del gw.tmp
echo "%gw%"

This does, unfortunately, use a temporary file (gw.tmp), and I didn’t find a non-clunky way around that. But it works, for now 🙂

Suggestions on more elegant ways of doing this in a pure batch script are welcome.

Recovering versioned files from Owncloud’s data directory

Tasked with recovering files you don’t have access to through the web interface, you can always find these through the file system if you’re the operator. You could, of course, give yourself access, but let’s save that for another time. Right now we want the files our users removed and/or can’t find.

We start by looking for them in Owncloud’s data directory.

# find /mnt/data/owncloud/ -name "AWOL Excel file.xlsm*"
/mnt/data/owncloud/files/user/files_versions/Important/Stuff/AWOL Excel file.xlsm.v1542980717
/mnt/data/owncloud/files/user/files_versions/Important/Stuff/AWOL Excel file.xlsm.v1542029758
/mnt/data/owncloud/files/user/files_versions/Important/Stuff/AWOL Excel file.xlsm.v1542303810
/mnt/data/owncloud/files/user/files_versions/Important/Stuff/AWOL Excel file.xlsm.v1542631862
/mnt/data/owncloud/files/user/files_versions/Important/Stuff/AWOL Excel file.xlsm.v1541686827
/mnt/data/owncloud/files/user/files_versions/Important/Stuff/AWOL Excel file.xlsm.v1543226434
/mnt/data/owncloud/files/user/files_versions/Important/Stuff/AWOL Excel file.xlsm.v1542889051
/mnt/data/owncloud/files/user/files_versions/Important/Stuff/AWOL Excel file.xlsm.v1541775363
/mnt/data/owncloud/files/user/files_versions/Important/Stuff/AWOL Excel file.xlsm.v1542722172
/mnt/data/owncloud/files/user/files_versions/Important/Stuff/AWOL Excel file.xlsm.v1541922874

Oh hey, there they are!

So, now we could simply copy them and ship them to our user, but let’s be a bit more fancy, since we can.

# FILE_SUFFIX=".xlsm"
# find /mnt/data/owncloud/ -name "AWOL Excel file.xlsm*" |
>   grep -E "\.v[0-9]+(\.d[0-9]+)?$" |
>   while read -r file; do
>       ts="${file##*.v}"; ts="${ts%.*}";
>       date=$(date --date="@${ts}" +"%Y%m%dT%H%M%S")
>       base="$(basename "${file%.v*}")"
>       name="${base%${FILE_SUFFIX:-}}"
>       echo "${name}-${date}${FILE_SUFFIX:-}"
>   done
AWOL Excel file-20181123T134517.xlsm
AWOL Excel file-20181112T133558.xlsm
AWOL Excel file-20181115T174330.xlsm
AWOL Excel file-20181119T125102.xlsm
AWOL Excel file-20181108T142027.xlsm
AWOL Excel file-20181126T100034.xlsm
AWOL Excel file-20181122T121731.xlsm
AWOL Excel file-20181109T145603.xlsm
AWOL Excel file-20181120T135612.xlsm
AWOL Excel file-20181111T075434.xlsm

So that’s the list of our files, with all the names neatly timestamped (translated from the UNIX timestamp at the end of Owncloud’s file names).
Not setting the $FILE_SUFFIX variable will work fine too, but you will end up with names such as “AWOL Excel file.xlsm-20181126T100034

Now, let’s copy the files to their new names in our current directory.

# FILE_SUFFIX=".xlsm"
# find /mnt/data/owncloud/ -name "AWOL Excel file.xlsm*" |
>   grep -E "\.v[0-9]+(\.d[0-9]+)?$" |
>   while read -r file; do
>       ts="${file##*.v}"; ts="${ts%.*}";
>       date=$(date --date="@${ts}" +"%Y%m%dT%H%M%S")
>       base="$(basename "${file%.v*}")"
>       name="${base%${FILE_SUFFIX:-}}"
>       cp -v "$file" "${name}-${date}${FILE_SUFFIX:-}"
>   done
'/mnt/data/owncloud/files/user/files_versions/Important/Stuff/AWOL Excel file.xlsm.v1542980717' -> 'AWOL Excel file-20181123T134517.xlsm'
'/mnt/data/owncloud/files/user/files_versions/Important/Stuff/AWOL Excel file.xlsm.v1542029758' -> 'AWOL Excel file-20181112T133558.xlsm'
[... and so on ... ]

Cool.

Here it is as a one-liner, which your browser might add a line break to anyway, in case you’re interested:

FILE_SUFFIX=".xlsm"; find /mnt/data/owncloud/ -name "AWOL Excel file.xlsm*" | grep -E "\.v[0-9]+(\.d[0-9]+)?$" | while read -r file; do ts="${file##*.v}"; ts="${ts%.*}"; date=$(date --date="@${ts}" +"%Y%m%dT%H%M%S"); base="$(basename "${file%.v*}")"; name="${base%${FILE_SUFFIX:-}}"; cp -v "$file" "${name}-${date}${FILE_SUFFIX:-}"; done

Who deleted the files from the Windows file server?

Classic whodunit – the file is gone. Who deleted it?

Well, unless you’ve already prepared for this, Windows has no log for you. Sorry.
The good news is that this is an excellent time to prepare for the next time. So let’s do that.

Enable the auditing of file operations to the Windows Event Log

Netwrix has documented the procedure for this. The relevant steps for us are:

  1. Navigate to the file share, right-click it and select “Properties” Select the “Security” tab → “Advanced” button → “Auditing” tab → Click “Add” button:
    • Select Principal: “Everyone”; Select Type: “All”; Select Applies to: “This folder, subfolders and files”; Select the following “Advanced Permissions”: “Delete subfolders and files” and “Delete”.
  2. Run gpedit.msc, create and edit new GPO → Computer Configuration → Policies → Windows Settings → Security Settings → Go to Local Policies → Audit Policy:
    • Audit object access → Define → Success and Failures.
  3. Go to “Advanced Audit Policy Configuration” → Audit Policies → Object Access:
    • Audit File System → Define → Success and Failures
    • Audit Handle Manipulation → Define → Success and Failures.
  4. Link new GPO to File Server and force the group policy update.
  5. Open Event viewer and search Security log for event ID 4656 with “File System” or “Removable Storage” task category and with “Accesses: DELETE” string. “Subject: Security ID” will show you who has deleted a file.

Extract the logs

I wrote a short PowerShell script to do this. It’s not very efficient, but it does the job, and is relatively readable 🙂

"Chewing through the log files..."
Get-WinEvent -LogName "Security" | Where-Object { $_.Id -eq 4656 } | ForEach-Object { [xml]($_.ToXml()) } | Select -ExpandProperty "Event" | Where-Object `
{
    ( $_.EventData.Data | Where-Object { $_.Name -eq "AccessList" -and $_."#text" -like "*%%1537*" } ) -and `
    ( $_.EventData.Data | Where-Object { $_.Name -eq "ObjectName" -and $_."#text" -like "R:\*" } )
} | ForEach-Object `
{
    New-Object PSObject -Property (@{
        "TimeStamp"  = ($_.System.TimeCreated.SystemTime)
        "UserName"   = ($_.EventData.Data | Where-Object { $_.Name -eq "SubjectUserName" } | Select -ExpandProperty "#text")
        "UserDomain" = ($_.EventData.Data | Where-Object { $_.Name -eq "SubjectDomainName" } | Select -ExpandProperty "#text")
        "ObjectType" = ($_.EventData.Data | Where-Object { $_.Name -eq "ObjectType" } | Select -ExpandProperty "#text")
	"ObjectName" = ($_.EventData.Data | Where-Object { $_.Name -eq "ObjectName" } | Select -ExpandProperty "#text")
    })
} | Format-Table TimeStamp,UserDomain,UserName,ObjectType,ObjectName -AutoSize
"DONE"
"Press <enter> to quit"
Read-Host | Out-Null

Pro tip: For faster printout (line by line) but worse formatting, remove the -AutoSize parameter to Format-Table

This script specifically looks for files on R:\, but you can change this to whatever you want, or remove that condition all together.

Sample output

And there you go! 🙂

ClipWriter, a PowerShell script that emulates keyboard input to transfer files and text

I wrote this tool today, as I was annoyed having to do a bunch of changes to a configuration file on a client site through a restrictive remote tool that doesn’t let you paste or transfer any files for “security reasons” (read: BOFH)

It pastes text, files and entire directory structures by simulating keyboard input. I found some tools that do stuff like this, but all of them were somewhat shady and closed source. I thought someone else might enjoy this, so here it is.

https://github.com/b01t/clipwriter

Enable remote management of Windows Server Core and Hyper-V Core

This is a reference for the commands to enable the firewall rules necessary to remotely manage Windows Server Core and Hyper-V Core.

I keep having to look these up…

  • Enable-NetFireWallRule -DisplayName “Windows Management Instrumentation (DCOM-In)”
  • Enable-NetFireWallRule -DisplayGroup “Remote Event Log Management”
  • Enable-NetFireWallRule -DisplayGroup “Remote Service Management”
  • Enable-NetFireWallRule -DisplayGroup “Remote Volume Management”
  • Enable-NetFireWallRule -DisplayGroup “Remote Scheduled Tasks Management”
  • Enable-NetFireWallRule -DisplayGroup “Windows Firewall Remote Management”

You also have to run one of them on the computer you intend to manager it from. Yes, the client.

  • Enable-NetFirewallRule -DisplayGroup “Remote Volume Management”

Woop!

Forcing Cygwin to create sane permissions on Windows

If you use Cygwin to mainly manipulate files in your regular Windows filesystem, under /cygdrive/…, you have probably seen this message more than a few times:

“The permissions on <node> are incorrectly ordered, which may cause some entries to be ineffective”

You have also likely seen “NULL SID” as the top entry in permission lists.

The Cygwin website has a page about filemodes, which explains why this happens.

In short, you have to edit /etc/fstab in Cygwin, and add “noacl” to the mount options for /cygdrive. Here is my /etc/fstab, for reference:

# /etc/fstab
# 
#    This file is read once by the first process in a Cygwin process tree.
#    To pick up changes, restart all Cygwin processes.  For a description
#    see https://cygwin.com/cygwin-ug-net/using.html#mount-table

# This is default anyway:
#none /cygdrive cygdrive binary,posix=0,user 0 0
none /cygdrive cygdrive binary,noacl,posix=0,user 0 0

After editing this option, you have to stop every single Cygwin process for it to take effect. The easy way out is to reboot your system.