Duplicate bind mounts with chroots on systemd

When setting up jails, I commonly end up with structures like this in my /etc/fstab:

/dev /jail/test/dev auto bind 0 0
/dev/pts /jail/test/dev/pts auto bind 0 0
/dev/shm /jail/test/dev/shm auto bind 0 0
/proc /jail/test/proc auto bind 0 0
/sys /jail/test/sys auto bind 0 0

/dev /jail/test2/dev auto bind 0 0
/dev/pts /jail/test2/dev/pts auto bind 0 0
/dev/shm /jail/test2/dev/shm auto bind 0 0
/proc /jail/test2/proc auto bind 0 0
/sys /jail/test2/sys auto bind 0 0

Now, if you take a bind mount and attempt to mount it while it’s already mounted, your system will most likely let you…

# findmnt /jail/test/dev
TARGET            SOURCE FSTYPE   OPTIONS
/jail/test/dev udev   devtmpfs rw,relatime,size=10240k,nr_inodes=1012462,mode=755
# mount /jail/test/dev
# findmnt /jail/test/dev
TARGET            SOURCE FSTYPE   OPTIONS
/jail/test/dev udev   devtmpfs rw,relatime,size=10240k,nr_inodes=1012462,mode=755
/jail/test/dev udev   devtmpfs rw,relatime,size=10240k,nr_inodes=1012462,mode=755

That’s just an effect of how bind mounts happen to work. However, I found that as soon as I had several of these jails, all bind mounting /dev and various paths in it into the jails, I began seeing duplicates. Lots of duplicates.

# findmnt /jail | wc -l
5172

…that’s bad

Turns out it’s caused by the mount propagation feature, in which a bind mount is supposed to receive additional binds if something is mounted in its parent. Here from man mount (8):

Since Linux 2.6.15 it is possible to mark a mount and its submounts
as shared, private, slave or unbindable. A shared mount provides the
ability to create mirrors of that mount such that mounts and unmounts
within any of the mirrors propagate to the other mirror. A slave
mount receives propagation from its master, but not vice versa. A
private mount carries no propagation abilities. An unbindable mount
is a private mount which cannot be cloned through a bind operation.
The detailed semantics are documented in
Documentation/filesystems/sharedsubtree.txt file in the kernel source
tree.

What happens is that the default “shared” propagation causes any bind below /jail/test/dev, such as /jail/test/dev/pts get propagated up to /dev. It gets even worse when there are many jails, as the propagation down to /dev continues back up to any previously mounted jails:

# mount | grep "type dev"
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=61165,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
# mount /jail/test/dev && mount | grep "type dev" && echo "so far so good..."
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=61165,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
udev on /jail/test/dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=61165,mode=755)
so far so good...

# mount /jail/test/dev/pts && mount | grep "type dev" && echo "wait, wtf?"
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=61165,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
udev on /jail/test/dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=61165,mode=755)
devpts on /jail/test/dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
wait, wtf?

# mount /jail/test2/dev && mount /jail/test2/dev/pts && mount | grep "type dev"
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=61165,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
udev on /jail/test/dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=61165,mode=755)
devpts on /jail/test/dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
udev on /jail/test2/dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=61165,mode=755)
devpts on /jail/test2/dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /jail/test/dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)

As seen in the documentation, a private mount “carries no propagation abilities”. Thus, all I had to do was make all my jail bind mounts private, as so:

/dev /jail/test/dev auto bind,private 0 0
/dev/pts /jail/test/dev/pts auto bind,private 0 0
/dev/shm /jail/test/dev/shm auto bind,private 0 0
/proc /jail/test/proc auto bind,private 0 0
/sys /jail/test/sys auto bind,private 0 0

/dev /jail/test2/dev auto bind,private 0 0
/dev/pts /jail/test2/dev/pts auto bind,private 0 0
/dev/shm /jail/test2/dev/shm auto bind,private 0 0
/proc /jail/test2/proc auto bind,private 0 0
/sys /jail/test2/sys auto bind,private 0 0

And, after a while…

# findmnt /jail | wc -l
27

That’s better 🙂

Now, I could probably make mount propagation work for me and not manually specify binds for pts and shm and such, but I didn’t feel like spending the time to work out the implications of that.

Setting all SQL Server databases to “simple” recovery model and deleting all the transaction logs

Sometimes, you just want things brutally simple and stupid. I was searching for how to do this, and stumbled upon this post on SQL Server Central.

Turns out the query listed there doesn’t handle databases with weird names, containing version numbers with “.” in them, for instance.

Thus, I modified it slightly, adding brackets and such, and came up with the following:

USE MASTER
DECLARE
@isql varchar(2000),
@dbname varchar(64),
@logfile varchar(128)
    
DECLARE c1 CURSOR FOR
    SELECT  d.name, mf.name as logfile--, physical_name AS current_file_location, size
    FROM sys.master_files mf
    INNER JOIN sys.databases d
    ON mf.database_id = d.database_id
    WHERE recovery_model_desc <> 'SIMPLE'
    AND d.name NOT IN ('master','model','msdb','tempdb') 
    AND mf.type_desc = 'LOG'
OPEN c1
FETCH NEXT FROM c1 INTO @dbname, @logfile
WHILE @@fetch_status <> -1
    BEGIN
    SELECT @isql = 'ALTER DATABASE [' + @dbname + '] SET RECOVERY SIMPLE'
    PRINT @isql
    --EXEC(@isql)
    SELECT @isql='USE [' + @dbname + '] checkpoint'
    PRINT @isql
    --EXEC(@isql)
    SELECT @isql='USE [' + @dbname + '] DBCC SHRINKFILE ([' + @logfile + '], 0)'
    PRINT @isql
    --EXEC(@isql)
    
    FETCH NEXT FROM c1 INTO @dbname, @logfile
    END
CLOSE c1
DEALLOCATE c1

NOTE: The “EXEC()” statements are commented out, so you can inspect what’s going to happen before adding them back in.

Debugging SQL Server Query Performance

To enable timing of your query:

SET STATISTICS TIME ON

Time statistics provides output like this in the “Messages” tab of SSMS after running a query:

(127 row(s) affected)

 SQL Server Execution Times:
   CPU time = 0 ms,  elapsed time = 42 ms.

To show IO statistics:

SET STATISTICS IO ON

..which provides stuff like this:

(74394 row(s) affected)
Table 'Audit'. Scan count 1, logical reads 284, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Users'. Scan count 1, logical reads 27, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

This shows you things like logical and physical reads (memory vs. disk reads). As you can see, my current query runs entirely in memory, but we can change that…

To completely flush the disk cache, query cache, and whatever other cache and start with a clean slate

DBCC FREESYSTEMCACHE ('All')
DBCC FREESESSIONCACHE
DBCC FREEPROCCACHE
CHECKPOINT
DBCC DROPCLEANBUFFERS

Lastly, I’d like to point in the general direction of Erland Sommarskog‘s excellent article with a very long title:
Slow in the Application, Fast in SSMS? – Understanding Performance Mysteries” – archived here.

Happy debugging!

Force SQL Server Database offline

If you right click a database in SSMS and select Tasks => Take Offline, you might find yourself staring at this dialog for hours, if there are active sessions running queries on your database.

Offline

Here’s how to force it to go offline, rolling back any current transactions. Replace [dbname] with the name of your database.

USE master
GO
ALTER DATABASE [dbname]
SET OFFLINE WITH ROLLBACK IMMEDIATE
GO

To get it back online, you can do this

USE master
GO
ALTER DATABASE [dbname]
SET ONLINE
GO

Automatic Windows Logon on Domain Member Machines

On a Windows machine, you can normally use Start => Run => control userpasswords2 to enable automatic login for a given user when the system boots. On computers that are part of a domain, this is not the case. However, while the functionality is removed from the user interface when you join a domain, it is still available if you’re comfortable mucking about with the registry.

Using Regedit, go to “Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\WinLogon“.

There are 4 keys you need to set to allow automatic logon:

  • AutoAdminLogon, REG_SZ => 1
  • DefaultDomainName, REG_SZ => YourDomainNameHere
  • DefaultPassword, REG_SZ => YourSecretPasswordInClearText
  • DefaultUserName, REG_SZ => YourUserName

Add or edit the keys to match your domain setup and reboot. Autologin should function as intended after this.

AutoLoginWindowsDomain

Fixing empty search results in the Windows 10 Settings (the shiny new Control Panel)

This happened on a few machines I am responsible for, after the upgrade to Windows 10.
You click “Start”, type “updates”, it suggests “Check for Updates” in the “Settings” app, you click <enter> and it opens an empty settings window with no search results. Great.

After looking around for a while, I stumbled over the fix, detailed by winaero.com and rchived here.

In short:

  1. Win+R
  2. %LocalAppData%\Packages\windows.immersivecontrolpanel_cw5n1h2txyewy\LocalStatez
  3. Right click the directory called “Indexed”, go to Properties => Advanced
  4. Click “Allow files in this folder to have contents indexed in addition to file properties”
    If this option is already selected, de-select it, finish point 5, then go back through this list again to re-select it
  5. Click OK a few times, and agree to propagate the settings to subfolders and files

Hooray! Search works.

Working Search

W: [pulseaudio] authkey.c: Failed to open cookie file ‘/home/user/.config/pulse/cookie’: No such file or directory

W: [pulseaudio] authkey.c: Failed to open cookie file '/home/user/.config/pulse/cookie': No such file or directory
W: [pulseaudio] authkey.c: Failed to load authorization key '/home/user/.config/pulse/cookie': No such file or directory

I had this problem on an embedded box, with no X11, or even a screen. Turns out that at least some versions of Debian have an issue where not all programs agree on where the cookie file should be stored. Doing ls -la in my home directory revealed I had a ~/.pulse-cookie file, but I didn’t, as the warning message stated, have a ~/.config/pulse/cookie file.

I made this warning go away by symlinking where one program was looking for the cookie to where the cookie actually was.

mkdir -p ~/.config/pulse
cd ~/.config/pulse
ln -s ../../.pulse-cookie cookie

..and Bob’s your uncle. No more warnings.

Export certificates marked as not exportable in the Windows certificate manager

So, you need the private key for a certificate on Windows, for some innocent snooping around with Wireshark, but someone marked it as not exportable. Now what?

Cue Gentil Kiwi and his tool Mimikatz.

For future reference for myself, I’ve archived a copy of the source here, and the binaries here.

The following commands will extract the certificates from the local store:

crypto::capi
crypto::certificates /systemstore=CERT_SYSTEM_STORE_LOCAL_MACHINE /export

The password for the pfx files is “mimikatz” (no quotes).

To convert a pfx to a pem file, you can do something like this:

openssl pkcs12 -in CERT_SYSTEM_STORE_LOCAL_MACHINE_nicecert.pfx -out cert.pem -nodes

If it’s for use in Wireshark, you also need to add -nocerts:

openssl pkcs12 -in CERT_SYSTEM_STORE_LOCAL_MACHINE_nicecert.pfx -out cert.pem -nodes -nocerts