boltblog

Before reading this, please note that much of the information in mcelog is hardware dependent. Your mileage may vary.

Memory gone bad

So one of the servers, running an X99-WS/IPMI board from Asus, began putting errors into /var/log/mcelog. Thankfully, they were all the same, telling me the following:

mcelog: failed to prefill DIMM database from DMI data
Hardware event. This is not a software error.
MCE 0
CPU 0 BANK 11 
MISC 90840080008228c ADDR 9ce494000 
TIME 1499161840 Tue Jul 4 09:50:40 2017
MCG status:
MCi status:
Corrected error
MCi_MISC register valid
MCi_ADDR register valid
MCA: MEMORY CONTROLLER MS_CHANNEL2_ERR
Transaction: Memory scrubbing error
MemCtrl: Corrected patrol scrub error
STATUS 8c000051000800c2 MCGSTATUS 0
MCGCAP 7000c16 APICID 0 SOCKETID 0 
CPUID Vendor Intel Family 6 Model 79

So, what does this mean?

The first few lines tell us this happened from CPU 0 on BANK 11. This wasn’t much help, as the board only has 8 memory banks. It was suggested in #debian on freenode that the high bank number might be due to dual channel memory, but then how do I pinpoint the physical stick?

Enter dmidecode

mcelog tells us a weird bank number, but it has something else that’s vitally important; the address. ADDR 9ce494000 is a memory address on the faulty stick, and dmidecode can tell us which stick is responsible for that address:

# dmidecode -t 20
[...snip...]
Handle 0x005E, DMI type 20, 35 bytes
Memory Device Mapped Address
    Starting Address: 0x00800000000
    Ending Address: 0x00BFFFFFFFF
    Range Size: 16 GB
    Physical Device Handle: 0x005D
    Memory Array Mapped Address Handle: 0x0058
    Partition Row Position: 1
[...snip...]

This should be the problematic RAM stick, as address 0x009CE494000 is between 0x00800000000 and 0x00BFFFFFFFF. The stick has “Physical Device Handle” 0x005D. dmidecode can show us more information about this handle:

# dmidecode -t 17
[...snip...]
Handle 0x005D, DMI type 17, 40 bytes
Memory Device
    Array Handle: 0x0057
    Error Information Handle: Not Provided
    Total Width: 72 bits
    Data Width: 72 bits
    Size: 16384 MB
    Form Factor: RIMM
    Set: None
    Locator: DIMM_B1
    Bank Locator: NODE 1
    Type: DDR4
    Type Detail: Synchronous
    Speed: 2133 MHz
    Manufacturer: Samsung
    Serial Number: 32BFE65D
    Asset Tag: DIMM_B1_AssetTag
    Part Number: M393A2G40DB0-CPB 
    Rank: 2
    Configured Clock Speed: 2133 MHz
    Minimum Voltage: Unknown
    Maximum Voltage: Unknown
    Configured Voltage: Unknown
[...snip...]

Here you can look at the Locator, or the Asset Tag fields. Both show the memory slot as DIMM_B1. Now that’s something we can use! Looking in the motherboard manual, available online, one can see where DIMM_B1 is:

So that’s the bad stick, which will be going back to the supplier with an RMA.

Here are a couple of useful functions I wrote to kill a process tree. It’s useful when dealing with tcpserver, which refuses to disconnect a client for as long as a process it started, or a child of that process, still runs. It’s also good for tackling runaway unintentional fork bombs, rogue daemons and other mishaps.

The following script provides two functions:

Script:

function sf_killtree
{
    local ppid=$1 sig=${2:-"TERM"}
    if ! kill -0 "$ppid"; then
        builtin echo "killtree: ($ppid) - No such process" >&2
        return 1
    fi
    (( ppid == $$ )) || kill -STOP "$ppid" 2>/dev/null || :
    local pids=( $({ pgrep -P $ppid || ps -o pid= --ppid $ppid || :; } 2>/dev/null ) )
    if [[ -n "${pids[@]:-}" ]]; then
        for pid in "${pids[@]}"; do
            sf_killtree "$pid" "$sig" 2>/dev/null || :
        done
    fi
    kill "-${sig}" "$ppid" 2>/dev/null || :
    (( ppid == $$ )) || kill -CONT "$ppid" 2>/dev/null || :
    wait "$ppid" 2>/dev/null || :
}

function sf_killchildren
{
    local _term=$(builtin trap -p TERM)
    builtin trap : TERM
    sf_killtree $$ TERM;
    builtin trap - TERM
    builtin eval "$_term"
}

Addendum

Please also note the following from the KILL (2) man page:
“If pid equals 0, then sig is sent to every process in the process group of the calling process.”
While this does kill the script itself, unlike sf_killchildren, it can be a useful thing to end a script with, or to put in an EXIT trap.

Telnet servers are everywhere. They are simple, insecure by nature, outdated, and often badly implemented. Let’s do that, shall we? 🙂

TL;DR: Here’s the library.

A simple way to make any command line interface listen to incoming TCP connections is tcpserver, provided as part of the ucspi-tcp or ucspi-tcp-ipv6 package on Debian. Using this, we can make any Bash script available on the network.

Take a look at the following command:

# /usr/bin/tcpserver -c 10 -HR -u 65534 -g 65534 0.0.0.0 2048 /usr/local/bin/script

Here, we’re running tcpserver as root. This is to allow it to later change to a lower privilege level by switching to UID 65534 and GID 65534, nobody:nogroup. We’re also limiting the number of simultaneous connections to 10 and skipping a few reverse lookups. We’re listening on all available interfaces on port 2048.

Each incoming connection will start one instance of the script /usr/local/bin/script, which will be running as the user and group provided. If you do not specify -u and -g, the script will be running as the user that started tcpserver. Be careful with this.

Everything printed by the script to stdout will go to the client, and attempting to read from stdin will read from the client. We can test this with a short script:

#!/usr/bin/env bash
echo -n "Please enter your name: "
read -r name
echo "Hello, $name"
echo

Running this script through tcpserver, I can now connect to it with telnet:

$ telnet localhost 2048
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Please enter your name: Testy McTest
Hello, Testy McTest

Connection closed by foreign host.

That was easy. Now for the annoying stuff. RFC 854 specifies all sorts of things we should pay attention to, such as option negotiation and escape sequences. Most of this isn’t important, but we do need to pay attention to some of it if we want our telnet server to be smarter than a bag of hammers. Handling ^C (ctrl+c) would be good, as would proper telnet line endings, as if you try to connect to the above script with telnet on Windows, it won’t be pretty.

Responding properly to escape sequences requires reading incoming characters one by one – we can’t wait for a line ending. Properly formatting our line endings requires either printing every line like printf “Hello!\r\n”. Alternatively, all output could be run though another command that replaces Linux style newlines (\n) with Telnet style line endings (\r\n).

I wrote a set of functions to handle all of this for me. I included them in the Cathedral ShellFunc library, but they can run on their own if a few supporting functions are exported as well.
You can get shellfunc_telnet here if you’re interested.

With it, our script will work with Windows clients, line editing functions correctly, ctrl+c will disconnect properly, line endings are corrected, and we have login support with characters displayed as asterisks.

Here’s an example script that uses shellfunc_telnet:

#!/usr/bin/env bash
source shellfunc_telnet

echo "$TCPREMOTEIP:$TCPREMOTEPORT ++ ($$)" >&2
trap 'sf_killchildren || :; echo "$TCPREMOTEIP:$TCPREMOTEPORT -- ($$)" >&2' EXIT

# the password is "test"
testhash="9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08"

sf_tn_init echo

if ! sf_tn_read -t 120 -P "Please enter your name: " name || [[ -z "$name" ]]; then
    echo -e "\nNo name, no entry"
    exit 0
fi
echo "Hello, $name"

if ! sf_tn_read -t 120 -P "Password: " -p pass; then
    echo
    exit 0
fi
echo "Your pass hash is $pass"

if [[ "$pass" != "$testhash" ]]; then
    echo "Password incorrect. Go away."
    exit 0
fi

echo "Password correct. Welcome."
echo "I will now loop and print your lines back to you."
echo "Press ctrl+c or give me an empty line to quit"

while true; do
    sf_tn_read -t 120 line
    if (( $? == 69 )); then # interrupt
        echo "Interrupted!"
    elif (( $? > 128 )); then # timeout
        echo "Too slow!"
    fi

    [[ -n "$line" ]] || break

    echo "Your line: $line"
done

echo "Bye."
echo
sleep 1

Here’s a sample session:

$ telnet localhost 2048
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Please enter your name: Testy McTest
Hello, Testy McTest
Password: ****
Your pass hash is 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
Password correct. Welcome.
I will now loop and print your lines back to you.
Press ctrl+c or give me an empty line to quit
line 1
Your line: line 1
foo
Your line: foo
Q
Your line: Q
Interrupted!
Bye.

Connection closed by foreign host.

It even works from Windows!

We can make the script run at boot time by adding it to root’s crontab as such:

@reboot /usr/bin/tcpserver -c 10 -HR -u 65534 -g 65534 0.0.0.0 2048 /usr/local/bin/script 2>&1 | logger -t tcp-2048 -p user.notice

Here, we’re running our tcpserver at boot time, we’re taking anything our script prints to standard error, which is currently a log of new connections and disconnects, and putting it in the system log.

If you’re curious about sf_killchildren, you can read more about it here.

Have fun. Drop me a line if you do something cool with this.

Resource used: The Telnet Protocol (archived here)

So.. this shit again.

# apt-get update
Ign http://ftp.debian.org jessie-updates InRelease
Ign http://ftp.debian.org jessie-updates Release.gpg
Ign http://ftp.debian.org jessie-updates Release
Ign http://ftp.debian.org jessie-updates/main Sources/DiffIndex
Ign http://ftp.debian.org jessie-updates/main i386 Packages/DiffIndex
Ign http://ftp.no.debian.org jessie InRelease
Ign http://ftp.no.debian.org jessie Release.gpg
Ign http://ftp.no.debian.org jessie Release
Ign http://ftp.no.debian.org jessie/main Sources/DiffIndex
Ign http://ftp.no.debian.org jessie/main i386 Packages/DiffIndex
Ign http://security.debian.org jessie/updates InRelease
Ign http://security.debian.org jessie/updates Release.gpg
Ign http://security.debian.org jessie/updates Release
Ign http://security.debian.org jessie/updates/main Sources/DiffIndex
Ign http://security.debian.org jessie/updates/main i386 Packages/DiffIndex
Err http://security.debian.org jessie/updates/main Sources 404 Not Found [IP: 212.211.132.250 80]
Err http://security.debian.org jessie/updates/main i386 Packages 404 Not Found [IP: 212.211.132.250 80]
Err http://ftp.debian.org jessie-updates/main Sources 404 Not Found [IP: 130.89.148.12 80]
Err http://ftp.debian.org jessie-updates/main i386 Packages 404 Not Found [IP: 130.89.148.12 80]
Err http://ftp.no.debian.org jessie/main Sources 404 Not Found [IP: 130.239.18.173 80]
Err http://ftp.no.debian.org jessie/main i386 Packages 404 Not Found [IP: 130.239.18.173 80]
W: Failed to fetch http://ftp.debian.org/debian/dists/jessie-updates/main/source/Sources 404 Not Found [IP: 130.89.148.12 80]
W: Failed to fetch http://ftp.debian.org/debian/dists/jessie-updates/main/binary-i386/Packages 404 Not Found [IP: 130.89.148.12 80]
W: Failed to fetch http://ftp.no.debian.org/debian/dists/jessie/main/source/Sources 404 Not Found [IP: 130.239.18.173 80]
W: Failed to fetch http://ftp.no.debian.org/debian/dists/jessie/main/binary-i386/Packages 404 Not Found [IP: 130.239.18.173 80]
W: Failed to fetch http://security.debian.org/dists/jessie/updates/main/source/Sources 404 Not Found [IP: 212.211.132.250 80]
W: Failed to fetch http://security.debian.org/dists/jessie/updates/main/binary-i386/Packages 404 Not Found [IP: 212.211.132.250 80]
E: Some index files failed to download. They have been ignored, or old ones used instead.

If you really want to see what happens here, you can go after it with Wireshark and look at the raw http calls. I won’t go into detail, but the problem is the partially downloaded package lists, which on Debian reside in /var/lib/apt/lists/partial/

# ll /var/lib/apt/lists/partial/*
-rw-r--r-- 1 root root 140K Aug 16 2016 /var/lib/apt/lists/partial/ftp.debian.org_debian_dists_jessie-updates_InRelease.reverify
-rw-r--r-- 1 root root 2.4K Jun 4 2016 /var/lib/apt/lists/partial/ftp.no.debian.org_debian_dists_jessie_Release.gpg.reverify
-rw-r--r-- 1 root root 62K Aug 15 2016 /var/lib/apt/lists/partial/security.debian.org_dists_jessie_updates_InRelease.reverify

(your file listing may, and probably will, differ)

Anyway, remove them by issuing rm /var/lib/apt/lists/partial/* and update again.

Note that this is but one of the errors that can cause this kind of a mess. Another one I’ve experienced is a somewhat overzealous firewall. If the above does not resolve a problem, check if you’re in any way filtering outbound traffic.