The Advisory Boar

By Abhijit Menon-Sen <>

What is fsck up to now?

We had unexpectedly heavy snowfall the other day and, as always, the mains power supply came back only after a few days of repairing broken power lines in the forest. Meanwhile, the days were so overcast that the solar inverter couldn't charge the batteries enough to keep up with our minimal domestic load.

Which meant that when the sun came out again, I was left staring at something like this for a long time:

root@soot:~# ps -eo pid,cmd|grep '[f]sck'
756 /lib/systemd/systemd-fsck /dev/mapper/sdb1_crypt
757 /sbin/fsck -a -T -l -M -C4 /dev/mapper/sdb1_crypt
758 /lib/systemd/systemd-fsckd
759 fsck.ext4 -a -C4 /dev/mapper/sdb1_crypt

Long enough, in fact, that I began to wonder if I could tell what it was doing. (The volume in question is exported via iSCSI from a Synology NAS and fsck is still running long after the machine has otherwise finished booting up, so I have ordinary shell access.)

I could see (from the parent pid, not shown above) that systemd-fsck had started fsck, which had started fsck.ext4 in turn. I would have guessed that systemd-fsck was started by systemd-fsckd, but both processes were children of init by the time I looked.

Here's an excerpt of what strace had to say about fsck.ext4:

pread64(3, "\0\0\0\0"..., 32768, 2403706052608) = 32768
pread64(3, "\0\0\0\0"..., 32768, 2403706085376) = 32768
pread64(3, "\0\0\0\0"..., 32768, 2403706118144) = 32768
pread64(3, "\0\0\0\0"..., 16384, 2403706150912) = 16384
fadvise64(3, 2403973734400, 868352, POSIX_FADV_WILLNEED) = 0
write(4, "1 17910 59104 /dev/mapper/sdb1_c"..., 37) = 37
pread64(3, "\0\0\0\0"..., 32768, 2403839516672) = 32768
pread64(3, "\0\0\0\0"..., 32768, 2403839549440) = 32768
pread64(3, "\0\0\0\0"..., 32768, 2403839582208) = 32768
pread64(3, "\0\0\0\0"..., 32768, 2403839614976) = 32768

So it was reading from (presumably) the block device in 32KB chunks and telling the kernel's readahead mechanism that it meant to keep going past the 2TB or so it had reached. But what's that write in the middle?

$ strace -etrace=write -s 50 -p 759
write(4, "1 19120 59104 /dev/mapper/sdb1_crypt\n", 37) = 37
write(4, "1 19121 59104 /dev/mapper/sdb1_crypt\n", 37) = 37
write(4, "1 19122 59104 /dev/mapper/sdb1_crypt\n", 37) = 37

Well, that looked promising. I could just wait for the counter (19120) to catch up to its destination (59104) and fsck would be done!

write(4, "1 59102 59104 /dev/mapper/sdb1_crypt\n", 37) = 37
write(4, "1 59103 59104 /dev/mapper/sdb1_crypt\n", 37) = 37
write(4, "1 59104 59104 /dev/mapper/sdb1_crypt\n", 37) = 37
write(4, "2 0 158345 /dev/mapper/sdb1_crypt\n", 34) = 34
write(4, "2 1 158345 /dev/mapper/sdb1_crypt\n", 34) = 34
write(4, "2 2 158345 /dev/mapper/sdb1_crypt\n", 34) = 34

Oh, wait, fsck operates in multiple passes, doesn't it?

I ran apt-get source to download a copy of e2fsprogs (the package that installs /sbin/fsck.ext4, as identified by 'dpkg -S'), and a wild guess that the output above was generated with a printf of some kind led me to a function named e2fsck_update_progress.

$ rg -A1 -tc '%d %.*%.*%s\\n'
594: snprintf(buf, sizeof(buf), "%d %lu %lu %s\n",
595-          pass, cur, max, ctx->device_name);

A few lines below is a call to a calc_percent function.

struct percent_tbl {
    int max_pass;
    int table[32];
static struct percent_tbl e2fsck_tbl = {
    5, { 0, 70, 90, 92,  95, 100 }

static float calc_percent(struct percent_tbl *tbl,
                          int pass, int curr, int max)
    float percent;

    if (pass <= 0)
        return 0.0;
    if (pass > tbl->max_pass || max == 0)
        return 100.0;
    percent = ((float) curr) / ((float) max);
    return ((percent * (tbl->table[pass]-tbl->table[pass-1]))
        + tbl->table[pass-1]);

We can compute the percentage by substituting the numbers from the last line of strace output above.

write(4, "2 2 158345 /dev/mapper/sdb1_crypt\n", 34) = 34

We're neither at 0 nor a 100%, so the return value is 2/158345*(90-70), which is close to 0, plus 70; which leaves us at a smidgen over 70% at the start of pass 2, approaching 90% as curr approaches max (158345). Depending on how much faith you have in the "pass x takes roughly y%" e2fsck_tbl defined above, this may or may not be comforting.

That left me with one more question: why was "max" changing between passes? The answer is that it represents different values during the different passes, as seen below (results edited slightly for clarity).

255:  (ctx->progress)(ctx, 0, 0, 0);

2108: (ctx->progress)(ctx, 1, group+1,

1008: (ctx->progress)(ctx, 2, cd->count++, cd->max)

106:  (ctx->progress)(ctx, 3, count++, maxdirs)

191:  (ctx->progress)(ctx, 4, group, maxgroup)

496:  (ctx->progress)(ctx, 5, group,

If only I had thought to look at the fsck manpage at the time, I might have noticed what the -C4 option was meant to do:

-C [fd]
    Display completion/progress bars for those
    filesystem checkers (currently only for ext[234])
    which support them. …

Given enough time—and of time, there was no shortage—I may even have made the mental leap to checking if systemctl status might show me the progress of fsck in some easily comprehensible form.

I wonder if it will snow again this year.

Update 2020-03-16: Mid-March brought a surprise spell of battery-draining weather (heavy rain, hail, snow a bit further North, and three overcast days without mains power) and another opportunity to find out what systemd-fsck does with fsck progress information.

I had naïvely hoped that "systemctl status" would show this information, but it did not. Nor did the systemd-fsck(8) man page say anything about it. So I had a quick glance at the systemd-fsck source code and found this in the process_progress function:

p = percent(pass, cur, max);
fprintf(console, "\r%s: fsck %3.1f%% complete...\r%n", …);

systemd-fsck calls the process_progress function to read progress output from fsck and write a summary to the console if /run/systemd/show-status exists. (The percent function is a copy of e2fsck's calc_percent above.)

arg_show_progress =
    access("/run/systemd/show-status", F_OK) >= 0;

console = fopen("/dev/console", "we");
if (console &&
    arg_show_progress &&
    pipe(progress_pipe) < 0)
        return log_error_errno(errno, "pipe(): %m");

if (console)
    (void) process_progress(progress_pipe[0], console);

Searching for show-status led me to the boot parameter and the equivalent ShowStatus setting in systemd.conf:

Takes a boolean argument or the constant auto. If yes, the systemd manager (PID 1) shows terse service status updates on the console during bootup …

Setting this parameter controls the presence of this trigger file:

void manager_set_show_status(Manager *m, ShowStatus mode, …)
    bool enabled = IN_SET(mode, SHOW_STATUS_YES, …);

    if (enabled)
        (void) touch("/run/systemd/show-status");
        (void) unlink("/run/systemd/show-status");

Nice to know, but not helpful for an already-running fsck: systemd-fsck checks for the file only at startup to set up the progress pipeline and call process_progress.

Still, it's interesting to look at an excerpt from the code to construct the -Cn option to tell fsck where to write progress information.

r = safe_fork("(fsck)", …);
if (r == 0) {
    int progress_socket = -1;

    /* Child */

     * Try to connect to a progress management daemon,
     * if there is one
    progress_socket = fsck_progress_socket();
    if (progress_socket >= 0) {
             * If this worked we close the progress pipe
             * early, and just use the socket
            progress_pipe[1] = safe_close(progress_pipe[1]);
            xsprintf(dash_c, "-C%i", progress_socket);
    } else if (progress_pipe[1] >= 0) {
             * Otherwise if we have the progress pipe to our
             * own local handle, we use it
            xsprintf(dash_c, "-C%i", progress_pipe[1]);

A “progress management daemon”! That sounds promising. The fsck_progress_socket function tries to connect to an AF_UNIX socket named /run/systemd/fsck.progress, but searching for fsck.progress in the systemd git history led me to this commit from 2015:

fsck: remove fsckd again, but keep the door open for external replacement

This introduces /run/systemd/fsck.progress as a simply AF_UNIX/SOCK_STREAM socket. If it exists and is connectable we'll connect fsck's -c switch with it. If external programs want to get progress data they should hence listen on this socket and will get all they need via that socket. To get information about the connecting fsck client they should use SO_PEERCRED.

Unless /run/systemd/fsck.progress is around and connectable this change reverts back to v219 behaviour where we'd forward fsck output to /dev/console on our own.

The linked post explains that systemd-fsckd (apparently an Ubuntu innovation) was so broken that it had no place even in systemd. It was replaced with the fsck.progress socket so that it could continue to exist and work the same as before, just out of sight of systemd itself.

But systemd-fsckd is still running on my systemd, so what exactly does it do?

systemd-fsckd.service is a service responsible for receiving file system check progress, and communicating some consolidated data to console and plymouth (if running). It also handles possible check cancellations.

I didn't have any easy way to go back and read the console logs from this machine. So what's Plymouth, then?

Plymouth is an application that runs very early in the boot process (even before the root filesystem is mounted!) that provides a graphical boot animation while the boot process happens in the background.

This machine doesn't have a monitor connected, nor Plymouth installed, so I'm missing out on the graphical animations, not to mention that the boot process had already finished long ago.

In short, systemd-fsckd and systemd-fsck didn't do anything especially convenient for me with the progress information that it requests from fsck, and I didn't miss out on anything by reaching for strace first.

I like volumeicon

My headphone cable has become flaky, so I've been using a small external speaker for a while. I now need to change my volume settings so often that running alsamixer in a terminal each time was beginning to annoy me. So I looked for a systray-based mixer (I use xmobar with xmonad), and I found volumeicon (packaged in Debian as volumeicon-alsa).

I really like volumeicon. The default behaviour is perfectly sensible. Click the icon to mute, click again to unmute. Hover to see the current volume level, use the scroll wheel to change it. Middle click to "Open Mixer", which runs alsamixer in a terminal. Right click to set "Preferences".

What's more, the preferences are also remarkably sensible. You can pick the ALSA device and channel to control, and there's a straightforward menu to change the appearance and behaviour of the icon. volumeicon preferences

It's been quite a while since I encountered a new program that did exactly what I wanted with so little fuss.

Resolving a conflict between manpages-dev and libbsd-dev

I use xmonad, which needs ghc.

On my Debian 9/stretch machine, the manpages-dev package is declared to break libbsd-dev, which ghc depends on. So in practice, one must choose between xmonad and manpages-dev, which has annoyed me for a long time.

Today I tracked down a related Debian bug report and happened to notice that my libbsd-dev 0.8.3 was just one minor revision too old to avoid the conflict declared by manpages-dev 4.10-2:

$ apt-cache show manpages-dev|grep Breaks
Breaks: libbsd-dev (<< 0.8.4-1), manpages (<< 4.13-3)

I also noticed that libbsd-dev 0.9.1-1 packages were available in unstable/sid, and followed the instructions to create a backport of the package to stretch.

$ dget -x
$ cd libbsd-0.9.1
$ sudo mk-build-deps --install --remove
$ dch --local ~bpo9+ --distribution stretch-backports "Rebuild for stretch-backports."
$ dpkg-buildpackage -us -uc

I installed the updated libbsd0 and libbsd-dev packages, and was then able to install manpages-dev for the first time in… several months, at least.

I have manpages again!

Relaying from Postfix to gmail

Here's how I configured Postfix to relay mail from through using the credentials set up for on Google Apps.

There are three parts to this: making Postfix relay mail based on the sender address, teaching it to authenticate to gmail, and configuring gmail to accept the relayed mail. (Postfix was already configured to send outgoing mail directly.)

Sender-dependent relay

I created /etc/postfix/relay_hosts with the following contents:

Then I ran «postmap /etc/postfix/relay_hosts» and set sender_dependent_relayhost_maps in /etc/postfix/

sender_dependent_relayhost_maps =

SMTP SASL authentication

I created /etc/postfix/sasl_passwords (mode 0600) with the following contents:

Then I ran «postmap /etc/postfix/sasl_passwords» and added the following to /etc/postfix/

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps =
smtp_sasl_security_options = noanonymous

That enables SMTP AUTH in the Postfix SMTP client and tells Postfix where to look up the username and password for a domain.

Gmail will accept SMTP AUTH only in a TLS session, so TLS client support must be configured in Postfix (which means setting smtp_tls_security_level to "may"). But even once that's done, gmail advertises only the following authentication mechanisms:


I didn't want to worry about OAUTH, so I was left with PLAIN was the only reasonable choice. Postfix will not use plaintext authentication mechanisms by default, so I also had to remove "noplaintext" from the default value for smtp_sasl_security_options.

As an additional precaution, I also set smtp_tls_policy_maps to change the default TLS policy from "may" to "encrypt" for

Gmail configuration

When I tried to send mail through the relay, Postfix wasn't able to authenticate:

SASL authentication failure: No worthy mechs found

SASL authentication failed; cannot authenticate to server[]: no mechanism available

Google considers password authentication to be “less secure”, and you have to explicitly enable it on the less secure apps settings page. There are some other alternatives, but I was happy to take the path of least resistance here.

I did that and tried again, only for mail to bounce with this error:

Invalid credentials for relay []. The IP address you've registered in your G Suite SMTP Relay service doesn't match domain of the account this email is being sent from. If you are trying to relay mail from a domain that isn't registered under your G Suite account or has empty envelope-from, you must configure your mail server either to use SMTP AUTH to identify the sending domain or to present one of your domain names in the HELO or EHLO command. For more information, please visit

This message is misleading, as I found out by using openssl's s_client to establish a TLS session and then authenticating by hand. SMTP AUTH succeeded, but MAIL FROM was subsequently rejected. I followed the link in the message, which led me to the SMTP relay service support page.

The Google Apps admin console doesn't use sensible URLs, but I followed the breadcrumb trail to an “Advanced settings” page where I was able to edit the SMTP relay service settings to set “Allowed senders” to “Only addresses in my domains”, as well as to “Require SMTP authentication” and “Require TLS encryption”. Remember to “Save” the changes.

The error I got was because the “Only accept mail from specified IP addresses” option was checked for this particular domain. I could have added the IP address of my server to the list, but SMTP authentication was what was I wanted to use anyway.

Raspberry Pi murder

I wanted to restore a clean Raspbian image on the Raspberry Pi that was running in the battery room. I could have gone and got it, but I would have had to climb a ladder.

So I thought “Why not just remount the filesystems 'ro' and dd the image to /dev/mmcblk0?”

I remounted / and /boot (after "systemctl isolate", which stopped sshd but left my ssh session running) read-only, and dd'ed the image from an NFS mount onto /dev/mmcblk0. The dd worked fine. I used /proc/sysrq-trigger afterwards to sync and reboot (I couldn't run any binaries after the dd, which wasn't much of a surprise). The machine fell off the network as expected…

…and never came back up. So I climbed up the ladder and brought the Pi down to my desk and plugged it into my monitor. It did start to boot, and got a fair distance before the kernel panicked. I didn't bother to try to figure out the problem, just extracted and rewrote the SD card; and all was well thereafter.

But I still think my plan ought to have worked.

Reading about wireguard

I have more than a passing interest in VPN software, and have looked at and used many different implementations over the years. I haven't found much to cheer about, which led me to write tappet for my personal use.

I've been reading about Wireguard for the past few weeks, and I really like it so far. It follows through on many of the same goals that I had with tappet, and goes much further in areas important to more widespread adoption. The author, Jason Donenfeld, articulates the project's design goals in this presentation.

Keeping the code small and easy to review was a primary consideration for me (tappet is under a thousand lines of code, not including NaCl). By this measure, Wireguard does an admirable job of staying small at around 15,000 lines including crypto code and tests.

When I wrote tappet, the Noise Protocol did not exist in a usable (or recommended) form. Wireguard's adoption of this framework brings a host of desirable properties that tappet lacks, notably including perfect forward secrecy.

One of my major frustrations with OpenVPN is the extraordinary time it takes to establish a TLS connection on a high-latency link. Very often, when tethered via GPRS, it will retry forever and never succeed. Tappet goes to the other extreme—it requires zero setup for encrypted links (at the expense of perfect forward secrecy). Wireguard restricts its handshake to a single round-trip, which is an entirely acceptable compromise in practice.

Wireguard runs in the kernel, thereby avoiding the need to copy packets in and out of userspace. I didn't care nearly as much about performance. Tappet is fast enough in userspace that it keps up with the fastest link I've tried it on (42.2Mbps DCHSPA+), and I didn't need anything more.

Wireguard accepts multiple peers per interface, while tappet is limited to setting up point-to-point encrypted links. The former is obviously more practical in realistic deployments. (On the other hand, Wireguard is a Layer-3 VPN, while tappet operates at L2 and forwards Ethernet frames instead of IP packets. How much that matters depends on the circumstances.)

I look forward to a time when I can use Wireguard in production.

Debian 8 on the Intel NUC5PPYH

Hassath's birthday present this year was an Intel NUC5PPYH (with 8GB of Kingston DDR3L RAM and a 250GB Samsung SSD 750 Evo) to stand in at home for her ageing Thinkpad X131E.

It took some time for the machine to reach our remote mountain abode, but we have it working nicely after spending a few hours wrestling with it. Here's a quick summary of our experience (InstallingDebianOn/Intel/NUC5PPYH wasn't really useful).

Display problems

Hassath loves her old Samsung SyncMaster 172s monitor (1024x768, VGA) and resists the idea of a new wide-format monitor. Getting the NUC to work properly with this display took the most time (but none of it was the display's fault).

We connected the monitor to the NUC's VGA port and were greeted with a "Video mode not supported" error on the monitor. The debian installer's text-mode display worked fine after boot, but we couldn't see any of the UEFI setup menus. Fortunately, we were able to sidestep the problem by using an HDMI→VGA adapter that we had ordered “just in case”. Using the HDMI output resolved the display problems with the UEFI menus.

After we installed Debian (8.1 from a USB stick created from the DVD image), X wouldn't start. The intel driver didn't work, and Xorg fell back to the VESA driver, and died while trying to set the video mode. (Also, virtual terminals didn't work at all until we added an xorg.conf snippet to force the resolution to 1024x768.) It didn't work even with the DVI-D input (via another “just in case” HDMI→DVI-D cable) on my monitor.

We stumbled around for a while, but we eventually got it working. The key was to apt-get dist-upgrade against jessie-backports to install a new kernel and drivers (e.g., libdrm-intel1). We also updated the BIOS from revision 0054 to revision 0058, but I am not sure that this was necessary, or even helpful. Xorg works with the new kernel and Intel driver. We didn't bother to check if the VESA driver would also work if we forced its use.

(Aside: we had no UEFI boot-related problems at all. We didn't even need to try the legacy boot option, either for the installation from the USB stick or to boot the installed system.)

Everything else worked

The Ethernet controller is a Realtek RTL8168h, which works out of the box with the r8169 driver. Installing the firmware-realtek package got rid of an “unable to load firmware patch” message, but the adapter worked fine without it.

The wireless controller is an Intel dual band wireless-AC 3165, which required the new kernel from backports (4.8, though 4.2+ should have worked from what we read) and the firmware-iwlwifi package to be installed. It worked fine thereafter.

The audio controller is an Intel "Braswell" 2284, which works out of the box with the snd_hda_intel driver. Audio output goes simultaneously to the headphone connector on the front panel and the glowing red S/PDIF plus headphone connector on the back. We did not try S/PDIF audio (no cable, no devices) or HDMI audio (no audio port on the HDMI→VGA adapter) or recording (no mic—or at least, no mic on my desk).

The Intel Bluetooth 4.0 controller (8087:0a2a) works out of the box with the btusb driver. We were able to pair with an Android phone and a Bluetooth speaker. We were not able to play audio to the speaker, but that is probably not a problem with the NUC, because we didn't manage to get it working with any of our other machines either.

We didn't try the SDXC card slot or the infrared sensor.

Update (2017-01-18): The SDXC card slot works fine. I used it to write a Raspbian image.

Debian 8 on the Lenovo Ideapad S206

I got my dad a Lenovo Yoga 300 a few days ago, and installed Debian 8 on his old Ideapad S206 before giving it to Ammu, who has been using my old (and increasingly broken) Thinkpad X120e for months now.

The Linux Laptop Wiki has a page about the Ideapad S206; the quick summary is that everything works with only a little tweaking.

Wifi worked out of the box with brcmsmac. Bluetooth worked fine after installing the firmware-atheros package. Sound input and output worked fine out of the box. The hotkeys to change the volume (and brightness, etc.) also worked. I didn't try the card reader or the webcam.

I didn't bother with the fglrx video drivers, but I had to install firmware-linux-nonfree to enable KMS (kernel mode setting) to get suspend/resume to work properly. (Hibernation worked fine anyway.)

This is a sleek and light machine, quite a step up from the earlier Ideapad models I've used. The screen is a bit shiny, but stops short of being annoying. The keypad is unfortunately very jittery—unless you are very deliberate about tapping-to-click, you'll most likely just move the pointer a bit rather than clicking. This was my father's biggest problem with the machine (and it wasn't just a matter of acceleration settings).

But it's working nicely otherwise, and I hope Ammu gets some good use out of it.

Vodafone India snoops on e-mail

An article about Vodafone injecting javascript into web pages reminded me of a problem I investigated last year when Hassath couldn't send mail when connected through her phone's mobile hotspot.

My first response to any network problems is to run tcpdump, and I saw the following EHLO response from my own SMTP server.
250-SIZE 307200000
250 DSN

Vodafone is transparently proxying outgoing SMTP traffic and replacing STARTTLS in the EHLO response with XXXXXXXA, so that the client doesn't try to negotiate TLS. If you issue STARTTLS anyway—which no normal SMTP client would, but openssl's s_client can do—the TLS negotiation fails. So it's not just a downgrade attack, it's actively sabotaging TLS connections too.

This was the case in mid-2014, and it's still the case at the time of writing. I wonder how many terabytes of email logs they have collected in the meantime, how they are stored, and who is reading them.

While I was tethered to my phone, I did a bit more testing. Vodafone India doesn't seem to mess with HTTPS connections, and IMAP connections are not downgraded either (i.e., the server's STARTTLS advertisement is not modified, and the TLS negotiation succeeds). Nor did it inject any Javascript into the web pages I tried (yet).

Update (2017-10-01): I happened to read a 2014 post by Steve Atkins titled STARTTLS and misplaced outrage, which says this is a "very, very, very well known" problem with the configuration of a PIX firewall feature named "MailGuard". He writes:

The most likely scenario, by far, is that the mailserver operator is behind a PIX, and has it configured like that. As port forwarding is specific to the interface that traffic comes in on, it’s quite possible that it’s only misconfigured for traffic coming over some networks. Drastically less likely is that there was a PIX installed – backwards – on the cellular providers network. Somewhat less likely still is that they’re simply lying about what they’re seeing. But those are the only three options.

In this case, I'm the operator of the mail server in question, and I know there is no PIX involved anywhere, and I know I'm not simply lying either. I also know that the problem happens only on Vodafone's network, so—unlikely as it may be—maybe there's a PIX installed backwards on the Vodafone network?

Understanding sudoers(5) syntax

This straightforward guide to configuring sudo is for anyone who didn't expect to see “Don't despair” and a “Quick guide to EBNF” in the sudoers(5) manpage.

Sudo (su "do") allows a system administrator to delegate authority to give certain users (or groups of users) the ability to run some (or all) commands as root or another user while providing an audit trail of the commands and their arguments.

This guide is intended to supplement the manpage. The various environment, security, and logging options are not covered; the explanations in the manpage are easy to follow.

The first 90%

It's possible that the only sudo explanation you will ever need is:


This means “any user in the adm group on any host may run any command as any user without a password”. The first ALL refers to hosts, the second to target users, and the last to allowed commands. A password will be required if you leave out the "NOPASSWD:".

Read more…