This story involves Debian 12, running on a VPS at Hostwinds.com.
I was getting ready to spin up a new WordPress site and was doing the traditional aptitude update/upgrade process to make sure everything was up to date.
The update process came off fine. The upgrade process, not so much.
A number of packages got successfully upgraded, but when it came to util-linux the system just sat there. And sat there. And sat there. It wouldn’t even respond to ctrl-C interrupts…probably for reasons we’ll get to in a bit. I ended up killing the aptitude process and trying again. Same result.
Worse yet, none of the other pending upgrades could be processed until the util-linux upgrade successfully completed.
Digging around the aptitude log files I discovered util-linux was in a “half configured” state. This is a (documented) situation where necessary configuration has started but not completed successfully.
With input from web searches, Hostwinds tech support and some folks on r/debian, I tried a ton of different approaches to get past the stall. I tried doing the upgrade via both aptitude and dpkg. I tried doing aptitude clean and aptitude autoclean. I downloaded the relevant util-linux deb file and tried to do the upgrade locally. I basically spent several hours trying stuff, all to no avail.
I then reached out to a Debian expert I know – Kevin – and turned him loose on the problem.
After a ton of research, Kevin noticed that every stalled upgrade attempt manifested a particular pattern of processes. When looking at the process tree hierarchically, he saw each upgrade attempt spawned a series of subsidiary processes…the last one of which always involved the systemd-tty-ask-password-agent. Which just sat there:
Neither Kevin nor I are experts on what systemd-tty-ask-password-agent does, but its documentation suggests it handles password requests made by the system, prompting the user to enter a valid password if necessary.
systemd-tty-ask-password-agent works closely with the systemd-ask-password-console service. In fact, the console service will launch the agent as needed.
But on my system, for whatever reason, the systemd-ask-password-console service was dead/not running.
Kevin also noticed the systemd-ask-password-console service (which spins up systemd-tty-ask-password-agent) was dead/not running. We tried starting the systemd-ask-password-console service manually. Unfortunately, that didn’t solve the upgrade stall problem. In fact, some further research Kevin did later implies its death might be a red herring (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=774153 and “https://github.com/systemd/systemd/issues/9507#issuecomment-404078079“).
But we ran into an unexpected bit of luck.
Where I had killed the stalled upgrades by killing aptitude, Kevin started by killing the systemd-tty-ask-password-agent child process and then killing each of the parent processes in turn until he killed off aptitude.
Apparently, killing the processes that way enabled the package manager, the next time we tried to do the upgrade, to decide everything was hunky-dory with util-linux, and proceed with the rest of the upgrades.
Kevin’s theory is that the stall was taking place at the very last step of package configuration, which may have involved re-launching util-linux services (if any such are defined) but apparently does require some kind of password authentication (hence the spinning up of systemd-tty-ask-password-agent).
Killing the upgrade process from the top down, like I did, the configuration step was never completed and dpkg would not have had the opportunity to mark util-linux as configured (since dpkg would’ve been killed). As a result, the next time an installation was attempted, aptitude would detect the half-configured util-linux package and insist on trying to finish configuring it…leading to the same stall.
Killing the upgrade process from the bottom up, like Kevin did, resulted in the overall upgrade process being deemed successfully completed. Why? We didn’t dig into the source code so we don’t really know. Perhaps the calling process ignores the non-zero exit code caused by killing the systemd-tty-ask-password-agent. Whatever the reason, a bottom up kill set the stage for the next upgrade attempt to succeed.
A nearly identical situation was documented and diagnosed as a bug in systemd several years ago. Why it should still be present in Debian 12 today I have no idea.
Kevin did find an offhand comment online speculating that perhaps the version of systemd I was running was out-of-date. That might have happened if, somehow, util-linux (or some other package) was never properly configured, which might have caused my initramfs to be out of date. If systemd started from there (which Kevin thinks it is), my systemd might have been way out of date.
I don’t know enough about Debian or systemd to have an opinion. What I know, for now, is that upgrades complete properly (I did a new one the morning after Kevin fixed things). I’ll leave it up to those more knowledgeable to determine if there’s some bug that needs to be fixed.