I did the unthinkable the other day. I attempted to upgrade my Debian home
server (sherver) from Debian 10 to 11. In doing so, I fat fingered something
in /etc/apt/sources.list. I then proceeded to run the usual update and upgrade
commands, not paying any attention to any output. Then I rebooted the box.
Upon rebooting the box, a short stack Intel NUC, I was greeting with an
error about my private key not being valid or accepted. Adding in -vvv showed
more of the same.
I was locked out.
At this point, I jumped to the conclusion that I had borked something during the
upgrade and that caused the lock out. I figured the worst case scenario would be
that the system would need to be rebooted with a rescue disk and sorted out that
say.
Since it did appear that everything was running correctly on it, I waited a few
days before digging out a keyboard and hooking the machine up to a monitor.
Fast forward a few days later, I plugged in an HDMI cable and keyboard, and
attempted to log in.
Logging in with my user account didn’t work. A message flashed on the screen and
it a prompt was presented again. I was able to log into the root account,
fortunately. Said root account couldn’t be utilized over SSH as I keep
PermitRootLogin set to no on my entire server fleet.
Cool, so I’m logged into the machine now. When attempting to switch to my normal
user account, I was able to see the error in all it’s glory. Something about
zsh being missing.
Fantastic, let’s just reinstall that and we should be good to go!
Not quite, turns out during the upgrade process, there was a mismatch between a
required lib and the zsh version which resulted in zsh being removed. After
a bit of troubleshooting, I tracked down where I had an error, got that sorted
and was able to complete the upgrade and get zsh back.
Now I know what you’re thinking, “hey Josh, when are you going to tell me how to
SSH into a server with a broken or missing shell?”
Simply put, you cannot.
I did some digging after the fact, and there’s no way to pass in a different
shell to run in this scenario. If there were, things like /usr/sbin/nologin
would be rendered useless, wouldn’t they?
So that’s the hard truth of it, in a scenario where your shell is missing or
broken or set to something like nologin that’s it, you can’t log in.
Fortunately for me, I had physical access to the server in question, but that’s
not always the case with remote servers and cloud computing.
In those scenarios, you need to make sure you’re being more mindful of what
you’re doing. Keeping an active root shell logged in while you’re running
upgrades is a good start.
With an active session going, you can finish your upgrade and then attempt to
log in with your normal user. If things are borked, no big deal, just switch
back to your root session and attempt to sort things out!
