SSH into server with broken or missing shell

Josh Sherman
3 min read
Servers / Serverless Debian / Ubuntu

I did the unthinkable the other day. I attempted to upgrade my Debian home server (sherver) from Debian 10 to 11. In doing so, I fat fingered something in /etc/apt/sources.list. I then proceeded to run the usual update and upgrade commands, not paying any attention to any output. Then I rebooted the box.

Upon rebooting the box, a short stack Intel NUC, I was greeting with an error about my private key not being valid or accepted. Adding in -vvv showed more of the same.

I was locked out.

At this point, I jumped to the conclusion that I had borked something during the upgrade and that caused the lock out. I figured the worst case scenario would be that the system would need to be rebooted with a rescue disk and sorted out that say.

Since it did appear that everything was running correctly on it, I waited a few days before digging out a keyboard and hooking the machine up to a monitor.

Fast forward a few days later, I plugged in an HDMI cable and keyboard, and attempted to log in.

Logging in with my user account didn’t work. A message flashed on the screen and it a prompt was presented again. I was able to log into the root account, fortunately. Said root account couldn’t be utilized over SSH as I keep PermitRootLogin set to no on my entire server fleet.

Cool, so I’m logged into the machine now. When attempting to switch to my normal user account, I was able to see the error in all it’s glory. Something about zsh being missing.

Fantastic, let’s just reinstall that and we should be good to go!

Not quite, turns out during the upgrade process, there was a mismatch between a required lib and the zsh version which resulted in zsh being removed. After a bit of troubleshooting, I tracked down where I had an error, got that sorted and was able to complete the upgrade and get zsh back.

Now I know what you’re thinking, “hey Josh, when are you going to tell me how to SSH into a server with a broken or missing shell?”

Simply put, you cannot.

I did some digging after the fact, and there’s no way to pass in a different shell to run in this scenario. If there were, things like /usr/sbin/nologin would be rendered useless, wouldn’t they?

So that’s the hard truth of it, in a scenario where your shell is missing or broken or set to something like nologin that’s it, you can’t log in.

Fortunately for me, I had physical access to the server in question, but that’s not always the case with remote servers and cloud computing.

In those scenarios, you need to make sure you’re being more mindful of what you’re doing. Keeping an active root shell logged in while you’re running upgrades is a good start.

With an active session going, you can finish your upgrade and then attempt to log in with your normal user. If things are borked, no big deal, just switch back to your root session and attempt to sort things out!

Join the Conversation

Good stuff? Want more?

Weekly emails about technology, development, and sometimes sauerkraut.

100% Fresh, Grade A Content, Never Spam.

About Josh

Husband. Father. Pug dad. Born again Linux user. Founder of Holiday API, Engineering Manager and Emoji Specialist at Mailshake, and author of the best damn Lorem Ipsum Library for PHP.

Currently Reading

Antifragile: Things That Gain from Disorder
The House Across the Lake: A Novel
Batman: The Dark Knight Returns 30th Anniversary Edition

Previous Reads

Buy Me a Coffee Become a Sponsor

Related Articles