Fixing an Arch Linux system that is booting into emergency mode


3 min read
linux

File this under “crap I want to document in case it happens again later”.

So while in the middle of working today, my MacBook Pro running Arch Linux (recently clean installed) decided to lock up on me.

Pretty sure it was Firefox Developer Edition that did it, but that’s for another post.

So the system locked and I rebooted after I trying to switch to another terminal a few times.

Expecting to be back in action in a few seconds, I was greeted with some shit about booting into emergency mode and that it didn’t mount /boot.

Oh shit, did my drive crash? Was Firefox to blame??? O_O

To make matters worse, I couldn’t log in as root since I keep the account disabled.

Ok, NBD… just boot USB stick with Arch on it and get to fixing!

Except the aforementioned USB drive was at home.

Fortunately, I had another USB drive on me that I quickly cleaned off and put Arch on.

Huge takeaway here is to always have a Linux live CD or USB stick on you just in case.

On top of doing all of this, I was frantically searching for what the issue could have been and how to resolve it.

Obviously, it was related to upgrading packages on Arch (which I currently do pretty regularly) but I was quite overwhelmed with the amount of conflicting information out there.

Now armed with a bootable Linux system in the palm of my hand (thumb?), I was able to get to troubleshooting.

First thing I did was re-enable the root account and tried to get to the recovery console.

No dice!

Two keyboards and neither of them were responsive enough for me to be able to log into the root account.

After that, I decided to finally decided to follow the on screen directions and check what was in journalctl as it was supposed to give me some insight.

In doing that, I found that some kernel modules weren’t loading. Specifically, the stuff I had recently installed to get the FaceTime camera working in Linux.

Okay, easy enough, let me ax that module and go from there.

Still nothing :(

After doing some more digging, I came across some details that seemed to describe my situation.

Evidently if you run an upgrade that upgrades the Linux kernel and your boot partition isn’t mounted, shit will hit the fan.

But I mount /boot at boot time?

Not sure what may or may not have happened, but turned out that I definitely had upgraded the Linux kernel without the boot partition mounted.

The steps to resolve wasn’t hard, but I wasn’t able to find a single thing on the Arch forums that documented this set by step:

  1. Boot a Arch Linux live CD or USB drive
  2. Get connected to the Internet: wifi-menu
  3. Mount your root partition: mount /dev/sda# /mnt
  4. Mount your boot partition: mount /dev/sda# /mnt/boot
  5. Change your root directory: arch-chroot /mnt
  6. Reinstall the kernel: pacman -S linux
  7. Reboot and if successful
  8. Comment below about how awesome this post is (or you know… toss some BTC my way ;)

YMMV, especially if your issue isn’t the same as the one I had encountered. Like I said, this was mostly so I have a handy reference in case it happens to me again ;)