I run into my fair share of “gotchas” as an Arch Linux user. I also stand firmly that I’ve always ran into my fair share of “gotchas” running Linux in general.
In fact, I’ve even run into my fair share of “gotchas” on both macOS and Windows systems, so maybe the problem is just me.
Joking aside, I’m a power user. I run into issues. I fix those issues. Then I document those issues, first and foremost, for myself, so I know how to fix them in the future. Secondarily, I document things because I found they weren’t properly documented and to help other weary travelers experiencing the same thing.
My latest “gotcha” came in the form of running a well deserved Arch Linux upgrade recently. With well over 300 packages to be installed, what could possibly go wrong?
I went through my usual dance of saving all of my files and terminating
sessions and closing out every application sans my beloved terminal. I ran
with the usual flags and made the mistake of walking away for a few minutes to
do something else.
I came back a few minutes later to my system shutting down, I assumed to reboot. Generally speaking, this isn’t normal. While I do reboot after I run an upgrade, Arch Linux doesn’t do this automatically that I ever remember.
“Maybe I remembered wrong”, I thought.
Upon restarting, I was greeted with the unexpected (actual error message may vary as I forgot to snap a photo for posterity):
Missing vmlinuz from /boot
I was getting ready to sit down to finish the main story of Super Mario Odyssey to boot.
Side note, if you own a Switch and don’t own Super Mario Odyssey, AND want to help support my site because I just helped you out of a jam, you definitely should pick up a copy.
Sorry about that, gotta pay the bills and all ;)
So as mentioned, this was completely unexpected but not too shocking considering I had just ran an upgrade and witnessed some strange reboot behavior.
I never did get to the bottom of what actually went wrong. I’ve even updated my System76 Galago Pro recently as well, with well over 1,000 new packages, and didn’t run into the same issue.
Because this was my daily driver and it was Sunday afternoon, I had to get this resolved resolved quickly as to not impact my upcoming work day.
The error message I received was indicative of the Linux kernel going missing, so my first thought was to reboot into the LTS Linux Kernel. I always keep the LTS release of the Linux Kernel installed just in case I experience some issue with the mainline Kernel. At least once in my life the issue was the mainline Kernel’s fault and running LTS for a bit to wait for a fix was the solution.
So I rebooted and selected the LTS Linux Kernel with extremely high hopes that I’d be back to a working system.
No dice, same deal with the LTS version. Whatever went wrong borked both the
linux-lts packages I had installed.
Not a big deal, just like keeping multiple versions of the Linux Kernel installed, I also keep a thumb drive handy with Arch Linux on it in case I need to boot to a live disk to resolve my issue.
So that’s what I did. I booted up my trusty thumb drive, went through the steps
to get wireless networking up and running, mounted my partitions,
Arch way, and reinstalled the now missing packages.
It went something like this:
- Boot to my trusty Arch Linux USB thumb drive.
- Connected to my wireless network by running
wifi-menuand following the prompts.
- Mounted my root
/partition as such with
mount /dev/DEVICE /mntand immediately checked if I had lost any files (nothing obvious was gone).
- Mounted my boot
/bootpartition as well:
mount /dev/DEVICE /mnt/boot.
- Changed root directories to the mounted devices:
- Installed the missing packages:
pacman -S linux linux-lts.
- Rebooted to a working Arch Linux system!
Depending on your Linux distribution of choice, you could probably follow the same steps with a Live Disk for the distro and swap the Arch Linux specific commands for said distribution.
Sadly, this didn’t get me 100% back in business.
Upon rebooting, I was faced with messaging about
fsck running into some errors
and needing to be ran manually. I was able to do so from the prompt I was
While still not 100% sure of the cause, it does appear that it was perhaps related to either some underly issues with my SSD or some wires getting crossed with some bits being written to the same place and the system getting extremely confused about it.
Fortunately after running
fsck and letting it do what it does, I was in fact
able to boot to a working Linux desktop again.
Being a glutton for punishment, I went ahead ran another upgrade to make sure everything was up to date, as that was what I originally set out to do). A few reboots later, I was confident that the issues had been squashed.
Off to play Super Mario Odyssey I went.
Fast forward to the next day when I hopped on my daily stand up video call for work. Everything was aces, except my camera didn’t show up, so I had to actually “phone it in” without any video for the call.
Getting this crap resolved was my “blocker” for the day, obviously.
After jumping off the call, I immediately opened up
cheese to see which
devices were shown. Since I dock to my monitor and have an external camera, I
should have 2 separate devices. My thought was perhaps my external camera was
missing, but my laptop’s camera was still showing up.
Nope, both gone. Time to reboot to the LTS Kernel to see if it’s a mainline Kernel issue. Booted into the LTS Kernel, everything was as it should be. Both camera devices were showing up and working as expected.
After rebooting back into the mainline Kernel, I looked around the system logs
lsmod and all of that, none of which indicated that any of my camera
devices were being picked up at all.
My last ditch effort before deciding that I should just run with the LTS Kernel
for a bit was to forcefully reinstall the
linux package again, in case that
installing it from the live disk was part of the problem.
Sure enough, after reinstalling the
linux package and rebooting again,
everything was back to normal. Even though this sounds like a mountain of a
problem, I spent MAYBE an hour all in on it and I got this article out of it to
Oh, for anybody that’s been following the subplot, I did in fact beat the main story of Super Mario Odyssey!