I run into my fair share of “gotchas” as an Arch Linux user. I also stand firmly
that I’ve always ran into my fair share of “gotchas” running Linux in general.
In fact, I’ve even run into my fair share of “gotchas” on both macOS and Windows
systems, so maybe the problem is just me.
Joking aside, I’m a power user. I run into issues. I fix those issues. Then I
document those issues, first and foremost, for myself, so I know how to fix them
in the future. Secondarily, I document things because I found they weren’t
properly documented and to help other weary travelers experiencing the same
thing.
My latest “gotcha” came in the form of running a well deserved Arch Linux
upgrade recently. With well over 300 packages to be installed, what could
possibly go wrong?
I went through my usual dance of saving all of my files and terminating screen
sessions and closing out every application sans my beloved terminal. I ran yay
with the usual flags and made the mistake of walking away for a few minutes to
do something else.
I came back a few minutes later to my system shutting down, I assumed to reboot.
Generally speaking, this isn’t normal. While I do reboot after I run an upgrade,
Arch Linux doesn’t do this automatically that I ever remember.
“Maybe I remembered wrong”, I thought.
Upon restarting, I was greeted with the unexpected (actual error message may
vary as I forgot to snap a photo for posterity):
Missing vmlinuz from /boot
I was getting ready to sit down to finish the main story of Super Mario Odyssey
to boot.
Side note, if you own a Switch and don’t own Super Mario Odyssey, AND want to
help support my site because I just helped you out of a jam, you definitely
should pick up a copy.
Sorry about that, gotta pay the bills and all 😉
So as mentioned, this was completely unexpected but not too shocking considering
I had just ran an upgrade and witnessed some strange reboot behavior.
I never did get to the bottom of what actually went wrong. I’ve even updated my
System76 Galago Pro recently as well, with well over 1,000 new packages, and
didn’t run into the same issue.
Because this was my daily driver and it was Sunday afternoon, I had to get this
resolved resolved quickly as to not impact my upcoming work day.
The error message I received was indicative of the Linux kernel going missing,
so my first thought was to reboot into the LTS Linux Kernel. I always
keep the LTS release of the Linux Kernel installed just in case I experience
some issue with the mainline Kernel. At least once in my life the issue was the
mainline Kernel’s fault and running LTS for a bit to wait for a fix was the
solution.
So I rebooted and selected the LTS Linux Kernel with extremely high hopes that
I’d be back to a working system.
No dice, same deal with the LTS version. Whatever went wrong borked both the
linux and linux-lts packages I had installed.
Not a big deal, just like keeping multiple versions of the Linux Kernel
installed, I also keep a thumb drive handy with Arch Linux on it in case I need
to boot to a live disk to resolve my issue.
So that’s what I did. I booted up my trusty thumb drive, went through the steps
to get wireless networking up and running, mounted my partitions, chrooted the
Arch way, and reinstalled the now missing packages.
It went something like this:
- Boot to my trusty Arch Linux USB thumb drive.
- Connected to my wireless network by running
wifi-menuand following the
prompts. - Mounted my root
/partition as such withmount /dev/DEVICE /mntand
immediately checked if I had lost any files (nothing obvious was gone). - Mounted my boot
/bootpartition as well:mount /dev/DEVICE /mnt/boot. - Changed root directories to the mounted devices:
arch-chroot /mnt. - Installed the missing packages:
pacman -S linux linux-lts. - Rebooted to a working Arch Linux system!
Depending on your Linux distribution of choice, you could probably follow the
same steps with a Live Disk for the distro and swap the Arch Linux specific
commands for said distribution.
Sadly, this didn’t get me 100% back in business.
Upon rebooting, I was faced with messaging about fsck running into some errors
and needing to be ran manually. I was able to do so from the prompt I was
presented with.
While still not 100% sure of the cause, it does appear that it was perhaps
related to either some underly issues with my SSD or some wires getting crossed
with some bits being written to the same place and the system getting extremely
confused about it.
Fortunately after running fsck and letting it do what it does, I was in fact
able to boot to a working Linux desktop again.
Being a glutton for punishment, I went ahead ran another upgrade to make sure
everything was up to date, as that was what I originally set out to do). A few
reboots later, I was confident that the issues had been squashed.
Off to play Super Mario Odyssey I went.
Fast forward to the next day when I hopped on my daily stand up video call for
work. Everything was aces, except my camera didn’t show up, so I had to actually
“phone it in” without any video for the call.
Getting this crap resolved was my “blocker” for the day, obviously.
After jumping off the call, I immediately opened up cheese to see which
devices were shown. Since I dock to my monitor and have an external camera, I
should have 2 separate devices. My thought was perhaps my external camera was
missing, but my laptop’s camera was still showing up.
Nope, both gone. Time to reboot to the LTS Kernel to see if it’s a mainline
Kernel issue. Booted into the LTS Kernel, everything was as it should be. Both
camera devices were showing up and working as expected.
After rebooting back into the mainline Kernel, I looked around the system logs
and lsmod and all of that, none of which indicated that any of my camera
devices were being picked up at all.
My last ditch effort before deciding that I should just run with the LTS Kernel
for a bit was to forcefully reinstall the linux package again, in case that
installing it from the live disk was part of the problem.
Sure enough, after reinstalling the linux package and rebooting again,
everything was back to normal. Even though this sounds like a mountain of a
problem, I spent MAYBE an hour all in on it and I got this article out of it to
boot 🙂
Oh, for anybody that’s been following the subplot, I did in fact beat the main
story of Super Mario Odyssey!
