For 20 years, I've been a user of Linux (precisely: Arch Linux distribution) - for some time, it was the only system installed on my computer. Currently, I'm using it primarily for development-related work, and I regularly update my system using the brilliant pacman package manager available in Arch Linux. However, this devotion is sometimes hard, especially when the update goes wrong and you end up with a broken system.

This is a short story of a yesterday's failed system update that brought - among others - a new kernel version and nvidia-open-620.76.06 drivers for NVIDIA graphics card. They showed me two things:

  1. The new driver version is broken.
  2. I need a fallback, and I did not have one.

What Happened?

Morning. I ran the computer, I typed sudo pacman -Syu and I went to the kitchen to make coffee. Just an ordinary update. There was no "manual action needed" news on the website - nothing to worry about. I came back with the black drink, I restarted the machine, logged into KDE and... oh. The mouse cursor is lagging. All animations are lagging. Typing on the keyboard is lagging. Terminal is almost unusable.

I managed to find some information on the Internet. The main recommendation what to check was the video card driver - nvidia-open in my case. However, the reports were related to the older versions - I tried some workarounds, but obviously they did not work, and sometimes they made the issue even worse.

The challenge with debugging was that my desktop environment was unresponsive. I needed to log out, switch to the pure terminal, and look up all the information on the mobile phone.

Downgrade Fail

Eventually, I decided to downgrade the driver. Pacman has the downgrade ability, but you need to have an older version of the package on your disk. Fortunately, Pacman maintains a cache of previously installed packages in /var/cache/pacman/pkg, so I crafted the following command:

# CACHE_DIR=/var/cache/pacman/pkg # sudo pacman -U \ $CACHE_DIR/nvidia-open-615.78.08-4-x86_64.pkg.tar.zst \ $CACHE_DIR/nvidia-utils-615.78.08-1-x86_64.pkg.tar.zst \ $CACHE_DIR/nvidia-settings-615.78.08-1-x86_64.pkg.tar.zst \ $CACHE_DIR/lib32-nvidia-utils-615.78.08-1-x86_64.pkg.tar.zst 

It did not work. The first sign was that mkinitcpio did not rebuild correctly. The second sign was that after restart, the desktop environment did not start at all.

Root Cause

The root cause was again NVIDIA :). Precisely, the fact that nvidia-open package I've been using was compiled against a specific version of the kernel. This means I should also downgrade the kernel, which in turn could force me to downgrade more and more things, and the system could break completely meanwhile.

I recalled that there are in fact two variants of this package:

  • nvidia-open: precompiled against a specific kernel version
  • nvidia-open-dkms: dynamically compiled against current kernel during installation with DKMS (Dynamic Kernel Module Support)

I installed nvidia-open years earlier, because that was the recommendation many people gave: "use nvidia-open if you have a standard kernel, and nvidia-open-dkms for non-standard kernels". What they did not say is that nvidia-open also blocks the ability to downgrade the driver on the standard kernel :). Dynamic compilation has some cost: the system needs to compile all drivers during installation, but we do it once, that the process is not that long.

Second Downgrade

In order to switch to the DKMS driver, I needed to find the nvidia-open-dkms-615.78.08-2-x86_64.pkg.tar.gz package (older 615 version) in some trusted source, because obviously I did not have it in my cache. Then I discovered Arch Linux Archive, an official repository of older packages that I did not know about :). Usually you learn about such things, when things go wrong, because otherwise there is no use for it. And I did not have to downgrade any package for years.

I carefully retyped the full URI of the package from the mobile phone into the bare terminal, and I got the file. The downgrade with switching the driver was fairly easy:

# sudo pacman -U \ ~/nvidia-open-dkms-615.78.08-2-x86_64.pkg.tar.gz \ $CACHE_DIR/nvidia-utils-615.78.08-1-x86_64.pkg.tar.zst \ $CACHE_DIR/nvidia-settings-615.78.08-1-x86_64.pkg.tar.zst \ $CACHE_DIR/lib32-nvidia-utils-615.78.08-1-x86_64.pkg.tar.zst 

Pacman automatically detected that nvidia-open and nvidia-open-dkms are in conflict, and asked for removing the old variant. The compilation of DKMS modules took about 1 minute, mkinitcpio rebuilt normally, and after restart I had a working system again.

Lessons Learned

I don't know what exactly is wrong with the 620 version of NVIDIA driver. Later, I found reports of the problem on NVIDIA forums, so I wasn't the only person that ran into it. The lesson for me was that NVIDIA drivers were long known for causing issues on Linux for some users, but I did not prepare any fallback in case it happened to me.

nvidia-open-dkms will stay with me. The ability to downgrade the driver without touching the kernel is something that I really want to have. Most system upgrades go smoothly, but if they don't, I want to have my system back quickly.