Downgrading the kernel and fixing kernel modules after an incompatible new version in Fedora/Centos/RHEL

It happens quite often that a kernel module such as ZFS isn’t quite up to speed with the latest Linux kernels, and a distro such as Fedora decides to go ahead and release the latest kernel version without waiting for ZFS to be ready.

The obvious symptom of this, is when you boot your server and your ZFS array is gone, your ZFS utils may or may not be gone too, and if they aren’t, they’ll complain the ZFS kernel modules aren’t loaded, and when you try modprobe zfs Linux will tell you there’s no such thing.

Fret not! the dnf package manager allows you to downgrade the kernel to a previous version that ZFS is still compatible with. The command dnf downgrade kernel is pretty self explanatory, however, there are some caveats that need to be documented, and how to work around them.

Caveat 1: The ZFS module doesn’t get recompiled

The first thing you’ll probably notice is that, despite running dnf downgrade kernel and rebooting, your ZFS kernel modules still aren’t back.

The reason for this, is because dkms doesn’t always compile for older kernel versions, if there are multiple installed at the same time.

To fix this, after rebooting into the older kernel, simply run dnf reinstall zfs, this should be enough to fix it, although sometimes it may require a dnf remove zfs; dnf install zfs instead (don’t ask me why).

Caveat 2: “Oldest version already installed”

If you run into this problem more often with the same kernel version, it may happen that a new version of the newer kernel gets released, and dnf automatically upgrades to it, while not removing the older version.

When this happens, dnf downgrade won’t work, because the older kernel is already installed.

You CAN try and mess with Grub so that it automatically boots the correct older version, but your mileage may vary, especially as dnf overwrites it anyway if another new kernel arrives, so I don’t recommend this method.

To fix this, use dnf find kernel to find the exact version of the older kernel, copy the entire version string (e.g kernel-5.17.13-400.fc36.x86_64), and then run dnf remove 5.17.13-400.fc36.x86_64 to remove it. Afterwards, run dnf downgrade kernel again; it should work this time.

Update: You can also run dnf autoremove kernel and it should remove all the older versions:

[anthropy@stitch ~]$ sudo dnf autoremove kernel
Dependencies resolved.
 Package                                               Architecture                                          Version                                                          Repository                                               Size
 kernel                                                x86_64                                                5.19.6-200.fc36                                                  @updates                                                  0  
 kernel                                                x86_64                                                5.19.14-200.fc36                                                 @updates                                                  0  
 kernel                                                x86_64                                                5.19.16-200.fc36                                                 @updates                                                  0