Downgrading the kernel and fixing kernel modules after an incompatible new version in Fedora/Centos/RHEL
It happens quite often that a kernel module such as ZFS isn’t quite up to speed with the latest Linux kernels, and a distro such as Fedora decides to go ahead and release the latest kernel version without waiting for ZFS to be ready.
The obvious symptom of this, is when you boot your server and your ZFS array is gone, your ZFS utils may or may not be gone too, and if they aren’t, they’ll complain the ZFS kernel modules aren’t loaded, and when you try
modprobe zfs Linux will tell you there’s no such thing.
Fret not! the
dnf package manager allows you to downgrade the kernel to a previous version that ZFS is still compatible with. The command
dnf downgrade kernel is pretty self explanatory, however, there are some caveats that need to be documented, and how to work around them.
Caveat 1: The ZFS module doesn’t get recompiled
The first thing you’ll probably notice is that, despite running
dnf downgrade kernel and rebooting, your ZFS kernel modules still aren’t back.
The reason for this, is because dkms doesn’t always compile for older kernel versions, if there are multiple installed at the same time.
To fix this, after rebooting into the older kernel, simply run
dnf reinstall zfs, this should be enough to fix it, although sometimes it may require a
dnf remove zfs; dnf install zfs instead (don’t ask me why).
Caveat 2: “Oldest version already installed”
If you run into this problem more often with the same kernel version, it may happen that a new version of the newer kernel gets released, and dnf automatically upgrades to it, while not removing the older version.
When this happens,
dnf downgrade won’t work, because the older kernel is already installed.
You CAN try and mess with Grub so that it automatically boots the correct older version, but your mileage may vary, especially as dnf overwrites it anyway if another new kernel arrives, so I don’t recommend this method.
To fix this, use
dnf find kernel to find the exact version of the older kernel, copy the entire version string (e.g kernel-5.17.13-400.fc36.x86_64), and then run
dnf remove 5.17.13-400.fc36.x86_64 to remove it. Afterwards, run
dnf downgrade kernel again; it should work this time.
Update: You can also run
dnf autoremove kernel and it should remove all the older versions:
[anthropy@stitch ~]$ sudo dnf autoremove kernel Dependencies resolved. ============================================================================================================================================================================================================================================ Package Architecture Version Repository Size ============================================================================================================================================================================================================================================ Removing: kernel x86_64 5.19.6-200.fc36 @updates 0 kernel x86_64 5.19.14-200.fc36 @updates 0 kernel x86_64 5.19.16-200.fc36 @updates 0