Fixing "DB_RUNRECOVERY: Fatal error, run database recovery" when attempting to run yum update

Comcast is my current home ISP. Over the last year, I've had a ton of problems with them filtering all sorts of legitimate (outbound) traffic. The latest fun times I've had is the random dropping of SSH connections on both standard (22) and non-standard TCP ports. This occurred while I was running a `yum update` on one of my servers and I hadn't used `nohup` or `disown` to allow the processes I had spawned to continue to run.

By the time I had got a VPN connection up and running, the yum process had been terminated, which in turn caused yum's database to become corrupt. How can you tell that your server's yum database is corrupt? Running yum will generate this vaguely-terrifying error:

# yum update
error: rpmdb: BDB0113 Thread/process 4498/140039588845376 failed: BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 -  (-30973)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:
Error: rpmdb open failed

Here is what I did to resolve this issue:

1) I created a back of the yum database files referenced in the `cannot open Packages database in /var/lib/rpm` line of the error:

    # mv /var/lib/rpm/__db* /tmp/

You can never go wrong with backing up your data before troubleshooting. I do it reflexively.

2) I then used the following command to rebuild the database indices from the installed package headers:

    # rpm -rebuilddb

3) I cleaned out all of yum's caches and rpm header files.

    # yum clean all

At this point, the few solutions to this issue I found online before writing this article claim the issue should be resolved. Under some circumstances, this would probably work on its own. But for me, because my connection was severed by the Internet gurus at my ISP, I also had existing transactions pending in yum,

In my situation, running `yum update` again at this point produced this notification after yum began processing dependencies:

There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don't work you'll have to try removing/installing packages by hand (maybe package-cleanup can help).

So I needed to take one more step.

4) I finally ran `yum-complete-transaction` with the --cleanup-only flag. I wasn't entirely confident about how the database rebuild impacted the transaction list, so I wanted to get rid of it instead of trying to fix it.

    # yum-complete-transaction --cleanup-only

Fortunately, this fixed things for me and I didn't have to worry about "removing/installing packages by hand". However, I'd like to throw in a quick note about that and the "maybe package-cleanup can help" note in the same warning message.

The very, very first time I saw that package-cleanup error I wasn't sure what they were talking about. I thought it might have been a typo and they mean to refer to the similar `yum clean packages` command. After all, package-cleanup is:

1. not a native yum command
2. not a part of the core yum package and
3. not a separate package

So what is it? package-cleanup is part of the yum-utils package. I know my failure to understand what package-cleanup is might make me look like a dummy, but its doubtful I'm the only one who got a little confused. Check out this incredibly helpful wiki page for the application on the yum website (and be sure to mis-spell "PackageClenup" with 2 A's instead of 3 in the URI to get to the webpage):

Awesome.

I'm giving the yum guys a hard time, though. I've never read a wiki entry for `grep` and I use it every day. I only take issue with using yum-utils to resolve this specific problem: because I hadn't already installed yum-utils prior to this problem occurring, I wasn't crazy about the idea of installing the yum-utils package in the middle of trying to resolve a problem with the yum core package. I wouldn't want to install something like this from source, and installing the package using yum or rpm could very likely fail.

That's all for today. Have a good weekend, folks.