Git disaster recovery

By Abhijit Menon-Sen <ams@toroid.org>

2010-04-12

I typed git commit and git push, and a few seconds later, the mains power died. Normally, I wouldn't have noticed, but my trusty UPS is broken, so for the first time in many years, every power glitch makes its presence felt; and now, I can fully experience the joy of being bitten in the rear by Ext4's delayed allocation.

When my machine came up again, the newly-created commit object and some associated tree objects were corrupted. refs/heads/master pointed to that corrupted commit, so most git commands died with this error message:

$ git log
fatal: object 54590b644cb542d30ec962c138a763dddc26aac0 is corrupted

To my great good fortune, my git push had completed before the power failed, so I knew I could recover everything from the remote repository. I flailed around a little before finding out how, but here's what ultimately worked for me.

First, I kept running git fsck and deleting the objects it complained about:

$ git fsck
fatal: object 54590b644cb542d30ec962c138a763dddc26aac0 is corrupted
$ rm -f .git/objects/54/590b644cb542d30ec962c138a763dddc26aac0

Then I copied the corrupted objects back from the remote repository one by one, using a trick Sam Vilain showed me on IRC:

$ ssh remote.ho.st \
    "git cat-file commit 54590b644cb542d30ec962c138a763dddc26aac0" | \
    git hash-object -w -t commit --stdin

If I had deleted the corrupted objects and reset my HEAD to point to an older commit, a plain old git fetch should have retrieved the missing objects. I didn't think of that soon enough, and recovered the missing commit first, so git fetch thought everything was up to date. But fetching the objects one by one worked fine, and git fsck stopped complaining.

I'm not sure what I would have been able to do if the remote repository had not been updated in time. I would almost certainly have lost the most recent commit, and perhaps also its immediate parent.

I really hope my UPS gets fixed soon.