Mirroring GMail mailboxes with IMAP

By Abhijit Menon-Sen <ams@toroid.org>

2010-04-11

Hassath wanted to download a couple of gigabytes worth of mail from her GMail account for offline storage, but was frustrated by the IMAP server closing the connection during large transfers, which made Thunderbird go into a sulk, and forced her to restart the download by hand every time. Using POP was not an option, since it had no way to preserve the labels she had assigned to the messages (which show up as mailboxes via IMAP).

I tried some other programs, but they didn't work the way I wanted. Some of them provided little or no progress information. Some would fetch all of the messages into memory before writing anything to disk. Some would fetch messages one by one, with separate command-response cycles. Most importantly, none of them coped well with the frequent disconnections.

Having exhausted my patience, and being in need of a weekend hack, I set out to write a program to download the mail. I tried the IMAP modules on CPAN, but found them lacking in one way or another. IMAP::Client parses IMAP responses in a fundamentally broken way (scanning literal data for things which look like OK responses). Net::IMAP::Client doesn't provide debugging information or allow MSN fetches. Net::IMAP doesn't know about IMAP+SSL. Most importantly, none of the modules would allow me to stream messages to disk sensibly. So I wrote one from scratch.

My program connects to imap.gmail.com and downloads messages from each mailbox (except special ones like "All Mail" and "Spam") into local Maildir-format directories. It doesn't waste bandwidth, and saves data to disk as it is received, so that it can use partial fetches to resume after an interrupted download. It is safe to stop or restart it at any time. It never deletes any mail. When it completes, you will have an offline mirror of all your GMail messages.

Hassath felt I should make the program available, especially since a friend of ours with an even more flaky network connection also wants to use it.

Installation and usage instructions

Download gmail-mirror (from github.com/amenonsen/gmail-mirror) and place it somewhere in your PATH. (Use, modification, and distribution of this code is allowed without any limitations. There is no warranty, express or implied.)

You will need to have the IO::Socket::SSL module installed (Ubuntu users can apt-get install libio-socket-ssl-perl). There are no other module dependencies.

Run "gmail-mirror.pl username" to download all your mail, or specify a list of mailboxes explicitly: "gmail-mirror.pl username INBOX work personal". (Note: GMail doesn't really have mailboxes. An IMAP mailbox named x is the set of messages labelled x.) In either case, mail will be stored locally in a Maildir-format directory named "gmail/mailboxname/somenumber" in your current directory. You can run "gmail-mirror.pl -v username" for more verbose progress information. It is safe to interrupt (^C) and restart the program at any time.

Questions are welcome

Write to me if you need help or have any suggestions. If you find this program useful, I'd love to hear about it.