Gmail to Google Apps with Larch
Preamble: You've moved your primary email account from one Gmail account to another (e.g. regular Gmail to Google Apps).
Target: You want to move all your email archive from your old account to your new account and in the process of doing so you want to preserve your message dates.
In this article I show you how I managed to migrate all my mail with Larch.
There are a lot of article available online for migrating or consolodating email accounts. I'll list a few:
- Squidoo.com lists a few options from using simple forwarding, via Thunderbird to move emails to using Mailtraq for Windows.
- Migrationbox is a paid service that takes the hassle away. It gets good reviews but it costs $48,- to migrate one account (Bulk rates available).
- imapsync is a tool written in Perl that can migrate email from one IMAP account to another. There are tutorials here and here about how you can use it to move between Gmail accounts. However imapsync is no longer freely available (€ 30,-) and when I used it last year and now again it generated a lot of errors.
- Larch is what I finally settled for. It's predecessor was mentioned by one of the commenters on imapsync as a more stable successor. Reading the docs, it looked as a well maintained and stable tool, so I gave it a try.
Larch is written in Ruby and can be installed via Rubygems. Both need to be installed on the computer/server that will be running the migration. You can find a tutorial on how to set this up on OS X at Hivelogic.com. Because I needed to migrate thousands of messages I decided to set it up on my server (Debian Lenny). I used the latest version of Ruby via the Debian Backports repository.
sudo apt-get install ruby1.9.1-full
After Ruby and Rubygems are installed you install Larch with a single command.
sudo gem install larch
That's it. Larch should now run.
I hardly use Gmail labels and I generally keep all my messages in the Inbox as long as they require action from my side. As soon as a conversation thread is finished I archive the thread. If I want to find an old message I simply search.
I only use labels for email that I need to browse through. This always involves messages that are filtered out of my inbox. E.g. non-critical mail with server updates.
To migrate my email I needed to move the All Mail folder and the Sent Mail folder. You should note that every message exists only once on the Gmail server, but it can appear in different folders at the same time because it has been labeled multiple times.
The folders I needed to move are actually Gmail labels named
[Gmail]/All Mail and
[Gmail]/Sent Mail or different if you're using another language for your Gmail (e.g.
[Gmail]/Alle berichten and
[Gmail]/Verzonden berichten in Dutch). The
[Gmail]/ prefix is used for all default Gmail folders.
First I moved over all my messages by migrating the All Mail folder.
larch --from imaps://imap.gmail.com --to imaps://imap.gmail.com \ --from-folder '[Gmail]/All Mail' --to-folder '[Gmail]All Mail'
When you execute this command Larch will ask for your username (full email address) and password for both accounts.
It takes a while before Larch finishes moving your messages. This off course depends on the amount of messages you need to transfer. After the first transfer was done I moved over the mail from the Sent Mail folder.
larch --from imaps://imap.gmail.com --to imaps://imap.gmail.com \ --from-folder '[Gmail]/Sent Mail' --to-folder '[Gmail]/Sent Mail'
I specifically chose to migrate my emails in this order. Gmail always puts all messages in the All Mail folder even the messages sent by you. All the other "folders" in Gmail are actually labels. With the first command I already moved all my emails to the new Gmail account. With this command I will label all messages in the Sent Mail folder of my old account as Sent Mail in my new account. The messages will not be duplicated.
You can repeat this step for all subsequent folders/labels that you want to migrate. E.g. your starred messages (
[Gmail]/Met ster in Dutch).
larch --from imaps://imap.gmail.com --to imaps://imap.gmail.com \ --from-folder '[Gmail]/Starred' --to-folder '[Gmail]/Starred'
Etcetera, etcetera, etc.
Labeling your old messages
I wanted to label the messages from my old account with a legacy label in my new account. In my first test I tried to achieve this by moving all mail from the old account to a folder/label called
legacy in the new account. Gmail always puts all messages in the All Mail box automatically, so by migrating from All Mail to legacy (instead of All Mail too) I would label all messages that came from the old account. This approach works fine if your new account is empty. In my case there were duplicate messages because I already started using the new account and had forwarded messages between the two. The above approach also labels messages that are duplicate and already present in the new account.
In the end I chose to transfer from All Mail to All Mail. I still achieved to label the old mail by taking the following steps:
- I labeled the existing messages in my new account with a temporary label before I started the migration.
- Then I migrated all messages (also finishing all the folder migrations).
- After the migration was complete, I labeled all messages (You cannot select messages without label in Gmail) in the new account with the legacy label.
- Then I selected all messages with the temporary label and deleted both the legacy subsequently the temporary label from this selection.
- Finally only your old messages will be labeled legacy.
Once the migration of all mail is finished you might want to remember the following before you delete your old account.
- Contacts: You can export your contacts in Google CSV format and import them in your new account.
- Filters: If you go Settings → Labs, you can enable Filter import/export. Do this in both accounts and you can export your filters to an XML file and import them in your new account.
- Other settings: Place 2 browsers side to side and walk through the different settings to see what settings you might have tweaked in the time that you've used your old account.
If all your settings are ready, you can move on to the next step of your migration: Calendars, Documents, etc...