E-mail Import

From DreamHost
Jump to: navigation, search
This article or section may require a cleanup.
We are hoping to create articles that meet certain standards. Please discuss this issue on the talk page. Editing help is available.
The instructions provided in this article or section require shell access unless otherwise stated.

You can use the PuTTY client on Windows, or SSH on UNIX and UNIX-like systems such as Linux or Mac OS X.
Your account must be configured for shell access in the Control Panel.
More information may be available on the article's talk page.

The instructions provided in this article or section are considered advanced.

You are expected to be knowledgeable in the UNIX shell.
Support for these instructions is not available from DreamHost tech support.
Server changes may cause this to break. Be prepared to troubleshoot this yourself if this happens.
We seriously aren't kidding about this.

Contents

Overview of e-mail import

This article describes various methods for importing existing email archives (mailboxes, etc.; old messages) from another host to your new DreamHost account. There are a wide variety of tools and methods for accomplishing this task. Some of them work better than others, and some of them may or may not fit your situation. You may need to experiment with a few methods and tools before you find one that works for you.

Transfer using imapsync

    • NOTE that imapsync is no longer free. You must pay for the code and transfer it to your account.

(Note: I had difficulty with pop2imap freezing while transferring email. It also requires the Email::Simple perl module, which is not installed by default. imapsync seemed to work much better)

First, log into your shell account: 

mkdir temp
cd temp
(transfer latest .tgz file of imapsync up to this directory using FTP)
tar xzvf imapsync-1.286.tgz
cd imapsync-1.286
perl imapsync --host1 mail.oldimaphost.com --user1 user@oldimaphost.com --password1 XOXOXO --host2 mail.newdreamhosteddomain.com --user2 user@newdreamhosteddomain.com --password2 XOXOXO

Note: If you have not yet transferred your DNS to dreamhost (meaning that mail.yourdomain.com still points to the old host), you can obtain your DNS information by going to Domains > Manage Domains > DNS and locating the entry for 'mail'. Use the IP address of the mail server (e.g. 208.97.132.223) as the parameter to --host2

Note2: I had to explicitly add the switch:

--authmech2 PLAIN

to get imapsync to properly log in to the Dreamhost mail server

If transferring to Gmail, use:

--authmech2 LOGIN --ssl2

Transfer using pop2imap

Using pop2imap is super EASY.

First, log into your shell account. 

mkdir temp
cd temp
wget http://www.linux-france.org/prj/pop2imap/dist/pop2imap-1.18.tgz
tar xzvf  pop2imap-1.18.tgz
cd pop2imap-1.18
perl pop2imap --host1 mail.oldpophost.com --user1 user@oldpophost.com --password1 XOXOXO --host2 mail.newdreamhosteddomain.com --user2 user@newdreamhosteddomain.com --password2 XOXOXO

Done!

NOTE: If you have not yet transferred your DNS to dreamhost (meaning that mail.yourdomain.com still points to the old host), you can obtain your DNS information by going to Domains > Manage Domains > DNS and locating the entry for 'mail'. Use the IP address of the mail server (e.g. 208.97.132.223) as the parameter to --host2

Note: You need to edit the pop2imap script in order for it to work with certain hosts!

If you are not familiar with any *nix editors, Pico is relatively easy to get started with.

Find: 

$pop = new Mail::POP3Client( 
                           DEBUG    => $debugpop,
                           HOST     => $host1,
                           PORT     => $port1,
);

Replace with: 

$pop = new Mail::POP3Client( 
                           DEBUG    => $debugpop,
                           HOST     => $host1,
                           PORT     => $port1,
                           AUTH_MODE => 'PASS'
);

You are specifying the auth mode so the Perl POP3 module doesn't have to try to autodetect/guess. Other options if 'PASS' does not work are 'CRAM-MD5', 'APOP', and 'BEST'.

pop2imap uses similar arguments to imapsync. Put plaintext passwords in some files temporarily and then do something like: 

./pop2imap --host1 namethatresolvestoOldHost --user1 usernameatoldhost --passfile1 ~/mailMigration/userOldHost.secret --host2 mail.yourdomain.com --user2 m111111 --passfile2 ~/mailMigration/userDH.secret > logfile

If you are having trouble with pop2imap an alternative program that can be used is isync (http://isync.sourceforge.net).

Connecting to SSL-enabled POP3 sites

Some POP3 servers will only allow SSL connections (such as GMail). pop2imap will work with SSL POP but needs to first be modified.

Find: 

$pop = new Mail::POP3Client( 
                           DEBUG    => $debugpop,
                           HOST     => $host1,
                           PORT     => $port1,
);

Replace with: 

$pop = new Mail::POP3Client( 
                           DEBUG    => $debugpop,
                           HOST     => $host1,
                           PORT     => $port1,
                           USESSL   => "true",
);

Now, you can connect to the SSL POP server normally. Be sure to specify the TCP port of the SSL POP server as 995. 

./pop2imap --host1 namethatresolvestoOldHost --user1 usernameatoldhost --port1 995 --passfile1 ~/mailMigration/userOldHost.secret --host2 mail.yourdomain.com --user2 m111111 --passfile2 ~/mailMigration/userDH.secret > logfile

Transfer using MailUtil

MailUtil is a replacement (as of 2004) for several older mail utilities. MailUtil can be used to copy or move mail between numerous types of mail systems, including IMAP, POP3, mbox, mh, and so on. MailUtil does not support the Maildir format; and since Dreamhost uses Maildir, this must be done indirectly via IMAP, which is probably a good thing.

The syntax for mailutil is as follows: 

mailutil operation [options] source destination

Importing mail backups into your IMAP account (mbox to IMAP)

Note: Contrary to other entries on this page, MailUtil is able to read mbox files!

MailUtil can only be used to import mail "remotely" from your local system or your web server to your DreamHost email address. You must upload your mbox files to Dreamhost, either as the raw mbox files to be processed from a shell, or by using MailUtil in your local workstation since the import uploads each one of the individual emails in the mbox file. If you have sufficient upload bandwidth, we recommend using a local MailUtil. If not, we suggest you compress the mboxes, upload them to Dreamhost, and continue from a shell. When the upload finishes, expand the file, and then do a MailUtil import.

The following shows an example using Mac OS X (the binary works on both Intel-based and PowerPC Macs).

Note: MailUtil binary can be found at http://www.washington.edu/pine/getpine/unix.html 

 wget ftp://ftp.cac.washington.edu/pine/bin/mailutil-bin.osx-10.4.Z
 zcat mailutil-bin.osx-10.4.2 > mailutil
 chmod u+x mailutil

Suppose your mbox file(s) are:

  • ~/oldmail/Sent
  • ~/oldmail/Friends
  • ~/oldmail/INBOX
  • ~/oldmail/ToDo

Issue the following command for each mbox file: 

  mailutil copy -debug -verbose ~/oldmail/<file> {mail.<domain>/novalidate-cert/user=<mailuser>@<domain>}INBOX.<destination>

Where:

  • <file> is the mbox filename
  • <domain> is your hosted domain name (ie. example.com)
  • <mailuser>@<domain> is the user you use to login to your IMAP account (ie. me@example.com).
  • <destination> is the IMAP folder where your mbox content should be sent. The folder will be created if it doesn't exists yet.

Note: Relative paths DO NOT work for mailbox locations. You must use ~/oldmail/<file> instead of ./<file>

Please allow some time for MailUtil to process the mbox file. Then, you are prompted for the IMAP account's password (for <mailuser>@<domain>). When finished, you must subscribe to any new folders in your IMAP client(s).

You can obtain helpful information by calling MailUtil without parameters. The following is a Mac binary example: 

 usage: ./mailutil check [-debug] [-verbose] [mailbox]
         ;; report number of messages and new messages
        ./mailutil create [-debug] [-verbose] new_mailbox
       ;; create new mailbox
        ./mailutil delete [-debug] [-verbose] mailbox
         ;; delete existing mailbox
        ./mailutil rename [-debug] [-verbose] source destination
         ;; rename mailbox to a new name
        ./mailutil (copy | move) [-debug] [-verbose] old_mailbox new_mailbox
         ;; create new mailbox and copy/move messages
        ./mailutil (append | appenddelete) [-debug] [-verbose] source destination
         ;; copy/move messages to existing mailbox
        ./mailutil prune [-debug] [-verbose] mailbox search_criteria
         ;; prune mailbox of messages matching criteria
        ./mailutil transfer [-debug] [-verbose] [-merge mode] source destination
         ;; make copy of source hierarchy to destination
         ;;  -merge modes are prompt, append, or suffix=xxxx
  • For different operations, use 'append' instead of 'copy'.
  • The following command gives you a detailed description of various options: 
 man mailutil

See the next section if you wish to transfer mailboxes from one IMAP server to another.

Transferring an entire mailbox using MailUtil

MailUtil can be used to copy an entire mailbox, including subfolders, from one account to another (even across ISPs so long as both support IMAP).

To transfer an entire mailbox, use the following command (all on one line, as the arguments are separated for readability): 

 mailutil -d -v append {source.mail.server.com/novalidate-cert/user=<source username>}Sent
 {destination.mail.server.com/novalidate-cert/user=<destination username>}INBOX.Sent

The above example assumes you are moving sent items from mbox mailbox format to maildir format (which is what DreamHost uses). If you do not specify a source and destination folder, Inbox is used by default. If you use a source mbox folder and no destination maildir folder, it appends it to the destination inbox.

Note: The <source username> you use for each argument must be the username of the account in question, not its email address. Thus, it's the "mxxxxxxx" you see next to the account holder's real name on the {Mail -> Manage Email} page of the DreamHost panel.)

You are then prompted twice for a password: the first is for the source system, and the second is for the destination. You may see a message stating that the "INBOX already exists!". This is not an error, but instead is an informational message.

Transfer by copying files via ftp/scp (and possibly mb2md)

It is possible to transfer mail archives by copying the mailbox files directly from your old host to dreamhost, via ftp or scp.

Requirements

First, you need to create your email users from the 'Users'->'Manage Users' section of the web panel. The randomly named mailboxes that are created at the time of creating a new email address don't have FTP access. You will need FTP access to be able to copy in the converted old email. (The crucial hint provided by the DreamHost folks.)

Second, you need to convert the mbox format mail messages into Maildir format. One tool that can do this is Mb2md (there may be others). Once you have the old email in Maildir format, FTP the the contents into your 'Maildir' folder (in your FTP root directory).

NB: Webmail is no longer stored in the user's FTP-accessible 'Maildir' folder. Webmail now lives on DH mailservers to which users do not have direct access. Contact DreamHost Support for further support accessing your mailserver. --Jeffslutz1 22:57, 6 April 2009 (UTC)

Example conversion

Below is described one person's experience performing a conversion/transfer operation.

First attempt converting files from mbox to maildir

Unpacked backup.tar.gz from my old host. The old mail in mbox format was in subdirectories under: 

 backup-2.22.2006_03-18-19_preston/homedir/mail/bannister.us

The domain bannister.us is mine and the end goal is to convert old mail sent to addresses in that domain. The above path may or may not be similar in your case. Created a rather more convenient soft-link (from my home directory): 

 ln -s backup-2.22.2006_03-18-19_preston/homedir/mail/bannister.us oldmail

Downloaded and ran the mb2md conversion program according to the instructions on the Mb2md page on my local Ubuntu 5.10 box ... or at least tried to: 

 preston@mite:~/backup-2.22.2006_03-18-19_preston/homedir/mail/bannister.us$ ./mb2md -s oldmail/asg -R -d _asg
 Can't locate Date/Parse.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.8.7 /usr/local/share/perl/5.8.7 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl .) at ./mb2md line 385.
 BEGIN failed--compilation aborted at ./mb2md line 385.

Hmmm ... need to grab a missing module. If you don't get the above error, you can skip the next section.

Upgrading Perl modules using CPAN

Spent some quality time with CPAN (http://search.cpan.org/) loading modules, including eventually Date::Parse. If you do not see the above complaint from perl about a missing module, you can skip this part. This was on my local system (rather than directly on the DreamHost shared machine).

The starting point is: 

sudo perl -MCPAN -e shell

CPAN will ask you a lot of questions. You should be fine with the defaults. Otherwise you want to look elsewhere for information about using CPAN. Within the CPAN shell I ran the commands: 

install Bundle::CPAN
install Date::Parse

If you see install -- OK at the end, you should be good to go.

Second attempt converting files from mbox to maildir

I had a bunch of mailboxes to convert. 

preston@mite:~$ ls oldmail
maria  paul  preston  wendy

Commands to convert all the mailboxes from mbox to Maildir format: 

cd
mkdir tmp
for i in `ls oldmail` ; do rm -rf tmp/Maildir ; ./mb2md -R -s oldmail/$i -d tmp/Maildir && ( cd tmp && zip -rm ../maildir-$i Maildir; ) ; done

The end result is a ZIP file containing the Maildir for each account. 

preston@mite:~$ ls maildir*.zip
maildir-maria.zip  maildir-paul.zip  maildir-preston.zip  maildir-wendy.zip

Copying converted mail to DreamHost

In the DreamHost Control Panel under Users/ Manage Users (https://panel.dreamhost.com/index.cgi?tree=users.users) you will want to enable SSH login (at least temporarily) for each user.

Using your FTP client login to each account in turn, and copy down the maildir-*.zip file.

Using SSH, login to each account and run: 

unzip maildir-*.zip

At this point your old email should now appear in your mail client under the new account.

Warning

Note: The procedure above assumes you are copying old email into a new account. If you have existing email in your new account, it could be overwritten (This was not tested during the creation of this article).

Problems and Resolutions

  • Fatal: Source is not an mbox file or a directory!
    • The mb2md.pl script assumes that the mail directories provided are paths relative to the current users home directory.
    • In the context of this guide, this means that the symlink oldmail must be in your home ($HOME) directory.

External Links

Links for Windows Users

Retrieved from "http://wiki.dreamhost.com/index.php?title=E-mail_Import&oldid=32753"
Personal tools