E-mail Import
From DreamHost
| This article or section may require a cleanup. We are hoping to create articles that meet certain standards. Please discuss this issue on the talk page. Editing help is available. |
| The instructions provided in this article or section require shell access unless otherwise stated. You can use the PuTTY client on Windows, or SSH on UNIX and UNIX-like systems such as Linux or Mac OS X. |
| The instructions provided in this article or section are considered advanced. You are expected to be knowledgeable in the UNIX shell. |
Contents |
Overview of e-mail import
This article describes various methods for importing existing email archives (mailboxes, etc.; old messages) from another host to your new dreamhost account. There are a wide variety of tools and methods for accomplishing this task. Some of them work better than others, and some of them may or may not fit your situation. You may need to experiment with a few methods and tools before you find one that works for you.
Transfer using pop2imap
Using pop2imap is super EASY.
First, log into your shell account.
mkdir temp cd temp wget http://www.linux-france.org/prj/pop2imap/dist/pop2imap-1.12.tgz tar xzvf pop2imap-1.12.tgz cd pop2imap-1.12 perl pop2imap --host1 mail.oldpophost.com --user1 user@oldpophost.com --password1 XOXOXO --host2 mail.newdreamhosteddomain.com --user2 user@newdreamhosteddomain.com --password2 XOXOXO
Done!
NOTE: If your olddomain.com and newdomain.com are the same; for --host2 you will need to specify your Dreamhost mail server. Probably, spaceymail-master.dreamhost.com
More detailed instructions that also explain what the above commands do:
Start by retrieving the pop2imap tarball, untar it, and cd into the newly created directory:
wget http://www.linux-france.org/prj/pop2imap/dist/pop2imap-1.12.tgz tar zxf pop2imap-1.12.tgz cd pop2imap-1.12
Note: You need to edit the pop2imap script in order for it to work with certain hosts!
If you are not familiar with any *nix editors, Pico is relatively easy to get started with.
Find:
$pop = new Mail::POP3Client(
DEBUG => $debugpop,
HOST => $host1,
PORT => $port1,
);
Replace with:
$pop = new Mail::POP3Client(
DEBUG => $debugpop,
HOST => $host1,
PORT => $port1,
AUTH_MODE => 'PASS'
);
You are specifying the auth mode so the Perl pop3 module doesn't have to try to autodetect/guess. Other options if 'PASS' does not work are 'CRAM-MD5', 'APOP', and 'BEST'.
pop2imap uses similar arguments to imapsync. Put plaintext passwords in some files temporarily and then do something like:
./pop2imap --host1 namethatresolvestoOldHost --user1 usernameatoldhost --passfile1 ~/mailMigration/userOldHost.secret --host2 mail.yourdomain.com --user2 m111111 --passfile2 ~/mailMigration/userDH.secret > logfile
If you are having trouble with pop2imap an alternative program that can be used is isync (http://isync.sourceforge.net).
Connecting to SSL-enabled POP3 sites
Some POP3 servers will only allow SSL connections (such as GMail). pop2imap will work with SSL POP but needs to first be modified.
Find:
$pop = new Mail::POP3Client(
DEBUG => $debugpop,
HOST => $host1,
PORT => $port1,
);
Replace with:
$pop = new Mail::POP3Client(
DEBUG => $debugpop,
HOST => $host1,
PORT => $port1,
USESSL => "true",
);
Now, you can connect to the SSL POP server normally. Be sure to specify the TCP port of the SSL POP server as 995.
./pop2imap --host1 namethatresolvestoOldHost --user1 usernameatoldhost --port1 995 --passfile1 ~/mailMigration/userOldHost.secret --host2 mail.yourdomain.com --user2 m111111 --passfile2 ~/mailMigration/userDH.secret > logfile
Transfer using mailutil
Mailutil is a replacement (circa 2004) for several older mail utilities and is already installed on Dreamhost's systems. Mailutil can be used to copy or move mail between numerous types of mail systems including: IMAP, POP3, mbox, mh, etc. Interestingly, mailutil does not support the Maildir format. Since Dreamhost uses Maildir, must be made indirectly via IMAP - this is probably a good thing.
The syntax for mailutil is as follows:
mailutil operation [options] source destination
Importing mail backups into your IMAP account (mbox to IMAP)
Took me some time to figure out, but mailutil reads perfectly well mbox files! I was misleaded by other entries in this page.
mailutil is available in Dreamhost's servers and can also be used "remotetly". If you need/want mailutil local in your machine you can grab a binary from http://www.washington.edu/pine/getpine/unix.html. You'll have to upload your mbox files one way or another to Dreamhost, either as the raw mbox files to be processed from a shell or by using mailutil in your local workstation since the import does upload one by one each of the indivual emails in the mbox file. If you have good upstream bandwidth you'll be fine by using a local mailutil, if not I suggest you compress the mboxes, upload them to Dreamhost and continue from a shell (expand, do mailutil import).
Since I'm a Mac user I downloaded the OS X binary (its for PowerPC, but works OK in my Intel Mac thanks to Rosetta).
- Get a mailutil binary if you don't have already from http://www.washington.edu/pine/getpine/unix.html (not needed if working from Dreamhost servers)
- Example for Mac OS X:
wget ftp://ftp.cac.washington.edu/pine/bin/mailutil-bin.osx-10.4.Z zcat mailutil-bin.osx-10.4.2 > mailutil chmod u+x mailutil
Suppose your mbox file(s) are:
- ~/oldmail/Sent
- ~/oldmail/Friends
- ~/oldmail/INBOX
- ~/oldmail/ToDo
You simply need to issue the following command for each mbox file:
mailutil copy -debug -verbose ~/oldmail/<file> {mail.<domain>/novalidate-cert/user=<mailuser>@<domain>}INBOX.<destination>
The binary in Dreamhost seems to expect a slighly different parameter order:
mailutil -d -v copy ~/oldmail/<file> {mail.<domain>/novalidate-cert/user=<mailuser>@<domain>}INBOX.<destination>
Where:
- <file> is the mbox filename
- <domain> is your hosted domain name (ie. example.com)
- <mailuser>@<domain> is the user you use to login to your IMAP account (ie. me@example.com).
- <destination> is the IMAP folder where your mbox content should be sent. The folder will be created if it doesn't exists yet.
(Note that relative paths DO NOT work for mailbox locations that is, you need to have e.g. ~/oldmail/<file> instead of ./<file> )
mailutil will take some time processing the mbox file, then will prompt for the IMAP account's password (for <mailuser>@<domain>). When all is done you'll need to subscribe to any new folders in your IMAP client(s).
Calling mailutil without parameters outputs some help info. The Mac binary I got issues:
usage: ./mailutil check [-debug] [-verbose] [mailbox]
;; report number of messages and new messages
./mailutil create [-debug] [-verbose] new_mailbox
;; create new mailbox
./mailutil delete [-debug] [-verbose] mailbox
;; delete existing mailbox
./mailutil rename [-debug] [-verbose] source destination
;; rename mailbox to a new name
./mailutil (copy | move) [-debug] [-verbose] old_mailbox new_mailbox
;; create new mailbox and copy/move messages
./mailutil (append | appenddelete) [-debug] [-verbose] source destination
;; copy/move messages to existing mailbox
./mailutil prune [-debug] [-verbose] mailbox search_criteria
;; prune mailbox of messages matching criteria
./mailutil transfer [-debug] [-verbose] [-merge mode] source destination
;; make copy of source hierarchy to destination
;; -merge modes are prompt, append, or suffix=xxxx
For different operations (ie. use 'append' instead of 'copy') you already got the basic idea ;)
Also, the command
man mailutil
will give a detailed description of various options.
See next section if you need to transfer mailboxes from one IMAP server to another.
Transferring an entire mailbox using MailUtil
Mailutil can be used to copy an entire mailbox, including subfolders from one account to another (even across ISPs so long as both support IMAP).
To transfer an entire mailbox, use the following command (all on one line--the arguments are separated for readability):
mailutil -d -v append {source.mail.server.com/novalidate-cert/user=<source username>}Sent
{destination.mail.server.com/novalidate-cert/user=<destination username>}INBOX.Sent
The above example assumes you are moving sent items from mbox mailbox format to maildir format (which is what DreamHost uses). If you do not specify a source and destination folder, Inbox will be used by default. If you use a source mbox folder and no destination maildir folder, it will append it to the destination inbox.
Also note that the <source username> you use for each argument must be username of the account in question, not its email address. (So, it's the "mxxxxxxx" you see next to the account holder's real name on your Dreamhost "Manage email" control panel.)
You will be prompted twice for a password. The first is for the source system and the second is for the destination. You may see a message stating that "INBOX already exists!". This is not an error, just an informational message.
Transfer by copying files via ftp/scp (and possibly mb2md)
It is possible to transfer mail archives by copying the mailbox files directly from your old host to dreamhost, via ftp or scp.
Requirements
First, you need to create your email users from the 'Users'->'Manage Users' section of the web panel. The randomly named mailboxes that are created at the time of creating a new email address don't have FTP access. You will need FTP access to be able to copy in the converted old email. (The crucial hint provided by the DreamHost folks.)
Second, you need to convert the mbox format mail messages into Maildir format. One tool that can do this is Mb2md (there may be others). Once you have the old email in Maildir format, FTP the the contents into your 'Maildir' folder (in your FTP root directory).
Example conversion
Below is described one person's experience performing a conversion/transfer operation.
First attempt converting files from mbox to maildir
Unpacked backup.tar.gz from my old host. The old mail in mbox format was in subdirectories under:
backup-2.22.2006_03-18-19_preston/homedir/mail/bannister.us
The domain bannister.us is mine and the end goal is to convert old mail sent to addresses in that domain. The above path may or may not be similar in your case. Created a rather more convenient soft-link (from my home directory):
ln -s backup-2.22.2006_03-18-19_preston/homedir/mail/bannister.us oldmail
Downloaded and ran the mb2md conversion program according to the instructions on the Mb2md page on my local Ubuntu 5.10 box ... or at least tried to:
preston@mite:~/backup-2.22.2006_03-18-19_preston/homedir/mail/bannister.us$ ./mb2md -s oldmail/asg -R -d _asg Can't locate Date/Parse.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.8.7 /usr/local/share/perl/5.8.7 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl .) at ./mb2md line 385. BEGIN failed--compilation aborted at ./mb2md line 385.
Hmmm ... need to grab a missing module. If you don't get the above error, you can skip the next section.
Upgrading Perl modules using CPAN
Spent some quality time with CPAN (http://search.cpan.org/) loading modules, including eventually Date::Parse. If you do not see the above complaint from perl about a missing module, you can skip this part. This was on my local system (rather than directly on the DreamHost shared machine).
The starting point is:
sudo perl -MCPAN -e shell
CPAN will ask you a lot of questions. You should be fine with the defaults. Otherwise you want to look elsewhere for information about using CPAN. Within the CPAN shell I ran the commands:
install Bundle::CPAN install Date::Parse
If you see install -- OK at the end, you should be good to go.
Second attempt converting files from mbox to maildir
I had a bunch of mailboxes to convert.
preston@mite:~$ ls oldmail maria paul preston wendy
Commands to convert all the mailboxes from mbox to Maildir format:
cd mkdir tmp for i in `ls oldmail` ; do rm -rf tmp/Maildir ; ./mb2md -R -s oldmail/$i -d tmp/Maildir && ( cd tmp && zip -rm ../maildir-$i Maildir; ) ; done
The end result is a ZIP file containing the Maildir for each account.
preston@mite:~$ ls maildir*.zip maildir-maria.zip maildir-paul.zip maildir-preston.zip maildir-wendy.zip
Copying converted mail to DreamHost
In the DreamHost Control Panel under Users/ Manage Users (https://panel.dreamhost.com/index.cgi?tree=users.users) you will want to enable SSH login (at least temporarily) for each user.
Using your FTP client login to each account in turn, and copy down the maildir-*.zip file.
Using SSH, login to each account and run:
unzip maildir-*.zip
At this point your old email should now appear in your mail client under the new account.
Warning
Note: The procedure above assumes you are copying old email into a new account. If you have existing email in your new account, it could be overwritten (This was not tested during the creation of this article).
Problems and Resolutions
- Fatal: Source is not an mbox file or a directory!
- The mb2md.pl script assumes that the mail directories provided are paths relative to the current users home directory.
- In the context of this guide, this means that the symlink oldmail must be in your home ($HOME) directory.
External Links
- imapsync (http://directory.fsf.org/imapsync.html)
- pop2imap (http://directory.fsf.org/pop2imap.html)
- imap_migrate (http://directory.fsf.org/imap_migrate.html)
- migrationtool (http://directory.fsf.org/migrationtool.html)
- mb2md (http://batleth.sapienti-sat.org/projects/mb2md/)
Links for Windows Users
- Imap backup&restore(http://broobles.com/imapsize)
