Delete Duplicate emails

Deleting duplicate emails.
Note: This only works if you have a shell account associated with your email.

So you have a maildir folder that is full of duplicates. If you have hundreds or thousands of duplicates this method may work for you.

PCRE
Download and install PCRE libs because it's a dependency of maildrop.

Setup your path
Add $HOME/local/bin to your $PATH

Maildrop
Downlaod and install maildrop. We need this for the reformail tool

Go to the maildir directory
cd $HOME/.maildir/.Folder-name

Make sure there's nothing sitting in the new subdirectory.
ls new If there are messages in the new subdirectory, open the mailbox in a user agent to get it to move them into cur.

See how many messages you have:
ls cur | wc -l 842

Note: In this example we counted a total of 5746 messages

Check they all have Message-IDs:
Note: This may take a long time on a large folder for i in cur/*; do nice reformail -x Message-ID: <$i; done | wc -l 842

See how many you have if you filter out duplicate Message-IDs:
Note: This may take a long time on a large folder for i in cur/*; do nice reformail -x Message-ID: <$i; done | sort -u | wc -l 698

See how many we're going to delete:
rm /tmp/$USER.dups for i in cur/*; do nice reformail -D 20000 /tmp/$USER.dups <$i && echo $i; done | wc -l 144 expr 698 + 144 842

If this total doesn't match you should increase the 20000 - reformail isn't remembering enough Message-IDs to spot all the duplicates.

Make a backup of cur (optional):
Just incase you screw up. cp -a cur backup

Delete the messages and check things look right afterwords:
rm /tmp/$USER.dups for i in cur/*; do nice reformail -D 20000 /tmp/$USER.dups <$i && rm -fv $i; done | nl ls cur | wc -l 698

Cleanup your temporary file
rm /tmp/$USER.dups

Remove Duplicate Messages Plugin
Remove Duplicate Messages Plugin Note: If you delete more than 300 message or so at a time, there's a good chance thunderbird will timeout, and resend the command, resulting in lots of duplicate messages in your trash folder.