Tutorial - How To Sync IMAP Mailboxes Using Imapsync

How To Sync IMAP Mailboxes Using Imapsync

Warnings
• We recommend that only experienced system administrators perform these steps. SSH access and root-level privileges are required to execute the commands included in this tutorial.
• cPanel Technical Support does not provide assistance with the steps in this tutorial.

Summary
The purpose of this article is to provide information about using the imapsync utility to sync emails from one server to another. This may be needed to sync any emails that have arrived at the original server before the DNS changes could fully propagate. While cPanel Support does not directly support the installation or use of imapsync, we believe a short tutorial explaining its basic usage could be of great help.

Preparation
We will be referring to multiple servers in this article, the source server will be the server you are syncing emails from and will be represented by the IP address - 10.2.33.1 - in our examples. The destination server will be the server you are syncing emails to, and will be represented by the term 'localhost' in our examples. As the primary purpose of this article is to sync emails after a migration, we will also be assuming that the email accounts exist on both servers already. It is important to note that the imapsync utility can be used as the cPanel user as well as the root user, and can be run from the source server, the destination server or even a third/separate server. For the sake of security, simplicity and the scope of this article, we will be operating as the root user on the destination server.

To prepare to follow the steps in this article, you will need:

an SSH connection will need to be established on the destination server as the root user.
the screen RPM installed (optional, but recommended)
the cPanel username and password of the account you would like to sync (for both servers)

Beginning

(Optional) - Start a screen:
Code:
```
screen -S imapsync
```
Install the imapsync utility:
Code:
```
yum install imapsync -y
```
Once installed, you may view the help page by running:
Code:
```
imapsync --help
```
The imapsync utility will need 3 primary components for each server, the host, username and password. To prepare for this, we will want to a create a Dovecot 'masteruser' on both servers which will allow us to use one set of credentials for the entire process. To do this, we will need to make some modifications to the Dovecot configuration on both servers. If you have already customized your Dovecot installation, this step can be skipped:
Code:
```
cp -via /var/cpanel/templates/dovecot2.3/main.default /var/cpanel/templates/dovecot2.3/main.local
```
Once the 'main.local' file has been created, it will need to be opened with your preferred file editor. Then find the line that looks like this (usually around line #1234) - ## Password and user databases. Just below that line, the following section will need to be added:
Code:
```
auth_master_user_separator = *
passdb {
driver = passwd-file
args = scheme=ssha512 /etc/dovecot/passwd.masterusers
master = yes
pass = yes
}
```
Next, we want to add the ACL which will allow our Dovecot master user to access each of the user's mailboxes. We will need to find the lines that match this (around line #1898):
Code:
```
[%- IF plugin.acl.defined %]
acl = [% plugin.acl %]
```
and we need to add this line just below that:
Code:
```
acl = vfile:/etc/dovecot/dovecot-acl
```
which leaves that section looking like this:
Code:
```
[%- IF plugin.acl.defined %]
acl = [% plugin.acl %]
acl = vfile:/etc/dovecot/dovecot-acl
[%- ELSE %]
```
Once the above has been added, we will need to create the password for the master user. That can be accomplished with the following:
Code:
```
doveadm pw -s SSHA512
```
you will be prompted to enter your desired password. I would recommend using the same user/password on the source and destination servers for the sake of simplicity, we will be removing the master users at the end of this process for the sake of security. The output received should look like this:
Code:
```
{SSHA512}hX3ylWTr89xjIkXY5cca2FwMyXiB9bwqStfPpUAg3hm2cV+XlNn0Y3e2WG/A91DVN3BM4VJ9n5z7sAC0CU8L5ZtDjH4=
```
Save this line (and remember your chosen password!), as it will be needed in the following steps!

Now we need to create the master user's password file and the dovecot-acl file:

Code:

touch /etc/dovecot/{passwd.masterusers,dovecot-acl} && chmod -v 600 /etc/dovecot/{passwd.masterusers,dovecot-acl} && chown -v dovecot. /etc/dovecot/{passwd.masterusers,dovecot-acl}

Next, we will need to add our username and the SHA encrypted password from step 5 into the passwd.masterusers file in the format of 'user:password', I chose 'imapsync' as my user, here is an example of what it should look like:
Code:
```
cat /etc/dovecot/passwd.masterusers
imapsync:{SSHA512}3UQuYuUwDmyfma4+ToZ1i/cQeRmvsixTMIGpWsvxG6RNYEMVLhbA8vp7s32rS42nxfTKGrQtAfzvmkSrAMr7rYRvKCg=
```
We now need to add the necessary ACLs to our dovecot-acl file. On the source server, we do not need as many permissions, just list and read; so the following must be added on the source server:
Code:
```
* user=imapsync lr
```
The destination server needs a bit more permission as it has to create files, write them, etc. Add the following line to the dovecot-acl file on the destination server:
Code:
```
* user=imapsync lrwstipekx
```
Finally, we need to rebuild the Dovecot configuration file and restart the dovecot service:
Code:
```
/scripts/builddovecotconf && /scripts/restartsrv_dovecot
```
(remember to perform steps 3-8 on both the source and destination server!)
Now we will need to set up our working directory, in this example, we are on the source server syncing emails to the destination server:
Code:
```
mkdir -pv /root/imapsync && cd /root/imapsync
```
In this directory, we will want to create a password file for imapsync to use, this will need to contain the password of the Dovecot master user we created above Create the file and add the password using your preferred editor.
Code:
```
touch pass-file.txt && vim pass-file.txt
```
Now we will want to create a list of all of the email accounts on the server so that we can sync them all at once. The following will pull all of the email accounts for each user and place them in a file called 'email-accounts.txt':
Code:
```
while read -r domain user; do uapi --user="${user}" Email list_pops | grep -oP 'email:\s+\K.+'; done < /etc/trueuserdomains 2>/dev/null 1> email-accounts.txt
```

The Transfer

Now it is time to formulate the imapsync loop, the basic outline of the command will look like this:

Code:

while read -r eacct; do imapsync --dry --host1 ${source-server-IP} --authuser1 "${eacct}*{your-dovecot-master-user}" --user1 "${eacct}*${your-dovecot-master-user}" -passfile1 pass-file.txt -ssl1 --host2 ${destination-server-IP} --authuser2 "${eacct}*${your-dovecot-master-user}" --user2 "${eacct}*${your-dovecot-master-user}" -passfile2 pass-file.txt -ssl2 --dry; done < email-accounts.txt

so if we plug our information in as an example, it will look like this:

Code:

while read -r eacct; do imapsync --dry --host1 localhost --authuser1 "${eacct}*imapsync" --user1 "${eacct}" -passfile1 pass-file.txt -ssl1 --host2 10.2.33.178 --authuser2 "${eacct}*imapsync" --user2 "${eacct}" -passfile2 pass-file.txt -ssl2; done < email-accounts.txt

Please note that both of the examples above contain the '--dry' flag. This prevents imapsync from performing any actions, but will run a test (dry) run and will report any errors it would have encountered. With that in mind, plug in your source/destination server IP address and try to run the command. Here is some truncated output from a test cPanel account:

Code:

Host2 folder 7/12 [INBOX.cptech@cptech_testing] Size: 104914 Messages: 5 Biggest: 50462
Host2 folder 8/12 [INBOX.cptest@cptech_testing] Size: 101507 Messages: 3 Biggest: 50454
Host2 folder 9/12 [INBOX.random1sa@cptech_testing] Size: 50549 Messages: 1 Biggest: 50549
Host2 folder 10/12 [INBOX.random2sa@cptech_testing] Size: 50549 Messages: 1 Biggest: 50549
Host2 folder 11/12 [INBOX.sa@cptech_testing] Size: 0 Messages: 0 Biggest: 0
Host2 folder 12/12 [INBOX.spam] Size: 0 Messages: 0 Biggest: 0
Host2 Nb folders: 12 folders
Host2 Nb messages: 11 messages
Host2 Total size: 307900 bytes (300.684 KiB)
Host2 Biggest message: 50549 bytes (49.364 KiB)
Host2 Time spent: 0.1 seconds
++++ Statistics
Transfer started on : Thu May 16 01:19:12 2019
Transfer ended on : Thu May 16 01:19:14 2019
Transfer time : 2.3 sec
Folders synced : 12/12 synced
Messages transferred : 0 (could be 0 without dry mode)
Messages skipped : 7
Messages found duplicate on host1 : 0
Messages found duplicate on host2 : 0
Messages void (noheader) on host1 : 0
Messages void (noheader) on host2 : 0
Messages deleted on host1 : 0
Messages deleted on host2 : 0
Total bytes transferred : 0 (0.000 KiB)
Total bytes duplicate host1 : 0 (0.000 KiB)
Total bytes duplicate host2 : 0 (0.000 KiB)
Total bytes skipped : 105886 (103.404 KiB)
Total bytes error : 0 (0.000 KiB)
Message rate : 0.0 messages/s
Average bandwidth rate : 0.0 KiB/s
Memory consumption : 303.1 MiB
Biggest message : 0 bytes (0.000 KiB)
Memory/biggest message ratio : NA
Start difference host2 - host1 : 4 messages, 202014 bytes (197.279 KiB)
Final difference host2 - host1 : 4 messages, 202014 bytes (197.279 KiB)
Detected 0 errors

If no errors are reported, remove the '--dry' flag and run the command once more to begin the sync.
Once that is running, we will want to remove the previously created credential file and email-accounts list file:
Code:
```
rm -v /root/imapsync/pass-file.txt /root/imapsync/email-accounts.txt
```
If you are connected to a screen session, use 'ctrl + a + d' to detach from the screen session and let the sync run in the background.

Post-Sync Cleanup

After the sync has finished, we will want to remove the Dovecot master users and ACL customizations that we created. There are two ways to go about doing this, this first way will be for users who did not previously have any customizations in place:
Code:
```
rm -v /etc/dovecot/passwd.masterusers /var/cpanel/templates/dovecot2.3/main.local && /scripts/builddovecotconf && /scripts/restartsrv_dovecot
```
If you previously had customizations in place, the 'passdb' section we added in step 4 of the 'Beginning' section will need to be manually removed, along with the dovecot-acl line:
Code:
```
auth_master_user_separator = *
passdb {
driver = passwd-file
args = scheme=ssha512 /etc/dovecot/passwd.masterusers
master = yes
pass = yes
}

acl = vfile:/etc/dovecot/dovecot-acl
```
Next, the passwd.masterusers file and the dovecot-acl file can be removed:
Code:
```
rm -vi /etc/dovecot/{passwd.masterusers,dovecot-acl}
```
Then the following command will need to be run to ensure dovecot is fully cleaned up:
Code:
```
/scripts/builddovecotconf && /scripts/restartsrv_dovecot
```

***Remember to perform this clean up on both servers!***

Additional Questions / Feedback
Feel free to click on the Discussion tab to let us know if you have any questions or feedback about the information in this tutorial.

Reactions: RadWebHosting

Search

Search

Tutorial How To Sync IMAP Mailboxes Using Imapsync

Share this resource