Streaming homedir in copy process doesn't work

[MediaServe]

cPanel 11.24.4-C35075 - WHM 11.24.2 - X 3.9

The account copy feature is really frustrating me!

We've started putting upgraded servers online and moving sites to them, but the copy process doesn't seem to work larger sites (which we tend to have a lot of.)

For example, this morning I attempted to move a 15GB site to a new server.

The tail end of the homedir streaming process of the copy reports this:

..11254784550..
Done
xferstream tag passed!

The homedir has 15GB but has only transfered 11GB, and yet it is reporting success! What is up with that?

A du -sh on the old server's homedir shows 15GB, on the new server it's 11GB. I have to tar the whole homedir across after the copy to make sure everything is there.

The problem is, I really need to do some bulk copies, but I can't with the process being this unreliable. Anyone else noticing this?

Also, SSL certificates and keys are never copied properly - they always have to be manually corrected afterwards.

 

[cPanelDavidG]

cPanel 11.24.4-C35075 - WHM 11.24.2 - X 3.9

The account copy feature is really frustrating me!

We've started putting upgraded servers online and moving sites to them, but the copy process doesn't seem to work larger sites (which we tend to have a lot of.)

For example, this morning I attempted to move a 15GB site to a new server.

The tail end of the homedir streaming process of the copy reports this:

..11254784550..
Done
xferstream tag passed!

The homedir has 15GB but has only transfered 11GB, and yet it is reporting success! What is up with that?

A du -sh on the old server's homedir shows 15GB, on the new server it's 11GB. I have to tar the whole homedir across after the copy to make sure everything is there.

The problem is, I really need to do some bulk copies, but I can't with the process being this unreliable. Anyone else noticing this?

Also, SSL certificates and keys are never copied properly - they always have to be manually corrected afterwards.

Please submit a support ticket so our technical analysts can assist you with this. Also mention the SSL certificate issue so we can take a look at what is causing that. You can contact our technical analysts at: http://tickets.cPanel.net/submit

 

[wwwcad]

Same problem here

Today i had copied a lot of accounts with the whm multicopy script, but for several accounts, like 20/150, it report sucess with missing data from the old homedir. Sometimes the public_html had missing files or the email accounts were missing in the new cpanel.

These transfer scripts seems to have problems with the old step: Streaming Homedir from source server.... It should compare at least the size of the home directory before reporting success.

Regards

 

[Vision]

Same problem here with multiple copy. 10/40 accounts were missing data from public_html and no email accounts created in cpanel.

 

[cPanelDavidG]

I recommend anyone experiencing these issues to submit a support ticket so we can determine the causes of this situation.

 

[cPanelKenneth]

Please verify that all directories and files in the home directory on the source server either:

1) are owned by user:user; or

2) have permissions 0755 for directories and 0644 for files

The streaming process is performed by the account user name ( not root ). Hence any file or directory the user cannot access or read will be skipped by the streaming process.

 

[MediaServe]

Please verify that all directories and files in the home directory on the source server either:

1) are owned by user:user; or

2) have permissions 0755 for directories and 0644 for files

The streaming process is performed by the account user name ( not root ). Hence any file or directory the user cannot access or read will be skipped by the streaming process.

This is not the problem in our case - all ownership/permissions are good.

I'll try to open a ticket soon, as we really need this fixed so we can start migrating sites to new servers.

 

[MediaServe]

I never did open a ticket on this, but probably will soon. For anyone who comes across this thread, I thought I'd post some current observations.

This problem seems to happen for sites that are over about 5GB. Sites under 5GB don't exhibit the problem. For sites over 5GB, the process seems to stop at the same relative spot close to 5GB - like there is some sort of 5GB limitation to the file streaming.

I've started moving sites from one data center to another, and I started with the big sites, 10GB-20GB in size. None of them are completely transferring, and I started noticing with the last couple that they seem to only transfer about 5GB of files. Oddly, the process says the streaming is complete - i.e. no errors, just not everything is transferred.

I just did a 4.7GB site to see if it worked properly, and it did. I'm going to do a couple more transfers of >5GB sites and will post exactly where they stop in the streaming process, then open a ticket for further investigation.

 

[dave_g]

Hello,

I used cpanel for many years, now I have big accounts too (5-10Go) of different customers. What is the best way to transfer this kind of account when we have to change of server?


Dave G

 

[MediaServe]

Here are the results of a couple tests:

First site actual size (per du -s --bytes in home directory) is 8984700792.

Last few lines of the streaming process:

..5436783600..
..5439568838..
...........
..5442354075..
...........
..5445139313..
Done
xferstream tag passed!

Second site actual size is 8037912962

Last few lines of the streaming process:

..4941011325..
..4943796563..
..4946581800..
...........
..4949367038..
Done
xferstream tag passed!

When I used rsync to finish transferring files for these sites, the files which had been missed were files with proper ownership and permissions, so no reason for them not to transfer.

 

[cPanelCory]

Can I get the version and build numbers of both servers?

Thanks!

 

[MediaServe]

Can I get the version and build numbers of both servers?

Thanks!

Both servers are: cPanel 11.25.0-C42399 - WHM 11.25.0 - X 3.9

I am meaning to take the content from one of these large sites and create a dummy site that can be used for testing, and then provide access information in a ticket.

Would this be useful, or can you perhaps duplicate this in house?

 

[cPanelDon]

Both servers are: cPanel 11.25.0-C42399 - WHM 11.25.0 - X 3.9

I am meaning to take the content from one of these large sites and create a dummy site that can be used for testing, and then provide access information in a ticket.

Would this be useful, or can you perhaps duplicate this in house?

Thank you for the version information. So that we may examine the issue more in-depth, obtaining more detail to reproduce the symptoms, and investigate the surrounding circumstances please submit a support request/ticket referencing this thread; when available, please PM me your new ticket ID number so I may follow-up internally. For reference, the links in my forums signature may be used to initiate a ticket submission.

 

[cPanelCory]

When you do start a ticket with access to your servers where this is occurring please put ATTN QA: 5GB Transfer Limit in the subject.

I could not reproduce this with the builds you mentioned on a 10G account:

..11088030488..
..11090815725..
..11093600963..
..11096386200..
..11099171438..
..11101956675..
..11104741913..
..11107527150..
..11110312388..
...........

..11113097625..
..11115882863..
..11118668100..
Done
xferstream tag passed!


The servers you are seeing this on will give up invaluable information as to what is causing this issue on your servers and we can go from there =)

Thanks!

 

[urano007]

I have the same problem, but my site is 4 GB.
How I can create a ticket?

 

[JaredR.]

You can submit a ticket by clicking the following link:

https://tickets.cpanel.net/submit/

 

[urano007]

The tiicket is 1031677.

Thank you.