Migration to new server almost cost us a big client

Operating System & Version
CloudLinux 8
cPanel & WHM Version
v108.0.14

DennisMidjord

Well-Known Member
Sep 27, 2016
352
78
78
Denmark
cPanel Access Level
Root Administrator
This is really a big issue for us.
We have a client that has a few VPS services with us. We needed to migrate their cPanel/WHM servers to a server running a newer version of CloudLinux.

We made sure to transfer the configuration for cPanel, EasyApache and stuff like that using the transfer tool. Once we made sure everything was as similar to the old server as possible, we started migrating.
This is where the first issue started... The Transfer Tool refused to transfer more than 1 account at a time. I stopped the transfer, made sure that the transfer threads was set to 5 and started again. It still refused to transfer more than 1. I had to do this a couple of times before it managed to transfer 5 accounts at a time. Both servers had 12 CPU cores and 12GB of RAM.
I started researching the issue and came across this: https://support.cpanel.net/hc/en-us...-at-once-Transfer-and-Restore-Thread-options-
The server almost always had a load of 1 so it wasn't overloaded.

Well, moving on because that wasn't deal breaking for us. We managed to get all accounts transferred.
Now the annoying part starts...

After migrating all accounts, I took a few samples to make sure everything was alright. I checked around 20 websites, clicked around them and noticed that everything seemed to be working. I notify the client that the migration is over and we close the case.
Then hell broke lose. I received a call saying that one of the websites suddenly stopped working. I check the website, and sure enough - it throws a 500 error.
I find out that the PHP version has been changed for the website. It's set to PHP 8.0 (the default PHP version on the new server - the old one had PHP 8.1 as the default) while it should be running PHP 7.0. I know it's a deprecated version but it's needed for some websites.
It's worth noting that at this point, our client didn't even have access to the server. Neither did our client's clients.

No biggie, I fixed the issue. However, these exact problem kept happening - cPanel somehow changed the PHP version after the accounts were migrated. I fixed them and went to bed.
Then the cPanel cron job ran... Holy f****** hell. We suddenly had a huge list of websites not working anymore (that had been working for hours previously). A lot of these websites had had their .htaccess updated exactly at 04:00 - and the cPanel cron job runs at 03:59. I started changing their PHP versions back and thought everything was fine.
Nope, some websites still didn't work. According to MultiPHP Manager, they were running the correct version of PHP. I checked .htaccess to see if everything was fine - and it was indeed using the correct PHP version.
After troubleshooting for a few minutes, I tried changing to another PHP version and then back to the original - and then it started working all of a sudden.

We've performed many of these migrations before and while there's often small issues, we've never had major issues like these.
Needless to say, our client's clients are not happy, our client is not happy and we're not happy. What was supposed to be a very simple transfer of less than 100 accounts turned out to be a big pain.
We were supposed to migrate multiple servers for the client but these migrations has been put on hold indefinitely.
During the summer, we have to migrate all of our own servers as well, hosting tens of thousands of websites. We cannot afford to have those migrations go wrong.

I've already created ticket #94767181 regarding this issue but would be happy to know if anyone else has experienced something similar. I've never seen cPanel just randomly change PHP version like this. I was the only one with access to the server and I know I didn't change them.
 
Last edited by a moderator:

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
14,307
2,239
363
cPanel Access Level
Root Administrator
Hey there! Thanks for sharing this experience. As the Migrations Coordinator for cPanel, I am especially interested to see what we are able to find in the ticket, as I can't say I've heard of anything similar happening in the past. I'm following along with that on my end now as well.
 
  • Like
Reactions: retechpro

DennisMidjord

Well-Known Member
Sep 27, 2016
352
78
78
Denmark
cPanel Access Level
Root Administrator
Hey @cPRex

Thanks!
Unfortunately, it seems like we're not really progressing very much at this point.
My original post in the ticket explained that something automatically changed the PHP version for websites on the new server. How this is happening, I have no idea.

Your colleagues has been hard at work and I appreciate that.
But, the support ticket has unfortunately gotten pretty long and 5 different people have been covering it - and it doesn't feel like everyone is taking the time to read all the information about the case.
We're now back to square one with this reply:

Hello again,

Looking into this, I'm seeing that the [[SOURCE SERVER]] server has the same PHP version for the main domain and the addon domains which share it's document root as shown below. As such, I would not expect any changes to the PHP version used.
[09:35:57 vps [email protected] ~]cPs# grep "phpversion" /var/cpanel/userdata/username/*|egrep -v "cache|SSL"
/var/cpanel/userdata/username/subdomain.domain.com:phpversion: ea-php70
/var/cpanel/userdata/username/othersubdomain.domain.com:phpversion: ea-php70
/var/cpanel/userdata/username/domain.com:phpversion: ea-php70

However, on the [[MIGRATED SERVER] server, I see there are differing PHP versions. This could result in the PHP version used flipping depending on which version was set most recently.
[09:24:25 [email protected] ~]cPs# grep "phpversion" /var/cpanel/userdata/username/*|egrep -v "cache|SSL"
/var/cpanel/userdata/username/subdomain.domain.com:phpversion: ea-php70
/var/cpanel/userdata/username/othersubdomain.domain.com:phpversion: ea-php70
/var/cpanel/userdata/username/domain.com:phpversion: ea-php80

As Steven mentioned, having the same document root use two different PHP versions is impossible. So the issue here is that cPanel allows domains with a shared document root to be set to different PHP versions. The case Steven opened with our developers (CPANEL-42597) would address this if implemented.

Unfortunately, we do not have a workaround to prevent this situation at this time.

Please let me know if you have any questions.
Somehow the issue is now that cPanel allows clients to set different PHP versions for domains that share the same document root - but as you can see, the domains all had the same PHP version on the source server. Something in cPanel automatically changed the PHP version during or after the transfer, and I have no idea what it is.
On the migration server, I made no changes to anything. I literally booted the server and started migrating accounts (and unsuspended them since they were suspended on the source because of the previous live transfer). Then I went to bed. Nobody has been changing the PHP versions - and that's the issue. They're being changed without anyone doing anything.

It might be my fault since English is not my native language so I might've been unclear when trying to communicate the issue.
If you understand what the issue is (which I really hope you do, because I don't know how I could otherwise explain it ), could you put a note on the ticket explaining what the issue is?
 
Last edited:

DennisMidjord

Well-Known Member
Sep 27, 2016
352
78
78
Denmark
cPanel Access Level
Root Administrator
Hi,

I initially thought that but the domains doesn't have different PHP versions on the source server. I've also provided this info in the ticket (along with images and a login to the source server).
I am completely at a loss regarding this issue - I've never seen this behavior before.

I don't know if it happens during the transfer or during the cPanel nightly maintenance but the PHP versions of some websites suddenly changes.
 

DennisMidjord

Well-Known Member
Sep 27, 2016
352
78
78
Denmark
cPanel Access Level
Root Administrator
Hey @cPRex

I would appreciate if you took a look at the ticket and assigned it to a single staff member. 8 different people have been looking at the case now, and we're just going in circles where I have to explain everything over and over again.
The last guy started transferring accounts to the live server (which is labelled "LIVE server - no changes". Thankfully, it was just a test account so it didn't break anything or overwrite a live account.

I guess this is a case where too many cooks spoil the broth :)

We don't mind having to wait for the same person to get back to us - our client has put the other migrations on hold and we've manually fixed the bad cases on the live server.
 
Last edited:

DennisMidjord

Well-Known Member
Sep 27, 2016
352
78
78
Denmark
cPanel Access Level
Root Administrator
Hi,

Steven was indeed able to clear up much of the confusion and shortly after Christopher managed to replicate the behavior on his test server.
He also created the following article regarding the issue: https://support.cpanel.net/hc/en-us...P-version-set-differently-after-live-transfer

I'm happy to say that based on the information they've provided, I think we can work around this issue - we don't actually need to enable Live Transfer for the transfers, we just didn't disable it because it was a few extra clicks and we still needed to suspend the accounts on the source server.
 
  • Like
Reactions: cPRex