In Progress CPANEL-40433 - WHM Backup Configuration page loading slowly; previously it was timing out

[jcalvert]

Hi,

CentOS v7.9.2009
WHM/cPanel v100.0.12

The situation has improved from "Backup Configuration" not loading at all (timing out), to the present situation which is, sometimes it loads right away, other times it takes about 40 seconds to load. I haven't experienced a delay loading any other WHM pages.

The timeout seems to not be happening anymore, since I made some modifications to our custom iptables firewall.

But this intermittent delay loading the page is now happening.

It seems there is some resource that "Backup Configuration" needs, which is involved in causing this delay.

Does anyone know what this is?

thanks,
JC

 

[cPRex]

Hey there! I think this would be related to this recent case we just opened, as the timing you mentioned of 30 seconds or so sounds right:

https://support.cpanel.net/hc/en-us/articles/5418953288599

 

[cPanelAdamF]

Hi,

CentOS v7.9.2009
WHM/cPanel v100.0.12

The situation has improved from "Backup Configuration" not loading at all (timing out), to the present situation which is, sometimes it loads right away, other times it takes about 40 seconds to load. I haven't experienced a delay loading any other WHM pages.

The timeout seems to not be happening anymore, since I made some modifications to our custom iptables firewall.

But this intermittent delay loading the page is now happening.

It seems there is some resource that "Backup Configuration" needs, which is involved in causing this delay.

Does anyone know what this is?

thanks,
JC

In playing with this, I do see opportunity for us to load things in a more optimized way (I used browser network throttling to do this). Can you help us understand what kind of network bandwidth you are working with so that we can test our fixes using those settings?

 

[jcalvert]

I tested again last night on a very fast cable broadband connection, and got the timeout again after about 1 min. 40 sec. Timeout while trying to load the "Backup Configuration" page. Tested at ~1:40am Pacific. I don't think this has anything to do with the network.

Tested again at 1:44am Pacific... this time page loads immediately.

This is looking a lot like a cPanel resource is sometimes overloaded, or perhaps there are one or more cPanel IP addresses that I don't know about, which are currently not open in the firewall for http or https. I can imagine possibly that WHM is trying to reach some remote cPanel service, at an IP address, and there are a set of mirror addresses, and none of them are available, hence the timeout. At random times, the top ones on the list are available, and the page loads immediately.

Do I need to open an account to view this?
https://support.cpanel.net/hc/en-us/articles/5418953288599

 

[jcalvert]

Hey there! I think this would be related to this recent case we just opened, as the timing you mentioned of 30 seconds or so sounds right:

https://support.cpanel.net/hc/en-us/articles/5418953288599

I doubt it is related. What I'm seeing is specific to the loading of the "Backup Configuration" page, and there is either no delay, a delay of some amount (e.g. ~40 sec.), or a timeout and the page doesn't load at all (after about 1 min. 40 sec.).

The timeout is apparently only happening when our custom iptables firewall is operating, leading me to suspect that there is an inbound http:// or https:// request from an IP that we have not whitelisted, or there is some other port needed that is currently blocked by the firewall. Keep in mind that we are using Cloudflare, and our firewall whitelists all Cloudflare IPs (for inbound http:// and https://), in addition to 17 cPanel IP addresses. Also please note that I have updated our firewall according to this article:

https://docs.cpanel.net/knowledge-base/general-systems-administration/how-to-configure-your-firewall-for-cpanel-services/98

Can someone please tell me how WHM could possibly get a timeout trying to load the "Backup Configuration"? What resource is it trying to get to and can't, or what would cause it to time out with no error message at all?

thanks,
JC

 

[cPRex]

There isn't a good reason that should be happening, and I don't see that on my servers at this time.

Could you make a ticket with our team so we can investigate this directly on the system?

 

[jcalvert]

@cPRex If we turn off our custom firewall, there are no problems.

With our firewall enabled, "Backup Configuration" sometimes (as right now) times out loading after 1 min. 40 sec. This happens consistently. When my testing produces this result, I also see a delay of 20 sec. loading the home page of WHM. Loading other pages, such as "List Accounts" takes 1 second.

I have opened the required ports (INPUT, OUTPUT) for cPanel in our firewall, and whitelisted 17 cPanel IP addresses for inbound http:// (80) and https:// (443).

Can you tell me what ports, specifically, need to be open for "Backup Configuration" to load normally? (i.e. other than 80, 443)

If that isn't the issue, then which service is Backup Configuration trying to access which would require a port 80 (http) or port 443 (https) connection, either inbound or outbound?

Note also that WHM doesn't report in the error log why it timed out.

thanks,
JC

 

[cPRex]

There aren't any special ports or connections happening with the Backup Configuration page. Do you have everything open from this page? How to Configure Your Firewall for cPanel & WHM Services | cPanel & WHM Documentation

 

[jcalvert]

@cPRex I have just discovered the problem... our firewall is restricting access for FTP to specific IPs and IP ranges. When I remove the FTP rules, WHM is working normally. Specifically, it's the "passive mode" FTP port range that needs to be open, 49152:65534.

Can you tell me what cPanel IPs need to be whitelisted for this port range, 49152:65534? Is cPanel using one or more ports in the range for something other than FTP?

Note that the "How to Configure..." page doesn't say anything about which cPanel IP addresses need to be whitelisted for inbound requests on TCP ports. We currently have 17 cPanel IPs whitelisted for inbound http and https requests (80, 443). Can you provide me with your official list of which IPs need to be whitelisted?

thanks,
JC

 

[cPRex]

I wouldn't expect any passive ports need to be open for the page itself to load. Those ports are used by FTP clients to make connections, and not something that is needed for the WHM interface to function.

 

[jcalvert]

I wouldn't expect any passive ports need to be open for the page itself to load. Those ports are used by FTP clients to make connections, and not something that is needed for the WHM interface to function.

The problems I'm seeing with WHM are intermittent. Sometimes I see delays, e.g. it takes ~20 sec. to reach home page, and the issue of timeout loading "Backup Configuration" page. These conditions happen together. When this occurs, and I open the FTP passive port range, 49152:65534, WHM immediately works normally again.

At the moment, I'm not seeing the WHM problems. I will keep trying to test this.

 

[jcalvert]

@cPRex I just finished another round of testing on this issue. This time I monitored the TCP ports and found something very interesting. I should say first that I also verified again that turning off our firewall immediately eliminated the delays and timeout in WHM.

What I found is that WHM is, in fact, using ports that are within the FTP passive port range, and it's not using those for FTP – it's using them for https (443).

In tonight's testing, after I clicked on "Backup Configuration", I then monitored TCP ports from the shell...

This is the relevant output from the command, /usr/sbin/ss -tpn (show here vertically for clarity):

State: SYN-SENT
Recv-Q: 0
Send-Q: 1
Local Address:Port: 67.231.17.190:51196
Peer Address:Port: 104.18.17.164:443
users ("whostmgr - back",pid=21791,fd=4)

67.231.17.190 is the local host, vps.wwk.com.

104.18.17.164 is cpanel.net's IP address on Cloudflare (also 104.18.16.164). (See https://intodns.com/cpanel.net, or from the shell, %host www.cpanel.net)

In the /usr/sbin/ss output above, we see that the local host has TCP port 51196 open for communication with the peer, cpanel.net. The remote peer is using port 443 (https).

The local process using this connection is whostmgr, which is WHM. Another process I've seen using ports in this range for https is queueprocd.

For now I will solve this problem by opening 49152:65534 for output from localhost, and for input from 104.18.17.164, 104.18.16.164, and localhost.

Tested and is working.

 

[cPRex]

Do you happen to have a remote FTP server configured on this particular WHM instance?

 

[jcalvert]

Do you happen to have a remote FTP server configured on this particular WHM instance?

We are using WHM to manage the FTP service on our VPS. We are using Pure-FTPd. My client is connecting via FTP to the VPS. We are running a custom firewall that whitelists my client's IP addresses for FTP and SSH.

 

[cPRex]

That's definitely odd as I just don't have a good explanation. Could you submit a ticket to our team so we can check this out directly?

 

[jcalvert]

@cPRex I recommend your team look at the code that loads the Backup Configuration page and confirm what I have observed, which is that WHM is communicating with cpanel.net using https, with the remote port being 443, and the local port being in the FTP passive mode range. You should be able to use CSF to block the passive mode range, which should allow you to see the delays and timeout in WHM.

 

[cPRex]

By default, when CSF is installed it does not include the passive port ranges. I did the following for my test:

-created a fresh cPanel 102 install on AlamLinux 8
-made sure an FTP server was installed in WHM
-installed CSF

At this point I checked /etc/csf/csf.conf and found this configuration, showing that the passive ports aren't allowed by default:

# Allow incoming TCP ports
TCP_IN = "20,21,22,25,53,80,110,143,443,465,587,993,995,2077,2078,2079,2080,2082,2083,2086,2087,2095,2096,8443"

# Allow outgoing TCP ports
TCP_OUT = "20,21,22,25,37,43,53,80,110,113,443,587,873,993,995,2086,2087,2089,2703"

This really does seem to be a unique situation to your server environment. If the page worked as you described, every server that doesn't have the passive FTP ports enabled would be experiencing this issue, and so far this is the only report we have of this behavior.

 

[jcalvert]

@cPRex Thanks for testing.

I see some inconsistency with your CSF allowed ports and what I have programmed in our firewall.

First, why is port 8443 open for input? Isn't this used by Plesk? Port 8443 is not listed in the knowledge base article. It turns out we have been getting some timeouts when the WHM/cPanel auto updates run, and one of those timeouts happens while waiting to download from https://wp-toolkit.plesk.com/cPanel/wp-toolkit. I thought it was odd that WHM is trying to access plesk.com. Another timeout is, "The /usr/local/cpanel/scripts/updatesigningkey command (process 823) ended prematurely... reached the timeout of 3,600 seconds."

Next, I see that you have port 20 (FTP data) open for output, whereas we only have it open for input. Does WHM need port 20 to be open to establish outgoing FTP connections? Note that my client is regularly FTP'ing into the server with no problems.

Next, I see that you have port 22 (SSH) open for output. The knowledge-base article says it only needs to be open for input. Does WHM need port 22 to be open to establish outgoing SSH connections?

Next, I see that you have 2086 and 2087 open for output, whereas we have them open for input only, as per the knowledge base article.

I wrote:

For now I will solve this problem by opening 49152:65534 for output from localhost, and for input from 104.18.17.164, 104.18.16.164, and localhost.

Tested again and still working OK.

You wrote:

This really does seem to be a unique situation to your server environment. If the page worked as you described, every server that doesn't have the passive FTP ports enabled would be experiencing this issue, and so far this is the only report we have of this behavior.

I think the situation is unique because we are running a custom firewall, not CSF or whatever else WHM/cPanel offers. Your test shows that the CSF defaults are different than the official knowledge base article for setting up ports for WHM/cPanel in a firewall, so this may help explain why nobody else is seeing what we've seen.

 

[cPRex]

Right - I just did the default CSF install with no additional configuration necessary. The point of that exercise was to show that the passive ports aren't included by default, and don't need to be opened for the WHM interface to work properly. I do agree the issue is with the custom firewall.

 

[jcalvert]

@cPRex I asked several basic questions in my last post, and you appear unable to answer any of them. We are considering switching to Plesk.