SOLVED cPanel not accessible - services issue Cent OS 7.9

[inspyre210]

Hi,

Hoping someone can help with the following.

It seems after a recent OS update to Cent OS 7.9 I came across the issue of services not starting with a PID error.


System details:


VPS Host running
OpenVZ

Operating System
# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)

Kernel
# uname -r
2.6.32-042stab113.11

Cpanel version
# /usr/local/cpanel/cpanel -V
92.0 (build 7)




The error was very similar to the information provided in this thread.

[UPS-177] Service Failures after CentOS 7.7 update

Hello Everyone, We have received a number of reports regarding service failures with the error "Too many levels of symbolic links". This is an upstream issue involving the latest update to systemd included in CentOS 7.7. Technical Summary The recent CentOS 7.7 update includes an update to...

forums.cpanel.net

I followed the steps for Workaround 2 as recommended in that thread.
After, it seems all services changed with the override workaround say they have started (Except for those that say they are disabled), it seems that resolved the problem for others in that thread.
On this instance however, the services say they are started, but I cannot access the WHM / Cpanel interface on port 2087 from the web browser.

I have also attempted to restart the apache service, the panel services and rebooted the server to ensure I am not missing something.


I ran the below command to see what services were listening on which ports. It seems apache is on 80, 443 and cpsrvd are listening for 2087.
However, if I attempt to perform a telnet to any of those ports, I get a connect failed response.


netstat -tulpn


Active Internet connections (only servers)


Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name


tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 604/pdns_server


tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN 169/exim


tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 838/httpd


tcp 0 0 0.0.0.0:993 0.0.0.0:* LISTEN 594/dovecot


tcp 0 0 0.0.0.0:2082 0.0.0.0:* LISTEN 710/cpsrvd (SSL) -


tcp 0 0 0.0.0.0:2083 0.0.0.0:* LISTEN 710/cpsrvd (SSL) -


tcp 0 0 0.0.0.0:2086 0.0.0.0:* LISTEN 710/cpsrvd (SSL) -


tcp 0 0 0.0.0.0:2087 0.0.0.0:* LISTEN 710/cpsrvd (SSL) -


tcp 0 0 0.0.0.0:587 0.0.0.0:* LISTEN 169/exim


tcp 0 0 127.0.0.1:878 0.0.0.0:* LISTEN 672/sw-cp-server: m


tcp 0 0 127.0.0.1:783 0.0.0.0:* LISTEN 671/spamd-dormant:


tcp 0 0 0.0.0.0:143 0.0.0.0:* LISTEN 594/dovecot


tcp 0 0 0.0.0.0:2095 0.0.0.0:* LISTEN 710/cpsrvd (SSL) -


tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 838/httpd


tcp 0 0 0.0.0.0:2096 0.0.0.0:* LISTEN 710/cpsrvd (SSL) -


tcp 0 0 0.0.0.0:465 0.0.0.0:* LISTEN 169/exim


tcp 0 0 0.0.0.0:53 0.0.0.0:* LISTEN 604/pdns_server


tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 595/sshd


tcp6 0 0 :::25 :::* LISTEN 169/exim


tcp6 0 0 :::443 :::* LISTEN 838/httpd


tcp6 0 0 :::993 :::* LISTEN 594/dovecot


tcp6 0 0 :::3306 :::* LISTEN 708/mysqld


tcp6 0 0 :::587 :::* LISTEN 169/exim


tcp6 0 0 :::143 :::* LISTEN 594/dovecot


tcp6 0 0 :::80 :::* LISTEN 838/httpd


tcp6 0 0 :::465 :::* LISTEN 169/exim


tcp6 0 0 :::53 :::* LISTEN 604/pdns_server


tcp6 0 0 :::22 :::* LISTEN 595/sshd


udp 0 0 0.0.0.0:53 0.0.0.0:* 604/pdns_server


udp6 0 0 :::53 :::* 604/pdns_server





Below are the current failed services.


UNIT LOAD ACTIVE SUB DESCRIPTION


[B]●[/B] [B]cpanel_php_fpm.service [/B] loaded [B]failed failed[/B] FPM service for cPanel Daemons


[B]●[/B] [B]cpdavd.service [/B] loaded [B]failed failed[/B] cPanel dav services


[B]●[/B] [B]cpgreylistd.service [/B] loaded [B]failed failed[/B] cPanel Greylisting Daemon


[B]●[/B] [B]cphulkd.service [/B] loaded [B]failed failed[/B] cPanel brute force detector services


[B]●[/B] [B]mailman.service [/B] loaded [B]failed failed[/B] mailman services


[B]●[/B] [B]systemd-sysctl.service [/B] loaded [B]failed failed[/B] Apply Kernel Variables


[B]●[/B] [B]systemd-tmpfiles-setup.service[/B] loaded [B]failed failed[/B] Create Volatile Files and Directories


[B]●[/B] [B]systemd-vconsole-setup.service[/B] loaded [B]failed failed[/B] Setup Virtual Console


[B]●[/B] [B]tailwatchd.service [/B] loaded [B]failed failed[/B] tailwatchd



The mailman.service was one of the PID's I changed.

List of services that I had to create the override file for:

Modified override
cpanel.service
cpanellogd.service
cphulkd.service
dnsadmin.service
mailman.service
queueprocd.service
spamd.service


If I check the systemctl status for cpanel.service, it says it is running after performing the override fix, however, the service logs at the bottom don't seem to show any new information since January 10th (yesterday, current time on server of writing 11th January 2021 - 15:15)


systemctl status cpanel.service


[B]●[/B] cpanel.service - cPanel services


Loaded: loaded (/etc/systemd/system/cpanel.service; enabled; vendor preset: disabled)


Drop-In: /etc/systemd/system/cpanel.service.d


└─override.conf


Active: [B]active (running)[/B] since Mon 2021-01-11 13:24:37 CET; 1h 48min ago


Main PID: 710 (cpsrvd (SSL) - )


CGroup: /system.slice/cpanel.service


└─710 cpsrvd (SSL) - dormant mode - accepting connections





Jan 10 07:16:50 server systemd[1]: [B]Unit cpanel.service entered failed state.[/B]


Jan 10 07:16:50 server systemd[1]: [B]cpanel.service failed.[/B]


Jan 10 07:16:51 server systemd[1]: Starting cPanel services...


Jan 10 07:16:51 server restartsrv_cpsrvd[13331]: License is valid and has already updated recently.


Jan 10 07:16:51 server systemd[1]: Can't open PID file /var/run/cpsrvd.pid (yet?) after start: Too many levels of symbolic links


Jan 10 07:16:51 server restartsrv_cpsrvd[13331]: Starting PID 13384: /usr/local/cpanel/libexec/cpsrvd-dormant


Jan 10 07:18:21 server systemd[1]: [B]cpanel.service start operation timed out. Terminating.[/B]


Jan 10 07:18:21 server systemd[1]: [B]Failed to start cPanel services.[/B]


Jan 10 07:18:21 server systemd[1]: [B]Unit cpanel.service entered failed state.[/B]


Jan 10 07:18:21 server systemd[1]: [B]cpanel.service failed.[/B]



I feel like I am extremely close to getting the server up and running again, but cannot figure out why I cannot access any of the ports or cannot connect via the web.

Any help would be greatly appreciated or advice on the next step to try / logs to check.
Happy to provide any further information on request. Not too sure what else may be needed.

Thanks.

 

[cPRex]

Hey there! At this point it almost sounds like a firewall issue of some sort since you're not even able to telnet, but it might be best to get a ticket submitted with our team so we can check things on our side. Please post the ticket number here so I can follow along and keep the community updated with our findings.

 

[inspyre210]

Thanks cPRex.

I have created a ticket with ID #94118281

I did wonder if it was a firewall issue, but I am just at the stage of complete loss what to try next.
The system was working over the weekend before it was rebooted by the host. At that point I had the issue with the PID service error.

tailwatchd doesn't seem to want to startup, so not sure if there is some other services causing issues.

 

[cPRex]

Thanks so much - due to the nature of this issue I've given this ticket some priority in our queue, so you should be getting a reply soon.

 

[inspyre210]

Hi,

Just to update on the issue. It looks like the fix in the original post worked and the services were functional.
So far, all looks OK with cPanel and WHM again.

The issue however was that the server had enabled firewalld and had then blocked all ports except ssh. This was part of a fail2ban install which I hadn't realised had turned itself on during the issue.

After testing by stopping both services, the ports were accessible again.


systemctl stop fail2ban.service
systemctl stop firewalld.service
systemctl disable fail2ban.service
systemctl disable firewalld.service



Thanks to the assistance of the cPanel team, it managed to point me in the right direction and resolve.

Thank you all.

 

[cPRex]

I'm glad we were able to help track that down!