Please see this post for the official cPanel response with available solutions.
I was prompted for a server graceful reboot for software updates and server did not come back.
Edit: it came up after host intervention after 15 minutes but with some notes regarding correcting software config. Cpanel Ticket id for this: 10885171
If you haven't done so already, I'd strongly recommend checking that you have working IPMI / KVM / Console access available to you. In a situation like this, it's the one (and only) hope available to revive a server that's inaccessible via normal means like SSH, WHM, etc. Personally speaking, I like to audit such out-of-band access systems every month or so, just to ensure they still work properly.
[Removed Links - these threads were merged into this one]
Can't restart the server after applying update. I am using the kernel care free simlink protection patch on this server and when I click on restart I see the message:
kcarectl --info reporting unknown patch type: free:15xxxxxx
at /usr/local/cpanel/Cpanel/KernelCare.pm line 80.
Cpanel::KernelCare::get_kernelcare_state() called at /usr/local/cpanel/Cpanel/KernelCare/Suggest.pm line 65
Cpanel::KernelCare::Suggest::get_suggestion() called at whostmgr/bin/whostmgr.pl line 5204
main::graceful_reboot_landing("graceful_reboot_landing") called at /usr/local/cpanel/Whostmgr/Dispatch.pm line 259
Whostmgr::Dispatch::_do_call("graceful_reboot_landing", HASH(0x2b9d000), HASH(0x2b9d988)) called at /usr/local/cpanel/Whostmgr/Dispatch.pm line 157
Whostmgr::Dispatch::dispatch("graceful_reboot_landing", 1, ARRAY(0x2b9cf88), HASH(0x2b9d988)) called at whostmgr/bin/whostmgr.pl line 393
I performed a graceful restart and the server failed to reload.
Then I restarted server at host - server won't restart.
In host console I get the message: !!!!!failed to load selinux policy
The solution is to modify the Root config in the GRUB boot (you need to be administrator)
I did this at my server host console (NOTE: use at your own risk. This worked for me (CentOS KVM) - it might be a different process for you depending on your machine / OS / Host. Please check with cPanel your host if you are not 100%
1. Access host console and click the send CTRL+ALT+DEL button on the top right, or click [RESTART] to restart the server. As soon as the boot process starts, press ESC to bring up the GRUB boot prompt. You may need to stop the server first, then restart it to reach the GRUB boot prompt.
2. You will see a GRUB boot prompt - press E to edit the first boot option. (If you do not see the GRUB prompt, you may need to press any key to bring it up before the machine boots)
3. Find the kernel line (it starts with "linux16"), REPLACE ro with rw init=/sysroot/bin/sh and leave the rest of the line unchanged.
4. Press CTRL+X or F10 to boot into single user mode.
5. Access the system with the command: chroot /sysroot
6. Edit the file with vi: eg;
#vi /etc/selinux/config
and change the line "enforcing" to "disabled"
7. reboot or restart the machine at host control panel.
I hope this helps someone !
There is also a tutorial here asafshoval.wordpress.com/2014/11/18/overcome-fail-to-load-selinux-policy-freezing-error-message-while-booting-linux (I didn't use it, but it is similar instruction)
Indeed. Thanks for the instructions on that thread. I think I will not do anything for now, I will just wait for Cpanel techs to check out my ticket and take the appropriate action.
I can confirm this is an active problem as well, atleast for servers running on Vultr OS Templates.
For what ever reason, when system updates are automatically updated and the machine is rebooted as needed, selinux somehow becomes enabled which is definitely not supposed to happen being a cpanel server.
Those are the same instructions vultr support confirmed with my when I brought this to their attention after having 15 servers crap out at once with them. I was able to just disable selinux and it resolved the issue.
Ill be shooting in a ticket as well with access to a now fixed machine for further investigation if needed.
EDIT: Ticket submitted with access to server that was affected: 10888269
We've received multiple reports of service failures and failed reboot attempts after a system is upgraded to CentOS version 7.6.
The upgrade to CentOS 7.6 can occur automatically as part of nightly cPanel & WHM updates on CentOS 7 systems with Operating System Package Updates set to Automatic in WHM Home » Server Configuration » Update Preferences, or when a system administrator manually runs the yum update command.
Upon investigating the issue, it appears the issue is isolated to systems missing the /etc/selinux/config file that's part of the selinux-policy RPM. This RPM and file exist by default on stock CentOS 7 installations, however it looks like some providers intentionally exclude the /etc/selinux/config file in the CentOS and cPanel & WHM images provided to their customers.
Furthermore, an updated Bind package is included as part of the CentOS 7.6 upgrade which includes a hard dependency for the selinux-policy RPM. This leads to the automatic installation of the selinux-policy RPM on systems where it was previously uninstalled. The selinux-policy RPM includes the following post-script:
Code:
# rpm -q --scripts selinux-policy-3.13.1-229.el7_6.6.noarch
postinstall scriptlet (using /bin/sh):
if [ ! -s /etc/selinux/config ]; then
#
# New install so we will default to targeted policy
#
echo "
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of three values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
" > /etc/selinux/config
This post script automatically creates the /etc/selinux/config file with the SELINUX=enforcing value enabled if the file did not previously exist. Once enforcing mode is enabled, issues such as service failures and failed reboot attempts will occur.
Available solutions:
Modify the following value in the /etc/selinux/config file:
Code:
SELINUX=enforcing
To:
Code:
SELINUX=disabled
Once you've modified the /etc/selinux/config file, reboot the system to enable the change.
You will need to connect to your server's console using the method documented by your server provider. If console access is not available, contact your provider and ask them to review and perform the workaround steps provided by @WorkinOnIt below:
1. Access host console and click the send CTRL+ALT+DEL button on the top right, or click [RESTART] to restart the server. As soon as the boot process starts, press ESC to bring up the GRUB boot prompt. You may need to stop the server first, then restart it to reach the GRUB boot prompt.
2. You will see a GRUB boot prompt - press E to edit the first boot option. (If you do not see the GRUB prompt, you may need to press any key to bring it up before the machine boots)
3. Find the kernel line (it starts with "linux16"), REPLACE ro with rw init=/sysroot/bin/sh and leave the rest of the line unchanged.
4. Press CTRL+X or F10 to boot into single user mode.
5. Access the system with the command: chroot /sysroot
6. Edit the file with vi: eg;
#vi /etc/selinux/config
and change the line "enforcing" to "disabled"
7. reboot or restart the machine at host control panel.
The following steps explain how it's possible for reboot attempts to take longer than normal on affected systems:
The /etc/selinux/config file does not exist on the system (SELinux defaults to disabled in this state).
The server was rebooted at least once in the past (with /etc/selinux/config missing).
As part of the initial reboot, the /.autorelabel touch file is created during the shutdown phase.
With SELinux disabled, the reboot process finishes normally and the relabeling process is not initiated.
Later, the /etc/selinux/config is created and SELinux is enabled (see the beginning of this post for how this can happen).
Upon the next reboot, the /.autorelabel touch file initiates the relabeling process this time because it detects SELinux as enabled.
The relabeling process can take anywhere from a few minutes to several hours depending on the server hardware and usage.
Once the relabeling process is complete, the server will automatically reboot again.
If you have not yet upgraded your system to CentOS 7.6, and the /etc/selinux/config file does not exist on your system, ensure that it's created with the SELINUX=disabled value before performing the upgrade as seen in the output below:
Code:
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
I'm on OVH and have just had this happen.. naturally its exactly 20min before phone support closed and I can not get through!
Also after booting to a rescue image and looking for the
SELINUX=enforcing flag only to discover it was already set to
SELINUX=disabled I'm totally lost as to what to do and now looking at 12 hours until phone support comes online - I have emailed them so hopefully they can work out what has happened and get it back online.
WHM/cPanel auto-update is enabled.
I've received message that after a new kernel update the server needs to reboot. I've logged into WHM and clicked the "You must reboot the server to apply software updates."
Then the server never comes back.
I'm on OVH as a physical machine.
They have IPMI java console. I access the machine through this and I get
grub>
?
This is not my area of expertise. But after I find the link above and boot into a rescue image to get disk access, I check the raid is OK and SSH to look for the
SELINUX=enforcing
line in /etc/selinux/config and it was already set to 'disabled' now I am at a loss as to how to get the server back up. I have ticketed OVH as it is just after hours for phone support... I'm hoping now they can resolve it an understand my notes (I'm also hoping they will not discard my request if they see me connected by the IPMI).
I'm on OVH and have just had this happen.. naturally its exactly 20min before phone support closed and I can not get through!
Also after booting to a rescue image and looking for the
SELINUX=enforcing flag only to discover it was already set to
SELINUX=disabled I'm totally lost as to what to do and now looking at 12 hours until phone support comes online - I have emailed them so hopefully they can work out what has happened and get it back online.
When booted into the rescue image, did you go to /etc/selinux/config on SSH, or did you mount your partition on /mnt and changed it there? the /etc/selinux/config file in rescue, is the selinux config for the rescue image, and not your server.
@Webdew@LucasRolff is correct, you need to access the GRUB menu and modify the file there if the server is not booting. The instructions I gave in post #5 above may assist you if you are confident and still offline - could be worth a go!
Disable SELinux
If your server runs an operating system from a source other than the cPanel & WHM installer, you must disable SELinux to make your system compatible with cPanel & WHM.
To disable SELinux security features, use one of the following methods:
Use the graphical interface to disable SELinux while you configure your operating system, and then reboot the server.
Edit the /etc/selinux/config file to set the SELINUX parameter to disabled, and then reboot the server.
The contents of the /etc/selinux/config file should resemble the following example...
Code:
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
# targeted - Only targeted network daemons are protected.
# strict - Full SELinux protection.
SELINUXTYPE=targeted
Warning:
To run cPanel & WHM on your server, SELinux must remain disabled.
SELinux in enforcing mode does not allow cPanel & WHM to function properly. For more information about SELinux modes, read the SELinux Mode documentation.
While cPanel & WHM can function with SELinux in permissive mode, we recommend that you do not use it. Permissive mode generates a large number of log entries.
To check the status of SELinux on your server, run the sestatus command.
Do not transfer the SELinux configuration file between computers. It may destroy the file's integrity.
I updated my previous post with some additional information for hosting providers and system administrators explaining why rebooting a system affected by this issue can take longer than normal if SELinux is not disabled prior to the reboot. I've also quoted this information below:
The following steps explain how it's possible for reboot attempts to take longer than normal on affected systems:
The /etc/selinux/config file does not exist on the system (SELinux defaults to disabled in this state).
The server was rebooted at least once in the past (with /etc/selinux/config missing).
As part of the initial reboot, the /.autorelabel touch file is created during the shutdown phase.
With SELinux disabled, the reboot process finishes normally and the relabeling process is not initiated.
Later, the /etc/selinux/config is created and SELinux is enabled (see the beginning of this post for how this can happen).
Upon the next reboot, the /.autorelabel touch file initiates the relabeling process this time because it detects SELinux as enabled.
The relabeling process can take anywhere from a few minutes to several hours depending on the server hardware and usage.
Once the relabeling process is complete, the server will automatically reboot again.
I don't have an OpenVZ system available for immediate testing, but as I understand the /etc/selinux/config file exists by default with any stock CentOS 7 installation (with the SELINUX=disabled value). Are you using a particular image or template to provision the servers?
I just happen to have 1 OpenVZ CentOS 7 server. It does not have an /etc/selinux/config file and yum check-update does not list an selinux-policy package to be installed (I haven't yet updated to CentOS 7.6).
I'm not sure how CentOS 7 was installed on this particular OpenVZ VPS. Perhaps others might chime in?
I suspect I can add an /etc/selinux/config with SELINUX=disabled just to be safe, but I don't think it would make any difference.
The issue appears to only pop up on systems where selinux-policy is updated.
These are the contents of my selinux config, what should I do?
Code:
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=permissive
# SELINUXTYPE= can take one of three two values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
This was the host reply (whom I pointed out to this thread):
Your server got updated and with the new update is not able to boot correctly. Since the monitoring was enabled our system detected that and putted an intervention to check why the server was offline. The message you have received is the result of that intervention advising you to fix the software problems preventing your server from correctly booting into the OS.
I checked the link that you have send to me and it refers to a configuration file that does not exist. You can reboot the server in rescue mode and create the missing configuration in order for the server to boot.
From what I saw the server now is online and you are using one of the kernels of OVH. I suggest in this case you are able to log in to your server and start the configuration of the kernel in order to boot from your kernel and have the services running.
