Error in MySQL upgrade from 5.6 to 5.7

[ejde]

I'm seeing this error after the upgrade in WHM apparently completed, but now MySQL won't run:

(XID h2bb74) The system failed to connect to the “MySQL” database “mysql” because of an error: CR_CONNECTION_ERROR (Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)) at /usr/local/cpanel/Whostmgr/Mysql/Upgrade.pm line 234.
   Whostmgr::Mysql::Upgrade::__ANON__(Cpanel::Exception::Database::ConnectError=HASH(0x3e98a28)) called at /usr/local/cpanel/3rdparty

Also, when attempting to restart mysqld, I get the following failure:

Waiting for “mysqld” to start ……[2018-08-27 22:32:27 -0400] warn [restartsrv_mysql] The system encountered an error while starting the “mysqld” service with the command “/etc/init.d/mysqld start”: The subprocess reported error number 1 when it ended. at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 1403. Cpanel::ServiceManager::Base::warn(Cpanel::ServiceManager::Services::Mysql=HASH(0x21a1390), "The system encountered an error while starting the \x{e2}\x{80}\x{9c}mysqld\x{e2}"...) called at /usr/local/cpanel/Cpanel/ServiceManager/Manager/Initd.pm line 103 Cpanel::ServiceManager::Manager::Initd::start(Cpanel::ServiceManager::Manager::Initd=HASH(0x2513498), Cpanel::ServiceManager::Services::Mysql=HASH(0x21a1390)) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 848 Cpanel::ServiceManager::Base::start(Cpanel::ServiceManager::Services::Mysql=HASH(0x21a1390)) called at /usr/local/cpanel/Cpanel/ServiceManager/Services/Mysql.pm line 60 Cpanel::ServiceManager::Services::Mysql::start(Cpanel::ServiceManager::Services::Mysql=HASH(0x21a1390)) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 707 Cpanel::ServiceManager::Base::restart(Cpanel::ServiceManager::Services::Mysql=HASH(0x21a1390), "graceful", undef) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 540 Cpanel::ServiceManager::Base::run_from_argv(Cpanel::ServiceManager::Services::Mysql=HASH(0x21a1390), "--verbose", "--html", "--wait", "--verbose") called at bin/restartsrv_base.pl line 96 scripts::restartsrv_base::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 99 eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 90 Try::Tiny::try(CODE(0x21a1870), Try::Tiny::Catch=REF(0xb3ea58)) called at bin/restartsrv_base.pl line 130 scripts::restartsrv_base::run("--html", "--wait", "--verbose") called at bin/restartsrv_base.pl line 33 …[2018-08-27 22:32:33 -0400] warn [restartsrv_mysql] The system encountered an error while starting the “mysqld” service with the command “/etc/init.d/mysqld start”: The subprocess reported error number 1 when it ended. at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 1403. Cpanel::ServiceManager::Base::warn(Cpanel::ServiceManager::Services::Mysql=HASH(0x21a1390), "The system encountered an error while starting the \x{e2}\x{80}\x{9c}mysqld\x{e2}"...) called at /usr/local/cpanel/Cpanel/ServiceManager/Manager/Initd.pm line 103 Cpanel::ServiceManager::Manager::Initd::start(Cpanel::ServiceManager::Manager::Initd=HASH(0x2513498), Cpanel::ServiceManager::Services::Mysql=HASH(0x21a1390)) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 848 Cpanel::ServiceManager::Base::start(Cpanel::ServiceManager::Services::Mysql=HASH(0x21a1390)) called at /usr/local/cpanel/Cpanel/ServiceManager/Services/Mysql.pm line 60 Cpanel::ServiceManager::Services::Mysql::start(Cpanel::ServiceManager::Services::Mysql=HASH(0x21a1390)) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 707 Cpanel::ServiceManager::Base::restart(Cpanel::ServiceManager::Services::Mysql=HASH(0x21a1390), "graceful", undef) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 540 Cpanel::ServiceManager::Base::run_from_argv(Cpanel::ServiceManager::Services::Mysql=HASH(0x21a1390), "--verbose", "--html", "--wait", "--verbose") called at bin/restartsrv_base.pl line 96 scripts::restartsrv_base::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 99 eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 90 Try::Tiny::try(CODE(0x21a1870), Try::Tiny::Catch=REF(0xb3ea58)) called at bin/restartsrv_base.pl line 130 scripts::restartsrv_base::run("--html", "--wait", "--verbose") called at bin/restartsrv_base.pl line 33 …[2018-08-27 22:32:39 -0400] warn [restartsrv_mysql] The system encountered an error while starting the “mysqld” service with the command “/etc/init.d/mysqld start”: The subprocess reported error number 1 when it ended. at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 1403. Cpanel::ServiceManager::Base::warn(Cpanel::ServiceManager::Services::Mysql=HASH(0x21a1390), "The system encountered an error while starting the \x{e2}\x{80}\x{9c}mysqld\x{e2}"...) called at /usr/local/cpanel/Cpanel/ServiceManager/Manager/Initd.pm line 103 Cpanel::ServiceManager::Manager::Initd::start(Cpanel::ServiceManager::Manager::Initd=HASH(0x2513498), Cpanel::ServiceManager::Services::Mysql=HASH(0x21a1390)) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 848 Cpanel::ServiceManager::Base::start(Cpanel::ServiceManager::Services::Mysql=HASH(0x21a1390)) called at /usr/local/cpanel/Cpanel/ServiceManager/Services/Mysql.pm line 60 Cpanel::ServiceManager::Services::Mysql::start(Cpanel::ServiceManager::Services::Mysql=HASH(0x21a1390)) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 707 Cpanel::ServiceManager::Base::restart(Cpanel::ServiceManager::Services::Mysql=HASH(0x21a1390), "graceful", undef) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 540 Cpanel::ServiceManager::Base::run_from_argv(Cpanel::ServiceManager::Services::Mysql=HASH(0x21a1390), "--verbose", "--html", "--wait", "--verbose") called at bin/restartsrv_base.pl line 96 scripts::restartsrv_base::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 99 eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 90 Try::Tiny::try(CODE(0x21a1870), Try::Tiny::Catch=REF(0xb3ea58)) called at bin/restartsrv_base.pl line 130 scripts::restartsrv_base::run("--html", "--wait", "--verbose") called at bin/restartsrv_base.pl line 33 …failed.

 

[24x7server]

Hi,

Execute the below command in one SSH session:
# tail -f /var/lib/mysql/<hostname>.err

Open second SSH session and execute below command:
# service mysqld restart
OR
# service mysql restart

Check what error is occuring and paste the error here.

 

[ejde]

Thank you for the reply. Here is the output from the first terminal window after issuing both commands as instructed:

2018-08-28T04:50:03.329696Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2018-08-28T04:50:03.329740Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2018-08-28T04:50:03.329758Z 0 [ERROR] Failed to initialize builtin plugins.
2018-08-28T04:50:03.329763Z 0 [ERROR] Aborting

2018-08-28T04:50:03.329774Z 0 [Note] Binlog end
2018-08-28T04:50:03.329820Z 0 [Note] Shutting down plugin 'MyISAM'
2018-08-28T04:50:03.329837Z 0 [Note] Shutting down plugin 'CSV'
2018-08-28T04:50:03.330240Z 0 [Note] /usr/sbin/mysqld: Shutdown complete

2018-08-28T11:31:34.707254Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2018-08-28T11:31:34.742280Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.23) starting as process 1526 ...
2018-08-28T11:31:34.829121Z 0 [Note] InnoDB: PUNCH HOLE support available
2018-08-28T11:31:34.829151Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-08-28T11:31:34.829165Z 0 [Note] InnoDB: Uses event mutexes
2018-08-28T11:31:34.829177Z 0 [Note] InnoDB: GCC builtin __sync_synchronize() is used for memory barrier
2018-08-28T11:31:34.829188Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.3
2018-08-28T11:31:34.829199Z 0 [Note] InnoDB: Using Linux native AIO
2018-08-28T11:31:34.829390Z 0 [Note] InnoDB: Number of pools: 1
2018-08-28T11:31:34.829489Z 0 [Note] InnoDB: Using CPU crc32 instructions
2018-08-28T11:31:34.830733Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2018-08-28T11:31:34.836990Z 0 [Note] InnoDB: Completed initialization of buffer pool
2018-08-28T11:31:34.838782Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page ofsetpriority().
2018-08-28T11:31:34.936705Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2018-08-28T11:31:35.061035Z 0 [ERROR] InnoDB: Upgrade after a crash is not supported. This redo log was created before MySQL 5.7.9, and we did not find avalid checkpoint. Please follow the instructions at MySQL :: MySQL 5.7 Reference Manual :: 2.11.1 Upgrading MySQL
2018-08-28T11:31:35.061073Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2018-08-28T11:31:35.461390Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2018-08-28T11:31:35.461424Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2018-08-28T11:31:35.461435Z 0 [ERROR] Failed to initialize builtin plugins.
2018-08-28T11:31:35.461443Z 0 [ERROR] Aborting

2018-08-28T11:31:35.461624Z 0 [Note] Binlog end
2018-08-28T11:31:35.461672Z 0 [Note] Shutting down plugin 'MyISAM'
2018-08-28T11:31:35.462057Z 0 [Note] Shutting down plugin 'CSV'
2018-08-28T11:31:35.467876Z 0 [Note] /usr/sbin/mysqld: Shutdown complete

 

[cPanelMichael]

Hello @ejde,

It looks like InnoDB is corrupted on the server. We provide a guide for you to follow in order to attempt to repair InnoDB at:

Tutorial - InnoDB Corruption Repair Guide

Thank you.