MySQL Restarting - Note -usr-sbin-mysqld: Normal shutdown

[kieran_kkcs]

Hi, my first post here! I've been using cPanel and WHM to manage a MySQL 5.5 database which I've just upgraded to 5.6 and which has unearthed a myriad of issues, mostly memory related. Anyway, they're sorted I think but it appears now that MySQL is shutting itself down and restarting in the mdidle of the night. In itself, this isn't an issue except tha tI have scripts running every 15 minutes, which email me in the event of errors and I can get 1000+ emails in the time the MySQL server is down.

My question is, why is it restarting? Can I find that out in a log somewhere? If its not necessary for it to restart, can I turn that off? If its advisable that its left restarting etc. is there any advice on how to schedule my scripts so that perhaps they dont run between, say 01:15 and 02:00?

Thanks in advance

Kieran

 

[Babene7]

Hello,

For how long does MySQL stay down?
Does it happen at the same time every night?
Do you have any crons running at the same time that the issue occurs?

Normally, MySQL doesn't restart on its own, usually it's something else that triggers the restart, for example, cPanel's monitor function (WHM > Service Manager) if the MySQL service is detected to be down.

If the issue happens always at the same time I bet that it's triggered by a cron.

Regarding logs, you can check MySQL's log if you have it enabled. By default the path is /var/lib/mysql/yourhostname.err

 

[kieran_kkcs]

Hi, I have cron jobs running, but they're php scripts.
It looks like it shutdown and restarted immediately. I've attached two files, one is the section from mysqld.log and the other is the cron jobs.
It happens at the same time, but not every night.
Thanks
Kieran

 

[cPanelWilliam]

Hello!

It looks like it shutdown and restarted immediately. I've attached two files, one is the section from mysqld.log and the other is the cron jobs.

Does the time of the MySQL restart align with any of the cron jobs running on the server? I checked your reply but didn't see any files attached for review.

You can check /var/log/cron to compare the timestamps of crons that were executed around the time MySQL was last restarted. If you notice that the mysqluserstore cron is restarting MySQL, then the following article would guide you on resolving the issue:

How to resolve "[mysqluserstore] Restarting MySQL due to my.cnf modifications."

 

[kieran_kkcs]

Hi, apologies, the files didn't attach because they were in the wrong format and I didn't spot it. Should be there now.

 

[cPRex]

Was there a minor update to MySQL? You can run "dnf history" to see if a MySQL package was updated, as shortly after 1AM is when I would expect that to happen.

 

[kieran_kkcs]

Hi, thanks for that. I couldn't find the dnf command, I don't think I have the right version of centos for that. It restarted last night again at 01:28. In the cron logs there were nothing that was running at that time. I grepped Out Of Memory in /var/log, and the last entry was at the end of June when I did run out of RAM. However, when I do sar -r, it says the average memory usage is 97% (screenshot attached) but when I look at the process manager in WHM there doesnt appear to be anything that's hogging that amount of memory. It is a pain as I'm getting 2K+ error message emails per night.

 

[cPRex]

If you don't have dnf, you can check /var/log/yum.log to see if the package update happened then. However, that wouldn't account for all the error messages you're seeing.

It might be best to submit a ticket to our team so we can check this directly on your machine.

 

[kieran_kkcs]

Hi, thanks.
The error messages are coming from my php scripts as they continue to run despite the MySQL server being down for a short while. I checked yum.log as per your suggestionand the last entry was from March,so that can't be it. I'll submit a ticket. Thank you.

 

[cPRex]

Once you have it, let me know the ticket number so I can follow along and keep this thread updated.

 

[kieran_kkcs]

Thanks, it looks like its 94463287

 

[cPRex]

Thanks for that - I'm following along with that ticket on my end now.

 

[cPRex]

I don't see our team was able to find much either, even with direct access to the system. I do see where we recommended having a MySQL administrator examine the machine to see if more information could be found, but it seems there just isn't much to go on with the details we found.