Major Issues Lately with MariaDB Upgrades and Updates via WHM

[celiac101]

After going through 2 days of hell in the last few months, I am sharing my thoughts here:

1) I no longer trust anything to do with WHM's ability to run any type of upgrade or update on MariaDB databases. Within the last few months I've had 2 major crashed database incidents due to:

• a) a more major upgrade from MariaDB 10.5 to 10.6 using WHM update mysql app;
• b) a "minor" update to MariaDB 10.6 using the standard yum update command;

I am not sure what is going on since WHM was purchased by a larger company, but to me it's all bad and after more than 10 years I'm now considering switching to an alternative. I've run WHM DB upgrade and updates for over 10 years without such issues, so having such major problems lately breaks my trust in WHM.



2) Why are there not more built in options to automatically repair DBs that have issues, or in an emergency dump a DB and re-create the DB within the WHM interface? Certainly you could offer far more tools to fix database issues than what you currently offer, which is nearly nothing. How hard would it be to offer an emergency tool that would try to perform a mysql dump and extraction in to a new DB with a "-copy" name added to the DB name. After it ran, if successful, it could display a simple message saying:

• "success-a new DB was created that was tested to have no issues with the name of db-name-copy. To test this with your app be sure to change the DB name in the config file of your site,"

Adding more repair options might at least mitigate the many problems now created when using WHM to upgrade or update MariaDB.

 

[cPRex]

Hey there! If you're referring to the issues yesterday, that was a problem with MariaDB breaking their own repositories, so there wasn't anything we could do on our end for that. MariaDB said that was resolved as of 12:30 yesterday, although some people reported odd behavior for another few hours while the mirrors were getting back in sync.

If you have ideas about adding repair features, please submit a feature request and I can bring that up during our weekly developer meeting. We're always working on and entertaining new features so I'd be happy to discuss that with them.

 

[celiac101]

No, I'm not referring to that. The auto update that ran did alert me to an issue, but when I ran yup update ~6 hours later the update ran fine, found the repositories, and then broke my databases--all of them on different web sites.

 

[cPRex]

Any chance you could submit a ticket so we can see that? I'm not saying it's not possible, but a Yum transaction wouldn't touch any data in the database itself, so I'm not sure how the two would be related without us doing some investigation.

 

[celiac101]

I did, but fixed this myself in the several hours I was waiting for anyone there to respond...oh...did I mention my issue with how long it is now to get any help from you guys?

 

[celiac101]

and yes, yum update, which updated MariaDB, did crash every database on my server to the point where the only way to repair them was a full export import routine, which took a ton of time.

 

[cPRex]

How long did you wait for a ticket response? Is your license direct through us so you receive priority support? Could you let me know a ticket number so I could review that? I'm happy to listen to feedback, but I do need some data to act on in order to help.

As far as the Yum issues, I suppose anything is possible but I'm just not sure how a Yum transaction would corrupt database data. If that did happen between Yum and MariaDB, at least cPanel tools wouldn't have been involved.

 

[celiac101]

#94475125 is the ticket number, and I believe I'm on the misnomer "Priority" support

Apparently the paid upgraded "Premium" support is no longer an option, as it now throws an error when I try to do that.

I can't tell the exact times on the ticket, but it looks like I submitted it at ~4:30PM Pacific time, and the first response on the ticket was ~4 hours later. It was enough time for a non-expert like me to research how, and to rebuild 5 mysql database, 3 of them being over 5 gigs. I'm only glad I was able successfully dump all the data, so that nothing was lost.

But really, I'd love to hear a comment about #2 above. Why the heck, given that essentially all of your customers run mysql databases, doesn't WHM offer more automated recovery options (especially if your software updates/upgrades might be breaking them more often going forward)?

BTW, I posted a reply on that ticket 6 hours ago and nobody has replied. I guess it's a good think my sites are not still down and I was able to recover the DBs myself!

 

[cPRex]

Thanks for the details. I checked the ticket and confirmed that your license isn't purchased through us, so the ticket is part of the complimentary queue, which has no official timeframe for a reply.

You are correct that there are issues with the support bump in the ticket system, and our developers are working to address that.

I see that Justin created case CPANEL-41340 with our development team to have them explore the possibility of making some type of automatic database copy during the upgrade process on the server, just in case something goes wrong. Traditionally we try and stay hands-off with data, but we'll let the team explore this and see what they come up with.

We also weren't able to access the server in the ticket, so for more details on what may have caused the MySQL problems we'd have to get access to the affected system.

 

[celiac101]

Thanks for looking into this...regardless of the license type I'm paying your regularly for my license, which should include reasonable support times, but again, your escalation system has been broken for months now, and I have no issues paying $30 or so in an emergency, but I could not even do that the last two emergencies I had (both caused by WHM). I hope you can fix it soon, but I would fire the person who's been working on it. I could have gotten that fixed for you within a few days...

 

[cPRex]

I've got someone looking into this today, so hopefully that is resolved soon.

Since the license isn't purchased through cPanel, your first contact for support would normally be the hosting provider where the license was purchased.

 

[HenrikSCH]

Hi my server gives me an update error .. this is on my MariaDB106.repo


[MariaDB106]

name = MariaDB106

baseurl = http://yum.mariadb.org/10.6/centos8-amd64

gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB

gpgcheck=1



But that folder does not exist on the MariaDB server?



I am running Almalinux 8.6 with latest WHM/cPanel and MariaDB 10.6.9


[root@host ~]# yum -y update

MariaDB106 862 B/s | 10 B 00:00

Error: Failed to download metadata for repo 'MariaDB106': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

Is there a way to fix this? Because i can not even update EA now.

 

[HenrikSCH]

The is no centos8 folders on the MariaDB servers

 

[cPRex]

CentOS 8 is not a supported operating system, so that makes sense to me. You should be using Ubuntu, CentOS 7, AlmaLinux 8, or CloudLinux for a cPanel system at this time.

 

[HenrikSCH]

I am running Almalinux 8.6 with latest WHM/cPanel and MariaDB 10.6.9

 

[HenrikSCH]

this is the text in my MariaDB106.repo on the server.

[MariaDB106]

name = MariaDB106

baseurl = http://yum.mariadb.org/10.6/centos8-amd64

gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB

gpgcheck=1

 

[HenrikSCH]

I updated from MariaDB 10.6.8 to 10.6.9 a few day ago with no problems.

 

[HenrikSCH]

Do i need to change the baseurl in the MariaDB106.repo file?

And if yes ... what do i need to change it to?

 

[ServerHealers]

@HenrikSCH
The correct baseurl for AlmaLinux8 would be Index of /yum/10.6/rhel8-amd64/

The repository /etc/yum.repos.d/MariaDB106.repo file should look like:


[MariaDB106]
name = MariaDB106
baseurl = https://yum.mariadb.org/10.6/rhel8-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1


Replace baseurl correctly in your repository file, and try again after clearing yum cache

 

[cPRex]

Thanks @ServerHealers !