Couldn't execute 'show fields from `messages`': Unknown storage engine 'InnoDB' (1286)

[ttremain]

MariaDB 10.2 at this moment.

We first noticed message came up in a backup report a couple hundred of these.

[2023-02-26 02:11:15 -0800] mysqldump: Couldn't execute 'show fields from `messages`': Unknown storage engine 'InnoDB' (1286)
[2023-02-26 02:11:15 -0800] mysqldump: Couldn't execute 'show fields from `messages`': Unknown storage engine 'InnoDB' (1286)
[2023-02-26 02:11:15 -0800] mysqldump: Couldn't execute 'show fields from `messages`': Unknown storage engine 'InnoDB' (1286)
[2023-02-26 02:11:15 -0800] mysqldump: Couldn't execute 'show fields from `messages`': Unknown storage engine 'InnoDB' (1286)
[2023-02-26 02:11:15 -0800] mysqldump: Couldn't execute 'show fields from `messages`': Unknown storage engine 'InnoDB' (1286)

Then tonight I was having trouble with some queries, and accessing table in phpMyAdmin with what I believe was also: "Unknown storage engine 'InnoDB'"


some of /var/lib/mysql/---HOSTNAME---.err

2023-02-27 22:09:06 139746927843520 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-02-27 22:09:06 139746927843520 [Note] InnoDB: Uses event mutexes
2023-02-27 22:09:06 139746927843520 [Note] InnoDB: Compressed tables use zlib 1.2.7
2023-02-27 22:09:06 139746927843520 [Note] InnoDB: Using Linux native AIO
2023-02-27 22:09:06 139746927843520 [Note] InnoDB: Number of pools: 1
2023-02-27 22:09:06 139746927843520 [Note] InnoDB: Using SSE2 crc32 instructions
2023-02-27 22:09:06 139746927843520 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2023-02-27 22:09:06 139746927843520 [ERROR] InnoDB: mmap(137232384 bytes) failed; errno 12
2023-02-27 22:09:06 139746927843520 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2023-02-27 22:09:06 139746927843520 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2023-02-27 22:09:06 139746927843520 [Note] InnoDB: Starting shutdown...
2023-02-27 22:09:06 139746927843520 [ERROR] Plugin 'InnoDB' init function returned error.
2023-02-27 22:09:06 139746927843520 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2023-02-27 22:09:06 139746927843520 [ERROR] mysqld: Out of memory (Needed 128663552 bytes)
2023-02-27 22:09:06 139746927843520 [Note] Plugin 'FEEDBACK' is disabled.
2023-02-27 22:09:06 139746927843520 [Note] Server socket created on IP: '::'.
2023-02-27 22:09:06 139746926884608 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1286: Unknown storage engine 'InnoDB'
2023-02-27 22:09:06 139746927843520 [Note] Reading of all Master_info entries succeeded
2023-02-27 22:09:06 139746927843520 [Note] Added new Master_info '' to hash table
2023-02-27 22:09:06 139746927843520 [Note] /usr/sbin/mysqld: ready for connections.

A few things here imply memory, but I'd like to find what I should specifically check.

Please advise!

 

[andrew.n]

Check your available memory with "free -m" command and set the buffer size in /etc/my.cnf accordingly:

innodb_buffer_pool_size=XXXM

 

[cPRex]

I agree with @andrew.n - it looks like your MySQL server is running out of memory before it fully restarts.

 

[ttremain]

Do you see anything new now?


<code>
Version: '10.2.44-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
2023-03-06 12:18:05 140649880459008 [Note] InnoDB: Buffer pool(s) load completed at 230306 12:18:05
2023-03-06 13:11:59 140027518089408 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-03-06 13:11:59 140027518089408 [Note] InnoDB: Uses event mutexes
2023-03-06 13:11:59 140027518089408 [Note] InnoDB: Compressed tables use zlib 1.2.7
2023-03-06 13:11:59 140027518089408 [Note] InnoDB: Using Linux native AIO
2023-03-06 13:11:59 140027518089408 [Note] InnoDB: Number of pools: 1
2023-03-06 13:11:59 140027518089408 [Note] InnoDB: Using SSE2 crc32 instructions
2023-03-06 13:12:00 140027518089408 [Note] InnoDB: Initializing buffer pool, total size = 8G, instances = 8, chunk size = 128M
2023-03-06 13:12:00 140027518089408 [Note] InnoDB: Completed initialization of buffer pool
2023-03-06 13:12:00 140018012681984 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2023-03-06 13:12:00 140027518089408 [Note] InnoDB: Highest supported file format is Barracuda.
2023-03-06 13:12:00 140027518089408 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1343910767
2023-03-06 13:12:01 140027518089408 [Note] InnoDB: 128 out of 128 rollback segments are active.
2023-03-06 13:12:01 140027518089408 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2023-03-06 13:12:01 140027518089408 [Note] InnoDB: Creating shared tablespace for temporary tables
2023-03-06 13:12:01 140027518089408 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2023-03-06 13:12:01 140027518089408 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2023-03-06 13:12:01 140027518089408 [Note] InnoDB: Waiting for purge to start
2023-03-06 13:12:01 140027518089408 [Note] InnoDB: 5.7.38 started; log sequence number 1343910776
2023-03-06 13:12:01 140018238326528 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2023-03-06 13:12:01 140027518089408 [Note] Plugin 'FEEDBACK' is disabled.
2023-03-06 13:12:01 140027518089408 [Note] Recovering after a crash using tc.log
2023-03-06 13:12:01 140027518089408 [Note] Starting crash recovery...
2023-03-06 13:12:01 140027518089408 [Note] Crash recovery finished.
2023-03-06 13:12:01 140027518089408 [Note] Server socket created on IP: '::'.
2023-03-06 13:12:02 140027518089408 [Note] Reading of all Master_info entries succeeded
2023-03-06 13:12:02 140027518089408 [Note] Added new Master_info '' to hash table
2023-03-06 13:12:02 140027518089408 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.2.44-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
2023-03-06 13:12:03 140018238326528 [Note] InnoDB: Buffer pool(s) load completed at 230306 13:12:03
2023-03-06 13:12:06 140018188326656 [ERROR] mysqld: Table './nodyss5_southrivercondo/visitors' is marked as crashed and should be repaired
2023-03-06 13:12:06 140018188326656 [ERROR] mysqld: Table 'visitors' is marked as crashed and should be repaired
2023-03-06 13:12:06 140018188326656 [Warning] Checking table: './nodyss5_southrivercondo/visitors'
2023-03-06 13:12:07 140018188633856 [ERROR] mysqld: Table './nodyss5_lexingtonplaceorlando/visitors' is marked as crashed and should be repaired
2023-03-06 13:12:07 140018188633856 [ERROR] mysqld: Table 'visitors' is marked as crashed and should be repaired
2023-03-06 13:12:07 140018188633856 [Warning] Checking table: './nodyss5_lexingtonplaceorlando/visitors'
2023-03-06 13:13:31 140018188019456 [ERROR] mysqld: Table './nodyss5_southseaswest/users' is marked as crashed and should be repaired
2023-03-06 13:13:31 140018188019456 [ERROR] mysqld: Table 'users' is marked as crashed and should be repaired
2023-03-06 13:13:31 140018188019456 [Warning] Checking table: './nodyss5_southseaswest/users'
2023-03-06 13:13:31 140018187405056 [ERROR] mysqld: Table './nodyss5_southseaswest/visitors' is marked as crashed and should be repaired
2023-03-06 13:13:31 140018187405056 [ERROR] mysqld: Table 'visitors' is marked as crashed and should be repaired
2023-03-06 13:13:31 140018187405056 [Warning] Checking table: './nodyss5_southseaswest/visitors'
2023-03-06 13:17:36 140018188633856 [ERROR] mysqld: Table './nodyss5_myccstuart/visitors' is marked as crashed and should be repaired
2023-03-06 13:17:36 140018188633856 [ERROR] mysqld: Table 'visitors' is marked as crashed and should be repaired
2023-03-06 13:17:36 140018188633856 [Warning] Checking table: './nodyss5_myccstuart/visitors'
2023-03-06 13:20:21 140018188019456 [ERROR] mysqld: Table './nodyss5_1800atlantickw/visitors' is marked as crashed and should be repaired
2023-03-06 13:20:21 140018188019456 [ERROR] mysqld: Table 'visitors' is marked as crashed and should be repaired
2023-03-06 13:20:21 140018188019456 [Warning] Checking table: './nodyss5_1800atlantickw/visitors'
2023-03-06 13:21:29 140018188019456 [ERROR] mysqld: Table './nodyss5_whisperwalkgreenleaf/visitors' is marked as crashed and should be repaired
2023-03-06 13:21:29 140018188019456 [ERROR] mysqld: Table 'visitors' is marked as crashed and should be repaired
2023-03-06 13:21:29 140018188019456 [Warning] Checking table: './nodyss5_whisperwalkgreenleaf/visitors'


</code>

 

[cPRex]

Thanks for posting that. While I don't see the memory error any longer, it does look like there are several crashed InnoDB tables. While cPanel support can't help you fix those, we have some helpful details on dealing with that here: https://support.cpanel.net/hc/en-us/articles/1500006010582-InnoDB-Corruption-Repair-Guide

 

[ttremain]

Those logs start showing after we get the Unknown storage engine 'InnoDB' errors, but the "visitors" tables listed are in the logs aren't even Innodb. I try repairing in phpMyAdmin, but there they show fine. Do you still think I should be looking at that link for more crash fixes?

The "crashed" tables appear to me to be a symptom of the service failing, but that's why I'm asking.

<code>
2023-03-06 21:03:55 140318766040832 [ERROR] mysqld: Table './nodyss5_labrisacondo/users' is marked as crashed and should be repaired
2023-03-06 21:03:55 140318766040832 [ERROR] mysqld: Table 'users' is marked as crashed and should be repaired
2023-03-06 21:03:55 140318766040832 [Warning] Checking table: './nodyss5_labrisacondo/users'
2023-03-06 21:03:55 140318766040832 [ERROR] mysqld: Table './nodyss5_labrisacondo/log' is marked as crashed and should be repaired
2023-03-06 21:03:55 140318766040832 [ERROR] mysqld: Table 'log' is marked as crashed and should be repaired
2023-03-06 21:03:55 140318766040832 [Warning] Checking table: './nodyss5_labrisacondo/log'
2023-03-06 21:27:28 140318766348032 [Warning] Access denied for user 'nodyss5_oAkH2m0TH0A'@'localhost' (using password: YES)
2023-03-06 22:53:39 140318765426432 [Warning] Access denied for user 'nodyss5_oAkH2m0TH0A'@'localhost' (using password: YES)
2023-03-06 23:11:03 140318764812032 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2023-03-06 23:21:30 140318765119232 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2023-03-07 6:28:39 140250486614208 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-03-07 6:28:39 140250486614208 [Note] InnoDB: Uses event mutexes
2023-03-07 6:28:39 140250486614208 [Note] InnoDB: Compressed tables use zlib 1.2.7
2023-03-07 6:28:39 140250486614208 [Note] InnoDB: Using Linux native AIO
2023-03-07 6:28:39 140250486614208 [Note] InnoDB: Number of pools: 1
2023-03-07 6:28:39 140250486614208 [Note] InnoDB: Using SSE2 crc32 instructions
2023-03-07 6:28:39 140250486614208 [Note] InnoDB: Initializing buffer pool, total size = 8G, instances = 8, chunk size = 128M
2023-03-07 6:28:40 140250486614208 [Note] InnoDB: Completed initialization of buffer pool
2023-03-07 6:28:40 140240971593472 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2023-03-07 6:28:40 140250486614208 [Note] InnoDB: Highest supported file format is Barracuda.
2023-03-07 6:28:40 140250486614208 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1352375362
2023-03-07 6:28:40 140250486614208 [Note] InnoDB: 128 out of 128 rollback segments are active.
2023-03-07 6:28:40 140250486614208 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2023-03-07 6:28:40 140250486614208 [Note] InnoDB: Creating shared tablespace for temporary tables
2023-03-07 6:28:40 140250486614208 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2023-03-07 6:28:40 140250486614208 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2023-03-07 6:28:40 140250486614208 [Note] InnoDB: Waiting for purge to start
2023-03-07 6:28:41 140250486614208 [Note] InnoDB: 5.7.38 started; log sequence number 1352375371
2023-03-07 6:28:41 140241197225728 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2023-03-07 6:28:41 140250486614208 [Note] Plugin 'FEEDBACK' is disabled.
2023-03-07 6:28:41 140250486614208 [Note] Recovering after a crash using tc.log
2023-03-07 6:28:41 140250486614208 [Note] Starting crash recovery...
2023-03-07 6:28:41 140250486614208 [Note] Crash recovery finished.
2023-03-07 6:28:41 140241197225728 [Note] InnoDB: Buffer pool(s) load completed at 230307 6:28:41
2023-03-07 6:28:41 140250486614208 [Note] Server socket created on IP: '::'.
2023-03-07 6:28:41 140250486614208 [Note] Reading of all Master_info entries succeeded
2023-03-07 6:28:41 140250486614208 [Note] Added new Master_info '' to hash table
2023-03-07 6:28:41 140250486614208 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.2.44-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
2023-03-07 6:28:54 140037555460288 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-03-07 6:28:54 140037555460288 [Note] InnoDB: Uses event mutexes
2023-03-07 6:28:54 140037555460288 [Note] InnoDB: Compressed tables use zlib 1.2.7
2023-03-07 6:28:54 140037555460288 [Note] InnoDB: Using Linux native AIO
2023-03-07 6:28:54 140037555460288 [Note] InnoDB: Number of pools: 1
2023-03-07 6:28:54 140037555460288 [Note] InnoDB: Using SSE2 crc32 instructions
2023-03-07 6:28:54 140037555460288 [Note] InnoDB: Initializing buffer pool, total size = 8G, instances = 8, chunk size = 128M
2023-03-07 6:28:55 140037555460288 [Note] InnoDB: Completed initialization of buffer pool
2023-03-07 6:28:55 140028040726272 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2023-03-07 6:28:55 140037555460288 [Note] InnoDB: Highest supported file format is Barracuda.
2023-03-07 6:28:55 140037555460288 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1352375399
2023-03-07 6:28:56 140037555460288 [Note] InnoDB: 128 out of 128 rollback segments are active.
2023-03-07 6:28:56 140037555460288 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2023-03-07 6:28:56 140037555460288 [Note] InnoDB: Creating shared tablespace for temporary tables
2023-03-07 6:28:56 140037555460288 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2023-03-07 6:28:56 140037555460288 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2023-03-07 6:28:56 140037555460288 [Note] InnoDB: Waiting for purge to start
2023-03-07 6:28:56 140037555460288 [Note] InnoDB: 5.7.38 started; log sequence number 1352375408
2023-03-07 6:28:56 140028266350336 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2023-03-07 6:28:56 140028266350336 [Note] InnoDB: Buffer pool(s) load completed at 230307 6:28:56
2023-03-07 6:28:56 140037555460288 [Note] Plugin 'FEEDBACK' is disabled.
2023-03-07 6:28:56 140037555460288 [Note] Recovering after a crash using tc.log
2023-03-07 6:28:56 140037555460288 [Note] Starting crash recovery...
2023-03-07 6:28:56 140037555460288 [Note] Crash recovery finished.
2023-03-07 6:28:56 140037555460288 [Note] Server socket created on IP: '::'.
2023-03-07 6:28:56 140037555460288 [Note] Reading of all Master_info entries succeeded
2023-03-07 6:28:56 140037555460288 [Note] Added new Master_info '' to hash table
2023-03-07 6:28:56 140037555460288 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.2.44-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
2023-03-07 6:29:03 140028215736064 [ERROR] mysqld: Table './nodyss5_mycitrusvalley/users' is marked as crashed and should be repaired
2023-03-07 6:29:03 140028215736064 [ERROR] mysqld: Table 'users' is marked as crashed and should be repaired
2023-03-07 6:29:03 140028215736064 [Warning] Checking table: './nodyss5_mycitrusvalley/users'
2023-03-07 6:29:03 140028215736064 [ERROR] mysqld: Table './nodyss5_mycitrusvalley/visitors' is marked as crashed and should be repaired
2023-03-07 6:29:03 140028215736064 [ERROR] mysqld: Table 'visitors' is marked as crashed and should be repaired
2023-03-07 6:29:03 140028215736064 [Warning] Checking table: './nodyss5_mycitrusvalley/visitors'
2023-03-07 6:29:26 140028215428864 [ERROR] mysqld: Table './nodyss5_pacc2info/users' is marked as crashed and should be repaired
2023-03-07 6:29:26 140028215428864 [ERROR] mysqld: Table 'users' is marked as crashed and should be repaired
2023-03-07 6:29:26 140028215428864 [Warning] Checking table: './nodyss5_pacc2info/users'
2023-03-07 6:31:03 140028216043264 [ERROR] mysqld: Table './nodyss5_burgundy1/visitors' is marked as crashed and should be repaired
2023-03-07 6:31:03 140028216043264 [ERROR] mysqld: Table 'visitors' is marked as crashed and should be repaired
2023-03-07 6:31:03 140028216043264 [Warning] Checking table: './nodyss5_burgundy1/visitors'
2023-03-07 6:33:29 140028214814464 [ERROR] mysqld: Table './nodyss5_riverbendfl/visitors' is marked as crashed and should be repaired
2023-03-07 6:33:29 140028214814464 [ERROR] mysqld: Table 'visitors' is marked as crashed and should be repaired
2023-03-07 6:33:29 140028214814464 [Warning] Checking table: './nodyss5_riverbendfl/visitors'
2023-03-07 6:49:33 139923511498944 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-03-07 6:49:33 139923511498944 [Note] InnoDB: Uses event mutexes
2023-03-07 6:49:33 139923511498944 [Note] InnoDB: Compressed tables use zlib 1.2.7
2023-03-07 6:49:33 139923511498944 [Note] InnoDB: Using Linux native AIO
2023-03-07 6:49:33 139923511498944 [Note] InnoDB: Number of pools: 1
2023-03-07 6:49:33 139923511498944 [Note] InnoDB: Using SSE2 crc32 instructions
2023-03-07 6:49:33 139923511498944 [Note] InnoDB: Initializing buffer pool, total size = 8G, instances = 8, chunk size = 128M
2023-03-07 6:49:34 139923511498944 [Note] InnoDB: Completed initialization of buffer pool
2023-03-07 6:49:34 139914009155328 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2023-03-07 6:49:34 139923511498944 [Note] InnoDB: Highest supported file format is Barracuda.
2023-03-07 6:49:34 139923511498944 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1352375436
2023-03-07 6:49:47 140519621306560 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-03-07 6:49:47 140519621306560 [Note] InnoDB: Uses event mutexes
2023-03-07 6:49:47 140519621306560 [Note] InnoDB: Compressed tables use zlib 1.2.7
2023-03-07 6:49:47 140519621306560 [Note] InnoDB: Using Linux native AIO
2023-03-07 6:49:47 140519621306560 [Note] InnoDB: Number of pools: 1
2023-03-07 6:49:47 140519621306560 [Note] InnoDB: Using SSE2 crc32 instructions
2023-03-07 6:49:48 140519621306560 [Note] InnoDB: Initializing buffer pool, total size = 8G, instances = 8, chunk size = 128M
2023-03-07 6:49:48 140519621306560 [Note] InnoDB: Completed initialization of buffer pool
2023-03-07 6:49:48 140510116435712 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2023-03-07 6:49:48 140519621306560 [Note] InnoDB: Highest supported file format is Barracuda.
2023-03-07 6:49:48 140519621306560 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1352375436
2023-03-07 6:49:49 140519621306560 [Note] InnoDB: 128 out of 128 rollback segments are active.
2023-03-07 6:49:49 140519621306560 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2023-03-07 6:49:49 140519621306560 [Note] InnoDB: Creating shared tablespace for temporary tables
2023-03-07 6:49:49 140519621306560 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2023-03-07 6:49:49 140519621306560 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2023-03-07 6:49:49 140519621306560 [Note] InnoDB: Waiting for purge to start
2023-03-07 6:49:49 140519621306560 [Note] InnoDB: 5.7.38 started; log sequence number 1352375445
2023-03-07 6:49:49 140510342076160 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2023-03-07 6:49:49 140519621306560 [Note] Plugin 'FEEDBACK' is disabled.
2023-03-07 6:49:49 140519621306560 [Note] Recovering after a crash using tc.log
2023-03-07 6:49:49 140519621306560 [Note] Starting crash recovery...
2023-03-07 6:49:49 140519621306560 [Note] Crash recovery finished.
2023-03-07 6:49:49 140510342076160 [Note] InnoDB: Buffer pool(s) load completed at 230307 6:49:49
2023-03-07 6:49:49 140519621306560 [Note] Server socket created on IP: '::'.
2023-03-07 6:49:49 140519621306560 [Note] Reading of all Master_info entries succeeded
2023-03-07 6:49:49 140519621306560 [Note] Added new Master_info '' to hash table
2023-03-07 6:49:49 140519621306560 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.2.44-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
2023-03-07 6:50:12 140510290847488 [ERROR] mysqld: Table './nodyss5_oakbrookcondominium/visitors' is marked as crashed and should be repaired
2023-03-07 6:50:12 140510290847488 [ERROR] mysqld: Table 'visitors' is marked as crashed and should be repaired
2023-03-07 6:50:12 140510290847488 [Warning] Checking table: './nodyss5_oakbrookcondominium/visitors'
2023-03-07 6:50:35 140510290847488 [ERROR] mysqld: Table './nodyss5_southrivercondo/visitors' is marked as crashed and should be repaired
2023-03-07 6:50:35 140510290847488 [ERROR] mysqld: Table 'visitors' is marked as crashed and should be repaired
2023-03-07 6:50:35 140510290847488 [Warning] Checking table: './nodyss5_southrivercondo/visitors'
2023-03-07 6:51:05 140510291461888 [ERROR] mysqld: Table './nodyss5_beachandtennisclub/visitors' is marked as crashed and should be repaired
2023-03-07 6:51:05 140510291461888 [ERROR] mysqld: Table 'visitors' is marked as crashed and should be repaired
2023-03-07 6:51:05 140510291461888 [Warning] Checking table: './nodyss5_beachandtennisclub/visitors'
2023-03-07 6:55:48 140510290233088 [ERROR] mysqld: Table './nodyss5_auburnvillage/users' is marked as crashed and should be repaired
2023-03-07 6:55:48 140510290233088 [ERROR] mysqld: Table 'users' is marked as crashed and should be repaired
2023-03-07 6:55:48 140510290233088 [Warning] Checking table: './nodyss5_auburnvillage/users'
2023-03-07 6:59:34 140510290847488 [ERROR] mysqld: Table './nodyss5_auburnvillage/visitors' is marked as crashed and should be repaired
2023-03-07 6:59:34 140510290847488 [ERROR] mysqld: Table 'visitors' is marked as crashed and should be repaired
2023-03-07 6:59:34 140510290847488 [Warning] Checking table: './nodyss5_auburnvillage/visitors'
2023-03-07 7:20:22 140510289311488 [ERROR] mysqld: Table './nodyss5_riverbendfl/realestate' is marked as crashed and should be repaired
2023-03-07 7:20:22 140510289311488 [ERROR] mysqld: Table 'realestate' is marked as crashed and should be repaired
2023-03-07 7:20:22 140510289311488 [Warning] Checking table: './nodyss5_riverbendfl/realestate'
2023-03-07 7:30:40 140510289618688 [Warning] Access denied for user 'nodyss5_oAkH2m0TH0A'@'localhost' (using password: YES)
2023-03-07 8:20:01 140510289618688 [Warning] Access denied for user 'nodyss5_oAkH2m0TH0A'@'localhost' (using password: YES)
2023-03-07 8:42:56 140510288389888 [ERROR] mysqld: Table './nodyss5_princesscondo/visitors' is marked as crashed and should be repaired
2023-03-07 8:42:56 140510288389888 [ERROR] mysqld: Table 'visitors' is marked as crashed and should be repaired
2023-03-07 8:42:56 140510288389888 [Warning] Checking table: './nodyss5_princesscondo/visitors'
2023-03-07 8:56:39 140510288697088 [Warning] Access denied for user 'nodyss5_oAkH2m0TH0A'@'localhost' (using password: YES)
2023-03-07 8:59:57 139879386273984 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-03-07 8:59:57 139879386273984 [Note] InnoDB: Uses event mutexes
2023-03-07 8:59:57 139879386273984 [Note] InnoDB: Compressed tables use zlib 1.2.7
2023-03-07 8:59:57 139879386273984 [Note] InnoDB: Using Linux native AIO
2023-03-07 8:59:57 139879386273984 [Note] InnoDB: Number of pools: 1
2023-03-07 8:59:57 139879386273984 [Note] InnoDB: Using SSE2 crc32 instructions
2023-03-07 8:59:59 139879386273984 [Note] InnoDB: Initializing buffer pool, total size = 8G, instances = 8, chunk size = 128M
2023-03-07 8:59:59 139879386273984 [Note] InnoDB: Completed initialization of buffer pool
2023-03-07 8:59:59 139869871138560 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2023-03-07 8:59:59 139879386273984 [Note] InnoDB: Highest supported file format is Barracuda.
2023-03-07 8:59:59 139879386273984 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1352757841
2023-03-07 9:00:00 139879386273984 [Note] InnoDB: 128 out of 128 rollback segments are active.
2023-03-07 9:00:00 139879386273984 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2023-03-07 9:00:00 139879386273984 [Note] InnoDB: Creating shared tablespace for temporary tables
2023-03-07 9:00:00 139879386273984 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2023-03-07 9:00:00 139879386273984 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2023-03-07 9:00:00 139879386273984 [Note] InnoDB: Waiting for purge to start
2023-03-07 9:00:00 139879386273984 [Note] InnoDB: 5.7.38 started; log sequence number 1352757850
2023-03-07 9:00:00 139870096762624 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2023-03-07 9:00:00 139879386273984 [Note] Plugin 'FEEDBACK' is disabled.
2023-03-07 9:00:00 139879386273984 [Note] Recovering after a crash using tc.log
2023-03-07 9:00:00 139879386273984 [Note] Starting crash recovery...
2023-03-07 9:00:00 139879386273984 [Note] Crash recovery finished.
2023-03-07 9:00:00 139870096762624 [Note] InnoDB: Buffer pool(s) load completed at 230307 9:00:00
2023-03-07 9:00:00 139879386273984 [Note] Server socket created on IP: '::'.
2023-03-07 9:00:00 139879233410816 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown
2023-03-07 9:00:00 139879386273984 [Note] Reading of all Master_info entries succeeded
2023-03-07 9:00:00 139879386273984 [Note] Added new Master_info '' to hash table
2023-03-07 9:00:00 139879386273984 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.2.44-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
2023-03-07 9:00:00 139879233410816 [Note] Event Scheduler: Purging the queue. 0 events
2023-03-07 9:00:00 139870138726144 [Note] InnoDB: FTS optimize thread exiting.
2023-03-07 9:00:00 139879233410816 [Note] InnoDB: Starting shutdown...
2023-03-07 9:00:00 139870096762624 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2023-03-07 9:00:00 139870096762624 [Note] InnoDB: Buffer pool(s) dump completed at 230307 9:00:00
2023-03-07 9:00:03 139879233410816 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2023-03-07 9:00:03 139879233410816 [Note] InnoDB: Shutdown completed; log sequence number 1352757869
2023-03-07 9:00:03 139879233410816 [Note] /usr/sbin/mysqld: Shutdown complete

2023-03-07 9:00:04 139870554659008 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-03-07 9:00:04 139870554659008 [Note] InnoDB: Uses event mutexes
2023-03-07 9:00:04 139870554659008 [Note] InnoDB: Compressed tables use zlib 1.2.7
2023-03-07 9:00:04 139870554659008 [Note] InnoDB: Using Linux native AIO
2023-03-07 9:00:04 139870554659008 [Note] InnoDB: Number of pools: 1
2023-03-07 9:00:04 139870554659008 [Note] InnoDB: Using SSE2 crc32 instructions
2023-03-07 9:00:04 139870554659008 [Note] InnoDB: Initializing buffer pool, total size = 8G, instances = 8, chunk size = 128M
2023-03-07 9:00:04 139870554659008 [Note] InnoDB: Completed initialization of buffer pool
2023-03-07 9:00:05 139861058160384 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2023-03-07 9:00:05 139870554659008 [Note] InnoDB: Highest supported file format is Barracuda.
2023-03-07 9:00:05 139870554659008 [Note] InnoDB: 128 out of 128 rollback segments are active.
2023-03-07 9:00:05 139870554659008 [Note] InnoDB: Creating shared tablespace for temporary tables
2023-03-07 9:00:05 139870554659008 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2023-03-07 9:00:05 139870554659008 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2023-03-07 9:00:05 139870554659008 [Note] InnoDB: 5.7.38 started; log sequence number 1352757869
2023-03-07 9:00:05 139861265630976 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2023-03-07 9:00:05 139861265630976 [Note] InnoDB: Buffer pool(s) load completed at 230307 9:00:05
2023-03-07 9:00:05 139870554659008 [Note] Plugin 'FEEDBACK' is disabled.
2023-03-07 9:00:05 139870554659008 [Note] Server socket created on IP: '::'.
2023-03-07 9:00:05 139870554659008 [Note] Reading of all Master_info entries succeeded
2023-03-07 9:00:05 139870554659008 [Note] Added new Master_info '' to hash table
2023-03-07 9:00:05 139870554659008 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.2.44-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
2023-03-07 9:01:19 139861215938304 [ERROR] mysqld: Table './nodyss5_islandiaone/visitors' is marked as crashed and should be repaired
2023-03-07 9:01:19 139861215938304 [ERROR] mysqld: Table 'visitors' is marked as crashed and should be repaired
2023-03-07 9:01:19 139861215938304 [Warning] Checking table: './nodyss5_islandiaone/visitors'
2023-03-07 9:02:01 139870375315200 [ERROR] mysqld: Table './nodyss5_polyvill/visitors' is marked as crashed and should be repaired
2023-03-07 9:02:01 139870375315200 [ERROR] mysqld: Table 'visitors' is marked as crashed and should be repaired
2023-03-07 9:02:01 139870375315200 [Warning] Checking table: './nodyss5_polyvill/visitors'
2023-03-07 9:02:37 139861215323904 [ERROR] mysqld: Table './nodyss5_southseaswest/visitors' is marked as crashed and should be repaired
2023-03-07 9:02:37 139861215323904 [ERROR] mysqld: Table 'visitors' is marked as crashed and should be repaired
2023-03-07 9:02:37 139861215323904 [Warning] Checking table: './nodyss5_southseaswest/visitors'
2023-03-07 9:03:48 139861215323904 [ERROR] mysqld: Table './nodyss5_lakesidevillagefla/visitors' is marked as crashed and should be repaired
2023-03-07 9:03:48 139861215323904 [ERROR] mysqld: Table 'visitors' is marked as crashed and should be repaired
2023-03-07 9:03:48 139861215323904 [Warning] Checking table: './nodyss5_lakesidevillagefla/visitors'
2023-03-07 9:04:44 139870375315200 [ERROR] mysqld: Table './nodyss5_polyvill/users' is marked as crashed and should be repaired
2023-03-07 9:04:44 139870375315200 [ERROR] mysqld: Table 'users' is marked as crashed and should be repaired
2023-03-07 9:04:44 139870375315200 [Warning] Checking table: './nodyss5_polyvill/users'
2023-03-07 9:12:55 139870375315200 [ERROR] mysqld: Table './nodyss5_forestlakescondos/visitors' is marked as crashed and should be repaired
2023-03-07 9:12:55 139870375315200 [ERROR] mysqld: Table 'visitors' is marked as crashed and should be repaired
2023-03-07 9:12:55 139870375315200 [Warning] Checking table: './nodyss5_forestlakescondos/visitors'
2023-03-07 9:13:19 139861214709504 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown
2023-03-07 9:13:19 139861214709504 [Note] Event Scheduler: Purging the queue. 0 events
2023-03-07 9:13:19 139861307594496 [Note] InnoDB: FTS optimize thread exiting.
2023-03-07 9:13:19 139861214709504 [Note] InnoDB: Starting shutdown...
2023-03-07 9:13:19 139861265630976 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2023-03-07 9:13:19 139861265630976 [Note] InnoDB: Buffer pool(s) dump completed at 230307 9:13:19
2023-03-07 9:13:21 139861214709504 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2023-03-07 9:13:21 139861214709504 [Note] InnoDB: Shutdown completed; log sequence number 1352757897
2023-03-07 9:13:21 139861214709504 [Note] /usr/sbin/mysqld: Shutdown complete


</code>

 

[cPRex]

Yes, as long as the logs show the "marked as crashed and should be repaired" error you'll have to go through that process. Or, you can restore the tables from a backup.

 

[ttremain]

Yes, as long as the logs show the "marked as crashed and should be repaired" error you'll have to go through that process. Or, you can restore the tables from a backup.

How about export the table, drop it, and re-import it? All the data seems to be in tact.

 

[cPRex]

Sure, that's basically a restore with just a different process. Although if you're working with the currently crashed table, I wouldn't expect good results.

 

[ttremain]

Sure, that's basically a restore with just a different process. Although if you're working with the currently crashed table, I wouldn't expect good results.

The visitors table is temporary anyway. It's a logs table, I could truncate it without worry, so drop and re-create with no concern of data integ.

 

[ttremain]

Watching the logs, each table that said it was corrupt so far, has been deleted and restored.

As of this time, mysql hasn't failed in about 8 hours.

Thank you!

 

[cPRex]

Nice - I'm so glad to hear that!

 

[ttremain]

Nice - I'm so glad to hear that!

15(ish) more tables corrupt before morning. Running MySQL Tuning-Primer, even though mysql hasn't been up for 48 hours.

 

[cPRex]

I wonder if there is some underlying disk issues on the machine that could be causing this type of behavior to happen.

 

[ttremain]

I wonder if there is some underlying disk issues on the machine that could be causing this type of behavior to happen.

We are wondering the same thing. Probably time to migrate to a new dedicated anway. This one is 6 years old, that's about 60 in computer years.

 

[ttremain]

I wonder if there is some underlying disk issues on the machine that could be causing this type of behavior to happen.

Ran tuning-primer and adjusted a few settings. Seems to have held for the last 48 hours! WOOHOO!!!!

 

[cPRex]

Nice - so there's hope!

 

[andrew.n]

Good job @ttremain!