S3 Compatible backup configuration

[nixon7606]

Attempting to configure an S3Compatible backup destination in WHM. S3 server is running on MinIO, have tested the S3 server using the "mc" MinIO S3 client from the WHM server, this works fine. When attempting to configure the destination in cPanel, "Validation" always fails. I am pretty sure we're not falling into the trap of bucketname.domain.com where bucketname needs to be specified separately. Regardless of how we've attempted to configure this, the validation step fails. Logs are scarce as far as I can tell. The typical error we get is "Error: Validation for transport “cpbackup” failed: Could not upload test file: Timeout"; (cpbackup is the name we've given this destination). Has anyone succeeded in getting cPanel to use MinIO as a backup destination? What logs can I be checking? Are there any OS dependencies that should be present for S3 clients to work? Furthermore, tcpdump on the MinIO server while doing validation shows no packets actually hitting the S3 server from the cPanel server.

Any help would be appreciated.

[2022-04-18 11:31:23 -0600] info [xml-api] Validation for transport “cpbackup” failed: Could not upload test file: Timeout [backup_destination_validate] version [1].

 

[Spirogg]

Attempting to configure an S3Compatible backup destination in WHM. S3 server is running on MinIO, have tested the S3 server using the "mc" MinIO S3 client from the WHM server, this works fine. When attempting to configure the destination in cPanel, "Validation" always fails. I am pretty sure we're not falling into the trap of bucketname.domain.com where bucketname needs to be specified separately. Regardless of how we've attempted to configure this, the validation step fails. Logs are scarce as far as I can tell. The typical error we get is "Error: Validation for transport “cpbackup” failed: Could not upload test file: Timeout"; (cpbackup is the name we've given this destination). Has anyone succeeded in getting cPanel to use MinIO as a backup destination? What logs can I be checking? Are there any OS dependencies that should be present for S3 clients to work? Furthermore, tcpdump on the MinIO server while doing validation shows no packets actually hitting the S3 server from the cPanel server.

Any help would be appreciated.

[2022-04-18 11:31:23 -0600] info [xml-api] Validation for transport “cpbackup” failed: Could not upload test file: Timeout [backup_destination_validate] version [1].

not sure if you read the docs here might be helpful ?

Backup Configuration | cPanel & WHM Documentation

The Backup Configuration interface allows system administrators to customize their scheduled backups.

docs.cpanel.net

Setting​	Description​
Destination Name​	Required — Enter a destination name for your backup files. This name appears in your destination table.​
Transfer System Backups to this Destination​	Select this checkbox to transfer system backups to this additional destination. To use this setting, you must also select the Back up System Files checkbox in the Backup Settings tab.​
Folder​	Enter the name of the directory where you wish to store your backup.​
S3 Endpoint​	Required — Enter the fully qualified domain name (FQDN) for the remote server where your bucket resides. The system backs up files to the destination server through a secured connection via SSL/TLS. The FQDN that you enter must be the same as the FQDN on the server’s SSL/TLS certificate, or the connection will fail.​
Bucket​	Required — Enter the name of the bucket where you wish to store your backup. You must first create the bucket through your S3-compatible provider.​
Access Key ID​	Required — Enter the access key ID in this text box. Your server uses the access key to authenticate with your S3-compatible account. You must first generate an access key ID through your S3-compatible provider.​
Secret Access Key​	Required — Enter the secret access key in this text box. You must first generate a secret access key ID through your S3-compatible provider.​
Timeout​	Required— The maximum amount of time, in seconds, that you want the server to wait for a response from the remote server before it generates errors. You must enter a number between 30 and 300. If the server does not respond before the time expires, it makes two additional attempts to contact the server. If the server does not respond after those attempts, the system administrator receives an email that notes the failed attempts. The system will retry the transfer when it runs the backup process again.

 

[nixon7606]

not sure if you read the docs here might be helpful ?

Backup Configuration | cPanel & WHM Documentation

The Backup Configuration interface allows system administrators to customize their scheduled backups.

docs.cpanel.net

Setting​	Description​
Destination Name​	Required — Enter a destination name for your backup files. This name appears in your destination table.​
Transfer System Backups to this Destination​	Select this checkbox to transfer system backups to this additional destination. To use this setting, you must also select the Back up System Files checkbox in the Backup Settings tab.​
Folder​	Enter the name of the directory where you wish to store your backup.​
S3 Endpoint​	Required — Enter the fully qualified domain name (FQDN) for the remote server where your bucket resides. The system backs up files to the destination server through a secured connection via SSL/TLS. The FQDN that you enter must be the same as the FQDN on the server’s SSL/TLS certificate, or the connection will fail.​
Bucket​	Required — Enter the name of the bucket where you wish to store your backup. You must first create the bucket through your S3-compatible provider.​
Access Key ID​	Required — Enter the access key ID in this text box. Your server uses the access key to authenticate with your S3-compatible account. You must first generate an access key ID through your S3-compatible provider.​
Secret Access Key​	Required — Enter the secret access key in this text box. You must first generate a secret access key ID through your S3-compatible provider.​
Timeout​	Required— The maximum amount of time, in seconds, that you want the server to wait for a response from the remote server before it generates errors.​ You must enter a number between 30 and 300. If the server does not respond before the time expires, it makes two additional attempts to contact the server. If the server does not respond after those attempts, the system administrator receives an email that notes the failed attempts. The system will retry the transfer when it runs the backup process again.

We have read the documentation. Is there a way to increase the log level so that we can see what is happening? We aren't even seeing traffic via tcpdump from cpanel to the minio server. What S3 client does cpanel use? is there a way to use it via the command line for testing?

 

[cPRex]

I don't have a good way to change the backup verbosity. I'm not sure there is a specific "client" that we used, although it is based around an S3 perl module.

This sounds like it would be a good request to open a ticket with our team so we can do some more investigating directly on an affected system.

 

[greenspike-it]

Was this ever resolved? I am having the same issue. Should I be opening a ticket as well?

 

[cPRex]

I don't see a resolution on this, so feel free to create a ticket and post the number here!

 

[greenspike-it]

I don't see a resolution on this, so feel free to create a ticket and post the number here!

Ticket #95097115

 

[cPRex]

Thanks - I'm following along with that now!

 

[greenspike-it]

After working/troubleshooting with support, it looks/sounds like the S3 Compatible backup feature, doesn't support Path based S3 connections (ie https://subdomain.example.com/bucket-name/filename.type) that is the default for my version of MINIO. cPanel only supports Virtual-host based S3 connections (ie https://bucket-name.subdomain.example.com/filename.type). There is a way to change that setting for MINIO, but I have yet to set that up yet. First I have to update my version to more current one, then I will try it after that. If I run into more issues or that doesn't work, I will come back post again .