Solution for large files being sent to Dropbox?

[martin MHC]

I have set up a backup system to my Dropbox -- external to cPanel and the local server environment. I have used the script described here: Custom Backup Transport: Dropbox | cPanel Blog as well as various Forum posts to set up the Dropbox backup.

The script and the WHM backup interface appear to work correctly for most accounts, but there are persistent accounts that fail on _most_ but not all backups.

I believe the reason may be something like the time taken to transport the data to the external Dropbox fails due to the size of the transfers (each failed account is well over 1Gb+ due to email accounts).

Backups are compressed - backups are maximised timeout (50,000 seconds) - smaller backups are transported ok.

Example result error message is:

Unable to send “/backup/2019-01-22/accounts/accountname.tar.gz” to destination “Dropbox”

So;

It seems to be a timeout issue for the 50,000 seconds being needed to transport 1Gb+ (so on a minimum sized 1Gb backup file the backup needs to be transported at 1Mbps as a MINIMUM, so for a 3-4Gb file (of which we have a few accounts); thy need to be transported consistently at 3-4Mbps which from one server to another across the open net seems to be unlikely to transport :-(

Can cPanel provide any assistance to making this work? IS this likely to be a Dropbox issue or a Panel issue?

 

[martin MHC]

To clarify further:

1) Can we increase the 50k seconds hardlimit timeout value and see if this helps move larger files across to the backup destination?

2) If not, can the 50k timeout limit be set on a per-file basis rather than a per-transaction basis, if not already? (I am unclear on this)

3) Can the transport mechanism be able to report why the backup failed, for instance; if the timeout was reached?


Thanks!

 

[martin MHC]

Yes there is a support ticket now open. I will update here as news comes through.

 

[cPanelMichael]

Hello @martin MHC,

Thanks for opening the ticket. I'll monitor it and update this thread with more information as it becomes available. Note that for reference, the corresponding report on the GitHub repo for this script is available at:

TECH-577: large size files are not being uploaded · Issue #6 · CpanelInc/backup-transport-dropbox

Thank you.

 

[martin MHC]

Hello @cPanelMichael - sure, as referenced just a couple of posts above - are you able to give any answers to points a,b,c -if these are possible currently and if so how to achieve them, or if not currently possible any chance they'd be candidates for future versions?

Searching the online feature requests system I can't see anything for extending the 50k time limit or asking for more verbose error reporting in email (if possible), should I put these as future feature requests, or is there somethig I'm unaware of as to these topics?

Cheers

 

[cPWilliamL]

Hi @martin MHC

This should now be corrected in the 'development' branch. Please update the Github issues if you have any problems.

Thanks!

 

[quanin]

@cPWilliamL, have you corrected for the issue of large files filling up /tmp in the development branch as well? I understand the original version of this script can't feed files over 1 GB due to API limitations. I had opened a support ticket previously without coming to the forums, and was told this was being worked on.

Also, please tell me you're advocating for this to be added to CPanel by default. Please? I find Dropbox a lot less finicky than Google Drive, which I only signed up for because CPanel backups.

 

[cPanelMichael]

Hello @quanin,

1) Can we increase the 50k seconds hardlimit timeout value and see if this helps move larger files across to the backup destination?

2) If not, can the 50k timeout limit be set on a per-file basis rather than a per-transaction basis, if not already? (I am unclear on this)

3) Can the transport mechanism be able to report why the backup failed, for instance; if the timeout was reached?

It looks like the issue was addressed, but I did want to answer your earlier questions:

1. A timeout value between 300 and 50,000 seconds is required and entering a value higher than 50,000 seconds is not supported.

2. The timeout is enforced on each upload attempt (e.g. each attempt to upload account.tar.gz to the remote destination). This is not a configurable option.

3. Transport failures/errors are logged to the /usr/local/cpanel/logs/cpbackup_transporter.log file. Is there a specific event/error that isn't currently logged?

Also, please tell me you're advocating for this to be added to CPanel by default. Please? I find Dropbox a lot less finicky than Google Drive, which I only signed up for because CPanel backups.

I don't have anything new to report on the status of native support for DropBox integrated into the product as a backup destination, but I do see a fair amount of votes on the official feature request page:

Add remote backup destination Dropbox

I suggest adding voting and adding feedback directly to the feature request page if you haven't already done so. I've voted for it myself, and will push others to vote for it whenever I see questions/feedback about the use of DropBox as a backup destination. This is the best approach to getting native support added to the product.

Thank you.

 

[martin MHC]

Hello @cPanelMichael
Just as a final note - William has put a lot of effort into this issue for me/us (and others) and the development branch on the GitHub is now the main branch/master.

CpanelInc/backup-transport-dropbox

This script now successfully and correctly resolves my issues raised in this thread and causes complete file transfers to my Dropbox destination.

Many thanks to all for this