Server's disk space will soon be full while backing up

[T1531]

I thought I'd make a new thread after my previous one about possibly using JetBackup so I could get some advice about my actual issue.

I'm currently using cPanel's built-in backup configuration, using an S3 destination, and then not retaining the backups in the default backup directory after they're uploaded. The backup type is compressed. My server is getting closer and closer to running out of disk space every time it creates the backup. If I don't address this soon, it will no longer be able to back up. (This is about space on the server itself, not the backup destination.) Once it moves the backup to the remote destination, everything is back to normal since I'm not retaining the backups locally.

What are some ways I can reduce the amount of space the server uses while it creates the backup? Do uncompressed and incremental also need to use all of this space, or is this specifically related to using the compressed method?

My current idea is to replace my Amazon S3 remote destination with some other host that supports rsync so I can use incremental, but I'm not sure exactly how these other methods work. Do they still need to be copied to the backup directory before uploading remotely or can they be uploaded directly from their source? About half of the data I'm backing up is from databases. I realize these would take up extra space either way since they need to be dumped, but I'm hoping at least regular files don't need to be copied to the backup directory before uploading.

 

[ffeingol]

This is the way that I belive the compressed backup works.

- For each account it creates all the files and folders needed for the backup in a temporary location (off of /home from memory)
- Once that folder structure is created, it creates the compressed backup file (a tar.gz)
- Once the compressed backup is created the temp folder structure is removed
- After the compressed backup is transferred, it's removed

So you need 2x the space of your largest account + room to create the compressed file.

Using rsync does not need all the temp space for the sites files/folders as they are synced from their current location. You do need some temp space for MySQL backups, as those have to be created, transferred and then removed.

I'm sure that others will confirm my understanding, but that should give you a general guide.

 

[cPRex]

So you need 2x the space of your largest account + room to create the compressed file.

That's the rule we go by, and what we say in our docs. 2x the space + 1G is our rule to ensure backups and migrations work well.

 

[T1531]

That's the rule we go by, and what we say in our docs. 2x the space + 1G is our rule to ensure backups and migrations work well.

Is that only for compressed though? Is using uncompressed/incremental instead of compressed a viable option for saving space during the backup process?

 

[cPRex]

Yes, although the first run of the incremental backups will effectively be a full backup.

 

[T1531]

Yes, although the first run of the incremental backups will effectively be a full backup.

I know incremental backups will save space where the backups are stored, but I mean on the server itself (while the backup is being created, before the backup is moved remotely and deleted locally). Do both uncompressed and incremental not add local disk space other than the database dumps during the process of backing up like ffeingol thought?

 

[cPRex]

Yes, that is also true - it will take much less space during the backup process as well, after the initial backup is generated.

 

[Ed Bergavera]

If possible, in your situation, I recommend mounting a separate disk for backup purposes alone. This saves the root disk from filling up where cpanel accounts and services consume some space over time.

 

[SimpleSonic]

I feel like everything discussed at this point is more of a band-aid rather than a resolution.

At some point you will just need to seriously consider moving to a server with adequate resources even if a migration to a new server is a headache for you because low disk space will always be a concern.

 

[T1531]

Yes, that is also true - it will take much less space during the backup process as well, after the initial backup is generated.

But if an incremental backup needs to use that space on the initial one, does that mean using the uncompressed one will use that space every time?

I feel like everything discussed at this point is more of a band-aid rather than a resolution.

At some point you will just need to seriously consider moving to a server with adequate resources even if a migration to a new server is a headache for you because low disk space will always be a concern.

If possible, in your situation, I recommend mounting a separate disk for backup purposes alone. This saves the root disk from filling up where cpanel accounts and services consume some space over time.

That's my solution for the future. After I streamline the backup process and it lasts me some time until it eventually gets too big again, I'll either mount a separate disk for backups or for a very large image folder (to store those separately). Meanwhile, I'm also trying to reduce data. I was at 53% disk usage when I started addressing this. After deleting some data, I've been able to get it down to 48% so far. Obviously, that's not 2x of free space, which is why I'm wondering how the other two backup types use space while creating the backups.

 

[cPRex]

But if an incremental backup needs to use that space on the initial one, does that mean using the uncompressed one will use that space every time?

Incremental is the uncompressed one. After the initial backup run, it will take considerably less space, both to create and store.

 

[T1531]

Incremental is the uncompressed one. After the initial backup run, it will take considerably less space, both to create and store.

What about the other one named "uncompressed" though?


I'm considering using this one because Incremental only works with Rsync, so my remote destination options are more limited. While it would obviously take up more space to store, I'm interested to know if it will take up less space to create.

 

[cPRex]

Slightly less space to create, but not much.