- Notifications
You must be signed in to change notification settings - Fork2
centminmod/cpanel-backup
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
cpanel-backup.sh is a cPanel/WHM backup script I was hired to write for a client to allow data migration from cPanel/WHM toCentmin Mod LEMP stack as a result of thecPanel announced price increase and licensing changes. This is the backup part with an accompanying import part to come and backup data transfer part as well - when combined will make up the data migrator for importing into Centmin Mod LEMP stack based servers. Centmin Mod LEMP stack isn't for shared hosting, so isn't always an alternative to cPanel. However, Centmin Mod LEMP stack is best suited for single site owner managed own sites where there's trust. Note: there is already a manual guide forcPanel/WHM to Centmin Mod data migration.
cpanel-backup.shwill have features fromdbbackup.sh including conditional db character set detection &pure innodb database detection for conditional single-transaction options and using nice/ionice for tar backups.cpanel-backup.shwill also have sar & pidstat cpu, memory and disk usage stats recorded allowing you to fine tune your backup and compression parameters for your specific server and backup requirements i.e. lower memory & cpu usage.- pidstat logs and backup logs also also zstd level 1 compressed to save space.
cpanel-backup.shwill support the following compression algorithms, gzip via multi-threaded pigz, xz via multi-threaded pxz and zstd (zstd is the default due to speed and compression ration performance)cpanel-backup.shcompression algorithm level settings have finer granular control on a per backup target basis, so public_html, mail, logs, ssl or database backup targets each have their own compression level control so you can optimise compression speed and compressed file size as well as control the amount of server resources used (cpu, memory etc)- zstd compression is the default compression used with tar backups and has both normal and a low memory mode to reduce memory used for compression.
- For cPanel log files if you opt to back them up,
cpanel-backup.shwill conditionally reduce zstd compression level to lowest negative 10 (--fast=10) levels to not waste time if the script detects there are already a mix of compressed & uncompressed version of your logs (due to logrotate). FYI, zstd has compression levels from fastest to slowest (smallest compressed file size) from -10 to 19 and then 3 ultra levels 20-22. cpanel-backup.shwill support backing up all cpanel user accounts in same session as well as per cpanel user account backups on command line.cpanel-backup.shwill support alias name masking for domain name, username and database name to allow you to publicly demo Slack notifications on live cPanel user data but still keep actual domain name, username, and database names private.cpanel-backup.shwill backup cPanel user's domain mapping for main domain, subdomains, parked domains, git repositories, cronjobs and DNS zone files if they exist.cpanel-backup.shwill also optionally sending slack channel notifications on successful or failed backup targets i.e. public_html, mail, logs, ssl or database backups. All notifications are colour coded - green = successful or red = failed for backup status for each backup target. This allows quick visual inspection of which backup targets failed their backup runs.cpanel-backup.shoptionally can upload backed up files toBackBlaze via API- first comparisontest run of
cpanel-backup.shvs cPanel pkgacct backup methods showcpanel-backup.shmethod being~29.45%faster +~6.33%smaller compressed backup files than cPanel native pkgacct method for a single cPanel user account of ~1.45GB in size for data files + MySQL databases for default set zstd compression levels incpanel-backup.shand~21.26%faster than cPanel pkgacct when using higher zstd compression levels incpanel-backup.sh+12.88%smaller compressed file backups than pkgacct method
cpanel user = cpuser1 public_html backup
mysql database backup using zstd level 3 compression
slack channel searching backup notification logs
For privacy reasons, alias masked notifictions for public demos of Slack channel notifications.
cpanel user = cpuser1 public_html backup
mysql database backup using zstd level 3 compression
Example of failed mysql backup with alias db name masking enabled. Colour coded Slack notification, red = failed backup
cpanel-backup.sh preview for single cpanel user backup mode to back cpanel user =cpuser1 with alias domain name, cPanel username and database name masking enabled only for Slack channel notifications (script outputs the real names) and default zstd compression algorithm used and zstd compressed logs.
with all backup targets enabled for public_html, logs, mail, mysql, ssl and git repositories
# backup /home/username/public_htmlBACKUP_WEBROOT='y'# backup /home/username/mailBACKUP_MAIL='y'# backup MySQL databases prefixed with cpanel username_*BACKUP_MYSQL='y'# backup /home/username/logsBACKUP_LOGS='y'# backup /home/username/sslBACKUP_SSL='y'# backup /home/username/repositoriesBACKUP_GITREPOS='y'where/home/cpuser1 size wise is ~1.1GB for files and ~1.2GB for MySQL databases
du -s /home/cpuser11058348 /home/cpuser1du -s /var/lib/mysql/cpuser1*1204552 /var/lib/mysql/cpuser1_db18 /var/lib/mysql/cpuser1_db28 /var/lib/mysql/cpuser1_db3/root/tools/cpanel-backup.sh cpuser1-------------------------------cPanel/WHM backup script 0.8for data migration to Centmin Mod LEMP stack importswritten by George Liu (centminmod.com)----------------------------------------------------------------------------------------------------------------list cpanel users--------------------------------------------------------cpuser1--------------------------------------------------------list cpuser1 domain mapping--------------------------------------------------------{ "parked_domains": [ "domain1.biz", "domain1.info" ], "addon_domains": {}, "main_domain": "domain1.com", "sub_domains": [ "ads.domain1.com", "m.domain1.com" ]}main_domain=domain1.comsub_domains:ads.domain1.comm.domain1.comparked_domains:domain1.bizdomain1.infocpuser_domainlist=domain1.com ads.domain1.com m.domain1.com domain1.biz domain1.info--------------------------------------------------------domain mapping saved:/home/backup-accounts/cpuser1/domain-map-cpuser1-040719-094323.txt----------------------------------------------------------------------------------------------------------------backup cpuser1 domain related DNS zone files--------------------------------------------------------backup /var/named/domain1.com.dbcp -af /var/named/domain1.com.db /home/backup-accounts/cpuser1/named/domain1.com.dbbackup /var/named/ads.domain1.com.dbcp -af /var/named/ads.domain1.com.db /home/backup-accounts/cpuser1/named/ads.domain1.com.dbbackup /var/named/m.domain1.com.dbcp -af /var/named/m.domain1.com.db /home/backup-accounts/cpuser1/named/m.domain1.com.dbbackup /var/named/domain1.biz.dbcp -af /var/named/domain1.biz.db /home/backup-accounts/cpuser1/named/domain1.biz.dbbackup /var/named/domain1.info.dbcp -af /var/named/domain1.info.db /home/backup-accounts/cpuser1/named/domain1.info.db--------------------------------------------------------list cpuser1 cronjobs--------------------------------------------------------#SHELL="/usr/local/cpanel/bin/jailshell"#*/15 * * * * echo "dummy cronjob"--------------------------------------------------------cpuser1 cronjobs saved:/home/backup-accounts/cpuser1/cronjobs-cpuser1-040719-094323.txt----------------------------------------------------------------------------------------------------------------backup cpanel /home/cpuser1/public_html web root--------------------------------------------------------/bin/nice -n 12 /bin/ionice -c2 -n7 tar cpf - public_html | zstd -6 -T2 -f --rsyncable > "/home/backup-accounts/cpuser1/public_html-cpuser1-040719-094323.tar.zst"backup start time: Thu Jul 4 09:43:23 UTC 2019backup end time: Thu Jul 4 09:43:25 UTC 2019backup time: 1.745zstd normal compression mode: enabledcompression level: 6compression ratio: 2.193 (22470527 / 10246463)compression ratio: 0.455 (10246463 / 22470527)slack notification sendingok--------------------------------------------------------backup cpanel /home/cpuser1/mail directory/bin/nice -n 12 /bin/ionice -c2 -n7 tar cpf - mail | zstd -6 -T2 -f --rsyncable > "/home/backup-accounts/cpuser1/mail-cpuser1-040719-094323.tar.zst"backup start time: Thu Jul 4 09:43:25 UTC 2019backup end time: Thu Jul 4 09:43:25 UTC 2019backup time: 0.008zstd normal compression mode: enabledcompression level: 6compression ratio: 81.681 (24586 / 301)compression ratio: 0.012 (301 / 24586)slack notification sendingok--------------------------------------------------------backup cpanel /home/cpuser1/logs directory/bin/nice -n 12 /bin/ionice -c2 -n7 tar cpf - logs | zstd -1 -T2 -f --rsyncable > "/home/backup-accounts/cpuser1/logs-cpuser1-040719-094323.tar.zst"backup start time: Thu Jul 4 09:43:26 UTC 2019backup end time: Thu Jul 4 09:43:40 UTC 2019backup time: 13.998zstd normal compression mode: enabledcompression level: 1compression ratio: 12.733 (1049492951 / 82422177)compression ratio: 0.078 (82422177 / 1049492951)slack notification sendingok--------------------------------------------------------backup cpanel /home/cpuser1/ssl directory/bin/nice -n 12 /bin/ionice -c2 -n7 tar cpf - ssl | zstd -6 -T2 -f --rsyncable > "/home/backup-accounts/cpuser1/ssl-cpuser1-040719-094323.tar.zst"backup start time: Thu Jul 4 09:43:40 UTC 2019backup end time: Thu Jul 4 09:43:40 UTC 2019backup time: 0.007zstd normal compression mode: enabledcompression level: 6compression ratio: 46.545 (4096 / 88)compression ratio: 0.021 (88 / 4096)slack notification sendingok--------------------------------------------------------backup cpanel /home/cpuser1/repositories directory/bin/nice -n 12 /bin/ionice -c2 -n7 tar cpf - repositories | zstd -6 -T2 -f --rsyncable > "/home/backup-accounts/cpuser1/repositories-cpuser1-040719-094323.tar.zst"backup start time: Thu Jul 4 09:43:41 UTC 2019backup end time: Thu Jul 4 09:43:41 UTC 2019backup time: 0.066zstd normal compression mode: enabledcompression level: 6compression ratio: 1.623 (932311 / 574381)compression ratio: 0.616 (574381 / 932311)slack notification sendingok--------------------------------------------------------list cpanel user: cpuser1 mysql databases--------------------------------------------------------cpuser1_db1cpuser1_db2cpuser1_db3--------------------------------------------------------list cpanel user: cpuser1 mysql usernames & grants--------------------------------------------------------mysql username: cpuser1_dbuserGrants for cpuser1_dbuser@localhostGRANT USAGE ON *.* TO 'cpuser1_dbuser'@'localhost' IDENTIFIED BY PASSWORD '*1F315FCA7B4226EFD886026773262F1429508DF3'GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, TRIGGER ON `cpuser1_db1`.* TO 'cpuser1_dbuser'@'localhost'GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, TRIGGER ON `cpuser1_db2`.* TO 'cpuser1_dbuser'@'localhost'GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, TRIGGER ON `cpuser1_db3`.* TO 'cpuser1_dbuser'@'localhost'replay commands via SSH command linemysql -e "GRANT USAGE ON *.* TO 'cpuser1_dbuser'@'localhost' IDENTIFIED BY PASSWORD '*1F315FCA7B4226EFD886026773262F1429508DF3'" mysqlmysql -e "GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, TRIGGER ON cpuser1_db1.* TO 'cpuser1_dbuser'@'localhost'" mysqlmysql -e "GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, TRIGGER ON cpuser1_db2.* TO 'cpuser1_dbuser'@'localhost'" mysqlmysql -e "GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, TRIGGER ON cpuser1_db3.* TO 'cpuser1_dbuser'@'localhost'" mysqlbackup cpanel user's mysql databases/bin/nice -n 12 /bin/ionice -c2 -n7 mysqldump --default-character-set=utf8 -Q -K --max_allowed_packet=256M --net_buffer_length=65536 --routines --events --triggers --hex-blob "cpuser1_db1" | zstd -3 -T2 -f --rsyncable > "/home/backup-accounts/cpuser1/mysqlbackup-cpuser1_db1-040719-094323.sql.zst"backup start time: Thu Jul 4 09:43:41 UTC 2019backup end time: Thu Jul 4 09:44:23 UTC 2019backup time: 42.023zstd normal compression mode: enabledcompression level: 3compression ratio: 4.309 (1232429907 / 285958928)compression ratio: 0.232 (285958928 / 1232429907)slack notification sendingok/bin/nice -n 12 /bin/ionice -c2 -n7 mysqldump --default-character-set=utf8 --single-transaction -Q -K --max_allowed_packet=256M --net_buffer_length=65536 --routines --events --triggers --hex-blob "cpuser1_db2" | zstd -3 -T2 -f --rsyncable > "/home/backup-accounts/cpuser1/mysqlbackup-cpuser1_db2-040719-094323.sql.zst"backup start time: Thu Jul 4 09:44:24 UTC 2019backup end time: Thu Jul 4 09:44:24 UTC 2019backup time: 0.014zstd normal compression mode: enabledcompression level: 3compression ratio: 7.873 (4157 / 528)compression ratio: 0.127 (528 / 4157)slack notification sendingok/bin/nice -n 12 /bin/ionice -c2 -n7 mysqldump --default-character-set=utf8 --single-transaction -Q -K --max_allowed_packet=256M --net_buffer_length=65536 --routines --events --triggers --hex-blob "cpuser1_db3" | zstd -3 -T2 -f --rsyncable > "/home/backup-accounts/cpuser1/mysqlbackup-cpuser1_db3-040719-094323.sql.zst"backup start time: Thu Jul 4 09:44:25 UTC 2019backup end time: Thu Jul 4 09:44:25 UTC 2019backup time: 0.013zstd normal compression mode: enabledcompression level: 3compression ratio: 7.888 (4157 / 527)compression ratio: 0.126 (527 / 4157)slack notification sendingok--------------------------------------------------------List cpuser1 backups at /home/backup-accounts/cpuser1--------------------------------------------------------+-- [ 76] cronjobs-cpuser1-040719-094323.txt+-- [ 190] domain-map-cpuser1-040719-094323.json+-- [ 533] domain-map-cpuser1-040719-094323.txt+-- [ 79M] logs-cpuser1-040719-094323.tar.zst+-- [ 301] mail-cpuser1-040719-094323.tar.zst+-- [273M] mysqlbackup-cpuser1_db1-040719-094323.sql.zst+-- [ 528] mysqlbackup-cpuser1_db2-040719-094323.sql.zst+-- [ 527] mysqlbackup-cpuser1_db3-040719-094323.sql.zst+-- [ 835] mysql-grants-040719-094323.log+-- [ 861] mysql-grants-ssh-040719-094323.log+-- [4.0K] named| +-- [ 0] ads.domain1.com.db| +-- [ 0] domain1.biz.db| +-- [ 0] domain1.com.db| +-- [ 0] domain1.info.db| +-- [ 0] m.domain1.com.db+-- [9.8M] public_html-cpuser1-040719-094323.tar.zst+-- [561K] repositories-cpuser1-040719-094323.tar.zst+-- [ 88] ssl-cpuser1-040719-094323.tar.zst--------------------------------------------------------pidstat stats saved at:/home/backup-accounts/logs/backup_pidstat_stats_040719-094323.log.zst--------------------------------------------------------read command:zstdcat /home/backup-accounts/logs/backup_pidstat_stats_040719-094323.log.zst--------------------------------------------------------sar usage stats saved at:/home/backup-accounts/logs/backup_sar_stats_040719-094323--------------------------------------------------------read via sar command:cpu load avgsar -q -f /home/backup-accounts/logs/backup_sar_stats_040719-094323 | sed -e "s|$(hostname)|hostname|g"09:43:23 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blockedAverage: 3 147 1.04 0.28 0.16 01min 5min 15min min:0.00 0.01 0.071min 5min 15min avg:1.04 0.28 0.161min 5min 15min max:1.96 0.56 0.251min 5min 15min 95%:1.94 0.55 0.25cpu utilisationsar -u -f /home/backup-accounts/logs/backup_sar_stats_040719-094323 | sed -e "s|$(hostname)|hostname|g"09:43:23 AM CPU %user %nice %system %iowait %steal %idleAverage: all 72.65 7.34 3.78 3.81 0.24 12.18%user %nice %system $iowait %steal %idle min:0.50 0.00 1.01 0.00 0.00 0.00%user %nice %system $iowait %steal %idle avg:72.66 7.33 3.79 3.83 0.24 12.15%user %nice %system $iowait %steal %idle max:90.36 16.92 8.00 51.79 1.55 98.49%user %nice %system $iowait %steal %idle 95%:89.85 15.66 6.31 29.02 0.52 64.84memory usagesar -r -f /home/backup-accounts/logs/backup_sar_stats_040719-094323 | sed -e "s|$(hostname)|hostname|g"09:43:23 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirtyAverage: 103236 911500 89.83 64251 441303 1372402 66.51 246046 527449 21496kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty min:70180.00 855908.00 84.35 64168.00 376364.00 1279200.00 62.00 180744.00 465176.00 760.00kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty avg:103236.44 911499.56 89.83 64251.05 441303.30 1372402.35 66.51 246046.29 527449.33 21496.44kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty max:158828.00 944556.00 93.08 64776.00 518116.00 1435092.00 69.55 302512.00 612688.00 37280.00kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 95%:142049.60 937571.20 92.39 64768.00 511324.80 1435092.00 69.55 302402.80 602478.40 35894.00disk I/O usagesar -d -f /home/backup-accounts/logs/backup_sar_stats_040719-094323 | sed -e "s|$(hostname)|hostname|g"09:43:23 AM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %utilAverage: dev253-0 176.77 61579.22 10937.98 410.24 0.30 1.74 0.28 5.00tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util min:7.00 7.00 520.00 0.00 11.81 0.00 0.00 0.00tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util avg:176.85 176.85 61571.99 10930.42 725.01 0.30 3.06 0.45tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util max:2624.00 2624.00 180435.64 76546.53 945.40 1.16 7.69 1.20tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util 95%:351.60 351.60 163840.00 67071.20 914.42 0.84 5.57 0.74--------------------------------------------------------/root/tools/cpanel-backup.sh run log saved at:/home/backup-accounts/logs/cpanel-backup-040719-094323.log.zst--------------------------------------------------------read command:zstdcat /home/backup-accounts/logs/cpanel-backup-040719-094323.log.zst--------------------------------------------------------backup completed--------------------------------------------------------total cpanel backup time: 64.361 secondsslack notification sendingok--------------------------------------------------------Inspecting pidstat zstd compressed logs
At public_html backup stage
zstdcat /home/backup-accounts/logs/backup_pidstat_stats_020719-233152.log.zstLinux 3.10.0-957.21.3.el7.x86_64 (cpanelhost.domain.com) 07/02/2019 _x86_64_ (2 CPU)# Time UID PID %usr %system %guest %CPU CPU minflt/s majflt/s VSZ RSS %MEM kB_rd/s kB_wr/s kB_ccwr/s Command 1562110313 0 9 0.00 0.97 0.00 0.97 0 0.00 0.00 0 0 0.00 0.00 0.00 0.00 rcu_sched 1562110313 0 21125 0.00 0.00 0.00 0.00 1 42.72 0.00 113704 1540 0.15 0.00 0.00 0.00 /bin/bash /root/tools/cpanel-backup.sh cpuser1 1562110313 0 21127 0.00 0.00 0.00 0.00 0 10.68 0.00 108036 784 0.08 0.00 0.00 0.00 sar -o /home/backup-accounts/logs/backup_sar_stats_020719-233152 1 1562110313 0 21128 0.00 0.97 0.00 0.97 0 318.45 0.00 108184 1052 0.10 0.00 0.00 0.00 pidstat -durhl 1 1562110313 0 21132 0.00 0.00 0.00 0.00 0 36.89 0.00 113244 996 0.10 0.00 11.65 0.00 sadc 1 -z -S ALL /home/backup-accounts/logs/backup_sar_stats_020719-233152 1562110313 0 21143 0.97 13.59 0.00 14.56 0 830.10 0.00 123532 1312 0.13 23840.78 0.00 0.00 tar cpf - public_html 1562110313 0 21144 100.00 0.97 0.00 100.00 0 8762.14 0.00 234432 35032 3.45 0.00 5231.07 0.00 zstd -6 -T2 -f --rsyncableAt start of mysql database backup stage via mysqldump
# Time UID PID %usr %system %guest %CPU CPU minflt/s majflt/s VSZ RSS %MEM kB_rd/s kB_wr/s kB_ccwr/s Command 1562110331 0 1264 0.00 1.00 0.00 1.00 0 0.00 0.00 0 0 0.00 0.00 19368.00 0.00 jbd2/vda1-8 1562110331 0 1707 0.00 0.00 0.00 0.00 0 0.00 0.00 55572 180 0.02 0.00 4.00 0.00 /sbin/auditd 1562110331 0 2201 0.00 0.00 0.00 0.00 1 2.00 0.00 21584 408 0.04 0.00 0.00 0.00 /usr/sbin/irqbalance --foreground 1562110331 997 4592 25.00 4.00 0.00 29.00 0 1019.00 7.00 517624 60604 5.97 37008.00 1784.00 1784.00 /usr/sbin/mysqld 1562110331 0 21125 0.00 0.00 0.00 0.00 0 393.00 0.00 113704 1600 0.16 4996.00 8.00 0.00 /bin/bash /root/tools/cpanel-backup.sh cpuser1 1562110331 0 21126 0.00 0.00 0.00 0.00 0 0.00 0.00 107952 672 0.07 0.00 4.00 0.00 tee /home/backup-accounts/logs/cpanel-backup-020719-233152.log 1562110331 0 21128 0.00 1.00 0.00 1.00 1 330.00 0.00 108184 1128 0.11 0.00 0.00 0.00 pidstat -durhl 1 1562110331 0 21132 0.00 0.00 0.00 0.00 0 38.00 0.00 113244 996 0.10 0.00 8.00 0.00 sadc 1 -z -S ALL /home/backup-accounts/logs/backup_sar_stats_020719-233152 1562110331 0 21396 24.00 2.00 0.00 26.00 0 1824.00 1.00 54284 4924 0.49 3980.00 0.00 0.00 mysqldump --default-character-set=utf8 -Q -K --max_allowed_packet=256M --net_buffer_length=65536 --routines --events --triggers 1562110331 0 21397 90.00 3.00 0.00 93.00 0 9848.00 0.00 211320 38196 3.76 0.00 4320.00 0.00 zstd -3 -T2 -f --rsyncableInspecting sar recorded resource usage
cpu load avearages
sar -q -f /home/backup-accounts/logs/backup_sar_stats_020719-233152 | sed -e "s|$(hostname)|hostname|g"Linux 3.10.0-957.21.3.el7.x86_64 (hostname) 07/02/2019 _x86_64_ (2 CPU)11:31:52 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked11:31:53 PM 3 147 0.00 0.06 0.16 111:31:54 PM 1 144 0.00 0.06 0.16 011:31:55 PM 1 144 0.00 0.06 0.16 011:31:56 PM 3 147 0.00 0.06 0.16 011:31:57 PM 3 147 0.00 0.06 0.16 011:31:58 PM 3 147 0.24 0.11 0.17 011:31:59 PM 0 147 0.24 0.11 0.17 211:32:00 PM 3 147 0.24 0.11 0.17 011:32:01 PM 3 147 0.24 0.11 0.17 011:32:02 PM 3 147 0.24 0.11 0.17 011:32:03 PM 0 147 0.38 0.14 0.18 211:32:04 PM 1 147 0.38 0.14 0.18 211:32:05 PM 3 148 0.38 0.14 0.18 111:32:06 PM 3 147 0.38 0.14 0.18 011:32:07 PM 3 147 0.38 0.14 0.18 011:32:08 PM 3 147 0.51 0.17 0.19 011:32:09 PM 1 144 0.51 0.17 0.19 011:32:10 PM 2 146 0.51 0.17 0.19 011:32:11 PM 2 147 0.51 0.17 0.19 111:32:12 PM 3 147 0.51 0.17 0.19 011:32:13 PM 3 147 0.87 0.25 0.22 111:32:14 PM 6 149 0.87 0.25 0.22 111:32:15 PM 4 148 0.87 0.25 0.22 211:32:16 PM 1 150 0.87 0.25 0.22 311:32:17 PM 3 149 0.87 0.25 0.22 011:32:18 PM 5 149 1.12 0.31 0.24 011:32:19 PM 3 149 1.12 0.31 0.24 011:32:20 PM 3 149 1.12 0.31 0.24 011:32:21 PM 3 149 1.12 0.31 0.24 011:32:22 PM 3 149 1.12 0.31 0.24 011:32:23 PM 3 149 1.35 0.38 0.26 011:32:24 PM 2 149 1.35 0.38 0.26 011:32:25 PM 4 149 1.35 0.38 0.26 011:32:26 PM 4 147 1.35 0.38 0.26 011:32:27 PM 2 147 1.35 0.38 0.26 011:32:28 PM 3 147 1.49 0.42 0.27 011:32:29 PM 3 147 1.49 0.42 0.27 011:32:30 PM 3 147 1.49 0.42 0.27 011:32:31 PM 3 147 1.49 0.42 0.27 011:32:32 PM 3 147 1.49 0.42 0.27 011:32:33 PM 2 147 1.61 0.46 0.29 011:32:34 PM 3 149 1.61 0.46 0.29 011:32:35 PM 4 149 1.61 0.46 0.29 011:32:36 PM 1 149 1.61 0.46 0.29 011:32:37 PM 4 149 1.61 0.46 0.29 011:32:38 PM 4 149 1.64 0.49 0.30 011:32:39 PM 3 149 1.64 0.49 0.30 011:32:40 PM 2 149 1.64 0.49 0.30 011:32:41 PM 2 149 1.64 0.49 0.30 011:32:42 PM 3 149 1.64 0.49 0.30 011:32:43 PM 3 149 1.75 0.53 0.31 011:32:44 PM 3 149 1.75 0.53 0.31 011:32:45 PM 2 147 1.75 0.53 0.31 011:32:46 PM 4 147 1.75 0.53 0.31 011:32:47 PM 1 146 1.75 0.53 0.31 011:32:48 PM 0 144 1.61 0.52 0.31 011:32:49 PM 0 144 1.61 0.52 0.31 011:32:50 PM 0 144 1.61 0.52 0.31 0Average: 3 147 1.03 0.31 0.24 0sar cpu load average extended stats for min, avg, max and 95th percentile values
cat /home/backup-accounts/logs/sar-cpuload-data-formatted-020719-233152.log 1min 5min 15min min:0.00 0.06 0.161min 5min 15min avg:1.03 0.31 0.241min 5min 15min max:1.75 0.53 0.311min 5min 15min 95%:1.75 0.53 0.31sar disk I/O usage extended stats for min, avg, max and 95% percentile values
cat /home/backup-accounts/logs/sar-disk-data-formatted-030719-002643.logtps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util min:0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util avg:166.84 166.84 132239.08 9618.89 790.93 0.66 3.63 0.56tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util max:279.00 279.00 194312.87 57968.00 914.07 2.34 11.32 1.06tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util 95%:231.06 231.06 181390.35 55487.20 910.22 1.20 5.81 0.88cpanel-backup.sh optionally supports uploading backed up cPanel files to Backblaze API
# backblaze api settingsBB_ENABLE='n'BB_DEBUG='n'BB_APPKEYID=''BB_APPKEY=''example excerpt from backblaze API upload routine section withBB_DEBUG='y' set for more verbose output
--------------------------------------------------------List cpuser2 backups at /home/backup-accounts/cpuser2--------------------------------------------------------+-- [ 76] cronjobs-cpuser2-040719-193257.txt+-- [ 190] domain-map-cpuser2-040719-193257.json+-- [ 533] domain-map-cpuser2-040719-193257.txt+-- [ 115] mail-cpuser2-040719-193257.tar.zst+-- [ 529] mysqlbackup-cpuser2_db1-040719-193257.sql.zst+-- [ 528] mysqlbackup-cpuser2_db2-040719-193257.sql.zst+-- [ 528] mysqlbackup-cpuser2_db3-040719-193257.sql.zst+-- [ 835] mysql-grants-040719-193257.log+-- [ 861] mysql-grants-ssh-040719-193257.log+-- [4.0K] named| +-- [ 0] ads.domain2.com-040719-193257.db| +-- [ 0] domain2.biz-040719-193257.db| +-- [ 0] domain2.com-040719-193257.db| +-- [ 0] domain2.info-040719-193257.db| +-- [ 0] m.domain2.com-040719-193257.db+-- [ 122] public_html-cpuser2-040719-193257.tar.zst+-- [561K] repositories-cpuser2-040719-193257.tar.zst+-- [ 88] ssl-cpuser2-040719-193257.tar.zst--------------------------------------------------------b2 upload_file --threads 2 B2BUCKETNAME /home/backup-accounts/cpuser2/domain-map-cpuser2-040719-193257.txt cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/domain-map-cpuser2-040719-193257.txt--------------------------------------------------------/home/backup-accounts/cpuser2/domain-map-cpuser2-040719-193257.txt: 100%|██████████| 533/533 [00:01<00:00 356B/s] fileName: cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/domain-map-cpuser2-040719-193257.txt size: 533 uploadTimestamp: 1562268783000--------------------------------------------------------b2 upload_file --threads 2 B2BUCKETNAME /home/backup-accounts/cpuser2/mysqlbackup-cpuser2_db1-040719-193257.sql.zst cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/mysqlbackup-cpuser2_db1-040719-193257.sql.zst--------------------------------------------------------/home/backup-accounts/cpuser2/mysqlbackup-cpuser2_db1-040719-193257.sql.zst: 100%|██████████| 529/529 [00:01<00:00 334B/s] fileName: cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/mysqlbackup-cpuser2_db1-040719-193257.sql.zst size: 529 uploadTimestamp: 1562268785000--------------------------------------------------------b2 upload_file --threads 2 B2BUCKETNAME /home/backup-accounts/cpuser2/cronjobs-cpuser2-040719-193257.txt cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/cronjobs-cpuser2-040719-193257.txt--------------------------------------------------------/home/backup-accounts/cpuser2/cronjobs-cpuser2-040719-193257.txt: 100%|██████████| 76.0/76.0 [00:01<00:00 50.5B/s] fileName: cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/cronjobs-cpuser2-040719-193257.txt size: 76 uploadTimestamp: 1562268788000--------------------------------------------------------b2 upload_file --threads 2 B2BUCKETNAME /home/backup-accounts/cpuser2/mysql-grants-ssh-040719-193257.log cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/mysql-grants-ssh-040719-193257.log--------------------------------------------------------/home/backup-accounts/cpuser2/mysql-grants-ssh-040719-193257.log: 100%|██████████| 861/861 [00:01<00:00 544B/s] fileName: cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/mysql-grants-ssh-040719-193257.log size: 861 uploadTimestamp: 1562268790000--------------------------------------------------------b2 upload_file --threads 2 B2BUCKETNAME /home/backup-accounts/cpuser2/mysqlbackup-cpuser2_db3-040719-193257.sql.zst cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/mysqlbackup-cpuser2_db3-040719-193257.sql.zst--------------------------------------------------------/home/backup-accounts/cpuser2/mysqlbackup-cpuser2_db3-040719-193257.sql.zst: 100%|██████████| 528/528 [00:01<00:00 377B/s] fileName: cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/mysqlbackup-cpuser2_db3-040719-193257.sql.zst size: 528 uploadTimestamp: 1562268793000--------------------------------------------------------b2 upload_file --threads 2 B2BUCKETNAME /home/backup-accounts/cpuser2/ssl-cpuser2-040719-193257.tar.zst cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/ssl-cpuser2-040719-193257.tar.zst--------------------------------------------------------/home/backup-accounts/cpuser2/ssl-cpuser2-040719-193257.tar.zst: 100%|██████████| 88.0/88.0 [00:01<00:00 61.5B/s] fileName: cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/ssl-cpuser2-040719-193257.tar.zst size: 88 uploadTimestamp: 1562268795000--------------------------------------------------------b2 upload_file --threads 2 B2BUCKETNAME /home/backup-accounts/cpuser2/mail-cpuser2-040719-193257.tar.zst cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/mail-cpuser2-040719-193257.tar.zst--------------------------------------------------------/home/backup-accounts/cpuser2/mail-cpuser2-040719-193257.tar.zst: 100%|██████████| 115/115 [00:01<00:00 73.6B/s] fileName: cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/mail-cpuser2-040719-193257.tar.zst size: 115 uploadTimestamp: 1562268798000--------------------------------------------------------b2 upload_file --threads 2 B2BUCKETNAME /home/backup-accounts/cpuser2/domain-map-cpuser2-040719-193257.json cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/domain-map-cpuser2-040719-193257.json--------------------------------------------------------/home/backup-accounts/cpuser2/domain-map-cpuser2-040719-193257.json: 100%|██████████| 190/190 [00:01<00:00 136B/s] fileName: cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/domain-map-cpuser2-040719-193257.json size: 190 uploadTimestamp: 1562268800000--------------------------------------------------------b2 upload_file --threads 2 B2BUCKETNAME /home/backup-accounts/cpuser2/public_html-cpuser2-040719-193257.tar.zst cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/public_html-cpuser2-040719-193257.tar.zst--------------------------------------------------------/home/backup-accounts/cpuser2/public_html-cpuser2-040719-193257.tar.zst: 100%|██████████| 122/122 [00:01<00:00 86.6B/s] fileName: cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/public_html-cpuser2-040719-193257.tar.zst size: 122 uploadTimestamp: 1562268803000--------------------------------------------------------b2 upload_file --threads 2 B2BUCKETNAME /home/backup-accounts/cpuser2/mysqlbackup-cpuser2_db2-040719-193257.sql.zst cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/mysqlbackup-cpuser2_db2-040719-193257.sql.zst--------------------------------------------------------/home/backup-accounts/cpuser2/mysqlbackup-cpuser2_db2-040719-193257.sql.zst: 100%|██████████| 528/528 [00:01<00:00 369B/s] fileName: cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/mysqlbackup-cpuser2_db2-040719-193257.sql.zst size: 528 uploadTimestamp: 1562268808000--------------------------------------------------------b2 upload_file --threads 2 B2BUCKETNAME /home/backup-accounts/cpuser2/repositories-cpuser2-040719-193257.tar.zst cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/repositories-cpuser2-040719-193257.tar.zst--------------------------------------------------------/home/backup-accounts/cpuser2/repositories-cpuser2-040719-193257.tar.zst: 100%|██████████| 574k/574k [00:03<00:00 190kB/s] fileName: cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/repositories-cpuser2-040719-193257.tar.zst size: 574302 uploadTimestamp: 1562268810000--------------------------------------------------------b2 upload_file --threads 2 B2BUCKETNAME /home/backup-accounts/cpuser2/mysql-grants-040719-193257.log cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/mysql-grants-040719-193257.log--------------------------------------------------------/home/backup-accounts/cpuser2/mysql-grants-040719-193257.log: 100%|██████████| 835/835 [00:01<00:00 617B/s] fileName: cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/mysql-grants-040719-193257.log size: 835 uploadTimestamp: 1562268814000--------------------------------------------------------b2 upload_file --threads 2 B2BUCKETNAME /home/backup-accounts/cpuser2/named/domain2.info-040719-193257.db cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/named/domain2.info-040719-193257.db--------------------------------------------------------/home/backup-accounts/cpuser2/named/domain2.info-040719-193257.db: 0.00B [00:00 ?B/s] fileName: cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/named/domain2.info-040719-193257.db size: 0 uploadTimestamp: 1562268816000--------------------------------------------------------b2 upload_file --threads 2 B2BUCKETNAME /home/backup-accounts/cpuser2/named/domain2.com-040719-193257.db cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/named/domain2.com-040719-193257.db--------------------------------------------------------/home/backup-accounts/cpuser2/named/domain2.com-040719-193257.db: 0.00B [00:00 ?B/s] fileName: cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/named/domain2.com-040719-193257.db size: 0 uploadTimestamp: 1562268832000--------------------------------------------------------b2 upload_file --threads 2 B2BUCKETNAME /home/backup-accounts/cpuser2/named/m.domain2.com-040719-193257.db cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/named/m.domain2.com-040719-193257.db--------------------------------------------------------/home/backup-accounts/cpuser2/named/m.domain2.com-040719-193257.db: 0.00B [00:00 ?B/s] fileName: cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/named/m.domain2.com-040719-193257.db size: 0 uploadTimestamp: 1562268834000--------------------------------------------------------b2 upload_file --threads 2 B2BUCKETNAME /home/backup-accounts/cpuser2/named/domain2.biz-040719-193257.db cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/named/domain2.biz-040719-193257.db--------------------------------------------------------/home/backup-accounts/cpuser2/named/domain2.biz-040719-193257.db: 0.00B [00:00 ?B/s] fileName: cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/named/domain2.biz-040719-193257.db size: 0 uploadTimestamp: 1562268836000--------------------------------------------------------b2 upload_file --threads 2 B2BUCKETNAME /home/backup-accounts/cpuser2/named/ads.domain2.com-040719-193257.db cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/named/ads.domain2.com-040719-193257.db--------------------------------------------------------/home/backup-accounts/cpuser2/named/ads.domain2.com-040719-193257.db: 0.00B [00:00 ?B/s] fileName: cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/named/ads.domain2.com-040719-193257.db size: 0 uploadTimestamp: 1562268840000--------------------------------------------------------list b2 bucket uploaded files--------------------------------------------------------2019-07-04 19:33:08 76 cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/cronjobs-cpuser2-040719-193257.txt2019-07-04 19:33:20 190 cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/domain-map-cpuser2-040719-193257.json2019-07-04 19:33:03 533 cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/domain-map-cpuser2-040719-193257.txt2019-07-04 19:33:18 115 cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/mail-cpuser2-040719-193257.tar.zst2019-07-04 19:33:34 835 cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/mysql-grants-040719-193257.log2019-07-04 19:33:10 861 cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/mysql-grants-ssh-040719-193257.log2019-07-04 19:33:05 529 cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/mysqlbackup-cpuser2_db1-040719-193257.sql.zst2019-07-04 19:33:28 528 cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/mysqlbackup-cpuser2_db2-040719-193257.sql.zst2019-07-04 19:33:13 528 cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/mysqlbackup-cpuser2_db3-040719-193257.sql.zst2019-07-04 19:34:00 0 cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/named/ads.domain2.com-040719-193257.db2019-07-04 19:33:56 0 cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/named/domain2.biz-040719-193257.db2019-07-04 19:33:52 0 cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/named/domain2.com-040719-193257.db2019-07-04 19:33:36 0 cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/named/domain2.info-040719-193257.db2019-07-04 19:33:54 0 cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/named/m.domain2.com-040719-193257.db2019-07-04 19:33:23 122 cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/public_html-cpuser2-040719-193257.tar.zst2019-07-04 19:33:30 574302 cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/repositories-cpuser2-040719-193257.tar.zst2019-07-04 19:33:15 88 cpanelhost.domain.com/cpanel-backup/cpuser2/040719-193257/ssl-cpuser2-040719-193257.tar.zst