Backups
Server Backups
AllStarLink, Inc uses a per server backup method based on borg.
Installing Borg on Ubuntu 16
the default package that ships with ubuntu 16.04 LTS is borg 1.0 based and we use the latest 1.1 version as it has several security fixes.
Install nessary packages
apt-get install libacl1-dev python3-dev libssl-dev gcc g++
install pip
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py python3 get-pip.py
Install Borg
pip3 install borgbackup
Test for proper version
root:~# borg -V borg 1.1.7
which borg /usr/local/bin/borg
Install the scripts
This will install a new ssh key for root that matches the Rsync.net account and place the borg.inc program in the /root/ directory.
cd / tar -xvf /root/borg-root-config.tar
Edit the borg script
On the Db servers we add a command to dump the database to /var/mysql-backup-current.sql.bz2
Initialize the repo
Copy the export lines to your shell and run then do
echo $BORG_REPO ASLUSER@host.rsync.net:borg/ASL/db-ord
The borg/ASL/db-ord needs to be created on the server
ssh -t ASLUSER@host.rsync.net mkdir -p borg/ASL/db-ord
now we need to init the repo at that location
borg init -e keyfile-blake2 -p -v
export the key
borg key export --paper
Email this output GPG encrypted to the admin team members.
If we lose this key, there is no way to restore the backup. This means if the server dies, we need the paper key record and the passphrase.
Do the first backup
/root/borg.inc
Check that it's succeed
move borg.inc to cron
mv /root/borg.inc /srv/borg.sh
edit crontab
edit /etc/crontab to run daily at 8am UTC
#borg backup 30 8 * * * root /srv/borg.sh
Reload the crontab file
service cron reload
Mounting and restoring
Borg makes it easy to mount a backup on the server using "borgfs".
If this is done on a different server or during a restore operation on new servers, the key files need to be imported from the paper key. Selected admin users have the paper keys backed up in encrypted email.
For example on a our db-fnt server:
Take the first 5 lines from the /srv/borg.sh file and export them on the cli:
export BORG_REMOTE_PATH=/usr/local/bin/borg1/borg1 export HOST= export BORG_REPO= export BORG_PASSPHRASE= export BORG=/usr/local/bin/borg
Note if doing this on a different server, BORG_REPO= must be the server you intend to restore from.
#import the paper key borg key import --paper $BORG_REPO #make a directory to mount the backups on: mkdir /mnt/backups #now mount the borg repo borgfs $BORG_REPO /mnt/backups/
The files will now be in /mnt/backups:
ls /mnt/backups/ db-fnt.allstarlink.org-2018-10-31_08:30 db-fnt.allstarlink.org-2019-02-28_08:30 db-fnt.allstarlink.org-2018-11-30_08:30 db-fnt.allstarlink.org-2019-03-02_08:30 db-fnt.allstarlink.org-2018-12-31_08:30 db-fnt.allstarlink.org-2019-03-03_08:30 db-fnt.allstarlink.org-2019-01-31_08:30 db-fnt.allstarlink.org-2019-03-04_08:30 db-fnt.allstarlink.org-2019-02-03_08:30 db-fnt.allstarlink.org-2019-03-05_08:30 db-fnt.allstarlink.org-2019-02-10_08:30 db-fnt.allstarlink.org-2019-03-06_08:30 db-fnt.allstarlink.org-2019-02-17_08:30 db-fnt.allstarlink.org-2019-03-07_08:30 db-fnt.allstarlink.org-2019-02-24_08:30 db-fnt.allstarlink.org-2019-03-08_08:30
You can now go into any of these directories and restore files as of the date for the respective directory using normal UNIX utilities.