Backing up to Yandex Object Storage with Bacula
You can use Bacula
Bacula consists of multiple components:
- Bacula Director: Manages backup and recovery.
- File Daemon: Provides access to backup files.
- Storage Daemon: Reads files and writes them to the hard disk.
- Catalog: Maintains the backup file catalog that resides in the MariaDB
database. - Bacula Console: Management console for working with the Bacula Director.
To configure Bacula backup and recovery:
- Get your cloud ready.
- Create a VM.
- Configure the AWS CLI.
- Install Bacula and additional components.
- Configure MariaDB.
- Configure a storage.
- Configure the Bacula components.
- Run a backup job.
- Recover the files.
If you no longer need the resources you created, delete them.
Getting started
Sign up for Yandex Cloud and create a billing account:
- Navigate to the management console
and log in to Yandex Cloud or create a new account. - On the Yandex Cloud Billing
page, make sure you have a billing account linked and it has theACTIVEorTRIAL_ACTIVEstatus. If you do not have a billing account, create one and link a cloud to it.
If you have an active billing account, you can navigate to the cloud page
Learn more about clouds and folders here.
Required paid resources
The backup and recovery support costs include:
- Fee for VM computing resources and disks (see Yandex Compute Cloud pricing).
- Fee for bucket data storage and data operations (see Yandex Object Storage pricing).
- Fee for using a dynamic or static external IP address (see Yandex Virtual Private Cloud pricing).
Create a bucket
To create a backup bucket in Object Storage:
- Navigate to the Yandex Cloud management console
and select the folder to work with. - On the folder dashboard, click Create resource and select Bucket.
- In the ** Name** field, specify the bucket name.
- Set public access to read objects in the bucket, get an object list, and read bucket settings, i.e.,
With authorization. - In the Storage class field, select Cold.
- Click Create bucket.
Create a service account
Create a service account and assign it the editor role.
Create static access keys
Create static access keys.
Save key_id and the secret key right away. You will not be able to get the key again.
Create a VM
To create a VM:
-
In the management console
, select the folder where you want to create your VM. -
In the list of services, select Compute Cloud.
-
In the left-hand panel, select
Virtual machines. -
Click Create virtual machine.
-
Select Advanced setup.
-
Under Boot disk image, select a public CentOS 7 image.
-
Under Location, select the availability zone where your VM will reside.
-
Under Computing resources, navigate to the Custom tab and specify these settings:
- Platform:
Intel Ice Lake - vCPU:
2 - Guaranteed vCPU performance:
20% - RAM:
2 GB
- Platform:
-
Under Network settings:
-
In the Subnet field, enter the ID of a subnet in the new VM’s availability zone. Alternatively, select a cloud network from the list.
-
Each network must have at least one subnet. If there is no subnet, create one by selecting Create subnet.
-
If you do not have a network, click Create network to create one:
- In the window that opens, specify the network name and select the folder where it will be created.
- Optionally, enable the Create subnets setting to automatically create subnets in all availability zones.
- Click Create network.
-
-
In the Public IP address field, select
Autoto assign a random external IP address from the Yandex Cloud pool to the VM. Alternatively, select a static address from the list if you reserved one.
-
-
Under Access, select SSH key and specify the VM access credentials:
-
In the Login field, enter a name for the user you want to create on the VM, e.g.,
yc-user.Alert
Do not use
rootor other reserved usernames. To perform operations requiring root privileges, use thesudocommand. -
In the SSH key field, select the SSH key saved in your organization user profile.
If there are no SSH keys in your profile or you want to add a new key:
-
Click Add key.
-
Enter a name for the SSH key.
-
Select one of the following:
-
Enter manually: Paste the contents of the public SSH key. You need to create an SSH key pair on your own. -
Load from file: Upload the public part of the SSH key. You need to create an SSH key pair on your own. -
Generate key: Automatically create an SSH key pair.When adding a new SSH key, an archive containing the key pair will be created and downloaded. In Linux or macOS-based operating systems, unpack the archive to the
/home/<user_name>/.sshdirectory. In Windows, unpack the archive to theC:\Users\<user_name>/.sshdirectory. You do not need additionally enter the public key in the management console.
-
-
Click Add.
The system will add the SSH key to your organization user profile. If the organization has disabled the ability for users to add SSH keys to their profiles, the added public SSH key will only be saved in the user profile inside the newly created resource.
-
-
-
Under General information, specify the VM name:
bacula-vm. -
Click Create VM.
-
Wait until the VM status switches to
RUNNING.
Configure the AWS CLI
To configure the AWS CLI on bacula-vm:
-
In the management console
, navigate to the VM page and get the VM’s public IP address. -
Connect to the VM over SSH.
We recommend using a key pair when authenticating over SSH. Configure the generated key pair so that the private key matches the public key sent to the VM.
-
Update the installed packages by running this command in the terminal:
yum update -y -
Install the AWS CLI:
yum install awscli -y -
Configure the AWS CLI:
sudo aws configureSpecify these settings:
AWS Access Key ID:key_idyou got when creating the static key.AWS Secret Access Key: Thesecretkey you got when creating the static key.Default region name:ru-central1.Default output format:json.
-
Make sure the
key_idandsecretvalues in the/root/.aws/credentialsfile are correct:sudo cat /root/.aws/credentials -
Make sure the
Default region nameandDefault output formatvalues in the/root/.aws/configfile are correct:sudo cat /root/.aws/config
Install Bacula and additional components
-
Install the Bacula components:
sudo yum install -y bacula-director bacula-storage bacula-console bacula-client -
Install MariaDB
:sudo yum install -y mariadb-server -
Install the
s3fstool to mount the Object Storage bucket to the file system:sudo yum install -y epel-release sudo yum install -y s3fs-fuse -
Install the
nanotext editor:sudo yum install -y nano
Configure MariaDB
-
Run MariaDB:
sudo systemctl start mariadb -
Make sure MariaDB is running:
sudo systemctl status mariadb | grep Active -
Configure MariaDB to run at system startup:
sudo systemctl enable mariadb -
Create database tables and configure access permissions:
/usr/libexec/bacula/grant_mysql_privileges /usr/libexec/bacula/create_mysql_database -u root /usr/libexec/bacula/make_mysql_tables -u bacula -
Configure database security:
sudo mysql_secure_installationFor these queries:
Enter current password for root (enter for none): Press Enter to skip this field.Set root password? [Y/n]: EnterY, specify the root password, and confirm it. You will need the password at the next step.Remove anonymous users? [Y/n]: Press Enter to accept the default value.Disallow root login remotely? [Y/n]: Press Enter to accept the default value.Remove test database and access to it? [Y/n]: Press Enter to accept the default value.Reload privilege tables now? [Y/n]: Press Enter to accept the default value.
-
Log in to the database and enter the
rootpassword you created at the previous step:mysql -u root -p -
Create
bacula_db_passwordfor thebaculauser:UPDATE mysql.user SET Password=PASSWORD('bacula_db_password') WHERE User='bacula'; FLUSH PRIVILEGES; exit -
Configure the MySQL library for Bacula:
sudo alternatives --config libbaccats.soEnter
1to select MySQL:Selection Command ----------------------------------------------- 1 /usr/lib64/libbaccats-mysql.so 2 /usr/lib64/libbaccats-sqlite3.so *+ 3 /usr/lib64/libbaccats-postgresql.so Enter to keep the current selection[+], or type selection number: 1
Configure a storage
Prepare a backup folder
-
Create a backup folder named
/tmp/bacula:sudo mkdir /tmp/bacula -
Configure access permissions for the
/tmp/baculafolder:sudo chown -R bacula:bacula /tmp/bacula sudo chmod -R 700 /tmp/bacula sudo semanage permissive -a bacula_t
Mount the bucket to the file system
-
To upload backups to Object Storage, use
s3fsto mount the bucket by running the command below and specifying the bucket name:sudo s3fs <bucket_name> /tmp/bacula \ -o url=https://storage.yandexcloud.net \ -o use_path_request_style \ -o allow_other \ -o nonempty \ -o uid=133,gid=133,mp_umask=077Where:
uid=133:baculauser ID from the/etc/passwdfile.gid=133:baculagroup ID from the/etc/passwdfile.
-
Check the access permissions for the
/tmp/baculafolder:sudo ls -la /tmp/bacula/Result:
drwx------. 2 bacula bacula 31 Sep 18 09:16 . drwxrwxrwt. 10 root root 265 Sep 18 08:59 .. -
Make sure the
baculauser can create files in the/tmp/baculafolder:-
Enable the
bashshell for thebaculauser:sudo sed -i "/^bacula/ s@/sbin/nologin@/bin/bash@" /etc/passwd -
In the
/tmp/baculafolder, create a file:sudo runuser -l bacula -c 'touch /tmp/bacula/test.test' -
Make sure you have created the
test.testfile in the/tmp/baculafolder:sudo ls -la /tmp/bacula | grep test.test -
On the folder dashboard of the management console
, select Object Storage and check whether thetest.testfile is in the bucket. -
Delete the test file:
sudo runuser -l bacula -c 'rm -f /tmp/bacula/test.test' -
Disable the
bashshell for thebaculauser:sudo sed -i "/^bacula/ s@/bin/bash@/sbin/nologin@" /etc/passwd
-
Configure the Bacula components
Configure the Bacula Director
-
Open the Bacula Director configuration file:
sudo nano /etc/bacula/bacula-dir.conf -
To set up a connection to the Bacula Director, add the
DirAddress = 127.0.0.1line in theDirectorconfiguration section:... Director { # define myself Name = bacula-dir DIRport = 9101 # Specify the port (a positive integer) on which the Director daemon will listen for Bacula Console connections. # This same port number must be specified in the Director resource of the Console configuration file. # The default is 9101, so normally this directive need not be specified. # This directive should not be used if you specify the DirAddresses (plural) directive. QueryFile = "/etc/bacula/query.sql" WorkingDirectory = "/var/spool/bacula" PidDirectory = "/var/run" Maximum Concurrent Jobs = 1 Password = "@@DIR_PASSWORD@@" # Console password Messages = Daemon DirAddress = 127.0.0.1 } ... -
For your convenience, rename the
BackupClient1task toBackupFiles:... Job { Name = "BackupFiles" JobDefs = "DefaultJob" } ... -
To assign
/tmp/bacula-restoresas the folder for your recovered files, add theWhere = /tmp/bacula-restoresline to theRestoreFilesjob configuration section:... Job { Name = "RestoreFiles" Type = Restore Client=bacula-fd FileSet="Full Set" Storage = File Pool = Default Messages = Standard Where = /tmp/bacula-restores } ... -
Under
Include, in theFileSetconfiguration section namedFull Set:- Add the
compression = GZIPline to theOptionssection to enable backup compression. - Specify
File = /to back up the entire file system.
... FileSet { Name = "Full Set" Include { Options { signature = MD5 compression = GZIP } File = / } Exclude { File = /var/spool/bacula File = /tmp File = /proc File = /tmp File = /.journal File = /.fsck } } ... - Add the
-
In the management console
, navigate to the VM page and get the VM’s internal IP address. -
To set up an outbound connection to the Storage Daemon, specify the VM’s internal IP address in the
Addressfield of theStorageconfiguration section:... Storage { Name = File # Do not use "localhost" here Address = <VM_internal_IP_address> # N.B. Use a fully qualified name here SDPort = 9103 Password = "@@SD_PASSWORD@@" Device = FileStorage Media Type = File } ... -
To connect to the DB, specify
dbpassword = "bacula_db_password"that you created when setting up MariaDB in theCatalogconfiguration section:... # Generic catalog service Catalog { Name = MyCatalog # Uncomment the following line if you want the dbi driver # dbdriver = "dbi:postgresql"; dbaddress = 127.0.0.1; dbport = dbname = "bacula"; dbuser = "bacula"; dbpassword = "bacula_db_password" } ... -
Save the file.
-
Make sure the
bacula-dir.conffile has no syntax errors:sudo bacula-dir -tc /etc/bacula/bacula-dir.confIf there are no error messages, the configuration is correct.
Configure Storage Daemon
-
Open the Storage Daemon configuration file:
sudo nano /etc/bacula/bacula-sd.conf -
To set up an outbound connection to the Storage Daemon, specify the VM’s internal IP address in the
SDAddressfield of theStorageconfiguration section:... Storage { # definition of myself Name = BackupServer-sd SDPort = 9103 # Specifies port number on which the Storage daemon listens for Director connections. The default is 9103. WorkingDirectory = "/var/spool/bacula" Pid Directory = "/var/run/bacula" Maximum Concurrent Jobs = 20 SDAddress = <VM_internal_IP_address> # This directive is optional, and if it is specified, # it will cause the Storage daemon server (for Director and File daemon connections) to bind to the specified IP-Address, # which is either a domain name or an IP address specified as a dotted quadruple. # If this directive is not specified, the Storage daemon will bind to any available address (the default). } ... -
In the
Deviceconfiguration section, specifyArchive Device = /tmp/baculaas the backup folder:... Device { Name = FileStorage Media Type = File Archive Device = /tmp/bacula LabelMedia = yes; # lets Bacula label unlabeled media Random Access = Yes; AutomaticMount = yes; # when device opened, read it RemovableMedia = no; AlwaysOpen = no; } ... -
Save the file.
-
Make sure the
bacula-sd.conffile has no syntax errors:sudo bacula-sd -tc /etc/bacula/bacula-sd.confIf there are no error messages, the configuration is correct.
Create passwords for the Bacula components
The Bacula Director, the Storage Daemon, and the File Daemon all use passwords for cross-component authentication.
To set passwords for the Bacula components:
-
Generate passwords for the Bacula Director, the Storage Daemon, and the File Daemon:
DIR_PASSWORD=`date +%s | sha256sum | base64 | head -c 33` SD_PASSWORD=`date +%s | sha256sum | base64 | head -c 33` FD_PASSWORD=`date +%s | sha256sum | base64 | head -c 33` -
Store the passwords in the configuration files:
sudo sed -i "s/@@DIR_PASSWORD@@/${DIR_PASSWORD}/" /etc/bacula/bacula-dir.conf sudo sed -i "s/@@DIR_PASSWORD@@/${DIR_PASSWORD}/" /etc/bacula/bconsole.conf sudo sed -i "s/@@SD_PASSWORD@@/${SD_PASSWORD}/" /etc/bacula/bacula-sd.conf sudo sed -i "s/@@SD_PASSWORD@@/${SD_PASSWORD}/" /etc/bacula/bacula-dir.conf sudo sed -i "s/@@FD_PASSWORD@@/${FD_PASSWORD}/" /etc/bacula/bacula-dir.conf sudo sed -i "s/@@FD_PASSWORD@@/${FD_PASSWORD}/" /etc/bacula/bacula-fd.conf
Run the Bacula components
-
Start the Bacula components:
sudo systemctl start bacula-dir sudo systemctl start bacula-sd sudo systemctl start bacula-fd -
Make sure the Bacula components are running:
sudo systemctl status bacula-dir sudo systemctl status bacula-sd sudo systemctl status bacula-fd -
Configure the Bacula components to run at system startup:
sudo systemctl enable bacula-dir sudo systemctl enable bacula-sd sudo systemctl enable bacula-fd
Run a backup job
-
Open the Bacula Console:
sudo bconsole -
To set up a backup profile, create a label:
label -
Name the new volume as
MyVolume:Enter new Volume name: MyVolume -
To select the
Filepool, enter2:Defined Pools: 1: Default 2: File 3: Scratch Select the Pool (1-3): 2 -
Run a backup job:
runTo run the
BackupFilesjob, enter1:A job name must be specified. The defined Job resources are: 1: BackupFiles 2: BackupCatalog 3: RestoreFiles Select Job resource (1-3): 1To confirm, enter
yes:OK to run? (yes/mod/no): yes -
Check the backup status:
status directorResult if the backup is in progress:
Running Jobs: Console connected at 12-Sep-19 07:22 JobId Level Name Status ====================================================================== 2 Full BackupFiles.2019-09-12_07.22.56_03 is runningResult if the backup is complete:
Running Jobs: Console connected at 12-Sep-19 07:25 No Jobs running. ==== Terminated Jobs: JobId Level Files Bytes Status Finished Name ==================================================================== 2 Full 32,776 483.6 M OK 12-Sep-19 07:24 BackupFiles -
Wait for the backup to complete and exit the Bacula Console:
exit
Check the backup
To make sure the backup has been completed:
- On the folder dashboard of the management console
, select Object Storage. - Open the bucket.
- Make sure it contains the
MyVolumeobject.
Recover the files
-
To test recovery, begin by deleting a file, e.g., the
pingtool:sudo rm -f /bin/ping -
Make sure you have deleted the
pingtool:pingResult:
bash: ping: command not found -
Log in to the Bacula Console:
sudo bconsole -
Run a full recovery:
restore allTo run a recovery from the most recent backup, enter
5:To select the JobIds, you have the following choices: 1: List last 20 Jobs run 2: List Jobs where a given File is saved 3: Enter list of comma separated JobIds to select 4: Enter SQL list command 5: Select the most recent backup for a client 6: Select backup for a client before a specified time 7: Enter a list of files to restore 8: Enter a list of files to restore before a specified time 9: Find the JobIds of the most recent backup for a client 10: Find the JobIds for a backup for a client before a specified time 11: Enter a list of directories to restore for found JobIds 12: Select full restore to a specified Job date 13: Cancel Select item: (1-13): 5To confirm full recovery, enter
done:You are now entering file selection mode where you add (mark) and remove (unmark) files to be restored. No files are initially added, unless you used the "all" keyword on the command line. Enter "done" to leave this mode. cwd is: / doneTo confirm running the recovery, enter
yes:OK to run? (yes/mod/no): yes -
Check the recovery status:
status directorResult if the recovery is in progress:
Running Jobs: Console connected at 12-Sep-19 07:25 JobId Level Name Status ====================================================================== 3 RestoreFiles.2019-09-12_07.27.42_05 is runningResult if the recovery is complete:
Terminated Jobs: JobId Level Files Bytes Status Finished Name ==================================================================== 2 Full 32,776 483.6 M OK 12-Sep-19 07:24 BackupFiles 3 32,776 1.136 G OK 12-Sep-19 07:27 RestoreFiles -
Wait for the recovery to complete and exit the Bacula Console:
exit
Check the files you recovered
-
Check whether the
/tmp/bacula-restoresfolder contains the files you recovered:sudo ls -la /tmp/bacula-restoresResult:
total 16 dr-xr-xr-x. 15 root root 201 Sep 12 07:09 . drwx------. 4 bacula bacula 35 Sep 12 07:09 .. lrwxrwxrwx 1 root root 7 Sep 12 07:27 bin -> usr/bin dr-xr-xr-x 5 root root 4096 Sep 12 07:01 boot drwxr-xr-x 2 root root 6 Sep 12 07:22 dev drwxr-xr-x 79 root root 8192 Sep 12 07:07 etc drwxr-xr-x 3 root root 18 Sep 12 07:01 home lrwxrwxrwx 1 root root 7 Sep 12 07:27 lib -> usr/lib lrwxrwxrwx 1 root root 9 Sep 12 07:27 lib64 -> usr/lib64 drwxr-xr-x 2 root root 6 Apr 11 2018 media drwxr-xr-x 2 root root 6 Apr 11 2018 mnt drwxr-xr-x 2 root root 6 Apr 11 2018 opt dr-xr-x--- 3 root root 217 Sep 12 07:21 root drwxr-xr-x 2 root root 6 Sep 12 07:22 run lrwxrwxrwx 1 root root 8 Sep 12 07:27 sbin -> usr/sbin drwxr-xr-x 2 root root 6 Apr 11 2018 srv dr-xr-xr-x 2 root root 6 Sep 12 07:22 sys drwxr-xr-x 13 root root 155 Mar 4 2019 usr drwxr-xr-x 19 root root 267 Sep 12 07:01 var -
Make sure
pingis in the/tmp/bacula-restoresdirectory:sudo ls -la /tmp/bacula-restores/bin/pingResult:
-rwxr-xr-x 1 root root 66176 Aug 4 2017 /tmp/bacula-restores/bin/ping -
Copy the
pingtool to the main file system:sudo cp /tmp/bacula-restores/bin/ping /bin/ping -
Make sure that
pingworks:sudo ping 127.0.0.1 -c 1Result:
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data. 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.016 ms --- 127.0.0.1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.016/0.016/0.016/0.000 ms -
To free up disk space, delete the copy of the recovered files:
sudo rm -rfd /tmp/bacula-restores/*
How to delete the resources you created
To stop paying for the resources you created: