Backup to Yandex Object Storage via Bacula
You can use Yandex Object Storage for VM backup and recovery via the Bacula
Bacula consists of several components:
- Bacula Director: Controls the backup and recovery process.
- File Daemon: Provides access to backup files.
- Storage Daemon: Reads and writes files to the hard disk.
- Catalog: Maintains the file catalog used for backup. The catalog is stored in a MariaDB
database. - Bacula Console: A management console for interacting with Bacula Director.
To set up backup and recovery via Bacula:
- Prepare your cloud.
- Create a VM.
- Set up the AWS CLI.
- Install Bacula and additional components.
- Configure MariaDB.
- Set up a storage.
- Configure Bacula components.
- Create a backup.
- Recover the files.
If you no longer need the resources you created, delete them.
Getting started
Sign up for Yandex Cloud and create a billing account:
- Go to the management console
and log in to Yandex Cloud or create an account if you do not have one yet. - On the Yandex Cloud Billing
page, make sure you have a billing account linked and it has theACTIVE
orTRIAL_ACTIVE
status. If you do not have a billing account, create one.
If you have an active billing account, you can go to the cloud page
Learn more about clouds and folders.
Required paid resources
The cost for backup and recovery includes:
- Fee for VM computing resources and disks (see Yandex Compute Cloud pricing).
- Fee for data storage in a bucket and operations with data (see Yandex Object Storage pricing).
- Fee for using a dynamic or static external IP address (see Yandex Virtual Private Cloud pricing).
Create a bucket
To create a bucket for backups in Object Storage:
- Go to the Yandex Cloud management console
and select the folder where you will perform the operations. - On the folder page, click Create resource and select Bucket.
- In the ** Name** field, enter a name for the bucket.
- Set the public access parameters to read objects in the bucket, get a list of objects, and read bucket settings, i.e.,
Restricted
. - In the Storage class field, select Cold.
- Click Create bucket.
Create a service account
Create a service account and assign it the editor
role.
Create static access keys
Create static access keys.
Save key_id
and the secret
key right away. You will not be able to get the key value again.
Create a VM
To create a VM:
-
In the management console
, select the folder to create your VM in. -
In the list of services, select Compute Cloud.
-
In the left-hand panel, select
Virtual machines. -
Click Create virtual machine.
-
Under Boot disk image, select a public CentOS 7 image.
-
Under Location, select an availability zone to place your VM in.
-
Under Computing resources, navigate to the Custom tab and specify the parameters as follows:
- Platform:
Intel Ice Lake
- vCPU:
2
- Guaranteed vCPU performance:
20%
- RAM:
2 GB
- Platform:
-
Under Network settings:
-
In the Subnet field, specify the ID of a subnet in the new VM’s availability zone. Alternatively, you can select a cloud network from the list.
-
Each network must have at least one subnet. If there is no subnet, create one by selecting Create subnet.
-
If you do not have a network, click Create network to create one:
- In the window that opens, specify the network name and select the folder to host the network.
- (Optional) Select the Create subnets option to automatically create subnets in all availability zones.
- Click Create network.
-
-
In the Public IP field, select
Auto
to assign the VM a random external IP address from the Yandex Cloud pool or select a static address from the list if you reserved one in advance.
-
-
Under Access, specify the information required to access the VM:
-
In the Login field, enter the username of the user to be created on the VM, e.g.,
yc-user
.Alert
Do not use
root
or other usernames reserved by the operating system. To perform operations requiring superuser permissions, use thesudo
command. -
In the SSH key field, paste the contents of the public key file.
You need to create a key pair for the SSH connection yourself. To learn how, see Connecting to a VM via SSH.
-
-
Under General information, specify the VM name:
bacula-vm
. -
Click Create VM.
-
Wait for the VM status to change to
RUNNING
.
Set up the AWS CLI
To set up the AWS CLI utility on your bacula-vm
instance:
-
In the management console
, go to the VM page and find out its public IP address. -
Connect to the VM via SSH.
The recommended authentication method when connecting over SSH is using a key pair. Set up the generated key pair: the private key must match the public key sent to the VM.
-
Update the packages installed in the system. For this, in the terminal, run:
yum update -y
-
Install the AWS CLI:
yum install awscli -y
-
Set up the AWS CLI:
sudo aws configure
Specify the parameter values:
AWS Access Key ID
:key_id
that you got when creating the static key.AWS Secret Access Key
:secret
key that you got when generating the static key.Default region name
:ru-central1
.Default output format
:json
.
-
Check that the
/root/.aws/credentials
file contains the correctkey_id
andsecret
values:sudo cat /root/.aws/credentials
-
Check that the
/root/.aws/config
file contains the correctDefault region name
andDefault output format
values:sudo cat /root/.aws/config
Install Bacula and additional components
-
Install the Bacula components:
sudo yum install -y bacula-director bacula-storage bacula-console bacula-client
-
Install MariaDB
:sudo yum install -y mariadb-server
-
Install the
s3fs
utility to mount the Object Storage bucket to the file system:sudo yum install -y epel-release sudo yum install -y s3fs-fuse
-
Install the
nano
text editor:sudo yum install -y nano
Configure MariaDB
-
Run MariaDB:
sudo systemctl start mariadb
-
Check that MariaDB is running:
sudo systemctl status mariadb | grep Active
-
Enable MariaDB to run at system startup:
sudo systemctl enable mariadb
-
Create database tables and configure access rights:
/usr/libexec/bacula/grant_mysql_privileges /usr/libexec/bacula/create_mysql_database -u root /usr/libexec/bacula/make_mysql_tables -u bacula
-
Secure your database:
sudo mysql_secure_installation
For the following queries:
Enter current password for root (enter for none)
: Press Enter to skip the field.Set root password? [Y/n]
: EnterY
, set the root password, and confirm it. You will need the password in the next step.Remove anonymous users? [Y/n]
: To accept the default value, press Enter.Disallow root login remotely? [Y/n]
: To accept the default value, press Enter.Remove test database and access to it? [Y/n]
: To accept the default value, press Enter.Reload privilege tables now? [Y/n]
: To accept the default value, press Enter.
-
Log in to the DB command line and enter the
root
password created in the previous step:mysql -u root -p
-
Create
bacula_db_password
for thebacula
user:UPDATE mysql.user SET Password=PASSWORD('bacula_db_password') WHERE User='bacula'; FLUSH PRIVILEGES; exit
-
Enable the MySQL library for Bacula:
sudo alternatives --config libbaccats.so
Enter
1
to select MySQL:Selection Command ----------------------------------------------- 1 /usr/lib64/libbaccats-mysql.so 2 /usr/lib64/libbaccats-sqlite3.so *+ 3 /usr/lib64/libbaccats-postgresql.so Enter to keep the current selection[+], or type selection number: 1
Set up the storage
Prepare a backup folder
-
Create the
/tmp/bacula
backup folder:sudo mkdir /tmp/bacula
-
Set up access permissions for the
/tmp/bacula
folder:sudo chown -R bacula:bacula /tmp/bacula sudo chmod -R 700 /tmp/bacula sudo semanage permissive -a bacula_t
Mount the bucket to the file system
-
Use
s3fs
to mount the bucket to upload backups to Object Storage. To do this, run the command below and specify the bucket name:sudo s3fs <bucket_name> /tmp/bacula \ -o url=https://storage.yandexcloud.net \ -o use_path_request_style \ -o allow_other \ -o nonempty \ -o uid=133,gid=133,mp_umask=077
Where:
uid=133
: ID of thebacula
user from the/etc/passwd
filegid=133
: ID of thebacula
group from the/etc/passwd
file
-
Check the access permissions for the
/tmp/bacula
folder:sudo ls -la /tmp/bacula/
Result:
drwx------. 2 bacula bacula 31 Sep 18 09:16 . drwxrwxrwt. 10 root root 265 Sep 18 08:59 ..
-
Make sure that the
bacula
user can create files in the/tmp/bacula
folder:-
Temporarily enable the
bash
shell for thebacula
user:sudo sed -i "/^bacula/ s@/sbin/nologin@/bin/bash@" /etc/passwd
-
In the
/tmp/bacula
folder, create a file:sudo runuser -l bacula -c 'touch /tmp/bacula/test.test'
-
Make sure that the file
test.test
was created in the/tmp/bacula
folder:sudo ls -la /tmp/bacula | grep test.test
-
In the management console
, on the folder page, select Object Storage and make sure thetest.test
file is in the bucket. -
Delete the test file:
sudo runuser -l bacula -c 'rm -f /tmp/bacula/test.test'
-
Temporarily disable the
bash
shell for thebacula
user:sudo sed -i "/^bacula/ s@/bin/bash@/sbin/nologin@" /etc/passwd
-
Configure Bacula components
Configure Bacula Director
-
Open the Bacula Director configuration file:
sudo nano /etc/bacula/bacula-dir.conf
-
To set up a connection to Bacula Director, go to the
Director
configuration section and add theDirAddress = 127.0.0.1
line:... Director { # define myself Name = bacula-dir DIRport = 9101 # Specify the port (a positive integer) on which the Director daemon will listen for Bacula Console connections. # This same port number must be specified in the Director resource of the Console configuration file. # The default is 9101, so normally this directive need not be specified. # This directive should not be used if you specify the DirAddresses (plural) directive. QueryFile = "/etc/bacula/query.sql" WorkingDirectory = "/var/spool/bacula" PidDirectory = "/var/run" Maximum Concurrent Jobs = 1 Password = "@@DIR_PASSWORD@@" # Console password Messages = Daemon DirAddress = 127.0.0.1 } ...
-
For your convenience, rename the
BackupClient1
task toBackupFiles
:... Job { Name = "BackupFiles" JobDefs = "DefaultJob" } ...
-
To specify
/tmp/bacula-restores
as the folder for your recovered files, add the lineWhere = /tmp/bacula-restores
to theRestoreFiles
job configuration:... Job { Name = "RestoreFiles" Type = Restore Client=bacula-fd FileSet="Full Set" Storage = File Pool = Default Messages = Standard Where = /tmp/bacula-restores } ...
-
In the
FileSet
coniguration section namedFull Set
, underInclude
:- Add the
compression = GZIP
line to theOptions
section to enable compression during backup. - Specify
File = /
to back up the entire file system.
... FileSet { Name = "Full Set" Include { Options { signature = MD5 compression = GZIP } File = / } Exclude { File = /var/spool/bacula File = /tmp File = /proc File = /tmp File = /.journal File = /.fsck } } ...
- Add the
-
In the management console
, go to the VM page and look up its internal IP address. -
To set up an outbound connection to the Storage Daemon, in the
Storage
configuration section, enter the VM's internal IP address in theAddress
field:... Storage { Name = File # Do not use "localhost" here Address = <VM_internal_IP_address> # N.B. Use a fully qualified name here SDPort = 9103 Password = "@@SD_PASSWORD@@" Device = FileStorage Media Type = File } ...
-
To connect to the DB, in the
Catalog
configuration section, specify thedbpassword = "bacula_db_password"
database password you created when setting up MariaDB:... # Generic catalog service Catalog { Name = MyCatalog # Uncomment the following line if you want the dbi driver # dbdriver = "dbi:postgresql"; dbaddress = 127.0.0.1; dbport = dbname = "bacula"; dbuser = "bacula"; dbpassword = "bacula_db_password" } ...
-
Save the file.
-
Make sure that the
bacula-dir.conf
file has no syntax errors:sudo bacula-dir -tc /etc/bacula/bacula-dir.conf
If there are no error messages, the configuration is correct.
Configure Storage Daemon
-
Open the Storage Daemon configuration file:
sudo nano /etc/bacula/bacula-sd.conf
-
To set up an outbound connection to the Storage Daemon, in the
Storage
configuration section, specify the VM's internal IP address in theSDAddress
field:... Storage { # definition of myself Name = BackupServer-sd SDPort = 9103 # Specifies port number on which the Storage daemon listens for Director connections. The default is 9103. WorkingDirectory = "/var/spool/bacula" Pid Directory = "/var/run/bacula" Maximum Concurrent Jobs = 20 SDAddress = <VM_internal_IP_address> # This directive is optional, and if it is specified, # it will cause the Storage daemon server (for Director and File daemon connections) to bind to the specified IP-Address, # which is either a domain name or an IP address specified as a dotted quadruple. # If this directive is not specified, the Storage daemon will bind to any available address (the default). } ...
-
In the
Device
configuration section, specifyArchive Device = /tmp/bacula
as the backup folder:... Device { Name = FileStorage Media Type = File Archive Device = /tmp/bacula LabelMedia = yes; # lets Bacula label unlabeled media Random Access = Yes; AutomaticMount = yes; # when device opened, read it RemovableMedia = no; AlwaysOpen = no; } ...
-
Save the file.
-
Make sure the
bacula-sd.conf
file has no syntax errors:sudo bacula-sd -tc /etc/bacula/bacula-sd.conf
If there are no error messages, the configuration is correct.
Create passwords for Bacula components
Bacula Director, Storage Daemon, and File Daemon use passwords for inter-component authentication.
To set passwords for Bacula components:
-
Generate passwords for Bacula Director, Storage Daemon, and File Daemon:
DIR_PASSWORD=`date +%s | sha256sum | base64 | head -c 33` SD_PASSWORD=`date +%s | sha256sum | base64 | head -c 33` FD_PASSWORD=`date +%s | sha256sum | base64 | head -c 33`
-
Put the passwords in the configuration files:
sudo sed -i "s/@@DIR_PASSWORD@@/${DIR_PASSWORD}/" /etc/bacula/bacula-dir.conf sudo sed -i "s/@@DIR_PASSWORD@@/${DIR_PASSWORD}/" /etc/bacula/bconsole.conf sudo sed -i "s/@@SD_PASSWORD@@/${SD_PASSWORD}/" /etc/bacula/bacula-sd.conf sudo sed -i "s/@@SD_PASSWORD@@/${SD_PASSWORD}/" /etc/bacula/bacula-dir.conf sudo sed -i "s/@@FD_PASSWORD@@/${FD_PASSWORD}/" /etc/bacula/bacula-dir.conf sudo sed -i "s/@@FD_PASSWORD@@/${FD_PASSWORD}/" /etc/bacula/bacula-fd.conf
Run the Bacula components
-
Run the Bacula components:
sudo systemctl start bacula-dir sudo systemctl start bacula-sd sudo systemctl start bacula-fd
-
Check that the Bacula components are running:
sudo systemctl status bacula-dir sudo systemctl status bacula-sd sudo systemctl status bacula-fd
-
Set up the Bacula components to launch at system startup:
sudo systemctl enable bacula-dir sudo systemctl enable bacula-sd sudo systemctl enable bacula-fd
Create a backup
-
Open Bacula Console:
sudo bconsole
-
To set up a backup profile, create a label:
label
-
Name the new volume as
MyVolume
:Enter new Volume name: MyVolume
-
To select the
File
pool, enter2
:Defined Pools: 1: Default 2: File 3: Scratch Select the Pool (1-3): 2
-
Run the backup process:
run
To run the
BackupFiles
job, select1
:A job name must be specified. The defined Job resources are: 1: BackupFiles 2: BackupCatalog 3: RestoreFiles Select Job resource (1-3): 1
To confirm, enter
yes
:OK to run? (yes/mod/no): yes
-
Check the backup status:
status director
Result if the backup is running:
Running Jobs: Console connected at 12-Sep-19 07:22 JobId Level Name Status ====================================================================== 2 Full BackupFiles.2019-09-12_07.22.56_03 is running
Result if the backup is complete:
Running Jobs: Console connected at 12-Sep-19 07:25 No Jobs running. ==== Terminated Jobs: JobId Level Files Bytes Status Finished Name ==================================================================== 2 Full 32,776 483.6 M OK 12-Sep-19 07:24 BackupFiles
-
Wait for the backup to complete and exit Bacula Console:
exit
Check the backup
To make sure that the backup is complete:
- In the management console
, on the folder page, select Object Storage. - Open the bucket.
- Make sure it contains the
MyVolume
object.
Recover the files
-
To test recovery, first, delete a file, e.g., the
ping
utility:sudo rm -f /bin/ping
-
Make sure that
ping
is deleted:ping
Result:
bash: ping: command not found
-
Log in to Bacula Console:
sudo bconsole
-
Run a full recovery:
restore all
To run a recovery from the most recent backup, enter
5
:To select the JobIds, you have the following choices: 1: List last 20 Jobs run 2: List Jobs where a given File is saved 3: Enter list of comma separated JobIds to select 4: Enter SQL list command 5: Select the most recent backup for a client 6: Select backup for a client before a specified time 7: Enter a list of files to restore 8: Enter a list of files to restore before a specified time 9: Find the JobIds of the most recent backup for a client 10: Find the JobIds for a backup for a client before a specified time 11: Enter a list of directories to restore for found JobIds 12: Select full restore to a specified Job date 13: Cancel Select item: (1-13): 5
To confirm full recovery, enter
done
:You are now entering file selection mode where you add (mark) and remove (unmark) files to be restored. No files are initially added, unless you used the "all" keyword on the command line. Enter "done" to leave this mode. cwd is: / done
To confirm running the recovery, enter
yes
:OK to run? (yes/mod/no): yes
-
Check the recovery status:
status director
Here's the result if the recovery is in progress:
Running Jobs: Console connected at 12-Sep-19 07:25 JobId Level Name Status ====================================================================== 3 RestoreFiles.2019-09-12_07.27.42_05 is running
Here's the result if the recovery is complete:
Terminated Jobs: JobId Level Files Bytes Status Finished Name ==================================================================== 2 Full 32,776 483.6 M OK 12-Sep-19 07:24 BackupFiles 3 32,776 1.136 G OK 12-Sep-19 07:27 RestoreFiles
-
Wait for the recovery to complete and exit Bacula Console:
exit
Check the recovered files
-
Make sure the
/tmp/bacula-restores
folder now contains the recovered files:sudo ls -la /tmp/bacula-restores
Result:
total 16 dr-xr-xr-x. 15 root root 201 Sep 12 07:09 . drwx------. 4 bacula bacula 35 Sep 12 07:09 .. lrwxrwxrwx 1 root root 7 Sep 12 07:27 bin -> usr/bin dr-xr-xr-x 5 root root 4096 Sep 12 07:01 boot drwxr-xr-x 2 root root 6 Sep 12 07:22 dev drwxr-xr-x 79 root root 8192 Sep 12 07:07 etc drwxr-xr-x 3 root root 18 Sep 12 07:01 home lrwxrwxrwx 1 root root 7 Sep 12 07:27 lib -> usr/lib lrwxrwxrwx 1 root root 9 Sep 12 07:27 lib64 -> usr/lib64 drwxr-xr-x 2 root root 6 Apr 11 2018 media drwxr-xr-x 2 root root 6 Apr 11 2018 mnt drwxr-xr-x 2 root root 6 Apr 11 2018 opt dr-xr-x--- 3 root root 217 Sep 12 07:21 root drwxr-xr-x 2 root root 6 Sep 12 07:22 run lrwxrwxrwx 1 root root 8 Sep 12 07:27 sbin -> usr/sbin drwxr-xr-x 2 root root 6 Apr 11 2018 srv dr-xr-xr-x 2 root root 6 Sep 12 07:22 sys drwxr-xr-x 13 root root 155 Mar 4 2019 usr drwxr-xr-x 19 root root 267 Sep 12 07:01 var
-
Make sure
ping
is in the/tmp/bacula-restores
folder:sudo ls -la /tmp/bacula-restores/bin/ping
Result:
-rwxr-xr-x 1 root root 66176 Aug 4 2017 /tmp/bacula-restores/bin/ping
-
Copy the
ping
utility to the main file system:sudo cp /tmp/bacula-restores/bin/ping /bin/ping
-
Make sure that
ping
works:sudo ping 127.0.0.1 -c 1
Result:
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data. 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.016 ms --- 127.0.0.1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.016/0.016/0.016/0.000 ms
-
To free up disk space, delete the copy of the recovered files:
sudo rm -rfd /tmp/bacula-restores/*
How to delete the resources you created
To stop paying for the resources you created: