Contact UsGet started

Backup to Yandex Object Storage via Bacula

Written by
Updated at November 27, 2024

You can use Yandex Object Storage for VM backup and recovery via the Bacula utility.

Bacula consists of several components:

  • Bacula Director: Controls the backup and recovery process.
  • File Daemon: Provides access to backup files.
  • Storage Daemon: Reads and writes files to the hard disk.
  • Catalog: Maintains the file catalog used for backup. The catalog is stored in a MariaDB database.
  • Bacula Console: A management console for interacting with Bacula Director.

To set up backup and recovery via Bacula:

  1. Prepare your cloud.
  2. Create a VM.
  3. Set up the AWS CLI.
  4. Install Bacula and additional components.
  5. Configure MariaDB.
  6. Set up a storage.
  7. Configure Bacula components.
  8. Create a backup.
  9. Recover the files.

If you no longer need the resources you created, delete them.

Getting started

Sign up for Yandex Cloud and create a billing account:

  1. Go to the management console and log in to Yandex Cloud or create an account if you do not have one yet.
  2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one.

If you have an active billing account, you can go to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

The cost for backup and recovery includes:

Create a bucket

To create a bucket for backups in Object Storage:

  1. Go to the Yandex Cloud management console and select the folder where you will perform the operations.
  2. On the folder page, click Create resource and select Bucket.
  3. In the ** Name** field, enter a name for the bucket.
  4. Set the public access parameters to read objects in the bucket, get a list of objects, and read bucket settings, i.e., Restricted.
  5. In the Storage class field, select Cold.
  6. Click Create bucket.

Create a service account

Create a service account and assign it the editor role.

Create static access keys

Create static access keys.

Save key_id and the secret key right away. You will not be able to get the key value again.

Create a VM

To create a VM:

  1. In the management console, select the folder to create your VM.

  2. In the list of services, select Compute Cloud.

  3. In the left-hand panel, select Virtual machines.

  4. Click Create virtual machine.

  5. Under Boot disk image select a public CentOS 7 image.

  6. Under Location, select an availability zone to place your VM in.

  7. Under Computing resources, navigate to the Custom tab and specify the parameters as follows:

    • Platform: Intel Ice Lake.
    • vCPU: 2.
    • Guaranteed vCPU performance: 20%.
    • RAM: 2 GB

  8. Under Network settings:

    • In the Subnet field, enter the ID of a subnet in the new VM’s availability zone. Alternatively, you can select a cloud network from the list.

      • Each network must have at least one subnet. If there is no subnet, create one by selecting Create subnet.

      • If you do not have a network, click Create network to create one:

        • In the window that opens, enter the network name and select the folder to host the network.
        • (Optional) Select the Create subnets option to automatically create subnets in all availability zones.
        • Click Create network.

    • In the Public IP field, select Auto to assign the VM a random external IP address from the Yandex Cloud pool or select a static address from the list if you reserved one in advance.

  9. Under Access, specify the information required to access the VM:

    • In the Login field, enter a name for the user you want to create on the VM, e.g., yc-user.

      Alert

      Do not use root or other usernames reserved by the operating system. To perform operations requiring superuser permissions, use the sudo command.

    • In the SSH key field, select the SSH key saved in your organization user profile.

      If there are no saved SSH keys in your profile, or you want to add a new key:

      • Click Add key.
      • Enter a name for the SSH key.
      • Upload or paste the contents of the public key file. You need to create a key pair for the SSH connection to a VM yourself.
      • Click Add.

      The SSH key will be added to your organization user profile.

      If users cannot add SSH keys to their profiles in the organization, the added public SSH key will only be saved to the user profile of the VM being created.

  10. Under General information, specify the VM name: bacula-vm.

  11. Click Create VM.

  12. Wait for the VM status to change to RUNNING.

Set up the AWS CLI

To set up the AWS CLI utility on your bacula-vm instance:

  1. In the management console, go to the VM page and find out its public IP address.

  2. Connect to the VM via SSH.

    The recommended authentication method when connecting over SSH is using a key pair. Set up the generated key pair: the private key must match the public key sent to the VM.

  3. Update the packages installed in the system. For this, in the terminal, run:

    yum update -y

  4. Install the AWS CLI:

    yum install awscli -y

  5. Set up the AWS CLI:

    sudo aws configure

    Specify the parameter values:

  6. Check that the /root/.aws/credentials file contains the correct key_id and secret values:

    sudo cat /root/.aws/credentials

  7. Check that the /root/.aws/config file contains the correct Default region name and Default output format values:

    sudo cat /root/.aws/config

Install Bacula and additional components

  1. Install the Bacula components:

    sudo yum install -y bacula-director bacula-storage bacula-console bacula-client

  2. Install MariaDB:

    sudo yum install -y mariadb-server

  3. Install the s3fs utility to mount the Object Storage bucket to the file system:

    sudo yum install -y epel-release
sudo yum install -y s3fs-fuse

  4. Install the nano text editor:

    sudo yum install -y nano

Configure MariaDB

  1. Run MariaDB:

    sudo systemctl start mariadb

  2. Check that MariaDB is running:

    sudo systemctl status mariadb | grep Active

  3. Enable MariaDB to run at system startup:

    sudo systemctl enable mariadb

  4. Create database tables and configure access rights:

    /usr/libexec/bacula/grant_mysql_privileges
/usr/libexec/bacula/create_mysql_database -u root
/usr/libexec/bacula/make_mysql_tables -u bacula

  5. Secure your database:

    sudo mysql_secure_installation

    For the following queries:

    • Enter current password for root (enter for none): Press Enter to skip the field.
    • Set root password? [Y/n]: Enter Y, set the root password, and confirm it. You will need the password in the next step.
    • Remove anonymous users? [Y/n]: To accept the default value, press Enter.
    • Disallow root login remotely? [Y/n]: To accept the default value, press Enter.
    • Remove test database and access to it? [Y/n]: To accept the default value, press Enter.
    • Reload privilege tables now? [Y/n]: To accept the default value, press Enter.

  6. Log in to the DB command line and enter the root password created in the previous step:

    mysql -u root -p

  7. Create bacula_db_password for the bacula user:

    UPDATE mysql.user SET Password=PASSWORD('bacula_db_password') WHERE User='bacula';
FLUSH PRIVILEGES;
exit

  8. Enable the MySQL library for Bacula:

    sudo alternatives --config libbaccats.so

    Enter 1 to select MySQL:

      Selection    Command
-----------------------------------------------
   1           /usr/lib64/libbaccats-mysql.so
   2           /usr/lib64/libbaccats-sqlite3.so
*+ 3           /usr/lib64/libbaccats-postgresql.so

Enter to keep the current selection[+], or type selection number: 1

Set up the storage

Prepare a backup folder

  1. Create the /tmp/bacula backup folder:

    sudo mkdir /tmp/bacula

  2. Set up access permissions for the /tmp/bacula folder:

    sudo chown -R bacula:bacula /tmp/bacula
sudo chmod -R 700 /tmp/bacula
sudo semanage permissive -a bacula_t

Mount the bucket to the file system

  1. Use s3fs to mount the bucket to upload backups to Object Storage. To do this, run the command below and specify the bucket name:

    sudo s3fs <bucket_name> /tmp/bacula \
  -o url=https://storage.yandexcloud.net \
  -o use_path_request_style \
  -o allow_other \
  -o nonempty \
  -o uid=133,gid=133,mp_umask=077

    Where:

    • uid=133: ID of the bacula user from the /etc/passwd file
    • gid=133: ID of the bacula group from the /etc/passwd file

  2. Check the access permissions for the /tmp/bacula folder:

    sudo ls -la /tmp/bacula/

    Result:

    drwx------.  2 bacula bacula        31 Sep 18 09:16 .
drwxrwxrwt. 10 root   root         265 Sep 18 08:59 ..

  3. Make sure that the bacula user can create files in the /tmp/bacula folder:

    1. Temporarily enable the bash shell for the bacula user:

      sudo sed -i "/^bacula/ s@/sbin/nologin@/bin/bash@" /etc/passwd

    2. In the /tmp/bacula folder, create a file:

      sudo runuser -l  bacula -c 'touch /tmp/bacula/test.test'

    3. Make sure that the file test.test was created in the /tmp/bacula folder:

      sudo ls -la /tmp/bacula | grep test.test

    4. In the management console, on the folder page, select Object Storage and make sure the test.test file is in the bucket.

    5. Delete the test file:

      sudo runuser -l  bacula -c 'rm -f /tmp/bacula/test.test'

    6. Temporarily disable the bash shell for the bacula user:

      sudo sed -i "/^bacula/ s@/bin/bash@/sbin/nologin@" /etc/passwd

Configure Bacula components

Configure Bacula Director

  1. Open the Bacula Director configuration file:

    sudo nano /etc/bacula/bacula-dir.conf

  2. To set up a connection to Bacula Director, go to the Director configuration section and add the DirAddress = 127.0.0.1 line:

    ...
Director {                              # define myself
  Name = bacula-dir
  DIRport = 9101                        # Specify the port (a positive integer) on which the Director daemon will listen for Bacula Console connections.
                                        # This same port number must be specified in the Director resource of the Console configuration file.
                                        # The default is 9101, so normally this directive need not be specified.
                                        # This directive should not be used if you specify the DirAddresses (plural) directive.
  QueryFile = "/etc/bacula/query.sql"
  WorkingDirectory = "/var/spool/bacula"
  PidDirectory = "/var/run"
  Maximum Concurrent Jobs = 1
  Password = "@@DIR_PASSWORD@@"         # Console password
  Messages = Daemon
  DirAddress = 127.0.0.1
}    
...

  3. For your convenience, rename the BackupClient1 task to BackupFiles:

    ...
Job {
  Name = "BackupFiles"
  JobDefs = "DefaultJob"
}
...

  4. To specify /tmp/bacula-restores as the folder for your recovered files, add the line Where = /tmp/bacula-restores to the RestoreFiles job configuration:

    ...
Job {
  Name = "RestoreFiles"
  Type = Restore
  Client=bacula-fd
  FileSet="Full Set"
  Storage = File
  Pool = Default
  Messages = Standard
  Where = /tmp/bacula-restores
}
...

  5. In the FileSet coniguration section named Full Set, under Include:

    • Add the compression = GZIP line to the Options section to enable compression during backup.
    • Specify File = / to back up the entire file system.
    ...
FileSet {
  Name = "Full Set"
  Include {
	Options {
	  signature = MD5
	  compression = GZIP
	}
	File = /
  }
  Exclude {
	File = /var/spool/bacula
	File = /tmp
	File = /proc
	File = /tmp
	File = /.journal
	File = /.fsck
  }
}
...

  6. In the management console, go to the VM page and look up its internal IP address.

  7. To set up an outbound connection to the Storage Daemon, in the Storage configuration section, enter the VM's internal IP address in the Address field:

    ...
Storage {
  Name = File
# Do not use "localhost" here
  Address = <VM_internal_IP_address>  # N.B. Use a fully qualified name here
  SDPort = 9103
  Password = "@@SD_PASSWORD@@"
  Device = FileStorage
  Media Type = File
}
...

  8. To connect to the DB, in the Catalog configuration section, specify the dbpassword = "bacula_db_password" database password you created when setting up MariaDB:

    ...
# Generic catalog service
Catalog {
  Name = MyCatalog
# Uncomment the following line if you want the dbi driver
# dbdriver = "dbi:postgresql"; dbaddress = 127.0.0.1; dbport =
  dbname = "bacula"; dbuser = "bacula"; dbpassword = "bacula_db_password"
}
...

  9. Save the file.

  10. Make sure that the bacula-dir.conf file has no syntax errors:

    sudo bacula-dir -tc /etc/bacula/bacula-dir.conf

    If there are no error messages, the configuration is correct.

Configure Storage Daemon

  1. Open the Storage Daemon configuration file:

    sudo nano /etc/bacula/bacula-sd.conf

  2. To set up an outbound connection to the Storage Daemon, in the Storage configuration section, specify the VM's internal IP address in the SDAddress field:

    ...
Storage {                                      # definition of myself
  Name = BackupServer-sd
  SDPort = 9103                                # Specifies port number on which the Storage daemon listens for Director connections. The default is 9103.
  WorkingDirectory = "/var/spool/bacula"
  Pid Directory = "/var/run/bacula"
  Maximum Concurrent Jobs = 20
  SDAddress = <VM_internal_IP_address>         # This directive is optional, and if it is specified,
                                               # it will cause the Storage daemon server (for Director and File daemon connections) to bind to the specified IP-Address,
                                               # which is either a domain name or an IP address specified as a dotted quadruple.
                                               # If this directive is not specified, the Storage daemon will bind to any available address (the default).
}
...

  3. In the Device configuration section, specify Archive Device = /tmp/bacula as the backup folder:

    ...
Device {
  Name = FileStorage
  Media Type = File
  Archive Device = /tmp/bacula 
  LabelMedia = yes;                   # lets Bacula label unlabeled media
  Random Access = Yes;
  AutomaticMount = yes;               # when device opened, read it
  RemovableMedia = no;
  AlwaysOpen = no;
}
...

  4. Save the file.

  5. Make sure the bacula-sd.conf file has no syntax errors:

    sudo bacula-sd -tc /etc/bacula/bacula-sd.conf

    If there are no error messages, the configuration is correct.

Create passwords for Bacula components

Bacula Director, Storage Daemon, and File Daemon use passwords for inter-component authentication.

To set passwords for Bacula components:

  1. Generate passwords for Bacula Director, Storage Daemon, and File Daemon:

    DIR_PASSWORD=`date +%s | sha256sum | base64 | head -c 33`
SD_PASSWORD=`date +%s | sha256sum | base64 | head -c 33`
FD_PASSWORD=`date +%s | sha256sum | base64 | head -c 33`

  2. Put the passwords in the configuration files:

    sudo sed -i "s/@@DIR_PASSWORD@@/${DIR_PASSWORD}/" /etc/bacula/bacula-dir.conf
sudo sed -i "s/@@DIR_PASSWORD@@/${DIR_PASSWORD}/" /etc/bacula/bconsole.conf
sudo sed -i "s/@@SD_PASSWORD@@/${SD_PASSWORD}/" /etc/bacula/bacula-sd.conf
sudo sed -i "s/@@SD_PASSWORD@@/${SD_PASSWORD}/" /etc/bacula/bacula-dir.conf
sudo sed -i "s/@@FD_PASSWORD@@/${FD_PASSWORD}/" /etc/bacula/bacula-dir.conf
sudo sed -i "s/@@FD_PASSWORD@@/${FD_PASSWORD}/" /etc/bacula/bacula-fd.conf

Run the Bacula components

  1. Run the Bacula components:

    sudo systemctl start bacula-dir
sudo systemctl start bacula-sd
sudo systemctl start bacula-fd

  2. Check that the Bacula components are running:

    sudo systemctl status bacula-dir
sudo systemctl status bacula-sd
sudo systemctl status bacula-fd

  3. Set up the Bacula components to launch at system startup:

    sudo systemctl enable bacula-dir
sudo systemctl enable bacula-sd
sudo systemctl enable bacula-fd

Create a backup

  1. Open Bacula Console:

    sudo bconsole

  2. To set up a backup profile, create a label:

    label

  3. Name the new volume as MyVolume:

    Enter new Volume name: MyVolume

  4. To select the File pool, enter 2:

    Defined Pools:
     1: Default
     2: File
     3: Scratch
Select the Pool (1-3): 2

  5. Run the backup process:

    run

    To run the BackupFiles job, select 1:

    A job name must be specified.
The defined Job resources are:
    1: BackupFiles
    2: BackupCatalog
    3: RestoreFiles
Select Job resource (1-3): 1

    To confirm, enter yes:

    OK to run? (yes/mod/no): yes

  6. Check the backup status:

    status director

    Result if the backup is running:

    Running Jobs:
Console connected at 12-Sep-19 07:22
 JobId Level   Name                       Status
======================================================================
     2 Full    BackupFiles.2019-09-12_07.22.56_03 is running

    Result if the backup is complete:

    Running Jobs:
Console connected at 12-Sep-19 07:25
No Jobs running.
====

Terminated Jobs:
 JobId  Level    Files      Bytes   Status   Finished        Name 
====================================================================
     2  Full     32,776    483.6 M  OK       12-Sep-19 07:24 BackupFiles

  7. Wait for the backup to complete and exit Bacula Console:

    exit

Check the backup

To make sure that the backup is complete:

  1. In the management console, on the folder page, select Object Storage.
  2. Open the bucket.
  3. Make sure it contains the MyVolume object.

Recover the files

  1. To test recovery, first, delete a file, e.g., the ping utility:

    sudo rm -f /bin/ping

  2. Make sure that ping is deleted:

    ping

    Result:

    bash: ping: command not found

  3. Log in to Bacula Console:

    sudo bconsole

  4. Run a full recovery:

    restore all

    To run a recovery from the most recent backup, enter 5:

    To select the JobIds, you have the following choices:
    1: List last 20 Jobs run
    2: List Jobs where a given File is saved
    3: Enter list of comma separated JobIds to select
    4: Enter SQL list command
    5: Select the most recent backup for a client
    6: Select backup for a client before a specified time
    7: Enter a list of files to restore
    8: Enter a list of files to restore before a specified time
    9: Find the JobIds of the most recent backup for a client
    10: Find the JobIds for a backup for a client before a specified time
    11: Enter a list of directories to restore for found JobIds
    12: Select full restore to a specified Job date
    13: Cancel
Select item:  (1-13): 5

    To confirm full recovery, enter done:

    You are now entering file selection mode where you add (mark) and
remove (unmark) files to be restored. No files are initially added, unless
you used the "all" keyword on the command line.
Enter "done" to leave this mode.

cwd is: /
done

    To confirm running the recovery, enter yes:

    OK to run? (yes/mod/no): yes

  5. Check the recovery status:

    status director

    Here's the result if the recovery is in progress:

    Running Jobs:
Console connected at 12-Sep-19 07:25
 JobId Level   Name                       Status
======================================================================
     3         RestoreFiles.2019-09-12_07.27.42_05 is running

    Here's the result if the recovery is complete:

    Terminated Jobs:
 JobId  Level    Files      Bytes   Status   Finished        Name 
====================================================================
     2  Full     32,776    483.6 M  OK       12-Sep-19 07:24 BackupFiles
     3           32,776    1.136 G  OK       12-Sep-19 07:27 RestoreFiles

  6. Wait for the recovery to complete and exit Bacula Console:

    exit

Check the recovered files

  1. Make sure the /tmp/bacula-restores folder now contains the recovered files:

    sudo ls -la /tmp/bacula-restores

    Result:

    total 16
dr-xr-xr-x. 15 root   root    201 Sep 12 07:09 .
drwx------.  4 bacula bacula   35 Sep 12 07:09 ..
lrwxrwxrwx   1 root   root      7 Sep 12 07:27 bin -> usr/bin
dr-xr-xr-x   5 root   root   4096 Sep 12 07:01 boot
drwxr-xr-x   2 root   root      6 Sep 12 07:22 dev
drwxr-xr-x  79 root   root   8192 Sep 12 07:07 etc
drwxr-xr-x   3 root   root     18 Sep 12 07:01 home
lrwxrwxrwx   1 root   root      7 Sep 12 07:27 lib -> usr/lib
lrwxrwxrwx   1 root   root      9 Sep 12 07:27 lib64 -> usr/lib64
drwxr-xr-x   2 root   root      6 Apr 11  2018 media
drwxr-xr-x   2 root   root      6 Apr 11  2018 mnt
drwxr-xr-x   2 root   root      6 Apr 11  2018 opt
dr-xr-x---   3 root   root    217 Sep 12 07:21 root
drwxr-xr-x   2 root   root      6 Sep 12 07:22 run
lrwxrwxrwx   1 root   root      8 Sep 12 07:27 sbin -> usr/sbin
drwxr-xr-x   2 root   root      6 Apr 11  2018 srv
dr-xr-xr-x   2 root   root      6 Sep 12 07:22 sys
drwxr-xr-x  13 root   root    155 Mar  4  2019 usr
drwxr-xr-x  19 root   root    267 Sep 12 07:01 var

  2. Make sure ping is in the /tmp/bacula-restores folder:

    sudo ls -la /tmp/bacula-restores/bin/ping

    Result:

    -rwxr-xr-x 1 root root 66176 Aug  4  2017 /tmp/bacula-restores/bin/ping

  3. Copy the ping utility to the main file system:

    sudo cp /tmp/bacula-restores/bin/ping /bin/ping

  4. Make sure that ping works:

    sudo ping 127.0.0.1 -c 1

    Result:

    PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.016 ms

--- 127.0.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.016/0.016/0.016/0.000 ms

  5. To free up disk space, delete the copy of the recovered files:

    sudo rm -rfd /tmp/bacula-restores/*

How to delete the resources you created

To stop paying for the resources you created:

  1. Delete the VM.
  2. Delete all objects from the Object Storage bucket:
  3. Delete the Object Storage bucket.
  4. Delete the static public IP if you reserved one.
Previous
Deploying the GlusterFS parallel file system in high performance mode
Next
Overview