Mounting Yandex Object Storage buckets to the file system of Yandex Data Processing hosts

In Yandex Data Processing, you can use initialization actions to configure hosts.

You can use them to automate the installation and setup of GeeseFS – the software enabling Yandex Data Processing cluster hosts to mount Yandex Object Storage buckets via FUSE.

To set up GeeseFS:

1. Prepare the initialization action.
2. Create a cluster that uses the initialization action.
3. Check bucket availability.

If you no longer need the resources you created, delete them.

Getting startedGetting started

1. Create a service account with the dataproc.agent and dataproc.provisioner roles.

2. Create an Yandex Object Storage bucket.

3. Grant to the service account the read access for the bucket. For this, use one of the following methods:

   • Set up the bucket's ACL and grant the READ access to the service account.

     The service account will get read access only for the specified bucket.

   • Assign the service account the storage.viewer role.

     The service account will get read access to all buckets in the folder.

Prepare the initialization actionPrepare the initialization action

1. Create the initialization action file named geesefs_mount.sh accepting two positional arguments: the name of the Yandex Object Storage bucket and the host's file system directory where you are going to mount it.

   #!/bin/bash
   
   set -e
   
   BUCKET=$1
   MOUNT_POINT=$2
   
   # Downloading GeeseFS
   wget https://github.com/yandex-cloud/geesefs/releases/latest/download/geesefs-linux-amd64 -O /opt/geesefs
   chmod a+rwx /opt/geesefs
   mkdir -p "${MOUNT_POINT}"
   
   # Preparing a script to run on every boot
   BOOT_SCRIPT="/var/lib/cloud/scripts/per-boot/80-geesefs-mount.sh"
   echo "#!/bin/bash" >> ${BOOT_SCRIPT}
   echo "/opt/geesefs -o allow_other --iam ${BUCKET} ${MOUNT_POINT}" >> ${BOOT_SCRIPT}
   chmod 755 ${BOOT_SCRIPT}
   
   # Running the script
   ${BOOT_SCRIPT}
   

2. Upload the geesefs_mount.sh file to the previously created Yandex Object Storage bucket.

Create a cluster that would use the initialization actionCreate a cluster that would use the initialization action

Create a Yandex Data Processing cluster with the following settings:

• In the Service account field, select the service account you previously created.

• In the Custom scripts field, click Add and set the script parameters:

  • In the URI field, specify the path to the script file in the bucket, such as:

    s3a://<bucket_name>/geesefs_mount.sh
    

  • In the Arguments field, specify the name of the previously created bucket and /mnt/test as your mount point. Arguments are specified on separate lines:

    <bucket_name>
    /mnt/test
    

• In the Bucket name field, select the previously created bucket.

• In the Subclusters menu, enable Public access in the settings. This will enable you to connect to subcluster hosts without an intermediate virtual machine.

Check bucket availability.Check bucket availability.

1. After the cluster status changes to Alive, connect via SSH to any of its hosts as the ubuntu user.

2. To make sure that the bucket has been mounted successfully, run the command:

   ls /mnt/test/<bucket_name>
   

   As a result, it will output the list of objects stored in the root folder of the bucket. In this case, the file name geesefs_mount.sh.

Delete the resources you createdDelete the resources you created

Delete the resources you no longer need to avoid paying for them:

1. Delete the Yandex Data Processing cluster.
2. If you used static public IP addresses to access the cluster hosts, release and delete them.
3. Delete the Yandex Object Storage bucket.