s3fs
s3fs
Note
We recommend using s3fs version 1.84 or higher.
If you experience any issues with s3fs, check the official documentation
s3fs performance depends on the speed of your local disk. Use high-speed disks, especially if you store a lot of small files a few hundred kilobytes in size or smaller. You can increase s3fs performance by enabling caching (--use_cache <directory> parameter). The s3fs cache grows indefinitely, so you need to clear it at regular intervals. You can read more in the s3fs documentation
Getting started
-
Assign to the service account the roles required for your project, e.g., storage.editor for a bucket (to work with a particular bucket) or a folder (to work with all buckets in this folder). For more information about roles, see Access management with Yandex Identity and Access Management.
To work with objects in an encrypted bucket, a user or service account must have the following roles for the encryption key in addition to the
storage.configurerrole:kms.keys.encrypter: To read the key, encrypt and upload objects.kms.keys.decrypter: To read the key, decrypt and download objects.kms.keys.encrypterDecrypter: This role includes thekms.keys.encrypterandkms.keys.decrypterpermissions.
For more information, see Key Management Service service roles.
-
As a result, you will get the static access key data. To authenticate in Object Storage, you will need the following:
key_id: Static access key IDsecret: Secret key
Save
key_idandsecret: you will not be able to get the key value again.
To access the HTTP API directly, you need static key authentication, which is supported by the tools listed in Supported tools.
Note
You can disable using static keys for bucket access. Once disabled, access will be denied to all tools using this access option: the AWS CLI, SDK, and third-party applications. Access via ephemeral keys, temporary Security Token Service access keys, and pre-signed URLs will also be disabled. Only access with an IAM token or anonymous access (if enabled) will remain.
You can use Yandex Lockbox to safely store the static key for access to Object Storage. For more information, see Using a Yandex Lockbox secret to store a static access key.
Note
A service account is only allowed to view a list of buckets in the folder it was created in.
A service account can perform actions with objects in buckets that are created in folders different from the service account folder. To enable this, assign the service account roles for the appropriate folder or its bucket.
Installation
-
Make sure the FUSE utilities are installed in the distribution:
apt list --installed | grep fuseWarning
Many Linux distributions have the utilities for working with FUSE pre-installed by default. Reinstalling or deleting them may lead to OS failures.
-
If the FUSE utilities are not installed, run this command:
sudo apt-get install fuse -
To install s3fs, follow this guide
in the project repository.
-
Make sure the FUSE utilities are installed in the distribution:
dnf list installed | grep fuseWarning
Many Linux distributions have the utilities for working with FUSE pre-installed by default. Reinstalling or deleting them may lead to OS failures.
-
If the FUSE utilities are not installed, run this command:
sudo dnf install fuse -
To install s3fs, follow this guide
in the project repository.
-
Install the macFUSE
package. -
Enable
support for third-party core extensions. This step is only required the first time you use MacFUSE on an Apple Silicon Mac. -
Allow
loading the MacFUSE core extension (Apple Silicon and Intel Mac).For more information on installing macFUSE, see this installation guide
in the macFUSE GitHub repository. -
To install s3fs, follow this guide
in the project repository.
Setup
To configure s3fs, save the previously obtained key ID and the secret key to the ~/.passwd-s3fs file in <key_ID>:<secret_key> format and restrict access to the ~/.passwd-s3fs file in the following way:
echo <key_ID>:<secret_key> > ~/.passwd-s3fs
chmod 600 ~/.passwd-s3fs
Mounting a bucket
-
Select the folder where you want to mount your bucket and make sure you have permissions for this operation.
-
Run this command:
s3fs <bucket_name> /mount/<folder_path> -o passwd_file=$HOME/.passwd-s3fs \ -o url=https://storage.yandexcloud.net -o use_path_request_styleTo allow other PC users to access the folder, specify the
-o allow_otheroption.To set access permissions to the folder you are going to mount the bucket to, specify the
-o mp_umask=<required_permissions>option.
You can configure mounting a bucket when starting the system. To do this, open the /etc/fstab file and add a line in the following format:
s3fs#<bucket_name> /mount/<folder_path> fuse _netdev,allow_other,use_path_request_style,url=https://storage.yandexcloud.net,passwd_file=/home/<username>/.passwd-s3fs 0 0
For descriptions of all s3fs parameters, see the project's wiki page