GeeseFS

GeeseFS is a utility for mounting Object Storage buckets using FUSE on Linux, MacOS, and Windows platforms. It was developed by the Yandex Cloud team as a goofys branch. It is a high-performance, POSIX-ish file system.

FeaturesFeatures

PerformancePerformance

Compared to goofys and s3fs, GeeseFS handles large amounts of small files (up to 1 MB) much faster and achieves similar or higher performance with large files. For more information about benchmark tests, see the GeeseFS repository on GitHub.

To make GeeseFS run faster, it implements:

• Parallel readahead.
• Heuristic readahead for random access: if the size of multiple blocks requested in a row is below the threshold, GeeseFS downloads smaller blocks from storage for upcoming requests.
• Parallel multipart uploads of objects to storage.
• Optimized object updates: the client and repository only exchange modified object parts.
• Background uploads of small object trees and directories: when a directory is requested, GeeseFS downloads the whole tree per request to storage.
• Asynchronous object write, rename, and delete.
• Disk cache for reads and writes.

POSIX compatibilityPOSIX compatibility

In addition to the basic functions of the POSIX standards (open, read, write, close, and so on), GeeseFS supports the following features:

• Read-after-write consistency.
• Partial writes (please note that partial writes in buckets with versioning may result in intermediate object versions).
• fsync: Synchronization of the contents of an object or directory between the VM memory and storage.
• truncate: Changing object size at will.
• Soft links (symlinks).
• xattr: Extended file attributes.
• Directory renames.
• readdir: Reads of directory metadata.

Partial updating and appending of objects dataPartial updating and appending of objects data

GeeseFS supports partial updating and appending of objects data to Object Storage buckets.

To enable partial object updates, use the --enable-patch option.

To learn more, see the GeeseFS repository on GitHub:

• Partial object updates: Description of partial updating and appending of objects data.
• Concurrent Updates: Description of how an object can be partially updated by multiple concurrent requests.

LimitationsLimitations

GeeseFS does not support the following:

• Working with file and directory access permissions, including the chmod and chown commands.

  When mounting the file system, you can specify:

  • Access permissions to all files or directories in the --file-mode and --dir-mode option values, respectively.
  • ID of the owner of all files and directories in the --uid option value.
  • ID of the group all files and directories belong to in the --gid option value.

  Here is an example:

  geesefs \
    --file-mode=0666 \
    --dir-mode=0777 \
    --uid=1000 \
    <bucket_name> <mount_point>
  

• Hard links.

• File locking.

• Correct time of the last access to the file (atime) and the last change of the file's attributes (ctime). Both fields always contain the time of the file's last modification: same as in the mtime field.

• Creating files larger than 1 TB.

Getting startedGetting started

1. Create a service account.

2. Assign to the service account the roles required for your project, e.g., storage.editor for a bucket (to work with a particular bucket) or a folder (to work with all buckets in this folder). For more information about roles, see Access management with Yandex Identity and Access Management.

   To work with objects in an encrypted bucket, a user or service account must have the following roles for the encryption key in addition to the storage.configurer role:

   • kms.keys.encrypter: To read the key, encrypt and upload objects.
   • kms.keys.decrypter: To read the key, decrypt and download objects.
   • kms.keys.encrypterDecrypter: This role includes the kms.keys.encrypter and kms.keys.decrypter permissions.

   For more information, see Key Management Service service roles.

3. Create a static access key.

   As a result, you will get the static access key data. To authenticate in Object Storage, you will need the following:

   • key_id: Static access key ID
   • secret: Secret key

   Save key_id and secret: you will not be able to get the key value again.

InstallationInstallation

You can also build GeeseFS yourself using its source code. For more information, see the guide in the GeeseFS repository on GitHub.

AuthenticationAuthentication

GeeseFS uses the static access key to Object Storage you got earlier. You can set it using one of the following methods:

When using GeeseFS on a Compute Cloud VM that has a linked service account, you can enable simplified authentication that does not require a static access key. To do this, use the --iam option when mounting the bucket.

Mounting a bucketMounting a bucket

Select the folder or disk where you want to mount the bucket. Make sure you have sufficient rights to perform this operation.

When mounting a bucket, you can also configure GeeseFS settings related to system performance and object access rights. To view the list of options and their descriptions, run geesefs --help.

• For one-time bucket mounting:

  Linux/macOS

  Windows

  1. Make sure the .aws/credentials file contains the up-to-date static key data or provide the path to it in the --shared-config parameter.

  2. Create a folder for mounting:

     mkdir <mount_point>
     

  3. Mount the bucket:

     geesefs <bucket_name> <mount_point>
     

     You should specify an existing folder as the mount point.

  1. Make sure the .aws/credentials file contains the up-to-date static key data or provide the path to it in the --shared-config parameter.

  2. Mount the bucket:

     geesefs <bucket_name> <mount_point>
     

     As the mount point, specify the name of the new folder that will be created when you mount the bucket. You cannot specify the name of an existing folder.

• To automatically mount a bucket at system startup:

  macOS

  Linux

  Windows

  1. Create a folder for automatic mounting:

     mkdir <mount_point>
     

  2. Create a file named com.geesefs.automount.plist with the autorun agent configuration:

     nano /Users/<username>/Library/LaunchAgents/com.geesefs.automount.plist
     

  3. Set the agent configuration by specifying the name of the bucket and the absolute path to the mount point:

     <?xml version="1.0" encoding="UTF-8"?>
     <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
     <plist version="1.0">
     <dict>
         <key>Label</key>
         <string>com.geesefs.automount</string>
         <key>ProgramArguments</key>
         <array>
             <string>/usr/local/bin/geesefs</string>
             <string><bucket_name></string>
             <string><absolute_path_to_mount_point></string>
         </array>
         <key>RunAtLoad</key>
         <true/>
         <key>KeepAlive</key>
         <dict>
             <key>NetworkState</key>
             <true/>
         </dict>
     </dict>
     </plist>
     

     Note

     Specify an existing empty folder as the mount point.

     For the bucket to be mounted correctly, provide the full absolute path to the mount point and to the key file without ~. e.g., /home/user/.

  4. Enable the agent you created:

     launchctl load /Users/<username>/Library/LaunchAgents/com.geesefs.automount.plist
     

  5. Reboot and check that the bucket has been mounted to the specified folder.

  To disable agent autorun, use this command:

  launchctl unload /Users/<username>/Library/LaunchAgents/com.geesefs.automount.plist
  

  1. Create a folder for automatic mounting:

     mkdir <mount_point>
     

  2. Open the /etc/fuse.conf file:

     sudo nano /etc/fuse.conf
     

  3. Add the following line to it:

     user_allow_other
     

  4. Open the /etc/fstab file:

     sudo nano /etc/fstab
     

  5. Add the following line to the /etc/fstab file:

     <bucket_name>    /home/<username>/<mount_point>    fuse.geesefs    _netdev,allow_other,--file-mode=0666,--dir-mode=0777,--shared-config=/home/<username>/.aws/credentials    0   0
     

     If you had created the .aws/credentials file for the root user, you do not need to specify the --shared-config parameter.

     Note

     For the bucket to be mounted correctly, provide the full absolute path to the mount point and to the key file without ~, e.g., /home/user/.

  6. Reboot and check that the bucket has been mounted to the specified folder.

  To disable automounting, remove the line with the bucket name from the /etc/fstab file.

  Create a Windows service that will automatically run at system startup:

  1. Run CMD as an administrator.

  2. Run this command:

     sc create <service_name> ^
       binPath="<command_for_mounting>" ^
       DisplayName= "<service_name>" ^
       type=own ^
       start=auto
     

     Where binPath is the path to the geesefs.exe file with the required mounting parameters. Here is an example: C:\geesefs\geesefs.exe <bucket_name> <mount_point>. As the mount point, specify the name of the new folder that will be created when you mount the bucket. You cannot specify the name of an existing folder.

     Result:

     [SC] CreateService: Success
     

  3. Click Start and start typing Services in the Windows search bar. Run the Services application as an administrator.

  4. In the window that opens, find the service you created earlier, right-click it, and select Properties.

  5. On the Log on tab, select This account and specify your Windows account name and password.

     If necessary, click Browse → Advanced → Search to find the user you need on the computer.

  6. Click OK.

  To delete the created service, open CMD as an administrator and run the following command:

  sc delete <service_name>
  

  Result:

  [SC] DeleteService: Success