Docker container for mounting an Object Storage bucket to an (S)FTP(S) server

To access a bucket over FTP, FTPS, or SFTP, you can deploy the server using a public Docker container provided by Object Storage.

A Docker container implements links between the Object Storage GeeseFS FUSE client and servers: vsftpd for FTP and FTPS and sftp-server (part of OpenSSH) for SFTP.

Getting startedGetting started

1. Create a service account.
2. Assign the service account the roles required for your project. For more information about roles, see the Identity and Access Management documentation.
3. Create a static access key.

InstallingInstalling

1. Install Docker.

2. Get authenticated in Yandex Container Registry.

3. Pull a Docker container:

   docker pull cr.yandex/crp9ftr22d26age3hulg/ftp-s3-gateway:1.0
   

4. Create a directory named secrets to store FTP server user data and bucket mounting settings:

   mkdir secrets
   

5. In the secrets directory:

   • Create a credentials file:

     [default]
       aws_access_key_id = <key_id>
       aws_secret_access_key = <key_value>
     

     Where:

     • aws_access_key_id: ID of the static access key obtained before starting.
     • aws_secret_access_key: Contents of the static access key.

   • If you are going to use SFTP, create a file named authorized_keys with a public SSH key:

     ssh-ed25519 AAAAB3Nz.....BdZoeQ==
     

     To learn how to create an SSH key pair, see the Compute Cloud documentation.

   • If you are going to use FTPS, add the ftp.pem TLS certificate and its ftp.key to the folder. For example, for testing purposes, you can issue a self-signed certificate:

     openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
       -keyout secrets/ftp.key -out secrets/ftp.pem
     

6. Create an env.list file with environment variables for the Docker container:

   <variable_name>=<variable_value>
   ...
   

   Supported variables:

   • S3_BUCKET: Bucket name or path to its folder to be mounted to the FTP server, in <bucket_name>:<folder_path> format. This is a required variable.
   • SFTP: Enables the use of SFTP. By default, it is set to YES.
   • FTP: Enables the use of FTP. By default, it is set to NO.
   • FTP_USER: Username for a server connection. By default, it is set to s3.
   • FTP_PASS: User password for a server connection. By default, a random password is generated and displayed in Docker container logs.
   • FTP_PASV_ENABLE: Enables passive FTP connection mode. By default, it is set to YES.
   • FTP_PASV_MIN_PORT: Start of the port range for passive mode. By default, it is set to 21100.
   • FTP_PASV_MAX_PORT: End of the port range for passive mode. By default, it is set to 21100.
   • FTP_PASV_ADDRESS: Server IP address or its domain name (if the FTP_PASV_ADDR_RESOLVE option is selected) for passive mode. By default, the IP address specified in the Docker container's route table (the ip route show command) is used as the default route target IP address (specified in a default via <IP_address> ... string).
   • FTP_PASV_ADDR_RESOLVE: Allows specifying the server domain name instead of its IP address in the FTP_PASV_ADDRESS variable. By default, it is set to YES.
   • FTP_PASV_PROMISCUOUS: Disables client IP address mapping for passive mode: a managing connection may be opened from one client address, while a connection for data exchange, from another. By default, it is set to NO. We do not recommend disabling this check.
   • FTP_PORT_PROMISCUOUS: Disables client IP address mapping for active mode: when a managing connection is established, a client can specify another client's address in the PORT command. By default, it is set to NO. We do not recommend disabling this check.
   • FTP_SSL_ENABLE: Enables the use of FTPS (over TLS 1.x) instead of FTP:
     • YES (default): FTPS is enabled but optional. Clients can establish non-secure FTP connections to the server.
     • FORCE: FTPS is enabled and required. Clients can only establish secure FTPS connections to the server.
     • NO: FTPS is disabled.
   • FTP_RSA_CERT_FILE: Path to the TLS certificate inside the Docker container. By default, it is set to /secrets/ftp.pem.
   • FTP_RSA_PRIVATE_KEY_FILE: Path to the private key of the TLS certificate inside the Docker container. By default, it is set to /secrets/ftp.key.

7. Run the Docker container:

   SFTP

   FTP(S)

   docker run -d -it \
     --cap-add SYS_ADMIN \
     --device /dev/fuse \
     --security-opt apparmor:unconfined \
     --env-file env.list \
     -v <full_path_to_the_secrets_folder>:/secrets \
     -p 1022:22 \
     --name ftp \
     cr.yandex/crp9ftr22d26age3hulg/ftp-s3-gateway:1.0
   

   The server will accept connections on port 1022.

   docker run -d -it \
     --cap-add SYS_ADMIN \
     --device /dev/fuse \
     --security-opt apparmor:unconfined \
     --env-file env.list \
     -v <full_path_to_the_secrets_folder>:/secrets \
     --expose 21 \
     -p 1021:21 \
     --expose 21100 \
     -p 21100:21100 \
     --name ftp \
     cr.yandex/crp9ftr22d26age3hulg/ftp-s3-gateway:1.0
   

   The server will accept connections on port 1021. In addition, for passive mode (the FTP_PASV_ENABLE variable), port 21100 is open: if you do not use this mode, the --expose 21100 and -p 21100:21100 options are not required.

Specifics of uploading files to a bucketSpecifics of uploading files to a bucket

The GeeseFS client that is part of a Docker container works with files asynchronously. It caches new files and uploads them to a bucket after a while. If an FTP server connection is broken between these two points of time, uploaded files may be lost either partially or completely.

To ensure data integrity when establishing SFTP connections, use the fsync@openssh.com extension so that file uploads are considered successful only after the fsync system call. For example, for the sftp client that is part of OpenSSH, the extension is enabled with the -f flag: sftp -f <server_address>. Waiting for fsync calls slows down operations with files.