Yandex Cloud
Search
Contact UsGet started
  • Blog
  • Pricing
  • Documentation
  • All Services
  • System Status
    • Featured
    • Infrastructure & Network
    • Data Platform
    • Containers
    • Developer tools
    • Serverless
    • Security
    • Monitoring & Resources
    • ML & AI
    • Business tools
  • All Solutions
    • By industry
    • By use case
    • Economics and Pricing
    • Security
    • Technical Support
    • Customer Stories
    • Cloud credits to scale your IT product
    • Gateway to Russia
    • Cloud for Startups
    • Education and Science
    • Yandex Cloud Partner program
  • Blog
  • Pricing
  • Documentation
© 2025 Direct Cursus Technology L.L.C.
Yandex Object Storage
    • All tutorials
    • Getting statistics on object queries with S3 Select
    • Getting website traffic statistics with S3 Select
    • Getting statistics on object queries using Yandex Query
    • Generating a resource-by-resource cost breakdown report using S3 Select
    • Server-side encryption
    • Integrating an L7 load balancer with CDN and Object Storage
    • Blue-green and canary deployment of service versions
    • Analyzing logs in DataLens
    • Mounting buckets to the file system of Yandex Data Processing hosts
    • Using Object Storage in Yandex Data Processing
    • Importing data from Object Storage, processing and exporting to Managed Service for ClickHouse®
    • Mounting a bucket as a disk in Windows
    • Migrating data from Yandex Data Streams using Yandex Data Transfer
    • Using hybrid storage in Yandex Managed Service for ClickHouse®
    • Loading data from Yandex Managed Service for OpenSearch to Yandex Object Storage using Yandex Data Transfer
    • Automatically copying objects from one bucket to another
    • Recognizing audio files in a bucket on a regular basis
    • Training a model in Yandex DataSphere on data from Object Storage
    • Connecting to Object Storage from VPC
    • Migrating data to Yandex Managed Service for PostgreSQL using Yandex Data Transfer
    • Uploading data to Yandex Managed Service for Greenplum® using Yandex Data Transfer
    • Uploading data to Yandex Managed Service for ClickHouse® using Yandex Data Transfer
    • Uploading data to Yandex Managed Service for YDB using Yandex Data Transfer
    • Exchanging data between Yandex Managed Service for ClickHouse® and Yandex Data Processing
    • Uploading data from Yandex Managed Service for YDB using Yandex Data Transfer
    • Hosting a static Gatsby website in Object Storage
    • Migrating a database from Managed Service for PostgreSQL to Object Storage
    • Exchanging data between Yandex Managed Service for ClickHouse® and Yandex Data Processing
    • Importing data from Yandex Managed Service for PostgreSQL to Yandex Data Processing using Sqoop
    • Importing data from Yandex Managed Service for MySQL® to Yandex Data Processing using Sqoop
    • Migrating data from Yandex Object Storage to Yandex Managed Service for MySQL® using Yandex Data Transfer
    • Migrating a database from Yandex Managed Service for MySQL® to Yandex Object Storage
    • Exporting Greenplum® data to a cold storage in Yandex Object Storage
    • Loading data from Yandex Direct to a Yandex Managed Service for ClickHouse® data mart using Yandex Cloud Functions, Yandex Object Storage, and Yandex Data Transfer
    • Migrating data from Elasticsearch to Yandex Managed Service for OpenSearch
    • Uploading Terraform states to Object Storage
    • Locking Terraform states using Managed Service for YDB
    • Visualizing Yandex Query data
    • Publishing game updates
    • VM backups using Hystax Acura
    • Backing up to Object Storage with CloudBerry Desktop Backup
    • Backing up to Object Storage with Duplicati
    • Backing up to Object Storage with Bacula
    • Backing up to Yandex Object Storage with Veeam Backup
    • Backing up to Object Storage with Veritas Backup Exec
    • Managed Service for Kubernetes cluster backups in Object Storage
    • Developing a custom integration in API Gateway
    • URL shortener
    • Storing application runtime logs
    • Developing a skill for Alice and a website with authorization
    • Creating an interactive serverless application using WebSocket
    • Deploying a web application using the Java Servlet API
    • Developing a Telegram bot
    • Replicating logs to Object Storage using Fluent Bit
    • Replicating logs to Object Storage using Data Streams
    • Uploading audit logs to ArcSight SIEM
    • Exporting audit logs to SIEM Splunk systems
    • Creating an MLFlow server for logging experiments and artifacts
    • Operations with data using Yandex Query
    • Federated data queries using Query
    • Recognizing text in image archives using Vision OCR
    • Converting a video to a GIF in Python
    • Automating tasks using Managed Service for Apache Airflow™
    • Processing files with usage details in Yandex Cloud Billing
    • Deploying a web app with JWT authorization in API Gateway and authentication in Firebase
    • Searching for Yandex Cloud events in Yandex Query
    • Searching for Yandex Cloud events in Object Storage
    • Creating an external table from a bucket table using a configuration file
    • Migrating databases from Google BigQuery to Managed Service for ClickHouse®
  • Terraform reference
  • Monitoring metrics
  • Audit Trails events
  • Bucket logs
  • Release notes
  • FAQ

In this article:

  • Required paid resources
  • Getting started
  • Set up your web server with a configuration file
  • Create an external table
  • Delete the resources you created
  1. Tutorials
  2. Creating an external table from a bucket table using a configuration file

Creating an external table from a bucket table using a configuration file

Written by
Yandex Cloud
Updated at May 5, 2025
  • Required paid resources
  • Getting started
  • Set up your web server with a configuration file
  • Create an external table
  • Delete the resources you created

To create an external table from a table in a Yandex Object Storage bucket, you need to provide a static access key for the service account in the query. You can do this using the S3 protocol and a configuration file stored on the HTTP server.

To create an external table using a configuration file:

  1. Set up your web server with a configuration file.
  2. Create an external table.

If you no longer need the resources you created, delete them.

Required paid resourcesRequired paid resources

The support cost includes:

  • Managed Service for Greenplum® cluster fee: Using computing resources allocated to hosts and disk space (see Greenplum® pricing).
  • NAT gateway fee (see Virtual Private Cloud pricing).
  • Object Storage bucket fee: Storing data and performing operations with it (see Object Storage pricing).
  • VM fee: Using computing resources, storage, and, optionally, public IP address (see Compute Cloud pricing).
  • Fee for using public IP addresses if public access is enabled for cluster hosts (see Virtual Private Cloud pricing).

Getting startedGetting started

Set up your infrastructure:

Manually
Terraform
  1. Create a Managed Service for Greenplum® cluster with any suitable configuration.

  2. In the cluster subnet, set up a NAT gateway and create a security group allowing all incoming and outgoing traffic from all addresses.

  3. Create a virtual machine on Linux in the same cloud network as the Managed Service for Greenplum® cluster.

  4. Create a Object Storage bucket with restricted access. Upload the example.csv file with a test table to it:

    10,2010
    
  5. Create a static access key for the service account.

  1. If you do not have Terraform yet, install it.

  2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.

  3. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.

  4. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.

  5. In the working directory, create a file called example.csv with a test table:

    10,2010
    
  6. Download the infrastructure configuration file to the same working directory.

    This file describes:

    • Network.
    • Subnet.
    • Service account with a static access key.
    • Managed Service for Greenplum® cluster.
    • Bucket to contain the example.csv file.
    • Virtual machine on Ubuntu 20.04.
  7. Specify the following in the greenplum-s3-vm.tf file:

    • Password for the user named user for access to the Managed Service for Greenplum® cluster.
    • Virtual machine image ID.
    • Username and path to the SSH key for virtual machine access.
    • Folder ID for the same service account as specified in the provider settings.
    • Bucket name that must be unique throughout Object Storage.
  8. In the terminal window, go to the directory containing the infrastructure plan.

  9. To verify that the config files are correct, run the command below:

    terraform validate
    

    If there are any errors in the configuration files, Terraform will point them out.

  10. Create the infrastructure required to follow the steps provided in this tutorial:

    1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

    2. If everything looks correct, apply the changes:

      1. Run this command:

        terraform apply
        
      2. Confirm updating the resources.

      3. Wait for the operation to complete.

    All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

  11. To retrieve the static key parameters, run the command below in the working directory:

    terraform output -raw access_key > static-key.txt && \
    echo \n >> static-key.txt && \
    terraform output -raw secret_key >> static-key.txt
    

    The command saves to a file named static-key.txt the static key ID and the static key you are going to need later.

  12. Go to the management console and set up a NAT gateway for the subnet hosting your cluster.

Set up your web server with a configuration fileSet up your web server with a configuration file

Set up a web server on your virtual machine and create an s3 configuration file on the web server:

  1. Connect to the virtual machine over SSH.

  2. Create a folder named /opt/gp_http_server and put into it the s3.config file with the static key parameters and the other configuration settings for the s3 protocol:

    sudo mkdir /opt/gp_http_server
    

    s3.config file contents:

    [default]
    accessid = "<static_key_ID>"
    secret = "<static_key>"
    threadnum = 4
    chunksize = 67108864
    low_speed_limit = 10240
    low_speed_time = 60
    encryption = true
    version = 2
    proxy = ""
    autocompress = true
    verifycert = true
    server_side_encryption = ""
    # gpcheckcloud config
    gpcheckcloud_newline = "\n"
    
  3. In the /etc/systemd/system/ folder, put a web server config file named gp_s3_config_http.service:

    [Unit]
    Description=HTTP server config to connect Greenplum with Object Storage
    After=network.target
    
    [Service]
    KillMode=process
    Restart=on-failure
    Type=simple
    User=root
    ExecStart=python3 -m http.server 8553 --directory /opt/gp_http_server
    
    [Install]
    WantedBy=multi-user.target
    
  4. Start your web server:

    sudo systemctl daemon-reload && \
    sudo systemctl enable gp_s3_config_http && \
    sudo systemctl start gp_s3_config_http
    
  5. To check the state of the web server, run the command below:

    sudo systemctl status gp_s3_config_http
    

Create an external tableCreate an external table

  1. Connect to the Managed Service for Greenplum® cluster.

  2. Run a query to create an external table referencing the example.csv table in the bucket:

    CREATE READABLE EXTERNAL TABLE s3_ext_table(id int, year int)
    LOCATION('s3://storage.yandexcloud.net/<bucket_name>/example.csv config_server=http://<VM_internal_IP_address>:8553/s3.config region=ru-central1-a')
    FORMAT 'csv';
    

    A successful query will return the following response:

    CREATE EXTERNAL TABLE
    
  3. Query the external table for data:

    SELECT * FROM s3_ext_table;
    

    Expected result:

     id | year
    ----+------
     10 | 2010
    (1 row)
    

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

Manually
Terraform
  1. Delete the VM.
  2. If you reserved a public static IP address for the VM, delete it.
  3. Delete the bucket in Object Storage.
  4. Delete the Managed Service for Greenplum® cluster.
  5. Delete the service account.
  6. Delete the subnet.
  7. Delete the route table.
  8. Delete the NAT gateway.
  9. Delete the cloud network.
  1. In the terminal window, go to the directory containing the infrastructure plan.

    Warning

    Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.

  2. Delete resources:

    1. Run this command:

      terraform destroy
      
    2. Confirm deleting the resources and wait for the operation to complete.

    All the resources described in the Terraform manifests will be deleted.

Greenplum® and Greenplum Database® are registered trademarks or trademarks of VMware, Inc. in the United States and/or other countries.

Was the article helpful?

Previous
Searching for Yandex Cloud events in Object Storage
Next
Migrating databases from Google BigQuery to Managed Service for ClickHouse®
© 2025 Direct Cursus Technology L.L.C.