Migrating PostgreSQL clusters
To migrate your database to Yandex Managed Service for PostgreSQL, you need to directly transfer the data, acquire a write lock for the old database, and switch over the load to the database cluster in Yandex Cloud.
Warning
Users are not transferred automatically to a Managed Service for PostgreSQL cluster. You need to create them again in the new cluster.
Performing migration with Data Transfer allows you to:
- Go without creating an intermediate VM or granting online access to your Managed Service for PostgreSQL target cluster.
- Minimize the DB downtime when migrating data.
- Migrate from older PostgreSQL versions to newer versions.
For more information, see Problems addressed by Yandex Data Transfer.
To use these migration methods, allow connecting to the source cluster from the internet.
Transferring data
Required paid resources
The support cost includes:
- Managed Service for PostgreSQL cluster fee: Using DB hosts and disk space (see Managed Service for PostgreSQL pricing).
- Fee for using public IP addresses if public access is enabled for cluster hosts (see Virtual Private Cloud pricing).
- Transfer fee: Using computing resources and the number of transferred data rows (see Data Transfer pricing).
Transfer the data
-
Set up your infrastructure:
ManuallyTerraform-
Create a Managed Service for PostgreSQL target cluster in any suitable configuration. In this case, the following applies:
- The PostgreSQL version must be the same or higher than in the source cluster. You cannot perform migration while downgrading PostgreSQL version.
- When creating a cluster, specify the same database name as in the source cluster.
- Enable the same PostgreSQL extensions as in the source cluster.
-
Create a source endpoint with the following parameters:
- Database type:
PostgreSQL - Endpoint parameters → Connection settings:
Custom installation
Specify the parameters for connecting to the source cluster.
- Database type:
-
Create a target endpoint with the following parameters:
- Database type:
PostgreSQL - Endpoint parameters → Connection settings:
Managed Service for PostgreSQL cluster
Specify the ID of the target cluster.
- Database type:
-
Create a transfer of the Snapshot and increment type that will use the created endpoints.
-
Warning
Abstain from making any changes to the data schema in the source and target clusters when the data transfer is running. To learn more, see Working with databases during transfer.
-
If you do not have Terraform yet, install it.
-
Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.
-
Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it
. -
Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.
-
Download the data-transfer-pgsql-mpg.tf
configuration file to the same working directory.This file describes:
- Network.
- Subnet.
- Security group and the rule required to connect to a cluster.
- Managed Service for PostgreSQL target cluster.
- Source endpoint.
- Target endpoint.
- Transfer.
-
Specify the following in the
data-transfer-pgsql-mpg.tffile:-
pg-extensions: List of PostgreSQL extensions in the source cluster. -
Target cluster parameters also used as target endpoint parameters:
target_pgsql_version: PostgreSQL version. Must be the same or higher than in the source cluster.target_userandtarget_password: Name and user password of the database owner.
-
Make sure the Terraform configuration files are correct using this command:
terraform validateIf there are any errors in the configuration files, Terraform will point them out.
-
Create the required infrastructure:
-
Run this command to view the planned changes:
terraform planIf you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
-
If everything looks correct, apply the changes:
-
Run this command:
terraform apply -
Confirm updating the resources.
-
Wait for the operation to complete.
-
All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console
.Once created, your transfer will be activated automatically.
-
-
-
Wait for the transfer status to change to Replicating.
-
Switch the source cluster to
read-only
. -
On the transfer monitoring page, wait for the Maximum data transfer delay metric to decrease to zero. This means that all changes that occurred in the source cluster after data copying was completed are transferred to the target cluster.
-
Deactivate the transfer and wait for its status to change to Stopped.
For more information about transfer statuses, see Transfer lifecycle.
-
Transfer the load to the target cluster.
-
Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:
Manually created resourcesResources created with Terraform- Delete the Managed Service for PostgreSQL cluster.
- Delete the stopped transfer.
- Delete the endpoints for both the source and target.
-
In the terminal window, go to the directory containing the infrastructure plan.
Warning
Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.
-
Delete resources:
-
Run this command:
terraform destroy -
Confirm deleting the resources and wait for the operation to complete.
All the resources described in the Terraform manifests will be deleted.
-
Quick data transfer to a new cluster
After creating a Managed Service for PostgreSQL cluster, you can immediately transfer data from another Managed Service for PostgreSQL cluster or custom database installation into it.
This migration method provides:
- Automatic creation of a transfer and endpoints.
- One-time transfer of the data snapshot from the source to the target.
Required paid resources
The support cost includes:
- Managed Service for PostgreSQL cluster fee: Using DB hosts and disk space (see Managed Service for PostgreSQL pricing).
- Fee for using public IP addresses (see Virtual Private Cloud pricing).
- Transfer fee: using computing resources and the number of transferred data rows (see Data Transfer pricing).
Transfer the data
-
Open the Overview page of the new Managed Service for PostgreSQL cluster.
-
At the top of the screen, click Migrate data.
-
Configure the source:
Custom installationManaged Service for PostgreSQL cluster-
Connecting:
-
Select existing connection: Select this option to use an existing connection.
-
Connection ID: Select the connection in Yandex Connection Manager.
-
Database: Enter the DB name.
-
-
Create new connection: Select this option to create a new connection.
-
Hosts: Specify the IP address or FQDN of the master host. If hosts have different ports open for connection, you can specify multiple host values in
host:portformat. In this case, the value of the Port field will be disregarded. -
Username: Enter a username for connecting to the DB.
-
Password: Enter the password to access the database.
-
Database: Enter the DB name.
-
TLS: Enable this setting to use TLS connection.
- CA certificate: Add the certificate file contents.
-
-
-
Connecting:
-
Select existing connection: Select this option to use an existing connection.
-
Cluster ID: Select the cluster to connect to.
-
Connection ID: Select the connection in Yandex Connection Manager.
-
Database: Specify the DB name in the cluster you selected.
-
-
Create new connection: Select this option to create a new connection.
-
Cluster ID: Select the cluster to connect to.
-
Username: Enter a username for connecting to the cluster DB.
-
Password: Enter the user password to access the cluster DB.
-
Database: Specify the DB name in the selected cluster.
-
-
Warning
To use a connection from Connection Manager, the user must have access permissions for this connection of
connection-manager.useror higher. -
-
Configure the target:
-
Connection ID: Select the connection in Connection Manager.
-
Database: Specify the target cluster’s DB name.
-
-
Optionally, configure advanced settings:
-
Cleanup policy: Select the method to clean up data in the target database:
-
Drop: Completely delete the tables included in the transfer (default). -
Truncate: Delete only the data from the tables included in the transfer but keep the schema.
-
-
Included tables: Only data from the tables listed here will be transferred.
-
Excluded tables: Data from the listed tables is not transferred.
The lists include the name of the schema
that describes the DB contents, structure, and integrity constraints, as well as the table name. Both lists support expressions in the following format:<schema_name>.<table_name>: Full table name.<schema_name>.*: All tables in the specified schema.
Table names must match this regular expression:
^"?[-_a-zA-Z0-9.]+"?\\."?[$-_a-zA-Z0-9.*]+"?$Double quotes within a table name are not supported. Outer quotes are only used as delimiters and will be deleted when processing paths.
-
Schema migration: Select the DB schema elements to migrate when activating or deactivating a transfer.
-
-
Click Create to create a transfer and endpoints.
-
Activate the transfer.
-
Wait for the transfer status to change to Completed.
-
Some resources are not free of charge. To avoid unnecessary charges, delete the resources you no longer need:
- Delete the Managed Service for PostgreSQL cluster if you transferred data from it.
- Delete the transfer.
- Delete the source and target endpoints.
Transferring tables with data types from PostgreSQL extensions
Data Transfer allows you to copy tables, the columns of which contain data types defined in PostgreSQL and tables with derived types, i.e., arrays of these types and composite types with fields of these types. Currently, there is a limitation: the data type must implement binary I/O functions. This means that, for the type of data in the pg_typetypsend and typreceive column values must not be zero.
For example, for the PostGISGEOMETRYGEOMETRY_DUMPGEOGRAPHYBOX2DBOX3D
See also
For other migration methods, see this Yandex Managed Service for PostgreSQL tutorial.