Managed Service for Greenplum® API, REST: PXFDatasource.create
Creates PXF datasource
HTTP request
POST https://mdb.api.cloud.yandex.net/managed-greenplum/v1/clusters/{clusterId}/pxf_datasources
Path parameters
Parameter | Description |
---|---|
clusterId | Required. The maximum string length in characters is 50. |
Body parameters
{
"datasource": {
"name": "string",
// `datasource` includes only one of the fields `s3`, `jdbc`, `hdfs`, `hive`
"s3": {
"accessKey": "string",
"secretKey": "string",
"fastUpload": true,
"endpoint": "string"
},
"jdbc": {
"driver": "string",
"url": "string",
"user": "string",
"password": "string",
"statementBatchSize": "integer",
"statementFetchSize": "integer",
"statementQueryTimeout": "integer",
"poolEnabled": true,
"poolMaximumSize": "integer",
"poolConnectionTimeout": "integer",
"poolIdleTimeout": "integer",
"poolMinimumIdle": "integer"
},
"hdfs": {
"core": {
"defaultFs": "string",
"securityAuthToLocal": "string"
},
"kerberos": {
"enable": true,
"primary": "string",
"realm": "string",
"kdcServers": [
"string"
],
"adminServer": "string",
"defaultDomain": "string",
"keytabBase64": "string"
},
"userImpersonation": true,
"username": "string",
"saslConnectionRetries": "integer",
"zkHosts": [
"string"
],
"dfs": {
"haAutomaticFailoverEnabled": true,
"blockAccessTokenEnabled": true,
"useDatanodeHostname": true,
"namenodes": "object",
"nameservices": "string"
},
"yarn": {
"resourcemanagerHaEnabled": true,
"resourcemanagerHaAutoFailoverEnabled": true,
"resourcemanagerHaAutoFailoverEmbedded": true,
"resourcemanagerClusterId": "string",
"haRm": "object"
}
},
"hive": {
"core": {
"defaultFs": "string",
"securityAuthToLocal": "string"
},
"kerberos": {
"enable": true,
"primary": "string",
"realm": "string",
"kdcServers": [
"string"
],
"adminServer": "string",
"defaultDomain": "string",
"keytabBase64": "string"
},
"userImpersonation": true,
"username": "string",
"saslConnectionRetries": "integer",
"zkHosts": [
"string"
],
"ppd": true,
"metastoreUris": [
"string"
],
"metastoreKerberosPrincipal": "string",
"authKerberosPrincipal": "string"
},
// end of the list of possible fields`datasource`
}
}
Field | Description |
---|---|
datasource | object |
datasource. name |
string Required. Data source name. The string length in characters must be 3-200. Value must match the regular expression |
datasource. s3 |
object Settings of an external S3 data source. datasource includes only one of the fields s3 , jdbc , hdfs , hive |
datasource. s3. accessKey |
string Public key to access S3 storage. The maximum string length in characters is 200. |
datasource. s3. secretKey |
string Secret key to access S3 storage. The maximum string length in characters is 200. |
datasource. s3. fastUpload |
boolean (boolean) Manages a fast upload of big files to S3 storage. In case of the The fast upload is enabled by default. |
datasource. s3. endpoint |
string S3 storage address. The default value is The maximum string length in characters is 200. Value must match the regular expression |
datasource. jdbc |
object Settings of an external JDBC data source. datasource includes only one of the fields s3 , jdbc , hdfs , hive |
datasource. jdbc. driver |
string JDBC driver class in Java. The possible values are the following:
The maximum string length in characters is 50. |
datasource. jdbc. url |
string URL that the JDBC driver uses to connect to the database. Examples:
The maximum string length in characters is 1000. |
datasource. jdbc. user |
string Username of the DB owner. The maximum string length in characters is 200. |
datasource. jdbc. password |
string Password of the DB owner. The maximum string length in characters is 200. |
datasource. jdbc. statementBatchSize |
integer (int64) Number of rows to read in an external table, in a batch. The default value is Acceptable values are 50 to 1000, inclusive. |
datasource. jdbc. statementFetchSize |
integer (int64) Number of rows to fetch (buffer) when reading from an external table. The default value is Acceptable values are 50 to 10000, inclusive. |
datasource. jdbc. statementQueryTimeout |
integer (int64) Amount of time (in seconds) the JDBC driver waits for a statement to run. This timeout applies to statements created for both read and write operations. The default value is Acceptable values are 5 to 43200, inclusive. |
datasource. jdbc. poolEnabled |
boolean (boolean) Determines whether JDBC connection pooling is used in a server configuration. By default, it is used. |
datasource. jdbc. poolMaximumSize |
integer (int64) Maximum number of connections to the DB backend. The default value is Acceptable values are 1 to 200, inclusive. |
datasource. jdbc. poolConnectionTimeout |
integer (int64) Maximum amount of time, in milliseconds, after which an inactive connection is considered idle. The default value is Acceptable values are 5000 to 600000, inclusive. |
datasource. jdbc. poolIdleTimeout |
integer (int64) Maximum amount of time, in milliseconds, after which an inactive connection is considered idle. The default value is Acceptable values are 5000 to 600000, inclusive. |
datasource. jdbc. poolMinimumIdle |
integer (int64) Minimum number of idle connections maintained in the connection pool. The default value is Acceptable values are 0 to 200, inclusive. |
datasource. hdfs |
object Settings of an external HDFS data source. datasource includes only one of the fields s3 , jdbc , hdfs , hive |
datasource. hdfs. core |
object Settings of the file system and security rules. |
datasource. hdfs. core. defaultFs |
string URI whose scheme and authority determine the file system implementation. The maximum string length in characters is 200. |
datasource. hdfs. core. securityAuthToLocal |
string Rules for mapping Kerberos principals to operating system user accounts. The maximum string length in characters is 1000. |
datasource. hdfs. kerberos |
object Settings of the Kerberos network authentication protocol. |
datasource. hdfs. kerberos. enable |
boolean (boolean) Determines whether the Kerberos authentication server is used. By default, it is not used. |
datasource. hdfs. kerberos. primary |
string Host of the primary KDC server (Key Distribution Center). The maximum string length in characters is 128. |
datasource. hdfs. kerberos. realm |
string Kerberos realm for a Greenplum® DB. The maximum string length in characters is 1000. |
datasource. hdfs. kerberos. kdcServers[] |
string KDC server hosts. The maximum number of elements is 200. The string length in characters for each value must be 1-200. |
datasource. hdfs. kerberos. adminServer |
string Administration server host. Usually, this is the primary Kerberos server. The maximum string length in characters is 128. |
datasource. hdfs. kerberos. defaultDomain |
string Domain that is used for the host name extension. Applicable when Kerberos 4 service members become Kerberos 5 service members (for example, when rcmd.hostname is replaced with host/hostname.domain). The maximum string length in characters is 128. |
datasource. hdfs. kerberos. keytabBase64 |
string Base64 encoded contents of the keytab file. The maximum string length in characters is 65536. Value must match the regular expression |
datasource. hdfs. userImpersonation |
boolean (boolean) Enables authentication on behalf of the Greenplum® user when connecting to the remote file storage or DBMS. The authentication is disabled by default. |
datasource. hdfs. username |
string Login username for the remote file storage or DBMS if authentication on behalf of the Greenplum® user is enabled. The maximum string length in characters is 128. |
datasource. hdfs. saslConnectionRetries |
integer (int64) Maximum number of times that PXF retries a SASL connection request after a refused connection returns a The default value is Acceptable values are 1 to 50, inclusive. |
datasource. hdfs. zkHosts[] |
string ZooKeeper server hosts. Specify values in the The maximum number of elements is 200. The string length in characters for each value must be 1-200. |
datasource. hdfs. dfs |
object Settings of the distributed file system. |
datasource. hdfs. dfs. haAutomaticFailoverEnabled |
boolean (boolean) Determines whether automatic failover is enabled for the high availability of the file system. The automatic failover is enabled by default. |
datasource. hdfs. dfs. blockAccessTokenEnabled |
boolean (boolean) If The check of access tokens is enabled by default. |
datasource. hdfs. dfs. useDatanodeHostname |
boolean (boolean) Determines whether the datanode hostname is used when connecting to datanodes. |
datasource. hdfs. dfs. namenodes |
object List of HDFS service logical names. Specify them separated by commas. The names can be arbitrary. No more than 10000 per resource. |
datasource. hdfs. dfs. nameservices |
string Corresponds well-known HDFS client setting "dfs.nameservices" for this datasource |
datasource. hdfs. yarn |
object Settings of the ResourceManager service that is responsible for tracking resources in a cluster and scheduling applications (e.g., MapReduce jobs). |
datasource. hdfs. yarn. resourcemanagerHaEnabled |
boolean (boolean) Determines whether high availability is enabled for YARN's ResourceManager services. The high availability is enabled by default. |
datasource. hdfs. yarn. resourcemanagerHaAutoFailoverEnabled |
boolean (boolean) Determines whether another ResourceManager should automatically become active when the active ResourceManager has failed and does not respond. The switch of ResourceManagers is enabled by default if the high availability is enabled. |
datasource. hdfs. yarn. resourcemanagerHaAutoFailoverEmbedded |
boolean (boolean) Determines whether the embedded ActiveStandbyElector method should be used for the election of the active ResourceManager. If the current active ResourceManager has failed and does not respond, the ActiveStandbyElector method makes another ResourceManager active which then takes over. |
datasource. hdfs. yarn. resourcemanagerClusterId |
string Cluster ID. Specify it, so the ResourceManager service does not become active for a different cluster. The maximum string length in characters is 1000. |
datasource. hdfs. yarn. haRm |
object Highly available ResourceManager service. No more than 10000 per resource. |
datasource. hive |
object Settings of an external Hive data source. datasource includes only one of the fields s3 , jdbc , hdfs , hive |
datasource. hive. core |
object Settings of the file system and security rules. |
datasource. hive. core. defaultFs |
string URI whose scheme and authority determine the file system implementation. The maximum string length in characters is 200. |
datasource. hive. core. securityAuthToLocal |
string Rules for mapping Kerberos principals to operating system user accounts. The maximum string length in characters is 1000. |
datasource. hive. kerberos |
object Settings of the Kerberos network authentication protocol. |
datasource. hive. kerberos. enable |
boolean (boolean) Determines whether the Kerberos authentication server is used. By default, it is not used. |
datasource. hive. kerberos. primary |
string Host of the primary KDC server (Key Distribution Center). The maximum string length in characters is 128. |
datasource. hive. kerberos. realm |
string Kerberos realm for a Greenplum® DB. The maximum string length in characters is 1000. |
datasource. hive. kerberos. kdcServers[] |
string KDC server hosts. The maximum number of elements is 200. The string length in characters for each value must be 1-200. |
datasource. hive. kerberos. adminServer |
string Administration server host. Usually, this is the primary Kerberos server. The maximum string length in characters is 128. |
datasource. hive. kerberos. defaultDomain |
string Domain that is used for the host name extension. Applicable when Kerberos 4 service members become Kerberos 5 service members (for example, when rcmd.hostname is replaced with host/hostname.domain). The maximum string length in characters is 128. |
datasource. hive. kerberos. keytabBase64 |
string Base64 encoded contents of the keytab file. The maximum string length in characters is 65536. Value must match the regular expression |
datasource. hive. userImpersonation |
boolean (boolean) Enables authentication on behalf of the Greenplum® user when connecting to the remote file storage or DBMS. The authentication is disabled by default. |
datasource. hive. username |
string Login username for the remote file storage or DBMS if authentication on behalf of the Greenplum® user is enabled. The maximum string length in characters is 128. |
datasource. hive. saslConnectionRetries |
integer (int64) Maximum number of times that PXF retries a SASL connection request after a refused connection returns a The default value is Acceptable values are 1 to 50, inclusive. |
datasource. hive. zkHosts[] |
string ZooKeeper server hosts. Specify values in the The maximum number of elements is 200. The string length in characters for each value must be 1-200. |
datasource. hive. ppd |
boolean (boolean) Specifies if predicate pushdown is enabled for queries on external tables. The predicate pushdown is enabled by default. |
datasource. hive. metastoreUris[] |
string List of URIs separated by commas. To request metadata, the remote DBMS connects to Metastore by one of these URIs. The maximum number of elements is 200. The string length in characters for each value must be 1-200. |
datasource. hive. metastoreKerberosPrincipal |
string Service principal for the Metastore Thrift server. The maximum string length in characters is 1000. |
datasource. hive. authKerberosPrincipal |
string Kerberos server principal. The maximum string length in characters is 1000. |
Response
HTTP Code: 200 - OK
{
"id": "string",
"description": "string",
"createdAt": "string",
"createdBy": "string",
"modifiedAt": "string",
"done": true,
"metadata": "object",
// includes only one of the fields `error`, `response`
"error": {
"code": "integer",
"message": "string",
"details": [
"object"
]
},
"response": "object",
// end of the list of possible fields
}
An Operation resource. For more information, see Operation.
Field | Description |
---|---|
id | string ID of the operation. |
description | string Description of the operation. 0-256 characters long. |
createdAt | string (date-time) Creation timestamp. String in RFC3339 text format. The range of possible values is from To work with values in this field, use the APIs described in the Protocol Buffers reference. In some languages, built-in datetime utilities do not support nanosecond precision (9 digits). |
createdBy | string ID of the user or service account who initiated the operation. |
modifiedAt | string (date-time) The time when the Operation resource was last modified. String in RFC3339 text format. The range of possible values is from To work with values in this field, use the APIs described in the Protocol Buffers reference. In some languages, built-in datetime utilities do not support nanosecond precision (9 digits). |
done | boolean (boolean) If the value is |
metadata | object Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any. |
error | object The error result of the operation in case of failure or cancellation. includes only one of the fields error , response |
error. code |
integer (int32) Error code. An enum value of google.rpc.Code. |
error. message |
string An error message. |
error. details[] |
object A list of messages that carry the error details. |
response | object includes only one of the fields error , response The normal response of the operation in case of success. If the original method returns no data on success, such as Delete, the response is google.protobuf.Empty. If the original method is the standard Create/Update, the response should be the target resource of the operation. Any method that returns a long-running operation should document the response type, if any. |