Managed Service for Greenplum® API, gRPC: PXFDatasourceService
Written by
Updated at June 4, 2024
Call | Description |
---|---|
List | List all PXF datasources |
Create | Creates PXF datasource |
Update | Update PXF datasource |
Delete | Delete PXF datasource |
Calls PXFDatasourceService
List
List all PXF datasources
rpc List (ListPXFDatasourcesRequest) returns (ListPXFDatasourcesResponse)
ListPXFDatasourcesRequest
Field | Description |
---|---|
cluster_id | string Required. The maximum string length in characters is 50. |
ListPXFDatasourcesResponse
Field | Description |
---|---|
datasources[] | PXFDatasource |
PXFDatasource
Field | Description |
---|---|
name | string Required. Data source name. The string length in characters must be 3-200. Value must match the regular expression ^[^\\|/*?.,;\"'<>]+$ . |
settings | oneof: s3 , jdbc , hdfs or hive |
s3 | PXFDatasourceS3 Settings of an external S3 data source. |
jdbc | PXFDatasourceJDBC Settings of an external JDBC data source. |
hdfs | PXFDatasourceHDFS Settings of an external HDFS data source. |
hive | PXFDatasourceHive Settings of an external Hive data source. |
PXFDatasourceS3
Field | Description |
---|---|
access_key | string Public key to access S3 storage. The maximum string length in characters is 200. |
secret_key | string Secret key to access S3 storage. The maximum string length in characters is 200. |
fast_upload | google.protobuf.BoolValue Manages a fast upload of big files to S3 storage. In case of the false value, the PXF generates files on disk before sending them to the S3 storage. In case of the true value, the PXF generates files in RAM (the PXF writes to disc only if there is not enough RAM). The fast upload is enabled by default. |
endpoint | string S3 storage address. The default value is storage.yandexcloud.net used for Yandex Object Storage. The maximum string length in characters is 200. Value must match the regular expression (((s3|https?)://)?([a-z0-9]([a-z0-9-]*[a-z0-9])?[.])+[a-z]{2,})? . |
PXFDatasourceJDBC
Field | Description |
---|---|
driver | string JDBC driver class in Java. The possible values are the following:
|
url | string URL that the JDBC driver uses to connect to the database. Examples:
|
user | string Username of the DB owner. The maximum string length in characters is 200. |
password | string Password of the DB owner. The maximum string length in characters is 200. |
statement_batch_size | google.protobuf.Int64Value Number of rows to read in an external table, in a batch. The default value is 100 . Acceptable values are 50 to 1000, inclusive. |
statement_fetch_size | google.protobuf.Int64Value Number of rows to fetch (buffer) when reading from an external table. The default value is 1000 . Acceptable values are 50 to 10000, inclusive. |
statement_query_timeout | google.protobuf.Int64Value Amount of time (in seconds) the JDBC driver waits for a statement to run. This timeout applies to statements created for both read and write operations. The default value is 60 . Acceptable values are 5 to 43200, inclusive. |
pool_enabled | google.protobuf.BoolValue Determines whether JDBC connection pooling is used in a server configuration. By default, it is used. |
pool_maximum_size | google.protobuf.Int64Value Maximum number of connections to the DB backend. The default value is 5 . Acceptable values are 1 to 200, inclusive. |
pool_connection_timeout | google.protobuf.Int64Value Maximum amount of time, in milliseconds, after which an inactive connection is considered idle. The default value is 30000 . Acceptable values are 5000 to 600000, inclusive. |
pool_idle_timeout | google.protobuf.Int64Value Maximum amount of time, in milliseconds, after which an inactive connection is considered idle. The default value is 30000 . Acceptable values are 5000 to 600000, inclusive. |
pool_minimum_idle | google.protobuf.Int64Value Minimum number of idle connections maintained in the connection pool. The default value is 0 . Acceptable values are 0 to 200, inclusive. |
PXFDatasourceHDFS
Field | Description |
---|---|
core | PXFDatasourceCore Settings of the file system and security rules. |
kerberos | PXFDatasourceKerberos Settings of the Kerberos network authentication protocol. |
user_impersonation | google.protobuf.BoolValue Enables authentication on behalf of the Greenplum® user when connecting to the remote file storage or DBMS. The authentication is disabled by default. |
username | string Login username for the remote file storage or DBMS if authentication on behalf of the Greenplum® user is enabled. The maximum string length in characters is 128. |
sasl_connection_retries | google.protobuf.Int64Value Maximum number of times that PXF retries a SASL connection request after a refused connection returns a GSS initiate failed error. The default value is 5 . Acceptable values are 1 to 50, inclusive. |
zk_hosts[] | string ZooKeeper server hosts. Specify values in the <address>:<port> format. The maximum number of elements is 200. The string length in characters for each value must be 1-200. |
dfs | PXFDatasourceHDFSDfs Settings of the distributed file system. |
yarn | PXFDatasourceHDFSYarn Settings of the ResourceManager service that is responsible for tracking resources in a cluster and scheduling applications (e.g., MapReduce jobs). |
PXFDatasourceCore
Field | Description |
---|---|
default_fs | string URI whose scheme and authority determine the file system implementation. The maximum string length in characters is 200. |
security_auth_to_local | string Rules for mapping Kerberos principals to operating system user accounts. The maximum string length in characters is 1000. |
PXFDatasourceKerberos
Field | Description |
---|---|
enable | google.protobuf.BoolValue Determines whether the Kerberos authentication server is used. By default, it is not used. |
primary | string Host of the primary KDC server (Key Distribution Center). The maximum string length in characters is 128. |
realm | string Kerberos realm for a Greenplum® DB. The maximum string length in characters is 1000. |
kdc_servers[] | string KDC server hosts. The maximum number of elements is 200. The string length in characters for each value must be 1-200. |
admin_server | string Administration server host. Usually, this is the primary Kerberos server. The maximum string length in characters is 128. |
default_domain | string Domain that is used for the host name extension. Applicable when Kerberos 4 service members become Kerberos 5 service members (for example, when rcmd.hostname is replaced with host/hostname.domain). The maximum string length in characters is 128. |
keytab_base64 | string Base64 encoded contents of the keytab file. Value must match the regular expression ^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$ . The maximum string length in characters is 65536. |
PXFDatasourceHDFSDfs
Field | Description |
---|---|
ha_automatic_failover_enabled | google.protobuf.BoolValue Determines whether automatic failover is enabled for the high availability of the file system. The automatic failover is enabled by default. |
block_access_token_enabled | google.protobuf.BoolValue If true , access tokens are used as capabilities for accessing datanodes. If false , no access tokens are checked on accessing datanodes. The check of access tokens is enabled by default. |
use_datanode_hostname | google.protobuf.BoolValue Determines whether the datanode hostname is used when connecting to datanodes. |
namenodes | map<string,PXFDatasourceHDFSDfsNamenode> List of HDFS service logical names. Specify them separated by commas. The names can be arbitrary. No more than 10000 per resource. |
nameservices | string Corresponds well-known HDFS client setting "dfs.nameservices" for this datasource |
PXFDatasourceHDFSYarn
Field | Description |
---|---|
resourcemanager_ha_enabled | google.protobuf.BoolValue Determines whether high availability is enabled for YARN's ResourceManager services. The high availability is enabled by default. |
resourcemanager_ha_auto_failover_enabled | google.protobuf.BoolValue Determines whether another ResourceManager should automatically become active when the active ResourceManager has failed and does not respond. The switch of ResourceManagers is enabled by default if the high availability is enabled. |
resourcemanager_ha_auto_failover_embedded | google.protobuf.BoolValue Determines whether the embedded ActiveStandbyElector method should be used for the election of the active ResourceManager. If the current active ResourceManager has failed and does not respond, the ActiveStandbyElector method makes another ResourceManager active which then takes over. |
resourcemanager_cluster_id | string Cluster ID. Specify it, so the ResourceManager service does not become active for a different cluster. The maximum string length in characters is 1000. |
ha_rm | map<string,PXFDatasourceHDFSYarnHaRm> Highly available ResourceManager service. No more than 10000 per resource. |
PXFDatasourceHive
Field | Description |
---|---|
core | PXFDatasourceCore Settings of the file system and security rules. |
kerberos | PXFDatasourceKerberos Settings of the Kerberos network authentication protocol. |
user_impersonation | google.protobuf.BoolValue Enables authentication on behalf of the Greenplum® user when connecting to the remote file storage or DBMS. The authentication is disabled by default. |
username | string Login username for the remote file storage or DBMS if authentication on behalf of the Greenplum® user is enabled. The maximum string length in characters is 128. |
sasl_connection_retries | google.protobuf.Int64Value Maximum number of times that PXF retries a SASL connection request after a refused connection returns a GSS initiate failed error. The default value is 5 . Acceptable values are 1 to 50, inclusive. |
zk_hosts[] | string ZooKeeper server hosts. Specify values in the <address>:<port> format. The maximum number of elements is 200. The string length in characters for each value must be 1-200. |
ppd | google.protobuf.BoolValue Specifies if predicate pushdown is enabled for queries on external tables. The predicate pushdown is enabled by default. |
metastore_uris[] | string List of URIs separated by commas. To request metadata, the remote DBMS connects to Metastore by one of these URIs. The maximum number of elements is 200. The string length in characters for each value must be 1-200. |
metastore_kerberos_principal | string Service principal for the Metastore Thrift server. The maximum string length in characters is 1000. |
auth_kerberos_principal | string Kerberos server principal. The maximum string length in characters is 1000. |
Create
Creates PXF datasource
rpc Create (CreatePXFDatasourceRequest) returns (operation.Operation)
Metadata and response of Operation:
Operation.metadata:CreatePXFDatasourceMetadata
Operation.response:PXFDatasource
CreatePXFDatasourceRequest
Field | Description |
---|---|
cluster_id | string Required. The maximum string length in characters is 50. |
datasource | PXFDatasource |
PXFDatasource
Field | Description |
---|---|
name | string Required. Data source name. The string length in characters must be 3-200. Value must match the regular expression ^[^\\|/*?.,;\"'<>]+$ . |
settings | oneof: s3 , jdbc , hdfs or hive |
s3 | PXFDatasourceS3 Settings of an external S3 data source. |
jdbc | PXFDatasourceJDBC Settings of an external JDBC data source. |
hdfs | PXFDatasourceHDFS Settings of an external HDFS data source. |
hive | PXFDatasourceHive Settings of an external Hive data source. |
PXFDatasourceS3
Field | Description |
---|---|
access_key | string Public key to access S3 storage. The maximum string length in characters is 200. |
secret_key | string Secret key to access S3 storage. The maximum string length in characters is 200. |
fast_upload | google.protobuf.BoolValue Manages a fast upload of big files to S3 storage. In case of the false value, the PXF generates files on disk before sending them to the S3 storage. In case of the true value, the PXF generates files in RAM (the PXF writes to disc only if there is not enough RAM). The fast upload is enabled by default. |
endpoint | string S3 storage address. The default value is storage.yandexcloud.net used for Yandex Object Storage. The maximum string length in characters is 200. Value must match the regular expression (((s3|https?)://)?([a-z0-9]([a-z0-9-]*[a-z0-9])?[.])+[a-z]{2,})? . |
PXFDatasourceJDBC
Field | Description |
---|---|
driver | string JDBC driver class in Java. The possible values are the following:
|
url | string URL that the JDBC driver uses to connect to the database. Examples:
|
user | string Username of the DB owner. The maximum string length in characters is 200. |
password | string Password of the DB owner. The maximum string length in characters is 200. |
statement_batch_size | google.protobuf.Int64Value Number of rows to read in an external table, in a batch. The default value is 100 . Acceptable values are 50 to 1000, inclusive. |
statement_fetch_size | google.protobuf.Int64Value Number of rows to fetch (buffer) when reading from an external table. The default value is 1000 . Acceptable values are 50 to 10000, inclusive. |
statement_query_timeout | google.protobuf.Int64Value Amount of time (in seconds) the JDBC driver waits for a statement to run. This timeout applies to statements created for both read and write operations. The default value is 60 . Acceptable values are 5 to 43200, inclusive. |
pool_enabled | google.protobuf.BoolValue Determines whether JDBC connection pooling is used in a server configuration. By default, it is used. |
pool_maximum_size | google.protobuf.Int64Value Maximum number of connections to the DB backend. The default value is 5 . Acceptable values are 1 to 200, inclusive. |
pool_connection_timeout | google.protobuf.Int64Value Maximum amount of time, in milliseconds, after which an inactive connection is considered idle. The default value is 30000 . Acceptable values are 5000 to 600000, inclusive. |
pool_idle_timeout | google.protobuf.Int64Value Maximum amount of time, in milliseconds, after which an inactive connection is considered idle. The default value is 30000 . Acceptable values are 5000 to 600000, inclusive. |
pool_minimum_idle | google.protobuf.Int64Value Minimum number of idle connections maintained in the connection pool. The default value is 0 . Acceptable values are 0 to 200, inclusive. |
PXFDatasourceHDFS
Field | Description |
---|---|
core | PXFDatasourceCore Settings of the file system and security rules. |
kerberos | PXFDatasourceKerberos Settings of the Kerberos network authentication protocol. |
user_impersonation | google.protobuf.BoolValue Enables authentication on behalf of the Greenplum® user when connecting to the remote file storage or DBMS. The authentication is disabled by default. |
username | string Login username for the remote file storage or DBMS if authentication on behalf of the Greenplum® user is enabled. The maximum string length in characters is 128. |
sasl_connection_retries | google.protobuf.Int64Value Maximum number of times that PXF retries a SASL connection request after a refused connection returns a GSS initiate failed error. The default value is 5 . Acceptable values are 1 to 50, inclusive. |
zk_hosts[] | string ZooKeeper server hosts. Specify values in the <address>:<port> format. The maximum number of elements is 200. The string length in characters for each value must be 1-200. |
dfs | PXFDatasourceHDFSDfs Settings of the distributed file system. |
yarn | PXFDatasourceHDFSYarn Settings of the ResourceManager service that is responsible for tracking resources in a cluster and scheduling applications (e.g., MapReduce jobs). |
PXFDatasourceCore
Field | Description |
---|---|
default_fs | string URI whose scheme and authority determine the file system implementation. The maximum string length in characters is 200. |
security_auth_to_local | string Rules for mapping Kerberos principals to operating system user accounts. The maximum string length in characters is 1000. |
PXFDatasourceKerberos
Field | Description |
---|---|
enable | google.protobuf.BoolValue Determines whether the Kerberos authentication server is used. By default, it is not used. |
primary | string Host of the primary KDC server (Key Distribution Center). The maximum string length in characters is 128. |
realm | string Kerberos realm for a Greenplum® DB. The maximum string length in characters is 1000. |
kdc_servers[] | string KDC server hosts. The maximum number of elements is 200. The string length in characters for each value must be 1-200. |
admin_server | string Administration server host. Usually, this is the primary Kerberos server. The maximum string length in characters is 128. |
default_domain | string Domain that is used for the host name extension. Applicable when Kerberos 4 service members become Kerberos 5 service members (for example, when rcmd.hostname is replaced with host/hostname.domain). The maximum string length in characters is 128. |
keytab_base64 | string Base64 encoded contents of the keytab file. Value must match the regular expression ^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$ . The maximum string length in characters is 65536. |
PXFDatasourceHDFSDfs
Field | Description |
---|---|
ha_automatic_failover_enabled | google.protobuf.BoolValue Determines whether automatic failover is enabled for the high availability of the file system. The automatic failover is enabled by default. |
block_access_token_enabled | google.protobuf.BoolValue If true , access tokens are used as capabilities for accessing datanodes. If false , no access tokens are checked on accessing datanodes. The check of access tokens is enabled by default. |
use_datanode_hostname | google.protobuf.BoolValue Determines whether the datanode hostname is used when connecting to datanodes. |
namenodes | map<string,PXFDatasourceHDFSDfsNamenode> List of HDFS service logical names. Specify them separated by commas. The names can be arbitrary. No more than 10000 per resource. |
nameservices | string Corresponds well-known HDFS client setting "dfs.nameservices" for this datasource |
PXFDatasourceHDFSYarn
Field | Description |
---|---|
resourcemanager_ha_enabled | google.protobuf.BoolValue Determines whether high availability is enabled for YARN's ResourceManager services. The high availability is enabled by default. |
resourcemanager_ha_auto_failover_enabled | google.protobuf.BoolValue Determines whether another ResourceManager should automatically become active when the active ResourceManager has failed and does not respond. The switch of ResourceManagers is enabled by default if the high availability is enabled. |
resourcemanager_ha_auto_failover_embedded | google.protobuf.BoolValue Determines whether the embedded ActiveStandbyElector method should be used for the election of the active ResourceManager. If the current active ResourceManager has failed and does not respond, the ActiveStandbyElector method makes another ResourceManager active which then takes over. |
resourcemanager_cluster_id | string Cluster ID. Specify it, so the ResourceManager service does not become active for a different cluster. The maximum string length in characters is 1000. |
ha_rm | map<string,PXFDatasourceHDFSYarnHaRm> Highly available ResourceManager service. No more than 10000 per resource. |
PXFDatasourceHive
Field | Description |
---|---|
core | PXFDatasourceCore Settings of the file system and security rules. |
kerberos | PXFDatasourceKerberos Settings of the Kerberos network authentication protocol. |
user_impersonation | google.protobuf.BoolValue Enables authentication on behalf of the Greenplum® user when connecting to the remote file storage or DBMS. The authentication is disabled by default. |
username | string Login username for the remote file storage or DBMS if authentication on behalf of the Greenplum® user is enabled. The maximum string length in characters is 128. |
sasl_connection_retries | google.protobuf.Int64Value Maximum number of times that PXF retries a SASL connection request after a refused connection returns a GSS initiate failed error. The default value is 5 . Acceptable values are 1 to 50, inclusive. |
zk_hosts[] | string ZooKeeper server hosts. Specify values in the <address>:<port> format. The maximum number of elements is 200. The string length in characters for each value must be 1-200. |
ppd | google.protobuf.BoolValue Specifies if predicate pushdown is enabled for queries on external tables. The predicate pushdown is enabled by default. |
metastore_uris[] | string List of URIs separated by commas. To request metadata, the remote DBMS connects to Metastore by one of these URIs. The maximum number of elements is 200. The string length in characters for each value must be 1-200. |
metastore_kerberos_principal | string Service principal for the Metastore Thrift server. The maximum string length in characters is 1000. |
auth_kerberos_principal | string Kerberos server principal. The maximum string length in characters is 1000. |
Operation
Field | Description |
---|---|
id | string ID of the operation. |
description | string Description of the operation. 0-256 characters long. |
created_at | google.protobuf.Timestamp Creation timestamp. |
created_by | string ID of the user or service account who initiated the operation. |
modified_at | google.protobuf.Timestamp The time when the Operation resource was last modified. |
done | bool If the value is false , it means the operation is still in progress. If true , the operation is completed, and either error or response is available. |
metadata | google.protobuf.Any Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any. |
result | oneof: error or response The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true , exactly one of error or response is set. |
error | google.rpc.Status The error result of the operation in case of failure or cancellation. |
response | google.protobuf.Any if operation finished successfully. |
CreatePXFDatasourceMetadata
Field | Description |
---|---|
cluster_id | string Required. The maximum string length in characters is 50. |
datasource_name | string Required. The string length in characters must be 3-200. Value must match the regular expression ^[^\\|/*?.,;\"'<>]+$ . |
PXFDatasource
Field | Description |
---|---|
name | string Required. Data source name. The string length in characters must be 3-200. Value must match the regular expression ^[^\\|/*?.,;\"'<>]+$ . |
settings | oneof: s3 , jdbc , hdfs or hive |
s3 | PXFDatasourceS3 Settings of an external S3 data source. |
jdbc | PXFDatasourceJDBC Settings of an external JDBC data source. |
hdfs | PXFDatasourceHDFS Settings of an external HDFS data source. |
hive | PXFDatasourceHive Settings of an external Hive data source. |
Update
Update PXF datasource
rpc Update (UpdatePXFDatasourceRequest) returns (operation.Operation)
Metadata and response of Operation:
Operation.metadata:UpdatePXFDatasourceMetadata
Operation.response:PXFDatasource
UpdatePXFDatasourceRequest
Field | Description |
---|---|
cluster_id | string Required. The maximum string length in characters is 50. |
update_mask | google.protobuf.FieldMask |
datasource | PXFDatasource |
PXFDatasource
Field | Description |
---|---|
name | string Required. Data source name. The string length in characters must be 3-200. Value must match the regular expression ^[^\\|/*?.,;\"'<>]+$ . |
settings | oneof: s3 , jdbc , hdfs or hive |
s3 | PXFDatasourceS3 Settings of an external S3 data source. |
jdbc | PXFDatasourceJDBC Settings of an external JDBC data source. |
hdfs | PXFDatasourceHDFS Settings of an external HDFS data source. |
hive | PXFDatasourceHive Settings of an external Hive data source. |
PXFDatasourceS3
Field | Description |
---|---|
access_key | string Public key to access S3 storage. The maximum string length in characters is 200. |
secret_key | string Secret key to access S3 storage. The maximum string length in characters is 200. |
fast_upload | google.protobuf.BoolValue Manages a fast upload of big files to S3 storage. In case of the false value, the PXF generates files on disk before sending them to the S3 storage. In case of the true value, the PXF generates files in RAM (the PXF writes to disc only if there is not enough RAM). The fast upload is enabled by default. |
endpoint | string S3 storage address. The default value is storage.yandexcloud.net used for Yandex Object Storage. The maximum string length in characters is 200. Value must match the regular expression (((s3|https?)://)?([a-z0-9]([a-z0-9-]*[a-z0-9])?[.])+[a-z]{2,})? . |
PXFDatasourceJDBC
Field | Description |
---|---|
driver | string JDBC driver class in Java. The possible values are the following:
|
url | string URL that the JDBC driver uses to connect to the database. Examples:
|
user | string Username of the DB owner. The maximum string length in characters is 200. |
password | string Password of the DB owner. The maximum string length in characters is 200. |
statement_batch_size | google.protobuf.Int64Value Number of rows to read in an external table, in a batch. The default value is 100 . Acceptable values are 50 to 1000, inclusive. |
statement_fetch_size | google.protobuf.Int64Value Number of rows to fetch (buffer) when reading from an external table. The default value is 1000 . Acceptable values are 50 to 10000, inclusive. |
statement_query_timeout | google.protobuf.Int64Value Amount of time (in seconds) the JDBC driver waits for a statement to run. This timeout applies to statements created for both read and write operations. The default value is 60 . Acceptable values are 5 to 43200, inclusive. |
pool_enabled | google.protobuf.BoolValue Determines whether JDBC connection pooling is used in a server configuration. By default, it is used. |
pool_maximum_size | google.protobuf.Int64Value Maximum number of connections to the DB backend. The default value is 5 . Acceptable values are 1 to 200, inclusive. |
pool_connection_timeout | google.protobuf.Int64Value Maximum amount of time, in milliseconds, after which an inactive connection is considered idle. The default value is 30000 . Acceptable values are 5000 to 600000, inclusive. |
pool_idle_timeout | google.protobuf.Int64Value Maximum amount of time, in milliseconds, after which an inactive connection is considered idle. The default value is 30000 . Acceptable values are 5000 to 600000, inclusive. |
pool_minimum_idle | google.protobuf.Int64Value Minimum number of idle connections maintained in the connection pool. The default value is 0 . Acceptable values are 0 to 200, inclusive. |
PXFDatasourceHDFS
Field | Description |
---|---|
core | PXFDatasourceCore Settings of the file system and security rules. |
kerberos | PXFDatasourceKerberos Settings of the Kerberos network authentication protocol. |
user_impersonation | google.protobuf.BoolValue Enables authentication on behalf of the Greenplum® user when connecting to the remote file storage or DBMS. The authentication is disabled by default. |
username | string Login username for the remote file storage or DBMS if authentication on behalf of the Greenplum® user is enabled. The maximum string length in characters is 128. |
sasl_connection_retries | google.protobuf.Int64Value Maximum number of times that PXF retries a SASL connection request after a refused connection returns a GSS initiate failed error. The default value is 5 . Acceptable values are 1 to 50, inclusive. |
zk_hosts[] | string ZooKeeper server hosts. Specify values in the <address>:<port> format. The maximum number of elements is 200. The string length in characters for each value must be 1-200. |
dfs | PXFDatasourceHDFSDfs Settings of the distributed file system. |
yarn | PXFDatasourceHDFSYarn Settings of the ResourceManager service that is responsible for tracking resources in a cluster and scheduling applications (e.g., MapReduce jobs). |
PXFDatasourceCore
Field | Description |
---|---|
default_fs | string URI whose scheme and authority determine the file system implementation. The maximum string length in characters is 200. |
security_auth_to_local | string Rules for mapping Kerberos principals to operating system user accounts. The maximum string length in characters is 1000. |
PXFDatasourceKerberos
Field | Description |
---|---|
enable | google.protobuf.BoolValue Determines whether the Kerberos authentication server is used. By default, it is not used. |
primary | string Host of the primary KDC server (Key Distribution Center). The maximum string length in characters is 128. |
realm | string Kerberos realm for a Greenplum® DB. The maximum string length in characters is 1000. |
kdc_servers[] | string KDC server hosts. The maximum number of elements is 200. The string length in characters for each value must be 1-200. |
admin_server | string Administration server host. Usually, this is the primary Kerberos server. The maximum string length in characters is 128. |
default_domain | string Domain that is used for the host name extension. Applicable when Kerberos 4 service members become Kerberos 5 service members (for example, when rcmd.hostname is replaced with host/hostname.domain). The maximum string length in characters is 128. |
keytab_base64 | string Base64 encoded contents of the keytab file. Value must match the regular expression ^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$ . The maximum string length in characters is 65536. |
PXFDatasourceHDFSDfs
Field | Description |
---|---|
ha_automatic_failover_enabled | google.protobuf.BoolValue Determines whether automatic failover is enabled for the high availability of the file system. The automatic failover is enabled by default. |
block_access_token_enabled | google.protobuf.BoolValue If true , access tokens are used as capabilities for accessing datanodes. If false , no access tokens are checked on accessing datanodes. The check of access tokens is enabled by default. |
use_datanode_hostname | google.protobuf.BoolValue Determines whether the datanode hostname is used when connecting to datanodes. |
namenodes | map<string,PXFDatasourceHDFSDfsNamenode> List of HDFS service logical names. Specify them separated by commas. The names can be arbitrary. No more than 10000 per resource. |
nameservices | string Corresponds well-known HDFS client setting "dfs.nameservices" for this datasource |
PXFDatasourceHDFSYarn
Field | Description |
---|---|
resourcemanager_ha_enabled | google.protobuf.BoolValue Determines whether high availability is enabled for YARN's ResourceManager services. The high availability is enabled by default. |
resourcemanager_ha_auto_failover_enabled | google.protobuf.BoolValue Determines whether another ResourceManager should automatically become active when the active ResourceManager has failed and does not respond. The switch of ResourceManagers is enabled by default if the high availability is enabled. |
resourcemanager_ha_auto_failover_embedded | google.protobuf.BoolValue Determines whether the embedded ActiveStandbyElector method should be used for the election of the active ResourceManager. If the current active ResourceManager has failed and does not respond, the ActiveStandbyElector method makes another ResourceManager active which then takes over. |
resourcemanager_cluster_id | string Cluster ID. Specify it, so the ResourceManager service does not become active for a different cluster. The maximum string length in characters is 1000. |
ha_rm | map<string,PXFDatasourceHDFSYarnHaRm> Highly available ResourceManager service. No more than 10000 per resource. |
PXFDatasourceHive
Field | Description |
---|---|
core | PXFDatasourceCore Settings of the file system and security rules. |
kerberos | PXFDatasourceKerberos Settings of the Kerberos network authentication protocol. |
user_impersonation | google.protobuf.BoolValue Enables authentication on behalf of the Greenplum® user when connecting to the remote file storage or DBMS. The authentication is disabled by default. |
username | string Login username for the remote file storage or DBMS if authentication on behalf of the Greenplum® user is enabled. The maximum string length in characters is 128. |
sasl_connection_retries | google.protobuf.Int64Value Maximum number of times that PXF retries a SASL connection request after a refused connection returns a GSS initiate failed error. The default value is 5 . Acceptable values are 1 to 50, inclusive. |
zk_hosts[] | string ZooKeeper server hosts. Specify values in the <address>:<port> format. The maximum number of elements is 200. The string length in characters for each value must be 1-200. |
ppd | google.protobuf.BoolValue Specifies if predicate pushdown is enabled for queries on external tables. The predicate pushdown is enabled by default. |
metastore_uris[] | string List of URIs separated by commas. To request metadata, the remote DBMS connects to Metastore by one of these URIs. The maximum number of elements is 200. The string length in characters for each value must be 1-200. |
metastore_kerberos_principal | string Service principal for the Metastore Thrift server. The maximum string length in characters is 1000. |
auth_kerberos_principal | string Kerberos server principal. The maximum string length in characters is 1000. |
Operation
Field | Description |
---|---|
id | string ID of the operation. |
description | string Description of the operation. 0-256 characters long. |
created_at | google.protobuf.Timestamp Creation timestamp. |
created_by | string ID of the user or service account who initiated the operation. |
modified_at | google.protobuf.Timestamp The time when the Operation resource was last modified. |
done | bool If the value is false , it means the operation is still in progress. If true , the operation is completed, and either error or response is available. |
metadata | google.protobuf.Any Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any. |
result | oneof: error or response The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true , exactly one of error or response is set. |
error | google.rpc.Status The error result of the operation in case of failure or cancellation. |
response | google.protobuf.Any if operation finished successfully. |
UpdatePXFDatasourceMetadata
Field | Description |
---|---|
cluster_id | string Required. The maximum string length in characters is 50. |
datasource_name | string Required. The string length in characters must be 3-200. Value must match the regular expression ^[^\\|/*?.,;\"'<>]+$ . |
PXFDatasource
Field | Description |
---|---|
name | string Required. Data source name. The string length in characters must be 3-200. Value must match the regular expression ^[^\\|/*?.,;\"'<>]+$ . |
settings | oneof: s3 , jdbc , hdfs or hive |
s3 | PXFDatasourceS3 Settings of an external S3 data source. |
jdbc | PXFDatasourceJDBC Settings of an external JDBC data source. |
hdfs | PXFDatasourceHDFS Settings of an external HDFS data source. |
hive | PXFDatasourceHive Settings of an external Hive data source. |
Delete
Delete PXF datasource
rpc Delete (DeletePXFDatasourceRequest) returns (operation.Operation)
Metadata and response of Operation:
Operation.metadata:DeletePXFDatasourceMetadata
Operation.response:google.protobuf.Empty
DeletePXFDatasourceRequest
Field | Description |
---|---|
cluster_id | string Required. The maximum string length in characters is 50. |
datasource_name | string Required. The string length in characters must be 3-200. Value must match the regular expression ^[^\\|/*?.,;\"'<>]+$ . |
Operation
Field | Description |
---|---|
id | string ID of the operation. |
description | string Description of the operation. 0-256 characters long. |
created_at | google.protobuf.Timestamp Creation timestamp. |
created_by | string ID of the user or service account who initiated the operation. |
modified_at | google.protobuf.Timestamp The time when the Operation resource was last modified. |
done | bool If the value is false , it means the operation is still in progress. If true , the operation is completed, and either error or response is available. |
metadata | google.protobuf.Any Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any. |
result | oneof: error or response The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true , exactly one of error or response is set. |
error | google.rpc.Status The error result of the operation in case of failure or cancellation. |
response | google.protobuf.Any if operation finished successfully. |
DeletePXFDatasourceMetadata
Field | Description |
---|---|
cluster_id | string Required. The maximum string length in characters is 50. |
datasource_name | string Required. The string length in characters must be 3-200. Value must match the regular expression ^[^\\|/*?.,;\"'<>]+$ . |