Managed Service for Greenplum® API, gRPC: PXFDatasourceService.Update
- gRPC request
- UpdatePXFDatasourceRequest
- PXFDatasource
- PXFDatasourceS3
- PXFDatasourceJDBC
- PXFDatasourceHDFS
- PXFDatasourceCore
- PXFDatasourceKerberos
- PXFDatasourceHDFSDfs
- PXFDatasourceHDFSDfsNamenode
- PXFDatasourceHDFSYarn
- PXFDatasourceHDFSYarnHaRm
- PXFDatasourceHive
- operation.Operation
- UpdatePXFDatasourceMetadata
- PXFDatasource
- PXFDatasourceS3
- PXFDatasourceJDBC
- PXFDatasourceHDFS
- PXFDatasourceCore
- PXFDatasourceKerberos
- PXFDatasourceHDFSDfs
- PXFDatasourceHDFSDfsNamenode
- PXFDatasourceHDFSYarn
- PXFDatasourceHDFSYarnHaRm
- PXFDatasourceHive
Update PXF datasource
gRPC request
rpc Update (UpdatePXFDatasourceRequest) returns (operation.Operation)
UpdatePXFDatasourceRequest
{
"clusterId": "string",
"updateMask": "google.protobuf.FieldMask",
"datasource": {
"name": "string",
// Includes only one of the fields `s3`, `jdbc`, `hdfs`, `hive`
"s3": {
"accessKey": "string",
"secretKey": "string",
"fastUpload": "google.protobuf.BoolValue",
"endpoint": "string"
},
"jdbc": {
"driver": "string",
"url": "string",
"user": "string",
"password": "string",
"statementBatchSize": "google.protobuf.Int64Value",
"statementFetchSize": "google.protobuf.Int64Value",
"statementQueryTimeout": "google.protobuf.Int64Value",
"poolEnabled": "google.protobuf.BoolValue",
"poolMaximumSize": "google.protobuf.Int64Value",
"poolConnectionTimeout": "google.protobuf.Int64Value",
"poolIdleTimeout": "google.protobuf.Int64Value",
"poolMinimumIdle": "google.protobuf.Int64Value"
},
"hdfs": {
"core": {
"defaultFs": "string",
"securityAuthToLocal": "string"
},
"kerberos": {
"enable": "google.protobuf.BoolValue",
"primary": "string",
"realm": "string",
"kdcServers": [
"string"
],
"adminServer": "string",
"defaultDomain": "string",
"keytabBase64": "string"
},
"userImpersonation": "google.protobuf.BoolValue",
"username": "string",
"saslConnectionRetries": "google.protobuf.Int64Value",
"zkHosts": [
"string"
],
"dfs": {
"haAutomaticFailoverEnabled": "google.protobuf.BoolValue",
"blockAccessTokenEnabled": "google.protobuf.BoolValue",
"useDatanodeHostname": "google.protobuf.BoolValue",
"namenodes": {
"rpcAddress": "string",
"serviceRpcAddress": "string",
"httpAddress": "string",
"httpsAddress": "string"
},
"nameservices": "string"
},
"yarn": {
"resourcemanagerHaEnabled": "google.protobuf.BoolValue",
"resourcemanagerHaAutoFailoverEnabled": "google.protobuf.BoolValue",
"resourcemanagerHaAutoFailoverEmbedded": "google.protobuf.BoolValue",
"resourcemanagerClusterId": "string",
"haRm": {
"resourcemanagerAddress": "string",
"resourcemanagerSchedulerAddress": "string",
"resourcemanagerResourceTrackerAddress": "string",
"resourcemanagerAdminAddress": "string",
"resourcemanagerWebappAddress": "string",
"resourcemanagerWebappHttpsAddress": "string"
}
}
},
"hive": {
"core": {
"defaultFs": "string",
"securityAuthToLocal": "string"
},
"kerberos": {
"enable": "google.protobuf.BoolValue",
"primary": "string",
"realm": "string",
"kdcServers": [
"string"
],
"adminServer": "string",
"defaultDomain": "string",
"keytabBase64": "string"
},
"userImpersonation": "google.protobuf.BoolValue",
"username": "string",
"saslConnectionRetries": "google.protobuf.Int64Value",
"zkHosts": [
"string"
],
"ppd": "google.protobuf.BoolValue",
"metastoreUris": [
"string"
],
"metastoreKerberosPrincipal": "string",
"authKerberosPrincipal": "string"
}
// end of the list of possible fields
}
}
Field |
Description |
clusterId |
string Required field. |
updateMask |
|
datasource |
PXFDatasource
Field |
Description |
name |
string Required field. Data source name. |
s3 |
Settings of an external S3 data source. Includes only one of the fields |
jdbc |
Settings of an external JDBC data source. Includes only one of the fields |
hdfs |
Settings of an external HDFS data source. Includes only one of the fields |
hive |
Settings of an external Hive data source. Includes only one of the fields |
PXFDatasourceS3
Field |
Description |
accessKey |
string Public key to access S3 storage. |
secretKey |
string Secret key to access S3 storage. |
fastUpload |
Manages a fast upload of big files to S3 storage. In case of the The fast upload is enabled by default. |
endpoint |
string S3 storage address. The default value is |
PXFDatasourceJDBC
Field |
Description |
driver |
string JDBC driver class in Java. The possible values are the following:
|
url |
string URL that the JDBC driver uses to connect to the database. Examples:
|
user |
string Username of the DB owner. |
password |
string Password of the DB owner. |
statementBatchSize |
Number of rows to read in an external table, in a batch. The default value is |
statementFetchSize |
Number of rows to fetch (buffer) when reading from an external table. The default value is |
statementQueryTimeout |
Amount of time (in seconds) the JDBC driver waits for a statement to run. This timeout applies to statements created for both read and write operations. The default value is |
poolEnabled |
Determines whether JDBC connection pooling is used in a server configuration. By default, it is used. |
poolMaximumSize |
Maximum number of connections to the DB backend. The default value is |
poolConnectionTimeout |
Maximum time, in milliseconds, to wait for a connection from the pool. The default value is |
poolIdleTimeout |
Maximum amount of time, in milliseconds, after which an inactive connection is considered idle. The default value is |
poolMinimumIdle |
Minimum number of idle connections maintained in the connection pool. The default value is |
PXFDatasourceHDFS
Field |
Description |
core |
Settings of the file system and security rules. |
kerberos |
Settings of the Kerberos network authentication protocol. |
userImpersonation |
Enables authentication on behalf of the Greenplum® user when connecting to the remote file storage or DBMS. The authentication is disabled by default. |
username |
string Login username for the remote file storage or DBMS if authentication on behalf of the Greenplum® user is enabled. |
saslConnectionRetries |
Maximum number of times that PXF retries a SASL connection request after a refused connection returns a The default value is |
zkHosts[] |
string ZooKeeper server hosts. Specify values in the |
dfs |
Settings of the distributed file system. |
yarn |
Settings of the ResourceManager service that is responsible for tracking resources in a cluster and scheduling applications (e.g., MapReduce jobs). |
PXFDatasourceCore
Field |
Description |
defaultFs |
string URI whose scheme and authority determine the file system implementation. |
securityAuthToLocal |
string Rules for mapping Kerberos principals to operating system user accounts. |
PXFDatasourceKerberos
Field |
Description |
enable |
Determines whether the Kerberos authentication server is used. By default, it is not used. |
primary |
string Host of the primary KDC server (Key Distribution Center). |
realm |
string Kerberos realm for a Greenplum® DB. |
kdcServers[] |
string KDC server hosts. |
adminServer |
string Administration server host. Usually, this is the primary Kerberos server. |
defaultDomain |
string Domain that is used for the host name extension. Applicable when Kerberos 4 service members become Kerberos 5 service members (for example, when rcmd.hostname is replaced with host/hostname.domain). |
keytabBase64 |
string Base64 encoded contents of the keytab file. |
PXFDatasourceHDFSDfs
Field |
Description |
haAutomaticFailoverEnabled |
Determines whether automatic failover is enabled for the high availability of the file system. The automatic failover is enabled by default. |
blockAccessTokenEnabled |
If The check of access tokens is enabled by default. |
useDatanodeHostname |
Determines whether the datanode hostname is used when connecting to datanodes. |
namenodes |
List of HDFS service logical names. Specify them separated by commas. The names can be arbitrary. |
nameservices |
string Corresponds well-known HDFS client setting "dfs.nameservices" for this datasource |
PXFDatasourceHDFSDfsNamenode
Field |
Description |
rpcAddress |
string |
serviceRpcAddress |
string |
httpAddress |
string |
httpsAddress |
string |
PXFDatasourceHDFSYarn
Field |
Description |
resourcemanagerHaEnabled |
Determines whether high availability is enabled for YARN's ResourceManager services. The high availability is enabled by default. |
resourcemanagerHaAutoFailoverEnabled |
Determines whether another ResourceManager should automatically become active when the active ResourceManager has failed and does not respond. The switch of ResourceManagers is enabled by default if the high availability is enabled. |
resourcemanagerHaAutoFailoverEmbedded |
Determines whether the embedded ActiveStandbyElector method should be used for the election of the active ResourceManager. If the current active ResourceManager has failed and does not respond, the ActiveStandbyElector method makes another ResourceManager active which then takes over. |
resourcemanagerClusterId |
string Cluster ID. Specify it, so the ResourceManager service does not become active for a different cluster. |
haRm |
Highly available ResourceManager service. |
PXFDatasourceHDFSYarnHaRm
Field |
Description |
resourcemanagerAddress |
string |
resourcemanagerSchedulerAddress |
string |
resourcemanagerResourceTrackerAddress |
string |
resourcemanagerAdminAddress |
string |
resourcemanagerWebappAddress |
string |
resourcemanagerWebappHttpsAddress |
string |
PXFDatasourceHive
Field |
Description |
core |
Settings of the file system and security rules. |
kerberos |
Settings of the Kerberos network authentication protocol. |
userImpersonation |
Enables authentication on behalf of the Greenplum® user when connecting to the remote file storage or DBMS. The authentication is disabled by default. |
username |
string Login username for the remote file storage or DBMS if authentication on behalf of the Greenplum® user is enabled. |
saslConnectionRetries |
Maximum number of times that PXF retries a SASL connection request after a refused connection returns a The default value is |
zkHosts[] |
string ZooKeeper server hosts. Specify values in the |
ppd |
Specifies if predicate pushdown is enabled for queries on external tables. The predicate pushdown is enabled by default. |
metastoreUris[] |
string List of URIs separated by commas. To request metadata, the remote DBMS connects to Metastore by one of these URIs. |
metastoreKerberosPrincipal |
string Service principal for the Metastore Thrift server. |
authKerberosPrincipal |
string Kerberos server principal. |
operation.Operation
{
"id": "string",
"description": "string",
"createdAt": "google.protobuf.Timestamp",
"createdBy": "string",
"modifiedAt": "google.protobuf.Timestamp",
"done": "bool",
"metadata": {
"clusterId": "string",
"datasourceName": "string"
},
// Includes only one of the fields `error`, `response`
"error": "google.rpc.Status",
"response": {
"name": "string",
// Includes only one of the fields `s3`, `jdbc`, `hdfs`, `hive`
"s3": {
"accessKey": "string",
"secretKey": "string",
"fastUpload": "google.protobuf.BoolValue",
"endpoint": "string"
},
"jdbc": {
"driver": "string",
"url": "string",
"user": "string",
"password": "string",
"statementBatchSize": "google.protobuf.Int64Value",
"statementFetchSize": "google.protobuf.Int64Value",
"statementQueryTimeout": "google.protobuf.Int64Value",
"poolEnabled": "google.protobuf.BoolValue",
"poolMaximumSize": "google.protobuf.Int64Value",
"poolConnectionTimeout": "google.protobuf.Int64Value",
"poolIdleTimeout": "google.protobuf.Int64Value",
"poolMinimumIdle": "google.protobuf.Int64Value"
},
"hdfs": {
"core": {
"defaultFs": "string",
"securityAuthToLocal": "string"
},
"kerberos": {
"enable": "google.protobuf.BoolValue",
"primary": "string",
"realm": "string",
"kdcServers": [
"string"
],
"adminServer": "string",
"defaultDomain": "string",
"keytabBase64": "string"
},
"userImpersonation": "google.protobuf.BoolValue",
"username": "string",
"saslConnectionRetries": "google.protobuf.Int64Value",
"zkHosts": [
"string"
],
"dfs": {
"haAutomaticFailoverEnabled": "google.protobuf.BoolValue",
"blockAccessTokenEnabled": "google.protobuf.BoolValue",
"useDatanodeHostname": "google.protobuf.BoolValue",
"namenodes": {
"rpcAddress": "string",
"serviceRpcAddress": "string",
"httpAddress": "string",
"httpsAddress": "string"
},
"nameservices": "string"
},
"yarn": {
"resourcemanagerHaEnabled": "google.protobuf.BoolValue",
"resourcemanagerHaAutoFailoverEnabled": "google.protobuf.BoolValue",
"resourcemanagerHaAutoFailoverEmbedded": "google.protobuf.BoolValue",
"resourcemanagerClusterId": "string",
"haRm": {
"resourcemanagerAddress": "string",
"resourcemanagerSchedulerAddress": "string",
"resourcemanagerResourceTrackerAddress": "string",
"resourcemanagerAdminAddress": "string",
"resourcemanagerWebappAddress": "string",
"resourcemanagerWebappHttpsAddress": "string"
}
}
},
"hive": {
"core": {
"defaultFs": "string",
"securityAuthToLocal": "string"
},
"kerberos": {
"enable": "google.protobuf.BoolValue",
"primary": "string",
"realm": "string",
"kdcServers": [
"string"
],
"adminServer": "string",
"defaultDomain": "string",
"keytabBase64": "string"
},
"userImpersonation": "google.protobuf.BoolValue",
"username": "string",
"saslConnectionRetries": "google.protobuf.Int64Value",
"zkHosts": [
"string"
],
"ppd": "google.protobuf.BoolValue",
"metastoreUris": [
"string"
],
"metastoreKerberosPrincipal": "string",
"authKerberosPrincipal": "string"
}
// end of the list of possible fields
}
// end of the list of possible fields
}
An Operation resource. For more information, see Operation.
Field |
Description |
id |
string ID of the operation. |
description |
string Description of the operation. 0-256 characters long. |
createdAt |
Creation timestamp. |
createdBy |
string ID of the user or service account who initiated the operation. |
modifiedAt |
The time when the Operation resource was last modified. |
done |
bool If the value is |
metadata |
Service-specific metadata associated with the operation. |
error |
The error result of the operation in case of failure or cancellation. Includes only one of the fields The operation result. |
response |
The normal response of the operation in case of success. Includes only one of the fields The operation result. |
UpdatePXFDatasourceMetadata
Field |
Description |
clusterId |
string Required field. |
datasourceName |
string Required field. |
PXFDatasource
Field |
Description |
name |
string Required field. Data source name. |
s3 |
Settings of an external S3 data source. Includes only one of the fields |
jdbc |
Settings of an external JDBC data source. Includes only one of the fields |
hdfs |
Settings of an external HDFS data source. Includes only one of the fields |
hive |
Settings of an external Hive data source. Includes only one of the fields |
PXFDatasourceS3
Field |
Description |
accessKey |
string Public key to access S3 storage. |
secretKey |
string Secret key to access S3 storage. |
fastUpload |
Manages a fast upload of big files to S3 storage. In case of the The fast upload is enabled by default. |
endpoint |
string S3 storage address. The default value is |
PXFDatasourceJDBC
Field |
Description |
driver |
string JDBC driver class in Java. The possible values are the following:
|
url |
string URL that the JDBC driver uses to connect to the database. Examples:
|
user |
string Username of the DB owner. |
password |
string Password of the DB owner. |
statementBatchSize |
Number of rows to read in an external table, in a batch. The default value is |
statementFetchSize |
Number of rows to fetch (buffer) when reading from an external table. The default value is |
statementQueryTimeout |
Amount of time (in seconds) the JDBC driver waits for a statement to run. This timeout applies to statements created for both read and write operations. The default value is |
poolEnabled |
Determines whether JDBC connection pooling is used in a server configuration. By default, it is used. |
poolMaximumSize |
Maximum number of connections to the DB backend. The default value is |
poolConnectionTimeout |
Maximum time, in milliseconds, to wait for a connection from the pool. The default value is |
poolIdleTimeout |
Maximum amount of time, in milliseconds, after which an inactive connection is considered idle. The default value is |
poolMinimumIdle |
Minimum number of idle connections maintained in the connection pool. The default value is |
PXFDatasourceHDFS
Field |
Description |
core |
Settings of the file system and security rules. |
kerberos |
Settings of the Kerberos network authentication protocol. |
userImpersonation |
Enables authentication on behalf of the Greenplum® user when connecting to the remote file storage or DBMS. The authentication is disabled by default. |
username |
string Login username for the remote file storage or DBMS if authentication on behalf of the Greenplum® user is enabled. |
saslConnectionRetries |
Maximum number of times that PXF retries a SASL connection request after a refused connection returns a The default value is |
zkHosts[] |
string ZooKeeper server hosts. Specify values in the |
dfs |
Settings of the distributed file system. |
yarn |
Settings of the ResourceManager service that is responsible for tracking resources in a cluster and scheduling applications (e.g., MapReduce jobs). |
PXFDatasourceCore
Field |
Description |
defaultFs |
string URI whose scheme and authority determine the file system implementation. |
securityAuthToLocal |
string Rules for mapping Kerberos principals to operating system user accounts. |
PXFDatasourceKerberos
Field |
Description |
enable |
Determines whether the Kerberos authentication server is used. By default, it is not used. |
primary |
string Host of the primary KDC server (Key Distribution Center). |
realm |
string Kerberos realm for a Greenplum® DB. |
kdcServers[] |
string KDC server hosts. |
adminServer |
string Administration server host. Usually, this is the primary Kerberos server. |
defaultDomain |
string Domain that is used for the host name extension. Applicable when Kerberos 4 service members become Kerberos 5 service members (for example, when rcmd.hostname is replaced with host/hostname.domain). |
keytabBase64 |
string Base64 encoded contents of the keytab file. |
PXFDatasourceHDFSDfs
Field |
Description |
haAutomaticFailoverEnabled |
Determines whether automatic failover is enabled for the high availability of the file system. The automatic failover is enabled by default. |
blockAccessTokenEnabled |
If The check of access tokens is enabled by default. |
useDatanodeHostname |
Determines whether the datanode hostname is used when connecting to datanodes. |
namenodes |
List of HDFS service logical names. Specify them separated by commas. The names can be arbitrary. |
nameservices |
string Corresponds well-known HDFS client setting "dfs.nameservices" for this datasource |
PXFDatasourceHDFSDfsNamenode
Field |
Description |
rpcAddress |
string |
serviceRpcAddress |
string |
httpAddress |
string |
httpsAddress |
string |
PXFDatasourceHDFSYarn
Field |
Description |
resourcemanagerHaEnabled |
Determines whether high availability is enabled for YARN's ResourceManager services. The high availability is enabled by default. |
resourcemanagerHaAutoFailoverEnabled |
Determines whether another ResourceManager should automatically become active when the active ResourceManager has failed and does not respond. The switch of ResourceManagers is enabled by default if the high availability is enabled. |
resourcemanagerHaAutoFailoverEmbedded |
Determines whether the embedded ActiveStandbyElector method should be used for the election of the active ResourceManager. If the current active ResourceManager has failed and does not respond, the ActiveStandbyElector method makes another ResourceManager active which then takes over. |
resourcemanagerClusterId |
string Cluster ID. Specify it, so the ResourceManager service does not become active for a different cluster. |
haRm |
Highly available ResourceManager service. |
PXFDatasourceHDFSYarnHaRm
Field |
Description |
resourcemanagerAddress |
string |
resourcemanagerSchedulerAddress |
string |
resourcemanagerResourceTrackerAddress |
string |
resourcemanagerAdminAddress |
string |
resourcemanagerWebappAddress |
string |
resourcemanagerWebappHttpsAddress |
string |
PXFDatasourceHive
Field |
Description |
core |
Settings of the file system and security rules. |
kerberos |
Settings of the Kerberos network authentication protocol. |
userImpersonation |
Enables authentication on behalf of the Greenplum® user when connecting to the remote file storage or DBMS. The authentication is disabled by default. |
username |
string Login username for the remote file storage or DBMS if authentication on behalf of the Greenplum® user is enabled. |
saslConnectionRetries |
Maximum number of times that PXF retries a SASL connection request after a refused connection returns a The default value is |
zkHosts[] |
string ZooKeeper server hosts. Specify values in the |
ppd |
Specifies if predicate pushdown is enabled for queries on external tables. The predicate pushdown is enabled by default. |
metastoreUris[] |
string List of URIs separated by commas. To request metadata, the remote DBMS connects to Metastore by one of these URIs. |
metastoreKerberosPrincipal |
string Service principal for the Metastore Thrift server. |
authKerberosPrincipal |
string Kerberos server principal. |