Google Cloud Storage file system support
Trino includes a native implementation to access Google Cloud Storage (GCS) with a catalog using the Delta Lake, Hive, Hudi, or Iceberg connectors.
Enable the native implementation with fs.gcs.enabled=true in your
catalog properties file.
General configuration
Use the following properties to configure general aspects of Google Cloud Storage file system support:
| Property | Description |
|---|---|
fs.gcs.enabled | Activate the native implementation for Google Cloud Storage support. Defaults to false. Set to true to use Google Cloud Storage and enable all other properties. |
gcs.project-id | Identifier for the project on Google Cloud Storage. |
gcs.endpoint | Optional URL for the Google Cloud Storage endpoint. Configure this property if your storage is accessed using a custom URL, for example http://storage.example.com:8000. |
gcs.client.max-retries | Maximum number of RPC attempts. Defaults to 20. |
gcs.client.backoff-scale-factor | Scale factor for RPC retry delays. Defaults to 3. |
gcs.client.max-retry-time | Total time duration limit for RPC call retries. Defaults to 25s. |
gcs.client.min-backoff-delay | Minimum delay duration between RPC retries. Defaults to 10ms. |
gcs.client.max-backoff-delay | Maximum delay duration between RPC retries. Defaults to 2s. |
gcs.read-block-size | Minimum data size for blocks read per RPC. Defaults to 2MiB. See com.google.cloud.BaseStorageReadChannel. |
gcs.write-block-size | Minimum data size for blocks written per RPC. The Defaults to 16MiB. See com.google.cloud.BaseStorageWriteChannel. |
gcs.page-size | Maximum number of blobs to return per page. Defaults to 100. |
gcs.batch-size | Number of blobs to delete per batch. Defaults to 100. Recommended batch size is 100. |
gcs.application-id | Specify the application identifier appended to the User-Agent header for all requests sent to Google Cloud Storage. Defaults to Trino. |
Authentication
Use one of the following properties to configure the authentication to Google Cloud Storage:
| Property | Description |
|---|---|
gcs.auth-type | Authentication type to use for Google Cloud Storage access. Default to SERVICE_ACCOUNT. Supported values are: * SERVICE_ACCOUNT: loads credentials from the environment. Either gcs.json-key or gcs.json-key-file-path can be set in addition to override the default credentials provider. * ACCESS_TOKEN: usage of client-provided OAuth 2.0 token to access Google Cloud Storage. * APPLICATION_DEFAULT: Attempts to obtain Google Application Default Credentials (ADC) from the environment. If no ADC is available, the filesystem falls back to NoCredentials.getInstance() to explicitly indicate unauthenticated GCS access. |
gcs.json-key | Your Google Cloud service account key in JSON format. Not to be set together with gcs.json-key-file-path. |
gcs.json-key-file-path | Path to the JSON file on each node that contains your Google Cloud Platform service account key. Not to be set together with gcs.json-key. |
Migration from legacy Google Cloud Storage file system
Previous Trino releases included a legacy Google Cloud Storage file system
implementation used by catalogs configured with fs.hadoop.enabled and
hive.gcs.* properties. That legacy support has been removed. Use the native
Google Cloud Storage file system implementation.
To migrate a catalog to use the native file system implementation for Google Cloud Storage, make the following edits to your catalog configuration:
- Add the
fs.gcs.enabled=truecatalog configuration property. - If your catalog enabled
fs.hadoop.enabledonly for legacy Google Cloud Storage access, remove that property. - Refer to the following table to rename your existing legacy catalog configuration properties to the corresponding native configuration properties. Supported configuration values are identical unless otherwise noted.
| Legacy property | Native property | Notes |
|---|---|---|
hive.gcs.use-access-token | gcs.auth-type | |
hive.gcs.json-key-file-path | gcs.json-key-file-path | Also see gcs.json-key in preceding sections |