Google Cloud Storage
gs://BUCKET[/PREFIX] lists a bucket's current objects through the JSON API, or those whose names start with the prefix; --all-versions lists every generation, live and noncurrent.
The smallest command
A public Landsat prefix, with no Google account:
swath du gs://gcp-public-data-landsat/LC08/01/001/ -d 1 --anonymousWithout --anonymous swath uses your Google credentials.
Authentication and permissions
Permissions. storage.objects.list on the bucket (the same with --all-versions). The token's scope is read-only.
Credentials. --gcs-credentials-file FILE (a service-account or authorized-user JSON file), else Application Default Credentials: GOOGLE_APPLICATION_CREDENTIALS, the gcloud credentials file, the metadata server on Google Cloud. Credentials that cannot be found or loaded (no ADC anywhere, a GOOGLE_APPLICATION_CREDENTIALS that names a missing file) stop the run before listing (auth). Rules shared by every object store are in Choose a source.
Options for this source
--gcs-billing-project PROJECTbills the requests to that project, for a requester-pays bucket. It is part of the run's identity.--endpoint-urlreplaces the JSON API's base URL,https://storage.googleapis.com/storage/v1/(for fake-gcs-server).--all-versionslists live and noncurrent generations; their metadata differs from S3's versions (List every version).
GCS's XML API also answers S3 requests: s3:// with --endpoint-url https://storage.googleapis.com and --s3-bearer-token-command 'gcloud auth print-access-token' lists through it (Amazon S3). The native gs:// source is the one to prefer.
Writing a dataset to GCS
-o gs://BUCKET/PREFIX --work-dir DIR writes the dataset to GCS; the command, what appears in the bucket and how to resume and clean up are in Writing to a bucket. What is GCS's own:
- Credentials: the destination's are its own:
--output-gcs-credentials-file FILE(a service account key or authorized-user JSON) or Application Default Credentials, with the read-write scope. The source's--anonymousor credentials never apply to it. A rotated key file is given toswath resume DIRagain. - Permissions:
storage.objects.create,getanddeleteon the prefix (delete removes this run's own orphans, by generation). - Endpoint:
--output-endpoint-url, as the SDK takes it, is fixed for the run. - Unfinished uploads are resumable upload sessions, which GCS keeps a week: no lifecycle rule is needed for them.
Limits and known differences
- Soft-deleted objects are not listed, with or without
--all-versions. - A prefix is a string prefix:
gs://bucket/logscoverslogs/andlogs-old/. - The run report estimates request cost at GCS's reference rate only with the default service endpoint.