Questions tagged [gsutil]
The gsutil tag has no usage guidance.
34
questions
29
votes
2
answers
39k
views
Using JSON keys with google cloud gsutil
I have a ssh private key in a key.json file, and I want to use this credential to access a storage bucket using gsutil.
I can't seem to find anything about how to include json keys as a ...
6
votes
1
answer
4k
views
Slow synchronisation stage on gsutil rsync?
I've just started to use GCS as backup for my web servers. One server has 1.2 million JPEGS (3.5TB) and this all rsynced over flawlessly over 10 hours or so.
The other has 2.5 million JPEGS (just ...
3
votes
1
answer
630
views
gsutil rsync "too many values to unpack" error
I'm trying to sync a folder with over 3k files to a bucket on Google Cloud with gsutil. The problem is that less than half of the files are uploaded and then I get this error:
too many values to ...
3
votes
1
answer
3k
views
How to use JSON keys with google cloud gsutil to manage multiple Keys
We have multiple GCP Service account keys from different environment such as DEV,STAGING,..PROD.
I would like to run a command in my jenkins box which is there in Dev environment to create access DEV ...
3
votes
1
answer
6k
views
Copy list of files (-I flag) with gsutil preserving path
I am trying to copy all pictures and static files to a bucket of mine in Google Cloud Platform.
I am attempting this command from the root dir of my app:
find -regextype posix-extended -iregex ".*\.(...
3
votes
1
answer
2k
views
Setting up logging for buckets in Google Cloud storage
Trying to set up logging on a GCS bucket. every time i enter the command to set logging i get the error below. "Command exception: "logging" command does not support "file://" URLs. did you mean to ...
3
votes
1
answer
922
views
ServiceException: 401 Unauthorized WebHook callback channel
Can anyone help me with issue on setting up "Object Change Notification". I am trying to execute "
gsutil notification watchbucket https://<my_project_id>.appspot.com/ gs://glrs_test_deepshikha
...
2
votes
1
answer
1k
views
Google Storage ACL - write only for public, but don't allow read, list, delete or update operations
I need to setup Google Storage access for 3rd parties (public) to allow them to upload one or more files, but they shouldn't be allowed to read, list, delete or update an existing file. This structure ...
2
votes
1
answer
954
views
gsutil ls -l fails when gsutil mb succeeded
I get the following when I attempt to list the buckets for the account I'm logged in with:
$ gsutil ls -l gs://bucket
AccessDeniedException: 403 Forbidden
$ gcloud config list
Your active ...
2
votes
1
answer
380
views
gsutil cp error: cannot have start index greater than total size
I'm trying to download a lot of images from a Google Bucket that my employer's web host (not so) helpfully set up. I run the command gsutil cp -r gs://bucket-link-info/images/ . and the download ...
2
votes
0
answers
385
views
gsutil rsync fails on Ubuntu with UnicodeDecodeError
I'm trying to rsync between google cloud storage and amazon s3. On my local machine (MacOS) this command works
gsutil rsync -d -r gs://mira-internal/data/exports/20191108 s3://mira-temp/raw/20191108
...
1
vote
1
answer
3k
views
gsutil from gcloud/Google Cloud SDK failing with "SyntaxError: invalid syntax" error in bcrypt
Trying to use gsutil today I got the following error:
+ gsutil cp -nra public-read a gs://b
/usr/local/lib/python2.7/dist-packages/cryptography/__init__.py:39: CryptographyDeprecationWarning: Python ...
1
vote
2
answers
1k
views
Convert filenames to lowercase [Google Cloud Storage] [gsutil]
We recently moved all product images from our website to Google Cloud Storage. Due to bad naming policies in the past the casing (lowercase, uppercase) in the file names does not always match the way ...
1
vote
1
answer
692
views
start/stop a Google Cloud VM from desktop
I have a VM running on Google cloud that I would like to start and stop.
I am able to start/stop it from another VM in the same project using the command
sudo gcloud compute instances stop my-daily-...
1
vote
1
answer
332
views
How/why would ALL our Google Cloud Platform Service account keys suddenly disappear/be deleted?
We had a major issue with Google Cloud Platform some time between 2019-04-16T11:00:00Z and 2019-04-17T02:30:00Z, when all of our service accounts lost their access keys for an unknown reason, so our ...
1
vote
2
answers
1k
views
Specify account for single gsutil call
I want to be able to perform simultaneous gsutil calls against Google Cloud Storage, using different accounts.
For that purpose, I'd like to know if there is a way to specify a Google Cloud account ...
1
vote
1
answer
4k
views
CommandException: arg [...] does not name a directory, bucket, or bucket subdir. for a directory on Compute Engine
I am trying to use gsutils command to deploy changes from CircleCI to a directory on my Compute Engine instance. The command I am using is:
gsutil -m rsync -d -r dev/ [VM_INSTANCE_NAME]:/var/www/dev/
...
1
vote
1
answer
864
views
Gcloud cannot detect network connection
I have internet connection on the ESXi server and I want to copy everything to GCP storage. But Gcloud couldn't authenticate. It looks like it can't reach the gloud login servers.
I'm not if the esxi ...
1
vote
1
answer
643
views
Attempt to copy files from google cloud bucket to VM instance using gsutil is falling after reaching size limit
I have a bucket with a file whose size is larger than 11 GB. I note that I can access all bucket's data from the standard google cloud console. Unfortunately, this console has very limited ...
1
vote
1
answer
6k
views
Access denied 403 when trying to download from the Google Cloud Platform, but user has all access
We are just trying to use gsutil from the command line to download some DCM data from the Google Cloud Platform:
gsutil -m cp -R gs://dcdt_-dcm_account75701/dcm_account75701_activity_201803* C:\Users\...
1
vote
0
answers
56
views
gsutil fails with "OSError: Operation not permitted" when running from cron
I have a php script that does backups. The backups have been failing, because one of the first commands throws an error and stops the script. When I run the script in the terminal, it works just fine. ...
0
votes
1
answer
144
views
Where can I find the gsutil.tar.gz file for version 4.66 of gsutil?
The situation on my end is really very complicated1. The bottomline, however, is that I desperately need to download an version of the gsutil.tar.gz file that is older than the one currently ...
0
votes
2
answers
1k
views
Using gsutil on Windows to download a Google Workspace Data Export
What is the easiest approach to automatically download all export files onto a Windows system?
I need to download a full Google Workspace Data Export using Windows.
The Google Workspace Data Export is ...
0
votes
1
answer
335
views
change storage class a folder gcp
change storage class in a folder into a bucket in GCP, I try with
gsutil rewrite -s coldline gs://bucket-name/folder/
I see the
gsutil stat gs://bucket-name/folder/
what am I doing wrong?
0
votes
3
answers
281
views
gcloud auth login: Limit Oauth2 scope
How can I limit the Oauth2 scope when logging in with my user principal using (non service account):
gcloud auth login
Background: I created a Gsuite Takeout, got the Takeout Bucket. Now I want to ...
0
votes
1
answer
461
views
gsutil error /bootstrapping/gsutil.py
gsutil is giving me the following error. It is installed in its latest version on a linux suse 12. Can you help me?
gsutil
Traceback (most recent call last):
File "/root/google-cloud-sdk/bin/...
0
votes
2
answers
1k
views
How to run gsutil rsync without auth?
aws s3 sync provides a parameter named --no-sign-request, so user can sync a publicly-visible bucket without any auth in advance.
gsutil help anon says that:
gsutil users can access publicly ...
0
votes
2
answers
549
views
Dedicated Interconnect to GCP Storage (gsutil)
We are have a dedicated interconnect setup from our data center to GCP using a cloud router.
We have a RHEL linux server setup in the datacenter with a nic port connected to the public Internet and a ...
0
votes
0
answers
414
views
What is the difference between -i and -u options in gsutil rsync command?
The gsutil rsync documentation describes two options which sound identical:
Option
Description
-i
This forces rsync to skip any files which exist on the destination and have a modified time that is ...
0
votes
1
answer
97
views
remotely connect to files via gs path [closed]
I'm using an R package that requires a list of file locations to work. Normally, if I have files stored in remote servers I provide the ftp, http or ssh paths and it works. If I do the same providing ...
0
votes
0
answers
82
views
Why would gsutil come up short when transferring files into a bucket?
I have a very large directory tree sitting in the file system of a Google Compute instance. If I do a "find foo -type f | wc -l" on it, it comes back with 34577, and I have no reason to assume that ...
0
votes
1
answer
1k
views
gsutil "Copying mtime from src to dst" having moved source files?
I use gsutil to backup photos from a dedicated box to google nearline storage.
Recently I moved all of my photos from that dedicated box to a new dedicated box. I made sure to use the relevant ...
0
votes
0
answers
670
views
Why does gsutil rsync on CoreOS fail with 'CommandException: arg () does not name a directory, bucket, or bucket subdir'
On a recent Google Cloud Platform CoreOS VM, the command gsutil rsync -r gs://some-bucket-here dest fails with:
CommandException: arg (dest) does not name a directory, bucket, or bucket subdir.
$ ...
-2
votes
1
answer
829
views
CommandException: Caught non-retryable exception - aborting rsync
.....
At destination listing 8360000...
CommandException: Caught non-retryable exception - aborting rsync
.....
I tried to locate the files with this sync problem but I am not able to do so. Is there ...