Copying different S3 bucket folders with space(s) in their names.

s3cmd is an usual and handy tool for dealing with data in S3 from the command line. Usage Doc.
For simple data copy between two buckets, I would generally use the “s3cmd cp” command.

Usage: s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]

However, recently I came across a two different needs while addressing this seemingly mundane task.

1. The folders to be copied were partially copied to the destination bucket; and it would have been expensive to simply copy them again.

2. The folders were strangely named with spaces in their name. eg. <source folder 1> instead of <source_folder_1> or other better naming practices.

The solution was to use “s3cmd sync” command between the folders and use double quotes (“) to wrap the complete folder name (S3 url).

s3cmd sync "s3://source-bucket/source folder 1/" "s3://destination-bucket/source folder 1/"

This is also a quick method to verify if both folders are in sync. If so, you get an output:
“Summary: 0 source files to copy, 0 files at destination to delete”

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s