Upload/Download from S3 CLI (rclone)

rclone is a command line program to manage files on cloud storage. It supports many cloud storage providers like Google Drive, Amazon S3, Dropbox, etc.

It is one of the most useful tools I have used.

Here is a simple script to upload and download files from s3/r2 bucket.

Usage -

./s3.sh <file_path> <bucket> <method> <prefix>
PUT Example: ./s3.sh path/to/file bucket-name PUT prefix/
Will upload file path/to/file to s3://bucket-name/prefix/path/to/file
GET Example: ./s3.sh path/to/file bucket-name GET prefix/
Will download file from s3://bucket-name/prefix/path/to/file to path/to/file
Note: prefix is optional

Note

• Requires rclone binary in path. (apt-get install rclone)
• Requires rclone config named s3conn or the following environment variables

RCLONE_CONFIG_S3CONN_TYPE=s3
RCLONE_CONFIG_S3CONN_PROVIDER=AWS
RCLONE_CONFIG_S3CONN_ENV_AUTH=false
RCLONE_CONFIG_S3CONN_REGION=us-east-1
RCLONE_CONFIG_S3CONN_ENDPOINT=
RCLONE_CONFIG_S3CONN_ACCESS_KEY_ID=XXXX
RCLONE_CONFIG_S3CONN_SECRET_ACCESS_KEY=XXXX

For Cloudflare

RCLONE_CONFIG_S3CONN_TYPE=s3
RCLONE_CONFIG_S3CONN_PROVIDER=Cloudflare
RCLONE_CONFIG_S3CONN_ENV_AUTH=false
RCLONE_CONFIG_S3CONN_REGION=auth
RCLONE_CONFIG_S3CONN_ENDPOINT=https://<ACCOUNT_ID>.r2.cloudflarestorage.com
RCLONE_CONFIG_S3CONN_ACCESS_KEY_ID=XXXX
RCLONE_CONFIG_S3CONN_SECRET_ACCESS_KEY=XXXX
RCLONE_CONFIG_S3CONN_no_head_object=true
RCLONE_CONFIG_S3CONN_no_list_bucket=true

https://developers.cloudflare.com/r2/api/s3/api/

Code on Github Gist -