Building on this article here is a simple ruby script, that copies files between two buckets of the same S3 account, omitting files already present (by name).
This variant adds a list of path prefixes, so you can selectively copy only certain directories of your buckets.
Furthermore it copies the original buckets ACLs for each key.
require 'rubygems' require 'right_aws' aws_access_key_id = 'YOUR AMAZON ACCESS KEY' aws_secret_access_key = 'YOUR AMAZON SECRET ACCESS KEY' source_bucket = 'SOURCE BUCKET NAME' target_bucket = 'TARGET BUCKET NAME' prefixes = [PATH_PREFIX1, PATH_PREFIX2, ...] s3 = RightAws::S3Interface.new(aws_access_key_id, aws_secret_access_key) copied_keys = Array.new (prefixes || ['']).each do |prefix| s3.incrementally_list_bucket(target_bucket, {:prefix => prefix}) do |key_set| copied_keys << key_set[:contents].map{|k| k[:key]}.flatten end end copied_keys.flatten! (prefixes || ['']).each do |prefix| s3.incrementally_list_bucket(source_bucket, {:prefix => prefix}) do |key_set| key_set[:contents].each do |key| key = key[:key] if copied_keys.include?(key) puts "#{target_bucket} #{key} already exists. Skipping..." else puts "Copying #{source_bucket} #{key}, setting acl" retries=0 begin s3.copy(source_bucket, key, target_bucket) acl = s3.get_acl(source_bucket, key) s3.put_acl(target_bucket, key, acl[:object]) rescue Exception => e puts "cannot copy key, #{e.inspect}\nretrying #{retries} out of 10 times..." retries += 1 retry if retries <= 10 end end end end end