Amazon Simple Storage Service: Copy Proposal

Summary

When you want to create a copy of an object in Amazon S3, today you must re-upload your existing object to the new name. If you do not have a copy of the object, you must first download the object and then re-uploaded to Amazon S3, incurring data transfer charges for both the download and the upload as well as a GET and PUT request charge.

TodaysCopy.png

By using copy, these operations are combined into a single operation which will save time and money.

UsingCopy.png

This document describes our design for a new Copy operation. We’d appreciate your feedback. As this design is a work in progress, please keep in mind that this specification may change before its public release.

Scenarios

Copy enables many use cases. Some of these include:

Requesting a Copy

To copy an object, you must provide us with a source bucket, source object, destination bucket, and destination object. You must be authenticated, and must have read access to the source object and write access to the destination bucket.

The metadata associated with the source object will be copied to the destination object by default. In order to change the metadata, you will need to set the Metadata Directive to REPLACE and specify all the metadata.

The following sections describe how to copy an object using REST and SOAP.

Requesting a Copy with REST

To request a copy from a source to a destination, perform a standard PUT request specifying the name of the new object and add a x-amz-copy-source header that identifies the source bucket and object. If the x-amz-copy-source argument does not specify the source bucket and key, Amazon S3 returns an immediate 400-based response.

As described in the preceding section, the copy operation will copy the metadata from the source object by default. If you wish to modify the metadata, you must respecify all the metadata. To change the default behavior, set the x-amz-metadata-directive header to either of the following values. If the value is set to a value other than these, Amazon S3 returns an immediate 400-based response.

The default ACL for the copied object is private regardless of the ACL of the source object. To use a different ACL, specify the x-amz-acl header as you would with a regular PUT request. For more information, refer to the Amazon S3 Developer Guide.

The copy operation does not accept a body. If the content-length is not zero or the request has a chunk-encoded body that is larger than 0 bytes, the request will fail.

Since we want to mitigate the possibility for time outs during long copy operations, the response to a copy is slightly different than other operations in Amazon S3—once Amazon S3 initiates the actual copy, it will respond with a 200 response. To determine whether the copy was successful, check the returned body for a CopyObjectResult body, which will contain the LastModified date and ETag of the uploaded object. If the copy fails, Amazon S3 might respond with a non-200 error as it does today.

The returned data is an XML document. After the XML declaration, there are a number of whitespace characters that you must read, after which there will be either a CopyObjectResult element or a standard error document. The number of whitespace characters sent to you will be determined by the time required for Amazon S3 to perform the copy operation.

Putting it all together for REST

The following is an example that copies sourcebucket/sourceObject to destinationbucket/destinationObject:

PUT /destinationObject HTTP/1.1
Host: destinationbucket.s3.amazonaws.com
x-amz-copy-source: /sourcebucket/sourceObject
Authorization: AWS 15B4D3461F177624206A:ENoSbxYByFA0UGLZUqJN5EUnLDg=
Date: Wed, 20 Feb 2008 22:12:21 +0000

Note: Although the preceding example uses the virtual hosted-style request, this API also works with path-style requests.

The signature scheme has not changed. All headers prefixed with x-amz- must be part of the signature, including x-amz-copy-source. For the preceding request, the string-to-sign is:

PUT\r\n
\r\n
\r\n
Wed, 20 Feb 2008 22:12:21 +0000\r\n
x-amz-copy-source:/sourcebucket/sourceObject\r\n
/destinationbucket/destinationObject

The following is an example of a successful response:

HTTP/1.1 200 OK
x-amz-id-2: Vyaxt7qEbzv34BnSu5hctyyNSlHTYZFMWK4FtzO+iX8JQNyaLdTshL0KxatbaOZt
x-amz-request-id: 6B13C3C5B34AF333
Date: Wed, 20 Feb 2008 22:13:01 +0000
Content-Type: application/xml
Transfer-Encoding: chunked
Connection: close
Server: AmazonS3

<?xml version="1.0" encoding="UTF-8"?>
some amount of whitespace
<CopyObjectResult>
   <LastModified>2008-02-20T22:13:01</LastModified>
   <ETag>&quot;7e9c608af58950deeb370c98608ed097&quot;</ETag>
</CopyObjectResult>

The following is an example of an error response:

HTTP/1.1 200 OK
x-amz-id-2: yUCAOv6P/njU/ERL0YkshcziXaqFlNje4UwlknCYiZC4P8Br9bGqH+SLk0ZXopQQ
x-amz-request-id: 65D5956C3B62B559
Date: Wed, 20 Feb 2008 23:19:01 +0000
Content-Type: application/xml
Connection: close
Server: AmazonS3

<?xml version="1.0" encoding="UTF-8"?>
<!-- some amount of whitespace -->
<Error>
   <Code>InternalError</Code>
   <Message>We encountered an internal error. Please try again.</Message>
   <RequestId>65D5956C3B62B559</RequestId>
   <HostId>yUCAOv6P/njU/ERL0YkshcziXaqFlNje4UwlknCYiZC4P8Br9bGqH+SLk0ZXopQQ</HostId>
</Error>

The following is another example of an error response:

HTTP/1.1 403 Forbidden
x-amz-request-id: E4CA6F6767D6685C
x-amz-id-2: BHzLOATeDuvN8Es1wI8IcERq4kl4dc2A9tOB8Yqr39Ys6fl7N4EJ8sjGiVvu6wLP
Content-Type: application/xml
Date: Wed, 20 Feb 2008 23:19:01 +0000
Connection: close
Server: AmazonS3

<?xml version="1.0" encoding="UTF-8"?>
<Error>
   <Code>AccessDenied</Code>
   <Message>Access Denied</Message>
   <RequestId>E4CA6F6767D6685C</RequestId>
   <HostId>BHzLOATeDuvN8Es1wI8IcERq4kl4dc2A9tOB8Yqr39Ys6fl7N4EJ8sjGiVvu6wLP</HostId>
</Error>

Requesting a Copy with SOAP

The copy API for SOAP works the same as our existing SOAP API.

The following is an example of an application-based copy using the WSDL and XSD described in the following sections:

        public CopyObjectResult copyObject( string srcBucket, string srcKey, string dstBucket, string dstKey, Metadata [] metadata, Grant[] accessControlList )
        {
            DateTime timestamp = AWSDateFormatter.GetCurrentTimeResolvedToMillis();
            string signature = makeSignature( "CopyObject", timestamp );
            return s3.CopyObject( srcBucket, srcKey, dstBucket, dstKey, MetadataDirective.REPLACE, metadata, accessControlList, StorageClass.STANDARD, awsAccessKeyId, timestamp, true, signature, null);
        }

As described in the preceding section, the copy operation will copy the metadata from the original object unless you specify a different constraint in the MetadataDirective. The MetadataDirective has two values:

The default ACL for the copied object is private regardless of the ACL of the source object. To use a different ACL, specify the Metadata field as you would with a PutObject request. For more information, refer to the Amazon S3 Developer Guide.

Unlike the REST-based Copy, Amazon S3 does not respond until the copy is successful. As a result, this operation is sensitive to a socket timeout and you might need to adjust it to work with this API.

Putting it all together for SOAP

The following is an example that copies sourcebucket/sourceObject to destinationbucket/destinationObject:

<CopyObject xmlns="http://doc.s3.amazonaws.com/2006-03-01">
  <SourceBucket>sourcebucket</SourceBucket>
  <SourceObject>sourceObject</SourceObject>
  <DestinationBucket>destinationbucket</DestinationBucket>
  <DestinationObject>destinationObject</DestinationObject>
  <AWSAccessKeyId>1D9FVRAYCP1VJEXAMPLE=</AWSAccessKeyId>
  <Timestamp>2008-02-18T13:54:10.183Z</Timestamp>
  <Signature>Iuyz3d3P0aTou39dzbq7RrtSFmw=</Signature>
</CopyObject>

The signature scheme has not changed. For the preceding request, the string-to-sign is:

AmazonS3CopyObject2008-02-18T13:54:10.183Z

The following is an example of a successful response:

<CopyObjectResponse xmlns="http://doc.s3.amazonaws.com/2006-03-01">
  <CopyObjectResponse>
    <ETag>&quot;828ef3fdfa96f00ad9f27c383fc9ac7f&quot;</ETag>
    <LastModified>2008-02-18T13:54:10.183Z</LastModified>
  </CopyObjectResponse>
</CopyObjectResponse>

Modified WSDL

The following is an excerpt of our WSDL for the copy operation:

<?xml version="1.0" encoding="UTF-8"?>
<wsdl:definitions targetNamespace="http://s3.amazonaws.com/doc/2006-03-01/">
<!-- ... -->
   <wsdl:message name="CopyObjectRequest">
      <wsdl:part element="tns:CopyObject" name="parameters"/>
   </wsdl:message>
   <wsdl:message name="CopyObjectResponse">
      <wsdl:part element="tns:CopyObjectResponse" name="parameters"/>
   </wsdl:message>

   <wsdl:portType name="AmazonS3">
<!-- ... -->
      <wsdl:operation name="CopyObject">
         <wsdl:input message="tns:CopyObjectRequest" name="CopyObjectRequest"/>
         <wsdl:output message="tns:CopyObjectResponse" name="CopyObjectResponse"/>
      </wsdl:operation>
   </wsdl:portType>

   <wsdl:binding name="AmazonS3SoapBinding" type="tns:AmazonS3">
<!-- ... -->
      <wsdl:operation name="CopyObject">
         <wsdlsoap:operation soapAction=""/>
         <wsdl:input name="CopyObjectRequest">
            <wsdlsoap:body use="literal"/>
         </wsdl:input>
         <wsdl:output name="CopyObjectResponse">
            <wsdlsoap:body use="literal"/>
         </wsdl:output>
      </wsdl:operation>
   </wsdl:binding>
<!-- ... -->
</wsdl:definitions>

Modified XSD Files

The following is an excerpt of our XSD file for the copy operation:

<?xml version="1.0" encoding="UTF-8"?>
<xsd:schema
  xmlns:tns="http://s3.amazonaws.com/doc/2006-03-01/"
  xmlns:xsd="http://www.w3.org/2001/XMLSchema"
  elementFormDefault="qualified"
  targetNamespace="http://s3.amazonaws.com/doc/2006-03-01/">
<!-- ... -->

  <xsd:simpleType name="MetadataDirective">
    <xsd:restriction base="xsd:string">
      <xsd:enumeration value="COPY"/>
      <xsd:enumeration value="REPLACE"/>
    </xsd:restriction>
  </xsd:simpleType>

  <xsd:element name="CopyObject">
    <xsd:complexType>
      <xsd:sequence>
        <xsd:element name="SourceBucket" type="xsd:string"/>
        <xsd:element name="SourceKey" type="xsd:string"/>
        <xsd:element name="DestinationBucket" type="xsd:string"/>
        <xsd:element name="DestinationKey" type="xsd:string"/>
        <xsd:element name="MetadataDirective" type="tns:MetadataDirective" minOccurs="0" maxOccurs="unbounded"/>
        <xsd:element name="Metadata" type="tns:MetadataEntry" minOccurs="0" maxOccurs="unbounded"/>
        <xsd:element name="AccessControlList" type="tns:AccessControlList" minOccurs="0"/>
        <xsd:element name="StorageClass" type="tns:StorageClass" minOccurs="0"/>
        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
      </xsd:sequence>
    </xsd:complexType>
  </xsd:element>

  <xsd:complexType name="CopyObjectResponse">
    <xsd:sequence>
      <xsd:element name="CopyObjectResult" type="tns:CopyObjectResult" />
    </xsd:sequence>
  </xsd:complexType>

  <xsd:complexType name="CopyObjectResult">
    <xsd:sequence>
      <xsd:element name="LastModified" type="xsd:dateTime"/>
      <xsd:element name="ETag" type="xsd:string"/>
    </xsd:sequence>
  </xsd:complexType>

</xsd:schema>

Pricing

Pricing for the copy operation has not yet been finalized.