Amazon Simple Storage Service: Copy Proposal
Summary
When you want to create a copy of an object in Amazon S3, today you must re-upload your existing object to the new name. If you do not have a copy of the object, you must first download the object and then re-uploaded to Amazon S3, incurring data transfer charges for both the download and the upload as well as a GET and PUT request charge.
By using copy, these operations are combined into a single operation which will save time and money.
This document describes our design for a new Copy operation. We’d appreciate your feedback. As this design is a work in progress, please keep in mind that this specification may change before its public release.
Scenarios
Copy enables many use cases. Some of these include:
- Creating additional copies of an object—The copy operation makes it easy to create a new object from an existing one.
- Renaming objects—Copy an object and Delete the object with the old name.
- Moving objects—Move an object to a new bucket, even to one with a different location constraint (EU, US).
- Updating object metadata—send a copy request with the new metadata that references the original object name.
Requesting a Copy
To copy an object, you must provide us with a source bucket, source object, destination bucket, and destination object. You must be authenticated, and must have read access to the source object and write access to the destination bucket.
The metadata associated with the source object will be copied to the destination object by default. In order to change the metadata, you will need to set the Metadata Directive to REPLACE and specify all the metadata.
The following sections describe how to copy an object using REST and SOAP.
Requesting a Copy with REST
To request a copy from a source to a destination, perform a standard PUT request specifying the name of the new object and add a x-amz-copy-source header that identifies the source bucket and object. If the x-amz-copy-source argument does not specify the source bucket and key, Amazon S3 returns an immediate 400-based response.
As described in the preceding section, the copy operation will copy the metadata from the source object by default. If you wish to modify the metadata, you must respecify all the metadata. To change the default behavior, set the x-amz-metadata-directive header to either of the following values. If the value is set to a value other than these, Amazon S3 returns an immediate 400-based response.
- COPY: Copy the metadata from the original object. If this is specified, any metadata in this request will be ignored. This is the default.
- REPLACE: Ignore the original object’s metadata and replace it with the metadata in this request.
The default ACL for the copied object is private regardless of the ACL of the source object. To use a different ACL, specify the x-amz-acl header as you would with a regular PUT request. For more information, refer to the Amazon S3 Developer Guide.
The copy operation does not accept a body. If the content-length is not zero or the request has a chunk-encoded body that is larger than 0 bytes, the request will fail.
Since we want to mitigate the possibility for time outs during long copy operations, the response to a copy is slightly different than other operations in Amazon S3—once Amazon S3 initiates the actual copy, it will respond with a 200 response. To determine whether the copy was successful, check the returned body for a CopyObjectResult body, which will contain the LastModified date and ETag of the uploaded object. If the copy fails, Amazon S3 might respond with a non-200 error as it does today.
The returned data is an XML document. After the XML declaration, there are a number of whitespace characters that you must read, after which there will be either a CopyObjectResult element or a standard error document. The number of whitespace characters sent to you will be determined by the time required for Amazon S3 to perform the copy operation.
Putting it all together for REST
The following is an example that copies sourcebucket/sourceObject to destinationbucket/destinationObject:
PUT /destinationObject HTTP/1.1 Host: destinationbucket.s3.amazonaws.com x-amz-copy-source: /sourcebucket/sourceObject Authorization: AWS 15B4D3461F177624206A:ENoSbxYByFA0UGLZUqJN5EUnLDg= Date: Wed, 20 Feb 2008 22:12:21 +0000
Note: Although the preceding example uses the virtual hosted-style request, this API also works with path-style requests.
The signature scheme has not changed. All headers prefixed with x-amz- must be part of the signature, including x-amz-copy-source. For the preceding request, the string-to-sign is:
PUT\r\n \r\n \r\n Wed, 20 Feb 2008 22:12:21 +0000\r\n x-amz-copy-source:/sourcebucket/sourceObject\r\n /destinationbucket/destinationObject
The following is an example of a successful response:
HTTP/1.1 200 OK x-amz-id-2: Vyaxt7qEbzv34BnSu5hctyyNSlHTYZFMWK4FtzO+iX8JQNyaLdTshL0KxatbaOZt x-amz-request-id: 6B13C3C5B34AF333 Date: Wed, 20 Feb 2008 22:13:01 +0000 Content-Type: application/xml Transfer-Encoding: chunked Connection: close Server: AmazonS3 <?xml version="1.0" encoding="UTF-8"?> some amount of whitespace <CopyObjectResult> <LastModified>2008-02-20T22:13:01</LastModified> <ETag>"7e9c608af58950deeb370c98608ed097"</ETag> </CopyObjectResult>
The following is an example of an error response:
HTTP/1.1 200 OK x-amz-id-2: yUCAOv6P/njU/ERL0YkshcziXaqFlNje4UwlknCYiZC4P8Br9bGqH+SLk0ZXopQQ x-amz-request-id: 65D5956C3B62B559 Date: Wed, 20 Feb 2008 23:19:01 +0000 Content-Type: application/xml Connection: close Server: AmazonS3 <?xml version="1.0" encoding="UTF-8"?> <!-- some amount of whitespace --> <Error> <Code>InternalError</Code> <Message>We encountered an internal error. Please try again.</Message> <RequestId>65D5956C3B62B559</RequestId> <HostId>yUCAOv6P/njU/ERL0YkshcziXaqFlNje4UwlknCYiZC4P8Br9bGqH+SLk0ZXopQQ</HostId> </Error>
The following is another example of an error response:
HTTP/1.1 403 Forbidden x-amz-request-id: E4CA6F6767D6685C x-amz-id-2: BHzLOATeDuvN8Es1wI8IcERq4kl4dc2A9tOB8Yqr39Ys6fl7N4EJ8sjGiVvu6wLP Content-Type: application/xml Date: Wed, 20 Feb 2008 23:19:01 +0000 Connection: close Server: AmazonS3 <?xml version="1.0" encoding="UTF-8"?> <Error> <Code>AccessDenied</Code> <Message>Access Denied</Message> <RequestId>E4CA6F6767D6685C</RequestId> <HostId>BHzLOATeDuvN8Es1wI8IcERq4kl4dc2A9tOB8Yqr39Ys6fl7N4EJ8sjGiVvu6wLP</HostId> </Error>
Requesting a Copy with SOAP
The copy API for SOAP works the same as our existing SOAP API.
The following is an example of an application-based copy using the WSDL and XSD described in the following sections:
public CopyObjectResult copyObject( string srcBucket, string srcKey, string dstBucket, string dstKey, Metadata [] metadata, Grant[] accessControlList ) { DateTime timestamp = AWSDateFormatter.GetCurrentTimeResolvedToMillis(); string signature = makeSignature( "CopyObject", timestamp ); return s3.CopyObject( srcBucket, srcKey, dstBucket, dstKey, MetadataDirective.REPLACE, metadata, accessControlList, StorageClass.STANDARD, awsAccessKeyId, timestamp, true, signature, null); }
As described in the preceding section, the copy operation will copy the metadata from the original object unless you specify a different constraint in the MetadataDirective. The MetadataDirective has two values:
- COPY: Copy the metadata from the original object. If this is specified, any metadata in this request will be ignored. This is the default.
- REPLACE: Ignore the original object’s metadata and replace it with the metadata in this request.
The default ACL for the copied object is private regardless of the ACL of the source object. To use a different ACL, specify the Metadata field as you would with a PutObject request. For more information, refer to the Amazon S3 Developer Guide.
Unlike the REST-based Copy, Amazon S3 does not respond until the copy is successful. As a result, this operation is sensitive to a socket timeout and you might need to adjust it to work with this API.
Putting it all together for SOAP
The following is an example that copies sourcebucket/sourceObject to destinationbucket/destinationObject:
<CopyObject xmlns="http://doc.s3.amazonaws.com/2006-03-01"> <SourceBucket>sourcebucket</SourceBucket> <SourceObject>sourceObject</SourceObject> <DestinationBucket>destinationbucket</DestinationBucket> <DestinationObject>destinationObject</DestinationObject> <AWSAccessKeyId>1D9FVRAYCP1VJEXAMPLE=</AWSAccessKeyId> <Timestamp>2008-02-18T13:54:10.183Z</Timestamp> <Signature>Iuyz3d3P0aTou39dzbq7RrtSFmw=</Signature> </CopyObject>
The signature scheme has not changed. For the preceding request, the string-to-sign is:
AmazonS3CopyObject2008-02-18T13:54:10.183Z
The following is an example of a successful response:
<CopyObjectResponse xmlns="http://doc.s3.amazonaws.com/2006-03-01"> <CopyObjectResponse> <ETag>"828ef3fdfa96f00ad9f27c383fc9ac7f"</ETag> <LastModified>2008-02-18T13:54:10.183Z</LastModified> </CopyObjectResponse> </CopyObjectResponse>
Modified WSDL
The following is an excerpt of our WSDL for the copy operation:
<?xml version="1.0" encoding="UTF-8"?> <wsdl:definitions targetNamespace="http://s3.amazonaws.com/doc/2006-03-01/"> <!-- ... --> <wsdl:message name="CopyObjectRequest"> <wsdl:part element="tns:CopyObject" name="parameters"/> </wsdl:message> <wsdl:message name="CopyObjectResponse"> <wsdl:part element="tns:CopyObjectResponse" name="parameters"/> </wsdl:message> <wsdl:portType name="AmazonS3"> <!-- ... --> <wsdl:operation name="CopyObject"> <wsdl:input message="tns:CopyObjectRequest" name="CopyObjectRequest"/> <wsdl:output message="tns:CopyObjectResponse" name="CopyObjectResponse"/> </wsdl:operation> </wsdl:portType> <wsdl:binding name="AmazonS3SoapBinding" type="tns:AmazonS3"> <!-- ... --> <wsdl:operation name="CopyObject"> <wsdlsoap:operation soapAction=""/> <wsdl:input name="CopyObjectRequest"> <wsdlsoap:body use="literal"/> </wsdl:input> <wsdl:output name="CopyObjectResponse"> <wsdlsoap:body use="literal"/> </wsdl:output> </wsdl:operation> </wsdl:binding> <!-- ... --> </wsdl:definitions>
Modified XSD Files
The following is an excerpt of our XSD file for the copy operation:
<?xml version="1.0" encoding="UTF-8"?> <xsd:schema xmlns:tns="http://s3.amazonaws.com/doc/2006-03-01/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified" targetNamespace="http://s3.amazonaws.com/doc/2006-03-01/"> <!-- ... --> <xsd:simpleType name="MetadataDirective"> <xsd:restriction base="xsd:string"> <xsd:enumeration value="COPY"/> <xsd:enumeration value="REPLACE"/> </xsd:restriction> </xsd:simpleType> <xsd:element name="CopyObject"> <xsd:complexType> <xsd:sequence> <xsd:element name="SourceBucket" type="xsd:string"/> <xsd:element name="SourceKey" type="xsd:string"/> <xsd:element name="DestinationBucket" type="xsd:string"/> <xsd:element name="DestinationKey" type="xsd:string"/> <xsd:element name="MetadataDirective" type="tns:MetadataDirective" minOccurs="0" maxOccurs="unbounded"/> <xsd:element name="Metadata" type="tns:MetadataEntry" minOccurs="0" maxOccurs="unbounded"/> <xsd:element name="AccessControlList" type="tns:AccessControlList" minOccurs="0"/> <xsd:element name="StorageClass" type="tns:StorageClass" minOccurs="0"/> <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/> <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/> <xsd:element name="Signature" type="xsd:string" minOccurs="0"/> <xsd:element name="Credential" type="xsd:string" minOccurs="0"/> </xsd:sequence> </xsd:complexType> </xsd:element> <xsd:complexType name="CopyObjectResponse"> <xsd:sequence> <xsd:element name="CopyObjectResult" type="tns:CopyObjectResult" /> </xsd:sequence> </xsd:complexType> <xsd:complexType name="CopyObjectResult"> <xsd:sequence> <xsd:element name="LastModified" type="xsd:dateTime"/> <xsd:element name="ETag" type="xsd:string"/> </xsd:sequence> </xsd:complexType> </xsd:schema>
Pricing
Pricing for the copy operation has not yet been finalized.