Amazon Simple Storage Service: Browser-Based Uploads using POST Proposal

• Summary
• Scenarios
• The HTML
  • The form declaration
    • Example
  • Form Fields
  • Access Control
    • Expiration date (Required)
    • The Access Conditions (Required)
      • Matching a Particular Value
      • Matching a Particular Prefix
      • Limiting Uploaded Content Size
    • Constraint Language Grammar
  • Dealing with Success
    • Example
  • Dealing with Failure
  • Location Constrained Buckets
• Putting it all together
  • Access Control Example
  • A Sample Form
    • Variation on the form
  • Sample Request
  • Sample Response
• Limitations
• Pricing

Summary

To upload content to Amazon S3, developers previously had the choice of uploading the content via our REST or SOAP APIs. Unfortunately, these APIs are not broadly accessible in web browsers. To allow developers the ability to upload content via web browsers, Amazon S3 will be releasing POST support by the end of 2007.

The purpose of this document is to describe our approach to browser uploads and to allow you a chance to help us refine it to better serve you. Please note that you should not start coding against the contents of this document as things will likely change before the feature is publicly available.

Scenarios

One of the primary advantages for using POST is to avoid proxying user uploads via your own webserver to Amazon S3.

For an example, if you have a photo sharing web site and want your users to upload content directly into Amazon S3, your webserver can now serve a standard HTML form that will allow your users to upload their content directly into Amazon S3 on your behalf. Once the content is successful uploaded in Amazon S3, you can optionally redirect the user to your web site.

The Browser Uploads via POST feature is designed to support all the use-cases presently supported by Object PUT, but to allow you to do that via a web browser with a standard HTML form. Most parameters that were passed in via a REST header will now be passed in via an input field, and the access control portion has been adapted to a more flexible system, allowing you to not be constrained when designing your applications.

The HTML

This feature will use HTML Forms to submit metadata, a single object, and other request parameters to Amazon S3.

The form declaration

The form declaration has three components to it, the action, the method, and the enclosure type (enctype).

The action defines the URL that will process the request; this must be set to the bucket's URL. For example, if your bucket's name is "johnsmith", then the URL would be "http://johnsmith.s3.amazonaws.com/".

Note: The key name is specified within an INPUT field within and not in this declaration.

The method must be POST.

The enclosure type (enctype) must be specified and must be set to multipart/form-data (Per RFC 1867).

If any of these values are not properly set, the request will fail with an error.

Example

Here is an example of a proper form declaration for the bucket "johnsmith":

<form action="http://johnsmith.s3.amazonaws.com/" method="post" enctype="multipart/form-data">

Form Fields

All metadata, request properties, access control, and the file will be submitted via standard HTML form fields. Please refer to the HTML Standard's Forms Section or any HTML reference guide for more information on the form fields themselves.

The following table shows a list of fields that may be submitted with the form. Anything outside of this set will be ignored. Any invalid information presented in this set will result in an error.

Note: All values in this table must also be in the Policy Document; see Access Control for more information.

Form Input Element Name	Description	Behavior
AWSAccessKeyId	The AWS Access Key ID of the owner of the bucket who will be granting Anonymous access for a request that satisfies the set of constraints in the Policy.
acl	One of our Canned Access Control Lists: private, public-read, public-read-write, or authenticated-read.	If unspecified, this defaults to private. If invalid, this will result in an error. See PUT Documentation.
Cache-Control Content-Type Content-Disposition Content-Encoding Expires	REST-specific Store-and-Forward headers.	See PUT Documentation.
file (required)	Data of the file being uploaded. This may either be a file input type or may be a textarea input field.	There can only be one file element per request. This must be the last element in the form except for the submit button.
key (required)	The name that the uploaded key should have.	See Using Keys
policy	Security Policy describing what is permitted in the request. Requests without a security policy are considered anonymous and only work on publicly writable buckets.	See Access Control
redirect	URL which client is redirected to on upload successful completion	See Dealing with Success
signature	HMAC signature using the secret key of the provided AWSAccessKeyID on the Policy.	See Access Control
Field names prefixed with x-amz-meta-	User-specified metadata. Amazon S3 will not validate the actual data.	See PUT Documentation

You are permitted to add additional form fields that are prefixed with x-ignore-. These values will be completely ignored.

There must only be a single file specified per upload. Anything following the file form field will be silently ignored. Typically the only thing that will follow the file form field will be the submit button.

Any of the form fields above may reference the uploaded filename via the

${filename}

, with the exception of the AWSAccessKeyID, Policy, or Signature fields. For example, if you wanted the key name to be the filename, you would set the key's value to

${filename}

. It should be noted that in the case of a non-file upload, this is not expanded or replaced.

Access Control

We have chosen an access control scheme that provides a flexible way to enable users ability to upload content, while still providing the ability to tightly control aspects of the upload that are important to your scenario. This flexibility is achieved by allowing you to set constraints on the specific fields being uploaded. You can choose how much to lock down the user's options.

Note: This scheme is currently only applicable to browser uploads using POST. In the future, we may expose it to other contexts.

We call the JSON document which describes the conditions on a user’s request the Policy Document. This document describes a set of conditions on the fields provided with the POST request as well as some implied by the request. An example of an implied field is the bucket field, as that is inferred from request URL, rather than from a POST field. The Policy Document provides the ability to express conditions that:

• A field is equal to a particular value.
• A field starts with a particular value.

As with other REST header processing in Amazon S3, field name lookup is not cases-sensitive. A Policy Document must completely describe a request. If the request includes any field that is not covered by Policy Document, an error will result. A field is considered covered by a Policy Document if the conditions mention that field.

The following example requires that an upload satisfy all of these conditions:

• the policy document expires on December 1st, 2007 at 12:00:00 GMT,
• the ACL must be public-read,
• must be to the johnsmith bucket, and
• the key must start with the prefix user/eric/.
• the content-length must be between 2K and 20MB.

{ "expiration": "2007-12-01T12:00:00.000Z",
  "conditions": [
    {"acl": "public-read" },
    {"bucket": "johnsmith" },
    ["starts-with", "$key", "user/eric/"],
    ["content-length-range", 2048, 20971520]
  ]
}

This Policy document is Base64 encoded and the Signature is the HMAC of the Base64 encoding.

Each form field mentioned in the form must have an entry in a Policy document. This prevents an end-user from changing your form for purposes that you do not wish to allow. If a form field is submitted that changes the meaning of the request and is not mentioned in the Policy document, a failure will result. If the policy references a parameter that is not submitted via a form field, an error will result.

Expiration date (Required)

Description: Developers are required to place an upper bound on how long the document is valid for. This is expressed by using an ISO8601 date GMT date format.

Example:

"expiration": "2007-12-01T13:54:23.000Z"

expresses that the policy document is not valid after 2007-12-01T13:54:23.000Z

The Access Conditions (Required)

Description: The conditions section enumerates what the properties that a request must have. If any of the conditions enumerated is not satisfied, the request will fail.

For example, if you required that the key parameter start with the prefix user/ and the user changed this to usr/, the request would fail.

Anything that is not mentioned in the conditions section will be silently ignored.

Syntax:

"conditions": [ ... ]

specifies a list of conditions. The ... represents one or more conditions that must all evaluate to true.

Matching a Particular Value

Description: There are certain fields that you want to match a particular value, such as matching a bucket name or requiring that an object is uploaded using the public-read access control policy. The field value is case sensitive, but the name is not.

Syntax: There are two ways to require that the field fieldname matches the string value. The value is case sensitive. If the value starts with a dollar sign ($), the dollar sign must be escaped with a backslash (\$)

• {"fieldname": "value" ],

• [ "eq", "$fieldname", "S" ],

  (Note the $ prefix).

Examples:

• { "bucket": "johnsmith" }

  ensures that the bucket name is johnsmith.

• [ "eq", "$key", "blog/11/17/00" ]

  ensures that the key name is blog/11/17/00.

• [ "eq", "$x-amz-meta-price", "\$10.00" ],

  ensures that the x-amz-meta-price is equal to $10.00; note how the $ is escaped though.

Matching a Particular Prefix

Description: If you wanted to allow a user to write to a particular portion of your namespace, you could accomplish this by using prefix matching. It is also applicable to other fields.

Syntax:

[ "starts-with", "$fieldname", "prefix" ]

requires that the fieldname (note that $) matches the prefix mentioned. The prefix is case-sensitive.

Examples:

• [ "starts-with", "$key", "users/eric/" ]

  ensures that the key starts with the prefix users/eric/.

• [ "starts-with", "$x-amz-meta-tag", "" ]

  ensures the x-amz-meta-tag field is specified in the request, but its exact value does not matter.

Limiting Uploaded Content Size

Description: If you want to limit the size of objects that users can upload to your bucket, you can accomplish this by limiting the Content-Length range. If this is not specified, this will default to the range 0 to 5GB.

Syntax:

[ "content-length-range", minimum, maximum ]

requires that the content length is the range stated. Both values must be numeric.

Example:

• [ "content-length-range", 2048, 20971520 ]

  requires that the uploaded content is between 2K and 20MB.

Constraint Language Grammar

Doc :=
    # Expiration and Conditions are actually allowed in any order
    '{' ( Expiration ',') ( Conditions ','? ) '}'

Expiration :=
    # expiration date is specified in ISO 8601 format.
    '"expiration"' ':' StringLiteral

Conditions :=
    '"conditions"' ':' '[' Condition (',' Condition)* ','? ']'

Condition :=
    SimpleCondition | ConditionTerm

SimpleCondition :=
    # first string is a field-name, and the second string is value for an equality test
    '{' StringLiteral ':' StringLiteral '}'

ConditionTerm :=
    '[' StringLiteral ( ',' ConditionValue )* ','? ']'

ConditionValue :=
    StringLiteral
    FieldReferenceLiteral
  | PositiveInteger
  | ConditionTerm

StringLiteral :=
   '"' /([^\"]|(\\[rnt$]))*/ '"'

FieldReferenceLiteral :=
   '"$' /([^\"]|(\\[rnt$]))*/ '"'

PositiveInteger :=
   /[0-9]+/

Dealing with Success

By default, Amazon S3 will return an empty document with the status code of 200 when an object is successfully uploaded.

Developers may choose to change this behavior by specifying a full URL within the redirect form field. If specified, on successful uploads, Amazon S3 will redirect users with a 303 to this URL and it will contain the following query string parameters:

Parameter	Description
bucket	Name of the bucket the object was uploaded to.
key	Name of the key that identifies the object.
etag	ETag of the object.

Note: If there is a problem interpreting the URL, Amazon S3 will ignore the redirect field.

Please note that the redirect is not guaranteed to be followed. It is possible that an upload would succeed, but that a networking problem on the end-users network prevents them from following the redirect. It is also possible that in certain failure conditions, that a file is actually uploaded but you are not notified about the upload.

Example

If you wanted to redirect your customers to http://johnsmith.s3.amazonaws.com/successful_upload.html, you would do this:

<input type="hidden" name="redirect" value="http://johnsmith.s3.amazonaws.com/successful_upload.html" >

If a user uploaded the key image.jpg to the johnsmith bucket, they would be redirected to: http://johnsmith.s3.amazonaws.com/successful_upload.html?bucket=johnsmith&key=image.jpg&etag=7ebcf2ee3b4dd720466bc4b85c63b54e .

Note: The policy document also needs to reflect the redirect field.

Dealing with Failure

When an error occurs when uploading content via POST, Amazon S3 will set the error code appropriately and send an XML document describing the error.

We do not currently expose the ability to redirect on failure.

Depending on the type of failure, users are encouraged to retry requests.

Please note that on certain errors, it is possible that a key is successfully committed to Amazon S3.

Location Constrained Buckets

If your bucket was created using CreateBucketConfiguration, it is possible for your end-users to require a redirect. If this does occur, various browsers will react differently and in some cases, incorrectly. This should be relatively rare.

Putting it all together

Access Control Example

The following policy requires that an upload match all of these requirements:

• the upload be started before December 1, 2007 at 12:00:00 GMT.
• the bucket must be johnsmith
• the key starts with the prefix user/eric/
• the ACL must be public-read
• Successfully uploaded objects must be redirected to http://johnsmith.s3.amazonaws.com/successful_upload.html
• the Content-Type must start with the prefix image/.
• the user metadata uuid field must have the value 14365123651274

{ "expiration": "2007-12-01T12:00:00.000Z",
  "conditions": [
    {"bucket": "johnsmith" },
    ["starts-with", "$key", "user/eric/"],
    {"acl": "public-read" },
    {"redirect": "http://johnsmith.s3.amazonaws.com/successful_upload.html" },
    ["starts-with", "$Content-Type", "image/"],
    {"x-amz-meta-uuid": "14365123651274"},
    ["starts-with", "$x-amz-meta-tag", ""],
  ]
}

The Base64 encoding of this document is as follows:

eyAiZXhwaXJhdGlvbiI6ICIyMDA3LTEyLTAxVDEyOjAwOjAwLjAwMFoiLAogICJjb25kaXRpb25zIjogWwogICAgeyJidWNrZXQiOiAiam9obnNtaXRoIiB9LAogICAgWyJzdGFydHMtd2l0aCIsICIka2V5IiwgInVzZXIvZXJpYy8iXSwKICAgIHsiYWNsIjogInB1YmxpYy1yZWFkIiB9LAogICAgeyJyZWRpcmVjdCI6ICJodHRwOi8vam9obnNtaXRoLnMzLmFtYXpvbmF3cy5jb20vc3VjY2Vzc2Z1bF91cGxvYWQuaHRtbCIgfSwKICAgIFsic3RhcnRzLXdpdGgiLCAiJENvbnRlbnQtVHlwZSIsICJpbWFnZS8iXSwKICAgIHsieC1hbXotbWV0YS11dWlkIjogIjE0MzY1MTIzNjUxMjc0In0sCiAgICBbInN0YXJ0cy13aXRoIiwgIiR4LWFtei1tZXRhLXRhZyIsICIiXSwKICBdCn0K

If your secret key id were uV3F3YluFJax1cknvbcGwgjvx4QpvB+leU8dUj2o, the signature for the above Policy document would be:

2qCp0odXe7A9IYyUVqn0w2adtCA=

A Sample Form

To support a POST to the johnsmith bucket, a form similar to the following would be used:

<form action="http://johnsmith.s3.amazonaws.com/" method="post" enctype="multipart/form-data">
Key to upload: <input type="input" name="key" value="user/eric/" /><br />
Tags for File: <input type="input" name="x-amz-meta-tag" value="" /><br />
<input type="hidden" name="acl" value="public-read" />
<input type="hidden" name="x-amz-meta-uuid" value="14365123651274" />
<input type="hidden" name="redirect" value="http://johnsmith.s3.amazonaws.com/successful_upload.html" >
<input type="hidden" name="AWSAccessKeyId " value="15B4D3461F177624206A" />
<input type="hidden" name="Policy" value="eyAiZXhwaXJhdGlvbiI6ICIyMDA3LTEyLTAxVDEyOjAwOjAwLjAwMFoiLAogICJjb25kaXRpb25zIjogWwogICAgeyJidWNrZXQiOiAiam9obnNtaXRoIiB9LAogICAgWyJzdGFydHMtd2l0aCIsICIka2V5IiwgInVzZXIvZXJpYy8iXSwKICAgIHsiYWNsIjogInB1YmxpYy1yZWFkIiB9LAogICAgeyJyZWRpcmVjdCI6ICJodHRwOi8vam9obnNtaXRoLnMzLmFtYXpvbmF3cy5jb20vc3VjY2Vzc2Z1bF91cGxvYWQuaHRtbCIgfSwKICAgIFsic3RhcnRzLXdpdGgiLCAiJENvbnRlbnQtVHlwZSIsICJpbWFnZS8iXSwKICAgIHsieC1hbXotbWV0YS11dWlkIjogIjE0MzY1MTIzNjUxMjc0In0sCiAgICBbInN0YXJ0cy13aXRoIiwgIiR4LWFtei1tZXRhLXRhZyIsICIiXSwKICBdCn0K" />
<input type="hidden" name="Signature" value="2qCp0odXe7A9IYyUVqn0w2adtCA=" />
<!-- The file must be the last meaningful element in the request; other elements after this will be ignored -->
File: <input type="file" name="file" /> <br />
<input type="submit" name="submit" value="Upload to Amazon S3" />
</form>

Variation on the form

Instead of uploading a file directly into S3, some developers may wish to have user-submitted content. For example, a simple blog-type application could be built using POST functionality by using a textarea instead of a file input field. Note that you will probably want to use the Content-Type header.

For our example, we will be using the following Policy which is similar to the above, except that the redirect field is different and the content-type must be text/html:

{ "expiration": "2007-12-01T12:00:00.000Z",
  "conditions": [
    {"bucket": "johnsmith" },
    ["starts-with", "$key", "user/eric/"],
    {"acl": "public-read" },
    {"redirect": "http://johnsmith.s3.amazonaws.com/new_post.html" },
    ["eq", "$Content-Type", "text/html"],
    {"x-amz-meta-uuid": "14365123651274"},
    ["starts-with", "$x-amz-meta-tag", ""],
  ]
}

And this would be the resulting form:

<form action="http://johnsmith.s3.amazonaws.com/" method="post" enctype="multipart/form-data">
Blog Post URL: <input type="input" name="key" value="user/eric/blog/2007/11/16/" /><br />
Tags for File: <input type="input" name="x-amz-meta-tag" value="" /><br />
<input type="hidden" name="acl" value="public-read" />
<input type="hidden" name="Content-Type" value="text/html" />
<input type="hidden" name="x-amz-meta-uuid" value="14365123651274" />
<input type="hidden" name="redirect" value="http://johnsmith.s3.amazonaws.com/new_post.html" >
<input type="hidden" name="AWSAccessKeyId " value="15B4D3461F177624206A" />
<input type="hidden" name="Policy" value="eyAiZXhwaXJhdGlvbiI6ICIyMDA3LTEyLTAxVDEyOjAwOjAwLjAwMFoiLAogICJjb25kaXRpb25zIjogWwogICAgeyJidWNrZXQiOiAiam9obnNtaXRoIiB9LAogICAgWyJzdGFydHMtd2l0aCIsICIka2V5IiwgInVzZXIvZXJpYy8iXSwKICAgIHsiYWNsIjogInB1YmxpYy1yZWFkIiB9LAogICAgeyJyZWRpcmVjdCI6ICJodHRwOi8vam9obnNtaXRoLnMzLmFtYXpvbmF3cy5jb20vbmV3X3Bvc3QuaHRtbCIgfSwKICAgIFsiZXEiLCAiJENvbnRlbnQtVHlwZSIsICJ0ZXh0L2h0bWwiXSwKICAgIHsieC1hbXotbWV0YS11dWlkIjogIjE0MzY1MTIzNjUxMjc0In0sCiAgICBbInN0YXJ0cy13aXRoIiwgIiR4LWFtei1tZXRhLXRhZyIsICIiXSwKICBdCn0K" />
<input type="hidden" name="Signature" value="QDMIU8m3GZ1KHPAKphYIvvIr0bE=" />
<!-- The file must be the last meaningful element in the request; other elements after this will be ignored -->
Post: <textarea name="file" cols="60" rows="10">
Your blog post would go here...
</textarea><br />
<input type="submit" name="submit" value="Post to Amazon S3" />
</form>

Sample Request

Requests are typically sent from a Web Browser; here is a sample request. This request assumes that the image uploaded is 117,108 bytes; the image data is not included.

POST / HTTP/1.1
Host: johnsmith.s3.amazonaws.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Content-Type: multipart/form-data; boundary=---------------------------13111570113246
Content-Length: 118954

-----------------------------13111570113246
Content-Disposition: form-data; name="key"

user/eric/MyPicture.jpg
-----------------------------13111570113246
Content-Disposition: form-data; name="x-amz-meta-tag"

Some,Tag,For,Picture
-----------------------------13111570113246
Content-Disposition: form-data; name="acl"

public-read
-----------------------------13111570113246
Content-Disposition: form-data; name="x-amz-meta-uuid"

14365123651274
-----------------------------13111570113246
Content-Disposition: form-data; name="redirect"

http://johnsmith.s3.amazonaws.com/successful_upload.html
-----------------------------13111570113246
Content-Disposition: form-data; name="AWSAccessKeyId"

15B4D3461F177624206A
-----------------------------13111570113246
Content-Disposition: form-data; name="Policy"

eyAiZXhwaXJhdGlvbiI6ICIyMDA3LTEyLTAxVDEyOjAwOjAwLjAwMFoiLAogICJjb25kaXRpb25zIjogWwogICAgeyJidWNrZXQiOiAiam9obnNtaXRoIiB9LAogICAgWyJzdGFydHMtd2l0aCIsICIka2V5IiwgInVzZXIvZXJpYy8iXSwKICAgIHsiYWNsIjogInB1YmxpYy1yZWFkIiB9LAogICAgeyJyZWRpcmVjdCI6ICJodHRwOi8vam9obnNtaXRoLnMzLmFtYXpvbmF3cy5jb20vc3VjY2Vzc2Z1bF91cGxvYWQuaHRtbCIgfSwKICAgIFsic3RhcnRzLXdpdGgiLCAiJENvbnRlbnQtVHlwZSIsICJpbWFnZS8iXSwKICAgIHsieC1hbXotbWV0YS11dWlkIjogIjE0MzY1MTIzNjUxMjc0In0sCiAgICBbInN0YXJ0cy13aXRoIiwgIiR4LWFtei1tZXRhLXRhZyIsICIiXSwKICBdCn0K
-----------------------------13111570113246
Content-Disposition: form-data; name="Signature"

2qCp0odXe7A9IYyUVqn0w2adtCA=
-----------------------------13111570113246
Content-Disposition: form-data; name="file"; filename="Picture.jpg"
Content-Type: image/jpeg

[Content of the image]
-----------------------------13111570113246
Content-Disposition: form-data; name="submit"

Upload to Amazon S3
-----------------------------13111570113246--

Sample Response

HTTP/1.1 303 Redirect
x-amz-request-id: 1AEE782442F35865
x-amz-id-2: cxzFLJRatFHy+NGtaDFRR8YvI9BHmgLxjvJzNiGGICARZ/mVXHj7T+qQKhdpzHFh
Content-Type: application/xml
Date: Wed, 14 Nov 2007 21:21:33 GMT
Connection: close
Location: http://johnsmith.s3.amazonaws.com/successful_upload.html?bucket=johnsmith&key=user/eric/MyPicture.jpg&etag=39d459dfbc0faabbb5e179358dfb94c3
Server: AmazonS3

Limitations

We are not allowing other requests that use the REST PUT verb via POST at this time. For example, bucket creates, changing ACLs, and End-User Logging must all be done via existing REST or SOAP APIs.

The file form field must be the last meaningful element in the form being uploaded. All fields following the file will be ignored.

POST shares the same file size limitations as PUT, however, it should be noted that uploading extremely large files via a web browser may not yield the best customer experience.

Pricing

A POST request will be charged in the same way that a PUT request is presently charged.