UPDATED : APRIL 2017

Amazon S3 Buckets are a cheap way to store your staticfiles and media files. It also seems like the easiest way when serving a Django App. We are going to go over the basic steps to set them up. They have been derived mostly from this amazing article with little modifications to make the process a little easier as the AWS console has been updated and to understand the process a little better.

After you have signed up with aws.amazon.com the first step would be to create a user using the IAM service. This way you can restrict the access to the bucket. Each user will have its own access id and key. To do this :

  • From the “Services” dropdown select “IAM”
  • Click on “Users” and select “Add User”.
  • Type the name of the user and make sure that “Programmatic access” is selected under “Access type” as you will need this to provide upload access for your S3 bucket. Click on “Next : Permissions”.
  • Select “Attatch existing policies directly”.
  • From the list that appears select “AmazonS3FullAccess” and again click on “Next : Review”.
  • Click on “Create User”
  • This will create the user and generate an access id and key for the user. The id-key pair can only be downloaded at this step so do that by clicking on “Download .csv”. Keep it secret. Keep it safe. Click on “Close”
  • From the Users Dashboard click on the user you have just created. Make a note of “User ARN”. You will be using this later.

If you made it this far great work. Your user has been set up. The next box that we have to check is creating a S3 Bucket.

  • From the Services dropdown select s3. This opens up the dashboard for the S3 buckets.
  • Click on “Create Bucket”.
  • Select the region you want.
  • Give your bucket a name and click on next that brings you to the “Set Properties” tab. Click on “Next” a couple more times and then click on “Create Bucket” to create your bucket.

Next you will have to define the policy for allowing restricted access to your bucket. This is where I diverged from the article mentioned before as I wanted to understand the process more rather than just copy and paste the policy rules mentioned there.

  • Click on the name of the bucket that you just created.
  • Click on the Permissions tab located at the top.
  • Since you have to set the permissions for access, click on “Bucket Policy”.
  • On the bottom left of the modal that appears click on “AWS Policy Generator” which opens up the tool Amazon provides for quick policy generation.

Now you have to generate two policy rules. The first one is to allow our hosted website to access your files from the bucket. This will be as follows :

Select Type of Policy : S3 Bucket Policy

Effect : Allow

Principal : *      //This gives everybody access

AWS Service : Amazon S3

Actions : GetObject

Amazon Resource Name : arn:aws:s3:::<your bucket name>/* //The * at    the end siginifies that access is being given to all the files

After setting the values as mentioned above click on “Add Statement”. Do not click on “Generate Policy” yet as you have to create a policy to allow the Django application to put files into the bucket on deployment. The policy will be as follows :

Select Type of Policy : S3 Bucket Policy

Effect : Allow

Principal : <User ARN> //This is the user arn that you kept a note of earlier.

AWS Service : Amazon S3

Actions : * //I gave full access, though I think GetObject,PutObject will be better. Will try it out soon. 

Amazon Resource Name : arn:aws:s3:::<your bucket name>/*,arn:aws:s3:::<your bucket name>  // Gives full access to buckets and its contents.

After setting the values mentioned above click on “Add Statement” and then “Generate Policy” . This will show you the policy which you can copy, paste in the dialogue box on S3 dashboard and click on Save.

The last step for setting up the bucket access is providing the application hosted on Heroku, in my case, access to the bucket content. This can be done by setting up the CORS configuration. To do this click onthe “ CORS Configuration” tab and click on Save. The default configuration wiill suffice.

You have the bucket access set up. Now all you have to do is set up your Django application to access the bucket which is fairly easy. The first step to achieve the goal will be to install Boto3 and DjangoStorages. Nifty APIs to make working with S3 buckets easy. This can be easily done using pip.

pip install django-storages boto3
pip freeze >> requirements.txt

and add “storages” to the list of INSTALLED_APPS in the settings.py file.

INSTALLED_APPS = (
          ...,
          'storages',
     )

Next you have to add the following settings to the settings.py file in your application.

//set S3 as the place to store your files.
DEFAULT_FILE_STORAGE = “storages.backends.s3boto3.S3Boto3Storage”
STATICFILES_STORAGE = “storages.backends.s3boto3.S3Boto3Storage”

AWS_ACCESS_KEY_ID = os.environ.get(“AWS_ACCESS_KEY_ID”, “”)
AWS_SECRET_ACCESS_KEY = os.environ.get(“AWS_SECRET_ACCESS_KEY”, “”)
AWS_STORAGE_BUCKET_NAME = os.environ.get(“AWS_STORAGE_BUCKET_NAME”, “”)

AWS_QUERYSTRING_AUTH = False //This will make sure that the file URL does not have unnecessary parameters like your access key.

AWS_S3_CUSTOM_DOMAIN = AWS_STORAGE_BUCKET_NAME + ‘.s3.amazonaws.com’

#static media settings
STATIC_URL = ‘https://' + AWS_STORAGE_BUCKET_NAME + ‘.s3.amazonaws.com/’
MEDIA_URL = STATIC_URL + ‘media/’
STATICFILES_DIRS = ( os.path.join(BASE_DIR, “static”), )
STATIC_ROOT = ‘staticfiles’
ADMIN_MEDIA_PREFIX = STATIC_URL + ‘admin/’

STATICFILES_FINDERS = (
‘django.contrib.staticfiles.finders.FileSystemFinder’,
‘django.contrib.staticfiles.finders.AppDirectoriesFinder’,
)

It is good security practice to store your settings values in the environment rather than writing them down directly in the settings file.

We are almost there. Make sure your main urls.py file does include the url configuration for your media files

from django.conf import settings

urlpatterns = [

] + static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

And as a last step there should be a folder called static in the root of your application (same level as the manage.py file). Git does not allow you to commit empty folders so you can inlcude a readme file in there to commit the folder.

Also if you are using Heroku before pushing your code use

heroku config:set DISABLE_COLLECTSTATIC=1

to disable collectstatic running automatically as your static folder is not yet present on the server and an error will be thrown regarding the same. You can run it manually using

heroku run python manage.py collectstatic --noinput

One important point before you go on this adventure. At the moment all the media files ( files that are uploaded by the user) used in our models have the property upload_to set for them and as a result don’t overwrite the static files. Else I would suggest reading the article mentioned above and going over the strategies mentioned by the author to mitigate the issue. We will probably implement it in the future if and when we have a product market fit and have time to worry about it.

So the long, arduous (?) and fun process is over and you are ready to serve static and media files from the S3 bucket. If you have any tips for doing this in a better way/spot a mistake do drop me a message and I will be more than happy to shout you a coffee!

Check out some of my apps if you like :

‎Vocab Watch
‎Improve your vocabulary by using this easy to use phone and watch app. Using Vocab Watch you can easily keep a track of…itunes.apple.com

‎Handwrite Me
‎Make your messages personal by sending them in your own handwriting. Using Handwrite Me you can create a personalised…itunes.apple.com

‎Lookie - Dictionary Made Easy
‎Lookie is the dictionary you were "looking" ( yes we are punny) for if you love the printed word. Lookie brings…itunes.apple.com