Django Media on Amazon S3 Fabric Deploy Script

Yesterday I moved all my static media files – css, images, and javascript over to S3 and wrote a short function for my Fabric deploy script to make it easy.  This move to S3 provided a noticeable improvement in the responsiveness of my web application.

I wrote a simple but effective performance tweak – only upload files that have changed. So it checks the modified dates on the files before uploading. This will save a bit of bandwidth costs and time during deployment.

The change I made to my settings.py was simply to point the MEDIA_URL to the bucket like this:

MEDIA_URL = 'https://s3.amazonaws.com/bucketname/media/'

And here’s the function I added to my fabfile:

from boto.s3.connection import S3Connection
from boto.s3.key import Key
import os
from stat import *
import time
 
def deploy_media():
    """Deploy the media files to S3
    """
    conn = S3Connection('ACCESS KEY ID', 'SECRET ACCESS KEY')
    bucket = conn.get_bucket('bucketname')
 
    #upload files
    for root, dirs, files in os.walk('media'):
        for f in files:
            if f.endswith('.swp') or f.startswith('.') :
                continue
            filename = root + '/' + f
            modify_time = os.stat(filename)[ST_MTIME]
            key = bucket.get_key(filename)
            if key is None:
                key = Key(bucket)
                key.key = filename
            if key.last_modified is None or time.localtime(modify_time) > time.strptime(key.last_modified, '%a, %d %b %Y %H:%M:%S %Z'):
                print filename
                fid = file(filename, 'r')
                key.set_contents_from_file(fid)
                key.set_acl('public-read')
                print 'file uploaded'
  • Steve

    Good idea. When you moved your static content over to S3, what type of response times were you getting? I know it depends on your location relative to the Amazon datacenter you chose.

    I’m curious to know how fast this is without having to use CloudFront.

  • http://halotis.com Matt

    I have found direct from S3 to be totally fine for performance. no real issues were raised from any of the performance tools like YSlow.

    seems much faster than serving directly from my own servers and 100x better than serving the files from heavily saturated shared hosting accounts.

    If it ever did become an issue it would be a pretty minor change to enable cloud front and modify the URL accordingly.