Category Archives: Python

One of the welcome additions to Amazon’s AWS offerings is a simplified server provisioning service to compete directly with Digital Ocean called Lightsail.  Lightsail provides a nicer web UI for launching instances, many quick launch options for common apps like WordPress or GitLab and simplified billing (yay!).  With Lightsail you don’t need to pre-pay for Reserved Instances to get a good price on an EC2 server.

Dokku is mini heroku you can run on your own servers.  It uses the same buildpacks that Heroku does to enable git push deployments. By building on top of Docker a collection of available dokku plugins make it easy to start up databases, caching or tie in other services.  In this tutorial I add Postgresql and get an SSL cert using Let’s Encrypt

Together, Lightsail and Dokku create an easy way to manage your application deployment on an inexpensive server.

Get started on Lightsail by starting up a new virtual server:

And then selecting an Ubuntu Image:

There’s a spot here for ‘Add launch script’ where you can drop in these commands to automatically install dokku on first boot:

sudo DOKKU_TAG=v0.7.2 bash

Give it a name and press Create to start booting up the server. You should be able to SSH to the new server very quickly though you can connect before dokku and package updates have been applied (it’ll take a couple minutes for the dokku command to become available)

After a couple of minutes have passed and things are installed and running visit your server in a web browser:

For the public key you’ll want to grab the key on your computer.  if you have linux or macOS you can grab the contents of ~/.ssh/  If you need to generate a key there’s a good How-To on Github about generating them.

Set the hostname you’ll use for the server if you have one and Finish Setup.

Next step is to SSH to the server and fiddle with it there using the private key you can download from Lightsail:

ssh -i LightsailDefaultPrivateKey.pem ubuntu@<YOUR PUBLIC IP ADDRESS>

And create the app you will be deploying:

dokku apps:create your-app

Add a postgres database (there are other great plugins available for dokku too)

sudo dokku plugin:install
dokku postgres:create database-name
dokku postgres:link database-name your-app

Now, back to your local app add a git remote to this new server and deploy it:

git remote add dokku dokku@<PUBLIC IP OR HOSTNAME> your-app
git push dokku master

If that is successful then the project should be visible online. Yay!

Then there are some next steps to help complete the app. Set any environment variables you need for the app:

dokku config:set your-app ENV=prod

You can install an SSL cert using Let’s Encrypt very easily:

sudo dokku plugin:install
dokku letsencrypt your-app

You can configure some pre and post deploy hooks inside the app.yaml file in your project repository to run checks or execute database migrations.

That’s about it! git push deploys to update your project whenever you want.

Finished the build for the Fermenter over the weekend. Here’s a video going over the physical components for the project:

The chamber itself is based on the Son of a Fermentation Chiller. with the only modification being that I made mine a bit smaller than their plans.

The django based web app is all open-source. The code & instructions for building it are available on GitHub.

The Pi will run a django based web interface that will provide simple and minimal control and logs of the brewing process.


The electronics are fairly simple, and as soon as I figure out how to generate a circuit diagram I’ll add instructions for how to solder together the prototype board.

Unfortunately Apple doesn’t have a nice easy API for downloading your revenue numbers from iAds. The only way around that right now is screen scraping the information, or manually downloading the reports.

Thanks to the awesome python requests library I was able to put something together in just 50 lines of code.

It’s a function you can integrate with your project to download daily iAd report files.

It is also simple enough that it should be relatively robust against future website changes by Apple. (hopefully)

If you decide to use this I would recommend storing your account’s publisherID to avoid having to get it each time which will speed it up.

(this code is also on github:

import requests
import re, os, datetime
url_base = ''
signin_url = url_base % '/WebObjects/iTunesConnect.woa'
download_csv_url = ''
def getPublisherID(session):
    publisher ID is a number that is buried deep... lets hope
    this code doesn't break often
    headers = {'Content-type': 'text/x-gwt-rpc; charset=UTF-8'}
    iad_service_url= ''
    body = '5|0|4||E8DB97D87973D76A7C9096DCF8A83BB5|com.qwapi.portal.itc.client.rpc.IITCStartupService|getStartupData|1|2|3|4|0|'
    r =, data=body, headers=headers)
    endpos = r.text.find('com.qwapi.portal.client.rpc.dto.UserDTO')
    endpos = r.text.find('"', endpos-4)
    startpos = r.text.rfind('"', endpos-20, endpos-4)
    pubID = r.text[startpos+1:endpos]
    if not int(pubID):
        raise Exception("Failed to get publisher ID")
    return pubID
def downloadiAdFiles(appleId, password, publisherId=None, outputDirectory='.', daysToDownload=14, outputFormat='iAd_D_%m-%d-%Y.txt'):
    session = requests.session()  #session used to maintain cookies etc.
    #get signin page to find url for posting signin credentials
    r = session.get(signin_url)
    match ='" action="(.*)"', r.text)
    #login to ITC
    params = {'theAccountName':appleId,
    r = %, params=params, headers={'Content-Length':'0'})
    r = session.get('')
    if publisherId is None:
        publisherId = getPublisherID(session)
    reportDates = [ - datetime.timedelta(i + 1 ) for i in range(daysToDownload)]
    filenames = []
    for downloadReportDate in reportDates:
        filename = os.path.join(outputDirectory, downloadReportDate.strftime(outputFormat))
        if (os.path.exists(filename)):
        dateString = downloadReportDate.strftime('%m/%d/%Y')
        params = {'pageName': 'app_homepage',
            'searchTerms':'Search Apps',
        r = session.get(download_csv_url, params=params)
        if r.status_code != 200:
            raise Exception("Script failed to dowload - check login credentials & publisher ID")
        with open(filename, 'wb') as f:
    return filenames

I’ve got a couple of projects on the go right now, but one of the more interesting ones is a script that walks a user through the process of cloning an Xcode project and do some variable substitutions. With the final result being an iOS app that is ready to submit to the App Store.

The reason for this project is that one of my app platforms lends itself to being very niche specific and it makes sense to go for a bunch of targeted niche apps rather than a few all encompassing ones. To that end I wanted a way to quickly take the app platform to as many niches where it makes sense. To do it in a reasonable amount of time required some automation.

The approach I’ve taken is to generate a master app. The code in the master app has some placeholder variables which I can easily do search and replace. Along with that I found a script to copy and rename an xcode project which deals with the xml files and folder names and such. The script to rename the project itself is from

One of the helpful things about this script is that by using PIL I can easily scale the images in the project down from the retina source graphics. I can walk myself through all the tedious stuff like getting provisioning profiles, setting up Facebook and Twitter keys so that I don’t forget to do things or do them in the wrong order.

It would be easy to see how this could be extended. Perhaps by adding a few more variable substitutions it would be easy to add/remove various features by setting pre-processor flags or tweak behaviours like, speeds, timeouts etc. Wrapping it all into a nice GUI would make this whole thing into a drag and drop app creator tool that anyone could use.

#!/usr/bin/env python
# Written By Matt Warren, Halotis Inc.
# This script is a wizard to walk you through duplicating an Xcode project that
# follows the patterns I've done for my previous projects.  It's not meant to
# be completely generic.
import subprocess
import zipfile
from PIL import Image
import glob, os
def main():
    the script asks questions and then unzips, renames and runs some variable substitions
    on the AppDelegate.h file.  the final product should be ready to test and submit
    #These are in no particular order...
    print """
Fear not about making mistakes.  All these values can be easily fixed in the final project.
Just fill things out as best you can and fix missing values during testing.
This script requires a file called which contains a project called SampleProject
The process is:
    1. create/rename the Xcode project
    2. Create the art assets and put them in place
    3. Tie the app to various services and set up in iTunesConnect
    4. replace constants in AppDelegate.h
    5. Test
    6. Upload binary to Apple
    7. Profit!!
    new_app_name = raw_input('Please enter New App Name:')
    new_proj_name = new_app_name
    raw_input('Renaming SampleProject to %s (enter to continue)' % new_proj_name)
    print 'Thanks, unzipping and renaming base project...'
    z = zipfile.ZipFile('')
    result =['./', 'SampleProject', new_proj_name])
    if result != 0:
    print 'On to the Art...'
    garbage = raw_input(' ## OVERWRITE %s/Default@2x.png (enter to continue) ##' % new_proj_name)
    image ='%s/Default@2x.png' % new_proj_name)
    small_image = image.resize((image.size[0] / 2, image.size[1] / 2), Image.ANTIALIAS)'%s/Default.png' % new_proj_name)
    garbage = raw_input(' ## OVERWRITE %s/Default~ipad@2x.png (enter to continue) ##' % new_proj_name)
    image ='%s/Default~ipad@2x.png' % new_proj_name)
    small_image = image.resize((image.size[0] / 2, image.size[1] / 2), Image.ANTIALIAS)'%s/Default~ipad.png' % new_proj_name)
    garbage = raw_input('Create Icons using PhotoShop template, (enter to continue)')
    print 'variable subsitutions are next:   (avoid using anything with quotes)'
    header_content = open('%s/Classes/AppDelegate.h' % new_proj_name, 'r').read()
    header_content.replace('<app_name>', new_app_name)
    website = raw_input('Please enter App Website:')
    header_content.replace('<website>', website)
    print ''
    apple_app_id = raw_input('Apple App ID:')
    header_content.replace('<app_id>', apple_app_id)
    print ''
    facebook_app_id = raw_input('Facebook App ID')
    header_content.replace('<facebook_app>', facebook_app_id)
    print ''
    twitter_consumer_key = raw_input('Twitter Consumer Key (use website above for callback)')
    header_content.replace('<twitter_consumer>', twitter_consumer_key)
    twitter_secret = raw_input('Twitter Secret')
    header_content.replace('<twitter_secret>', twitter_secret)
    print 'configure Facebook App ID in URLS - DO THIS MANUALLY!!'
    print 'Writing new App Delegate Header File'
    with open('%s/Classes/AppDelegate.h' % new_proj_name, 'w') as header:
if __name__ == '__main__':

Recently I’ve been building out my iPhone App server to provide a business dashboard with all the relevant services and numbers that I care about available at a glance. It avoids me having to sign in and out of many different sites to get the information and makes it easier to push things together – for example charting both Admob and iAd data on the same graph.

Thankfully the web is becoming more programmable every week and these things are becoming easier to put together quickly.

This is a chart I built last night to display the downloads and updates across all my apps for the past 31 days:

You can see the jumps in downloads that correspond to when I released updates to iTunes.

With these sorts of things I’m finding that there is a tipping point. If the custom page I have created is only 90% as good as going to the original source then I’ll just opt to login there but once it becomes as good as or better than that you’ll quickly forget about the 10 different logins you needed to get all those numbers.

Being in charge of it is even better. I use iAd and Admob for advertising and can pull those numbers in and compare them appropriately. On the same page I display data from Apple, Google, Linkshare, as well as numbers I collect myself such as traffic, link clicks and ad impressions. I only have to login to iTunes Connect to release new Apps.

I will continue open sourcing the components for this system over the next few weeks.

One of the ways to make money on the itunes store is to sign up for the itunes affiliate program. Linkshare runs the program in the USA and they will give you a 5% commission on all sales that you refer. It works through cookie based tracking that is valid for 72 hours… Meaning that if you follow one of my links (even to a free download) and then buy something 2 days later then I get credited for the referral and make a few cents.

Linkshare links currently make up about 15% of the revenue in my app business. It’s an extra little bit of money that takes very little effort to add to the bottom line.

I’m building up my back end platform for reporting on all the various numbers I get for the business and putting them in one place. This past friday I got an email about the new web services REST api for Linkshare which I can use to generate various reports. It took just a couple hours to put together a django app that can download the month to date numbers and store them in the database for reporting.

There might be a handful of people out there interested in using this sort of thing so I put the code up on github.


Fabric is a pretty awesome tool for deploying projects. But it turns out that it’s also pretty awesome for lots of other stuff.

At PyCon 2012 there was a talk given by Ricardo Kirkner (which you can watch here) that inspired me to play around with fabric in some new ways.

It’s possible to use fabric as a wrapper around the standard django ./ script, to help setting up virtualenvs and install packages. Using fabric to script around these things means that there are fewer tools that new developers will need to get set up and know how to use. Scripts that normally might have been loose bash files can now be collected, organized and documented.

I’m currently working on a large django project with 4 other developers who are new to python and django. Getting everyone’s development environment working was a big pain since there were multiple platforms (Mac and Linux) and different configurations for base packages. If I had thought of this sooner I might have been able to create a reliable fabric script so that “easy_install fabric; fab init_project” got them from zero to running django app.

There’s also several oneliners that I run fairly regularly which can be saved in and called much easier. For example:

def clean():
    local('find . -name "*\.pyc" -exec rm -r {} \;')


$ fab clean

will clear out any .pyc files in the project.

It’s also possible to manage the virtualenv environment through fabric:

VIRTUALENV = '.virtualenv/'
def setup_virtualenv():
    created = False
    virtual_env = os.environ.get('VIRTUAL_ENV', None)
    if virtual_env is None:
        if not os.path.exists(VIRTUALENV):
            created = True
        virtual_env = VIRTUALENV
    env.virtualenv = os.path.abspath(virtual_env)
    return created
def _activate_virtualenv():
    activate_this = os.path.abspath("%s/bin/" % env.virtualenv)
    execfile(activate_this, dict(__file__=activate_this))
def _create_virtualenv(clear=False):
    if not os.path.exists(VIRTUALENV) or clear:
        args = '--no-site-packages --distribute --clear'
        local("%s /usr/local/bin/virtualenv %s %s" % (sys.executable, args, VIRTUALENV), capture=False)
def virtualenv_local(command, capture=True):
    prefix = ''
    virtual_env = env.get('virtualenv', None)
    if virtual_env:
        prefix = ". %s/bin/activate && " % virtual_env
    command = prefix + command
    return local(command, capture=capture)
def manage(command, *args):
    virtualenv_local("python {0} {1}".format(command, ' '.join(args)), capture=False,)
def runserver(*args):
    manage('runserver', *args)

These functions let you create the virtualenv and run commands in the virtualenv (without having manually activated it). virtualenv_local wraps the call to fabric’s “local” function and sources the activate script before launching the command specified. the manage function provides a way to call django’s script, and the runserver function gives a way to use fabric to launch the server (in the virtualenv). So now the fab command can be used to consolidate both virtualenv tools and script into one document-able with a consistent command-line interface.

Technorati Tags:

I was surprised that there wasn’t already a script out there to download iAd reports from Apple’s iTunes Connect website.

Apple released a Java based command line tool to download the sales reports for Apps but neglected to provide something similar for iAd publishers. Some Googling around I was further surprised that I couldn’t find any 3rd party scripts to download this data.

I did however find a python script called which is a web scraper that downloads the sales reports. It handled a lot of the nasty http cookies and login stuff that is usually very tricky to do with a script. I modified that script and extended it with a function to download daily iAd reports.

The rather fascinating thing was just how complex the single page of the iAd publisher dashboard is. It was built with GWT which is perhaps the worst thing ever developed. It produced a webpage that contains 40,000+ lines of javascript and all it does is draw a few graphs. The code was terribly convoluted and nearly impossible to reverse engineer. But 6 frustrating hours later I was able to download the files I wanted.

I don’t wish this sort of struggle on anyone so I’m making the code available.

I am continually in awe of the power of Python code. The readability of the language combined with the lack of magic, and the massive number of easily obtainable libraries out there make getting things done insanely fast. It is the only language where I consistently experience writing code that works the first time.

Last night I added a geo-location aware link redirector with click tracking to my django web app. It took about 1 hour to code while I was listening to a panel discussion. Within that hour I had written, tested, added additional models, migrated the production database and deployed the new application.

At my day job, we have decided to migrate our web app from grails to django. The benefits have been numerous. The Groovy to Python conversion has resulted in significantly less lines of code, unit test runtime has dropped from 15 minutes to 3 seconds. The delay in starting the dev server dropped from 10 seconds to instantaneous. The functional style of code makes it far easier to avoid copy/pasting logic between subclasses. Python’s flexible import statement allows us to structure the code so that it makes more sense.

If you’re not using Python. You should be.

over the last week I have been seeing an odd error usually just once or twice per day out of the 10,000+ requests per day that hit my django web app backend for the iPhone apps.

It appears as though the GET parameters get dropped and I can’t explain why. Hoping someone out there has some suggestions.

The code in my iPhone apps check in to get data from the webserver. This is new content for the app. For each request it needs to have a parameter to determine which app is requesting the data – since there are a few using the same server.

The app code calls a statically defined url string like “”

However, about one in every 10,000 requests that come though create an exception because that app_id is not present.

The other bits seem to be on the mark. They don’t seem to be coming from a web browser manually.

So the most likely culprit is either that some small number of users are doing something sneaky, or there is a bug somewhere.

I’m curious if anyone has seen this issue before, or knows what might be causing it.

Solved – the problem was due to the string encoding on iOS side of things. Needed to enforce a utf8 encoding for all urls.