Category Archives: Python

Fermenter Built

Finished the build for the Fermenter over the weekend. Here’s a video going over the physical components for the project:

The chamber itself is based on the Son of a Fermentation Chiller. with the only modification being that I made mine a bit smaller than their plans.

The django based web app is all open-source. The code & instructions for building it are available on GitHub. https://github.com/mfwarren/Fermenter

The Pi will run a django based web interface that will provide simple and minimal control and logs of the brewing process.

Screenshot_2013-05-01_9_38_PM-3

The electronics are fairly simple, and as soon as I figure out how to generate a circuit diagram I’ll add instructions for how to solder together the prototype board.

iAd Download Script

Unfortunately Apple doesn’t have a nice easy API for downloading your revenue numbers from iAds. The only way around that right now is screen scraping the information, or manually downloading the reports.

Thanks to the awesome python requests library I was able to put something together in just 50 lines of code.

It’s a function you can integrate with your project to download daily iAd report files.

It is also simple enough that it should be relatively robust against future website changes by Apple. (hopefully)

If you decide to use this I would recommend storing your account’s publisherID to avoid having to get it each time which will speed it up.

(this code is also on github: https://github.com/mfwarren/iAdDownload)

import requests
import re, os, datetime
 
url_base = 'https://itunesconnect.apple.com%s'
signin_url = url_base % '/WebObjects/iTunesConnect.woa'
download_csv_url = 'https://iad.apple.com/itcportal/generatecsv'
 
def getPublisherID(session):
    """
    publisher ID is a number that is buried deep... lets hope
    this code doesn't break often
    """
    headers = {'Content-type': 'text/x-gwt-rpc; charset=UTF-8'}
    iad_service_url= 'https://iad.apple.com/itcportal/service/startup?requestId=GUEST_1331628761939_0__1331628761950__null'
    body = '5|0|4|https://iad.apple.com/itcportal/itcportal/|E8DB97D87973D76A7C9096DCF8A83BB5|com.qwapi.portal.itc.client.rpc.IITCStartupService|getStartupData|1|2|3|4|0|'
    r = session.post(iad_service_url, data=body, headers=headers)
    endpos = r.text.find('com.qwapi.portal.client.rpc.dto.UserDTO')
    endpos = r.text.find('"', endpos-4)
    startpos = r.text.rfind('"', endpos-20, endpos-4)
    pubID = r.text[startpos+1:endpos]
    if not int(pubID):
        raise Exception("Failed to get publisher ID")
    return pubID
 
 
def downloadiAdFiles(appleId, password, publisherId=None, outputDirectory='.', daysToDownload=14, outputFormat='iAd_D_%m-%d-%Y.txt'):
 
    session = requests.session()  #session used to maintain cookies etc.
 
    #get signin page to find url for posting signin credentials
    r = session.get(signin_url)
    match = re.search('" action="(.*)"', r.text)
 
    #login to ITC
    params = {'theAccountName':appleId,
            'theAccountPW':password,
            '1.Continue.x':'10',
            '1.Continue.y':'10',
            'theAuxValue':''}
    r = session.post(url_base % match.group(1), params=params, headers={'Content-Length':'0'})
    r = session.get('https://iad.apple.com/itcportal')
 
    if publisherId is None:
        publisherId = getPublisherID(session)
 
    reportDates = [datetime.date.today() - datetime.timedelta(i + 1 ) for i in range(daysToDownload)]
 
    filenames = []
    for downloadReportDate in reportDates:
        filename = os.path.join(outputDirectory, downloadReportDate.strftime(outputFormat))
        if (os.path.exists(filename)):
            continue
 
        dateString = downloadReportDate.strftime('%m/%d/%Y')
        params = {'pageName': 'app_homepage',
            'dashboardType':'publisher',
            'publisherId':publisherId,
            'dateRange':'customDateRange',
            'searchTerms':'Search Apps',
            'adminFlag':'false',
            'fromDate':dateString,
            'toDate':dateString,
            'dataType':'byName'
        }
        r = session.get(download_csv_url, params=params)
        if r.status_code != 200:
            raise Exception("Script failed to dowload - check login credentials & publisher ID")
 
        with open(filename, 'wb') as f:
            f.write(r.content)
        filenames.extend([filename])
 
    return filenames

Clone an Xcode Project

I’ve got a couple of projects on the go right now, but one of the more interesting ones is a script that walks a user through the process of cloning an Xcode project and do some variable substitutions. With the final result being an iOS app that is ready to submit to the App Store.

The reason for this project is that one of my app platforms lends itself to being very niche specific and it makes sense to go for a bunch of targeted niche apps rather than a few all encompassing ones. To that end I wanted a way to quickly take the app platform to as many niches where it makes sense. To do it in a reasonable amount of time required some automation.

The approach I’ve taken is to generate a master app. The code in the master app has some placeholder variables which I can easily do search and replace. Along with that I found a script to copy and rename an xcode project which deals with the xml files and folder names and such. The script to rename the project itself is from http://rudifa.wordpress.com/2011/07/23/clonexcodeproject-sh/.

One of the helpful things about this script is that by using PIL I can easily scale the images in the project down from the retina source graphics. I can walk myself through all the tedious stuff like getting provisioning profiles, setting up Facebook and Twitter keys so that I don’t forget to do things or do them in the wrong order.

It would be easy to see how this could be extended. Perhaps by adding a few more variable substitutions it would be easy to add/remove various features by setting pre-processor flags or tweak behaviours like, speeds, timeouts etc. Wrapping it all into a nice GUI would make this whole thing into a drag and drop app creator tool that anyone could use.

#!/usr/bin/env python
 
#######################################
#
# Written By Matt Warren, Halotis Inc.
# http://halotis.com
# http://mattwarren.co
#
# This script is a wizard to walk you through duplicating an Xcode project that
# follows the patterns I've done for my previous projects.  It's not meant to
# be completely generic.
#
#######################################
 
import subprocess
import zipfile
from PIL import Image
import glob, os
 
def main():
    """
    the script asks questions and then unzips, renames and runs some variable substitions
    on the AppDelegate.h file.  the final product should be ready to test and submit
    """
 
    #These are in no particular order...
    print """
################################################################################
 
MATTS SUPER AWESOME PROJECT MULTIPLIER WIZARD!!!
 
Fear not about making mistakes.  All these values can be easily fixed in the final project.
Just fill things out as best you can and fix missing values during testing.
 
This script requires a file called SampleProject.zip which contains a project called SampleProject
 
The process is:
    1. create/rename the Xcode project
    2. Create the art assets and put them in place
    3. Tie the app to various services and set up in iTunesConnect
    4. replace constants in AppDelegate.h
    5. Test
    6. Upload binary to Apple
    7. Profit!!
################################################################################
    """
    new_app_name = raw_input('Please enter New App Name:')
    new_proj_name = new_app_name
 
    raw_input('Renaming SampleProject to %s (enter to continue)' % new_proj_name)
    print 'Thanks, unzipping and renaming base project...'
 
    z = zipfile.ZipFile('SampleProject.zip')
    z.extractall('./')
 
    result = subprocess.call(['./RenameXCodeProj.sh', 'SampleProject', new_proj_name])
 
    if result != 0:
        print 'FAILED TO RUN RENAME SCRIPT'
        return
 
    print 'On to the Art...'
 
    garbage = raw_input(' ## OVERWRITE %s/Default@2x.png (enter to continue) ##' % new_proj_name)
    image = Image.open('%s/Default@2x.png' % new_proj_name)
    small_image = image.resize((image.size[0] / 2, image.size[1] / 2), Image.ANTIALIAS)
    small_image.save('%s/Default.png' % new_proj_name)
 
    garbage = raw_input(' ## OVERWRITE %s/Default~ipad@2x.png (enter to continue) ##' % new_proj_name)
    image = Image.open('%s/Default~ipad@2x.png' % new_proj_name)
    small_image = image.resize((image.size[0] / 2, image.size[1] / 2), Image.ANTIALIAS)
    small_image.save('%s/Default~ipad.png' % new_proj_name)
 
    garbage = raw_input('Create Icons using PhotoShop template, (enter to continue)')
 
    print 'variable subsitutions are next:   (avoid using anything with quotes)'
 
    header_content = open('%s/Classes/AppDelegate.h' % new_proj_name, 'r').read()
    header_content.replace('<app_name>', new_app_name)
 
    website = raw_input('Please enter App Website:')
    header_content.replace('<website>', website)
 
    print 'http://itunesconnect.apple.com'
    apple_app_id = raw_input('Apple App ID:')
    header_content.replace('<app_id>', apple_app_id)
 
    print 'https://developers.facebook.com/'
    facebook_app_id = raw_input('Facebook App ID')
    header_content.replace('<facebook_app>', facebook_app_id)
 
    print 'http://dev.twitter.com/apps/new'
    twitter_consumer_key = raw_input('Twitter Consumer Key (use website above for callback)')
    header_content.replace('<twitter_consumer>', twitter_consumer_key)
 
    twitter_secret = raw_input('Twitter Secret')
    header_content.replace('<twitter_secret>', twitter_secret)
 
    print 'configure Facebook App ID in URLS - DO THIS MANUALLY!!'
 
    print 'Writing new App Delegate Header File'
    with open('%s/Classes/AppDelegate.h' % new_proj_name, 'w') as header:
        header.write(header_content)
 
if __name__ == '__main__':
    main()
</twitter_secret></twitter_consumer></facebook_app></app_id></website></app_name>

iTunes Download Stats

Recently I’ve been building out my iPhone App server to provide a business dashboard with all the relevant services and numbers that I care about available at a glance. It avoids me having to sign in and out of many different sites to get the information and makes it easier to push things together – for example charting both Admob and iAd data on the same graph.

Thankfully the web is becoming more programmable every week and these things are becoming easier to put together quickly.

This is a chart I built last night to display the downloads and updates across all my apps for the past 31 days:

You can see the jumps in downloads that correspond to when I released updates to iTunes.

With these sorts of things I’m finding that there is a tipping point. If the custom page I have created is only 90% as good as going to the original source then I’ll just opt to login there but once it becomes as good as or better than that you’ll quickly forget about the 10 different logins you needed to get all those numbers.

Being in charge of it is even better. I use iAd and Admob for advertising and can pull those numbers in and compare them appropriately. On the same page I display data from Apple, Google, Linkshare, as well as numbers I collect myself such as traffic, link clicks and ad impressions. I only have to login to iTunes Connect to release new Apps.

I will continue open sourcing the components for this system over the next few weeks.

django-linkshare

One of the ways to make money on the itunes store is to sign up for the itunes affiliate program. Linkshare runs the program in the USA and they will give you a 5% commission on all sales that you refer. It works through cookie based tracking that is valid for 72 hours… Meaning that if you follow one of my links (even to a free download) and then buy something 2 days later then I get credited for the referral and make a few cents.

Linkshare links currently make up about 15% of the revenue in my app business. It’s an extra little bit of money that takes very little effort to add to the bottom line.

I’m building up my back end platform for reporting on all the various numbers I get for the business and putting them in one place. This past friday I got an email about the new web services REST api for Linkshare which I can use to generate various reports. It took just a couple hours to put together a django app that can download the month to date numbers and store them in the database for reporting.

There might be a handful of people out there interested in using this sort of thing so I put the code up on github.

LinkShare_336x280

Fabric For Development

Fabric is a pretty awesome tool for deploying projects. But it turns out that it’s also pretty awesome for lots of other stuff.

At PyCon 2012 there was a talk given by Ricardo Kirkner (which you can watch here) that inspired me to play around with fabric in some new ways.

It’s possible to use fabric as a wrapper around the standard django ./manage.py script, to help setting up virtualenvs and install packages. Using fabric to script around these things means that there are fewer tools that new developers will need to get set up and know how to use. Scripts that normally might have been loose bash files can now be collected, organized and documented.

I’m currently working on a large django project with 4 other developers who are new to python and django. Getting everyone’s development environment working was a big pain since there were multiple platforms (Mac and Linux) and different configurations for base packages. If I had thought of this sooner I might have been able to create a reliable fabric script so that “easy_install fabric; fab init_project” got them from zero to running django app.

There’s also several oneliners that I run fairly regularly which can be saved in fabfile.py and called much easier. For example:

def clean():
    local('find . -name "*\.pyc" -exec rm -r {} \;')

Now

$ fab clean

will clear out any .pyc files in the project.

It’s also possible to manage the virtualenv environment through fabric:

VIRTUALENV = '.virtualenv/'
def setup_virtualenv():
    created = False
    virtual_env = os.environ.get('VIRTUAL_ENV', None)
    if virtual_env is None:
        if not os.path.exists(VIRTUALENV):
            _create_virtualenv()
            created = True
        virtual_env = VIRTUALENV
    env.virtualenv = os.path.abspath(virtual_env)
    _activate_virtualenv()
    return created
 
def _activate_virtualenv():
    activate_this = os.path.abspath("%s/bin/activate_this.py" % env.virtualenv)
    execfile(activate_this, dict(__file__=activate_this))
 
def _create_virtualenv(clear=False):
    if not os.path.exists(VIRTUALENV) or clear:
        args = '--no-site-packages --distribute --clear'
        local("%s /usr/local/bin/virtualenv %s %s" % (sys.executable, args, VIRTUALENV), capture=False)
 
def virtualenv_local(command, capture=True):
    prefix = ''
    virtual_env = env.get('virtualenv', None)
    if virtual_env:
        prefix = ". %s/bin/activate && " % virtual_env
    command = prefix + command
    return local(command, capture=capture)
 
def manage(command, *args):
    virtualenv_local("python manage.py {0} {1}".format(command, ' '.join(args)), capture=False,)
 
def runserver(*args):
    manage('runserver', *args)

These functions let you create the virtualenv and run commands in the virtualenv (without having manually activated it). virtualenv_local wraps the call to fabric’s “local” function and sources the activate script before launching the command specified. the manage function provides a way to call django’s manage.py script, and the runserver function gives a way to use fabric to launch the server (in the virtualenv). So now the fab command can be used to consolidate both virtualenv tools and manage.py script into one document-able fabfile.py with a consistent command-line interface.

Technorati Tags:

iAd Report Downloader Script

I was surprised that there wasn’t already a script out there to download iAd reports from Apple’s iTunes Connect website.

Apple released a Java based command line tool to download the sales reports for Apps but neglected to provide something similar for iAd publishers. Some Googling around I was further surprised that I couldn’t find any 3rd party scripts to download this data.

I did however find a python script called appsalesdaily.py which is a web scraper that downloads the sales reports. It handled a lot of the nasty http cookies and login stuff that is usually very tricky to do with a script. I modified that script and extended it with a function to download daily iAd reports.

The rather fascinating thing was just how complex the single page of the iAd publisher dashboard is. It was built with GWT which is perhaps the worst thing ever developed. It produced a webpage that contains 40,000+ lines of javascript and all it does is draw a few graphs. The code was terribly convoluted and nearly impossible to reverse engineer. But 6 frustrating hours later I was able to download the files I wanted.

I don’t wish this sort of struggle on anyone so I’m making the code available. https://github.com/mfwarren/appdailysales

Python is like a Secret Weapon

I am continually in awe of the power of Python code. The readability of the language combined with the lack of magic, and the massive number of easily obtainable libraries out there make getting things done insanely fast. It is the only language where I consistently experience writing code that works the first time.

Last night I added a geo-location aware link redirector with click tracking to my django web app. It took about 1 hour to code while I was listening to a panel discussion. Within that hour I had written, tested, added additional models, migrated the production database and deployed the new application.

At my day job, we have decided to migrate our web app from grails to django. The benefits have been numerous. The Groovy to Python conversion has resulted in significantly less lines of code, unit test runtime has dropped from 15 minutes to 3 seconds. The delay in starting the dev server dropped from 10 seconds to instantaneous. The functional style of code makes it far easier to avoid copy/pasting logic between subclasses. Python’s flexible import statement allows us to structure the code so that it makes more sense.

If you’re not using Python. You should be.

Peculiar Puzzle – Missing GET Parameters

over the last week I have been seeing an odd error usually just once or twice per day out of the 10,000+ requests per day that hit my django web app backend for the iPhone apps.

It appears as though the GET parameters get dropped and I can’t explain why. Hoping someone out there has some suggestions.

The code in my iPhone apps check in to get data from the webserver. This is new content for the app. For each request it needs to have a parameter to determine which app is requesting the data – since there are a few using the same server.

The app code calls a statically defined url string like “http://appserver.com/app_data?app_id=aiuwbev”

However, about one in every 10,000 requests that come though create an exception because that app_id is not present.

The other bits seem to be on the mark. They don’t seem to be coming from a web browser manually.

So the most likely culprit is either that some small number of users are doing something sneaky, or there is a bug somewhere.

I’m curious if anyone has seen this issue before, or knows what might be causing it.

Edit:
Solved – the problem was due to the string encoding on iOS side of things. Needed to enforce a utf8 encoding for all urls.

Geolocation of Client with Django

My first thoughts yesterday when I started trying to add a lookup for a user’s country based on IP address was that this was going to be tricky. I figured I would have to create some models, fill them with data fixtures and do some manual queries against the database.

Turns out it was fairly trivial to do.

Django comes with some handy modules for doing it all for you. It just requires installing a C library from MaxMind and downloading their free Country data file.

To install the GeoIP MaxMind C Lib on Mac I used homebrew

brew install geoip

on the server I had to do the same thing on Linux:

sudo yum install geoip

Then I downloaded the free country data file from MaxMind and put it in my django project’s ‘geo’ directory:

$ curl -O http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
$ gunzip GeoIP.dat.gz 
$ mv GeoIP.dat <project_root>/geo/</project_root>

To finish the setup I needed to add a line to the settings.py file:

import os
PROJECT_ROOT = os.path.dirname(__file__)
GEOIP_PATH = os.path.join(PROJECT_ROOT, 'geo')

Getting the country of a connecting user was then rather simple.

from django.contrib.gis.utils import GeoIP
g = GeoIP()
country = g.country_code(request.META['REMOTE_ADDR'])

For the complete documentation on GeoIP check out the official documentation.