Category Archives: Software

Applying The Creative Writing Process to Software

Grabbing the best ideas from other industries and applying it to your own is a fantastic way to learn and discover new ways to do things.

I was listening to the Tim Ferriss podcast yesterday in an interview with Neil Strauss.  Two best selling authors discussing their creative process for writing a book.  It got me thinking if some of the approaches they discuss could apply to to software development -which is also a highly creative process.

Let’s break down Neil’s process a bit.

The first part is to do a lot of research.  For example, when Neil is doing an interview with a celebrity, he will read everything they’ve written, listen to ever song, devour every bit of information he can about the person.  He writes out a list of 100 questions for the person and studies it like you would for exam prep.  Finally when it’s time for the interview he puts the questions in his back pocket and forgets about them.  As a result of this level of preparation and study the interviews can flow naturally.  He has a backup of questions to draw on if the conversation stalls but usually he can riff off the bulk of knowlege he has collected to build rapport and get to the really interesting details.

With a cache of notes from research it’s time to start writing.  There will be several drafts and the goal of the first one is to “Write to the end”.  Convert the pile of notes into a story as quickly and rough as possible without self editing along the way until you get to the end of the book.  This first draft will be read by no body else.

After the first draft you are just 1/4 the way finished.  The second draft is done for the readers.  You cut out all the uninteresting pieces, ensure the story arc is solid, characters are developed enough, flow is good, and jokes are funny.  You do this by putting on the hat of a reader and ruthlessly editing your first draft.

The next step is to put on the hat of a hater.  To deflect criticism before it happens you go through it again with the perspective of someone who wants to debunk your stuff and criticize your ideas.  At this step you fact check everything, and make sure there’s nothing offensive, or at least that the counter points are acknowledged and dealt with.

Finally it goes to the editor and is test read by anyone who wants to read the manuscript. Listen to their criticism, and adjust your book to improve things and cut parts that people don’t find interesting.

After enough feedback you’ll know when to lock down the book to a final version and send it to be published.

For an average size book this whole process can take upwards of a year to complete.

There are some lessons that we can take from the best approaches to writing books and apply it to writing software.

Notice that this approach to writing a book is not Agile, and it’s not waterfall, it doesn’t involve sprints, scrums or user stories. It’s completely unique to the way most software is created.

The first part of Neil’s process is research.  You might roughly correlate this step to requirements gathering but that vastly understates the effort Neil goes through.  I have never met a developer that thoroughly masters the domain knowledge before typing anything the way Neil does.  Taking months and often years to interview people, take courses, and generally immerse yourself in the space you will eventually write about gives you skills and knowledge to be able to write “to the end” without pause.

Side note: Interruption is one of the most detrimental things to the creative process. Stopping to look something up, or ask a question of the client can waste half a day of productivity.  Agile methods promote this open communication during development… maybe there’s a better way.

Writing to the end for the first draft is something not possible without having enough domain knowledge to do it in the first place..  There really isn’t a comparable product in any software project I have personally worked on which compares to this first draft.  The scope of Neil’s first draft expands far beyond what is in the final book as he will likely cut 50% of the pages during editing.  It has me wondering if it would be a good software development approach to rip through a 20,000 line first cut of an application knowing that I’d delete 1/2 of the code before being finished, slashing out features along the way.

If we did write this quick rough first draft application with the intention of whittling away anything superfluous to arrive ultimately at a publishable product then the next step would be to put on the hat of someone who wants or needs to use this software. Can you make the interface more intuitive? Can you give it some life? Are there any features that are more complex than they’re worth? What can be cut without destroying the utility of the application?

It would be only after these changes have been made that anyone other than yourself would see a demo.

In Neil’s process the final step is essentially debugging. Going through the application yet again with an eye for correcting errors, fact checking things, getting a lawyer’s review for liability etc.  At this point you would also ask for the input from as many test subjects as you can, gathering and filtering feedback to fine tune rough edges.  Essentially you want to make sure that if someone is trying your software they don’t have a reason to ask for a refund and they can’t claim that you or your company are incompetent.  The final product has to be great.

Only after enough friends, family and colleagues have used the software that you feel confident in it do you call it done.

It’s interesting to wonder if this would produce a better product than the typical agile developed software.

There are a couple of significant differences in software vs book writing however that need to be addressed.  Software is rarely a single developer’s sole project.  There is usually a team to divvy the work between. This team adds some communication load to the project as people need to be in sync.  This team approach doesn’t work well if all the domain knowledge is stuck in scribbled notes and in the head of  one guy that did the research.  Would Neil’s approach scale to writing a 10,000 page book with 4 other writers?  The counter point would be that if you’re looking at writing a 10,000 page book maybe it would be better to write 20 500 page books and only have a single author per book.

Software is also not written linearly.  When you start to write a book chapter you may be able to start writing and continue without stopping the flow of words.  You probably wouldn’t be halfway through chapter 18 and realize that it would be good to foreshadow what’s happening earlier, then stop and go back through to find a good spot to add that foreshadow element to an earlier chapter. A disruption like that would completely disrupt the flow of your writing.  It’s difficult to write software without jumping around like that.

Programming languages are much more rigorous than English.  it’s not always possible to skip pieces where you need to know an answer to get a working program. There can be hard dependencies that prevent you from moving on until they get solved.  Again this is an opportunity for interruption.

It would be interesting to try this method on a software project to see how it translates.

If you are interested in getting more details from Neil Strauss and Tim Ferriss about the creative process check out this video:

Polyglot vs Specialization

I consider myself a fairly well rounded developer. So when it comes time to choose a technology to accomplish a task I will happily choose the one that is the best fit whether it requires Ruby, Python, Java, Objective-C, PHP, CSS, or Bash or anything else.

Recently I was given a project to take on.  At first it looked like it would be adding some PHP code to an inherited code base.  Upon starting it became apparent that the code was a disaster and too far gone to be recoverable… a full re-write would probably be best however it was way out of the budget for now.  Rather than continue patching in the new features we needed, we decided to create a new side project that tied into the existing database to accomplish what we needed.  It gave me a chance to evaluate how I wanted to build this application.

Generally we do Ruby on Rails work with my team.  However given that the existing database schema was not even close to normalized or conventional Rails would have been a difficult way to go.  There would need to be a lot of boilerplate code to map things to new field names.  It just wasn’t a good fit.

Instead I opted to go with a simple, lightweight Node.js App.  It would leverage the SQL queries copied out of the original PHP code with minimal overhead.  The app is heavy on database IO and has zero application logic so Node is a good fit.  We already have Javascript skills on the team and experience with testing frameworks for Javascript code, so support wouldn’t be difficult.

Turns out, it didn’t go over well with some people on the team.

“We use Rails as a  technology stack”

Passion can run high in technology circles over tools. Apple vs. Windows. VIM vs Emacs there are religious fights about which is better.

I like to be good at a breadth of technologies, that means staying sharp on a fistful of languages and tools, while always looking to learn new things.  Others like to focus on being world-class with one technology stack.

Which is the better approach?

Api First Web Development

The web is still suffering through a page refresh disaster. HTTP started with simple pages where full page reloads when navigating wasn’t so bad. Modern sites have boatloads of images, css, js and html files on each page that need to be requested each time a page loads and checked to see if the file has changed to re-download it.

It’s wasteful and slows down everyone’s experience with browsing.

There are various approaches to overcoming these shortfalls: gzip everything, image atlas files, concat and uglify javascript etc. These lead to further technical complexity on the server side and nuances that make web development more difficult to get right.

The rise of client side MVC frameworks is a breath of fresh air in web development.

Ajax was just a stepping stone to this new web development paradigm.

There are many competing frameworks right now: Ember.js, Backbone.js, Angular.js, and Meteor.js to name a few.  They provide several big wins for web developers: simplification on the server side (by removing the need for template engines, asset management, and a multitude of views)  This also allows for greater performance on the server side since template rendering is generally slow and rendering a full page is no longer required.  Web backends can be lean and optimized much more easily.  It lowers the bar to get other more performant languages like Go to compete with Django and Ruby on Rails.

With an API first approach the front-end web development can be completely isolated.  All front end files can reside on high performance, highly available and cheap hosting like Amazon S3 and CDNed around the world.

Done smartly the backend can be used against multiple front-end technologies: iOS, Android,  Windows Phone, and desktop apps can all use the same API.  It’s the best of both worlds. You get the superior feel of a native app while maintaining much of the code savings of consolidating business logic on the server.

For the past few years we’ve been living in a web development world where there is a tug of war to decide what pages, or parts of pages should be rendered on the server side and what should be done with javascript and DOM manipulation.  It’s a design decision that has to be thought about consistently when working on complex web sites and generally creates a messy code base with javascript scattered around everywhere.  The future of API first web design makes it very clear. The server provides JSON, and javascript on the client side does the rest.

From a testing perspective the backend becomes easier to unit test. CRUD interfaces and validations are simple to assert.

The messiness of testing the UI now falls entirely on the javascript application where they can be safely written without needing a complete web stack of infrastructure. The MVC frameworks nudge you towards more organized and easier to test javascript code.

For your next web development project consider taking this new approach to the web.  Simplify your backend by doing a RESTful JSON API only, and simplify your front-end by using one of the Javascript MVC frameworks.

Taking Rails Lessons to Django

For the past couple of months I’ve been working on a number of Ruby on Rails projects with some talented and experienced developers.  After years of working in the Django world on various projects, Rails is an interesting contrast.

My perspective on developing webapps has changed greatly since working with Rails. This is both due to technical differences as well as differences in the business goals of the projects I’ve worked on.

The rails community has embraced speed of delivering finished apps to an unprecedented level. Using the Heroku stack and add-on services it’s easy to build a fairly complete application by selecting a series of services you need and installing some carefully selected ruby gems. That gets you 90% to a workable project in a lot of cases. The last 10% is adding in the business logic and custom models and views.  Seeing this app development though the selection and configuration of 3rd party services and software was eye opening.

By embracing 3rd party services for things like logging, errors, email sending, and databases you are relieved of a significant amount of infrastructure and configuration setup time. Having a deployment process that ‘just works’ with a git push removes the need for writing custom scripts in Fabric, and Chef/Puppet recipes – so long as you’re willing to accept the higher costs for hosting.

The Python community, even though it’s supported on Heroku seems to have missed the boat on this a little.

I wanted to see how the speed of development could be tweaked for Django projects by making a couple changes especially on the early setup and bootstraping of new projects.

The Python community actually has some tools that make creating projects from scratch even better than I expected.

Cookie Cutter is an awesome tool for development teams that are creating new projects. You get to define a starting point that includes your favorite libraries and is organized the way you like. If you create lots of new projects from scratch then cookie cutter will save you a bunch of time. Rails bootstraps you with their preferred gems which gives developers a good starting point though it also includes things you might not like. Django developers need to invest in a bit of time to develop their own templates 

Embracing class based views – For a long time I was a hold out. Functional views are easy to understand and require less looking up of documentation to see what things need to be called or overridden in order to do what you want.  Class based views however eliminate a lot of the common patterns you would be writing over and over again in a common app. Using the class based views in Django feels a lot more like rails to me, and once you grasp it they will save you development time.

Use and love libraries. The variety of Ruby gems available is shocking. Even things like collecting in static files for something like jquery can save you some time. In Rails you can include jquery-rails gem and then include the js file in your template. two lines added to your project and everything is hooked up. The Django community hasn’t taken the same approach – adding jquery to a django project requires downloading the files from yourself and putting them in the right folder before adding it to your template.  It may not seem like much but it makes a difference.

Rails and Django are interesting to contrast. They’re similar enough and yet have taken different approaches to solve the same problems. Beyond the technical differences the community differences are fascinating.

Technorati Tags: ,

Playing with Docker, Vagrant, Dokku for deployment

homepage-docker-logoI’m looking at launching a series of new web services in the future and that got me thinking about finding ways to improve my development and deployment workflow to make it super trivial to launch new services and APIs.

Recent experience with Heroku’s PaaS got me looking in that direction. Pushing a git repository to kick off a server build is an eloquently simple approach that makes spinning up new services a snap.

However Heroku costs can quickly spiral up and out of control. Running it myself on raw Amazon EC2 or a local dev server would be ideal.

Thanks to several new projects it’s possible to create an easy to use mini-heroku on your own server.

Docker is a brilliantly simple idea that has sparked a huge following in the last few months.  It is one of the most exciting developments in Open Source at the moment.  Basically Docker lets you run an extremely lightweight virtual machine on top of Linux.  You can spin up and down services in 1-2 seconds that are isolated from the host server which allows you to play with various server configurations quickly, easily, and without touching the core system.

Docker allows you to specify your entire server configuration in a Dockerfile of which there are already thousands defined and shared and searchable at  Need a WordPress server? Redis? Memcached? you can now get them up and running in a virtualized environment extremely easily.

Dokku, builds on top of Docker with the ability to replicate the most impressive feature of Heroku: git push to deploy.  Once set up you can configure a git push remote and have Dokku automatically build and run a Docker for your app and host it on a subdomain.  Brilliant.

Managing all these Docker services can be done with another awesome tool called shipyard. Bring your virtualized multi-host environment under control.  It even supports automatic restarts of crashed services.

Getting it all up and running has been fairly straight forward. Once in place, it will make managing my own cloud of services much more robust.

Fermenter Built

Finished the build for the Fermenter over the weekend. Here’s a video going over the physical components for the project:

The chamber itself is based on the Son of a Fermentation Chiller. with the only modification being that I made mine a bit smaller than their plans.

The django based web app is all open-source. The code & instructions for building it are available on GitHub.

The Pi will run a django based web interface that will provide simple and minimal control and logs of the brewing process.


The electronics are fairly simple, and as soon as I figure out how to generate a circuit diagram I’ll add instructions for how to solder together the prototype board.

iAd Download Script

Unfortunately Apple doesn’t have a nice easy API for downloading your revenue numbers from iAds. The only way around that right now is screen scraping the information, or manually downloading the reports.

Thanks to the awesome python requests library I was able to put something together in just 50 lines of code.

It’s a function you can integrate with your project to download daily iAd report files.

It is also simple enough that it should be relatively robust against future website changes by Apple. (hopefully)

If you decide to use this I would recommend storing your account’s publisherID to avoid having to get it each time which will speed it up.

(this code is also on github:

import requests
import re, os, datetime
url_base = ''
signin_url = url_base % '/WebObjects/iTunesConnect.woa'
download_csv_url = ''
def getPublisherID(session):
    publisher ID is a number that is buried deep... lets hope
    this code doesn't break often
    headers = {'Content-type': 'text/x-gwt-rpc; charset=UTF-8'}
    iad_service_url= ''
    body = '5|0|4||E8DB97D87973D76A7C9096DCF8A83BB5|com.qwapi.portal.itc.client.rpc.IITCStartupService|getStartupData|1|2|3|4|0|'
    r =, data=body, headers=headers)
    endpos = r.text.find('com.qwapi.portal.client.rpc.dto.UserDTO')
    endpos = r.text.find('"', endpos-4)
    startpos = r.text.rfind('"', endpos-20, endpos-4)
    pubID = r.text[startpos+1:endpos]
    if not int(pubID):
        raise Exception("Failed to get publisher ID")
    return pubID
def downloadiAdFiles(appleId, password, publisherId=None, outputDirectory='.', daysToDownload=14, outputFormat='iAd_D_%m-%d-%Y.txt'):
    session = requests.session()  #session used to maintain cookies etc.
    #get signin page to find url for posting signin credentials
    r = session.get(signin_url)
    match ='" action="(.*)"', r.text)
    #login to ITC
    params = {'theAccountName':appleId,
    r = %, params=params, headers={'Content-Length':'0'})
    r = session.get('')
    if publisherId is None:
        publisherId = getPublisherID(session)
    reportDates = [ - datetime.timedelta(i + 1 ) for i in range(daysToDownload)]
    filenames = []
    for downloadReportDate in reportDates:
        filename = os.path.join(outputDirectory, downloadReportDate.strftime(outputFormat))
        if (os.path.exists(filename)):
        dateString = downloadReportDate.strftime('%m/%d/%Y')
        params = {'pageName': 'app_homepage',
            'searchTerms':'Search Apps',
        r = session.get(download_csv_url, params=params)
        if r.status_code != 200:
            raise Exception("Script failed to dowload - check login credentials & publisher ID")
        with open(filename, 'wb') as f:
    return filenames

Using tmux and Vim

Working on the same projects from multiple computers can be difficult.  There are lots of solutions for this problem such as using Dropbox to sync files between computers. but that leaves your workspace to start from scratch ie when working from the office last I had these 10 files open, and a couple of things running, then when I get home if the files sync over I have to reopen those files locally and configure that computer for anything else that needs to run to continue debugging.

Enter Tmux and Vim.

Lately I’ve been working 3 days at home, 2 days at the office each week.  As a result I have re-evaluated my workflow to find something a bit more efficient.  Vim in the terminal is brilliant once you’ve got it set up and have mastered it.  It has been my main editor for about 3 years now.  There are a ton of great resources out there to learn Vim for those of you still trying to find the way.

Tmux is new to me though.  It’s basically a better version of screen.  If your not familar it basically provides a way to do 2 main things.   First it allows you to create a persistent session that you can connect to and leave running.  Secondly in this session you can open up and layout many different consoles into tabs or windows. It’s sort of a command line version of VNC.

I have tmux installed on my Linux servers on AWS so that I can work efficiently on those machines, and I have it installed on my workstation at the office.  I can log into the environment with SSH and continue right where I left off even if I’m switching between my laptop and a friends computer.

With Vim I have the ability to keep my settings and plugins saved and configured the way I like them with a configurations version controlled on github.  Getting a new Linux or Mac machine up and running takes very little time to do with this setup.

I’ll give my best tips and tricks for Vim in another post.

Clone an Xcode Project

I’ve got a couple of projects on the go right now, but one of the more interesting ones is a script that walks a user through the process of cloning an Xcode project and do some variable substitutions. With the final result being an iOS app that is ready to submit to the App Store.

The reason for this project is that one of my app platforms lends itself to being very niche specific and it makes sense to go for a bunch of targeted niche apps rather than a few all encompassing ones. To that end I wanted a way to quickly take the app platform to as many niches where it makes sense. To do it in a reasonable amount of time required some automation.

The approach I’ve taken is to generate a master app. The code in the master app has some placeholder variables which I can easily do search and replace. Along with that I found a script to copy and rename an xcode project which deals with the xml files and folder names and such. The script to rename the project itself is from

One of the helpful things about this script is that by using PIL I can easily scale the images in the project down from the retina source graphics. I can walk myself through all the tedious stuff like getting provisioning profiles, setting up Facebook and Twitter keys so that I don’t forget to do things or do them in the wrong order.

It would be easy to see how this could be extended. Perhaps by adding a few more variable substitutions it would be easy to add/remove various features by setting pre-processor flags or tweak behaviours like, speeds, timeouts etc. Wrapping it all into a nice GUI would make this whole thing into a drag and drop app creator tool that anyone could use.

#!/usr/bin/env python
# Written By Matt Warren, Halotis Inc.
# This script is a wizard to walk you through duplicating an Xcode project that
# follows the patterns I've done for my previous projects.  It's not meant to
# be completely generic.
import subprocess
import zipfile
from PIL import Image
import glob, os
def main():
    the script asks questions and then unzips, renames and runs some variable substitions
    on the AppDelegate.h file.  the final product should be ready to test and submit
    #These are in no particular order...
    print """
Fear not about making mistakes.  All these values can be easily fixed in the final project.
Just fill things out as best you can and fix missing values during testing.
This script requires a file called which contains a project called SampleProject
The process is:
    1. create/rename the Xcode project
    2. Create the art assets and put them in place
    3. Tie the app to various services and set up in iTunesConnect
    4. replace constants in AppDelegate.h
    5. Test
    6. Upload binary to Apple
    7. Profit!!
    new_app_name = raw_input('Please enter New App Name:')
    new_proj_name = new_app_name
    raw_input('Renaming SampleProject to %s (enter to continue)' % new_proj_name)
    print 'Thanks, unzipping and renaming base project...'
    z = zipfile.ZipFile('')
    result =['./', 'SampleProject', new_proj_name])
    if result != 0:
    print 'On to the Art...'
    garbage = raw_input(' ## OVERWRITE %s/Default@2x.png (enter to continue) ##' % new_proj_name)
    image ='%s/Default@2x.png' % new_proj_name)
    small_image = image.resize((image.size[0] / 2, image.size[1] / 2), Image.ANTIALIAS)'%s/Default.png' % new_proj_name)
    garbage = raw_input(' ## OVERWRITE %s/Default~ipad@2x.png (enter to continue) ##' % new_proj_name)
    image ='%s/Default~ipad@2x.png' % new_proj_name)
    small_image = image.resize((image.size[0] / 2, image.size[1] / 2), Image.ANTIALIAS)'%s/Default~ipad.png' % new_proj_name)
    garbage = raw_input('Create Icons using PhotoShop template, (enter to continue)')
    print 'variable subsitutions are next:   (avoid using anything with quotes)'
    header_content = open('%s/Classes/AppDelegate.h' % new_proj_name, 'r').read()
    header_content.replace('<app_name>', new_app_name)
    website = raw_input('Please enter App Website:')
    header_content.replace('<website>', website)
    print ''
    apple_app_id = raw_input('Apple App ID:')
    header_content.replace('<app_id>', apple_app_id)
    print ''
    facebook_app_id = raw_input('Facebook App ID')
    header_content.replace('<facebook_app>', facebook_app_id)
    print ''
    twitter_consumer_key = raw_input('Twitter Consumer Key (use website above for callback)')
    header_content.replace('<twitter_consumer>', twitter_consumer_key)
    twitter_secret = raw_input('Twitter Secret')
    header_content.replace('<twitter_secret>', twitter_secret)
    print 'configure Facebook App ID in URLS - DO THIS MANUALLY!!'
    print 'Writing new App Delegate Header File'
    with open('%s/Classes/AppDelegate.h' % new_proj_name, 'w') as header:
if __name__ == '__main__':

App Control Demo

App Control is a Django App that I have been evolving over the last year for a backend service to support my (and my friends) iOS apps.

It allows you to update them, push new content to them, and provide some central storage for building things like a commenting engine and to tie everyone who uses your apps together within a community.

It also works to pull in marketing information, ad revenue, sales, updates, reviews etc and add them together to compute some higher level business numbers such as RPI (revenue per install).

Next features I plan on adding include a generic currency/wallet system and store for dynamically providing in game content and allow price changes without submitting an update to Apple.

I’m wondering if this sort of app would be of interest to anyone else out there who has iPhone Apps.