Category Archives: Software

Api First Web Development

The web is still suffering through a page refresh disaster. HTTP started with simple pages where full page reloads when navigating wasn’t so bad. Modern sites have boatloads of images, css, js and html files on each page that need to be requested each time a page loads and checked to see if the file has changed to re-download it.

It’s wasteful and slows down everyone’s experience with browsing.

There are various approaches to overcoming these shortfalls: gzip everything, image atlas files, concat and uglify javascript etc. These lead to further technical complexity on the server side and nuances that make web development more difficult to get right.

The rise of client side MVC frameworks is a breath of fresh air in web development.

Ajax was just a stepping stone to this new web development paradigm.

There are many competing frameworks right now: Ember.js, Backbone.js, Angular.js, and Meteor.js to name a few.  They provide several big wins for web developers: simplification on the server side (by removing the need for template engines, asset management, and a multitude of views)  This also allows for greater performance on the server side since template rendering is generally slow and rendering a full page is no longer required.  Web backends can be lean and optimized much more easily.  It lowers the bar to get other more performant languages like Go to compete with Django and Ruby on Rails.

With an API first approach the front-end web development can be completely isolated.  All front end files can reside on high performance, highly available and cheap hosting like Amazon S3 and CDNed around the world.

Done smartly the backend can be used against multiple front-end technologies: iOS, Android,  Windows Phone, and desktop apps can all use the same API.  It’s the best of both worlds. You get the superior feel of a native app while maintaining much of the code savings of consolidating business logic on the server.

For the past few years we’ve been living in a web development world where there is a tug of war to decide what pages, or parts of pages should be rendered on the server side and what should be done with javascript and DOM manipulation.  It’s a design decision that has to be thought about consistently when working on complex web sites and generally creates a messy code base with javascript scattered around everywhere.  The future of API first web design makes it very clear. The server provides JSON, and javascript on the client side does the rest.

From a testing perspective the backend becomes easier to unit test. CRUD interfaces and validations are simple to assert.

The messiness of testing the UI now falls entirely on the javascript application where they can be safely written without needing a complete web stack of infrastructure. The MVC frameworks nudge you towards more organized and easier to test javascript code.

For your next web development project consider taking this new approach to the web.  Simplify your backend by doing a RESTful JSON API only, and simplify your front-end by using one of the Javascript MVC frameworks.

Taking Rails Lessons to Django

For the past couple of months I’ve been working on a number of Ruby on Rails projects with some talented and experienced developers.  After years of working in the Django world on various projects, Rails is an interesting contrast.

My perspective on developing webapps has changed greatly since working with Rails. This is both due to technical differences as well as differences in the business goals of the projects I’ve worked on.

The rails community has embraced speed of delivering finished apps to an unprecedented level. Using the Heroku stack and add-on services it’s easy to build a fairly complete application by selecting a series of services you need and installing some carefully selected ruby gems. That gets you 90% to a workable project in a lot of cases. The last 10% is adding in the business logic and custom models and views.  Seeing this app development though the selection and configuration of 3rd party services and software was eye opening.

By embracing 3rd party services for things like logging, errors, email sending, and databases you are relieved of a significant amount of infrastructure and configuration setup time. Having a deployment process that ‘just works’ with a git push removes the need for writing custom scripts in Fabric, and Chef/Puppet recipes – so long as you’re willing to accept the higher costs for hosting.

The Python community, even though it’s supported on Heroku seems to have missed the boat on this a little.

I wanted to see how the speed of development could be tweaked for Django projects by making a couple changes especially on the early setup and bootstraping of new projects.

The Python community actually has some tools that make creating projects from scratch even better than I expected.

Cookie Cutter is an awesome tool for development teams that are creating new projects. You get to define a starting point that includes your favorite libraries and is organized the way you like. If you create lots of new projects from scratch then cookie cutter will save you a bunch of time. Rails bootstraps you with their preferred gems which gives developers a good starting point though it also includes things you might not like. Django developers need to invest in a bit of time to develop their own templates 

Embracing class based views – For a long time I was a hold out. Functional views are easy to understand and require less looking up of documentation to see what things need to be called or overridden in order to do what you want.  Class based views however eliminate a lot of the common patterns you would be writing over and over again in a common app. Using the class based views in Django feels a lot more like rails to me, and once you grasp it they will save you development time.

Use and love libraries. The variety of Ruby gems available is shocking. Even things like collecting in static files for something like jquery can save you some time. In Rails you can include jquery-rails gem and then include the js file in your template. two lines added to your project and everything is hooked up. The Django community hasn’t taken the same approach – adding jquery to a django project requires downloading the files from jquery.com yourself and putting them in the right folder before adding it to your template.  It may not seem like much but it makes a difference.

Rails and Django are interesting to contrast. They’re similar enough and yet have taken different approaches to solve the same problems. Beyond the technical differences the community differences are fascinating.

Technorati Tags: ,

Playing with Docker, Vagrant, Dokku for deployment

homepage-docker-logoI’m looking at launching a series of new web services in the future and that got me thinking about finding ways to improve my development and deployment workflow to make it super trivial to launch new services and APIs.

Recent experience with Heroku’s PaaS got me looking in that direction. Pushing a git repository to kick off a server build is an eloquently simple approach that makes spinning up new services a snap.

However Heroku costs can quickly spiral up and out of control. Running it myself on raw Amazon EC2 or a local dev server would be ideal.

Thanks to several new projects it’s possible to create an easy to use mini-heroku on your own server.

Docker is a brilliantly simple idea that has sparked a huge following in the last few months.  It is one of the most exciting developments in Open Source at the moment.  Basically Docker lets you run an extremely lightweight virtual machine on top of Linux.  You can spin up and down services in 1-2 seconds that are isolated from the host server which allows you to play with various server configurations quickly, easily, and without touching the core system.

Docker allows you to specify your entire server configuration in a Dockerfile of which there are already thousands defined and shared and searchable at http://index.docker.io.  Need a WordPress server? Redis? Memcached? you can now get them up and running in a virtualized environment extremely easily.

Dokku, builds on top of Docker with the ability to replicate the most impressive feature of Heroku: git push to deploy.  Once set up you can configure a git push remote and have Dokku automatically build and run a Docker for your app and host it on a subdomain.  Brilliant.

Managing all these Docker services can be done with another awesome tool called shipyard. Bring your virtualized multi-host environment under control.  It even supports automatic restarts of crashed services.

Getting it all up and running has been fairly straight forward. Once in place, it will make managing my own cloud of services much more robust.

Fermenter Built

Finished the build for the Fermenter over the weekend. Here’s a video going over the physical components for the project:

The chamber itself is based on the Son of a Fermentation Chiller. with the only modification being that I made mine a bit smaller than their plans.

The django based web app is all open-source. The code & instructions for building it are available on GitHub. https://github.com/mfwarren/Fermenter

The Pi will run a django based web interface that will provide simple and minimal control and logs of the brewing process.

Screenshot_2013-05-01_9_38_PM-3

The electronics are fairly simple, and as soon as I figure out how to generate a circuit diagram I’ll add instructions for how to solder together the prototype board.

iAd Download Script

Unfortunately Apple doesn’t have a nice easy API for downloading your revenue numbers from iAds. The only way around that right now is screen scraping the information, or manually downloading the reports.

Thanks to the awesome python requests library I was able to put something together in just 50 lines of code.

It’s a function you can integrate with your project to download daily iAd report files.

It is also simple enough that it should be relatively robust against future website changes by Apple. (hopefully)

If you decide to use this I would recommend storing your account’s publisherID to avoid having to get it each time which will speed it up.

(this code is also on github: https://github.com/mfwarren/iAdDownload)

import requests
import re, os, datetime
 
url_base = 'https://itunesconnect.apple.com%s'
signin_url = url_base % '/WebObjects/iTunesConnect.woa'
download_csv_url = 'https://iad.apple.com/itcportal/generatecsv'
 
def getPublisherID(session):
    """
    publisher ID is a number that is buried deep... lets hope
    this code doesn't break often
    """
    headers = {'Content-type': 'text/x-gwt-rpc; charset=UTF-8'}
    iad_service_url= 'https://iad.apple.com/itcportal/service/startup?requestId=GUEST_1331628761939_0__1331628761950__null'
    body = '5|0|4|https://iad.apple.com/itcportal/itcportal/|E8DB97D87973D76A7C9096DCF8A83BB5|com.qwapi.portal.itc.client.rpc.IITCStartupService|getStartupData|1|2|3|4|0|'
    r = session.post(iad_service_url, data=body, headers=headers)
    endpos = r.text.find('com.qwapi.portal.client.rpc.dto.UserDTO')
    endpos = r.text.find('"', endpos-4)
    startpos = r.text.rfind('"', endpos-20, endpos-4)
    pubID = r.text[startpos+1:endpos]
    if not int(pubID):
        raise Exception("Failed to get publisher ID")
    return pubID
 
 
def downloadiAdFiles(appleId, password, publisherId=None, outputDirectory='.', daysToDownload=14, outputFormat='iAd_D_%m-%d-%Y.txt'):
 
    session = requests.session()  #session used to maintain cookies etc.
 
    #get signin page to find url for posting signin credentials
    r = session.get(signin_url)
    match = re.search('" action="(.*)"', r.text)
 
    #login to ITC
    params = {'theAccountName':appleId,
            'theAccountPW':password,
            '1.Continue.x':'10',
            '1.Continue.y':'10',
            'theAuxValue':''}
    r = session.post(url_base % match.group(1), params=params, headers={'Content-Length':'0'})
    r = session.get('https://iad.apple.com/itcportal')
 
    if publisherId is None:
        publisherId = getPublisherID(session)
 
    reportDates = [datetime.date.today() - datetime.timedelta(i + 1 ) for i in range(daysToDownload)]
 
    filenames = []
    for downloadReportDate in reportDates:
        filename = os.path.join(outputDirectory, downloadReportDate.strftime(outputFormat))
        if (os.path.exists(filename)):
            continue
 
        dateString = downloadReportDate.strftime('%m/%d/%Y')
        params = {'pageName': 'app_homepage',
            'dashboardType':'publisher',
            'publisherId':publisherId,
            'dateRange':'customDateRange',
            'searchTerms':'Search Apps',
            'adminFlag':'false',
            'fromDate':dateString,
            'toDate':dateString,
            'dataType':'byName'
        }
        r = session.get(download_csv_url, params=params)
        if r.status_code != 200:
            raise Exception("Script failed to dowload - check login credentials & publisher ID")
 
        with open(filename, 'wb') as f:
            f.write(r.content)
        filenames.extend([filename])
 
    return filenames

Using tmux and Vim

Working on the same projects from multiple computers can be difficult.  There are lots of solutions for this problem such as using Dropbox to sync files between computers. but that leaves your workspace to start from scratch ie when working from the office last I had these 10 files open, and a couple of things running, then when I get home if the files sync over I have to reopen those files locally and configure that computer for anything else that needs to run to continue debugging.

Enter Tmux and Vim.

Lately I’ve been working 3 days at home, 2 days at the office each week.  As a result I have re-evaluated my workflow to find something a bit more efficient.  Vim in the terminal is brilliant once you’ve got it set up and have mastered it.  It has been my main editor for about 3 years now.  There are a ton of great resources out there to learn Vim for those of you still trying to find the way.

Tmux is new to me though.  It’s basically a better version of screen.  If your not familar it basically provides a way to do 2 main things.   First it allows you to create a persistent session that you can connect to and leave running.  Secondly in this session you can open up and layout many different consoles into tabs or windows. It’s sort of a command line version of VNC.

I have tmux installed on my Linux servers on AWS so that I can work efficiently on those machines, and I have it installed on my workstation at the office.  I can log into the environment with SSH and continue right where I left off even if I’m switching between my laptop and a friends computer.

With Vim I have the ability to keep my settings and plugins saved and configured the way I like them with a configurations version controlled on github.  Getting a new Linux or Mac machine up and running takes very little time to do with this setup.

I’ll give my best tips and tricks for Vim in another post.

Clone an Xcode Project

I’ve got a couple of projects on the go right now, but one of the more interesting ones is a script that walks a user through the process of cloning an Xcode project and do some variable substitutions. With the final result being an iOS app that is ready to submit to the App Store.

The reason for this project is that one of my app platforms lends itself to being very niche specific and it makes sense to go for a bunch of targeted niche apps rather than a few all encompassing ones. To that end I wanted a way to quickly take the app platform to as many niches where it makes sense. To do it in a reasonable amount of time required some automation.

The approach I’ve taken is to generate a master app. The code in the master app has some placeholder variables which I can easily do search and replace. Along with that I found a script to copy and rename an xcode project which deals with the xml files and folder names and such. The script to rename the project itself is from http://rudifa.wordpress.com/2011/07/23/clonexcodeproject-sh/.

One of the helpful things about this script is that by using PIL I can easily scale the images in the project down from the retina source graphics. I can walk myself through all the tedious stuff like getting provisioning profiles, setting up Facebook and Twitter keys so that I don’t forget to do things or do them in the wrong order.

It would be easy to see how this could be extended. Perhaps by adding a few more variable substitutions it would be easy to add/remove various features by setting pre-processor flags or tweak behaviours like, speeds, timeouts etc. Wrapping it all into a nice GUI would make this whole thing into a drag and drop app creator tool that anyone could use.

#!/usr/bin/env python
 
#######################################
#
# Written By Matt Warren, Halotis Inc.
# http://halotis.com
# http://mattwarren.co
#
# This script is a wizard to walk you through duplicating an Xcode project that
# follows the patterns I've done for my previous projects.  It's not meant to
# be completely generic.
#
#######################################
 
import subprocess
import zipfile
from PIL import Image
import glob, os
 
def main():
    """
    the script asks questions and then unzips, renames and runs some variable substitions
    on the AppDelegate.h file.  the final product should be ready to test and submit
    """
 
    #These are in no particular order...
    print """
################################################################################
 
MATTS SUPER AWESOME PROJECT MULTIPLIER WIZARD!!!
 
Fear not about making mistakes.  All these values can be easily fixed in the final project.
Just fill things out as best you can and fix missing values during testing.
 
This script requires a file called SampleProject.zip which contains a project called SampleProject
 
The process is:
    1. create/rename the Xcode project
    2. Create the art assets and put them in place
    3. Tie the app to various services and set up in iTunesConnect
    4. replace constants in AppDelegate.h
    5. Test
    6. Upload binary to Apple
    7. Profit!!
################################################################################
    """
    new_app_name = raw_input('Please enter New App Name:')
    new_proj_name = new_app_name
 
    raw_input('Renaming SampleProject to %s (enter to continue)' % new_proj_name)
    print 'Thanks, unzipping and renaming base project...'
 
    z = zipfile.ZipFile('SampleProject.zip')
    z.extractall('./')
 
    result = subprocess.call(['./RenameXCodeProj.sh', 'SampleProject', new_proj_name])
 
    if result != 0:
        print 'FAILED TO RUN RENAME SCRIPT'
        return
 
    print 'On to the Art...'
 
    garbage = raw_input(' ## OVERWRITE %s/Default@2x.png (enter to continue) ##' % new_proj_name)
    image = Image.open('%s/Default@2x.png' % new_proj_name)
    small_image = image.resize((image.size[0] / 2, image.size[1] / 2), Image.ANTIALIAS)
    small_image.save('%s/Default.png' % new_proj_name)
 
    garbage = raw_input(' ## OVERWRITE %s/Default~ipad@2x.png (enter to continue) ##' % new_proj_name)
    image = Image.open('%s/Default~ipad@2x.png' % new_proj_name)
    small_image = image.resize((image.size[0] / 2, image.size[1] / 2), Image.ANTIALIAS)
    small_image.save('%s/Default~ipad.png' % new_proj_name)
 
    garbage = raw_input('Create Icons using PhotoShop template, (enter to continue)')
 
    print 'variable subsitutions are next:   (avoid using anything with quotes)'
 
    header_content = open('%s/Classes/AppDelegate.h' % new_proj_name, 'r').read()
    header_content.replace('<app_name>', new_app_name)
 
    website = raw_input('Please enter App Website:')
    header_content.replace('<website>', website)
 
    print 'http://itunesconnect.apple.com'
    apple_app_id = raw_input('Apple App ID:')
    header_content.replace('<app_id>', apple_app_id)
 
    print 'https://developers.facebook.com/'
    facebook_app_id = raw_input('Facebook App ID')
    header_content.replace('<facebook_app>', facebook_app_id)
 
    print 'http://dev.twitter.com/apps/new'
    twitter_consumer_key = raw_input('Twitter Consumer Key (use website above for callback)')
    header_content.replace('<twitter_consumer>', twitter_consumer_key)
 
    twitter_secret = raw_input('Twitter Secret')
    header_content.replace('<twitter_secret>', twitter_secret)
 
    print 'configure Facebook App ID in URLS - DO THIS MANUALLY!!'
 
    print 'Writing new App Delegate Header File'
    with open('%s/Classes/AppDelegate.h' % new_proj_name, 'w') as header:
        header.write(header_content)
 
if __name__ == '__main__':
    main()
</twitter_secret></twitter_consumer></facebook_app></app_id></website></app_name>

App Control Demo

App Control is a Django App that I have been evolving over the last year for a backend service to support my (and my friends) iOS apps.

It allows you to update them, push new content to them, and provide some central storage for building things like a commenting engine and to tie everyone who uses your apps together within a community.

It also works to pull in marketing information, ad revenue, sales, updates, reviews etc and add them together to compute some higher level business numbers such as RPI (revenue per install).

Next features I plan on adding include a generic currency/wallet system and store for dynamically providing in game content and allow price changes without submitting an update to Apple.

I’m wondering if this sort of app would be of interest to anyone else out there who has iPhone Apps.

Mixing HTML5 and Native iOS

There’s a lot of people out there who have their favorite technologies and there are few more passionate debates than the issue of using HTML5 or native development for mobile apps.

It’s still early days for HTML5 on a mobile platform, the technologies are still maturing and so there are fewer libraries out there for doing things like writing games or interfacing with the hardware. Due to the nature of HTML and interpreted Javascript, performance will never match that of a native app, but with a fast enough phone and a good performing renderer performance issues will eventually not be issues at all.

Native development allows apps to build on the look and feel that users are used to on their devices. Usability should therefore be better on a native app because buttons will be using standard (tested) sizes and provide feedback that the user expects. Because native apps are compiled and optimized for the hardware they naturally have a performance advantage over HTML.

Personally I prefer to use native stuff whenever possible. But sometimes it is just far easier to hook into a web server to deliver some dynamic content in which case it can be simpler just to display it in it’s HTML form.

The current feature I’m working on is to add a commenting system to a bunch of my apps. Using the django commenting app and jquery mobile I was able to get a pretty good web based commenting system in place in a few hours. Translating that to a native form in Objective-C would have taken much longer. So I’m just going to throw up a web rendering widget in the app to load the webpage.

I see the lines blurring over the next few years in this way. HTML is catching up to native in a lot of ways but native will always have some natural advantages and is also improving. Any new hardware features will first be accessible to native APIs before they are wrapped up by webkit and exposed for use in HTML. And differing platforms will always result in HTML5 apps that have to things like “if iPhone then …”. The main benefit for HTML5 then is centralizing to servers that can be quickly updated with bug fixes and that people have HTML/JS/CSS skillsets and want to work on mobile apps.

Right now is simply too early to jump into a pure HTML5 mobile app for anything serious though. As seen by the HTML5 based Facebook App which is terribly slow on both iOS and Android and got bad user reviews as a result of those performance woes. A native Facebook app is in the works to address those shortcomings. But I would predict that it won’t be until iPhone 7 comes out that HTML5 will be able to compete from a user experience perspective with native apps.

iTunes Download Stats

Recently I’ve been building out my iPhone App server to provide a business dashboard with all the relevant services and numbers that I care about available at a glance. It avoids me having to sign in and out of many different sites to get the information and makes it easier to push things together – for example charting both Admob and iAd data on the same graph.

Thankfully the web is becoming more programmable every week and these things are becoming easier to put together quickly.

This is a chart I built last night to display the downloads and updates across all my apps for the past 31 days:

You can see the jumps in downloads that correspond to when I released updates to iTunes.

With these sorts of things I’m finding that there is a tipping point. If the custom page I have created is only 90% as good as going to the original source then I’ll just opt to login there but once it becomes as good as or better than that you’ll quickly forget about the 10 different logins you needed to get all those numbers.

Being in charge of it is even better. I use iAd and Admob for advertising and can pull those numbers in and compare them appropriately. On the same page I display data from Apple, Google, Linkshare, as well as numbers I collect myself such as traffic, link clicks and ad impressions. I only have to login to iTunes Connect to release new Apps.

I will continue open sourcing the components for this system over the next few weeks.