Category Archives: Internet

Api First Web Development

The web is still suffering through a page refresh disaster. HTTP started with simple pages where full page reloads when navigating wasn’t so bad. Modern sites have boatloads of images, css, js and html files on each page that need to be requested each time a page loads and checked to see if the file has changed to re-download it.

It’s wasteful and slows down everyone’s experience with browsing.

There are various approaches to overcoming these shortfalls: gzip everything, image atlas files, concat and uglify javascript etc. These lead to further technical complexity on the server side and nuances that make web development more difficult to get right.

The rise of client side MVC frameworks is a breath of fresh air in web development.

Ajax was just a stepping stone to this new web development paradigm.

There are many competing frameworks right now: Ember.js, Backbone.js, Angular.js, and Meteor.js to name a few.  They provide several big wins for web developers: simplification on the server side (by removing the need for template engines, asset management, and a multitude of views)  This also allows for greater performance on the server side since template rendering is generally slow and rendering a full page is no longer required.  Web backends can be lean and optimized much more easily.  It lowers the bar to get other more performant languages like Go to compete with Django and Ruby on Rails.

With an API first approach the front-end web development can be completely isolated.  All front end files can reside on high performance, highly available and cheap hosting like Amazon S3 and CDNed around the world.

Done smartly the backend can be used against multiple front-end technologies: iOS, Android,  Windows Phone, and desktop apps can all use the same API.  It’s the best of both worlds. You get the superior feel of a native app while maintaining much of the code savings of consolidating business logic on the server.

For the past few years we’ve been living in a web development world where there is a tug of war to decide what pages, or parts of pages should be rendered on the server side and what should be done with javascript and DOM manipulation.  It’s a design decision that has to be thought about consistently when working on complex web sites and generally creates a messy code base with javascript scattered around everywhere.  The future of API first web design makes it very clear. The server provides JSON, and javascript on the client side does the rest.

From a testing perspective the backend becomes easier to unit test. CRUD interfaces and validations are simple to assert.

The messiness of testing the UI now falls entirely on the javascript application where they can be safely written without needing a complete web stack of infrastructure. The MVC frameworks nudge you towards more organized and easier to test javascript code.

For your next web development project consider taking this new approach to the web.  Simplify your backend by doing a RESTful JSON API only, and simplify your front-end by using one of the Javascript MVC frameworks.

Playing with Docker, Vagrant, Dokku for deployment

homepage-docker-logoI’m looking at launching a series of new web services in the future and that got me thinking about finding ways to improve my development and deployment workflow to make it super trivial to launch new services and APIs.

Recent experience with Heroku’s PaaS got me looking in that direction. Pushing a git repository to kick off a server build is an eloquently simple approach that makes spinning up new services a snap.

However Heroku costs can quickly spiral up and out of control. Running it myself on raw Amazon EC2 or a local dev server would be ideal.

Thanks to several new projects it’s possible to create an easy to use mini-heroku on your own server.

Docker is a brilliantly simple idea that has sparked a huge following in the last few months.  It is one of the most exciting developments in Open Source at the moment.  Basically Docker lets you run an extremely lightweight virtual machine on top of Linux.  You can spin up and down services in 1-2 seconds that are isolated from the host server which allows you to play with various server configurations quickly, easily, and without touching the core system.

Docker allows you to specify your entire server configuration in a Dockerfile of which there are already thousands defined and shared and searchable at http://index.docker.io.  Need a WordPress server? Redis? Memcached? you can now get them up and running in a virtualized environment extremely easily.

Dokku, builds on top of Docker with the ability to replicate the most impressive feature of Heroku: git push to deploy.  Once set up you can configure a git push remote and have Dokku automatically build and run a Docker for your app and host it on a subdomain.  Brilliant.

Managing all these Docker services can be done with another awesome tool called shipyard. Bring your virtualized multi-host environment under control.  It even supports automatic restarts of crashed services.

Getting it all up and running has been fairly straight forward. Once in place, it will make managing my own cloud of services much more robust.

An Easy VPN on EC2

I’ve become a little bit more paranoid over the last few weeks about computer security. I used to think I will be fine – why would anyone want to hack my computer and find my personal files. But after a bit of reading I’m more concerned about the far more likely situation of mass spying and infiltration programs either churning through my network traffic or infecting my computers with a botnet. In those cases people will collect everything they can and data mine it afterwards.

Starting a VPN should help greatly with preserving my privacy.  It also gives me the added benefit of having an IP address in the USA which allows me to watch Hulu and other US only content sites.

After several false starts trying to get OpenVPN working I decided to try something easier.

Starting with an existing AMI was easy.  I searched for and launched a new instance using ami-744c971d which is pre-configured for pptp with a simple username/password.  Create a new security group for the instance when configuring it.

Once the server is launched and running you’ll need to open up some ports so that it is accessible.  PPTP uses TCP port 1723.

In the Security Group settings open up TCP port 1723 and TCP port 22 (SSH).

Ssh to the server using the key pair you selected when starting the instance and the username ‘ubuntu’ to edit the settings file.  Edit the file /etc/ppp/chap-secrets  change the username ‘seamus’ and password ‘wiles’ to something else.

Restart pptp: ‘sudo service pptpd restart’

You should be able to connect then to the new VPN server.

 

Mixing HTML5 and Native iOS

There’s a lot of people out there who have their favorite technologies and there are few more passionate debates than the issue of using HTML5 or native development for mobile apps.

It’s still early days for HTML5 on a mobile platform, the technologies are still maturing and so there are fewer libraries out there for doing things like writing games or interfacing with the hardware. Due to the nature of HTML and interpreted Javascript, performance will never match that of a native app, but with a fast enough phone and a good performing renderer performance issues will eventually not be issues at all.

Native development allows apps to build on the look and feel that users are used to on their devices. Usability should therefore be better on a native app because buttons will be using standard (tested) sizes and provide feedback that the user expects. Because native apps are compiled and optimized for the hardware they naturally have a performance advantage over HTML.

Personally I prefer to use native stuff whenever possible. But sometimes it is just far easier to hook into a web server to deliver some dynamic content in which case it can be simpler just to display it in it’s HTML form.

The current feature I’m working on is to add a commenting system to a bunch of my apps. Using the django commenting app and jquery mobile I was able to get a pretty good web based commenting system in place in a few hours. Translating that to a native form in Objective-C would have taken much longer. So I’m just going to throw up a web rendering widget in the app to load the webpage.

I see the lines blurring over the next few years in this way. HTML is catching up to native in a lot of ways but native will always have some natural advantages and is also improving. Any new hardware features will first be accessible to native APIs before they are wrapped up by webkit and exposed for use in HTML. And differing platforms will always result in HTML5 apps that have to things like “if iPhone then …”. The main benefit for HTML5 then is centralizing to servers that can be quickly updated with bug fixes and that people have HTML/JS/CSS skillsets and want to work on mobile apps.

Right now is simply too early to jump into a pure HTML5 mobile app for anything serious though. As seen by the HTML5 based Facebook App which is terribly slow on both iOS and Android and got bad user reviews as a result of those performance woes. A native Facebook app is in the works to address those shortcomings. But I would predict that it won’t be until iPhone 7 comes out that HTML5 will be able to compete from a user experience perspective with native apps.

Fabric For Development

Fabric is a pretty awesome tool for deploying projects. But it turns out that it’s also pretty awesome for lots of other stuff.

At PyCon 2012 there was a talk given by Ricardo Kirkner (which you can watch here) that inspired me to play around with fabric in some new ways.

It’s possible to use fabric as a wrapper around the standard django ./manage.py script, to help setting up virtualenvs and install packages. Using fabric to script around these things means that there are fewer tools that new developers will need to get set up and know how to use. Scripts that normally might have been loose bash files can now be collected, organized and documented.

I’m currently working on a large django project with 4 other developers who are new to python and django. Getting everyone’s development environment working was a big pain since there were multiple platforms (Mac and Linux) and different configurations for base packages. If I had thought of this sooner I might have been able to create a reliable fabric script so that “easy_install fabric; fab init_project” got them from zero to running django app.

There’s also several oneliners that I run fairly regularly which can be saved in fabfile.py and called much easier. For example:

def clean():
    local('find . -name "*\.pyc" -exec rm -r {} \;')

Now

$ fab clean

will clear out any .pyc files in the project.

It’s also possible to manage the virtualenv environment through fabric:

VIRTUALENV = '.virtualenv/'
def setup_virtualenv():
    created = False
    virtual_env = os.environ.get('VIRTUAL_ENV', None)
    if virtual_env is None:
        if not os.path.exists(VIRTUALENV):
            _create_virtualenv()
            created = True
        virtual_env = VIRTUALENV
    env.virtualenv = os.path.abspath(virtual_env)
    _activate_virtualenv()
    return created
 
def _activate_virtualenv():
    activate_this = os.path.abspath("%s/bin/activate_this.py" % env.virtualenv)
    execfile(activate_this, dict(__file__=activate_this))
 
def _create_virtualenv(clear=False):
    if not os.path.exists(VIRTUALENV) or clear:
        args = '--no-site-packages --distribute --clear'
        local("%s /usr/local/bin/virtualenv %s %s" % (sys.executable, args, VIRTUALENV), capture=False)
 
def virtualenv_local(command, capture=True):
    prefix = ''
    virtual_env = env.get('virtualenv', None)
    if virtual_env:
        prefix = ". %s/bin/activate && " % virtual_env
    command = prefix + command
    return local(command, capture=capture)
 
def manage(command, *args):
    virtualenv_local("python manage.py {0} {1}".format(command, ' '.join(args)), capture=False,)
 
def runserver(*args):
    manage('runserver', *args)

These functions let you create the virtualenv and run commands in the virtualenv (without having manually activated it). virtualenv_local wraps the call to fabric’s “local” function and sources the activate script before launching the command specified. the manage function provides a way to call django’s manage.py script, and the runserver function gives a way to use fabric to launch the server (in the virtualenv). So now the fab command can be used to consolidate both virtualenv tools and manage.py script into one document-able fabfile.py with a consistent command-line interface.

Technorati Tags:

iAd Report Downloader Script

I was surprised that there wasn’t already a script out there to download iAd reports from Apple’s iTunes Connect website.

Apple released a Java based command line tool to download the sales reports for Apps but neglected to provide something similar for iAd publishers. Some Googling around I was further surprised that I couldn’t find any 3rd party scripts to download this data.

I did however find a python script called appsalesdaily.py which is a web scraper that downloads the sales reports. It handled a lot of the nasty http cookies and login stuff that is usually very tricky to do with a script. I modified that script and extended it with a function to download daily iAd reports.

The rather fascinating thing was just how complex the single page of the iAd publisher dashboard is. It was built with GWT which is perhaps the worst thing ever developed. It produced a webpage that contains 40,000+ lines of javascript and all it does is draw a few graphs. The code was terribly convoluted and nearly impossible to reverse engineer. But 6 frustrating hours later I was able to download the files I wanted.

I don’t wish this sort of struggle on anyone so I’m making the code available. https://github.com/mfwarren/appdailysales

iPhone App Backend In Development

After getting UFO Invader published there were a couple of things that became obvious points of improvement from a marketing perspective.

Web sites have the advantage that you can count on having internet access when visiting a website so it’s easy to tie things in dynamically all over the  place.  Things such as Adsense ads are loaded and managed by other people using other services.  they change depending on who you are, what the page is like, and what time of day it is.

With a turn around time for publishing native apps to the iPhone being more than a few days – having anything from news messages, advertisements or dynamic content becomes a bit more complex.  You can’t simply push an app update out to update a message to say you’re working on a big update for example.  You also can’t count on internet access being available.

The idea for the iPhone App Control backend is to pair a web admin service with a library of Objective-C UIKit components.  An API call would be made out to the service to get marketing messages, news or alerts text or whatever else and it would also serve as a bit of an analytics service, tracking clicks on ads, and perhaps performing A/B split tests on offers.

Once built, this software will help control all of my current and future iOS Apps.

Will I release this software?  I’m not sure.  What do you think?  should I sell it? open-source it?  sell accounts and access?  Let me know your thoughts…

Perplexing Advertising Numbers

I’m still doing a lot of testing around how this iPhone Apps stuff works but yesterday I ran an experiment to see if mobile advertising would help boost sales of my game.

The results were very odd.

Firstly, I have been running various ad supported websites for about 8 years now so I’m pretty confident about how the revenue has been working for websites using Adsense.  I have also run quite a few AdWords campaigns so I’m familiar with how much that traffic usually costs.

Very quickly after launching an ad campaign on AdMob I blew through the $10 test budget at $0.03/click which netted 334 clicks with a click through rate of about 0.5%.  That’s pretty cheap traffic in comparison to my experience with AdWords which are often 10x to 100x more expensive.

Immediately following the campaign I could see that more people were playing the game.  More people had scores show up in the leader board, and I was seeing ad impressions from the in-app ads.  Ad impressions spiked roughly 20x compared to previous days so I was expecting to see significant sales.

Then I started seeing revenue come in from the ads.  It was shocking that every time I refreshed the page there would be a few more cents of revenue.  People seemed to be clicking the ads at roughly 20% CTR.  As of right now I can see four times more ad clicks than people who have bought the game!

Today, if the trend continues I expect my ad impressions and revenue from AdMob to exceed what I make with AdSense across all my websites combined!  This is crazy.  In the last week I have sold 30 copies of the game.  I can attribute 7 of those sales to the ad campaign I ran.  Before running the ad campaign I had 5 days of zero ad clicks and an average of roughly 10 ad impressions per day.  in the 24 hours following the ad campaign I saw ad impressions spike to nearly 1000 with 175 clicks.

Something doesn’t seem right.  These numbers are wacky compared to what I’m used to seeing with ad revenue.

Update: Found out that a hacked version of my game was uploaded to a forum at roughly the same time I started the ad campaign.  That explains it.

UFO Invader Submitted – Waiting Again

Waiting seems to be something that you have to get used to when dealing with Apple.  Granted, the whole process is vastly more efficient than waitingit was back in the days of physical distribution.  And I’m thankful that that I don’t have to hire a publisher and deal with box designs, CD duplication and the massive up front costs they would have incurred.

In fact the 1 to 2 week waiting time I expect for my first game to be approved is hardly enough time to get my marketing plan up and going.  There’s videos to create, stuff to write, lists to build and people to contact.  It’s all work that takes time.

However, it’s also frustrating when people who are anxious to play it can’t and have to wait weeks before they can – even though the development is finished.

These next few weeks are also a time to start looking forward to my next app.  The first one isn’t yet out the door and the next one is already getting sketched out on the whiteboard.  Getting the jump on development is going to be key to keeping the flow going.

So yes, there is some waiting involved, but it’s actually a welcome break to focus on next steps.

iPhone App Store SEO

The Apple iTunes App Store search engine is a bit of a black box. Unlike Google which has some transparency (though they keep a lot of things close to the chest). With Google you have the opportunity to find trends through Google Trends, or see competition and search volumes by keyword within the Adwords Keyword Tool. Through Google Analytics you have the ability to track back to find out what keywords provide your website with traffic in great detail.

With Apple you get none of that.

When publishing an iPhone App there are two fields that appear to provide some impact on how your app will rank which you have control over.

  1. the keywords field.  You can only update this field when you publish your app or an update and only get 100 characters to work with.  Max it out with as many of the  best keywords as you can.
  2. the title. Don’t be afraid to put something long in here or to embed a subtitle.  Make sure it is something unique enough that if a friend told you about the app you would be able to find it easily.
Other than these traditional SEO techniques for improving your ranking within the app store there’s one non-traditional SEO trick to getting ranked higher in the the results.  That is the number of downloads your app gets.  there are a number of ways your can influence this from outside the app store.
  • Create some youtube videos
  • Give away free download codes to review sites.
  • Create a webpage for the app
  • Create a Facebook Fan page
  • Tweet about it or create a twitter account
  • blog about your app
  • Send out email
  • Do a “Launch”
Besides these more traditional Internet Marketing approaches to drive traffic you also have several new abilities on the iPhone to market and drive traffic to the store.
  • Cross promote with your other apps.  Use things like house ads, a “More” screen with links to your other stuff, menu items, and pop ups.  Use your existing apps as a platform to advertise your newest apps.
  • Push notifications to your users so that they get notified of your latest promotions.  Directly communicating to your customers where ever they are at any time is HUGE.
  • Mobile advertising through places like AdMob allows you to target by region and device type.  If you want to get your App in front of people it’s always possible to pay to get it there.
There you have it.  There are three main factors you can influence to improve your ranking in the iPhone App Store.  Remember that getting your app published is really just the start of what needs to be done for a successful App business.  A great marketing strategy is key to get your stuff to stand out amongst the 285,000 apps in the App Store.