Category Archives: Software

My project for the month is a stock trading bot that will ingest tweets from accounts I consider to be market influencers and do some parsing and sentiment analysis to help create and execute a trade through my broker.

For years I’ve wanted to build something to do automated trading and this is something that seems simple enough to accomplish in a month.  That makes it a worthy experiment.

There are several steps to this process:

  1. connect to the twitter stream API and listen to specific user accounts
  2. for each tweet that comes in, parse it for a company name or CEO name
  3. if there is a company or CEO mentioned, find the ticker symbol
  4. run a sentiment analysis on the tweet
  5. look up the current price of the stock
  6. decide on a trade (long/short) and size, limits and stop loss
  7. execute trade through broker

This project will be open-source for those of you interested in watching the progress or curious to see how it works. Twitter Trading Project

Writing code everyday has been an interesting challenge.

In 2015 I started to work towards a long streak on GitHub which eventually capped out at 250 days. The questions I wanted to answer was:

  • Can I apply ‘deliberate practice‘ to programming and get better?
  • Can ‘free coding’ (like free writing) be effective way to push through writers block?
  • How important is memorising to your coding performance?
  • If syntax and API unknowns don’t present bottlenecks to your flow how fast can I translate an idea into code – can it be limited by typing speed?

I started a repository for my daily coding.  It had a simple script to generate a blank file everyday for me to code in and I would try to code something.  Sometimes it would be to explore a new python module, or fiddle with syntax, or challenge myself with a rosetta code example or replicate a previous day’s code from memory.  I wrote dozens of Flask apps, memorised the methods of lots of APIs, and gained a level of confidence with writing Python that I don’t have with any other language.

At the end of the streak I had a repository with hundreds of small scripts.  Only a handful of them were multi-day efforts or had any real value.  The variety of this collection proved to be useful on it’s own too – several times I have referred back to these examples to help with my actual work and to copy/paste snippets from.  Some of them started me down a path of exploration – like calculating the return on investment for solar panels.

Part of what enabled me to maintain this streak as long as I did was a simple script I wrote to check GitHub for daily activity and email me if I hadn’t yet committed any code.  This simple hack was enough of a reminder to keep me focused even when I was otherwise distracted.

This past week I turned that script into a web service anyone can use.  CodeStreak.io will watch your public github activity and email or SMS you if you haven’t yet pushed any code for the day.  This is the first project of 2017 that I plan on building to expand on my previous streak.

In 2017 I want to build 12 projects.  Each should be roughly 10-20 hours of effort and result in something that provides value for other people.  CodeStreak.io is an example of the kind of project that I want to undertake this coming year, but it is also a tool to help ensure that the momentum is sustained for 12 months.  Blocking out 4 hour chunks of time is a helpful way to really focus and be productive, but 4 hours once per week has been (for me) too sparse to maintain interest in something long enough to finish it.  A little bit everyday keeps a project on your mind.  Attempting to maintain a streak will be a tool to power through the bits that are otherwise uninteresting or difficult.  CodeStreak.io is a foundational tool necessary to accomplish my 2017 goal.

The questions I want to explore with this new goal are:

  1. Without a concern for generating revenue can I just write cool things and get them out there?
  2. Can I get deeper into something new and create something useful out of it with less than 20 hours of effort?
  3. Can you get good at seeing a project from start to finish – what skills or traits will improve the odds?

Hopefully, I’ll have some answers at the end of 2017.

Productivity is closely related to skills and talent so it’s been something that I have a keen interested in.  The actions you take to get better at what you do have fairly important role in your wealth and happiness.

I recently read the book “Talent is Overrated” by Geoff Colvin.  It was insightful in a lot of ways and built upon ideas that I had myself about what it takes to reach for the pinnacle of success.

Geoff busts a number of myths about successful people.

  • High IQ does not correlate to significantly more success
  • Fantastic memory is a skill you develop
  • “Deliberate Practice” is the (only) key to improvement
  • Experience in many cases correlates to lower job performance
  • Child prodigies rarely maintain their lead into adulthood

In a handful of skill sets there is a very clear framework for practice.  Music, athletics and chess have very well understood methods for improvement.  You start early, practice, learn and exercise until your skill improves. Stick with it long enough and it’s possible to reach world class standards.

With other domains of knowledge the frameworks for practice are not as well defined.  How do you practice software programming, project management or reading x-rays?

If deliberate practice is the determining factor in skill acquisition then how can we make sure that the education system supports it, and society builds on it to influence corporate decisions in an ongoing way?

The term “deliberate practice” is specific.  Simply working 9-5 at work related tasks where you’re expected to produce output is not considered practice.  Deliberate practice is hard by design, it is a process to continually push you slightly past your current limits, repeatably, and with feedback.

Hitting a bucket of golf balls at the driving range is not practice, but it provides the illusion of practice.  Without a coach to identify and provide feedback on each swing it will be very difficult to improve your skill.  Even Tiger Woods, at the top of his game, had a coach.

To answer the question for something close to my heart, how do you deliberately practice software development, I believe the answer lies in either finding someone better than you who can review or pair program with you who also understands how to coach, or by finding superior open source code to study and reproduce.

One of the examples from the book was how Benjamin Franklin learned to write. He picked articles from Spectator magazine that he wanted to emulate and he would take notes about the article.  After a few days when he had forgotten the text he would open his notes and try to re-create the original article. Then by comparing his against the original he could identify what he could have done better.  In the absence of a coach, this seems like a fairly ingenious way to get immediate feedback on your practice.

It would be good to test this sort of approach with programming skills.  It’s a concept I have started to test.  As with all practice, it’s is a long effort before the benefits are expected.

Everyone who does web development will probably find themselves doing devOps tasks at some point.  I keep an eye on servers hosting several different projects at different cloud providers and some private servers.  Until recently management of those servers wasn’t difficult enough to warrant using tools to deal with things.

Ansible is a commandline tool that allows you to manage your servers.  Lets say like me you want to introduce it into an environment with servers already in place.  Here’s how you can get started.

Install the Ansible commandline:

brew install ansible

Create a hosts file (file that lists all the servers you want to manage):

ansible_hosts:

[ec2]
ec2-.compute-1.amazonaws.com  ansible_ssh_private_key_file=/PATH/TO/key.pem  ansible_ssh_user=ec2-user
ec2-.compute-1.amazonaws.com  ansible_ssh_private_key_file=/PATH/TO/key.pem  ansible_ssh_user=ec2-user

Ping all the servers to see if they respond to ssh:

ansible ec2 -i ansible_hosts -m ping

Ansible comes with a large library of low level ‘modules’ for doing interesting things on your servers. You can call simple things right from the command line as in the previous example (uses the ping module). If you want to upgrade openssl package on all your (Ubuntu) servers:

ansible ec2 -i ansible_hosts -m apt -a "name=openssl update_cache=yes" --sudo

or upgrade all updates to a CentOS server:

ansible ec2 -i ansible_hosts -m yum -a "name=* state=latest" --sudo

There are over 200 modules built into the core of Ansible to make interacting with lots of different programs easy – git, shell commands, postgres databases, supervisorctl, django, gem packages. Doing a git checkout of your application across a pool of web servers can be done with a single command line.

If that was all you could do with Ansible I’d be happy. But that’s really just scratching the surface. Building on these low level modules is what Ansible calls ‘roles’. Roles encompass a series of commands to do a bigger task – like install and configure nginx. You can tie multiple roles together into a ‘playbook’ that can define how to build out your entire infrastructure.

In my next post I’ll cover how to go from zero to fully deployed web application with Ansible.

Programmers often take something for granted that would make them significantly better at their job.  No it’s not communication skills, or management.  The biggest differentiator between the best and average programmer is how often they have to refer to google to solve a problem.

Any programmer who knows the basics of programming in a C like language feels like they’re competent enough to write Javascript – even if they only have to program it occasionally.  What ends up happening is they stall on every other line of code to crawl stack overflow for solutions.  Despite their lack of skills they can still write complex code and get things done.

No doubt that Google and Stack Overflow are awesome resources that make it easier to write things you don’t really have the knowledge to do off the top of your head.  Each detour to the search engines eats time.  It takes 10x more time to find a good piece of code on the internet than if the programmer knows how to write it already.

The skill that most programmers overlook is memorizing the APIs and languages they use.

When you take away all the crutches we use to help write code by writing with pen and paper, without the help of reference material gaps in your knowledge become brutally apparent.  Try this from time to time.

Most programmers I know seem to think that picking up a new language or framework can be done in just a few weeks.  My experience is that unless they take deliberate time to read and learn and build expertise the typical result is getting to a passible skill level and then plateauing.

Do not underestimate the value of depth of knowledge.

Programming is a great mix of both the creative and technical skills. Problem solving on a daily basis makes it one of the best jobs imaginable. Staying ahead of the technology curve and continuing to get better at your core skill is what differentiates an average programmer from the superb.

The three most effective ways I have found to get better at programming is

Read Books

The quality of a curated, professionally edited book written by a talented author is hands down the quickest way to learn something new. If you want to pick up a new language, framework or learn new concepts reading at least one book is a good start. I am always reading at least one technical book.

Practice Programming

I write code at a full-time job for 40 hours / week. That isn’t practice and professional experience doesn’t provide many opportunities to actually get better. Deliberate practice for software coding can come in many forms. Learning a new language? Try solving the Rosetta Code problems. I believe some of the Basic Computer Science algorithms should be committed to memory – for example – being able to code merge sort quickly because you ‘know’ it rather than having to figure it out.  Practice writing Software Design Patterns and code idioms until they become second nature and intuitive.

Peer Review

Often you don’t know what you don’t know.  In these cases having someone else review your code can really open your eyes. However, in a work environment peer review often turns into a cursory sanity check, or after a while you have managed to learn all you can from the colleagues who typically review your work.  Venture into the world of open-source by making pull-requests.  This will introduce you to a wider community of developers where you have the chance to learn some new perspectives.

open sourceThere are so many good reasons to open source code.

  • Gain contributions from the wider community
  • Contribute back for all the awesome you’ve gotten from Open Source
  • To build the status of yourself or your company
  • Attract the best programmers
  • Get public feedback on the quality of your software
  • More people will use your software
  • open source reusable components actually get reused
  • Attract clients

With those great benefits for putting more open source code out there it still seems like a hard sale. Business types don’t always see the benefit to putting time and money into creating something only to turn around and give it away. “HEY!” they say “that’s valuable intellectual property”.

As developers we know the value of producing more open source code.  It is our job to convey that message as best we can to our clients, whether they be internal or external clients.

How do you identify part of a project that is a good candidate to open source?  Here’s the best criteria:

  • It’s not unique to the core business (ie Google shouldn’t open source their search engine)
  • It should be something that has some re-useability for other people or future projects
  • Should be small and single purpose.
  • It should be easy to understand and explain
  • Ideally it should be a package (cocoapod, gem, pypi, etc), or a service

If you are given a project and asked to design it’s architecture, estimate the cost or otherwise create an implementation plan you should take a moment to consider if there is any pieces of the project that could or should be made open source.  Isolating and open sourcing should add very little relative cost (you would have had to implement the functionality anyway) and you get the benefits of publishing open source mentioned earlier.  Try to up sell your client on open sourcing parts of their project.

In many cases open sourcing a piece of a larger project may be the best business decision to make.  MBA types just won’ t have an easy time grasping that concept so it could be a hard sell, but it’s often worth pushing for.

Keep a mindful eye and suggest an open source strategy on your next project.  You, your client and the wider community all benefit when new code is open sourced.

Free coding is the practice of writing code quickly off the top of your head.  It should be done as part of a daily ritual for at least 10 uninterrupted minutes.  The goal isn’t necessarily to produce something useful or even complete.  You should strive to open the taps of originating thoughts in your head and pouring them quickly into a text editor.

Here are five reasons you should be free coding:

Continue reading

Technology is getting more and more personal. As it does such it becomes easier to make things social in a real way.

Bringing a social experience to your application is more than just slapping a like button somewhere. Social is about making people aware that when they use your application they are entering a social situation.

A party is a real social situation. We know it’s a social event before we get to it. The host of a party tries to ensure that people engage in social behaviour such as chatting with each other, singing, dancing, and meeting new people by arranging for food, drinks and games to stir conversation.

Facebook is social in only a narrow sliver of what it means to be social in meatspace.

Now is the time to take a leap in what it means to create social software.

Creating software the provides a sense of presence, promotes social engagement, and enables social discovery and self expression is a real challenge from both a software and design perspective.  There will be new innovation required from both disciplines to get it right.

We are digging into this by experimenting within the scope of mobile games.  Mobile provides an interesting place because it is much more personal than sitting down at a computer in your office.  It allows us to go deeper than Facebook was able to go with their app when it was developed in 2004.

There is much work to be done yet in this area.  MMOs provide a glimpse at what should be possible to duplicate in mobile.  World of Warcraft and Second-Lift allowed people to meet and form relationships that became more intimate than would have happened outside of the games.  Technology has the potential to make social much more than it is in a physical setting.

Getting into the flow or zone as a coder is when you are your most productive and code spews from your fingers into your favourite text editor as fast as you can type. When in the zone you can seemingly pour out 100’s of lines of code in no time.

But how do you get into the zone other than by pure happenstance?

Authors have a technique called free writing which was popularized in Julia Cameron’s The Artist’s Way for which the rules are simple:

  • Select an amount of time: 5, 10, 20 minutes for example
  • Write continuosly and quickly for that amount of time ignoring spelling, grammar, formatting
  • If you can’t think of things to write, write about that or simply scribble. just keep things going from your brain to the page.
  • When your time is up, review what was written and mark anything interesting to explore further in a future free writing session

The goal of free writing is to build up enough momentum to blast through mental blocks and quickly transition to a state of flow for the real writing process. It is meant to be done daily as part of a morning routine before real work starts.

I’m curious if a similar technique could be used to get into the zone for coding. A quick Google search revealed no information about anyone exploring this idea so I thought I would.

Here are the rules I’m using for freecoding:

  1. Pick an amount of time before hand 5, 10, or 20 minutes for example
  2. Use a text editor where I can turn off syntax highlighting (to avoid the impulse to go back and fix typos)
  3. Code in a language that I know well enough to code without referring to documentation
  4. start coding and don’t stop typing. If you can’t think of something to code just code about that. it doesn’t have to run, it doesn’t have to make sense or be correct syntax. Try not to go back to code out of order.
  5. Don’t try to accomplish anything that will require you to stop to think. ie. don’t try to implement your favourite sorting algorithm.
  6. If you code yourself into a spot where you don’t know the API well enough, just fake it out with a call to a magical function that does what you want.

When time is up, turn on syntax highlighting and read what you wrote. Make notes about anything interesting that you’d like to explore again in the future. Note things that you were hung up on with syntax or APIs that you should brush up on.

The purpose of freecoding is to limber up your mind before starting real code. The secondary benefit is to identify gaps in your knowledge of a language or library where you might have to stop to look up the documentation.

Use the momentum built up in the freecoding session to launch into a real project. Try to keep the transition freecoding and real code somewhat smooth

As an example here’s part of what I did in my first 5 minute freecoding session:

a = 'hi there'
 
b = a.replace('hi', 'bye')
 
a. reverse()
 
c = fun(a)
 
def requestA(a):
    return GoogleThis(a)
 
if requstA(a) == 'no results':
    println("there's nothing in the google machine")
    return tryBingInstead(a)
 
requestA('matt warren')
 
b = b.capitalize()
b = b.reverse()
b = b.capitalize()
println(b)
 
not_sure_what_to_type = b
 
def star_trek_is_better_than_star_wars():
    return true
 
if star_trek_is_better_than_star_wars:
    print "hell yeah"

There’s nothing interesting going on here, it’s not complete thoughts and doesn’t accomplish anything nor does it run. the point is to simply start getting a flow of code coming by reducing any blocks that might prevent it (in this case by removing the need to create something that runs or even makes sense).

Upon reviewing this particular session I might decide that the concept of using bing as a backup for google is interesting, and that I need to increase my vocabulary of string functions to call upon.

In my limited trials so far it’s clear that there is a bit of a skill to freecoding successfully. For all other coding you do there is a goal to reach, something specific you want to implement. I suspect that after a couple weeks of regularly freecoding the code will start to get more complex, better structured, and longer thought streams of cohesive code segments.

I am still early on in my experimenting with this concept. However, if it is as effective as free writing is for authors, then perhaps free coding can be a daily ritual more of us can use to boost our productivity. I’d like to encourage every software developer to give it a try and report back (if they like) with how it worked or didn’t work for you.