After reading a reddit thread about interesting scripts people have written to automate things I was inspired to put together a quick web scraping script to check the sales at the LCBO and send me a message (on Telegram) if there is anything interesting.

Back in the day I would have done something like this with BeautifulSoup, but Scrapy is a full-on framework to help build these kinds of tools. It makes it easier to write a web scraping or spider script in a more normalized way and decompose the scraping into a more consise and easier to read functions.

This was also the first time I tried to integrate with Telegram as a bot to send myself messages.  Having built bots for Slack and Facebook Messenger in the past, Telegram’s integration was by far the simplest and quickest to get up and running.  A quick conversation with @BotFather and I had my bot created and a key to connect with.  Using the twx.botapi package I was sending messages in just 3 lines of code.

This past week I had a chance to work on a conversational UI.  It’s a neat way to interact with software and presents an interesting set of problems to implement.

Before even starting to build something I needed to decide on some tools to help with managing the conversations. There are a small number of libraries and services that can help with the natural language processing which would normally present a technical hurdle.

I evaluated 3 options:

Wit_—_landing.pngwit.ai is a service owned by Facebook that provides some very impressive ways to train a language parser to categorize and extract information from sentences.  It operates as a web service that if you send it a message it will return with a confidence measure and any extracted values.  It provides a level of fuzziness in it’s matching and as a web service it allows you application to improve over time without deploying new code.

Conversational_User_Experience_Platform_for_Web__Mobile_and_IoT_-_Api_ai

Api.ai is a similar Saas based solution that provides a way to convert a message into a more easily actionable format.  It has slightly different semantics than wit.ai in how you describe the message parsing.

RiveScript – is the option I decided to work with. It specifies what is comparable to a markdown type language for defining messages and responses. Unlike wit.ai or api.ai it is more simplistic in the language parsing aspects (closer to regular expression matching).  But has the nice property that it doesn’t rely on an external service. Implementations have been written in a bunch of languages which make it easy to get started writing bots.

For those of us lucky enough to be able to attend PyCon this year in Portland it’s a chance to meet and learn a ton about what’s about going on the wide world of Python.

For everyone else all the amazing talks and tutorials from PyCon are quickly being posted to YouTube, where they can be viewed at your leisure.  The amazing volunteers at PyCon are quickly processing and uploading the videos just hours after the talks are given so it’s easy for anyone to follow along.

Check out the PyCon 2016 Youtube Channel 

I’m going to be watching all the talks that I couldn’t get to see in Portland.  There’s just so much good stuff there to learn.  Here’s some good talks to start with:

Tomorrow I head to Cancun for a corporate retreat and will be taking advantage of the time away from my usual evening commitments (family) to catch up on my backlog of books.

What I’m primarily interested in right now is getting a better grasp of machine learning concepts.

 

I’m happy that Python has really become the go to language for doing machine learning. It’s one less thing to worry about having to learn myself when trying to really get a grasp of the concepts and putting it into practice.

There seems to be a huge boom in the demand for developers with the skills to develop machine learning algorithms. The recent Google IO keynote was notable in that the CEO talked of nothing other than Machine Learning and AI being the focus for Google in the coming years. It’s expected that by 2018 demand for experts in statistics and machine learning will exceed demand by at least 140,000. As a result people with the skills will have their choice of amazing opportunities.

This book is currently on sale for just $3 and it’s a hell of a deal.  I picked this up after hearing it referenced from several different groups of people in the span of a week, enough to make me curious.

“The obstacle is the way” presents an argument that much of what prevents us from achieving our goals in life is the perspective we apply to the objective reality of any given situation.  A challenging business negotiation can be seen as a stressful make or break deal, or a chance to further refine your negotiation skills.  The reality is that if you have one business negotiation it’s likely that you’ll have more in the future, thus taking the positive perspective of the situation will position you better for the next one regardless of the outcome.

When each obstacle you encounter can be re-framed into a learning experience, then everything compounds into more and more experience and skill.

The common approach that people take to a challenge is to give in – we say it can’t be done or we get angry when we try and fail.  There may be some learned helplessness, or emotional reactions that prevent clear thinking in these situations.  But stopping to get an external view can sometimes shead a lot of light.  We find it easy to suggest fixes to other people’s problems but often have a hard time thinking objectively about our own.

The book is a quick read, and worth checking out.

At the beginning of the year I, like many other people, set some goals for the year.  I also took the step of putting a reminder in my calendar to revisit my goals list every month to assess my progress.

After a couple of cycles of that I recognized the pattern, I had inadvertently applied a piece of the agile development method to my goals.

In typical agile we set a small set of attainable near term goals and work over the course of 1-2 weeks to accomplish them.  Then at the end of that sprint we pause to assess and examine 3 questions:

  1. what went well?
  2. what didn’t go well?
  3. what can be improved for the next sprint?

Small stretches of effort separated by pauses to assess and repoint are a fantastic way to reach a goal whether it is sailing across the ocean, building a software project or losing 10lbs.

The time to revisit and refocus allows you to keep all your goals front of mind and gives you a chance to re-evaluate if they were reasonable goals at all to begin with.

One of my goals was to be able to do 50 pull-ups.  After several weeks of exercise and research on training plans I’ve come to the conclusion that 50 is just an insanely unrealistic number.  So in light of this new perspective I’m re-targetting 20 reps.

Chat is becoming the next platform. Interacting with computers through channels that you already use to talk to people (Facebook Messenger, SMS, Slack etc) in a way that feels natural is a powerful way to accomplish many things for which web pages, mobile apps, desktop applications, or commandline scripts are cumbersome.

Chat is not a panacea for all user experience problems but there are many day to day things that might be able to transition to this way of interacting with computers or new ways that as yet haven’t been able to work.

In some ways this is evidence of more encroachment of technology into taking ever more jobs from humans.

Facebook has a bot they run on Messenger (available limited to Silicon Valley area) which can answer ‘any’ question and perform any number of activities on your behalf. Right now it is operating on a hybrid model. There is an ever growing list of queries for which it can automatically handle, and anything which the AI is not yet able to handle is directed back to an actual person to do. “what is a good pizza place nearby?” can be sourced from Yelp reviews. “Can you call and reserve a table for this evening” -> maybe can’t be handled automatically and will go to an actual person to do. As development continues the features that can be handled will be incrementally expanded.

I have a service running to maintain my healthy habits. Persistence is an app that messages me to do things when it can tell that I haven’t. It’s like a dependable friend that holds me accountable to my goals. There’s no reason why all this can’t happen through a communicative medium like chat. There’s no reason why this couldn’t feel like chatting to an actual friend.

After playing with the Slack and Facebook APIs around chat, it’s proving to be both more difficult and more interesting than I imagined. Difficult because in these early days creating interfaces that process natural language and conversation threads is missing a lot of tooling. Interesting because this is all so new that you really get a chance to build something that no one has really thought of yet. This is a new platform and there is still much to learn.

A number of things have come up in the last few weeks for a project I’m involved in that has shaken my beliefs and shifted my perspective.

The team I work with have all developed a deep level of experience within their domain of expertise.  This is part of the competitive advantage we use to win clients.  Smart people with a tightly focused specialization can work faster and produce better work than a generalist that hops to the new hotness on every other project.

The problem is that these specializations prevent individual developers from breaking out of their circle of perceived experience without great effort.  Specialization codifies itself into the how the team functions, who can work on what and who cannot work on what.

When a problem that demands some technology or tools which are outside of the normal comes up it can throw a wrench in things. Sometimes there are justifiable questions – can we support a solution written in Lua if the original developer leaves?  Other times it can devolve into a rather insulting “‘We’ don’t know how to do that” for something that can be learned in a few hours of reading or working through a tutorial.

The perceived difficulty and risk of something new can prompt only the most senior developers to get assigned to work on new things.

In the past at various different jobs we did one of two things.  We noticed that the existing technology stack was no longer meeting our needs so we evaluated some alternatives and then everyone was given some insight into the decision, how it was made and then either:

  1. Everyone was given training to get up to the same level of proficiency at the same time
  2. A hard cut over to the new project on a new stack and forced everyone to pick it up quickly on their own

Most good developers have no problem picking up a new language quickly. A significant portion of the knowlege you have as a Computer Science or Software Engineer is not tied to the syntax of a particular language.

Part of a good education in Computer Science is experience with a wide range of types of applications.  I did AI algorithms, wrote a real-time operating system, OpenGL and ray-tracing,  web applications along with the basics of algorithms and data structures.  Part of being a great developer is having the breadth of experience to know when to apply certain technologies over others.

A perceived specialization can negate all that past experience, and hinder an individual’s opportunity to tackle new challenges.  It demands a balance that must weigh company goals and efficiency won from deep expertise with the desires of each individual developer to work on interesting things and continue to learn.

 

One thing has come crystal clear over the last couple of weeks.  Distractions can easily destroy your productivity.

I remember fondly the days when I would write hundreds of lines of code per day.  Find a problem to work on and focus intensely on it for hours. Churning out features and fixes at a steady pace.

These last few weeks it feels like I barely manage to write 10’s of lines per day. Juggling multiple projects, getting messages on slack, emails,  and meetings all add up to a massive amount of context switching, loss of focus and limted progress.

The scary thing is just how effective these distractions are at killing productivity.  Judging by how much software I am writing now vs how much I used to write each day, these distractions have reduced my productivity by 10x.  What would have taken a good focused day now takes a week.

It’s clear that something needs to change.

I have slowly started to add some distraction reducing software to my toolbox.

Concentration – this is a python script which will block time sucking websites.

For Slack, the unread notification is a constant reminder that draws me back to read communications.  I changed the settings: https://get.slack.help/hc/en-us/articles/201675007-App-icon-notifications to hide this notification.

Email notifications similary are a way to keep drawing me back to gmail. I turned off email notifications to my phone and try to check email no more than 3 times per day.

400x40000bb.png Windy – an iOS app that I bought years ago and still use regularly when I need to drown out any noise.

 

Ah algorithms, that fundamental part of computer science education (and job interviews) which seem to very rarely come up in real world, day to day software development.

Sometimes I have a  real yearning to tackle a technical problem that involves writing an implementation of a red-black tree or a bucket sort, or hell, just write a function that uses recursion.  These opportunities come up so infrequently that when they do it’s a real pleasure to be savored.

It’s worth making those moments happen if they don’t come up very often at work.  Go out of your way to find projects that create the challenges you want to take on.  It’ll really remind you about what you love about writing software.