Progressive Enhancement

I recently sent this as email to a colleague. You’ll be glad to know we sorted things out :0)

The little chat we had last week about AngularJS has been playing on my mind. As you know I’m not against JavaScript (I love it, and have even written an Open Source JavaScript library) and I have a personal project partly planned to try out AngularJS, which looks awesome.

However, your comment that “progressive enhancement is one way to do it” (emphasis mine) bothers me. A lot. I’ve heard this attitude from a lot of developers, and I believe it’s wrong. Not because I believe every website or app (more on the difference between those two things later) should never use JavaScript, but because it ignores the fundamental layers of the web which cry out for a progressive enhancement approach. These layers are:

image004

Here’s how a browser works (more detail).

  1. It requests a URL and receives an HTML document
  2. It parses that document and finds the assets it needs to download: images and other media, CSS, JavaScript
  3. It downloads these assets (there are rules and constraints about how this happens which vary slightly from browser to browser, but here’s the basics):
    1. CSS files get downloaded and parsed generally very quickly, meaning the page is styled
    2. JavaScript files get downloaded one-by-one, parsed then executed in turn
    3. Images and other media files are downloaded

Let’s look at the absolute fundamental layer: the HTML document.

HTML

Way back in the beginning of the web there was only the HTML document on a web page; no CSS, no JavaScript (very early HTML didn’t even have images). In fact without HTML there is no web page at all: it’s the purpose of HTML to make a web page real. And a set of URLs, for example the URLs under a domain like my-site.com, which doesn’t return at least one HTML document is arguably not a website.

Yes, URLs can also be used to download other resources such as images, PDF files, videos etc. But a website is a website because it serves HTML documents, even if those document just act as indexes of the URLs of other (types of) resources.

HTML is the fundamental building block of the web, and software which can’t parse and render HTML (we’ve not even got to CSS or JavaScript yet) can’t call itself a web browser. That would be like software which can’t open a .txt file calling itself a text editor, or software which doesn’t understand .jpg files calling itself an image editor. So we have software – web browsers – which use URLs to parse and render HTML. That is the fundamental, non-negotiable, base layer of a web page, and therefore the web.

One of the great things about all HTML parsing engines is they are very forgiving about the HTML they read. If you make a mistake in the HTML (leave out a closing tag, whatever) they will do their best to render the entire page anyway. It’s not like XML where one mistake will make the whole document invalid. In fact that’s one of the main reasons why XHTML lost in favour of the looser HTML5 standard – because when XHTML was served with its proper MIME type a single syntax mistake would make the page invalid.

And if a web browser encounters elements or attributes it doesn’t recognise it will just carry on. This is the basis on which Web Components are built.

So even serving a broken HTML page, or one with unknown elements or attributes, will probably result in a readable document.

CSS

The next layer is CSS. At this point we’ve left the fundamentals and are onto optional components. That’s right – for a working web page CSS is optional! The page might not *look* very nice, but a well-structured page of HTML will be entirely usable even with no styling, in the same way that water is entirely drinkable without any flavourings.

But most – but not every – browser supports CSS (the basics of it, at least), so why do I treat it as optional? There are a few reasons. Firstly CSS might be turned off (which is very unlikely, but possible). But more importantly the CSS file might not be available or parsable:

  • The server hosting the CSS file may be offline
  • The file may be temporarily unreadable
  • The URL for the file may be wrong
  • DNS settings may be incorrect
  • The CDN may be down
  • The file may be empty
  • The file contains something that isn’t CSS or a syntax error

There may be lots of other problems, I’m sure you can think of some.

Now, any one of those errors could also be a problem with an HTML document. In fact if the document is being served from a CMS then there are a lot more things that could go wrong. But if any of those errors happen for an HTML document then the browser doesn’t have a web page to parse and render at all. Browsers have mechanisms to handle that, because everyone knows that URLs change all the time (even if they shouldn’t):

image007

So CSS is optional; it is layered on top of the fundamental layer (HTML) to provide additional benefits to the user – a nicely styled page.

And in the same way that web browsers are forgiving about the HTML they parse, they are also forgiving about CSS. You can serve a CSS file with syntax errors and the parts the rendering engine *can* parse correctly it will, and will ignore the rest.

So if an HTML document links to a CSS file which contains partially broken syntax the page will still be partially styled.

JavaScript

Now we come to the top layer, the client-side script.

I don’t need to tell you that JavaScript is sexy at the moment, and rightly so – it is a powerful and fun language. And the fact it’s built into every modern web browser arguably gives it a reach far wider than any other development platform. Even Microsoft are betting their house on it, with Windows 8 apps built on HTML and JavaScript.

But what happens when JavaScript Goes Bad on a web page? Here’s the same quick list of errors for a CSS file I wrote about above:

  • The server hosting the JS file may be offline
  • The file may be temporarily unreadable
  • The URL for the file may be wrong
  • DNS settings may be incorrect
  • The CDN may be down
  • The file may be empty
  • The file contains something that isn’t JavaScript or a syntax error

Let’s stop right there and look at the final error. What happens if a browser is parsing JavaScript and finds a syntax error (not quite all errors, but a lot) or something it can’t execute? It dies, that’s what happens:

“JavaScript has a surprisingly simple way of dealing with errors ? it just gives up and fails silently”

Browser JavaScript parsing engines are still pretty forgiving, but *way* less forgiving than HTML and CSS parsers. And different browsers forgive different things. Now, you might think “Just don’t have syntax errors” but the reality is bugs happen. Even something as innocuous as a trailing comma in an array will cause older Internet Explorer to fail, but not other browsers.

So JavaScript, while powerful and cool, is also brittle and can easily break a page. Therefore it *has to be optional*.

Web apps

You might be thinking “Yes, but my site is a Web Application, so I need JavaScript.” OK, but what is a web app exactly? I can’t say it better than Jeremy Keith, so I’ll just link to his article.

Progressive enhancement

This is the crux of progressive enhancement. Here’s the recipe:

  • Start with a basic web page that functions with nothing but the HTML, using standard semantic mark-up. Yes, it will require full-page GET or POST requests to perform actions, but we’re not going to stop there – we’re going to enhance the page.
  • Add CSS to style the page nicely; go to town with CSS3 animations if you want
  • Add JavaScript to enhance the UI and provide all the modern goodies: AJAX, client-side models, client-side validation etc

The benefits are obvious:

  • If the JavaScript or CSS files (or both) fail for any reason whatsoever the page still works
  • The use of semantic HTML means the page is fully understandable by search engine spiders
  • Because everything is rendered in HTML, not built up in JavaScript, it is understandable immediately by assistive devices
  • Serving fully-rendered HTML is quicker than building that same HTML client-side
  • Built-in support for older – and newer – browsers and devices

The best web developers on the planet all argue that progressive enhancement is the best way to approach web development. I honestly have no idea why anyone would think otherwise. There’s a good article (it’s actually the first chapter of Filament Group’s “Designing With Progressive Enhancement” book) on the case for progressive enhancement here.

Modern JavaScript

There are some people who use juicy headlines (this is called “link-baiting”) which doesn’t help those developers who are trying to promote progressive enhancement, instead causing JavaScript-loving developers to proclaim “you hate JavaScript!”. You know what developers are like: they get hot-headed. It’s much better to try to think clearly and objectively about development and come up with solutions based on real data.

The reality is that most modern JavaScript libraries don’t support progressive enhancement out of the box: AngularJS included. If there were a way to render standard HTML with real links and forms and then enhance that HTML with Angular I would be all over it, but unfortunately I haven’t found anything that explains how to do it yet.

This is something which I’ve been thinking about a lot, and I did a little proof of concept for a basic data-binding system. I wonder if it would be possible to apply the same techniques to Angular.

For me personally I’m a big believer in progressive enhancement, not just for accessibility reasons but for front-end performance as well. I do recognise it will probably add time to development, however the same can be said for Test Driven Development. The goal for progressive enhancement and TDD is the same: a better, more stable foundation for systems.

New WordPress plugins

I’ve released a couple of new WordPress plugins recently which I thought I’d waffle on about.

Theme Reset

I had a situation not too long ago on a WordPress MultiSite site I run where I had deleted some themes but there were still some sites using those themes. I needed to reset all the sites to use the same theme, but there wasn’t an easy way to do it. So I made a plugin.

And here it is: Theme Reset. There’s not much to it; you have to be a network admin to get the option, and you can choose any installed theme. That’s it.

Child Themes

The other plugin I released is also theme related. This one allows you to create a child theme from any installed theme. Just click “Create child theme” on the theme you want to be a parent, fill in a simple form and boom – the new child theme is created and installed.

screenshot-1

screenshot-2

screenshot-3

screenshot-4

I’m no designer (as you can probably tell) but this seemed like a good idea that could save people some time.

Namespaced SQL Columns in Queries

One of the things I really like about the C# language is the wide range of database ORM and data mapper packages available. In a nutshell an ORM can help remove the need to write SQL queries and stored procedures in your database, instead you tell it what the “model” of your data is (i.e. the schema of the tables and how they relate to each other) and the ORM constructs SQL queries to fetch, update and add data. Data mappers do much the same thing.

A simple example would probably be helpful. This is psuedo-code, it’s not from a real ORM:

// create the connection to the database which we have modelled
MyDatabaseModel myDb = new MyDatabaseModel("localhost", "username", "password");

// get all records from the MyTableType table and cast them to MyTableType objects
IEnumerable allTheThings = myDb.MyTableType.Get();

// create a new MyTableType object
MyTableType newThing = new MyTableType {
Column1 = "Hello";
Column2 = "World";
};

// insert it into the database
myDb.MyTableType.Insert(newThing);

You can imagine this saves a LOT of time and effort, particularly for applications with a simple database schema. I’ve used several ORMs including Entity Framework (big, powerful, complex) and Linq2SQL (small, quick, easy, only works with SQL Server). I do want to try some others, in particular Rob Conery’s massive.

In PHP there are ORM packages available, too, although I admit I’ve not used any of them. Sometimes, however, you need to inject a bit of cleverness into your database access code to help speed up development, but don’t have time (or inclination) to rewrite the system around an ORM.

Repetitive SQL

This was the situation I found myself in the other night. I had a medium-sized application with lots of queries, and some tables that were being accessed a lot. I found I had very repetitive SQL strings like this:

select t1.col1, t1.col2, t2.col1, t2.col2
from table1 t1
inner join table2 t2 on t2.parentid = t1.id

Fortunately I’d put all the queries into two sets of files:

  • The classes for my types, each one of which had its own Get(), Save() and Delete() methods
  • Manager classes which handle loading of lists of object, or loading sets of related objects

The problem was I had used the same column names across many tables. This code:

select t1.col1, t1.col2, t2.col1, t2.col2

Gives a set of results that looks like this:

col1 col2 col1 col2
Table 1, Row 1, Col 1 Table 1, Row 2, Col 2 Table 2, Row 1, Col 1 Table 2, Row 2, Col 2

Which “col1” column belongs to table1, and which to table2? I was in a pickle, and no mistake. What I needed was a way to namespace the columns I returned.

Namespacing

Namespacing is simply a way to group related functionality together in a system. You can imagine a folder structure on a hard drive being like many namespaces:

/My Documents/
/My Documents/Photos/
/My Documents/Photos/Holidays/
/My Documents/Music/
/My Documents/Music/County-AND-Western/

You know where all the music of both types (country AND western) are because the folder structure clearly shows you the way.

In C# the different levels in a namespace are separated by a dot, for example:

MyApp.UI.Helpers.DateHelpers

I thought I could use this same naming convention for my queries. Also the SQL language itself uses dots to denote the boundary between a table and its columns:

select table1.*
from table1
where table1.id = 123

After a bit of experimentation I found that as long as the column names were put in quotes I could return pretty much anything I wanted. For example:

select t1.id as 'this.is.my.test'
from table1 t1

So I realised I could namespace all the columns by the table they belong to, like this:

select
t1.col1 as 'table1.col1', t1.col2 as 'table1.col2',
t2.col1 as 'table2.col1', t2.col2 as 'table2.col2'
from table1 t1
inner join table2 t2 on t2.parentid = t1.id

Now the result set I get back looks like this:

table1.col1 table1.col2 table2.col1 table2.col2
Table 1, Row 1, Col 1 Table 1, Row 2, Col 2 Table 2, Row 1, Col 1 Table 2, Row 2, Col 2

Much better! I can query the columns for each table safely.

Making namespaced column queries easy to use

Now I’d proved the concept I did some work to make it easier to use these namespaced columns. Firstly I added a new method to each of my classes which returned a string which I could use in queries.

As this application is a plugin for WordPress I am using the global $wpdb object to do my database access. Here is PHP for a small, simple class:

class MyType {

// the properties of this class
var $id;
var $name;

// returns a string for all the columns in this class
function ColumnQuery( $prefix ) {
$s = "%1.id as '%2.id', %1.name as '%2.name'";
$s = str_replace( "%1", $prefix, $s );
$s = str_replace( "%2", "MyType", $s );
return $s;
}

// loads the row with the given ID
function Load( $id ) {
global $wpdb;
$sql = $wpdb->prepare( "select
" . MyType::ColumnQuery( "t1" ) . "
from {$wpdb->prefix}MyType t1
where b.id = %d;",
$id );
$row = $wpdb->get_row( $sql, ARRAY_A );
$this->MapRow( $row );
return $this;
}

// maps a database row to the current object
function MapRow( $row ) {
$this->id = $row["MyType.id"];
$this->name = $row["MyType.name"];
}
}

// create a new instance of a MyType object
$myType = new MyType();

// populate the details of the $myType variable with the details from row 123
$myType.Load(123);

One really important think to note is that I use the ARRAY_A parameter in the $wpdb->get_row() method. This means that the columns are returned as an associative array, which is the only way I could access the namespaced columns because of the dots in them.

Something has just been dropped in my brain…

Actually, and this has only just struck me, I could probably use a different character than a dot and use the default column methods. Here’s quick psuedo-code using underscores instead of dots:

// returns a string for all the columns in this class
function ColumnQuery( $prefix ) {
$s = "%1.id as '%2_id', %1.name as '%2_name'";
$s = str_replace( "%1", $prefix, $s );
$s = str_replace( "%2", "MyType", $s );
return $s;
}

// maps a database row to the current object
function MapRow( $row ) {
$this->id = $row->MyType_id;
$this->name = $row->MyType_name;
}

Hmm, I might have to go back and do that as it’s a bit cleaner…

Table variables

The $prefix parameter in the ColumnQuery method means I can set what the variable for the table is, for example:

$sql = "select
" . MyType::ColumnQuery( "t1" ) . "
from {$wpdb->prefix}MyType t1";

In this case the table variable is “t1”. The resulting SQL string will be:

select t1.id as 'MyType.id', t1.name as 'MyType.name'
from MyType t1

Related tables

That’s great for single tables, but the real power comes when you use this structure for multiple related tables.

Imagine we have two related tables like this:

  1. Child
  2. Parent

These tables are related using a foreign key; a parent_id column in the Child table which stored the id of the Parent.

Whenever we load a Child object we also want to know the details of the parent. This is the kind of thing that an ORM system eats for breakfast. Using our new namespaced column query it’s also pretty easy. Here are the classes for the Child and Parent type:

class Child {

// the properties of this class
var $id;
var $name;
var $parent_id;
var $parent;

// returns a string for all the columns in this class
function ColumnQuery( $prefix ) {
$s = "%1.id as '%2.id', %1.name as '%2.name', %1.parent_id as '%2.parent_id'";
$s = str_replace( "%1", $prefix, $s );
$s = str_replace( "%2", "Child", $s );
return $s;
}

// maps a database row to the current object
function MapRow( $row ) {
$this->id = $row["Child.id"];
$this->name = $row["Child.name"];
$this->parent_id = $row["Child.parent_id"];

// if we have columns from the parent table then populate the parent object
if ( $row["Parent.id"] != "" ) {
$this->Parent = new Parent();
$this->Parent->MapRow( $row );
}
}
}

class Child {

// the properties of this class
var $id;
var $name;

// returns a string for all the columns in this class
function ColumnQuery( $prefix ) {
$s = "%1.id as '%2.id', %1.name as '%2.name'";
$s = str_replace( "%1", $prefix, $s );
$s = str_replace( "%2", "Parent", $s );
return $s;
}

// maps a database row to the current object
function MapRow( $row ) {
$this->id = $row["Parent.id"];
$this->name = $row["Parent.name"];
}
}

This is the magic bit:

// if we have columns from the parent table then populate the parent object
if ( $row["Parent.id"] != "" ) {
$this->Parent = new Parent();
$this->Parent->MapRow( $row );
}

If the system detects that we have columns from the parent table then we can fully populate the parent property. Here’s the Load() method for the Child class that would load and populate a Child instance with the details of its Parent:

class Child {
function Load( $id ) {
global $wpdb;
$sql = $wpdb->prepare( "select
" . Child::ColumnQuery( "c" ) . ",
" . Parent::ColumnQuery( "p" ) . "
from {$wpdb->prefix}Child c
inner join {$wpdb->prefix}Parent p on p.id = c.parent_id
where c.id = %d;",
$id );
$row = $wpdb->get_row( $sql, ARRAY_A );
$this->MapRow( $row );
return $this;
}

So we would load a child object like this:

$child = new Child();
$child->Load( 123 );

And we can access details of the child and it’s parent like this:

print "ID: " . $child->id . "
";
print "Name: " . $child->name . "
";
print "Parent Name: " . $child->parent->name;

The same approach can be used when loading multiple rows. For example here’s some SQL to load all children:

$sql = $wpdb->prepare( "select
" . Child::ColumnQuery( "c" ) . ",
" . Parent::ColumnQuery( "p" ) . "
from {$wpdb->prefix}Child c
inner join {$wpdb->prefix}Parent p on p.id = c.parent_id;";

Wrapping up

This technique helped me out of a little hole this week, hopefully it can help you as well.

Responsified and updateified

That old proverb about the cobblers children going barefoot is true. For too long this site has been neglected and unloved. Well, no more! I’ve realigned (not redesigned) this site to take advantage of modern web development techniques. Of course, I’m talking about responsive design.

It’s not a radical departure from what was there before, and talking advantage of the excellent Bones starter theme meant the whole thing was done in a few hours. And even though I haven’t paid too much attention to performance I’ve still ended up with a respectable payload for the homepage:

Homepage payload

And even YSlow gives me a B grade:

YSlow grade B

I’ve also updated to WordPress 3.7, named in honour of the great Count Basie. The update was, it has to be said, not as slick as usual. I had “out of memory” problems which weren’t that hard to fix, but did cause me some stress.

WordPress 3.7 is the first version with auto-updating built in, so hopefully my installation will keep itself up to date and in tip-top condition while I’m asleep.

Bad, good and great programmers

A recent tweet by Cory House started my brain a-twitching:

One of my new favourite interview questions: What is the difference between a bad, good and great programmer? Which are you and why?

I’ve worked with all three kinds, and thinking about the question of what the differences are I realised (not for the first time) that pure coding skill is nowhere near all the story. This blog post explores what else may put you in one category or another. First some assumptions:

  • As this is an interview question I took “programmer” to mean “one who works in a software development team”
  • The basics of being a decent employee – turning up on time, dressed appropriately and at least partially sober – are a given
  • The programmer can actually code to a decent level, “decent” being whatever is appropriate to fulfil the job requirements
  • The employer doesn’t just want a code monkey. I hope and trust that there are very few companies who want programmers to just “shut up and write the code”, and if you find yourself working for one of those then get out as soon as you can.
  • I’m talking to traditional programmers (C#, PHP, Python, Ruby, JavaScript etc) rather than front-end (HTML, CSS) but many of the points are applicable for any type of developer

Bad programmers

Generally, bad programmers are easy to spot. They write code that generally works (if they didn’t they’d be out of a job), but suffer from many or all of these issues:

  • They have a bad attitude towards non-geeks, whether non-technical colleagues or users
  • They have a bad attitude towards other programmers whose views don’t closely match their own
  • They know it all, or at least everything that matters
  • They are opinionated about tech; either wanting to use cutting edge stuff because it’s cool (whether or not it gets the job done) or they only trust very established things and refuse to look at new tech
  • Their communication skills are poor, and they have little desire to improve
  • They may fulfil specification requirements but don’t go any further than coding for exactly what is written
  • Their sense of humour is non-existent, or is harsh and rude
  • They don’t ask questions, and don’t make suggestions

It’s important to take all these points on balance. Some people may be shy, which could mean they don’t ask questions or make suggestions. But a good manager or lead developer will help staff to fulfil their potential, for example by encouraging the quieter members of the team to contribute.

The code from a bad programmer may also have these hallmarks:

  • Lack of comments, or comments that state the obvious and don’t tell you *why* they coded something a particular way
  • Obtuse, gnarly, over-abstracted, over-complicated architecture (at any level: method, class, subsystem or app)
  • Inconsistent or otherwise poorly-considered naming of variables, methods and classes
  • Lack of attention to detail, such as proper sanitisation of inputs for security
  • Copy-and-paste-itis. Bonus points if it’s clear they didn’t understand how the copied code works.

You get the picture. We’ve all written code like this at some point and we should be careful not to think of a programmer with less experience, or one who is struggling with a complex task, as a bad programmer. Bad programmers have a bad attitude, and even if they can mask that attitude for a while, eventually the mask will slip.

Good programmers

Good programmers, like bad, are known by their attitude as well as their technical skill. Note that a good programmer may not have as powerful code-fu as a bad programmer, but their attitude and “soft” skills more than make up for any technical shortcoming. (This is not to say that someone who doesn’t know XML from XTC can be classed as a good programmer; the basics mentioned above still apply.)

You’ll know a good programmer because:

  • They have a good, helpful and professional attitude towards colleagues and customers/users
  • They have respect for other programmers and are willing to listen – and when necessary respectfully disagree – to other points of view
  • They have a good sense of humour and contribute to a good atmosphere in the team
  • They exhibit attention to detail, and can spot the holes and inconsistencies in a specification
  • They are respected and liked by other members of the team
  • They don’t make rash decisions and have learned to weigh the pros and cons of different technologies
  • They can put themselves into the users’ shoes and make good decisions about how their code affects the user interface and experience
  • They have good communication skills, and avoid jargon wherever possible when communicating non-technical people

These kind of people make a team better, and so help an organisation to be better. But it’s not just soft skills, they also have a good grasp of technology:

  • Their code is readable and well-formatted, adhering to any coding standards adopted by the team
  • They choose sensible, descriptive and unambiguous names for classes, methods, properties etc
  • Their comments are actually useful: not just telling you what a method does, but why it is used – and any non-obvious things you may need to watch out for
  • Their architecture choices (at method, class, subsystem or any other app level) are clear, well-abstracted and only as complicated as it needs to be. My rule of thumb for “good” architecture: either you would have written it the same way, or you see why they did it their way and don’t need to change it much.

No doubt anyone with a bit of experience working in a development team can think of several more hallmarks of good and bad programmers. But what about great programmers?

Great programmers

While many people would roughly agree what makes a programmer good or bad, deciding what criteria to use to distinguish the greats ones is much more subjective. Here are a few points I can think of that, in addition to everything I noted for a good programmer, would class a programmer as great:

  • They are aware not just of the users’ needs and the specification, but the organisational environment. They can guide the development of an application so that it serves wider organisational purposes: in the case of a business to serve users and be commercially successful.
  • Their communication and “people” skills are highly developed; for example they can explain complex technical concepts to non-geeks in an understandable, even entertaining, way
  • They are known in the team as the person who will help you out with a problem: partly because they have the experience and knowledge, but also because they make themselves available to help others
  • They keep themselves up to date with technology – not just their own area of expertise but the wider world of development, UI, UX, project management and business
  • Their thinking in terms of abstracting code takes into consideration not just the next 3 months but the next 3 years (that *almost* means they can tell the future)

In short a great programmer has an expansive view of technology, of people, of the future of the organisation and applications they work on. They are part psychologist, part entrepreneur, part designer, part storyteller.

A couple of years or so experience and taking pride in what you do will take you from being a bad programmer to being a good one. But to be a great programmer takes a special kind of person. The kind that, eventually, will do something really special. Like write a great book, start a great company, write an amazing Open Source project.

There’s lots I haven’t mentioned here (such as writing testable code).For more on this subject check out these links: some are articles and essays, some are great programmers.