Day & Night Phones (Or: Why I Carry Two Mobile Phones)

I carry two mobile phones, for most of the week. I have a personal phone which I use to call friends and family, to read my personal e-mail, access social media, and play games. My personal phone is almost always in my pocket, unless I'm in a work meeting or giving a presentation.

I also have a work phone. It is only used by clients or co-workers to call me about work-related matters, to access my work e-mail, and access our internal CRM systems. The nature of my job means that nobody typically calls me outside of working hours - so my work phone is in my pocket during the day, but very rarely at night.

Often, somebody will see two phones on my desk and ask the obvious question: "isn't one enough?" The recent talk about Silicon Valley executives carrying two phones got me thinking about this a little more.

There's no doubt that I'd prefer to only have one phone, however it isn't that big a deal. My work and personal lives rarely overlap, so I don't miss calls. Most of my work involves sitting at a desk with my phones on the desk, so I don't constantly have pockets full of mobile phones.

So why do I do it?

I want two phone numbers for the same reason I have two e-mail accounts - one for play, one for work.

My father has recently left his job. He's been there for over 20 years, so his work e-mail address was his first e-mail address, and his work-provided mobile phone was his first portable phone. Whenever somebody asked for his phone number or e-mail address, he gave out his work details.

Now, as he moves into retirement, he's left with the burden of updating his e-mail address with every person and organisation he wants to keep an online relationship with - from Facebook to overseas friends to his newspaper subscription. He's lucky that he's able to keep his phone number, so that's one less thing to change.

I won't be so lucky. When I move jobs, I'll most likely stay in an industry close to what I currently do - so my employer is not likely to let me keep access to my e-mail account or to let me take my phone number, along with all of the contacts and business benefits it brings.

As part of the generation who got mobile phones and e-mail addresses before getting a job, I didn't want to go through that hassle when I move on from my current job. Now, watching my dad go through the process as he wraps up his career, I'm glad I made that choice.

And, if I can let you in on a secret: Sometimes I carry three phones, so I can test stuff out for work. I'm the sole user of no less than five 3G/4G devices. Connectivity is actually rather useful.

Tools I Can't Live Without (2013 Edition)

I'm not one to jump from productivity tool to tool to try and become 5% better. In fact I use very few such tools, preferring to keep things simple instead. However over the past few years I've collected a short list of extremely valuable tools that I think it's safe to say I cannot comfortably work without. I often recommend one or more of these to friends and colleagues, so I figured I'd write up the list here for future use.

My day job and my hobbies are very different, however I use most of these tools for both. My hardware consists of a Windows 7 laptop for work, a Windows 7 desktop for home (hobbies, games, and other non-work stuff), and an iPad & iPhone (used for work & play). I also carry a Windows Phone 7.5 for work use.

Some context: I spend my day customising Microsoft Dynamics CRM for clients, which requires very little code cutting. When it does, it is usually plain JavaScript. I also write significant volumes of documentation, in MS Word. Say what you like about Word, but it's synchronisation and multi-user editing with SharePoint is fantastic. I also blog, both at work and at home. I take lots of photos, mostly of my kid, and don't share most of them online.

I've used every one of these for at least a year, in most cases many years. Where possible I pay for services so they don't disappear on me.


I distrust Google with my e-mail for various reasons, and I really dislike seeing ads next to my e-mail. FastMail (now owned by Opera) is my go-to for e-mail services. They give me multiple domain names with multiple aliases, a shared address book with my wife, and great IMAP support over any port I like (great for networks that specifically filter IMAP/SMTP). There's also alternate logins and two-factor authentication.

I am more than happy to pay the few bucks a month this costs me for what really is a premium e-mail service.

Tip: With a Family or Business account, you can share address books and files between users.


Evernote Screenshot

Evernote sucks as a text editor, but it's a great way to store snippets of text across multiple systems. I use it for note-taking in meetings, and I use it to write most of my blog posts (so that I can add to & edit on my iPad or work laptop).

The tagging & classification is really useful - I often find myself using Evernote to refer back to previous notes to make sure I haven't missed anything. My notes are kept in-sync between laptop, desktop, and iPad.

Tip: Create bullet lists but insert Checkboxes on each line instead. Your notes then instantly become a todo list that you can mark off. Evernote can also capture Win+S and save screenshots to a new Evernote note, or to your clipboard.

Sublime Text

Sublime Text is my go-to text editor in Windows. I use it for building HTML, writing JavaScript, and even editing long blocks of text. For a long time I used Vim32 on Windows, but it just doesn't seem right to me. It doesn't seem to work as fluidly as on a Linux machine, probably because of the lack of useful command line. (Yeah, yeah, PowerShell and Cygwin and such. Doesn't do it for me, sorry).

Tip: When you close ST, even with unsaved documents, your exact location is saved. When you reopen it, the unsaved buffers are shown. This means you can constantly have a scratchfile open that doesn't have to be saved, that maintains state through reboots. However, this also means that you close down ST sometimes without saving changes to disk, so the files you e-mail/upload aren't complete.


When I'm coding for fun, it's usually using Django on a Linux server. To edit this code I use Vim because it's what I know best. I have never been a huge fan of it on Windows, however, which is why I still use Sublime Text there.

Multiple buffers, good command-line support, syntax highlighting, and (for me) absurdly quick keyboard controls make this ugly duckling incredibly powerful and fast.


My BackBlaze Stats

I've had a computer stolen before, and it isn't nice. If you don't have sufficient backups, then all of your documents, photos, and more are gone. Just like that. A fire is possibly worse, as it wipes out your CD-Rs and USB hard-drives that you "safely" stored in the wardrobe.

I pay BackBlaze fifty bucks a year to store all of my junk on their servers in the United States. Photos, documents, prior years tax receipts, and more. Whenever I add new photos to my PC from my DSLR, they're automatically backed up.

Tip: Make sure your photo folder(s) are included in the backup, as well as any external hard drives you use. Test the backup by downloading a significant chunk of it, so you understand the restore process before you need it.

Royal TS

Royal TS with Multiple open connections

Royal TS is a great remote desktop tool that lets you save and connect to multiple RDP sessions at once within a single window. It's cheap. Apparently it's now available for OS X and iOS, however I only use it on Windows.

Tip: Group servers into Folders, and save login credentials (at least just Domain & Username) against the Folder. You then avoid storing login details against each server, making changes even quicker & easier.


The defacto SSH client on Windows - Using PuTTY with Pageant makes accessing Linux shells from Windows a piece of cake. It's also great for building secure tunnels through my home network when I'm on the road.

Tip: Put PuTTY in your $PATH and save your session details (eg which key to use, hostname, and window configuration). You can then run putty -load dev to login to your dev machine with pre-saved configuration.


It seems pretty common to use Dropbox for backups, however I don't use it for that - it's just a stupidly simple way for me to move files from one machine to another, or share online. Most commonly it'll be to move files between my work & home PCs; less regularly to move documents onto my iPad without using iTunes.

I only use a few hundred MB of storage so it's completely free. If you want to kick more storage space my way, you can use my Dropbox referral link (I get 250mb of storage if you signup this way, and it won't cost you any extra.)


1Password for Windows

I can't put a price on 1Password. Here's how it works: When you sign up for some new website, 1Password generates a password for you. It'll be something like 1yf~f6dUBmw[rY. It then saves your passwords for you, so you don't need to remember 1yf~f6dUBmw[rY. When you then visit that website again, just press Ctrl-\ in your web browser and 1Password logs in for you with the right password! The whole password list is secured by a single strong password; you type that password each web browsing session (eg type it once the first time you press Ctrl-\, it remembers it for the next half-hour or so).

Since using it, a number of high profile sites I use have had security breaches but I've never been overly concerned as I know that my passwords are never re-used anywhere. I don't know what most of my passwords are, and I'm happy with that.

1Password for iOS

Tip: Sync across devices using Dropbox. Ensure you've got your Dropbox and primary e-mail passwords stored elsewhere, in case you find yourself with no access to 1Password or Dropbox. Use 1Password on iOS to copy passwords to your clipboard, to then paste into Safari or other iOS apps - retyping those long random passwords is a shit task, especially on a touch-screen.

Pocket Casts

Pocket Casts - My preferred iOS podcast client

Pocket Casts is the only iOS Podcast client that I've found with push notifications. When new episodes come out, they pop up on my screen without me manually opening the app & refreshing the feeds.

I do wish that it had slightly better handling of 'show notes', but otherwise it's a great replacement for the buggy and incomplete Apple Podcasts app.


Spotify Playlists

I use Spotify on my iPhone and iPad, almost always streamed over AirPlay to my Apple TV. For now I've given up on buying MP3s, and listen to random playlists when I want background music.

There's something fantastic about sitting on my back verandah, listening to any songs I could ever want to listen to (OK, except for The Beatles and AC/DC), without any cables or physical media.


Instapaper Menu

Instapaper has a simple premise: When you find something you want to read later, you click 'Read Later' in your browser. Then, when you've got time to read, you open the Instapaper app, where the article is formatted in a reader-friendly fashion. It even works off-line.

Instapaper Article View

Instapaper is incredibly simple, and it works well. The basic service is free, and the iPad/iPhone apps cost a few bucks. It's initially hard to believe something so simple isn't free, but then you realise how useful it really is for those of us who like to consume long-form articles in our own time.

Tip: Don't forget to open Instapaper & sync it before getting on a plane. You've then got access to your reading backlog at 30,000 feet - images and all.

What else?

There are plenty of apps I use daily, but they're probably common enough to not need explanation. A few things you'll see me using if you watch long enough:

  • Outlook - paired with Exchange, it's the killer corporate e-mail & calendar system.
  • Google Chrome
  • Adobe Lightroom
  • Paint.NET
  • WinSCP


Occasionally I need to take WhisperGifts offline, but still show some parts of the site to users. This has included some system changes that require the site to be non-functional for a little while (such as doing a deployment with a bunch of backwards-incompatible changes, or large database migrations) and for server moves, whilst waiting for DNS changes to propogate.

To do this, I wrote a little library that I could toggle within my Django settings. I've just pulled it out of the WhisperGifts codebase, and django-readonly-site is now available on GitHub. I think it's pretty simple to use.

Install it with pip install django-readonly-site, add readonly to your Django projects' settings.INSTALLED_APPS, and set settings.SITE_READ_ONLY = True. More options are available to keep parts of your site online, see the README for more details.

By keeping parts of your site online (such as the homepage, about us page, and in my case a customers' registry listing) you can provide a transitional experience to users, while the database-intensive and high-integrity parts of the site (such as signup, account management, and checkout) are taken offline with a polite "Sorry, we're temporarily unavailable" message.

Just after I had to quickly move to Rackspace after an outage with my previous web host, Rackspace announced that they now had a public cloud offering in Australia. For performance reasons, I'll be moving from DFW to SYD soon - and I will use django-readonly-site to try and minimise the perceived downtime for my users.

Your thoughts, suggestions, and pull requests are welcome on the GitHub Project Page.

Pure CSS Image Accordian

A little while ago I rebuilt my homepage. The homepage became an 'About Me' page; more of a landing page than a blog archive. The blog URLs haven't changed, and it's still there - however the move to a landing page is an acknowledgement that I just don't write blog posts as frequently as I used to.

One of the fun things that I added at the top of my landing page is a series of eight images that show who I am and what I enjoy doing. They're all from my own photo collection, and although they're hardly perfect photos they do capture the essence of the past few years of my life.

The images are shown in a simple accordion - only the middle vertical 30% of an image is shown, then when you mouseover an image slice the other images are compressed to show the hovered image in it's entirety.

The images are all 400x400 pixel squares; they're floated left by -150px (with overflow: hidden) to show the 8 even slices. On hover, all of the slices are resized to 20%, except the image the mouse is over: That is changed to 100%.

To make it a little prettier, I use -webkit-transition (CSS3-only), however it isn't as smooth as I'd like.

I've tested in most modern browsers, but as my site isn't high-traffic I haven't tested every possible combination. YMMV.


<div id='hero'>
    <span><img src='/media/heroimages/hero1.jpg' title='Image Caption'></span>
    <span><img src='/media/heroimages/hero2.jpg' title='Image Caption'></span>
    <span><img src='/media/heroimages/hero3.jpg' title='Image Caption'></span>
    <span><img src='/media/heroimages/hero4.jpg' title='Image Caption'></span>
    <span><img src='/media/heroimages/hero5.jpg' title='Image Caption'></span>
    <span><img src='/media/heroimages/hero6.jpg' title='Image Caption'></span>
    <span><img src='/media/heroimages/hero7.jpg' title='Image Caption'></span>
    <span><img src='/media/heroimages/hero8.jpg' title='Image Caption'></span>


#hero span {
    /* Show each image as a 120x400 sliver. */
    width: 120px;
    height: 400px;
    float: left;
    overflow: hidden;

#hero span, #hero span img {
    /* Animate the image width changes. Not perfect, but good enough. */
    -webkit-transition: all .5s;
    -moz-transition: all .5s;
    -o-transition: all .5s;
    -ms-transition: all .5s;
    -webkit-transition-delay: .1s;
    -moz-transition-delay: .1s;
    -o-transition-delay: .1s;
    -ms-transition-delay: .1s;

#hero span img {
    /* Ensure the vertical slivers show the centre 30% of the image */
    text-align: center;
    position: relative;
    left: -150px;

#hero:hover span {
    /* When we hover over the #hero block, change all images to 80px (20%)... */
    width: 80px;

#hero span:hover {
    /* ... except for the actual image being hovered, which now becomes 400px (100%) */
    width: 400px;

#hero span:hover img {
    /* Reset the 'left', as it was previous 150px, on hover. */
    left: 0;

This code is hereby in the public domain; no warranties or support are provided. Good luck.

Getting Paid in Django with Pin Payments

Payments in Australia are controlled by the so-called "big four" banks, and it's been difficult for a long time for startups to get merchant facilities to process credit cards online. Accounts cost hundreds of dollars per month, with high transaction costs and minimum transaction volumes thrown in.

We watched with teary eyes as companies such as Stripe launched and made it easy for developers to process credit cards, and kept struggling with the PayPal "API" with hope that one day we'd see Stripe in Australia.

I was excited, then, to see that Pin Payments have just publicly launched in Australia. They're currently offering a $9/month account fee, which isn't free but it's certainly ideal for small companies. After June 2013 this will jump to $50/month, so it seems to make sense to sign up now if you've got any medium-term plans.

Like Stripe, Pin offer a JavaScript Library for processing payments on your page without ever handling credit card details. This makes it much quicker and easier to implement.

Because I process payments in multiple places on WhisperGifts, I needed a reusable way to render the payment form and process the payment in a worker queue. The resulting code has been packaged up as django-pinpayments, which is still simple and in an Alpha state but it's a great starting point.

The code is released under the BSD licence, and I'd love to get your suggestions, comments, and GitHub Pull Requests.

A few things on the to-do list include providing Celery tasks out of the box, and providing tests & documentation. If you can help with any of these I'd be very happy :)

In the meantime, I'm about to go live with credit cards on WhisperGifts for the first time. I'll keep PayPal around as an alternative, but if demand is low I won't hesitate to turn it off. I can't wait.

I've Screwed My Kid's Identity

Last month, this website passed it's ten-year anniversary. It's a quiet milestone, but it's humbling to know that my website has been online for more than half of the age of the "Commercial Internet" 1. I was online for a long time before 2002, but always in a less organised fashion than you see here.

Since I was an adolescent, I have been online. I was part of the first generation of teenagers to start getting themselves into trouble on the Internet by over-sharing personal details (This was never a problem for me, but I'm not sure why teenage-me shied away from being TOO personal.)

I try not to be too scared by stories carried by "news" and current affairs programs on TV where untold numbers of naive young Internet users share information, as in many cases it appears to be hyper-inflated news stories on top of parents looking for "justice" for their children, who most certainly understood what they were doing in the first place.

However I'm very cognisant of the fact that I'm putting much of my son's private information online. He'll be part of the first generation of internet users not only to not know of a pre-internet society, but whose history has already been shared without their knowledge.

Without his knowledge (and certainly without his permission), his mother, uncles, aunts, and I regularly and voluntarily publish information about his life in public arenas: Facebook, Twitter, Flickr, and even this blog contain voluminous details of my son's life from the moment he was born through to today. While most children & teenagers online are (hopefully) wise enough not to tell everybody their exact date of birth and their parents maiden names, we've already done that for our kids.

Based on the way Internet trust works today, in 2013, this information is benign to us but dangerous to our children. When my son signs up for his Facebook or Gmail account2 he's going to be asked for his mothers maiden name, his date of birth, and his first dogs' name to use as future proof of account ownership.

The problem is clearer now - my son's online accounts will not be safe because of the actions of his family decades in advance.

So what are we going to do about it? In reality it's too late; we cannot take back what we've already published about our kids. We also can't expect parents to stop bragging about their kids and posting photos - it's probably also not possible to expect parents to even understand the gravity of their oversharing because it isn't oversharing as far as they're concerned.

As an example, I have no qualms telling you that my dog is named Abby; this information is useless to you from a security viewpoint though as that's not my first dog's name. For my son, obviously, the timing is different to the tune of 27 years, so now you know the answer to that security question for him.

As new generations come online, it's time for the web to stop relying on static semi-personal information for identity. In 2013, it is no longer acceptable to rely on "security questions" to prove identity. The answer for lost online identities must be two factor.

Usefully enough, two factor authentication is already the industry-standard way to deal with high-security logins. Using hardware tokens, our mobile phones, or even old-fashioned e-mail and SMS to prove ownership of an account is the only way netizens can be safe for generations to come.

So my plea to websites on behalf of my kids is this: Stop using security questions.

  • Reset lost passwords using e-mail resets or SMS messages, not with security questions.
  • High-security accounts, such as bank accounts or e-mail accounts3, should use two-factor authentication by default
  • Security questions should die a quick and sudden death. The internet already has second generation users.

1. At least in the eyes of the general public, the Internet didn't really exist in Australia until at least the mid-nineties.

2. Or whatever takes their place in a decade from now - isn't that an exciting thought?

3. E-Mail is high-security because it is the single reset point for a persons entire online identity. It's no longer "just a Hotmail account".

The Definitive Answer, Explained

Yesterday I posted that Django was almost certainly suitable to use for your project. I've had some minor push-back, so I thought I'd explain a little.

When beginning a project, many businesses appear to spend an inordinate amount of time making technical decisions that are often outside their area of expertise. One such decision might be from a small business owner wanting to decide whether to build their shopping cart with Django, Rails, or PHP.

The hard truth is that for the most part, this decision doesn't matter. All three of the above can be used to successfully build exactly the sort of shopping cart that you want, no matter how bespoke.

An article I came across this week talked about the same theme but in a different context. It's by Forbes' Gene Marks, and is titled What Won't Tell You. The message here is that no matter what CRM solution you implement, you'll get results if you implement it well - and that means getting the right people to build/design it, getting your staff on board, and making sure somebody owns the system.

This applies to your website project, too.

  1. Ensure somebody at your company owns the website and makes decisions based on outcomes rather than technology
  2. Ensure you're working with somebody competent to build your website. Don't fuss over whether they use PHP, Django, Rails, or otherwise: defer to their experience (it's what you're paying for, after all)
  3. Embrace what you build.

My feeling is that when you're building something bespoke, #2 is the most important: make sure you work with somebody you trust. If you aren't able to let go of some control and let them make technical decisions for you, then your project is already doomed.

Of course there are exceptions here. If you're a Rails developer, just build with Rails (unless you want to try Django). If you know for a fact that a particular technology can't work for you, then don't use it. But if you aren't at all technical, then don't try to make technical decisions that impact your business: please find somebody who can make that decision for you.

After all, you wouldn't expect a web developer to tell you how to layout your retail store, would you?

The Definitive Answer To "Can I Use Django For This Project?"

Short: Yes.

Longer: Almost certainly. If you don't know any technical reason why Django isn't a good fit, then Django is probably a good fit.

WhisperGifts: The Tech That Let Me Launch

The WhisperGifts re-launch recently went very well! I promised a more technical follow-up with some details about what's new and what's changed, so if you want to know more about what makes WhisperGifts tick then you'll want to read on. Hint: It's a dash of Django, a pinch of Python, and a slathering of open-source software all around.

The primary technology behind WhisperGifts is Django, the "web framework for perfectionists with deadlines". My own deadline for this project was rather, ahem, flexible - the new design was a work in progress for 2 years (on-and-off, mostly off due to the birth of our baby) and the back-end re-write happened over a few months early in 2012.

Django allows us to iterate quickly. I find the language natural to use and the documentation is epic. There's a number of things that no framework does out of the box; I've tried to rely on open-source to fill as many gaps as possible rather than re-writing things from scratch like I did with the original WhisperGifts site - this is mostly because the open-source ecosystem around Django is now so much larger than it used to be.

As an example, originally I rolled my own authentication as the user management modules in early Django releases were rather inflexible. Building your own authentication is never a good idea, so I've migrated to using built-in Django logic. Similar changes are found throughout my codebase.

What I Use

Django, obviously. I use most of what comes with Django: The ORM and URL dispatcher, the included Admin, User and Cache apps, and more. Some might be interested to note that I don't use class-based views, simply because I don't see a need to change at this point.

Caching is done using Memcached and nested cache tags, as I've blogged about previously. I also use Django's site-wide caching middleware for anonymous users, which reduces load time for the marketing/static portions of the site.

Images are processed via the sorl-thumbnail package. I can generate thumbnails in any size on the fly. All of my images are stored locally - due to my current volume, the overhead of setting up a more formal CDN or even just using S3 isn't worthwhile.

Customer payments (for upgraded packages) are handled by PayPal. To interface with their IPN and to simplify the user-facing workflow as much as possible, I use django-paypal.

To track in-app metrics (such as number of signups, number of upgrades, number of new items) I use django-app-metrics and get a simple daily e-mail. I'm also testing out Mixpanel which although it isn't free lets me get much more detailed statistics for the same sorts of metrics. django-app-metrics even has a backend to automatically push data through to Mixpanel, so I might use that later on.

All e-mails are sent asynchronously (using django-celery) so they don't tie up front-end web serving. I deliver my e-mails via Postmark with the useful django-postmark library. All my outgoing e-mails include both HTML and plain-text components; I also embed a header image. In the geek world this is heavily frowned upon, but remember who my audience are: couples getting married and their wedding guests. Postmark makes these e-mails simple.

The front-end web server to all my websites is always nginx. It is small, easy to configure, does a wicked job of serving up any static files (both my own site static files and those customer images that have been uploaded) and integrates well with Django. To run Django for nginx I always use gunicorn managed by superisord.

My site-specific CSS and JavaScript files are hand-crafted during development then at runtime combined together effortlessly and minimised as much as possible using django-compressor.

To make sure that any gremlins are caught and dealt with, django-sentry catches any exceptions in my code and presents them in an interface that is incredibly useful: You can see which exceptions occur most often, what conditions trip them, and more.

In a similar vein I use django-timelog and occasionally review how long my views are taking to render in a live environment, django-debug-toolbar gives me similar data during development.

Bringing it all together

For much of the above all I need to do is pip install django-compressor and add the relevant code to my and templates. Very little of what I've mentioned above has changed the way I develop or deploy; they simply make life easier. The fact that I can pick up these bits of software (most of which weren't available 3-4 years ago) and use them off-the-shelf with some very minimal setup just makes me love Django development even more.

I therefore owe a big "thank you" to the Django community.

Previously I've manually written code to do many of the things I've mentioned above (and bad code, at that, given it's status as "helper code" rather than the main part of my projects). I owe a few people a beer or three.

WhisperGifts Re-Launch

Way back in 2007, my wife Lauren and I got married and went looking for a good bridal gift registry service. We didn't find one, so we built our own - WhisperGifts was born. It's now 5 years later, and today we are re-launching with a completely new look & feel and a 100% new code base under the hood.

I'm new here - what is WhisperGifts?

WhisperGifts lets couples who are getting married put their gift registry online - i.e. make available to their guests a list of the wedding gifts they'd like to receive.

In many cultures this is done by utilising the services of big chain department stores. This has two major downsides: the items are typically more expensive, and the range is moderate at best. Why not have a gift registry that lets you get gifts from anywhere?

So we built WhisperGifts - an online service that lets you list items from anywhere you like. In fact, you don't even list where the item is from - you simply say "We'd love a toaster that cooks crumpets." Your guests are still given enough latitude to select which toaster, from where, and at what price. It's win-win!

As a hobby, I can't say that WhisperGifts is too bad. Enough couples make use of it for me to say that it's worthwhile, and it's a great place for me to try out new tech and startup skills. But I've hardly put enough effort into it, which leads us to today: WhisperGifts 3.0.

WhisperGifts 3.0

I re-skinned WhisperGifts a few years back, but I've never been happy with the layout. A visitor to the website isn't really introduced to what WhisperGifts is, and there is no obvious call to action. There's a thousand and one problems that can be pointed out by anybody who has ever worked on a startup.

The old look and feel isn't the only thing that's gone, though. About 90% of the old codebase has been re-written, but not just for the sake of it - until now WhisperGifts has been running on 5-year-old code, a lifetime in web development. Every now and again I make minor updates to keep it compatible with up-to-date versions of Django, but purely for security purposes.

With a new back-end powered by all the goodness of Django 1.4, I'm able to have a website that runs faster, uses niceties like SQL aggregation, has proper form definitions (my old code still has comments referring to forms and newforms as two separate things) and is more maintainable to boot.

In terms of LOC there's about 50% less code to do the same end result.

So we've got a new visual style and a new back-end, but what else?

  • Our old "Premium" package is now "Standard" and a new Premium package has been created. It includes things like printed guest information cards and an unlimited number of items
  • Sharing of registry details via e-mail, Facebook and Twitter
  • Custom domains - so you don't have to have your registry at ''
  • Coming very soon will be custom registry templates

The new website is now live, at I'll be blogging again shortly with some more technical details of the change, which are probably of more interest to those who read this blog for the Django posts as none of it will be WhisperGifts-specific.

Getting married? E-mail me and I'll see what we can do about a few freebie premium accounts :)

Why you shouldn't buy cheap watches, and investigate odd sleeping patterns in kids.

A fun anecdote, showing why you should investigate issues that occur too regularly to be co-incidence.

December, 2009 - Tasmania, Australia

A few years ago, my wife and I went to Tasmania and walked the stunningly gorgeous Overland Track, a 65 kilometer (5-day) walk through isolated north-west-central Tasmania. If you're into hiking, you need to get yourself to Tasmania.

Wanting to keep track of time, Lauren bought a new watch. It was cheap, plastic, and bright purple - but at least it wasn't an expensive dress watch - perfect for the great outdoors.

Way before dawn one morning at 4am, sleeping in our expensive hiking tent on a camping platform in the middle of the bush (there are tent platforms throughout the park so as to protect the ground - this is a delicate area) the alarm went off. This little purple watch just started beeping, and never gave up.

In a bushwalking-induced tired haze, Lauren tried to disable the alarm. Floundering and quickly becoming frustrated, she commandeered my pocket knife and tried to stab the watch into silence - somehow avoiding tearing a hole in the floor of our nearly-new tent.

A week later, we returned home and packed our hiking gear away in our spare bedroom. Every now and then we take some of the gear out for a weekend, then it goes back into the cupboard. Unused items, such as the purple watch, slowly float to the bottom of our gear bags.

September, 2011 - Melbourne, Australia

In September last year, our little boy was born. We gave him our "spare room", and cleared out some of the junk in the cupboard. Half of our hiking gear got shifted to the garage, but for now half of it stayed at the bottom of the cupboard in the spare room.

Life's different, when you've got kids. Your "room to dump everything you don't need right now" room isn't yours, any more.

(I'm sure you know where this is going)

Saturday, 16th June, 2012 - Melbourne, Australia

Our kid's sick. First he was teething, which seems to hit the immune system pretty hard. So he got a cold, which he promptly gave to Lauren and I. Finally, it morphed into full-blown Tonsillitis, so it's safe to say none of us are getting a very good night's sleep right now.

Because he's been sick, the boy wakes regularly during the night. Poor fella. The really odd thing, though, was that he was waking right on 4am every night. The accuracy of his night waking was uncanny.

As parents, we are trying to teach our son to self-settle. If he wakes during the night, we try to give him 5-10 minutes by himself to re-settle, and much of the time it works. He learns that he doesn't need us by his side to fall asleep, and we get to stay just that little bit warmer.

So when you added all of this up, we had a chain of events something like this:

  • 4:00am The alarm went off at 4am, waking our son. Being in another room, it was too quiet for us to hear.
  • 4:01am We'd stay in bed for 5 minutes to see if he'd re-settle.
  • 4:04am After beeping for a few minutes, the watch would give up and turn it's alarm off.
  • 4:05am After 5 minutes, we'd sometimes have to go and settle him down. In a sleepy haze, we'd notice it was always 4:05am when we went in to see him.

With his illness this weekend, though, Lauren decided to settle him earlier - right away, in fact.

When she got into his bedroom at 4:01am on Sunday morning to help him get back to sleep, she heard it. The beeping. Flashes of pocket knives and cold tent floors came rushing back, and she managed to fish the wash out of it's bag. The "I'm going flat, dammit, gimme a new battery!" state of the watch had re-triggered it's incessant alarm at 4am each day.

Sunday, 17th June 2012 - Melbourne, Australia.

Revenge: we drowned the watch whilst cleaning the dishes.

Domain Names For Sale

I'm cleaning out my servers and have a few domains I want to sell or give away, including (and .net, .org); and E-mail me at if you're interested in any of these. (Note: the domains will require you to have an Australian address & ABN, I believe)

DjangoSites Move Complete

For the past few years I've hosted all of my projects on a single RimuHosting VPS. It's old, running Debian 5.0, and maintenance and upgrades have become headachy. It isn't easy to run Python 2.7 on old Debian versions, and since I only started using virtualenv relatively recently, things were a real mess.

As such, it was time to upgrade the VPS. I made the decision to look around for other hosting options, and eventually settled on Prgmr as the performance-for-dollar ratio seemed much higher to me.

I've taken the opportunity to break things down into multiple smaller VPS units to try and keep maintenance somewhat simpler. I can pull a single site's server down without breaking any other sites, and I can upgrade components individually for a single site.

Prgmr's pricing makes this possible: Their 'base' is very low - you pay $4 for each VPS plus RAM. now sits on it's own 512mb Prgmr Debian 6.0 instance. With 128mb taken up with memcached and a small portion to nginx and the operating system, there's plenty of RAM left for my gunicorn worker processes.

So how did I move everything across?

Although it took me 3 days to finish, the actual effort involved was really quite low. I hope these tips help you with server migrations in the future!

  1. In advance, drop the TTL on your DNS records low. I went to 300 seconds, but on the wrong domain. Doh!
  2. Deploy the new server environment. I have a shell script that creates the relevant users, installs all the Debian packages I need, installs system-wide tools like PIP, installs system-wide Python libraries I need, installs VirtualEnv, installs & configures Postgresql & Memcache, etc.
  3. Put your old app in Offline mode. For Djangosites, this means changing the 'accounts' and 'submit' URLs to point to flatpages instead of their views.
  4. Create a virtualenv on your new server
  5. Copy your (now not-being-written-to) database & code to your new server
  6. Configure nginx for the new app, spin up your app instances
  7. From your desktop, change your HOSTS file to let you access the new server directly as if it were live. Test your siste.
  8. Revert your HOSTS file, and update DNS.
  9. Revert your DNS TTL.

Users with stale DNS will continue to see the read-only site on the old server, whilst those with sensible ISPs will see the updated site within a few minutes.

Upgrading at the same time

Because moving code is boring I decided to take the opportunity to upgrade everything to Django 1.4. Djangosites was, until this week, running on 1.3 but it was using a number of deprecated features that wouldn't port to 1.4. It was also using the Postgres 8.3 contrib package for tsearch, which has different syntax to Postgres 8.4's built-in tsearch.

Unsurprisingly, the codebase has dropped due to this simplification. It's also let me clean out the cruft, simplifying things even further.

I also used the move to point DjangoSites at my Sentry instance - previously I just relied on the built-in Django error e-mails. Using Sentry made it easier to catch a few quirky bugs that were leftover from the move & upgrade.


DjangoSites now has it's own VPS environment, standing alone from my other projects (which, incidentally, also have their own VPS environment). This means I can do maintenance on it without impacting any other projects.

I can also manage memory/RAM with less fuss: I just ask Prgmr to increase the ram for one VPS instance, rather than try and juggle the way each application within a single server fights for memory.

At present, even on only a 512mb VPS, DjangoSites has plenty of headroom. With some basic Blitz benchmarking we're good for north of a million hits a day on a commodity hosting account... obviously we're nowhere near that at the moment, but it's good to know we can grow into this new t-shirt.

My monthly hosting costs have also dropped, but overall I've got allocations of more RAM, bandwidth, and disk space.


DjangoSites now has it's own VPS environment, which means my operations time is going to increase. Any updates and patches to non-app code (such as Postgresql updates or OS security fixes) need to be done multiple times, not just once.

I also suck at remembering hostnames, and need to stop instinctively logging into a single host server. I'm sure I'll get over this pretty quickly, though.


I can't recommend Prgmr highly enough. Their prices are great but more importantly the VPS instances I've got are rock solid. They're fast (including the disk IO) and very responsive. My old RimuHosting VPS wasn't too bad, but the IO was certainly nowhere near as good.

DjangoSites Downtime - Server Move

Heads-up: will be down, at some point in the next week, while I move it to a new server. Expected downtime is approximately 1 hour to transfer files & database; if your ISP has DNS servers that don't observe TTL then you might not see updates for a while longer.

This is happening as I'm moving DjangoSites to it's own VPS to make it easier to balance available resources between it and my other projects.

Key-based cache expiration with Django

Last week, the team over at 37Signals wrote up an article on their newly implemented Key-based cache expiration system and it hit me: It's such a simple idea with obvious benefits, why hadn't I implemented a similar caching mechanism before? Being a Django user, the Rails code didn't make much sense to me but the concept certainly did - so here's my take on it with a quick Django example.


I've just implemented this caching strategy for WhisperGifts for a re-launch that will go live in the next few weeks. We allow couples to publish an online gift list, then let people select items from that list. Pretty basic stuff, but rendering the gift list can require n+1 queries due to the way that my purchase data is kept. This hasn't been a big issue until now, when I've built new functionality and generally just extended things a bit.

The cache strategy is so simple it's taken longer to write up here than it did to alter my existing codebase.

My basic model is as follows:

  • Registry, the top-level "collection" of items for each wedding.
  • Item, of which there are many for each Registry
  • Buyer, of which there are 0-n for each Item

With that in mind, read on...

Model Setup

The first task to do is make sure your models have a last_modified timestamp. Mine already did, but just in case you need an example:

class Registry(models.Model):
    name = models.CharField(max_length=100)
    last_modified = models.DateTimeField(null=True, editable=False)

def save(self, *args, **kwargs):
    from datetime import datetime
    self.last_modified =
    super(Registry, self).save(*args, **kwargs)

Next, make sure that whenever you save an Item or Buyer the last_modified change cascades upwards to the Registry.

class Item(models.Model):
    registry = models.ForeignKey(Registry)
    name = models.CharField(max_length=100)
    last_modified = models.DateTimeField(null=True, editable=False)

def save(self, *args, **kwargs):
    from datetime import datetime
    self.last_modified =
    registry = self.registry
    registry.last_modified =
    super(Item, self).save(*args, **kwargs)

Note: I'd highly recommend this gets taken care of in an abstract base class. The above is just a (bad) example.

Define your cache keys

At the top of I defined a variable named DATA_SCHEMA_REVISION. For now I increment it whenever I make a change to my schema that should invalidate the entire cache - not that it's a regular occurrence.

On each model, now, define a cache_key property. For example, on my Item model:

def cache_key(self):
    return 'myproject/%s/item-%s-%s' % (DATA_SCHEMA_REVISION,, self.last_modified)

Again, as a good programmer you'll put this in your abstract model, won't you?

Configure Django caching.

This isn't hard. I use memcached. YMMV.

Update your templates.

My existing template was along these lines:

<b>{{ }}</b>
{% for item in registry.item_set.all %}
    {{ }}
    {{ item.expensive_calculation_and_rendering }}
{% endfor %}

The new, improved, fully cached version:

{% load cache %}
{% cache 9999999 registry registry.cache_key %}
<b>{{ }}</b>
{% for item in registry.item_set.all %}
    {% cache 9999999 item item.cache_key %}
    {{ }}
    {{ item.expensive_calculation_and_rendering }}
    {% endcache %}
{% endfor %}
{% endcache %}

Notice that we're caching this for a very long time - because the cache key will change whenever our data is changed. If an Item changes, that item's cache key (and the parent Registry cache key) will be changed due to the new timestamp, and they'll be re-rendered. However when we re-render the outer Registry cache fragment, it's primarily made up of still-valid cache components for the other Items.

This is an incredibly simple way to keep your site nice and snappy. There's some fantastic discussion over on the original 37Signals blog post - I strongly recommend you read & understand that post as well as it's comments, then consider my examples above as a bare minimum implementation in Django.

How quick is it?

A brief update, 10 minutes after this post was written. I have done some very rudimentary tests and come up with the following numbers. Note that these are NOT representative, however the difference is very clear. Keep in mind this is running in a development environment with debugging enabled and various other slowdowns not present in production!

Cold cache 17.34 seconds
Warm cache 1.11 seconds

This is the time taken for Django to process the request form start to finish, from the initial HTTP request to returning a complete HTML page. There's plenty of other gains to make but as a starting point this is not too bad at all!

Enabling your sales team with Mobile CRM

My day job is as a CRM consultant for Professional Advantage. I help companies implement the Microsoft Dynamics CRM platform so that they can streamline their business processes such as sales pipeline management, helpdesk / service management, and marketing automation.

I've just written a post over on the PA Blog about using Mobile devices to help your sales team get the most out of your CRM system. It talks about a few ways to help make sure your sales team fully utilise the system you've just implemented, rather than just see CRM system maintenance / usage as a chore.

While I'm guessing my typical blog audience won't find this too interesting, some of you may be intrigued as to what I actually do at work. Hopefully this helps... and of course, if your organisation has issues managing sales or service let me know and I'm sure we can help you out :)

2011 In Review

I'm not one to blog about personal topics, but I have to make an exception as 2011 has been a heck of a year on a number of fronts.

This is a pretty long post, so bookmark it if you're on an iPhone or just skip it entirely if you're not interested in a 20-minute read about some guy who reckons his personal experiences are any different to the next guy.

Learned I was to become a dad; learned to be a dad

A few days into the new year, my wife Lauren and I found out we were going to be first-time parents. There was the requisite excitement, nervousness and more excitement as we got used to the idea we were going to be parents.

By February we had told our best friends and family; others came a month or so later.

Sharing the news of your pregnancy is fun. For many people, including Lauren and I, it was kept a secret until we carefully shared it with initially few but later more people. We have very few true "secrets" these days, so it's nice to be able to play th secrecy game between yourselves for a few weeks.

In September, after very little sleep for a few days I became a dad for the first time. The lack of sleep suddenly didn't matter, the fact that it was a somewhat difficult labour (but not exceptionally so, in the grand scheme of things) didn't matter, and the fact that our lives had now been demarcated into "BC" (before children) and "AD" (after delivery) didn't matter.

As I write this, our little boy is just three months old, but it's hard to imagine life without him. The pain is there but made duller by the sheer joy he brings into our lives. Big gummy grins have become one of the most important things in my life... that's pretty big, for a guy who typically didn't like kids!

To show how my tolerance has changed I have a fun story from last week. We took our boy to a baby massage class, with Lauren's mothers group and everybodies partners. As us dads took over with the massage, there was a room full of screaming babies. All we could do was laugh... 6 months ago a babies cry was annoying, now it was amusing.

Completed my studies

Since 2005 I've been studying part-time at Swinburne University towards a Bachelor of Business (Accounting). I'm not a typical student; I finished my secondary schooling in 2001 and began working full-time in the IT industry immediately. I've been very lucky to have a fantastic career path that has grown out of that early work, however I identified (with some gentle pushing from family) that some formal tertiary qualifications weren't such a bad idea.

Ask any full-time student and they'll tell you study is hard. Ask any part-time student and they'll tell you the study is easy, but bloody hard to fit in around your other commitments.

Full-time students typically have their study as their primary focus. For part-time students like myself, study is a second (or third, or lower) priority after work, being a husband, homeowner, and now father. Competing pressures on your time make part-time study a serious undertaking.

It's now seven years later, and I've completed my degree. This final semester (during which my son was born) was hands-down the most challening I've faced, however I've recently received my results that show I have completed the required 24 units without failing a single one.

The maths nerds might notice that I have taken seven years to study 24 units, at 2 per semester. That's because both last year and this year I took off one semester to travel without having an impact on my grades.

Returned to Nepal

In March 2010, Lauren and I travelled to Nepal to trek in the Everest region. Unfortunately, a few days into the walk Lauren was bitten by a dog. Due to the Rabies endemic in Nepal, the recommendation from a western doctor and our insurance company was to evacuate to Kathmandu for post-exposure injections.

Rabies post-exposure injections are expensive and painful. Travel insurance is a lifeline, and as a result of our dealings I don't hesitate to recommend World Nomads to anybody who asks.

There were two positives to come out of the bite. Firstly, we were evacuated by helicopter. Yes, a helicopter flight through the Himalayas. Unfortunately it was very cloudy, so even though we had walked for a week and flown through the area I still hadn't had but a glimpse of Mt Everest, the worlds' highest mountain at 8,848 metres tall.

So, the second positive: As soon as I was happy that Lauren was safe back in Kathmandu, I consulted Asian Trekking (who had arranged our trek) and booked in a return trip.

Come March 2011: I returned to Nepal for another 3 weeks, this time with Lauren's cousin Rob and one of his climbing buddies. This time there were no dog bites, and I was luckly enough reach Gokyo Ri, one of the most beautiful places in the world.

On the day after our arrival in Gokyo, there was a snowstorm so we couldn't climb the Ri (mountain). Most of the trekkers in Gokyo left town, descending to lower altitudes to try and keep to their tight schedules. Our schedule was relatively relaxed, so we hung around for another day and I'm glad we did: the weather was perfect, there was a layer of snow over all of the surrounding mountains (including Cho Oyu and Everest), and there was hardly anybody around.

Climbing Gokyo Ri is perhaps the most physically challening thing I've ever done. I'm convinced it's no steeper than the street on which I live, but at 5000+ metres of altitude breathing is hard. A severe lack of protein in ones diet certainly doesn't help, either.

Arriving at the top of the Ri was rather emotional. The spectacular 360 degree views of some of the worlds' highest mountains, including Mt Everest (#1 at 8,848 metres), Lhotse (#4, 8,516m), Makalu (#5, 8,485m) and Cho Oyu (#6, 8,188m) are awe inspiring and make the hard work of seven-days of non-stop uphill trekking worthwhile. Although I had no phone reception, I used my iPhone and the Occipital 360 Panorama app to capture a panorama of the views. I reckon you should check it out, then contact Asian Trekking to arrange a trip there yourself.

Hillspotting: If you look West in the image above, find the person in the red jacket next to the cut-in-half person in Black: Mount Everest is up and to the left - the big black rock triangle in the distance with the snow spindrift. Lhotse is just to the right of that. Then, follow the glacier to the left (which is really North, but shows as South here for some reason). Cho Oyu is the tallest white triangle in that direction. It seems the 360 app has it's directions backwards, because what's West in this image is actually East and South is North. shrug

Nepal is somewhere I'll be returning to as soon as I can. Now that I'm a "family man" that may be a decade or more away, but I can't wait to take my wife and family back to Gokyo Ri.

The friendliness of the Nepalis is amazing. The cost of getting to and around Nepal is relatlively low (compared to Europe or the Americas, for us Australians) and despite their horrid political history I feel it's a safe enough place to take children.

Flew to Microsoft Convergence

From Nepal, I flew directly to Atlanta, USA for Convergence 2011. Not having been home to Australia, I arrived at the Hilton Hotel to meet a customer of mine looking and feeling rather shabby after the 24 hours of travel which were added onto two weeks of walking in the wilderness!

This was my first trip to America, and I was pleasantly surprised by a few things:

  • Southern hospitality rocks
  • Beer selection at hotels in the US rocks
  • The Microsoft Dynamics ecosystem is big, and full of very intelligent and friendly people.
  • The Atlanta aquarium is mightily impressive. Whale sharks indoors? Wow.

There were also a few things which I was unimpressed with.

  • The first meal offered to me in the US was on a Delta flight from Newark to Atlanta. The 7am breakfast offering was a bag of peanuts and a can of Coca-Cola.
  • Country fried steak. What the fuck, America?
  • Atlanta isn't a pretty city. It's obvious not much has been done to maintain the city since the '96 Olympics.

Overall though, the trip was very much worthwhile. Returning home I was lucky enough to be on a near-new V Australia 777. A pretty nice bit of kit.

Travelled too much

In 8 months I've done enough travel at work to go from zero to Gold membership with Virgin Australia. Note that my Nepal & US trips only included a single leg that calculated status, and it contributed less than 10% of my status points for the year.

That's a sign that you travel too much.

Earned my long service leave

Yup, I've been at the one place for 10 years. Kind of, anyway... Five years with my former employer who "merged" (read: were bought out) by Professional Advantage five years ago. That makes 10 years of employment without an interview.

Sped up on the Microsoft stuff

My day job is as a consultant for Professional Advantage. I specialise in, and spend all of my time on, the Microsoft Dynamics CRM application. Many people don't realise that Microsoft has a strong business division. No, not Office and Windows. I'm talking the Microsoft Dynamics suite.

Dynamics CRM is a pretty good product. It's flexible, it works well for small business through to enterprise, and it integrates bloody well with Microsoft Office.

The latest version, Dynamics CRM 2011, was released earlier this year. As a result I've had the opportunity to learn a new product and get more involved in our pre-sales process which has been great fun. I wouldn't call CRM2011 a "reinvention" of previous versions, but it's given me enough new work that I can keep working on exciting projects while I learn the ins and outs of a major upgrade.

Part of working at a Microsoft partner is that you're immersed into the Microsoft technical ecosystem. I've spent the past 12 months (and more) learning about SQL Server, Reporting Services, data migrations, and all sorts of other stuff that we never get exposed to in the open source world.

It's very easy to write off Microsoft products, simply because they come out of Redmond. The past few years have taught me that that's bullshit.

Let a few things slip

My open source commitments have fallen by the wayside a little, it seems.

Unfortunately this year I've neglected some of the Open Source stuff I've contributed to in the past. This includes DjangoSites (which I still maintain, but not too proactively), django-helpdesk, which needs some maintenance; and this blog.

I've managed to rebuild WhisperGifts but not relaunch. I've built GoHike but not launch. If this year was for building, next is for launching.


I think that's enough reflection for now. There has been plenty more happen over the past year and it's been tricky at times to keep up.

I've got few plans for next year. I want to slow down a little and get used to life as a dad. I won't be building anything big and new, but I'm committed to launching a few projects that are already 95% done.

I'm going to do more walking. Our first trip as a family will be early in the new year, and I'm hoping to spend some of it wandering the high plains of the Victorian north-east with baby in tow. I'm also keen on taking up rock climbing as an alternative to hiking - day trips where you achieve something are easier to manage vertically than horizontally, it seems.

And that's it. No huge projects to undertake. No big holidays planned. Just living & loving life.

Be safe over the holidays and I'll be back in the new year!

Small open-source release: django-menu

Many moons ago on this blog I wrote about a simple menuing system for Django. For the sake of convenience, I've just packaged up that code (plus a few minor improvements) into a package named django-menu which is also available via PyPi with pip install django-menu. Basic documentation is included in the package and in the git repository.

Please log any issues or suggested improvements via the GitHub issue tracker!


A while ago I released a helpdesk tool that I use to manage support requests, under the name of Jutda Helpdesk (named after my small consulting company). The project has received a slow but steady stream of patches and bug fixes, however it's always been a little tricky to manage with a single committer over at Google Code. To make life easier for everybody involved, I've renamed the project to django-helpdesk and shifted the source code and issue management to GitHub.

I thought I'd spend a few words talking about these two changes.

Firstly, the change in name. Because django-helpdesk was originally built for my own use internally at Jutda and WhisperGifts I released it under the name of "Jutda Helpdesk" when I opened the source up a few years back. There was a bit of a thought that I could release more products with a similar naming scheme, such as "Jutda Basket Weaver" and "Jutda Donut Maker". Those products never eventuated, leaving the only open-source product as "Jutda Helpdesk". This seems to have caused a bit of confusion, with people referring to the product simply as "Jutda" which does no good for either the helpdesk product or my business.

The next change was a move from Google Code to GitHub for the project site, including the source code management and issue tracking. The reason for this was twofold:

  1. For users to offer patches they had to log an issue with a patch and hope I could apply it sooner rather than later. By using GitHub, anybody can fork the project and begin making changes, meaning I don't need to be involved for people to share code changes.

  2. Every other Django project that I personally follow is on GitHub, and I prefer the git DVCS workflow over SVN. It seems most of the Django community is of a similar mindset, from what I can see.

So the project has been moved. I've migrated any issues that were open and that I feel need working on; most feature requests were culled out as I am not in a position to do custom development at the moment. Now that people can fork the project on GitHub I hope feature requests come in the format of pull requests or at least patches.

Lastly I've made a few other distribution improvements. The project is now on PyPi so you can install it using 'pip install django-helpdesk'. The listing over at DjangoPackages has also been updated so that you can see the PyPi downloads and mark yourself as a user of the package.

I would love to hear any comments or feedback you've got via e-mail or on Twitter (where I'm @RossPoulton. Enjoy!

DjangoSites Deployment Statistics

Every person that submits a site to DjangoSites gets a chance to include details about how they deployed their website: what database they use, what version of Django they use, and so on.

The aggregated statistics are now online for all to see. The deployment details for individual websites are not visible.

You can see the stats in a basic form along with some pretty charts on the DjangoSites website by clicking the Deployment Stats link in the navigation bar.

I'd love to hear any feedback you've got, please let me know your thoughts via e-mail or Twitter

Gracious E-Mail Bounce Handling in Django with Postmark

Recently I've been on a quest to simplify the way I deliver my websites to my customers. Not that my customers know: the primary changes are relating to server monitoring, being proactive about a few things, and getting rid of elements that I don't understand.

One of the things I really didn't understand that well and wanted to offload to somebody else was my email delivery. I already use a company called Tuffmail to handle my mailboxes (that is, for my actual email boxes - not email sent from within my Django applications) and I'm extremely happy with their service. I use and trust Tuffmail because I know I can't keep on top of everything I need to know to run SMTP and IMAP services properly and securely.

The next step in my outsourcing process was to find a company to deliver email sent from within my Django applications. Until now I've been running a local Postfix server used for delivering mail but not receiving it, and for most emails that works fine. As websites grow, though, it becomes harder to manage what happens with bounced email and also to guarantee delivery - many email providers seem more than happy to mark emails form virtual servers as spam!

Getting Somebody Else to Deliver Your Mail

My shortlist for outsourced SMTP came down to two companies: Sendgrid and Postmark. Both of these services charge a small fee to become your SMTP server for outgoing email with a number of great features.

Sendgrid, in particular, has some fantastic features that you usually only see from mailing list providers such as the ability to track e-mail opens and links within emails. Both providers deal well with unsubscribe requests and let you easily categorise emails so you can get statistics based on deliverability of signup emails compared to purchase receipts.

My final choice ended up being Postmark, primarily for two reasons:

  1. Postmark allow you to control your SPF & DKIM settings per-sender, so email looks as if it comes from who you say it comes from. Sendgrid only allow this on more expensive plans, which are overkill for my needs - the end result is that some e-mail clients display the sender as "Sendgrid on behalf of XYZ".
  2. Postmark allow you to configure multiple 'servers', each with different API credentials, to separate email delivery for each of your apps or websites. I want a feature like this to be able to see different statistics for WhisperGifts as opposed to DjangoSites.

Configuring Postmark

Getting started with Postmark is easy. Just sign up and follow the instructions to create a server, a sender signature and then configure your Django installation to use them as an SMTP server (which needs to be enabled per-server within the Postmark system). If you're on Django 1.2 there is an EmailBackend by Paul Martorana which you can use and gain even more features than using plain ol' SMTP.

One of the real benefits, as fast as I'm concerned, is managing your e-mail bounces. I've recently implemented this code over at DjangoSites to help deal with users who mistype their e-mail address. I currently get about 4-5 of these a week, and as the site gets more popular the volume of bounced email is growing.

Configuring Django to Handle Bounces via Postmark

I have created a Python file called in my DjangoSites project folder. It's contents are as follows:

from django.core.mail import send_mail
from django.http import HttpResponse
from django.contrib.auth.models import User
from django.conf import settings

    import json
except ImportError:
        import simplejson
        from django.utils import simplejson

def postHandler(request):
    Gets POST'd data from Postmark's BOUNCE handler. We put a message against
    the user letting them know their email bounced.

    Requires HTTP Basic authentication with a username set in settings.POSTMARK_BOUNCE_PASSWORD.

    Configure Postmark to use ''

    Sample of the received JSON data in the HTTP POST body is:
          "BouncedAt": "2010-06-03T21:00:19.0155096-04:00",
          "CanActivate": true,
          "Description": "Test bounce description",
          "Details": "Test bounce details",
          "DumpAvailable": true,
          "Email": "",
          "ID": 42,
          "Inactive": true,
          "Name": "Hard bounce",
          "Tag": "Test",
          "Type": "HardBounce",
          "TypeCode": 1
    authorised = False
    if request.META.has_key('HTTP_AUTHORIZATION'):
        (authmeth, auth) = request.META['HTTP_AUTHORIZATION'].split(' ',1)
        auth = auth.strip().decode('base64')
        username, password = auth.split(':',1)
        if username == settings.POSTMARK_BOUNCE_PASSWORD:
            authorised = True

    if not authorised:
        response =  HttpResponse('Authorization Required', mimetype="text/plain")
        response['WWW-Authenticate'] = 'Basic realm="Bounce Handler"'
        response.status_code = 401
        return response

    if request.method != 'POST':
        response = HttpResponse('Data must be provided via HTTP POST', mimetype="text/plain")
        response.status_code = 405 # Bad Method
        return response

    # The data we're interested in comes in via the body of the HTTP POST,
    # rather than as post parameters. As such we need to read in the
    # raw_post_data of the POST rather than using request.POST.
    raw_json = request.raw_post_data
    data = simplejson.loads(raw_json)
    email_address = data.get('Email', None)
    bounce_type = data.get('Type', None)
    bounce_desc = data.get('Description', None)

    # We only write a message if the bounce is a 'Hard Bounce', this means
    # we aren't bothering the user for transient errors such as DNS lookup
    # failures, offline mail servers, or full mailboxes. Of course, if
    # these errors persist eventually we'll get a hard bounce and let the
    # user know.
    if email_address and bounce_type == 'HardBounce':
            user = User.objects.get(email=email_address)
            # Using messaging system from Django pre-1.2.
            user.message_set.create(message="An email to %s seems to have bounced back to us - the response we got said '%s'. Can you please check it by clicking 'Account Settings'?" % (, bounce_desc))
            # Not the best way to do this - however we don't want to
            # raise an error if the user doesn't exist (eg we are getting
            # a bounce for an email not directly related to a user in
            # our system)

    # All Postmark needs to get is a response, any response. No templates or
    # fancy content needed here!
    return HttpResponse("OK")

You will note that this refers to a particular settings called POSTMARK_BOUNCE_PASSWORD - configure a random password in your file:


Due to the way HTTP Basic authentication works (which we'll be using later) you should avoid punctuation in this password.

Lastly, set up to point a URL to our new bounce handler:

(r'postmark_bounce/', 'postmarkBounces.postHandler'),

Reset your Django application (if required - depending on your deployment method) and you're good to go.

Telling Postmark To Tell Us About Bounces

Over at Postmark, log in to your control panel and go to the 'Settings & API Credentials' page for your server. Note that each Postmark 'server' has it's own settings screen, so bounces are handled differently per-application.

In the 'Bounce Hook' field, tell Postmark the full HTTP URL to your bounce handler, including the password you entered earlier:

It should look something like the below screenshot:

Click the 'Check' button and you should see a message letting you know that the response code was 200 - all is OK! If you get an authorisation error (HTTP error 401) then you have entered an incorrect password. If you see a HTTP error 500, check out the e-mail Django should have sent to your server administrator for more details.

Show The User

The last piece of the puzzle is to alert the user when the email has bounced. Over at DjangoSites, I do this by using Django's messaging framework (it's a Django 1.1 installation - the syntax for this has changed in 1.2) as seen in the above code snippet. Note that I don't force users to confirm their email address before they log in - if your users have to do that, you'll need to be more creative with your notification mechanisms!

Add some snazzy CSS and you're good to go. This is what a DjangoSites user will see while browsing the site if we realise that emails to them have bounced:

This is pretty unobtrusive, and it is only shown to the user once. If you want to be stricter about enforcing a valid e-mail address, just modify the way you store the 'failed email' flag and report it to the end-user.

In Closing...

I moved DjangoSites to Postmark about a fortnight ago, and configured the e-mail bouncing in less than an hour last week. That's not a long time to a solid comparison, but I can already report that I am happy with the service I've received from Postmark. My other Django sites including Jutda and WhisperGifts are already sending e-mail via the service, and the per-website reporting is wonderful.

If you want to do something more complex with your bounces, then I suggest you take a look at the Postmark Bounce Documentation which explains a full API to review bounces as needed.

Seeing fewer delivery errors come into my personal inbox as a result of the Postmark service is a welcome change. The fact that the process of alerting the user and fixing their account is now automated means I save a bit of time and as my websites grow the time saved will continue to grow. As a small business owner who continues to work a full-time "day job", I can only see that as a good thing.

Update Your DjangoSites Screenshots When You Want

You can now update your DjangoSites website screenshot by simply editing your listing and ticking the box that says "Re-do Screenshot".

To edit a website, first log in to DjangoSites using your username or OpenID. Once logged in, click 'My Sites' in the navigation bar, then click on the website you wish to have updated.

From the website detail page, click the 'Edit' link next to your username. Make any changes you wish to make, click the Re-do Screenshot checkbox, and click 'Save'.

It's important to note that when you edit a website, it becomes un-verified so it will not be publicly visible. I manually review all submissions and edits, so it might take me a few days to verify and publish your listing after you edit it.

This is something that has been on the books for a while, and I'd like to thank Richard Barran for prodding me into getting the change done.

While you're editing your listing, you can update your deployment details to help with our Django Deployment Survey. The results will be published shortly for all to see - thank you to those who have responded so far!

Django Site of the Week: The A.V. Club

For various reasons I haven't been able to post a Django Site of the Week for quite a while now, which is a little embarrassing. I've had this little interview sitting in my inbox for over a month, and I've only just been able to write it up and get it published.

Just about everybody online has come across The Onion at some stage. Whether you have interpreted a satirical news story as the truth, or just read it for a laugh, The Onion is a staple of online publishing.

A less-satirical yet still enjoyable feature of The Onion is The A.V. Club. Originally part of the printed Onion newspaper and then a PHP-powered Drupal based website, The A.V. Club is now running on Django - and the team who built it love Django to bits.

This week, I spoke to Tom Tobin, web developer at The Onion. We discussed how they built the A.V. Club website, how they manage with a million unique visitors a month, and what their plans are for the future of The Onion's online presence.

You can read the interview and leave your comments at the Django Site of the Week Website.

The Django Deployment Survey: How Are You Hosting Django Websites?

According to the official Django Documentation, there are around a half-dozen documented and supported ways of deploying Django in production environments. The recommended way these days is with Apache and mod_wsgi, but there are still a whole bunch of decisions that one must make when it comes to making their Django project public.

One of the great things about running DjangoSites is that it has exposed me to a number of fantastic Django-powered websites. Each of these is unique in one way or another, but they are all running Django. How they use it, though, is anybodies guess.

So what are our peers doing? There are proponents for and against each of the different pieces of the Django Deployment puzzle.

  • What operating system should be used? BSD, Linux or Windows?
  • What Database system should be used? Postgres, MySQL, or something else?
  • Which web server software? Apache, nginx, or one of the many python-powered web servers?
  • Which method should I use? mod_python, FastCGI, or something else?
  • Which version of Django? Do I stick to a release, or try and keep up with SVN?

To see what our Django-using peers are doing, I am starting a survey of Django website deployment methods. This will be an ongoing addition to DjangoSites.

The way it works is pretty straightforward. As of tonight, there are a number of new fields on the 'Submit a Site' and 'Edit' screens to let you select the way you have deployed your Django-powered website. I will not be publishing these details on a site-by-site basis, so you can rest easy knowing that I won't tell the world that your website is hosted on Windows with an Oracle database backend.

Once there are a material number of responses to those questions, I will publish the statistics in aggregate with some pretty colourful charts. These graphs will remain public (and dynamic - updating as the data grows) once the volume of submissions provides meaningful data.

What will this give us? An interesting look at the way that the wide public deploy their Django applications, and potentially longer-term trends showing the usage of particular deployment methods as time goes on.

Want to be involved? I'd love to hear your feedback and see as many sites as possible have their deployment details listed.

If your site is already listed at DjangoSites, you can simply log in and click 'My Sites' in the toolbar. Open each site that's listed, and click the 'Edit' link. Alternatively, if you have a whole bunch of websites you want to update, just email me your DjangoSites username and deployment details and I'll do the update for you.

Don't despair if your websites aren't listed at DjangoSites yet - you can simply sign up then submit your websites for free! Remember, the deployment details are optional so you can list your website without being included in the aggregated statistics if you prefer.

Lastly, I would like to thank Joshua Jonah of Symbiosis Marketing for planting the seed for this idea. I'm hoping it'll give an interesting insight into more of the 'behind the scenes' details of Django-powered websites.

Django Site of the Week: Deskography

Seeing how other people work is something that seems to be of interest to most developers. Whether it's because they want to become better workers themselves or because they're somewhat voyeuristic is open to debate - either way, Django-powered website Deskography is a well-designed social desk-sharing website. This week, I spoke to Gustaf Sjöberg of Distrop to find out why they chose Django to power Deskography, and what it's allowed them to do. You can read the interview over at the Django Site of the Week.


Want to see more? Check out the yearly archives below.