Tag Archive: programming.

Djangosites Open Sourced

Back in 2008 I started djangosites.org as a listing of websites powered by Django. Prior to that, we relied on a wiki page to see who was using Django, so an image-based website felt like a big improvement.

Since day one I've promised to release the source code that I use for the site. It's relatively simple, so I never stressed much about making it a high priority - but I continue to be asked and politely berated for not getting it published.

Today that's changed. I think it's too late for me to say I've come good on my promise, but the Djangosites source code is now available on GitHub.

The README has more details, but in short this is a dump of the code currently running the site. I'll continue to use this repository as changes are made to the live site, however I'm not actively working on djangosites at this point in time (other than reviewing & approving submissions)

There's a few pieces of this that might be useful for people new to Django, but otherwise this is really a collection of generic views. The useful bits might be:

Suggestions and pull requests are welcome, but I'm not actively soliciting changes. I should probably clean things up a bit given that this codebase hasn't changed materially since Django 0.96, other than slight refactors to allow upgrades to work - so I'm certainly not yet taking advantage of new functionality that's been made available in recent Django versions. Perhaps now you can see how bad the code is, I'll have more of an incentive to fix it :)

The source code is available right now from GitHub under a mixed licence: the Python code is MIT-licenced, and the rest (HTML etc) is not open source but included in the repository for completeness and as an example.

Tracking CPC Results in Django

Like many startups, I use CPC ads to attract attention to WhisperGifts. I bid a set fee per click for particular search words on Google, and for ads shown to my target demographic on Facebook. I wanted to track an individual signup to their source CPC campaign, so put together a really quick bit of Django middleware to help me out.

All I do is stash the value of utm_campaign (set by Google or Facebook) into a cookie called wgcookie.

class CampaignMiddleware(object):
    def process_response(self, request, response):
        from registry.models import Campaign
        if 'utm_campaign' in request.GET.keys():
            if 'wgcampaign' not in request.COOKIES.keys():
                # add campaign cookie
                c = request.GET.get('utm_campaign', None)
                response.set_signed_cookie('wgcampaign', c, max_age=31536000, httponly=True)
        return response

At signup time, I read out the cookie into the users table:

    campaign_code = request.get_signed_cookie('wgcampaign', None)
    campaign_code = None

user.campaign_code = campaign_code

Simple! I've now got the capability to track actual revenue by campaign source - a very handy tool to identify which campaigns might warrant higher or lower spending.

I'm aware this isn't rocket science, but I figured it's worthwhile sharing - it makes more sense to me to track these things directly in my own database, than to try and match data from the AdWords panel with various analytics services.

Happy CPC bidding!

New Podcast: Django Roundup

Lincoln Loop are one of the earlier Django-based development shops, and their various employees contribute in many ways to the open-source community. One new addition they've just made is the launch of Django Round-Up, a podcast covering the news in the Django community.

This is a podcast hosted by @kennethlove and @bkonkle from @lincolnloop that highlights recent articles and projects in the Django community. We love talking about web development, so our podcast focuses on casual conversations as we cover the latest blog posts and project releases.

I was surprised to hear my name coming through my headphones, only a minute into their first episode - with a quick review of my recently-published django-readonly-site package.

As a result of their comments I've made some minor updates to address questions and suggestions from the podcast team.

I want to publicly thank them for including my item in their inaugural episode, and suggest that anybody in the Django community goes out and checks out this valuable new resource!


Occasionally I need to take WhisperGifts offline, but still show some parts of the site to users. This has included some system changes that require the site to be non-functional for a little while (such as doing a deployment with a bunch of backwards-incompatible changes, or large database migrations) and for server moves, whilst waiting for DNS changes to propogate.

To do this, I wrote a little library that I could toggle within my Django settings. I've just pulled it out of the WhisperGifts codebase, and django-readonly-site is now available on GitHub. I think it's pretty simple to use.

Install it with pip install django-readonly-site, add readonly to your Django projects' settings.INSTALLED_APPS, and set settings.SITE_READ_ONLY = True. More options are available to keep parts of your site online, see the README for more details.

By keeping parts of your site online (such as the homepage, about us page, and in my case a customers' registry listing) you can provide a transitional experience to users, while the database-intensive and high-integrity parts of the site (such as signup, account management, and checkout) are taken offline with a polite "Sorry, we're temporarily unavailable" message.

Just after I had to quickly move to Rackspace after an outage with my previous web host, Rackspace announced that they now had a public cloud offering in Australia. For performance reasons, I'll be moving from DFW to SYD soon - and I will use django-readonly-site to try and minimise the perceived downtime for my users.

Your thoughts, suggestions, and pull requests are welcome on the GitHub Project Page.

Getting Paid in Django with Pin Payments

Payments in Australia are controlled by the so-called "big four" banks, and it's been difficult for a long time for startups to get merchant facilities to process credit cards online. Accounts cost hundreds of dollars per month, with high transaction costs and minimum transaction volumes thrown in.

We watched with teary eyes as companies such as Stripe launched and made it easy for developers to process credit cards, and kept struggling with the PayPal "API" with hope that one day we'd see Stripe in Australia.

I was excited, then, to see that Pin Payments have just publicly launched in Australia. They're currently offering a $9/month account fee, which isn't free but it's certainly ideal for small companies. After June 2013 this will jump to $50/month, so it seems to make sense to sign up now if you've got any medium-term plans.

Like Stripe, Pin offer a JavaScript Library for processing payments on your page without ever handling credit card details. This makes it much quicker and easier to implement.

Because I process payments in multiple places on WhisperGifts, I needed a reusable way to render the payment form and process the payment in a worker queue. The resulting code has been packaged up as django-pinpayments, which is still simple and in an Alpha state but it's a great starting point.

The code is released under the BSD licence, and I'd love to get your suggestions, comments, and GitHub Pull Requests.

A few things on the to-do list include providing Celery tasks out of the box, and providing tests & documentation. If you can help with any of these I'd be very happy :)

In the meantime, I'm about to go live with credit cards on WhisperGifts for the first time. I'll keep PayPal around as an alternative, but if demand is low I won't hesitate to turn it off. I can't wait.

The Definitive Answer To "Can I Use Django For This Project?"

Short: Yes.

Longer: Almost certainly. If you don't know any technical reason why Django isn't a good fit, then Django is probably a good fit.

How I Moved My Commercial Projects to Newforms-Admin

My projects were all running on an SVN checkout from late April 2007, after the Queryset-refactor branch was merged into trunk. This meant that I had to make a number of changes, on a public server, to incorporate modifications to file uploads, generic create/update views, and more.

In this post I'm only going to cover how I did the change to Newforms Admin, as the other changes were relatively simple for my projects.

I made these changes on a live server, whilst my projects were running. For the volume of changes I had to make, this was very straightforward. I use FastCGI, and because the FastCGI processes were already running I was able to modify the Python code without it taking effect until I restarted FastCGI.

As always, make sure you've got a backup that's easy to roll back to.

The first thing I did was create my admin.py files in each application. For example, for DjangoSites I removed this admin definition from my 'Website' model:

    class Admin:
        list_filter = ('verified', 'screenshot',)
        list_display = ('url', 'title', 'owner', 'created', 'verified', )
        search_fields = ('title',)

and moved it into websites/admin.py:

from django.contrib import admin
from djangosites.websites.models import Website

class WebsiteAdmin(admin.ModelAdmin):
    list_filter = ('verified', 'screenshot',)
    list_display = ('url', 'title', 'owner', 'created', 'verified', )
    search_fields = ('title',)

admin.site.register(Website, WebsiteAdmin)

It's useful to note that at this stage, if I restarted my Django FastCGI instances, nothing would be broken - but the Admin wouldn't work.

Next, my urls.py file had to be updated. Out with the old:

(r'^admin/', include('django.contrib.admin.urls')),

and in with the new:

from django.contrib import admin
urlpatterns = patterns('',
    (r'^admin/(.*)', admin.site.root),

The last step to do before restarting FastCGI was to update Django. All i did was CD into my Django trunk folder and run svn up.

Finally, I restarted FastCGI and browsed to /admin/.

This is very straightforward, as most of my models have basic Admin requirements. For applications where you don't need any fancy Admin functionality at all, your admin.py can be even simpler:

from django.contrib import admin
from mysite.models import FirstModel, SecondModel, ThirdModel

for model in [FirstModel, SecondModel, ThirdModel]:

The only real gotcha with doing an svn up is that there have been a heap of backwards-incompatible changes made recently, so you really need to make sure you work through the list and make the required changes. The main one that tripped me up was on WhisperGifts where I still used oldforms in a few places. After moving these to newforms to match the rest of the site, it all worked a charm.

I recommend updating Django incrementally - eg update to 7476, make changes required for queryset-refactor, update to 7814, make changes required for file uploads, etc. This makes it easier to manage than one huge update.

Luckily as we approach Django 1.0 backwards-incompatible changes shouldn't be as regular, and most people should be able to stick to the official 1.0 release. This will be a godsend for those distributing applications, as they will be able to recommend a particular Django release and know it is stable and it works correctly.

Congratulations to everybody on the Django team who contributed to newforms-admin going live. I've only covered a raw conversion from the old admin to the new here, and haven't even looked at the new features that are available like application-specific admin screens and admin-level managers (to filter data at the admin level). This is a fantastic change to Django and everybody involved has put in a mammoth effort. Kudos to you all.

Jutda: Django-powered Solution Provider

In my recent catchup blog post I mentioned in passing a few projects including Jutda and WhisperGifts. Now, I'd like to formally introduce the former of these (with the latter coming very soon now)

When I'm not 'at work', I spend a significant amount of time working on other projects both independently and together with other talented developers. These days, all of these projects are powered by Django.

Jutda is the glue that will pull these projects together under a single name.

The name comes from the nearly-extinct Wagiman language, spoken by a small number of indigenous Australians. It means "to point" or "to show", which I've taken on as the ethos of my new company: We’re aiming to show the way to others, to make life easier through the use of elegant online solutions.

The ultimate aim is to create simple solutions for everyday problems. No bells, no whistles, just beautiful outcomes for ugly problems.

More information about Jutda can be seen on the About Jutda page, and in the initial blog post I published on the Jutda website.

The first project that's been finished and published is WhisperGifts, about which I'll write more in a future blog post. I'm very excited about the possibilities of this, and the impact it may have on couples in the process of organising their wedding festivities.

I've also got a few other projects in the pipeline, with no fixed feature-set or timeline. Projects that are well on their way to adulthood include:

  • A personal fixed asset register: Do you know the serial number of your television? The warranty details for your refrigerator? The phone number of the guy who borrowed your Playstation for his son to use over Christmas? What would you do if your house burnt down tomorrow, taking all of your warranty paperwork with it? I'm helping you sort out this mess.
  • A personal information manager: There's been many takes on these, but none of them suit my needs quite well enough. Why can't I keep track of pages of information along with photos, embedded maps, lists, enhanced details of my contacts, and more - all in one place?
  • An open-source ticket tracker / helpdesk system. We're currently using a Django-powered helpdesk system internally to manage WhisperGifts support queries, and we're in the process of cleaning it up and abstracting it out from our other systems so that we can publish it for everybody to see and use. It's written in Django, it has full e-mail integration, it has both public and private web interfaces, and it provides open access to your data. It rocks.

So that's Jutda. Although I'm not planning for it to take over and become a full-time job, I hope to get some useful tools out in the wild for everyday consumers, whilst contributing further code back to the Django community wherever possible. Wish me luck!

Overdue Catchup

I've had a very busy few months in every way conceivable - everything from my Django projects, to my day job, to life as a whole has been running in fast-forward. Here's a quick summary of the

DjangoSites is coming along very well, with 1040 sites listed as of this evening. The quantity and quality is ever-increasing, and more and more sites are being claimed. There are still well over 300 unclaimed sites - is yours listed there? If so, drop an email with your DjangoSites username to djangosites@djangosites.org.

After a server move late last year my Django OpenID project went offline for a little while. After a handful of requests from the blogosphere I've put it back online - see my original blog posting on the topic for more details, although I'm guessing that Simon Willison has something up his sleeve that'll trump my hack-job soon enougy.

Over my Christmas holidays I launched Jutda, the 'corporate' face for my upcoming web projects built with Django. The word Jutda is from the Wagiman language, a dialect spoken by an ever-shrinking Aboriginal tribe in the Northern Territory of Australia. It means show the way, which is something I hope to do with my projects. This is by no means my day job, rather a single name with which to pull together a number of after-hours projects. Hopefully the name starts to mean something within the Django community after a little while :)

The first project to be released by Jutda is WhisperGifts, a service to allow you to publish your wedding gift registry online with minimum fuss. I used it for my wedding almost a year ago with no problems, and it's been used by others before and since with great praise from happily married couples and their guests alike. Of course, it's all built with Django, with a whole host of neat features. When I have a spare evening I'll write up a few more details, but in the mean time check it out and let me know what you think.

Last but not least, I thought I'd bring attention to a simple Django application I wrote a while ago but never 'released' per-se. Django-forum is a simple Forum application for Django, allowing you to leverage your existing templates and user accounts to add discussion-forum capability to your existing project. At least a few people are using it, with a few patches coming from the community to add new functionality. I don't currently have any public sites running it, but keep your eyes peeled.

That's all of my news for now - I've got plenty more to share, however just a little more time is needed to give a few projects some more polish before I go public. Let's just hope my next update isn't another three-months away!

Django Menuing System

On most of the websites that I've built with Django, I have had a desire to be able to manage little elements of the website from the Django administration screen without having to touch my templates. My intent is for the templates to become the presentation vehicle, with anything that matters being built out of the Django databases.

One such thing that I want to keep out of my templates is navigation. Sure, the template has a place for navigation (including an empty

    ), but the contents of my navigation bars are driven by a dynamic Django application.

    The application has but two files: the models and a template tag.

    First up is the model file, menu/models.py:

    from django.db import models
    class Menu(models.Model):
        name = models.CharField(maxlength=100)
        slug = models.SlugField()
        base_url = models.CharField(maxlength=100, blank=True, null=True)
        description = models.TextField(blank=True, null=True)
        class Admin:
        def __unicode__(self):
            return "%s" % self.name
        def save(self):
            Re-order all items at from 10 upwards, at intervals of 10.
            This makes it easy to insert new items in the middle of 
            existing items without having to manually shuffle 
            them all around.
            super(Menu, self).save()
            current = 10
            for item in MenuItem.objects.filter(menu=self).order_by('order'):
                item.order = current
                current += 10
    class MenuItem(models.Model):
        menu = models.ForeignKey(Menu)
        order = models.IntegerField()
        link_url = models.CharField(maxlength=100, help_text='URL or URI to the content, eg /about/ or http://foo.com/')
        title = models.CharField(maxlength=100)
        login_required = models.BooleanField(blank=True, null=True)
        class Admin:
        def __unicode__(self):
            return "%s %s. %s" % (self.menu.slug, self.order, self.title)

    Next is a template tag that builds named or path-based menus - menu/templatetags/menubuilder.py:

    from menu.models import Menu, MenuItem
    from django import template
    register = template.Library()
    def build_menu(parser, token):
        {% menu menu_name %}
            tag_name, menu_name = token.split_contents()
            raise template.TemplateSyntaxError, "%r tag requires exactly one argument" % token.contents.split()[0]
        return MenuObject(menu_name)
    class MenuObject(template.Node):
        def __init__(self, menu_name):
            self.menu_name = menu_name
        def render(self, context):
            current_path = template.resolve_variable('request.path', context)
            user = template.resolve_variable('request.user', context)
            context['menuitems'] = get_items(self.menu_name, current_path, user)
            return ''
    def build_sub_menu(parser, token):
        {% submenu %}
        return SubMenuObject()
    class SubMenuObject(template.Node):
        def __init__(self):
        def render(self, context):
            current_path = template.resolve_variable('request.path', context)
            user = template.resolve_variable('request.user', context)
            menu = False
            for m in Menu.objects.filter(base_url__isnull=False):
                if m.base_url and current_path.startswith(m.base_url):
                    menu = m
            if menu:
                context['submenu_items'] = get_items(menu.slug, current_path, user)
                context['submenu'] = menu
                context['submenu_items'] = context['submenu'] = None
            return ''
    def get_items(menu, current_path, user):
        menuitems = []
        for i in MenuItem.objects.filter(menu__slug=menu).order_by('order'):
            current = ( i.link_url != '/' and current_path.startswith(i.link_url)) or ( i.link_url == '/' and current_path == '/' )
            if not i.login_required or ( i.login_required and user.is_authenticated() ):
                menuitems.append({'url': i.link_url, 'title': i.title, 'current': current,})
        return menuitems
    register.tag('menu', build_menu)
    register.tag('submenu', build_sub_menu)

    Using this menu system is relatively easy:

    1. Filesystem setup

      1. Create a directory called menu in your Python path
      2. Drop models.py (above) into the menu folder
      3. Create a directory called templatetags inside the menu folder
      4. Copy the menuubilder.py (above) into the templatetags folder
      5. Create a blank file called _init.py_ and put a copy in each of your menu and templatetags folders
    2. Django setup

      1. Add menu to the INSTALLED_APPS list in your settings.py
      2. Run ./manage.py syncdb to create the relevant database tables
    3. Data setup (using Django's Admin tools)

      1. Add a Menu object for each menu set you wish to use. Give static menus (eg, those that are the same on each page) a slug such as main, footer or sidebar. For dynamic menus, that display different contents on different pages, add a base URL. A menu with a base URL or /about/ might contain links to your philosophy, your team photos, your history, and a few policies - but only when the user is visiting a page within /about/.
    4. Template setup

      1. In your template, wherever you want to use a particular named menu, add this code:

        {% load menubuilder %}{% menu main %} {% for item in menuitems %}
      • {{ item.title }}
      • {% endfor %}

      1. Replace main with the name of a static menu (eg footer or sidebar from the above example)
      2. For dynamic URL-based menus, add this code:

      {% load menubuilder %}{% submenu %}{% if submenu %}

      {% endif %}

      1. If a menu has been set up with a BASE URI that the user is currently seeing, it will be displayed. If no such menu exists, nothing will be displayed.

    I use named menus for link bars in header and footers. I use URI-based menus to display a sub-section navigation, for example within the about area on a website or within the products section. A URI-based menu will only be displayed with the user is within that URI.

    So there you have it. Database driven menus that let you build easy static or URI-based menus and submenus. If you look carefully there is code to only display links to logged in users if required, and the page that is currently being viewed is tagged with the current CSS class so it can be easily styled.

    If all of that is confusing, stay tuned for my Basic Django-powered website in a box that will pull together a number of elements such as this into a simple to install package you can use to get a website up and running quicksmart - with the fantasmical automated Admin screens to control it all.

DjangoPoweredSites Grows Up

After some e-mail discussions with Jacob Kaplan-Moss from the Django team, I've moved all of the sites listed on the old DjangoPoweredSites wiki page to Djangosites.org.

It took a fair bit of time to weed out some not-working pages and expired domains (which have, of course, been purchased by spammers and link farmers), and a small investment in extra WebThumb credits to allow me to take screenshots of a fresh 500+ websites, but I got there in the end.

We'll now be closing the DjangoPoweredSites wiki page for good, as it has become redundant and is not the best way to peruse 800+ websites.

What we're left with is a fantastic directory of websites powered by Django. I'll be putting together a 'Featured' list shortly, but in the meantime I need a little bit of help from the community.

All sites scraped from the wiki page are currently unowned. I encourage you to check out the list of unclaimed websites, and e-mail me to let me know which sites are yours so you can receive due credit. There are some great websites listed there, so it'd be fantastic to be able to show off who built them.

Thanks to Jacob and the rest of the Django team for getting behind this initiative.

Easy Multi-Part E-Mails with Django

Every time I send e-mail with Django I seem to do it slightly differently, especially when it comes to sending HTML-based emails.

So, I did what any good Djangonaut does and wrote a quick function that wraps up sending of a multi-part HTML / Text e-mail from templates you create.

Firstly, create your e-mail templates. You'll need both a HTML and a plain-text template; for the HTML template I strongly recommend the use of Premailer, which puts all CSS definitions inline.

Here's a sample HTML template - as it would come out of Premailer. We'll pretend it's in the emails template directory, with a name of newsite.html.

<h1 style='font: Arial; font-weight: bold; font-size: 15pt; color: #006;'>Thanks!</h1>

<p style='font: Arial,Helvetica; size: 10pt;'>Hello, {{ username }}.</p>

<p style='font: Arial,Helvetica; size: 10pt;'>Thank you for submitting your website, titled <strong>{{ title }}</strong>, to our listing.</p>

<p style='font: Arial,Helvetica; size: 10pt;'>We'll verify it, take a screenshot, and publish it within the next few days.</p>

<p style='font: Arial,Helvetica; size: 10pt;'>Regards,</p>

<p style='font: Arial,Helvetica; size: 10pt;'><em>Your Favourite Webmaster.</em></p>

And here's that same e-mail, in plain-text format - also in the e-mails folder, called newsite.txt.

Hello, {{ username }}.

Thank you for submitting your website, titled '{{ title }}', to our listing.

We'll verify it, take a screenshot, and publish it within the next few days.


Your Favourite Webmaster.

Those who are familiar with Django's template language will see that this template uses two variables, username and title. Let's populate them, shall we?

template_context = {
    'username': 'joecitizen',
    'title': 'My Movie Review Site',
    'url': 'http://www.moviewreviews.dom',

So far we have an e-mail message template in both HTML and text formats; all we need now to send an e-mail is a subject and a recipient.

recipient = 'joe@example.dom' # ['joe@example.dom',] would also be suitable
subject = 'Site submission: {{ url }}'

That's pretty straightforward - our subject also contains Django template variables, in this case the URL from our template context we created earlier.

Now it's time to send the message. I have been using this function with much success:

def send_multipart_mail(template_name, email_context, subject, recipients, sender=None, fail_silently=False):
    This function will send a multi-part e-mail with both HTML and
    Text parts.

    template_name must NOT contain an extension. Both HTML (.html) and TEXT
        (.txt) versions must exist, eg 'emails/public_submit' will use both
        public_submit.html and public_submit.txt.

    email_context should be a plain python dictionary. It is applied against
        both the email messages (templates) & the subject.

    subject can be plain text or a Django template string, eg:
        New Job: {{ job.id }} {{ job.title }}

    recipients can be either a string, eg 'a@b.com' or a list, eg:
        ['a@b.com', 'c@d.com']. Type conversion is done if needed.

    sender can be an e-mail, 'Name <email>' or None. If unspecified, the
        DEFAULT_FROM_EMAIL will be used.

    from django.core.mail import EmailMultiAlternatives
    from django.template import loader, Context
    from django.conf import settings

    if not sender:
        sender = settings.DEFAULT_FROM_EMAIL

    context = Context(email_context)

    text_part = loader.get_template('%s.txt' % template_name).render(context)
    html_part = loader.get_template('%s.html' % template_name).render(context)
    subject_part = loader.get_template_from_string(subject).render(context)

    if type(recipients) != list:
        recipients = [recipients,]

    msg = EmailMultiAlternatives(subject_part, text_part, sender, recipients)
    msg.attach_alternative(html_part, "text/html")
    return msg.send(fail_silently)

Lastly, to put together the previous example:

from multipart_email import send_multipart_email # Assuming, of course, you saved the above functoin in multipart_email.py

send_multipart_mail('emails/newsite', template_context, subject, recipient)

What this does, is uses our existing template context, subject, and recipient, to send a HTML and plain-text e-mail based off the templates we built earlier. Note that it automatically appends .html or .txt as required. Hopefully the extra documentation in the docstring is useful.

Recipients who can read HTML e-mail will see your nicely formatted HTML e-mail message, while everyone else will see the plain-text alternative.

Slowly, this method is finding it's way into each of my Django applications. Please, just don't overdo the HTML messages!

Django: Multiple Aliases for a Single Website

In these days of cheap domains, it's often desirable to own multiple domains for a single website. You've probably got each of the .com, .net and .org domain names, along with a country-specific domain. You want each of these to present exactly the same website to the world, but good design says that each web page should have one, and exactly one, URL. So what's the best way to serve this up without having an Apache config for each domain?

I've come across this whilst building a website recently whereby the primary domain is mydomain.com.au, while I've got secondary domains in other popular TLD's to try and reduce domain squatting and the like.

One option is to configure an Apache virtual host for each domain, which serves up a static redirect. Another is to have Apache aliases for the main host, so each of the domains serves up the same content. This works, but leaves each page with multiple URL's.

My solution is to set up Apache aliases, and use a Django middleware to identify any requests that aren't for the main domain name, redirecting them as they're found. The middleware code I use is as follows:

from django.http import HttpResponsePermanentRedirect

class ValidateHostMiddleware(object):
    Redirect all requests for a domain other than mysite.com.au
    def process_request(self, request):
        if not request.META['HTTP_HOST'].endswith('mysite.com.au'):
            return HttpResponsePermanentRedirect('http://www.mysite.com.au%s' % request.path)

This is nice and simple, and a useful way of having multiple domains (possibly increasing your virtual 'geographical spread') but keeping your search-engine optimisation efforts intact.

Update: Thanks to a note from Brice Carpenter, I've updated the code to do a permanent HTTP redirect (code 301) rather than a temporary (302) redirect. I've also added code from my live environment that sends the visitor to their request path on the new domain - so hitting www.mysite.com/about/policies/ refers the user to www.mysite.com.au/about/policies/.

Mixing OpenID into Django's authentication system

NOTE: This code is now outdated, and it's certainly not the best way to do OpenID in Django. I recommend you take a look at django-authopenid, a fantastic registration system that combines Django's authentication framework with OpenID sign-in. Ross, 17th April 2008

According to the OpenID website, from a consumers point of view OpenID is "the elimination of multiple user names and passwords and a smoother, more secure, online experience. ". What it provides is a single identity for you to use at multiple websites. Instead of having a username and password for each website you peruse, you have an identity (usually a URL to your blog or an OpenID provider) that you use to login. The only password you have to remember is that of your OpenID provider - and you don't have to provide your password to any websites you visit.

Simon Willison has been a fantastic campaigner for OpenID, especially for integration with Django. He's written the fantastic django-openidconsumer package which provides the framework for a Django Application to act as an OpenID consumer (that is, people login to your Django app using their OpenID).

Simon's package creates a new OpenID object within your application, but is unrelated to the existing authentication system. So what I've done is mixed some glue to pull together these fantastic standalone applications:

The 'glue' provides a useful process flow for new users to your website. It lets people register with a username/password as they would have before OpenID, lets people login with OpenID's, and keeps it all intertwined. The basic functions are as follows:

  • A new user can sign up with a username/password, using django-registration
  • A new user can enter an OpenID, which is authenticated before creating a standard Django user. The account is verified using django-registration's email checker.
  • A user who is already logged in with a username/password, can login again using an OpenID, which is automatically associated with their username
  • A user can login with either their username/password (if they have one), or with their OpenID(s) (if they have any), and you'll always see them as 'request.user' - the same as if they had signed up with a username/password
  • This gives users a choice when they sign up and lets them change their mind by adding or removing OpenID's from their account as they wish.
  • Why do e-mail verification when users sign up with an OpenID? Because OpenID doesn't guarantee a users identity. It's merely a replacement for a username & password. E-Mail verification helps cut down on automated registrations, and ensures users are providing you with a valid contact address.

How would I use this in the real world? For sign-ups, I would show one form, clearly divided, asking a user to enter their OpenID if they have one, or otherwise enter their username/password/e-mail address. For logins, show users a username/password box with a link to toggle an OpenID entry field. If the user logs in with an OpenID, set a cookie so you can remember them in the future, showing them the OpenID login by default rather than the Username-driven login. Make it as easy as possible for people to adopt this fantastic technology.

I'm not using this code in a live environment yet, as it needs more testing. However, I've put a testbed online and made the source code available.

I would love it if you could try out this code both on my server and on yours, and provide feedback on the flow from a users point of view. I plan on cleaning up the code significantly before packaging it into something that is safe to use in a production environment.

The online demo is at http://openid.rossp.org. There is a link to the source code there.

Note: The database isn't accessible via the web, and it's a database set up specifically for this demonstration. I will NOT be extracting lists of usernames, passwords, e-mail addresses, openid's, etc. I may pu tup some usage statistics later, but certainly nothing identifiable. Please feel safe using this.

Validating a Username via jQuery with Ajax

It all starts when John hits your website and clicks the big 'Register' link. John types his name, '_John_' into the username box, and hands over his e-mail address and password (unless you're cool and hip, and you let him sign up using his OpenID) and hit 'Submit', just like every other website he's signed up to in the past.

Except this time, somebody else called John (what are the chances, eh?) has already signed up using that username, so after waiting a few seconds John sees an error message asking him to select another username. He types a new username and tries 'Submit' again, unsure as to whether his new selection will be suitable. So we fix this problem easily - we tell your users, while they're entering their username, whether their selection is available.

To achieve this we're going to use jQuery with it's fantastic Ajax support.

To get started, we create a simple view in Django that confirms the presence of a given username. This view will be accessible at /check_username/; and will expect a username via a GET paramater.

def checkusername(request):
    from django.contrib.auth.models import User
    from django.http import HttpResponse
    username = request.POST.get('username', False)
    if username:
        u = User.objects.filter(username=username).count()
        if u != 0:
            res = "Already In Use"
            res = "OK"
        res = ""

    return HttpResponse('%s' % res)

You'll now find that accessing /checkusername/?username=john_ will return 'Already in Use' or 'OK', as required.

The next thing to do is access it from within your registration form. I use James Bennett's fantastic django-registration with a customised template. This means I don't have to alter the registration code at all! I've added a little element just next to the username field, which we'll update shortly. The username field on my form looks like this:

<dt><label for="id_username">Username:</label></dt>
<dd>{{ form.username }} <span id='username_status'></span> {% if form.username.errors %}<span class="formerror">{{ form.username.errors|join:", " }}</span>{% endif %}</dd>

We've got an empty there called usernamestatus_. Now, when the user types a username, we want to check that username via background AJAX call and update the <span%gt; appropriately.

In the header of your registration form (I use a block in my templates called {% block extrahead %} which is inside the of my base.html) you will need to add a handful of lines of JavaScript:

<script type='text/javascript' src='/media/jquery.js'></script>
<script type='text/javascript'>
var previous_username = '';
var in_ajax = 0;
function checkUsername() {
    username = $("#id_username").val();
    if ((previous_username != username) && (username != '') && (in_ajax != 1)) {
        in_ajax = 1;
        $("#username_status").html("<img src='/media/busy.gif' />");
        $("#username_status").load('/check_username/', {username: username}, function() {in_ajax = 0;});
    previous_username = username;
$(function() {
    setInterval("checkUsername()", 1000);

This code is relatively simple. Firstly, it loads jquery.js, which I hope by now you've downloaded and placed into a suitable media directory. Then, using the last 3 lines, it causes the checkUsername() function to be called every second. This function does an AJAX request to our checkusername_ view and puts the response into our usernamestatus_ .

We do a few niceties here, too. Firstly, we make sure the username isn't the same as when we last checked it (no point in checking the same username every second - it would increase server load with no real benefit). We also put a 'busy' image into the status area whilst doing the AJAX call - I use a simple image from ajaxload.info.

Net result? The user gets real-time feedback on whether their username is available, so they don't have to think as hard when the page re-loads with an error message.

Note that this won't stop the user submitting a form with an invalid username - if they do submit it, they'll get an error message as per usual. All this function does is provide some nice feedback to users who wish to use it. Signing up to a website should be easy, after all.

DjangoSites: We Want YOU!

I've just approved another batch of sites bringing the total to 260. I've also had to delete a few sites that were submitted due to a number of reasons: A handful were submitted that are obviously not Django (providing your link as /index.php is a give away) and a few have been submitted that are inaccessible URL's.

If you've got a website built with Django, we'd love for you to submit it. There are about 600-700 sites listed on the wiki page, if we can get most of those moved over to DjangoSites.org it'd be fantastic. It's a great way to show off what's out there in the wild that's powered by Django, and a great way to get feedback on your sites.

For those learning Django, it's also a great way to see how others have built their sites. Our listing of sites with publicly available source is growing every week, and is a great way to view how different problems have been tackled.

In the meantime, maddiin and I are working on a repository for Django-powered applications. There are plenty of projects out there (especially on Google Code), however having listed together in one place can only be a good thing. At a glance, you'll be able to see what version of Django and Python you need for a given application, and you can set your user preferences to only show applications that run on your version of Django so you aren't constantly looking at potentially incompatible applications. Keep your eyes here for more upcoming information about this.

Djangosites Updates

Last week I launched djangosites.org, a new website to show off other websites built with Django. Uptake has been great (as I write this there are 125 websites listed) and feedback has been even better.

Firstly there are more RSS feeds. You can view RSS feeds for latest entries, particular tags (eg business), or individual authors. The 'RSS' button at the top-right of most pages links to an RSS feed of your current view, more will be coming soon.

Next, listings with source-code available are now more easily identifiable from the listings by a new icon: (a good example of this is the blog tag listing which has a few listings with source-code available)

We've also done a fair few minor cosmetic changes, eg signup emails now come from a sane email address, and the signup/login forms match the rest of the site

Any other suggestions are more than welcome, in the meantime maddiin and I are working on our next Django community project! Stay tuned for more details.

Introducing Djangosites.org

Until now, if you wanted to see a few sites that were built with Django you had to wade through a list of a few hundred links on a wiki page. The links were somewhat organised into categories, but no matter who you ask that's a boring job.

Djangosites.org has been built as a showcase of what's out there in Django Land.

We hope it lets you easily see what's capable with the Django framework. Feel free to add your sites, post comments on other sites, and vote for your favorites. What I'd really love is if some of the Django heavyweights could post some sites up - it would kick ass if this can eventually replace the DjangoPoweredsites wiki page.

Thanks to maddiin for the site design, and to the django-users mailing list for feedback and suggestions. If you have any feedback please don't hesitate to email it to me at ross at this domain.

Using The WebThumb API with Python

First things first: Get an API key from bluga.net by signing up for an account.

Next, drop the following code into webthumb.py and enter your API key.

Python interface to Webthumb API (see http://bluga.net/webthumb/)

By Ross Poulton - www.rossp.org

License: Use this how you like, just don't claim it as your own because
         that isn't cool. I'm not responsible for what this script does.

Usage: Define WEBTHUMB_APIKEY with your API key, as per the above URL.

Then, just call get_thumbnail(url, output_path). It will return true on
success, false on anything else.

An optional third parameter can be passed for the image size.

import time
import os
import httplib

import xml.dom.minidom
from xml.dom.minidom import Node

WEBTHUMB_APIKEY='Enter your webthumb API key here'



def get_thumbnail(url, output_path, size='medium2'):
    if size not in VALID_SIZES:
        return False

    request = """
    """ % (WEBTHUMB_APIKEY, url)

    h = httplib.HTTPConnection(WEBTHUMB_HOST)
    h.request("GET", WEBTHUMB_URI, request)
    response = h.getresponse()

    type = response.getheader('Content-Type', 'text/plain')
    body = response.read()
    if type == 'text/xml':
        # This is defined as 'success' by the API. text/plain is failure.
        doc = xml.dom.minidom.parseString(body)

        for node in doc.getElementsByTagName("job"):
            wait = node.getAttribute('estimate')
            key = ""
            for node2 in node.childNodes:
                if node2.nodeType == Node.TEXT_NODE:
                    key = node2.data

        # We're given an approx time by the webthumb server,
        # we shouldn't request the thumbnail again within this
        # time.

        request = """
        """ % (WEBTHUMB_APIKEY, key, size)

        h = httplib.HTTPConnection(WEBTHUMB_HOST)
        h.request("GET", WEBTHUMB_URI, request)
        response = h.getresponse()
        img = file(output_path, "wb")
        return True
        return False

From another Python script you can now create thumbs with ease:

from webthumb import get_thumbnail
get_thumbnail('http://www.rossp.org/', '/var/www/static/screenshots/rossp_org.jpg')

The error-checking is almost non-existant but so far it's worked for my purposes - I'm using it from within a cron job to fetch screenshots for new websites listed on a website.

Enjoy :)

Using Subdomains with Django

As a part of my previously mentioned upcoming bridal gift registry project (which, by the way, performed outstandingly in it's most important private beta ever - my own wedding) I'm giving each user (in this sense, a user is a couple close to getting married) their own subdomain off of the main website - instead of having a URL to their registry like http://yourdomain.com/registries/view/?id=1048 there are beautiful URL's like http://couplesnames.yourdomain.com.

Getting this working was really quite simple, and it's something that I can see being useful for SAAS projects - just take a look at the 37signals projects such as Basecamp and the way they handle per-company logins: each company has it's own subdomain.

The first thing that needs to be done is server configuration. Your domain name needs to have a wildcard entry setup (eg *.yourdomain.com), using Bind9 this is as easy as adding the following line to your domain name config. For other setups, speak to your web hosting provider.

*       CNAME   yourdomain.com.

Next, configure your web server software to answer requests for *.yourdomain.com the same way as www.yourdomain.com. I'm assuming that www.yourdomain.com already works, and that you're using Apache - again, YMMV so speak to your hosting provider. In Apache, just add the following line to your yourdomain.com virtual host configuration:

ServerAlias *.yourdomain.com

After restarting your DNS & web server software, you'll find that going to http://anything.yourdomain.com shows up the same result as http://www.yourdomain.com. Perfect.

Lastly you need to configure Django to treat subdomains differently. In this case I'm making another assumption, that Django is handling requests for yourdomain.com, and that there is a view already in place (eg via urls.py) for handling requests for the root of the domain. I'm also assuming content for your "main page" (eg, anything that's not a subdomain) is handled by other views, and that your main page is under /mypage/.

My 'index' view therefore looks like this:

def index(request):
    django_site = Site.objects.get_current()
    if request.META['HTTP_HOST'] == django_site.domain:
        # The visitor has hit the main webpage, so redirect to /mypage/ 
        return HttpResponseRedirect('/mypage/')

    # Split the domain into it's parts, remove the main part of the domain 
    # from the requested host, and we're left with one variable: 'subdomain'.
    # We also strip out the 'www.' non web-savvy users often type 'www' in 
    # front of every website they visit, so let's not show them an error message!
    domain_parts = django_site.domain.split(".")
    domain = ".".join(domain_parts[1:])
    subdomain = request.META['HTTP_HOST'].replace(domain, '').replace('.', '').replace('www', '')
    # You can now access your data models using 'subdomain', eg:
    couple = Couple.objects.get(subdomain=subdomain)

You can see I'm referring to a model called Couple, with a field named 'Subdomain'. All you need to do is refer to your own model with a 'subdomain' key and you're in business - just remember to enforce unique subdomains at signup time! Lastly, ensure your current Django 'Site' entry (django.contrib.sites) is up to date with the domain set to 'www.yourdomain.com'.

From here you can customise what's shown to the visitor based off of the subdomain they've hit. Very easy to setup, and potentially very powerful - especially in environments where people need something as easy as possible to remember or as unique as possible.

Helping Sort Django's Error Emails

When you run a Django powered website with debugging turned off, whenever a condition happens that would cause a HTTP 500 Error (such as an uncaught exception), the administrators receive an e-mail with a traceback and other relevant details to help debug the issue. These e-mails are really, really handy.

If however you run multiple Django sites, all with Debugging turned off, it can become difficult to tell which emails and errors relate to which site.

There are two solutions:

  1. Write perfect code that never throws errors; or
  2. Get Django to prefix the subject of the error e-mails so you can tell what belongs where.

Since there are only 24 hours in a day (most of which when I'm either sleeping or at my non-Django day job which pays the bills) I've opted for #2. And doing that is easy.

All you need to do is set the EMAIL_SUBJECT_PREFIX variable in settings.py:

EMAIL_SUBJECT_PREFIX = '[myproject] '

Easy as pie!

Blog Spam Protection

I currently manually review and delete comment spam every few days. It doesn't take long, it's just frustrating.

When you regularly repeat an action over a long period of time, you begin to notice patterns of action. What I've noticed is that most comment spam is being posted to older blog posts - most likely due to their higher number of in-links, and I'm guessing higher google page rank.

To this end, what I've done is add a method to my blog Post model, which checks the post date and, if it's within the defined timeframe, will return 'True'. My Post detail template now checks this method and only displays the form when appropriate.

The first file to change is your blog applications models.py. At the top of the file add this import statement to bring in the 'datetime' module and required tools:

from datetime import datetime, timedelta

Next, in the same file, add the following method to your 'Post' class (just after the 'body' field is fine):

    def allow_comments(self):
        return (self.date + timedelta(21)) > datetime.now()

Lastly, open up your blog detail template and replace the existing call to {% freecomment_form ... %}_ with:

{% if object.allow_comments %}
{% free_comment_form for blog.post object.id %}
{% else %}
<p>Comments can only be posted for recent articles.</p>
{% endif %}

What this does is pretty straightforward. Using datetime and timedelta objects, we ensure that only posts within the last 21 days can have comments posted to them. For me, this timespan works well, you may need to experiment with longer or shorter times.

Next up for me is actually altering the 'comments' mechanism to add filtering via Akismet or similar processes - but I'm wary of modifying the contrib code as I dread upgrade-time!

Simple Javascript Spell Checking

The solution I chose in the end was Speller Pages, a very simple javascript-driven user interface with a number of different server-side options, each using the GNU Aspell application to do the actual spell checking.

Installation is extremely simple. Download the archive, place it on your web server (under /speller/ seems to be the quickest to get going), edit the JavaScript file to point it at either a Cold Fusion, PHP or Perl back-end, and add some extremely simple JavaScript calls to the page where you want the spell check to appear.

From the users point of view, all they have to do is click the obvious "Check Spelling" button next to the relevant textarea, and a logical, simple to use window appears highlighting the incorrect spellings and providing a list of relevant alternatives. Once they're finished selecting replacement words, the window automatically disappears and the user continues on their merry way - without having to worry about the embarrassment of sending their friends and family to a website full of typos or misspellings.

Total integration time, including making a new spell-check button on the page (with a bit of assistance from the icons by Mark James) , was about fifteen minutes. Not bad for a piece of functionality that makes life just that bit easier to use.

The Pareto principle (Or, why the 80/20 rule always bites us in the ass)

This rant bought to you by yet another unfinished project.

As hard as you work to avoid it, there seems to be no way to avoid the good old 80/20 rule. It seems that no matter what you're working on, 80% of the work takes 20% of the time, and the remaining 20% of the work takes (at least) 80% of the remaining time on the project.

The '80/20 rule', as it's commonly known, is an implentation of the Pareto Principle, originally conceived by observing that 80% of income in italy was received by 20% of the Italian population. (Source: Wikipedia).

In software development this is very visible. It doesn't take a developer long to get a working mock-up online, complete with a basic interface and the most important functionality. But one can almost guarantee that this mock-up will not include some small yet important features that a public, commercial product requires if it's to be used by the masses.

I'm a big fan of Django. But I'm also annoyed at the way it lets us emphasise Pareto's principles. Rapid Web Application Development is the current 'in thing' - using Django or Ruby on Rails, one can get a basic CRUD application online with a minimum of fuss. It works well, it's stable, and it probably has a funky logo and some AJAX magic. There is a good chance it'll have some Google Ads on it to try and cover some hosting costs.

However it'll be missing all the details that make an application truly kick ass. It'll be missing the automatic e-mail notifications that get sent on a monthly basis to subscribers. It'll be missing the extra database fields that make the user experience even better by keeping detailed analysis of user activity to allow the site to be customised to their specific needs. It'll be missing the automated reporting tools that make mundane data truly valuable (both to management types, and to advertisers).

Maybe I'm just impatient, but in this day and age of 'release early, release often' there seems to be too many half-done projects. There are too many almost-there Web-2.0 weblications that, if given that 80% of time, will be something quite new. Earth shattering, even. Something that's ready to make it's developer piles of cash.

It seems there are too many good ideas floating around, but people lose the motivation to get them to that finished and polished stage that makes people say wow. Of course, there are exceptions - just take a look at Tabblo and Flickr to see how a great idea was turned fantastic with the use of some cool technology. Looking at these two sites makes me wonder how many image sharing and social network sites were started but never made it past the beta stage.

Me? I'm working on an online gift registry service for engaged couples. I'm a victim: the site works (and, I must say, works very well), and couples have successfully used it for their bridal registries, but it needs some work around the edges if I want it to be valuable to anybody other than my own small group of friends. There is no reason it cannot be monetized - but people only pay for quality.

Damn that last 20%.

ChangeManipulators on only part of a model

Django has some cool forms management functionality that takes the hard work out of displaying forms and managing the users input. Unfortunately, the ChangeManipulator expects that all fields in a model will be updated by the ChangeManipulator, and cries about any required/not-null fields that you don't display to the user in the form.

This is a huge issue when you want to display a form to a user that only modifies a very small part of a model. For example, you have a 'product' model with a few boolean fields, and you want the user to only alter the 'description' field with a ChangeManipulator. If you were to use the standard ChangeManipulator code (from the above URL) and only displayed the field 'description' to your user, the resulting code would either wipe the values of the boolean fields, or if they're required, complain that they don't have a value.

The fix is simple, and it's been in Django since the new-admin branch late last year. Why it isn't documented, I don't know. It's somewhat explained in Django Ticket # 420.

When you call your ChangeManipulator, there is a second parameter you can use to define which fields will and won't be edited, named 'follow'. It can be used to control multi-level relationships (see the ticket for details) but in this case we want it to ignore a few non-user-editable fields:

follow = {'paid': False, 'package': False, 'reminder_list': False, 'reminder_guests': False, 'security_basic': False}
manipulator = Account.ChangeManipulator(user, follow)

What this will do is ignore the paid, package, reminder_list, reminder_guests, and security_basic fields - and the user won't be prompted with an error message for not entering required fields.

What I'd like to see is a way to exclude all fields (without writing a loop to manually make them all 'False'), and then only include a small subset of fields. In my example, I have three separate screens to edit different sections of the same model - for technical reasons the information is on one model, but for business reasons it's spread over a few screens Using {'*': False, 'editthis_field': True}_ would kick ass.


Want to see more? Check out the yearly archives below.