Skip to content

Automatically Extracting Tags from Nodes

2 min read

Automatically tagging content is becoming easier with services like OpenCalais and Yahoo Terms Extractor, offering their APIs for free semantic analysis of content. There’s even a great Drupal module, Auto Tagging (with a great writeup on usage) that ties these services together and makes it even easier.

However, there is still one common issue with these services: they really need nicely written, rich, keyword dense articles to produce the most logical, semantic tags.

Try any of those services with user generated content and you’ll see a common tag each time around: FAIL.

We experimented with over 20,000 pieces of content on MothersClick and our results showed that these semantic services weren’t producing quality & relevant tags: rather, we were getting very little, if any relevant tags for our user generated content.

After a little more trial and error, I then noticed a simple pattern: more often than not, the title to a user’s post usually had the most applicable keywords to what their post was about, rather than the body of the post.

So how to extract just the keywords and make tags from the title of a node?

Well, taking each word in the title could work, but that would also include a bunch of words like “a, an, the, with” etc.. The more technical term for those words is stop words and luckily with some Googling, there are some nice stop word lists out there for filtering.

Attached below is a simple JavaScript function for removing stop words from a string. Once the stop words are removed, you’re left with an array or string keyword candidates. Plug this into your tagging system and you’re off to a nice set of automatically generated tags. While not perfect, for user generated content these do work fairly well.

If you’re using Drupal and the active tags module, you can use the following code to automatically insert these suggested tags for the user as they create a piece of content.

Note: there is some extra code that strips non-alphanumeric characters and makes things lower case as well, this could be removed/changed based on your site’s requirements

 * Automatically determine Drupal taxonomy tags based
 * on the user entered form title.
Drupal.behaviors.autoTag = function() {
  var tagged = false;
  $("#edit-title").blur(function() {
    // only process this once to prevent tag oddness
    if (!tagged) {
      var words = ($(this).val()).split(" ");
      $.each(words, function(i, val) {
        // strip anything nonalphanumeric & make lowercase
        val = val.replace(/[\W]+/g,"").toLowerCase();
        // trim just in case of excessive spacing
        val = $.trim(val);

        // if this isnt a stopword add it as a tag
        if (!isStopWord(val)) {
          // add this to the first active tags enabled
          activeTagsAdd(Drupal.settings.active_tags[0], val);
      tagged = true;

While not perfect, we have found that this simple technique has resulted in quite an improvement in helping our users tag content, which when you consider the busy mom lifestyle, is a feat in and of itself! :)


Related Posts

Members Public

The Future of Automotive Mobility will be Powered by Open Source Software

Automotive companies are shifting from bending metal to bending bits. Soon they will be offering software and services to complement their manufactured metal. In order to win in the market, they will need to embrace open-source software to build leverage and enhance their offering.

Members Public

How to Host Your Static Website on Amazon S3: Step-by-Step Guide

1. Login to your Amazon AWS account 2. Create a new hosted zone for your domain using Route 53 3. Copy the NS records for this zone to your domain provider (e.g., GoDaddy) 4. Edit your domain on your domain registrar (e.g., GoDaddy) by clicking Set Nameservers 5.

Members Public

Book Review: Drupal 7 Module Development

Well, it’s been a while my Drupal friends, almost 18 months… dusting off this old Drupal 6 blog and time to start posting again. And what better way to get started again than by talking about Drupal 7, by far the best release of Drupal. And with almost 7