Automatically tagging content is becoming easier with services like OpenCalais and Yahoo Terms Extractor, offering their APIs for free semantic analysis of content. There’s even a great Drupal module, Auto Tagging (with a great writeup on usage) that ties these services together and makes it even easier.
However, there is still one common issue with these services: they really need nicely written, rich, keyword dense articles to produce the most logical, semantic tags.
Try any of those services with user generated content and you’ll see a common tag each time around: FAIL.
We experimented with over 20,000 pieces of content on MothersClick and our results showed that these semantic services weren’t producing quality & relevant tags: rather, we were getting very little, if any relevant tags for our user generated content.
After a little more trial and error, I then noticed a simple pattern: more often than not, the title to a user’s post usually had the most applicable keywords to what their post was about, rather than the body of the post.
So how to extract just the keywords and make tags from the title of a node?
Well, taking each word in the title could work, but that would also include a bunch of words like “a, an, the, with” etc.. The more technical term for those words is stop words and luckily with some Googling, there are some nice stop word lists out there for filtering.
If you’re using Drupal and the active tags module, you can use the following code to automatically insert these suggested tags for the user as they create a piece of content.
Note: there is some extra code that strips non-alphanumeric characters and makes things lower case as well, this could be removed/changed based on your site’s requirements
While not perfect, we have found that this simple technique has resulted in quite an improvement in helping our users tag content, which when you consider the busy mom lifestyle, is a feat in and of itself! :)