Unhandled Perception
From the mind of a developer.

Saturday, October 06, 2007

Robots.txt file for an SMF forum

To reduce the duplicate content issue with your SMF forum and the Google index (or any search engine) you should include the lines below (between the dashes) into your Robots.txt file. This will block all pages which can be considered duplicate from getting into a search engine index and/or get removed if already added.

--------------------------------
User-agent: *
Disallow: /forum/index.php?action=activate
Disallow: /forum/index.php?action=admin
Disallow: /forum/index.php?action=arcade
Disallow: /forum/index.php?action=calendar
Disallow: /forum/index.php?action=collapse
Disallow: /forum/index.php?action=deletemsg
Disallow: /forum/index.php?action=editpoll
Disallow: /forum/index.php?action=help
Disallow: /forum/index.php?action=helpadmin
Disallow: /forum/index.php?action=lock
Disallow: /forum/index.php?action=login
Disallow: /forum/index.php?action=logout
Disallow: /forum/index.php?action=markasread
Disallow: /forum/index.php?action=mergetopics
Disallow: /forum/index.php?action=mlist
Disallow: /forum/index.php?action=modifykarma
Disallow: /forum/index.php?action=movetopic
Disallow: /forum/index.php?action=notify
Disallow: /forum/index.php?action=notifyboard
Disallow: /forum/index.php?action=pm
Disallow: /forum/index.php?action=post
Disallow: /forum/index.php?action=profile
Disallow: /forum/index.php?action=register
Disallow: /forum/index.php?action=removetopic2
Disallow: /forum/index.php?action=reporttm
Disallow: /forum/index.php?action=search
Disallow: /forum/index.php?action=sendtopic
Disallow: /forum/index.php?action=splittopics
Disallow: /forum/index.php?action=stats
Disallow: /forum/index.php?action=sticky
Disallow: /forum/index.php?action=trackip
Disallow: /forum/index.php?action=unread
Disallow: /forum/index.php?action=unreadreplies
Disallow: /forum/index.php?action=who
Disallow: /forum/Themes/

Disallow: /forum/*.msg
--------------------------------

/forum/ should be replaced with the path to your forum

What is a robots.txt file:
http://www.robotstxt.org/wc/robots.html

Labels:

2 Comments:

  • Thanks! Exactly what I was looking for

    By Blogger CAD bloke, at 8:31 PM EST  

  • Thank you so much. This is what I've been looking for. My SMF site went down from 18,000 pages to 9,000 now. Perhaps this robots.txt you suggest will help. Thank you, again. from tubagbohol.mikeligalig.com

    By Anonymous Anonymous, at 4:47 AM EST  

Post a Comment

Links to this post:

Create a Link

<< Home