Stop Duplicate Content on WordPress with Robots.txt
Posted on February 4, 2008
Filed Under Blogging, SEO, Wordpress |
Is Duplicate Content an Issue?
Yes, you bet it is. It can negatively effect your search engine listings in three ways.
- Google selects a version that may not be your preferred copy.
- The duplicate content made rank higher than the original article/page.
- Google may detect your site has having a high volume of duplicate content and penalize your site.
A few resources about duplicate content:
- Supplemental Results & Duplicate Content Impact on SEO - A great video on youtube. A must watch.
- Duplicate Content - Google Webmaster Help Center - Comments on duplicate content from Google themselves.
- Deftly Dealing with Duplicate Content - A post on Google’s official blog about duplicate content.
Wordpress has a SEO weakness, and that is Duplicate Content!
I love Wordpress, it is fairly easy to SEO your site right out of the box with a good theme and a few plugins. As great as it is though, it is flawed. It produces many copies of the same content. One article may show on the homepage, archive, categories, tag pages, etc. This duplicate content can be reduce through good theme design, but there is a quick method that can be used to block Google and other search engines from ever seeing the duplicate content. This method is the use of robots.txt.
What is Robots.txt and How Does it Work?
Simply put, robots.txt is just a text file you place in the root directory of your site that contains information on how a search engine should archive your site. You can instruct bots to not access and archive various parts of your site. It is called the Robots Exclusion Protocol.
Visit the Wikipedia: Robot Exclusion Standard page for information on how to use robots.txt.
How to Set Up Your Robots.txt. for Wordpress
Open notepad and paste this into it:
User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /rss/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /date/
Disallow: /comments/
Then save the file as robots.txt, and upload to the root directory of your blog. The code is fairly self explanatory. It blocks bots from seeing the listed directory, and specifically blocks Googlebot from RSS and Feeds.
Warning: Be very careful when applying a robot.txt. Please read over the Wikipedia page, and do some Google research before you apply these changes. A mistake in your robot.txt could result in pages not being listed in Google. And pay special attention if you have your blog installed in a sub directory. The above code assumes the installation is in the site’s root directory.
While you’re at it, you may consider looking into selecting a preferred domain.
Related Posts:
Quick & Dirty Wordpress SEO Checklist
So you just installed Wordpress, and you’d like to SEO it out asap. Here is a quick list to SEO out any Wordpress install. A Good Theme This is a very important aspect of your sites SEO. You need a quality theme. When looking for a good theme, here are some points you need to consider. Clean and
WP Robots.txt Plugin
Yesterday I wrote an article on the benefit of having a robots.txt to prevent duplicate content on Wordpress. I just wanted to share with you KB Robots.txt, which a plugin that lets you edit your robots.txt directly from Wordpress. Here is my robots.txt
Yes WWW or No WWW
There are people on both sides of the issue. If you don’t know what I mean, let me quickly explain. When you enter the web address to a site, it will work with and without the www part. www.example.com goes to the same page as just example.com Now, which one is best? Well, that is a choice you
Important SEO Ranking Factors
We’re all trying to get an edge, and SEO provides us an ability to game the system a little bit, and give us a slight advantage. Many people are confused about SEO and struggle daily to improve their skills. For you new guys, here is a list of some of the most important on and off
Comments
Leave a Reply
