Use Categories in Product URLs for Magento SEO without Duplicate Content

Magento Open Source eCommerce logo Magento provides the option to use categories in your product URLs for the search engine optimization benefits it brings, but they’ve implemented it in a very unusual way. Basically, if you enable this option, you’ll end up with at least 4 URLs per product. Once people realize it, this often raises a number of concerns including possible duplicate content penalty from Google, fractured link juice for product pages, frustrated admins, etc.

With so many people also searching for a clever solution, I decided disabling the feature was an unacceptable solution, and took the challenge.

First you have to enable the “Use categories path for product URLs” feature under System > Configuration > Catalog > Search Engine Optimization.

Use categories path for product URLs – This determines how the URL Rewrites autogenerate. If you choose Yes, the URL rewrite for products will include the associated category, and a separate rewrite will generate for each associated category. If you choose No, the URL Rewrites will include only the product name, and there will be only one rewrite, regardless of how many categories the product is associated to.

Once this feature is enabled, you’ll be to use any of these URLs to visit the same product page:

  • /catalog/product/view/id/<product_id>
    example: http://www.domain.tld/catalog/product/view/id/6
    Internal to Magento; never actually exposed.

  • /catalog/product/view/id/<product_id>/category/<category_id>
    example: http://www.domain.tld/catalog/product/view/id/6/category/10
    Internal to Magento; never actually exposed.

  • /name-of-product
    example: http://www.domain.tld/super-dee-duper-tent-1000-olive
    Normally used on the front page, in content blocks, or anywhere other than a category page. This is because without being on a category page, Magento doesn’t know which category would be appropriate to display in the URL (since it provides the ability to have multiple categories per product). Therefore it opts to not display any category at all. A bad decision in my opinion.

  • /category-1/sub-category-1/name-of-product
    example: http://www.domain.tld/sporting-goods/camping-hiking/super-dee-duper-tent-1000-olive
    Used from category pages.

  • /category-2/name-of-product
    example: http://www.domain.tld/affordable-housing/super-dee-duper-tent-1000-olive
    Used from category pages.

  • etc.
    Depending on how many categories per product.

Personally, I don’t like my products having multiple URLs. Between the front page and the category pages, two separate URLs are introduced to GoogleBot as it crawls through the pages. Even though you may not be seriously penalized for duplicate content, which page do you want to appear in Google SERPs? Which one do you want your customers to link at? You’re splitting your link juice and causing confusion.

When I saw this feature for the first time, all I really expected was a single URL like:

http://www.domain.tld/sporting-goods/camping-hiking/super-dee-duper-tent-1000-olive

Well, it took about four hours to find, but I finally came up with a patch to Magento core that delivers this result. Check it out below in unified diff / patch file format:


Index: app/code/core/Mage/Catalog/Model/Resource/Eav/Mysql4/Product/Collection.php
===================================================================
--- app/code/core/Mage/Catalog/Model/Resource/Eav/Mysql4/Product/Collection.php	(revision 2102)
+++ app/code/core/Mage/Catalog/Model/Resource/Eav/Mysql4/Product/Collection.php	(working copy)
@@ -553,7 +553,11 @@
                 ->from($this->getTable('core/url_rewrite'), array('product_id', 'request_path'))
                 ->where('store_id=?', Mage::app()->getStore()->getId())
                 ->where('is_system=?', 1)
-                ->where('category_id=? OR category_id is NULL', $this->_urlRewriteCategory)
+//                excluding this clause to facilitate one URL per product, and one that includes the category
+//                if a product has multiple categories, the first one (by category_id) will be used
+//                in most cases you'll probably only have one category because you only want one page per product for SEO reasons
+//                for maximum link juice, no possibility of duplicate content, and a less confusing store
+//                ->where('category_id=? OR category_id is NULL', $this->_urlRewriteCategory)
                 ->where('product_id IN(?)', $productIds)
                 ->order('category_id DESC'); // more priority is data with category id
             $urlRewrites = array();

In my setup, each product only has one category. Although this will also work if you specify multiple categories per product, you won’t have complete control over which category is used for the official link.

18 thoughts on “Use Categories in Product URLs for Magento SEO without Duplicate Content

  1. Stephan Miller

    Great post. I have been wondering about this lately. I have been watching sites with 1000 products have over 14,000 urls indexed in Google. That’s a lot of PageRank leakage. It’s one thing to have a big site, but another one altogether to create it from duplicate content. This post will help that a bit.

    They need the ability to add nofollow to specific links in Magento, especially empty review pages and the layered attribute navigation. I will be experimenting with nofollowing some of these links in the future to see what’s happens.

  2. matt

    Stephan – nofollow would only work for the bots and spiders. but if a user comes along and finds a product they like and links to it on their blog, and another user links the same product under a different category URL, then you’ve essentially 1/2′d the link-juice between those two links. So there needs to be a way to “physically” restrict product URLs to just one.

    Thanks for sharing the diff, Mike. Gonna give it a shot and see how it works on our setup which we’re launching … tomorrow! :)

  3. matt

    hm, this code fix isn’t doing it for me. I’ve got products in multiple categories, and if I arrive at each product through a different category, then the URL ends up being unique for that category assignment …

  4. Mike Smullin Post author

    @matt: Yep, you’ve correctly identified it is because you’ve got products in multiple categories. My setup has only one category per product, so that’s as far as I wanted to develop the patch.

    If you end up taking it further to where it can work with multiple categories, please feel free to post your link here and let me know.

  5. René

    Thanks for sharing this solution, Mike. It worked out pretty good for all the links on my shop – except for the products in the my block “new products”. The URL is untouched and this in the form root/product.html.

    I check my template and it calls the product link via echo $_product->getProductUrl() and returns the “faulty” URL. Could you give me a hint how to change these as well into SEO friedly links without losing link juice? ;-)

  6. Pingback: Magento SEO

  7. Shaun O'Hagan

    Hi Mike,
    I’ve found an answer to my own question. I tested that before the fix doing a search on my store always brought back products with simple product urls. After applying the fix the results come back as full category based urls. Cool.
    Nice work
    Shaun.

  8. Tonkywonky

    Folks, this is what the canonical tag is for!

    http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html

    As the nice man at Google says in this article “Additional URL properties, like PageRank and related signals, are transferred as well.”

    Yes they are!

    So it will index your preferred URL if you have multiple category paths, and importantly transfer link juice to that single page from other links pointing to it.

    And you don’t have to force users to see a URL in another category from where they expect to be.

    Its what the canonical tag was invented for!

    We’ve found category URLs are much better for SEO, especially for a big site and it avoids having to put category, and product manufacturer info into each page name.

    Unfortunately Magento doesn’t appear to support canonical links as standard (not in my version) but google for add-ons that do this.

  9. James Anelay

    Hi, Nice solution, I used this on my last magento install and it worked great. Im in the process of another install, do you still feel this is the best solution to the problem, or have things changed? afterall its been a year.

  10. Andrew

    Hi Mike,

    Thank you very much for sharing your solution!

    We implemented your patch on a client’s website (powered by Magento 1.4.0) by commenting out exactly the line that you mentioned, and it worked perfectly! Great work on your part.

    We seem to have the same problem as RenĂ© for our section “new products” and we are still looking for a solution. I’ll post our findings here when we do find a solution.

    Best regards,

    Andrew

  11. Mike Smullin Post author

    if there is, i haven’t found it. i’ll leave this comment up for readers to reply.

  12. Shaun O'Hagan

    Mike,
    Have you hit the problem with magento canonical urls not having the categories in them. I’m on 1.4.1. I would have thought your code would work to fix this as well – just don’t know where the canonical url creating code is in magento.
    Shaun.

  13. Aleksey

    Wow, this actually worked for my Magento 1.4.2. Thanks! I spent several days trying to fix this issue.

  14. vit

    Thank you Mike for this solution.

    I found solution with advanced catalog search results url rewrite.

    Magento 1.5.1.0

    /app/code/core/Mage/CatalogSearch/Model/Advanced.php:287 add method addUrlRewrite() to $collection:

    $collection->addAttributeToSelect(Mage::getSingleton(‘catalog/config’)->getProductAttributes())
    ->setStore(Mage::app()->getStore())
    ->addMinimalPrice()
    ->addTaxPercents()
    ->addStoreFilter()
    ->addUrlRewrite();

    I override this core file (only prepareProductCollection() function) with my custom model in /app/code/local folder.

  15. Elio

    Thank you for your patch Mike.

    Like suggested by Vit, it’s better to override the core file (app/code/core/Mage/Catalog/Model/Resource/Eav/Mysql4/Product/Collection.php) with a local file (app/code/local/Mage/Catalog/Model/Resource/Eav/Mysql4/Product/Collection.php), so an upgrade of Magento won’t destroy your changes.

    So:
    1) create the directory app/code/local/Mage/Catalog/Model/Resource/Eav/Mysql4/Product/
    2) copy app/code/core/Mage/Catalog/Model/Resource/Eav/Mysql4/Product/Collection.php to app/code/local/Mage/Catalog/Model/Resource/Eav/Mysql4/Product/Collection.php
    3) edit app/code/local/Mage/Catalog/Model/Resource/Eav/Mysql4/Product/Collection.php as suggested by Mike

    Don’t forget to disable Canonical Url for products, or you will still have duplicate pages.

    I also made 301 redirects from /product.html to /category/subcategory/product.html

  16. Paul Bates

    Great article however being a bit thick I don’t really know how to patch the Collection.php file (obviously I have to add code somewhere in that file, but where?). ((we won’t upgrade our magento store as it’s too much hassle))

    I thought it was quite interesting to see the comment that Google doesn’t penalize duplicate content, this is as I understand is only part of the picture.

    If you have a relatively new site it is important to make it very clear to the search engine what your site is all about, effectively dumbing it down to start with so that the search engine gets a grip on exactly what your site is about – in my case bb guns. It was explained to me by my SEO team that the duplicate canonical links just makes it a bit confusing for the bots to quickly identify my sites content, so although I am not being penalized for the content indexing of my site takes longer than it could have.

    Anyhoo, If a bit of clarity could be thrown my way in regards to the code insertion it would be greatly appreciated, I am running 1.4.1.1 with multiple storefronts with Yoast canonical url extension installed. And this mod… http://www.theintegrationengineer.com/magento-seo-canonical-links/

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>