Search not respecting noindex

Question

I've got a content query web part embedded on a page layout in a publishing site. This content query web part rolls up the 10 most recent posts of a content type. In the page layout, I've wrapped the div with a noindex class and results in markup like this:

<div id="WebPartWPQ2" class="ms-WPBody noindex" style="" allowdelete="false" allowremove="false" allowminimize="false" onlyformepart="true" width="100%" haspers="true" webpartid="00000000-0000-0000-0000-000000000000">

I've also modified the ContentQueryMail.xsl file to wrap the div content with a noindex class as well.

<div id="cbqwpctl00_PlaceHolderMain_SocialContentDisplay_g_450c1fa5_7f6a_47aa_8af1_e43e0e2f8100" class="cbq-layout-main noindex">

In this site collection, I've got a particular content type scoped to it's own search scope. When I search against this scope for an article not in the top 10 of the content query, it correctly returns a limited number of items. When I search for an article within the top 10, it retuns the whole listing of the search scope.

It appears as if it isn't respecting the noindex attribute of the divs. Any others experiencing this? Using SP2010 Enterprise running SP1.

Answer 1

This is how we did something similar for our environment. On the master page or page layout you would create a declaration for this code and use a control to wrap anything you want to hide from the crawler.

Implementation and explanation from here: http://www.sharepointdesigners.net/profiles/blogs/sharepoint-search-partial-page-exclusion

was then translated into an implementation like this:

SearchCrawlExclusion.cs

//10/27/2010  This control was written to provide partial page exclusions when Sharepoint is crawling the page.
//It is applied to the master pages to "hide" specific content areas from the SharePoint crawler by not rendering them.  
//The business need was to exclude the content in the naviation and header/footer areas from being indexed with each page.


using System.Web.UI;
using Microsoft.SharePoint;


namespace NotMyCompany.SearchCrawlExclusion
{

    public class SearchCrawlExclusion : Control
    {
        protected override void Render(HtmlTextWriter writer)
        {
            bool render = true;
            try
            {
                SPUser cu = SPContext.Current.Web.CurrentUser;
                {
                    //Check to see if it's SharePoint crawling the page
                    if (Context.Request != null && Context.Request.UserAgent != null &&
                        Context.Request.UserAgent.ToLower().Contains("ms search"))
                    {
                        render = false;
                    }
                }
            }
            finally
            {
            }

            if (render)
            {
                base.Render(writer);
            }
        }
    }
}

None of this is my code, but it is used by permission...