Using javascript to apply "noindex" to get SharePoint to avoid crawling certain content

Question

I have a problem with navigation elements being crawled on our SharePoint site. These elements come from a custom control that's used for building the site navigation, so the output is just html that might look like this simple example:

<div class="nav_root">
  <ul class="nav_root_wrap">
    <li class="SomeLink twoline first" id="SomeLink">
      <a href="someLink.aspx">some link</a>
    </li>
    <li class="AnotherOption twoline second" id="AnotherOption">
      <div class="flyoutMenu">
        <a href="#">more links</a>
         ...
      </div>
    </li>
  </ul>
</div>

It looks like adding the "noindex" class to the elements in the html should do the trick. However, because I don't have very much control over the css classes that get applied to this content on the server-side, I'm wondering if I can add some jquery like below to apply the css classes on the client-side.

$('.nav_root *').addClass("noindex");

Does anyone know if this should work in SharePoint 2010's Enterprise Search (i.e. not FAST Search Server)? I can't tell if the search crawler is going to allow javascript to run before looking at the content of the page.

Update: I'm not having any luck getting this to work. This seems to indicate that the content of the page is crawled without javascript being taken into account. I really hope that I'm just doing something wrong, so please let me know if this is not accurate.

Update2: It sounds like some folks have had luck creating a user control that prevents content from rendering for a particular user agent. Certainly an interesting option, though it's not what I'd hoped for: http://underthehood.ironworks.com/2010/05/sharepoint-search-partial-page-exclusion.html

Answer 1

As far as I'm aware, the crawler won't 'run' the javascript on the generated page, therefore this technique probably won't actually work. Can't you add a noindex class to the container div of the nav control on the masterpage?

Answer 2

You can use the following code if you do not want the webpart to be crawled(if the navigation control is a separate webpart).

protected override void CreateChildControls()
{
if (!string.IsNullOrEmpty(Request.UserAgent))
{
  if (Request.UserAgent.ToUpperInvariant().IndexOf("MS SEARCH") > -1) 
  {
    return;
  }
}

}

Reference : How to prevent a SharePoint web part from being crawled

Answer 3

I've definitely seen this operate in reverse: I had a bear of a time getting content to show because of that darn noindex class.

Your issue looks like you're using the class (.) operator instead of the id (#) operator. I think your JQuery should look like this:

$("#nav_root").addclass("noindex");

If you just want to block the entire page and don't care that this process might take an extra second or two to load, you could just call every element on the page (warning: often several thousand on an SP page):

$("*").addclass("noindex");

The other thing you can take advantage of is that SharePoint won't crawl any information hidden behind a div tag with the "display:none;" element applied to it. You could just add the above style tag to your page if for some reason the noindex class continues to not work and then add some sort of toggle switch to display the data for end users.