1

I have a problem with navigation elements being crawled on our SharePoint site. These elements come from a custom control that's used for building the site navigation, so the output is just html that might look like this simple example:

<div class="nav_root">
  <ul class="nav_root_wrap">
    <li class="SomeLink twoline first" id="SomeLink">
      <a href="someLink.aspx">some link</a>
    </li>
    <li class="AnotherOption twoline second" id="AnotherOption">
      <div class="flyoutMenu">
        <a href="#">more links</a>
         ...
      </div>
    </li>
  </ul>
</div>

It looks like adding the "noindex" class to the elements in the html should do the trick. However, because I don't have very much control over the css classes that get applied to this content on the server-side, I'm wondering if I can add some jquery like below to apply the css classes on the client-side.

$('.nav_root *').addClass("noindex");

Does anyone know if this should work in SharePoint 2010's Enterprise Search (i.e. not FAST Search Server)? I can't tell if the search crawler is going to allow javascript to run before looking at the content of the page.

Update: I'm not having any luck getting this to work. This seems to indicate that the content of the page is crawled without javascript being taken into account. I really hope that I'm just doing something wrong, so please let me know if this is not accurate.

Update2: It sounds like some folks have had luck creating a user control that prevents content from rendering for a particular user agent. Certainly an interesting option, though it's not what I'd hoped for: http://underthehood.ironworks.com/2010/05/sharepoint-search-partial-page-exclusion.html

2
  • Can you add a wrapper around that control on the masterpage and apply the noindex class to the wrapper and not the actuanl navigation elements?
    – Luis
    Oct 3, 2013 at 22:13
  • Unfortunately, SharePoint requires "noindex" to be on each nested div. It's a good thought, and a bit frustrating that it doesn't work that way. However, I've verified this myself, and this link also refers to it: blogs.msdn.com/b/markarend/archive/2010/06/07/…
    – atcrawford
    Oct 4, 2013 at 2:18

3 Answers 3

5

As far as I'm aware, the crawler won't 'run' the javascript on the generated page, therefore this technique probably won't actually work. Can't you add a noindex class to the container div of the nav control on the masterpage?

3
  • I wish adding the noindex class to a container div would do the trick, but I guess the SharePoint team felt like that would be too easy or something :)
    – atcrawford
    Oct 4, 2013 at 2:25
  • Looks like that might work with FAST Search Server though, per this response to another question: sharepoint.stackexchange.com/a/46800/2995
    – atcrawford
    Oct 4, 2013 at 2:32
  • Marking this as the answer since it most closely addresses my question in the title. For the underlying problem, I actually wound up using the solution that I pointed out in "Update 2"; here's another link to the blog post.
    – atcrawford
    Oct 12, 2013 at 5:47
2

You can use the following code if you do not want the webpart to be crawled(if the navigation control is a separate webpart).

protected override void CreateChildControls()
{
if (!string.IsNullOrEmpty(Request.UserAgent))
{
  if (Request.UserAgent.ToUpperInvariant().IndexOf("MS SEARCH") > -1) 
  {
    return;
  }
}

}

Reference : How to prevent a SharePoint web part from being crawled

1
  • This technique is pretty cool actually - you can have a control with only this code in it, and wrap things on the masterpage you don't want indexed with this control, and it shouldn't crawl them. Used a similar technique on a project a few years ago and it worked like a charm.
    – James Love
    Oct 12, 2013 at 9:43
1

I've definitely seen this operate in reverse: I had a bear of a time getting content to show because of that darn noindex class.

Your issue looks like you're using the class (.) operator instead of the id (#) operator. I think your JQuery should look like this:

$("#nav_root").addclass("noindex");

If you just want to block the entire page and don't care that this process might take an extra second or two to load, you could just call every element on the page (warning: often several thousand on an SP page):

$("*").addclass("noindex");

The other thing you can take advantage of is that SharePoint won't crawl any information hidden behind a div tag with the "display:none;" element applied to it. You could just add the above style tag to your page if for some reason the noindex class continues to not work and then add some sort of toggle switch to display the data for end users.

2
  • I actually used the class operator intentionally as we have 3 generated navigation elements on each page (top, left, and bottom). I was able to verify that the javascript was correct by looking at the page in the browser.
    – atcrawford
    Oct 4, 2013 at 2:20
  • The part of your answer about "dispaly:none" is very interesting though... I might be able to hide the nav elements by default and use javascript to reveal them to normal users. I'll have to test out.
    – atcrawford
    Oct 4, 2013 at 2:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.