10

We have a custom web part for our SharePoint 2007 farm that calls a very large report (lots of views and stored procs being called) that is used on hundreds of pages on team sites and MySites. We get notified that the server running Reporting Services gets slammed during the nights when a full crawl is active. When we looked at the logs we can see that the full crawl is requesting the pages with the report on it which initiates this resource-heavy report.

What we want to do is keep the report from being crawled.

Things I've already thought about are,

Using robots meta tag, e.g. <META NAME="ROBOTS" CONTENT="NOHTMLINDEX"/>, but the web part doesn't have access to the <head> element, right?

Using crawl rules would work if we didn't have so many pages with the web part on it and new ones get created all the time.

So, is there code to exclude pages the web part is on from being crawled/indexed? We will be moving to SharePoint 2010 so the solution needs to work there as well.

[UPDATE1]

I've tried add the "noindex" class to a DIV that wraps the web part but the content of the web part still shows up in searches. I can see the DIV when I view the HTML source so I know that part is working which leaves me with SharePoint 2007 must not be honoring the 'noindex' class. First attempt:

    Panel noCrawlPanel = new Panel();
    noCrawlPanel.ID="noindex";
    noCrawlPanel.CssClass = "noindex";
    ReportView.ServerReport.SetParameters(parameters);
    noCrawlPanel.Controls.Add(ReportView);

    this.Controls.Add(noCrawlPanel);

[UPDATE2] Another question about noindex pointed to an MSDN article on using class="noindex" in a DIV tag. It mentions that the "noindex" class only works for the DIV tag that contains it and any nested DIV tags will still be crawled. My problem is the Report Viewer creates it's own DIV like so:

<div id="ctl00_m_g_c0778131_60eb_4115_8ec6_24cbffe6fc34_noindex" class="noindex">
   <div id="ctl00_m_g_c0778131_60eb_4115_8ec6_24cbffe6fc34_ctl00" onclick="if (document.getElementById('ctl00_m_g_c0778131_60eb_4115_8ec6_24cbffe6fc34_ctl00_ctl00') != null) document.getElementById('ctl00_m_g_c0778131_60eb_4115_8ec6_24cbffe6fc34_ctl00_ctl00').ParametersController.HideActiveDropDown();" onactivate="if (document.getElementById('ctl00_m_g_c0778131_60eb_4115_8ec6_24cbffe6fc34_ctl00_ctl00') != null) document.getElementById('ctl00_m_g_c0778131_60eb_4115_8ec6_24cbffe6fc34_ctl00_ctl00').ParametersController.HideActiveDropDown();" style="display:inline-block;">

and continues into even denser JavaScript. What I'm going to try is putting the "noindex" class directly on the ReportViewer control. Second attempt:

    ReportView.CssClass = "noindex";

Assuming that the DIV above is the actual ReportViewer control and now a SharePoint wrapper, I don't need to worry about an existing value for the class. It did occur to me that this might work for DIV's that already have a CSS class:

    ReportView.CssClass += " noindex";

[UPDATE3] Here is what I am using now. It is based on the code from this blog post by Scott Tindall which I found in one the answers. 

protected override void CreateChildControls()
{
    string userAgent = this.Context.Request.UserAgent;

    if (userAgent.ToLower().Contains("ms search"))
    {
        LOG.LogInfo("Page requested by search crawler, removing ReportViewer");
        this.Controls.Add(new LiteralControl("Report not available to be crawled for performance reasons - PXD001"));
        return;
    }

    ... <normal web part code here>
}

I added the literal text to provide something to test against. If the crawler requests the page then the term 'PXD001' would be easily searchable to prove that the page did get indexed while the report contents did not.

I hard coded the user agent because I doubt Microsoft will change that without going to a new version of SharePoint. It might not work for SharePoint 2010, I still need to test it there. If it changes or we need to support multiple variations, I'll just move it to the web config at that point.

Improve this question

3 Answers 3

Reset to default
6

You have several options of excluding content in SharePoint:

  1. Crawl Rules
  2. tag
  3. Search Scopes
  4. robots.txt
  5. custom code to exclude content
  6. "noindex" tag (--> warning will still make SSRS requests, only not index the content)

1. Crawl Rules You say you don't want to use crawl rules as you have many pages you would have to exclude. Fine.

2. tag With the standard Report WebPart you cannot add anything to the HEAD, correct. You could however add meta tags by using content placeholders.

3. Search Scopes Same thing as 1. I believe - if you don't have a criteria for your web part containing pages you cannot create custom search scopes.

4. robots.txt Yes, SharePoint also cares about robots.txt. You could exclude the path leading to your pages containing the WebParts, but I guess that also would be many paths to exclude

5. custom code Just like explained here you can hide parts of your page by using custom code.

6. noindex - tag Last, but not least: noindex! I'm not sure whether SharePoint 2007 will follow this (2010 does), but when you create a simple div like so

<div class="noindex">
    <p>I will not be indexed!</p>
</div>

I would opt for option 6. You would only have to wrap the WebPart in a div, and yes you would have to edit each page containing the WebPart. Or you could edit the Master page / Layout and add the noindex there for the WebPart Zone containing the WebPart.

EDIT: As stated in a comment below: Most probably noindex will still create SSRS requests as SP renders the page. If it were very smart it would exclude that part from rendering, but I doubt it works that way. So option 6 doesn't work.

Improve this answer
5
  • As the problem is the load of the web part puts on the reporting server I'd opt for 1, 4 or 5 as they are the ones that either can prevent the crawler from loading the page or prevent the web part from rendering. If you can change the custom web part I'd put the check in 5 into the web part itself, then it's automatically solved everywhere otherwise you have a maintainance nightmare
    – Per Jakobsen
    Mar 20, 2012 at 19:29
  • About noindex option: Correct me if I am wrong but it will not prevent the web part from loading when the crawler requests the page. It will simply tell the crawler to ignore the data that is within the div. This will not prevent the reporting service calls when the web crawler requests these pages.
    – justforkix09
    Mar 21, 2012 at 10:02
  • The noindex option doesn't work because nested DIVs are still included in the index. The ReportViewer control uses several layers of DIVs and it's not feasible to write a custom report viewer just like it's unlikely Microsoft will update the ReportViewer to add 'noindex' to all DIVs.
    – Kelly S. French
    Mar 21, 2012 at 14:17
  • 2
    a 7th option is also possible: use javascript/ajax/jqeury to populate the webpart, instead of a direct html content. It does not always apply, but it can work. using this technique, the crawler won't see the content as it won't be generated in the page source.
    – Steve B
    Mar 21, 2012 at 16:45
  • You're probably right about the noindex. If SP is smart it doesn't even access the content within the noindex and doesn't render it. Most probably it renders the whole page and creates SSRS requests. 7th option: true!
    – Dennis G
    Mar 22, 2012 at 18:36
2

As you stated in your last update, checking for useragent is an option.If the user agent contains MS SEARCH, the rendering is stopped. This works (tested) in SharePoint 2007 and 2010.

if (!string.IsNullOrEmpty(Request.UserAgent))
{
  if (Request.UserAgent.ToUpperInvariant().IndexOf("MS SEARCH") > -1) 
  {
    return;
  }
}
Improve this answer
0

From your code it seems you are using custom code to put the web part on page

Therefore best option in my opinion would be to have a specific user credential for crawling (you may already have this). Store this username in some config store on the portal. Then on page load check current user name and do not load the Report web part if it is the user used by crawler.

As I explained in my comment, noindex will not prevent your web part from loading (and thus hitting Reporting Services) even if it worked in SP2007.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.