Search not indexing search results / content search webpart on a page

Question

I have a page which features a search results webpart to show the results of a preconfigured search query. The display template of the webpart has been modified to include a number of custom metadata fields. Those are displayed on the page.

Now I want the user to be able to find that specific page by searching the text as rendered by the display template. So when the custom metadata field contains "foobar", "foobar" is displayed on the page and I want a search for "foobar" to return that page because it has that string on it.

Sounds trivial, but for some reason doesn't seem to work. I have set the search configuration to include webparts (which is normally switched off), but that didn't do the trick. I checked the page syntax, the search webpart does not include a "no-index" class any more (after disabling that setting).

So what could be blocking search from indexing the content on that page? Does it recognize search result webparts and skips indexing those? I can imagine it would do that, but in this case it shouldn't. Any ideas?

Update: I updated the page to include a search content webpart. Actually I used two search content webparts; one in client side rendering mode and one in server side rendering mode. Both display the same data including the text I want to be able to search for. But still, search is not returning this page when I search for terms displayed by those webparts. When I copy/paste the table as rendered by the server side rendering code and put it on the page itself, search does return the page. So either the content is not rendered properly when it is indexed by search, or search is filtering out that content for some reason.

Update 2: the scenario changed a little bit. We're using the catalog feature to show a managed metadata navigation structure linked to catalog templates. On the category pages, a search content webpart shows text from a list on the catalog site. This text we'd like to index via search so the specific category page pops up when the user searches for a word from this text. Due to the above reasons, this isn't working either.

Answer 1

I'm afraid your hypothesis:

Does it recognize search result webparts and skips indexing those?

is the good guess. Actually, to be precise, the Web part does recognize the crawler, not the other way arround!

Looking at the SharePoint code with ILSpy, we can see that:

1. ContentBySearchWebpart inherits from DisplayScriptWebPart
2. DisplayScriptWebPart has a protected member called IsSharePointCrawler
3. IsSharePointCrawler returns true if SPUtility.IsSearchRequestdoes (will come back to that one later)

So, the first question is: who calls IsSharePointCrawler? Different methods do, including RenderWebPart. And if we look at RenderWebPart, we can see that the Web part is actualy not rendered when IsSharePointCrawler returns true!

protected override void RenderWebPart(HtmlTextWriter output)
{
    using (new SearchUXMonitoredScope(this, "ResultScriptWebPart::RenderWebPart"))
    {
        if (!this.RenderOnServer && !base.IsSharePointCrawler())
        {
            base.RenderWebPart(output);
        }
    }
}

The second question is: how does SPUtility.IsSearchRequest determine the request comes from the crawler? Again, let's look at the code:

   int num = userAgent.IndexOf("MS Search", System.StringComparison.OrdinalIgnoreCase);
   if (num >= 0 && num < userAgent.IndexOf("Robot", System.StringComparison.OrdinalIgnoreCase))
   {
         result = true;
   }

So, each time a request comes in with the User-Agent string containing "MS Search [...] Robot", the Search Results Web part is not rendered!

I don't see a simple solution for you at the moment, but I'm still thinking...

Update
As you're running in Office365, most of our options are not possible (HTTP modules, inheriting and overriding the base Web part, ...). So, any solution would undoubtly be a (very) dirty patch...

One of these dirty workarounds would be to have a client "robot" hosted on your premises. That robot (a C# exe/service let's say) would regularly request the page in question, get the HTML rendered by the search Web part, and extract the search results from it.
Then, by using CSOM calls, you would update a Content Web part with the text retrieved previously (we can set Web part properties with the CSOM, see the WebPart property of WebPartDefinition, and its Properties... The text would be transparent or invisible, or something like that.

I'm sure I don't have to explain the 10+ reasons why this is dirty... :(

One variant: the code I'm talking about could be written in JavaScript, and hosted in the page itself. The first time a visitor comes on the page, if he has proper rights, would trigger the JavaScript code, that would get the HTML already rendered in the Content Search Web part frame, and inject it into a Content Web part... Yup, still very dirty... :(

Answer 2

I think the problem relies on the nature of the search results web part. When you access the page from the browser the search results are returned from the server in JSON format. You can check this by using Fiddler and checking the response of the ASPX page.

For obvious reasons, the crawler tends to ignore the contents of the "script" tags, thus not indexing the search results from the web part. I know that the code of the CSWP (content search web part) will detect crawlers and instead of using client-side rendering it will render an HTML table with the results (more info here). On the other hand, I think there is no server-side rendering mode on the SRWP (couldn't find info at least).

So, I would replace the SRWPs with CSWPs and hardcode the queries so they would return the same results. After that, crawl again the page and test if it is returned from search.

-Hope it helps

Answer 3

Under 'Site settings > Search and Offline Availability' ensure that 'Always index all Web Parts on this site' is selected under 'Indexing ASPX Page content'