7

I have a page which features a search results webpart to show the results of a preconfigured search query. The display template of the webpart has been modified to include a number of custom metadata fields. Those are displayed on the page.

Now I want the user to be able to find that specific page by searching the text as rendered by the display template. So when the custom metadata field contains "foobar", "foobar" is displayed on the page and I want a search for "foobar" to return that page because it has that string on it.

Sounds trivial, but for some reason doesn't seem to work. I have set the search configuration to include webparts (which is normally switched off), but that didn't do the trick. I checked the page syntax, the search webpart does not include a "no-index" class any more (after disabling that setting).

So what could be blocking search from indexing the content on that page? Does it recognize search result webparts and skips indexing those? I can imagine it would do that, but in this case it shouldn't. Any ideas?

Update: I updated the page to include a search content webpart. Actually I used two search content webparts; one in client side rendering mode and one in server side rendering mode. Both display the same data including the text I want to be able to search for. But still, search is not returning this page when I search for terms displayed by those webparts. When I copy/paste the table as rendered by the server side rendering code and put it on the page itself, search does return the page. So either the content is not rendered properly when it is indexed by search, or search is filtering out that content for some reason.

Update 2: the scenario changed a little bit. We're using the catalog feature to show a managed metadata navigation structure linked to catalog templates. On the category pages, a search content webpart shows text from a list on the catalog site. This text we'd like to index via search so the specific category page pops up when the user searches for a word from this text. Due to the above reasons, this isn't working either.

  • The user who performs the search has rights to display that page? – Stefano Paterno Jan 5 '15 at 13:43
  • 1
    Display templates display their results/HTML by using JavaScript. Not sure but maybe the display template code does not work correctly when visited by the crawler. In other words for the crawler there is nothing there. – Wout Jan 5 '15 at 14:10
  • If you simply add some text on the page with a content editor web part. Do you find the page when you search on the text entered in the content editor web part? – Wout Jan 5 '15 at 14:11
  • 1
    Silly question, but have you re-indexed the site after applying the changes to the search settings? See here – Rodrigo Molinas Jan 5 '15 at 16:52
  • 1
    My first guess would be that the page is unpublished and that you have draft item security in place preventing search from indexing your page. – Dave Jan 5 '15 at 17:57
6
+250

I'm afraid your hypothesis:

Does it recognize search result webparts and skips indexing those?

is the good guess. Actually, to be precise, the Web part does recognize the crawler, not the other way arround!

Looking at the SharePoint code with ILSpy, we can see that:

  1. ContentBySearchWebpart inherits from DisplayScriptWebPart
  2. DisplayScriptWebPart has a protected member called IsSharePointCrawler
  3. IsSharePointCrawler returns true if SPUtility.IsSearchRequestdoes (will come back to that one later)

So, the first question is: who calls IsSharePointCrawler? Different methods do, including RenderWebPart. And if we look at RenderWebPart, we can see that the Web part is actualy not rendered when IsSharePointCrawler returns true!

protected override void RenderWebPart(HtmlTextWriter output)
{
    using (new SearchUXMonitoredScope(this, "ResultScriptWebPart::RenderWebPart"))
    {
        if (!this.RenderOnServer && !base.IsSharePointCrawler())
        {
            base.RenderWebPart(output);
        }
    }
}

The second question is: how does SPUtility.IsSearchRequest determine the request comes from the crawler? Again, let's look at the code:

   int num = userAgent.IndexOf("MS Search", System.StringComparison.OrdinalIgnoreCase);
   if (num >= 0 && num < userAgent.IndexOf("Robot", System.StringComparison.OrdinalIgnoreCase))
   {
         result = true;
   }

So, each time a request comes in with the User-Agent string containing "MS Search [...] Robot", the Search Results Web part is not rendered!

I don't see a simple solution for you at the moment, but I'm still thinking...

Update
As you're running in Office365, most of our options are not possible (HTTP modules, inheriting and overriding the base Web part, ...). So, any solution would undoubtly be a (very) dirty patch...

One of these dirty workarounds would be to have a client "robot" hosted on your premises. That robot (a C# exe/service let's say) would regularly request the page in question, get the HTML rendered by the search Web part, and extract the search results from it.
Then, by using CSOM calls, you would update a Content Web part with the text retrieved previously (we can set Web part properties with the CSOM, see the WebPart property of WebPartDefinition, and its Properties... The text would be transparent or invisible, or something like that.

I'm sure I don't have to explain the 10+ reasons why this is dirty... :(

One variant: the code I'm talking about could be written in JavaScript, and hosted in the page itself. The first time a visitor comes on the page, if he has proper rights, would trigger the JavaScript code, that would get the HTML already rendered in the Content Search Web part frame, and inject it into a Content Web part... Yup, still very dirty... :(

  • I'm afraid you're right. I've granted you the bounty as this is the correct answer, but don't stop thinking about a solution please ;) I have a workaround to fulfil the client needs, but to get this working would be much easier. Too bad they didn't include an override property or something on the web part :( – Jasper Apr 14 '15 at 5:50
  • This design they had is most propably a security concern: the search crawler has access to all content. So, if it crawls the page with search results in it, this page would contain non-security trimmed results (actually results trimmed for the search account, i.e. all of them) that would fill the index. Then, a user would find in the search results of a subsequent search summaries of that page, potentially containing texts (that can't be security trimmed since they're now plain text) the user should not be able to access. – Evariste Apr 14 '15 at 7:07
  • That seems like a logical explanation. Still, a "I know know what I'm doing, index this content" property on the webpart would have been nice... – Jasper Apr 14 '15 at 7:11
3

I think the problem relies on the nature of the search results web part. When you access the page from the browser the search results are returned from the server in JSON format. You can check this by using Fiddler and checking the response of the ASPX page.

For obvious reasons, the crawler tends to ignore the contents of the "script" tags, thus not indexing the search results from the web part. I know that the code of the CSWP (content search web part) will detect crawlers and instead of using client-side rendering it will render an HTML table with the results (more info here). On the other hand, I think there is no server-side rendering mode on the SRWP (couldn't find info at least).

So, I would replace the SRWPs with CSWPs and hardcode the queries so they would return the same results. After that, crawl again the page and test if it is returned from search.

-Hope it helps

  • Completely true, but as the update already stated I have two content search webparts on the page. So also with CSWP on there, correctly configured with server side xslt rendering: it still doesn't work. – Jasper Jan 6 '15 at 22:02
2

Under 'Site settings > Search and Offline Availability' ensure that 'Always index all Web Parts on this site' is selected under 'Indexing ASPX Page content'

  • Jup, that flag is set correcly. There is no "noindex" class in the HTML source. – Jasper Jan 6 '15 at 10:21
  • This works for Content Editors but not for search-driven web parts. – Derek Gusoff Nov 23 '15 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.