2

I know a crawl is used to update the index in order to do a search on SharePoint quickly. But what I do not understand why one needs a crawl in the first place!

Whenever a page is updated, added or changed, why isn't the index updated in that very instance? This would mean the index is up-to-date immediately, and you don't have to run a 'crawl' ever. Wouldn't that be much easier?

Maybe I am missing the big picture here, so any insights would be great.

1

You can enable that feature in from SharePoint Server 2013 and forward, and you get the functionality you ask for. However, this feature "continuous crawl" is resource intensive and most organisations chose not to use it. You may need to double the memory from 16GB to 32GB on all your application servers.

See Manage continuous crawls in SharePoint Server 2013 for more on the topic.

But you can't avoid crawling and index itself. They need to be there to get Search to work.

4
  • I understand indexing is essential to make sure you can do a search in the first place. But why a crawl? A crawl looks to me like a waste of time and resources...
    – Alex
    Aug 18 '16 at 11:40
  • @Alex What goes into the index, and what permission does every item have? Should it be in the index or not? And what about metadata, do we need everything or just some? That's the crawl components job, which you can configure. You need it, SharePoint need it and Google need to crawl to make it possible to search. There may be other ways, but then we're talking Computer Science research and not business applications. Interesting question though :)
    – Benny Skogberg
    Aug 18 '16 at 11:45
  • @Alex But I'm generalizing too much. Actually you have six components in SSA: 1) Analytics processing component 2) Content processing component 3) Crawl component 4) Search administration component 5) Query processing component and the 6) Index component. Keep reading: technet.microsoft.com/en-us/library/jj862354.aspx and technet.microsoft.com/en-us/library/jj862355.aspx
    – Benny Skogberg
    Aug 18 '16 at 11:52
  • Maybe you could provide a more general insight in searching, indexing and the like, sharepoint independent? I cannot understand why this has been so very complicated? Is the creation of a search index so complicated in general? Or just because of an 'organical growth' of how sharepoint handles things? Or, asked differently, could sharepoint have been created to function much easier and faster?
    – Alex
    Aug 18 '16 at 11:59
1

Index can be auto updated if we configure the continuous crawling option as others mentioned.

But it can hit the performance of the SharePoint, Because from crawling to indexing is not a simple task. Below are the steps which search performed to index a single file.

  • New document added
  • Crawl Check the File Type and Path.
  • Now check the Crawl rule, if file type is exclude from crawling or this path is exclude from crawling.
  • After passing this step,Now it check the piece of software / Ifilter to read the content of file and all associated properties
  • During this process, it will skip the words which dont want to index.
  • Crawler store the information about the item in the Crawl Database i.e last crawl time, the last crawl ID, and the type of update during the last crawl .
  • Now hand over information to Content Processing components.
  • Content Processing the Process these items and add into the Index. ( Content Processing parsing and extracting document properties and various other tasks such as linguistic processing, property mapping etc).
  • this is how one file added into the Index.

I think this will explain how its work and why it is resource intensive.

Apart from that just think, while crawling something happen to the document and crawler should go back and start from 0.

So keeping all these kind of situation in mind they design this in way which cause less impact.

0

SharePoint crawl is the tool that creates the index to be searched against. An index is simply "a set of items each of which specifies one of the records of a file and contains information about its address" (Google definition).

The content is all there as soon as you click "Save", but the crawl is the thing that looks at new (or modified) content, parses the metadata and keywords, and creates connections with other existing data.

The reason to do interval crawls instead of a continuous crawl is for efficiency and server-load reasons. As you mentioned in our conversation, the following scenario would not be an efficient use of resources:

A user makes a change to a page, then again and then again. A crawl at, say, every 30 minutes would find maybe the final form (and index the page once) while with my suggestion the page would have indexed after each iteration (so n times).

To add to that, if it were to occur in the middle of the day in a large company, the servers may already be under load from other user interaction, and to crawl each and every modification (possibly tens of thousands per minute in a large enough company) would place significant additional load on the servers.

HowStuffWorks has a great breakdown of how/why search engines work, including the purposes and functions of crawlers. The same principles would apply to SharePoint as well.

http://computer.howstuffworks.com/internet/basics/search-engine1.htm

8
  • Thanks for this link, but it explains web searches. Here you need those spiders to search through content, because you do not know when something is changing on a web-page in the internet. But sharepoint is one site! Its a collection of servers in one place practically, and I see that you can use the same approach as working with 'spiders' to create a search index. But why make it so complicated? Why not indexing a sharepoint-site the moment it is created or modified? As a sharepoint site is created and/or modified, sharepoint know - it must store this information somewhere in the database.
    – Alex
    Aug 18 '16 at 13:17
  • Why not indexing it right away? Avoiding a later process, it appears right away in the search etc. Why this cumbersome approach?
    – Alex
    Aug 18 '16 at 13:18
  • Benny mentioned above that a continuous crawl would perform like you're suggesting, with an instantaneous indexing of new content. However, it's resource intensive for the server to be constantly searching for new content to add to the index, so the alternative is the more common approach, to crawl at regular intervals. Basically, something has to add the content to the index...that's the crawl's job. Whether you run it at an interval or continuously (providing instant search) is an administrative choice. Aug 18 '16 at 13:42
  • Why is it resource intensive? I do not understand. If you do not change any content, no indexing is done. It is only then done, if you CNANGE something on the webpage. Sharepoint must notice when someone adds a new document, or clicks the save button. Only then this particular piece of data/content is being indexed. Maybe there is still a misunderstanding? What I want to say: If a user 'clicks the save button', then and only then the new content is being indexed. No need to index all the time and use resources... I hope it is clearer now what I mean...
    – Alex
    Aug 18 '16 at 13:54
  • In your words I guess that something is the "click on the save-button"...
    – Alex
    Aug 18 '16 at 13:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.