When is SharePoint search treating a document as a duplicate?

I did some testing with 3 documents in a document library, different names ofcourse, but all the same file content. Performed a crawl and used an enterprise search center (keyword query) with a search core results webpart (setting Remove Duplicate Results checked) to perform a search. I expected just one result while searching on one of the words which all files contain, but I received all 3 documents as result.

I just wanted to test when an item is treated as a duplicate, what percentage of content has to be different or something. Appearently I don't understand the process, can somebody explain this to me?

Found this on 2007 http://blogs.technet.com/b/harikumh/archive/2008/11/14/some-interesting-facts-about-sharepoint-2007-search.aspx, is it still true for SharePoint 2010?

  • Ok, my VM died on me and I tested it on my new VM and the documents are displayed as duplicates. That part is solved then. But I'm still curious what percentage of content has to be different before it isn't a duplicate anymore. Jun 26, 2011 at 16:53
  • I ran into a strange and related issue today. For testing purposes I copied about 130 copies of a document with unique names, but the same content. I then wrote a powershell script that uploads the document to a site collection and applies unique content type and field attributes to it. I loaded two libraries in separate site collections for a total of over 250 documents. After reindexing/resetting my index a couple of times, I finalize realized that search only recognized this as a single document so there is a single result returned. Sep 10, 2011 at 20:01
  • In CSWP, under Refine tab, you can find "grouping" option. What is does isn't group items as view, it's eliminate duplicated documents (so you will only see one.) It works pretty well for me.
    – YogaPanda
    Mar 29, 2016 at 18:24

4 Answers 4


I will put my little research as answer (although it is not a real answer).

I followed link provided by @AnitaBoerboom and I believe it applies to 2010 version as well.


How does the duplicate document is identified when we do a search?

Document similarity for purposes of identifying duplicates is based only on a hash of the content of the document. No File properties (e.g. file name, type, author, create and modify dates) are input to this hash. The MSSDuplicateHashes table in the SSP’s search database holds, for each document, all the 64bit hashes necessary to determine if one document is a near-duplicate of another. This is read while doing a search if duplicate collapsing is enabled.

This is probably answer to @MikeOryszak strange issue. He uploaded 250 documents with same content - so this is just 1 document and 249 duplicates.

64bit hash is something that puzzles me. This hash is determined when document is crawled by preforming Full crawl.

After reading few documents from SharePoint Back-End Protocols and after exploring my local SharePoint SQL stored procedures I have found this: Duplicate Identifier Block

And it leaded me to 'something that is completely different' (or not):

Finding Duplicate Documents in SharePoint using PowerShell

This script uses MD5 Message-Digest Algorithm to determine if two files are same. And this was my first hunch after reading this question.

So without any real and hard evidence IMHO this is exact procedure that determines if two files are exact duplicates.

So possibility that 2 documents with different content have a same hash is almost none and way lower then possibility that 2 items in single list are determined by same GUID.

Disclaimer: I am really not an expert in this field so don't take my findings as granted. I would like to hear from someone with 'right knowledge'.

  • Thanks for your comment. It's not a real answer as you stated, but I appreciate your investigation and providing the links. Sep 12, 2011 at 5:48

Per the blog post for 2007, it uses a hash of the entire document and ignores the actual document properties. I did some tests today outlined in my comment above which showed all of the documents as duplicates, even when across site collections. Any documents that were changed afterwards showed as unique documents after another crawl.

  • you beat me with this by 5 minutes :) Sep 10, 2011 at 22:54
  • I cannot award myself the bounty anyway, so its up for grabs to somebody. ;) Sep 10, 2011 at 23:30
  • I wasn't aware of this, but even if my answer is accepted by question owner it is still in your hands to award bounty. True? FAQ doesn't cover this situation. Sep 13, 2011 at 22:43
  • Yes, your future bounty is in my hands. ;) Sep 13, 2011 at 22:51
  • I would trade 100 future points if only my answer could be verified by some 'higher authority' Sep 13, 2011 at 22:58

Sorry it is not the same. I Take a look into my database and can not find the MSSDuplicateHashes table in the SSP’s search database. http://blogs.technet.com/b/jpradeep/archive/2010/09/29/moss-2007-duplicate-search-results.aspx

Check it also on a other farm - still the same


I ran into this while trying to figure out why a specific page was NOT being returned by Search 2010. In my instance, the pages both had unique URLs but had some matching content in the title tag. Specifically, it was "matching" "Village Country Crafts" and "Country Crafts" as duplicates when searching for "Country Crafts", and dropping the second (shorter) result. I turned off the boolean and both results are now appearing. The page content itself was similar (same header, navigation, footer, similar listing info) -- but it was unique. This setting has definitely screwed up my results multiple times and seems to be a horrible matching algorithm. My site does not have duplicate pages, so running with this setting turned off will not affect anything (other than preventing obviously different pages from being removed).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.