Take the 2-minute tour ×
SharePoint Stack Exchange is a question and answer site for SharePoint enthusiasts. It's 100% free, no registration required.

When is SharePoint search treating a document as a duplicate?

I did some testing with 3 documents in a document library, different names ofcourse, but all the same file content. Performed a crawl and used an enterprise search center (keyword query) with a search core results webpart (setting Remove Duplicate Results checked) to perform a search. I expected just one result while searching on one of the words which all files contain, but I received all 3 documents as result.

I just wanted to test when an item is treated as a duplicate, what percentage of content has to be different or something. Appearently I don't understand the process, can somebody explain this to me?

Found this on 2007 http://blogs.technet.com/b/harikumh/archive/2008/11/14/some-interesting-facts-about-sharepoint-2007-search.aspx, is it still true for SharePoint 2010?

share|improve this question
    
Ok, my VM died on me and I tested it on my new VM and the documents are displayed as duplicates. That part is solved then. But I'm still curious what percentage of content has to be different before it isn't a duplicate anymore. –  Anita Boerboom Jun 26 '11 at 16:53
    
I ran into a strange and related issue today. For testing purposes I copied about 130 copies of a document with unique names, but the same content. I then wrote a powershell script that uploads the document to a site collection and applies unique content type and field attributes to it. I loaded two libraries in separate site collections for a total of over 250 documents. After reindexing/resetting my index a couple of times, I finalize realized that search only recognized this as a single document so there is a single result returned. –  Mike Oryszak Sep 10 '11 at 20:01
add comment

3 Answers 3

up vote 5 down vote accepted
+100

I will put my little research as answer (although it is not a real answer).

I followed link provided by @AnitaBoerboom and I believe it applies to 2010 version as well.

Quote:

How does the duplicate document is identified when we do a search?

Document similarity for purposes of identifying duplicates is based only on a hash of the content of the document. No File properties (e.g. file name, type, author, create and modify dates) are input to this hash. The MSSDuplicateHashes table in the SSP’s search database holds, for each document, all the 64bit hashes necessary to determine if one document is a near-duplicate of another. This is read while doing a search if duplicate collapsing is enabled.

This is probably answer to @MikeOryszak strange issue. He uploaded 250 documents with same content - so this is just 1 document and 249 duplicates.

64bit hash is something that puzzles me. This hash is determined when document is crawled by preforming Full crawl.

After reading few documents from SharePoint Back-End Protocols and after exploring my local SharePoint SQL stored procedures I have found this: Duplicate Identifier Block

And it leaded me to 'something that is completely different' (or not):

Finding Duplicate Documents in SharePoint using PowerShell

This script uses MD5 Message-Digest Algorithm to determine if two files are same. And this was my first hunch after reading this question.

So without any real and hard evidence IMHO this is exact procedure that determines if two files are exact duplicates.

So possibility that 2 documents with different content have a same hash is almost none and way lower then possibility that 2 items in single list are determined by same GUID.

Disclaimer: I am really not an expert in this field so don't take my findings as granted. I would like to hear from someone with 'right knowledge'.

share|improve this answer
    
Thanks for your comment. It's not a real answer as you stated, but I appreciate your investigation and providing the links. –  Anita Boerboom Sep 12 '11 at 5:48
add comment

Per the blog post for 2007, it uses a hash of the entire document and ignores the actual document properties. I did some tests today outlined in my comment above which showed all of the documents as duplicates, even when across site collections. Any documents that were changed afterwards showed as unique documents after another crawl.

share|improve this answer
    
you beat me with this by 5 minutes :) –  Vedran Rasol Sep 10 '11 at 22:54
    
I cannot award myself the bounty anyway, so its up for grabs to somebody. ;) –  Mike Oryszak Sep 10 '11 at 23:30
    
I wasn't aware of this, but even if my answer is accepted by question owner it is still in your hands to award bounty. True? FAQ doesn't cover this situation. –  Vedran Rasol Sep 13 '11 at 22:43
    
Yes, your future bounty is in my hands. ;) –  Mike Oryszak Sep 13 '11 at 22:51
    
I would trade 100 future points if only my answer could be verified by some 'higher authority' –  Vedran Rasol Sep 13 '11 at 22:58
show 1 more comment

Sorry it is not the same. I Take a look into my database and can not find the MSSDuplicateHashes table in the SSP’s search database. http://blogs.technet.com/b/jpradeep/archive/2010/09/29/moss-2007-duplicate-search-results.aspx

Check it also on a other farm - still the same

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.