Take the 2-minute tour ×
SharePoint Stack Exchange is a question and answer site for SharePoint enthusiasts. It's 100% free, no registration required.

I am completely frustrated with finding the actual list of SharePoint 2010 word breakers. Everything on Microsoft's site is just that they have them for English, BUT WHAT ARE THEY?

I keep reading contradicting articles about (_)underscore being and not being a word breaker!

Maybe I am completely lost with this, but I could really use that list and even the list for their stemmers and noise words, I don't understand why this isn't available.

share|improve this question
add comment

migrated from stackoverflow.com Jun 26 '12 at 20:19

This question came from our site for professional and enthusiast programmers.

3 Answers

up vote 5 down vote accepted

The search crawler uses an algorithm in the back-end to determine whether or not a character counts as a work breaker (it attempts to use grammar and other language-specific characteristics to determine the word breakers). For English, the closest documentation to a "list" is here: http://technet.microsoft.com/en-us/library/gg982951.aspx. Under tokenization is states "Spaces, tabs, periods, commas, dashes, question marks and quotation marks are considered delimiting characters." Unfortunately there isn't a specific list of word breakers as it attempts to guess them based on the characteristics of each language (and in some cases a character is a word breaker in one context but not in another).

share|improve this answer
    
Thank you! I am not sure if I was missing it, or MS just doesn't explain it in such concise terms on their site, but this kind of information is extrmemely helpful. –  Steve Jun 27 '12 at 12:33
add comment

If you want to test how a query is split into tokens you can modify the following test program:

using System;
using System.Collections;
using Microsoft.SharePoint;
using Microsoft.Office.Server.Search.Query;
using Microsoft.Office.Server.Search.Administration;
using Microsoft.Office.Server.Search.Query.LanguageResources;

namespace WordBreaker
{
    class Program
    {
        static void Main(string[] args)
        {
            using (var site = new SPSite("http://sharepoint"))
            {
                // Get the SearchServiceApplication
                SPServiceContext serviceContext = SPServiceContext.GetContext(site);
                var searchApplicationProxy = (SearchServiceApplicationProxy)serviceContext.GetDefaultProxy(typeof(SearchServiceApplicationProxy));
                Guid applicationId = searchApplicationProxy.GetSearchServiceApplicationInfo().SearchServiceApplicationId;
                var searchApplication = SearchService.Service.SearchApplications.GetValue<SearchServiceApplication>(applicationId);

                var queryInternal = new KeywordQueryInternal(searchApplication);
                var wordBreaker = queryInternal.CreateWordBreaker(false);
                var query = "Test:Test Test_Test2 Test3.Test4";
                wordBreaker.BreakText(query);
                foreach (ArrayList tokens in wordBreaker.GetTokens())
                {
                    Console.WriteLine("Token");
                    foreach (Token token in tokens)
                    {
                        Console.WriteLine("  {0}", token.GetToken());
                    }
                }
            }
        }
    }
}

The noice words are listed in "C:\Program Files\Microsoft Office Servers\14.0\Data\Office Server\Config\noiseenu.txt". See Manage stop word files (SharePoint Server 2010)

share|improve this answer
add comment

one additional suggestion: you can use custom dictionary if you are not completely satisfied with default word breaking rules. They are here: %ProgramFiles%\Microsoft Office Servers\14.0\bin\CustomNNNN.lex Where NNNN = Language Hex Code (1033 = 0009)

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.