List of SharePoint 2010 English Word Breakers?

Question

I am completely frustrated with finding the actual list of SharePoint 2010 word breakers. Everything on Microsoft's site is just that they have them for English, BUT WHAT ARE THEY?

I keep reading contradicting articles about (_)underscore being and not being a word breaker!

Maybe I am completely lost with this, but I could really use that list and even the list for their stemmers and noise words, I don't understand why this isn't available.

Answer 1

The search crawler uses an algorithm in the back-end to determine whether or not a character counts as a work breaker (it attempts to use grammar and other language-specific characteristics to determine the word breakers). For English, the closest documentation to a "list" is here: http://technet.microsoft.com/en-us/library/gg982951.aspx. Under tokenization is states "Spaces, tabs, periods, commas, dashes, question marks and quotation marks are considered delimiting characters." Unfortunately there isn't a specific list of word breakers as it attempts to guess them based on the characteristics of each language (and in some cases a character is a word breaker in one context but not in another).

Answer 2

If you want to test how a query is split into tokens you can modify the following test program:

using System;
using System.Collections;
using Microsoft.SharePoint;
using Microsoft.Office.Server.Search.Query;
using Microsoft.Office.Server.Search.Administration;
using Microsoft.Office.Server.Search.Query.LanguageResources;

namespace WordBreaker
{
    class Program
    {
        static void Main(string[] args)
        {
            using (var site = new SPSite("http://sharepoint"))
            {
                // Get the SearchServiceApplication
                SPServiceContext serviceContext = SPServiceContext.GetContext(site);
                var searchApplicationProxy = (SearchServiceApplicationProxy)serviceContext.GetDefaultProxy(typeof(SearchServiceApplicationProxy));
                Guid applicationId = searchApplicationProxy.GetSearchServiceApplicationInfo().SearchServiceApplicationId;
                var searchApplication = SearchService.Service.SearchApplications.GetValue<SearchServiceApplication>(applicationId);

                var queryInternal = new KeywordQueryInternal(searchApplication);
                var wordBreaker = queryInternal.CreateWordBreaker(false);
                var query = "Test:Test Test_Test2 Test3.Test4";
                wordBreaker.BreakText(query);
                foreach (ArrayList tokens in wordBreaker.GetTokens())
                {
                    Console.WriteLine("Token");
                    foreach (Token token in tokens)
                    {
                        Console.WriteLine("  {0}", token.GetToken());
                    }
                }
            }
        }
    }
}

The noice words are listed in "C:\Program Files\Microsoft Office Servers\14.0\Data\Office Server\Config\noiseenu.txt". See Manage stop word files (SharePoint Server 2010)

Answer 3

one additional suggestion: you can use custom dictionary if you are not completely satisfied with default word breaking rules. They are here: %ProgramFiles%\Microsoft Office Servers\14.0\bin\CustomNNNN.lex Where NNNN = Language Hex Code (1033 = 0009)