Hot answers tagged crawling
8
Yes we do need it. Search indexing is really made up of two items: the actual index file which contains keyword AND the search database that contains metadata. The two are combined when a search is initiated
The search database contains settings related to search (content sources, confguration, all that good stuff). In addition it will also contain the list ...
5
Maybe you are running into a loopback authentication issue. This will happen if content is accessed from a server to the same server using an alternate DNS name. For example:
Server name: srvSP2010wfe
Portal URL pointing to server (1.): http://portal
DNS config: portal A IP.OF.Server.1
The resolution is described here:
...
5
You also need to install the iFilter from Adobe in order to allow SharePoint to index the PDFs. The configuration changes you made simply tells SharePoint to index PDF, but the iFilter is actually what does the work.
Start here: http://support.microsoft.com/kb/2293357
4
Out of the box SharePoint will not be able to crawl the contents of these files. I would check 2 things. First, make sure the file type is in the allowed file types to make sure that the results will be available. Second, if you want to be able to search by contents, you will need to have an iFilter for these file types. Then SharePoint will be able to crack ...
4
XML files are searched and indexed natively in SharePoint 2010 using the out-of-the-box engine (FAST is just a search add-on/alternative but uses the native engine under the covers so it's a supplement and will search XML files using the native engine).
It's a little confusing because you say you're using FAST (generally a product related to SharePoint ...
4
Yes, you can crawl file shares with SharePoint 2007. This is how you can configure crawling, then you should configure full crawl and incremental crawl schedule.
4
SharePoint Search does by default not index unapproved items as well as draft versions (minor versions). This is so because the search crawling account only has read access to the content and only users with contribute access or better can see unapproved documents and draft versions. Consequently, you could get around the issue by granting contribute access ...
3
HA! Nailed it.
I had a look at the logs of an IIS box in our DMZ that was showing the same activity. It turns out in IIS anonymous access was turned on (of course) as well as Windows Integrated Authentication. So what's happening is that the spider is trying to use its credentials (which are no good on this machine as it's not on the domain) instead of ...
3
You can add a robots.txt to your root.
in this robots.txt you can put information about what pages may and may not be indexed. For more information check:
http://www.robotstxt.org/
http://www.robotstxt.org/robotstxt.html
edit:
waldek mastykarz has made a solution that automatically creates a robots.txt for you: ...
3
I doubt you would be able to implement an iFilter on Office 365/SharePoint Online as it requires changes to the Server File system.
As an alternative I would suggest you could use tagging and other metadata fields on the libraries where you store the DWG files. This might be inconvenient if you have many fields, but without an indexed search, and the ...
3
If you change your code to use "yield return" you should give out id's as the crawler asks for it, instead of building up a long list internally which is then returned. I haven't tried this myself, but using yield is the perfect way to fetch data as needed. Could be worth a try.
I suspect you can do something similar when it ask for each item, if you keep a ...
3
Have you considered splitting the content Databases into smaller chunks?
I see a few benefits for doing this :
Crawler performance
Page Response Times
Reduce App Pool Recycle Times
Since you are talking about Mirroring DB's I am going to take the liberty of thinking that you may have the capability of having multiple SQL Servers. Consider moving ...
2
You have virtually zero admin dials available for search in WSS 3.0. Just forget about it and opt for Microsoft Search Server 2008 Express, which is also free. Technically, MSS 2008 is WSS 3.0 + MOSS 2007 Search. Only thing missing is the BDC and people search - but it sounds like you can survive without that. You can upgrade your WSS 3.0 to MSS 2008 or you ...
2
The simple, one-word reason for all of the above is (I believe) scalability.
If you have a SharePoint site with tens of thousands of users, all performing possibly dozens of searches per second, you can vastly improve search performance by farming off the Search Database to another server or place its tablespace onto another drive array.
2
After doing a lot of matrix testing on some troublesome URLs, there appear to be at least three things which generate this same message.
1) There is, in fact, a bot tag in a page which is having its content crawled.
2) Your SharePoint SSA is not configured to crawl the file type reflected in the URL. We are using a custom protocol handler to crawl in ...
2
The ranking algorithm is based on the BM25 ranking model and takes into account various factors such as Contextual Relevance, Proximity, Metadata, Language, File Type, Anchor Text, URL Depth and URL Matching.
You can influence it by going to the Search Administration page and going to Authoratative Pages in the Queries and Results section. This allows you ...
2
Here is how I addressed the problem.
I created two crawl rules, one to exclude all files and one to include the desired file extension. Put include rule above the exclude rule. That way only the desired files get included and everything else matches the exclude rule.
Example:
Include Rule
file://hostname/sharename/*/*.las
Exclude Rule
...
2
I know this post is rather old, but I wanted to provide some insite to this as well.
If your Web Applications that you created in SP 2010 are "Claims Based" Web Apps, you will see this behavior.
Central Admin is not a CLAIMS Based Web App and cannot properly access the Items in the Scope for a CLaims Based Web App.
If you browse into the "Search Scopes" ...
2
This blog post provides an answer:
http://www.moss2007.be/blogs/vandest/archive/2010/03/18/sharepoint-search-scopes-approximate-item-count-is-incorrect.aspx
Apparently the behaviour is that it gets the number of items in a scope via a search query. If for any reason that query fails (permissions, probably) you'll get a result of 0.
I had this problem but ...
2
In addition to Mikes answer, and if you are thinking of rebuilding your SSP to try and resolve the issue, I would do the following list of things first. These have helped me resolve search related issues before and are quicker to test than a creating a new SSP.
Note: My assumption is you find no errors in the logs mentioned previously and your service packs ...
2
Toni, application pages are not crawled by the crawler when SharePoint is crawled. The crawler uses the SiteData.asmx/GetContent to find content to crawl for a web application. Since application pages are not a part of the actual content they are not being discovered.
If you want to crawl application pages you need to set up a standard web crawl content ...
2
Hard to get details on this, but I found an MSDN forum comment by Nick Swan indicating that returning a byte[] array in a BDC finder breaks the crawler. I don't think it is possible to crawl document data this way.
What about writing an ASP.NET application that renders hyperlinks for the document content and then crawl as an external website? I am not sure ...
2
Read this : http://blogs.msdn.com/b/dinaayoub/archive/2010/04/22/sharepoint-2010-how-to-change-the-list-view-threshold.aspx
Also : http://technet.microsoft.com/en-us/library/cc262813.aspx
Hope it helps
2
I could find no OOB way to do this in stsadm or PowerShell, so you could either write a server specific custom timer job that does this at pre-scheduled interval, or create a PowerShell cmdlet that is called using schedule on the server that runs the Search Service Application.
using Microsoft.Office.Server.Administration;
using ...
2
Verify that the account has permission to see these draft versions. The libraries may be set up so that only approvers and the original author can see the unapproved content. That means the crawl account needs to be in the Approvers group
Keep in mind that this will result in draft versions appearing in search results for everyone to see. Then, some users ...
2
The Technet article Estimate performance and capacity requirements for SharePoint Server 2010 Search provides the following formula to estimate the size of index partitions:
ContentDBSum = Size of content that is crawled.
TotalIndezSize = ContentDBSum * 0.35
QueryComponentIndexSize = TotalIndexSize / Number of index partitions
The size of the index ...
2
Looks like your Index location is Corrupt.
Do you have a test environment where a similar behavior is occuring? If so I would recommend a backup of your SP server then do a hard delete of the Index files (Gather n stuff). Then do an Index reset and start a full crawl once again.
Any other SharePoint Logs being written during this error?
2
If your data source is within a site collection, use a Content Query Web Part and filter on the Task content type, or otherwise use SPSiteDataQuery.
If it's split across site collections, you can only use search, and the indexing can only occur on a content source, as Anita mentioned.
2
Not sure, it makes any difference but a detailed thread from someone who faced similar issue. The person seemed to have contacted MS Support as well, so you'll see a lot many updates. Here's the article.
Only top voted, non community-wiki answers of a minimum length are eligible
