Good afternoon everyone.
I've been working on the implementation of a SharePoint 2010 farm for over a week know, which is far longer than I should have been working on it. I'm running into a problem provisioning the Managed Metadata Service application that has become more than just a headache. Here is some background:
Web Applications:
- Central Administration, application pool account is domain\_SPFarm, host is APP01
- Corporate Portal, application pool is domain\_SPPortalAppPool, host is WEB01
- Content Type Hub, application pool is domain\_SPMetadataAppPool, host is WEB01
Each Web application has been setup in Classic Mode using Windows Authentication with Kerberos as the provider. Service principal names using the FQDN and NetBIOS name have been configured for each application pool account. I have validated that there are no duplicate SPNs in the forest. I have configured Windows Authentication to use the application pool credentials and Kernal Mode is enabled (in the webServer section of the applicationHost.config). Multiple IP addresses have been assigned to the host named WEB01. Each web application has its address bound to a unique IP address; I'm not using host headers. DNS entries have been validated.
Using the domain's Certificate Authority I have issued a SAN certificate for Central Admin that can use both localhost as well as APP01. Each of the other Web applications has also been issued a standard SSL certificate. Bindings have been added for each website in IIS using port 443 for HTTPs and port 80 HTTP communications.
The Active Directory domain is running in a Server 2003 Operations mode. There are no 2008 R2 domain controllers in the forest anywhere. Each server in the farm does have a 2008 R2 operating system. They have been fully updated with all applicable updates that are available. SharePoint has been upgrade to and including the December 2011 Cumulative Update. I have updated the application using PSConfigUI, and the databases using the command line PSConfig program. I've validated that each server in the farm is current using Central Administration. Project Server 2010 is also installed and patched, but the Service Application has not been created.
Exclusions have been added to the loopback check as well. A domain group named SharePoint Application Pool Accounts, which contains each account used to host an application pool, has been added to the Distributed COM Operators group and the Cryptographic Operators group. It has also been granted the following user rights via Group Policy Assignment on APP01 and WEB01:
- Adjust Memory Quotas
- Allow Logon Locally
- Impersonate a Client After Authentication
- Log On As a Batch Job
- Log On As a Service
- Replace a Process Level Token
The Managed Metadata service is running on the host WEB01. I have tried running the service on both APP01 and WEB01, just APP01, and just WEB01. In each case the behavior is identical. Every time I try to create the Managed Metadata Service Application SharePoint tells me that the it is created successfully; however, when I try to manage the Service Application I receive that all too popular error:
The Managed Metadata Service or Connection is currently not available. The Application Pool or Managed Metadata Web Service may not have been started. Please Contact your Administrator.
If I try to view the proxy connection settings the dialog window will open, but I receive a 403 response.
During provisioning of the Service Application I specify "spcontenttype" prepended with HTTP (I can't put the full address in because it's converted to a hyperlink) as the content type hub, which is the DNS name assigned to the IP address under which the Content Type Hub Web application is running. I have tried this using "portal" as well with the same results (the Corporate Portal Web application). I use the managed account _SPMetadataAppPool as the application pool account.
The Content Type Syndication feature is added and started in the site collection, but the hidden content type list named "TaxonomyHiddenList" is not created. If I browse to http://spcontenttype/Lists/TaxonomyHiddenList I'm hit with a 404 response.
When I look at the ULS logs I can see that the user name is not being submitted during provisioning. The logs show __Current User= in the trace, there no is username at all, it's blank. I have tried provisioning the Service Application with Anonomyous Authentication disabled so only Windows Authentication running and with both Authentication methods enabled.
There are no audit failures in the Security Event Log, and no Kerberos errors reported in the System Event log. The only Event Log entries are in the Application Event Log which indicate that the Managed Metadata Service is not availalbe.
Because the application pool account _SPFarm under whose context Central Administration is running needs to talk with the application pool account _SPMetadataAppPool on a different server in a different Web applicaiton I configured constrained delegation such that both accounts can delegate to their self and all other service accounts. I was not 100% sure if constrained delegation was required in this scenario, but it seemed to fit the description. Regardless, the exact same behavior is observed.
The other strange piece of information in the ULS logs is that even though the application pools are running in Classic Mode, calls are being made to a WCF endpoint using Claims. The following lines are recorded in logs just prior to the WCF request to initialize the Service Application:
Entering monitored scope (SPClaimsAuthenticationManager.Authenticate) Leaving Monitored Scope (SPClaimsAuthenticationManager.Authenticate). Execution Time=0.117752395905066
I have searched all over for an answer to this problem without success. I've tried everything suggested in various forum posts on the Web. I've even completely uninstalled SharePoint on each server in the farm and rebuilt it only to arrive at the same problem.
Can anyone think of anything that I may have missed here? Any help is very much appreciated; thanks in advance.
UPDATE: I found out that the environment in which this farm is being constructed uses a Cisco ASA device to facilitate VPN access. I started looking into this after looking at my local System Event Log and saw that there were no Kerberos errors at all. Under normal operations I would, at the very least, expect to see errors about a "bad option" or "pre-auth required." I did not see anything at all, which made me think that Kerberos was not being used; Negotiate was reverting to NTLM.
After some research, I found this link:
Apparently there can be problems using Kerberos over UDP in a VPN tunnel that is established using a Cisco ASA device. I do not have access to Cisco's site to obtain further information, but this is start. What if during the initial farm installation and configuration, during which time I was accessing Central Administration from a Web browser on my local system, there was a double hop scenario where Kerberos could not be used? SharePoint may not have even reported any potential errors outside of the ULS logs, and without suspecting it, I would have never known.
It's a long shot, but since I'm at somewhat of a loss, as it would appear everyone else seems to be, it's worth taking the chance. I'm going to try uninstalling everything again, and then rebuilding the farm. All configurations will now take place from the host WEB01. This should eliminate the VPN/Kerberos variable.
If anyone has further insight into using Kerberos through a VPN tunnel I would be very interested in hearing more. Just an FYI, I've already configured my local client to use TCP instead of UDP and that made no difference. The link provided above does provide information about the firewall failing to switch to TCP with Kerberos however, so I think this is still a viable cause.
Thanks everyone.
UPDATE: After rebuilding this farm yet again, only this time from the server named WEB01 instead of using a Web browser on my local client, I found myself in the same exact place. However, I noticed something in the System Event Log that should not have been there. A Kerberos error was reported on the machine that I was using to configure the farm (WEB01). It indicated that the SPN LDAP// was missing.
I had previously added that SPN at least two times. The domain controller that the SPN was referencing was NOT the server that I had been using to configure service accounts and other directory objects. After logging into the server to obtain more information I found several very interesting - and by interesting I mean completely screwed up - facts.
This was a Server 2003 Domain Controller that had only been patched to Service Pack 1. There were thousands of WINS errors indicating that the Domain Controller was not able to synchronize with other Domain Controllers in the domain. In addition, there were thousands of Kerberos errors indicating that the ticket had bad options related to the SPN assigned to HOST.
Microsoft released at hot-fix for the WINS issues that was eventually rolled into Service Pack 2 for Server 2003. I attempted to install the service pack but could not due to a lack of drive space. Apparently when this server was provisioned the system partition on the drive was only allocated with 6 GB of disk space; there is less than 1 GB left.
Between the ULS logs and Network Monitor traces I can see that the authentication request occurring while the Managed Metadata Service Application is provisioned is being sent to this server. I validated that the Global Catalog Server is hosted on that Domain Controller (as well as every other Domain Controller in this particular environment).
I'm confident at this point that the problem is with this domain controller. I've suggested that we decommission the box all together and stand up a new 2008 R2 Domain Controller, or just an additional 2003 R2 Domain Controller while maintaining the current operations level. I do not thing there is a way to free up the addition 400 MB of disk space required to install Service Pack 2, but that would probably fix the problem as well.
An additional update will be provided once this has been done with the hope that maybe this enormous post can save someone the headache I've been dealing with in the future.
Thanks everyone.
