SharePoint 2016 - ADFS - persistent cookie - office client integration - authentication prompt

Question

After migration from sp2013 to new sp2016 server farm we have problems with office client integration.

on sp2013 farm, if there was no persistant cookie written from IE, the client application redirected to the adfs sign in page and did an auth for the office client application.

on the new sp2016 farm, a windows authentication prompt pops up.

ULS Log from sp2013 server gives me the correct behavior:

OPTION call to the Document with an 403 FORBIDDEN => goes to 302 adfs login redirect

ULS Log from the sp2016 gives me a strange behavior:

OPTION call to the document with 403 FORBIDDEN => After that: END OF CALL

strange: fiddler doesnt show me the 403... he shows me 401 access denied...

stranger: the windows authentication prompt shows the url of the portal and not of the adfs sign in page. The authentication prompt doesnt acceppt any credentials, which I pass into it => makes sense, because there is no NTLM or Basic auth configured on this zone.

The settings are the same on both farms (sp2013/sp2016)... https, LDAPCP, same ADFS, same office online server

I want to force sharepoint open the adfs sign in page for login :/ not windows auth prompt

ULS from non working farm:

01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x23FC  SharePoint Foundation   Logging Correlation Data    xmnv    Medium  Name=Request (OPTIONS:https://[url_of_my_portal])   be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x23FC  SharePoint Foundation   Asp Runtime avwhz   Medium  SPRequestModule.BeginRequestHandler End be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x23FC  Web Content Management  Publishing  aytib   Medium  ObjectCachePerRequest Global:True, Enabled:False    be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x23FC  SharePoint Foundation   Application Authentication  bjvyg   Medium  SPApplicationAuthenticationModule: Clear outgoing token context from SpThreadContext    be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x23FC  SharePoint Foundation   Asp Runtime avwh6   Medium  SPRequestModule.PostAuthenticateRequestHandler Begin    be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x23FC  SharePoint Foundation   Authentication Authorization    agb9s   Medium  Non-OAuth request. IsAuthenticated=False, UserIdentityName=, ClaimsCount=0  be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x23FC  SharePoint Foundation   Runtime ajd6k   Medium  Value for isAnonymousAllowed is : True  be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x23FC  SharePoint Foundation   Runtime ajd6l   Medium  Value for checkAuthenticationCookie is : False  be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x23FC  SharePoint Foundation   Asp Runtime avwh7   Medium  SPRequestModule.PostAuthenticateRequestHandler End  be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x23FC  SharePoint Foundation   Asp Runtime avwh8   Medium  SPRequestModule.PostAuthorizeRequestHandler Begin   be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x23FC  SharePoint Foundation   Asp Runtime avwh0   Medium  SPRequestModule.PostResolveRequestCacheHandler Begin    be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x23FC  SharePoint Foundation   Asp Runtime avwh1   Medium  SPRequestModule.PostResolveRequestCacheHandler End  be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x23FC  SharePoint Foundation   Asp Runtime aj1kn   Medium  SPRequestModule.AcquireRequestStateHandler  be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x23FC  SharePoint Foundation   Asp Runtime avwh2   Medium  SPRequestModule.PostAcquireRequestStateHandler Begin    be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x23FC  SharePoint Foundation   Asp Runtime avwh3   Medium  SPRequestModule.PostAcquireRequestStateHandler End  be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x23FC  SharePoint Foundation   Asp Runtime avwhu   Medium  SPRequestModule.PreRequestExecuteAppHandler Begin   be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x23FC  SharePoint Foundation   Claims Authentication   airze   Verbose Current identity context: '{"anonymous":"true","nameid":"@","userId":"@"}'  be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x23FC  SharePoint Foundation   Claims Authentication   anvuv   Medium  Context has no SMTP/UPN claims. IdentityContext: '{"anonymous":"true","nameid":"@","userId":"@"}'   be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x23FC  SharePoint Foundation   Asp Runtime avwhv   Medium  SPRequestModule.PreRequestExecuteAppHandler End be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x0EE8  SharePoint Foundation   General af71    Medium  HTTP Request method: OPTIONS    be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x0EE8  SharePoint Foundation   General af75    Medium  Overridden HTTP request method: OPTIONS be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x0EE8  SharePoint Foundation   General af74    Medium  HTTP request URL: https://[url_of_my_portal]    be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x0EE8  SharePoint Foundation   General a08yd   Medium  Getting content database b0a853e3-6682-4ddb-a267-988b92539eaf from webApp 47db0ff9-ccf4-4518-9831-3b91e1f8bf8f version 2928535. be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x0EE8  SharePoint Foundation   Runtime an85c   Medium  recordStatus called with status(0x001E0002): Access Denied. be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x13CC  SharePoint Foundation   Asp Runtime avwia   Medium  SPRequestModule.PostLogRequestHandler Begin be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x13CC  SharePoint Foundation   Asp Runtime avwib   Medium  SPRequestModule.PostLogRequestHandler End   be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x13CC  SharePoint Foundation   Asp Runtime avwic   Medium  SPRequestModule.EndRequestHandler Begin be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x13CC  SharePoint Foundation   Runtime ftc9    Medium  Access Denied: PreSendRequestHeaders, insufficient permission.  be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x13CC  SharePoint Foundation   Site Cache  az4z8   Medium  Looking up SPSite by ID 64da4339-0bc1-4970-b4bf-3510105f867a in memory. be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x13CC  SharePoint Foundation   Flighting Infrastructure    awjmz   Medium  Using legacy MSO-FBA header.    be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x13CC  SharePoint Foundation   Site Cache  az4z8   Medium  Looking up SPSite by ID 64da4339-0bc1-4970-b4bf-3510105f867a in memory. be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x13CC  SharePoint Foundation   General b6p2    Medium  Sending HTTP response 403 - text/plain:403 FORBIDDEN.   be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.03  w3wp.exe (0x0E8C)   0x13CC  SharePoint Foundation   Runtime aigd0   Medium  An Exception occurred when setting the headers in PreSendRequestHeaders. This is error is ignorable in most cases. Error: System.Threading.ThreadAbortException: Thread was being aborted.     at System.Threading.Thread.AbortInternal()     at System.Threading.Thread.Abort(Object stateInfo)     at System.Web.HttpResponse.AbortCurrentThread()     at Microsoft.SharePoint.Utilities.SPUtilityInternal.SendResponse(HttpContext context, Int32 code, String strBody, String strContentType, Dictionary`2 headers)     at Microsoft.SharePoint.Utilities.SPUtilityInternal.Send403(HttpContext context)     at Microsoft.SharePoint.Utilities.SPUtility.HandleAccessDenied(HttpContext context)     at Microsoft.SharePoint.ApplicationRuntime.SPRequestModule.PreSendRequestHeaders(Object oSender, EventArgs ea) be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.04  w3wp.exe (0x0E8C)   0x13CC  SharePoint Foundation   Monitoring  b4ly    Medium  Leaving Monitored Scope: (Request (OPTIONS:https://[url_of_my_portal])) Execution Time=9.0951; CPU Milliseconds=4; SQL Query Count=0; Parent=None   be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.04  w3wp.exe (0x0E8C)   0x13CC  SharePoint Foundation   Asp Runtime avwid   Medium  SPRequestModule.EndRequestHandler End   be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.04  w3wp.exe (0x0E8C)   0x13CC  SharePoint Foundation   Application Authentication  arftr   Medium  SPApplicationAuthenticationModule.IsBearerChallengeRequested: Return 'True'.    be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.04  w3wp.exe (0x0E8C)   0x13CC  SharePoint Foundation   Claims Authentication   ah26c   Verbose SPOAuthHttpChallenge: Setting WWW-Authenticate header to:Bearer realm="87d64469-386e-4b12-bfc2-a405b44ca0e2",client_id="00000003-0000-0ff1-ce00-000000000000",trusted_issuers="00000005-0000-0000-c000-000000000000@*,5148fd17-f9b2-4739-aa9d-462fedfcba7c@87d64469-386e-4b12-bfc2-a405b44ca0e2,00000003-0000-0ff1-ce00-000000000000@87d64469-386e-4b12-bfc2-a405b44ca0e2",cookie_uri="https://[url_of_my_portal]/_api/SP.OAuth.NativeClient/Authenticate"  be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.04  w3wp.exe (0x0E8C)   0x13CC  SharePoint Foundation   Claims Authentication   a1dea   Medium  SPOAuthHttpChallenge: Adding OAuth WWW-Authenticate challenge without clearing others.  be74d09d-6a91-200e-ee4e-df98ffb0a421
01.31.2017 16:22:51.04  w3wp.exe (0x0E8C)   0x13CC  SharePoint Foundation   Claims Authentication   ax3f1   Verbose SPSuspendedFeaturesHttpHeader: Setting x-ms-suspended-features header to:features=""    be74d09d-6a91-200e-ee4e-df98ffb0a421

Answer 1

And, here's an update: KB3203432 - https://support.microsoft.com/en-us/help/3203432/descriptionofthesecurityupdateforsharepointserver2016june13-2017 did seem to fix this problem (the 401 vs 403 issue) without using the module above. However, it then created another problem for us in our on-premises SP 2016 system with Office 2016 (and claims auth via AD FS). A note in the KB says this:

Administrators who wish to suppress modern authentication with Office 2016 applications can now configure the SPSecurityTokenServiceConfig object when the SuppressModernAuthForOfficeClients property is set to $false.

But in fact, after the update, the default value of SuppressModernAuthForOfficeClients is set to $false, which causes Office clients to fail authentication with the cryptic "Your organization's policies..." message. To get back to normal, you have to do this:

$c = get-spsecuritytokenserviceconfig
$c.SuppressModernAuthForOfficeClients=$true
$c.update()

Answer 2

Try this solution:

1. HKCU\SOFTWARE\Microsoft\Office\16.0\Common\Identity
2. Check if EnableADAL key is present
3. If not present then create new REG_DWORD key with name EnableADAL and value 0

This worked for me

Answer 3

I was having the exact same problem. Using SharePoint 2016, Office 2016, and ADFS 4.0/2016, the only way I can get it to work is to disable ADAL. ADAL is not supported for on-premise Exchange, so I wonder if the same is true for SharePoint as well. SharePoint 2013 in the exact same environment works OK.

Poking around in Fiddler, I can see a few differences with ADAL enabled/disabled. With it enabled, the request headers are different, the server returns a 401 vs a 403, and there is also a bit about hitting an OAuth URL.

With ADAL enabled: OPTIONS hxxps://sharepoint.domain/Shared%20Documents/ HTTP/1.1 Connection: Keep-Alive Authorization: Bearer User-Agent: Microsoft Office Word 2014 (16.0.4456) Windows NT 10.0 X-Office-Major-Version: 16 X-MS-CookieUri-Requested: t X-FeatureVersion: 1 X-MSGETWEBURL: t X-IDCRL_ACCEPTED: t Host: sharepoint.domain

HTTP/1.1 401 Unauthorized Content-Type: text/plain; charset=utf-8 Server: Microsoft-IIS/8.5 X-SharePointHealthScore: 0 SPRequestGuid: 75fed49d-4537-0085-da92-b195d2c7ea26 request-id: 75fed49d-4537-0085-da92-b195d2c7ea26 X-Forms_Based_Auth_Required: hxxps://sharepoint.domain/_login/default.aspx?ReturnUrl=/_layouts/15/error.aspx X-Forms_Based_Auth_Return_Url: hxxps://sharepoint.domain/_layouts/15/error.aspx X-MSDAVEXT_Error: 917656; Access+denied.+Before+opening+files+in+this+location%2c+you+must+first+browse+to+the+web+site+and+select+the+option+to+login+automatically. x-ms-suspended-features: features="" X-Powered-By: ASP.NET MicrosoftSharePointTeamServices: 16.0.0.4483 X-Content-Type-Options: nosniff X-MS-InvokeApp: 1; RequireReadOnly WWW-Authenticate: Bearer realm="{Removed GUID}",client_id="00000003-0000-0ff1-ce00-000000000000",trusted_issuers="00000003-0000-0ff1-ce00-000000000000@{Removed GUID}",cookie_uri="https://sharepoint.domain/_api/SP.OAuth.NativeClient/Authenticate" Date: Tue, 14 Feb 2017 17:45:15 GMT Content-Length: 13

403 FORBIDDEN

With ADAL disabled: OPTIONS hxxs://sharepoint.domain/Shared%20Documents/ HTTP/1.1 Connection: Keep-Alive User-Agent: Microsoft Office Word 2014 (16.0.4456) Windows NT 10.0 X-Office-Major-Version: 16 X-MSGETWEBURL: t X-IDCRL_ACCEPTED: t Host: sharepoint.domain

HTTP/1.1 403 FORBIDDEN Content-Type: text/plain; charset=utf-8 Server: Microsoft-IIS/8.5 X-SharePointHealthScore: 0 SPRequestGuid: da02d59d-d5b7-0085-da92-b5fe7b8c3434 request-id: da02d59d-d5b7-0085-da92-b5fe7b8c3434 X-Forms_Based_Auth_Required: hxxps://sharepoint.epi.ophth.wisc.edu/_login/default.aspx?ReturnUrl=/_layouts/15/error.aspx X-Forms_Based_Auth_Return_Url: hxxs://sharepoint.epi.ophth.wisc.edu/_layouts/15/error.aspx X-MSDAVEXT_Error: 917656; Access+denied.+Before+opening+files+in+this+location%2c+you+must+first+browse+to+the+web+site+and+select+the+option+to+login+automatically. X-Powered-By: ASP.NET MicrosoftSharePointTeamServices: 16.0.0.4483 X-Content-Type-Options: nosniff X-MS-InvokeApp: 1; RequireReadOnly Date: Tue, 14 Feb 2017 19:02:05 GMT Content-Length: 13

403 FORBIDDEN

Answer 4

I'm currently having the same problem, and I have an open support case with Microsoft.

My environment is mostly Mac users, running various versions of Office For Mac 2016. There are also a few stragglers who haven't upgraded, and are still running Office for Mac 2011.

Unfortunately, I didn't catch this issue while testing my upgrade from 2010->2013->2016, so it's affecting my production environment.

While working with Microsoft to address the root cause of the issue, I developed workaround that transparently proxies the HTTPS requests and rewrites the headers. I am running this proxy in production.

Specifically, it removes the "Authorization" header from the client's request BEFORE the request is delivered to the server. This causes the server not to respond with a WWW-Authenticate header, and instead it sets the FedAUTH cookie - thereby enabling the Office Client application to load the document.

There are still some occasional issues: * Sometimes the client application displays the Site Collection Home Page instead of the ADFS login page * The Office for Mac client needs to be 15.30. Older versions of 2016 do not work (though 2011 works fine).

This forces the clients to always display the ADFS Sign in page.

I posted the sanitized server setup script here: https://github.com/crossan007/SharePoint-Office-16-Claims-Proxy

After cloning the repository, and installing vagrant / virtual box, you can run "vagrant up" and it will automatically start and configure the proxy server locally on your machine.

I'll follow up on this post when I get more details.

Answer 5

I dug into this, and here's the answer and a solution. The short answer is that SharePoint 2016 is returning a 401 status when it should return a 403 (as noted by the OP) and this is a bug. The specific use case that it breaks is that you open Word, then access a document that's stored in SharePoint from your recently accessed documents list. Assuming that you weren't previously authenticated, you are then prompted for Windows credentials even though (a) you're off-domain, and (b) the web application doesn't have NTLM enabled.

What's actually happening is that SP is generating the 403, but it's getting changed into a 401 by one of the modules in the pipeline. The solution that I found to work is to create an HttpModule and use that with your web application. The sole purpose of this module is to end the request with a 403 status before it can go awry. It does it only under a certain set of circumstances: the user agent is MS Office, and the http verb is "OPTIONS". Here's the code:

using System;
using System.Web;

namespace SP2016FilterModule
{
    public class SP2016FilterModule : IHttpModule
    {
        /// <summary>
        /// You will need to configure this module in the Web.config file of your
        /// web and register it with IIS before being able to use it. For more information
        /// see the following link: http://go.microsoft.com/?linkid=8101007
        /// </summary>
        #region IHttpModule Members

        public void Dispose()
        {
            //clean-up code here.
        }

        public void Init(HttpApplication context)
        {
            context.EndRequest += new EventHandler(OnEndRequest);
        }

        #endregion

        public void OnEndRequest(Object sender, EventArgs e)
        {
            var context = sender as HttpApplication;
            var request = context.Request;
            var response = context.Response;

            if (request.HttpMethod == System.Net.Http.HttpMethod.Options.Method
                && !request.IsAuthenticated
                && request.UserAgent.ToLower().Contains("microsoft office")
                && (response.StatusCode == 401 || response.StatusCode == 403))
            {
                response.StatusCode = 403;
                response.Write("403 FORBIDDEN");
                response.Flush();
                context.CompleteRequest();
            }
        }
    }
}

When you put the reference to this module in your (SharePoint web application) web.config, be sure to put it at the top, so it gets run first, because it's one of the other modules below that's responsible for the breakage. It looks like this (note that you'll have to sign your assembly in visual studio and generate your own strong name):

<modules runAllManagedModulesForAllRequests="true">
  <remove name="AnonymousIdentification" />
  <remove name="FileAuthorization" />
  <remove name="Profile" />
  <remove name="WebDAVModule" />
  <remove name="Session" />
  <add name="SP2016FilterModule" type="SP2016FilterModule.SP2016FilterModule, SP2016FilterModule, Version=1.0.0.0, Culture=neutral, PublicKeyToken=**your PKT here ***" />
  <add name="SPNativeRequestModule" preCondition="integratedMode" />

As a side note, the business in the referenced articles about "modern authentication" (what does that even mean?) is completely misguided and not helpful. Applying registry settings to every client that might access your system is also probably not a viable solution for most people.

Answer 6

This seems to have been fixed by KB3203432 - https://support.microsoft.com/en-us/help/3203432/descriptionofthesecurityupdateforsharepointserver2016june13,2017

Answer 7

Scott, this saved my day. After several days of investigation this was finally the solution. Thanks a lot for this post!!

We have NTLM and ADFS Trusted Provider activated on the Web Apps Default Zone and Used LDAPCP and SPBypassLoginPage which works great without the need of extending the Web App and use two different Zones for NTLM and ADFS Trusted Provider.

Answer 8

The post above by Scott D Carson Solves the "Your organizations policies are preventing you from completing this action" I have been working on this and on the phone with our identity management providers helpdesk since CHRISTMAS!!!!!. Thank you so much Scott for probably saving my job. I have googled my brains off and finally came across this by accident. Im a one man show and I thought it was never going to happen.

$c = get-spsecuritytokenserviceconfig
$c.SuppressModernAuthForOfficeClients=$true
$c.update()