Access Denied on all Sharepoint 2013 Site collections

Question

I was toying around with the Application Initialization IIS feature, and something strange appears to have happened. I was making some edits to the applicationConfig file per the MSDN instructions but couldn't seem to get it working. I rolled back everything I had done, but now all of my web application prompt me with the Access Required screen, even when logged in as Farm Admin. I tried changing the site collection admin to a different user account, but that account also gets the access required page.

I can still access the CA, but it doesn't seem like I can do much from here to fix the problem. I tried using stsadm to unlock one of the web applications but I still am hitting the access required screen. Event logs are spotless as well, so I am confused as to what had happened.

I should also note I had to restore a backup of the "master" table in the Sharepoint DB this morning due to a bad program deciding that was a good place to put all of its new tables. All other aspects of SQL seems to be working OK though.

Even our Portal site that utilize FBA will present the Access Required page once good FBA credentials have been passed to it. Even stranger is that I can still access the site from SharePoint Designer, and view the membership lists. They still contain all the correct user accounts.

If I create a new site collection under any of my existing web applications I can log right into it. However, after a few moments then the new site will lock me out as well!

Answer 1

I discovered the cause of all of my pain was restoring a VMware snapshot that was two days old, without restoring the content DB. This led to my content database being out of sync with the SharePoint timer services.

I was finally able to restore functionality with the following steps:

1. Stop the VMware SP frontend
2. Revert the SP frontend back to the most recent snapshot
3. Restore ALL content databases, including SharePoint_config from backups made the same day as the VM snapshot.
4. Turn the SP frontend VM back on

The ONLY reason this worked was because I had backups of the SQL server that were created close enough to the VMware snapshot of the frontend. They was about 6 hours in between the snapshot of the DB and the SP server itself but it was close enough that it managed to hook back up successfully. I couldn't tell you if this would work again if I had to repeat the process!

The environment is now working just fine as though nothing ever happened. Lesson to be learned is VM snapshots don't work well unless you having matching DB backups as well. It was a rookie mistake and a hard lesson learned, but I managed to make it out alive.