Adventures With SharePoint 2010 Security Updates and Troubleshooting PSConfig Errors

Recently we installed some security updates to our SharePoint servers, only to find out later that we were having issues caused by our incorrect application of those updates. The problems surfaced when we were trying to do a routine backup from our production site collection and then restore that site collection down to a dev and test box.  We’ve done this numerous times without issue but all of a sudden it was failing for no (known to us) reason.  We were getting the following error message:

Restore-SPSite : Your backup is from a different version of Microsoft SharePoint Foundation and cannot be restored to a server running the current version. The backup file should be restored to a server with version ‘4.0.145.0’ or later.

What seemed strange to us was the fact that all of our boxes were in fact the same version, so what was going on?  The rest of this article details all the troubleshooting steps we took, other issues we encountered along the way, and some other useful tips that we picked up on our journey.

Realization – You must manually run the Products Configuration Wizard (PSConfig.exe) after you install SharePoint security patches and updates

So we knew that you had to run PSConfig after installing Service Pack 1; however, we didn’t realize you had to run it after installing security updates as well.  It took some digging but we finally verified that this was the likely reason why our backup/restores were no longer working (searching on the actual error message above led us on a wild goose chase).  No problem, we’ll just run PSConfig and that will fix our problem, I thought.  That is, unless it fails to complete due to errors… and lots of them!

Before I get into the actual errors we received and the subsequent troubleshooting we did, I want to mention some prerequisites you should do before running PSConfig, and then actually how to run the utility via a command line for reference.

Prerequisites for Running PSConfig

This is the excellent post we finally found that informed us we had to run PSConfig in order to complete the security patch installation.  It also details a PowerShell command you can run to determine if you even need to complete the update process:

1. Open a SharePoint PowerShell command window (All Programs — Microsoft SharePoint 2010 Products — SharePoint 2010 Management Shell)
2. Type the following command and press Enter:  (get-spserver $env.computername).NeedsUpgrade (this will return either True or False)

Note:  One thing we discovered about this command – if you run PSConfig and it fails, then subsequently running the above command will return False, even though PSConfig did not complete successfully.  For this reason, a better way to determine whether you need to run PSConfig may be to navigate to a page in Central Admin that will tell you whether any of the SharePoint databases need upgrading:

• Central Admin — Upgrade and Migration — Review database status

In addition, something we learned the hard way is that instead of just jumping right in and running PSConfig, we probably should have run the test-spcontentdatabase PowerShell cmdlet first.  Running this identifies any missing features, solutions, or other elements that might cause PSConfig to fail, so you can fix any issues ahead of time.  To run it, you should do the following:

1. Open a SharePoint PowerShell command window (All Programs — Microsoft SharePoint 2010 Products — SharePoint 2010 Management Shell)
2. Type the following command and press Enter:  test-spcontentdatabase -Identity [name of content database]

If you run this first and resolve all errors that appear (if any), there is a much greater chance that running PSConfig will be successful.

Running the PSConfig Utility

The same article mentioned above explains how to run the PSConfig utility via a command prompt:

1. Open an Administrative command prompt
2. Change directory to C:Program FilesCommon FilesMicrosoft SharedWeb Server Extensions14BIN
3. Run PSConfig.exe -cmd upgrade -inplace b2b -force -cmd applicationcontent -install -cmd installfeatures
4. Run iisreset

In the command window you will see a status as it’s running, and whether it completed successfully or not.  You can also see much of this information in Central Admin – this page will tell you whether the upgrade completed or failed, and error messages if it failed:

• Central Admin — Upgrade and Migration — Upgrade and migration

As mentioned we just ran PSConfig without any kind of prerequisite preparation and we garnered many errors which caused it to fail.  We had to troubleshoot one by one until all the issues were resolved before it would run successfully.  We had quite a time knowing where to even begin, and the next part of this article explains how we went about troubleshooting the issues.

Where the Heck are the (Useful) Error Logs?

The first challenge we faced in troubleshooting was locating the actual error messages as to why running PSConfig failed.  Turns out that the error log you are directed to after failure does not contain any useful troubleshooting information.  There is another log located in the SharePoint root folder that does provide more detailed error messages, however.  Waldek Mastykarz wrote an excellent article recently that explains this in great detail – basically you want to navigate to the location of the log file mentioned in the output (by default it’s C:Program FilesCommon FilesMicrosoft SharedWeb Server Extensions14LOGS) and locate the file with the same name but with “-error” appended at the end of the file name.  For example, Upgrade-20111205-17899-833-error.log.

Troubleshooting and Fixing the Errors

Once we located the useful error messages, the fun really began.  We were seeing numerous issues of missing features, missing setup files, and missing web parts in the error log.  We learned that there is an incorrect way to remove these artifacts from SharePoint, and evidently we had at one time done it “the wrong way”.  Apparently if you don’t remove an item correctly, a reference to it can remain in the content database, even though it doesn’t technically exist.

In addition, we discovered that the majority of the errors occurred on our dev server, with fewer on our test server, and then only 1 or 2 on our production servers.  Which makes perfect sense because it’s in development where we’re likely to throw stuff on the server that never makes it into production.  Due to the age of many of the items that were missing, we presume that they were items deployed to the servers while SharePoint 2010 was still in Beta, and perhaps Visual Studio at the time wasn’t removing all traces of the solutions when we retracted or removed them (well, that’s my story and I’m sticking to it!).

One solution I kept seeing over and over again was to simply install any missing features back to the server.  This wasn’t an option because these were things that we did not want on our servers.

So I soldiered on and finally found some articles written by Phil Childs that explained first how to find the offending references, and then how to remove them.  I placed all the different error messages, solution links, and notes that I accumulated throughout my trial and error of troubleshooting our development server into this handy-dandy cheat sheet:

My cheat sheet actually proved to be quite helpful when I had to go through the whole process again on the test and then production servers.

Some other miscellaneous tips that I learned throughout this whole ordeal:

• After you run PSConfig (whether successfully or not) you have to change the SecurityTokenServiceApplicationPool app pool back to the proper account.  Running PSConfig changes it to ‘LocalSystem’ but does not change it back when done (we discovered this because forms-based authentication stopped working after we ran PSConfig and we traced it back to this; also it only did this in our dev and test environments; it did not do it in production – not sure what that means)

• If PSConfig fails to run, you may need to restart the World Wide Web Publishing Service.  If this service isn’t running, none of your web sites on the server will run

• You must run PSConfig on every server in your production farm

• For multiple web front end servers, you should run PSConfig first on the server that has Central Admin installed, and when it’s finished you should run it on the other server(s).  While it is possible to start the utility on all servers at the same time since the process runs asynchronously, it’s probably better to do them one at a time.  If it errors out on any of the servers, you’ll have to manually restart it after you’ve fixed the problem anyway

• The first server that you run PSConfig on will take the longest because it is also upgrading the back end databases

Many thanks to the following folks for answering some of my questions throughout this whole ordeal via Twitter or webcasts:  @jdwade, @naupliustrevor, @ToddKlindt, @ShanesCows, @brianlala, @ericharlan, @warrtalon, @GrumpyTech, and @EVanRoy.

In Summary

The root cause of our issue was the fact that we had not “upgraded” our SharePoint databases after installing SharePoint security updates, and this broke SharePoint’s backup/restore feature.  The solution was to run PSConfig.exe via a command line; however, when running it we encountered many errors and issues that could have been avoided had we followed best practices for removing features and solutions that were no longer being used.  Once we resolved all the issues that caused PSConfig to fail to complete, we finally ran it successfully and the backup/restore issue we were facing was resolved.

Disclaimer:  This may all be common knowledge for all you SharePoint Admins out there, but since I’m a developer this is all new for me so I’m documenting in case I ever run into these issues again.  If anything appears to be inaccurate, please let me know via the comments!