Thursday, March 15, 2012

Rebuilding ArcGIS Server / Troubleshooting Proxy.ashx

Recently my ArcGIS Server went down.  For some unknown reason the SOM process would not start.

The error message in the system logs was:

The ArcGIS Server Object Manager service failed to start due to the following error: 
ArcGIS Server Object Manager is not a valid Win32 application. 

I also had this error in the Application log:

The application, C:\Program Files (x86)\ArcGIS\Server10.0\bin\ArcSOM.exe, generated an application error The error occurred on 03/12/2012 @ 13:58:12.647 The exception generated was c0000005 at address 0280C218 (ServerCore)

I started by doing many iterations of re-installing and configuring everything, which did not help.  I tried making some recommended registry changes which also did not help.  Eventually I came to believe that the problem was with the operating system, and started searching for similar cases where a windows service would not start.

I found this post which was very helpful:

http://www.technipages.com/error-193-1-is-not-a-valid-win32-application.html

It turned out that there was a file named "c:\program" that had been created by some unknown process the day before the application stopped running.   Because of the space in the folder name "Program Files", the system was getting confused and tried to run the "c:\program" file instead of my service file.  Deleting the file "c:\program" allowed my service to start.

Unfortunately I had basically nuked all my configuration settings with permissions / SSL etc, which had to be rebuild.  Of course that wasn't smooth and took several iterations of uninstall / install / configure to get right.  On the last iteration I removed the ArcGIS server configuration folder "c:\inetpub\wwwroot\arcgis", which allowed a clean recreate of all those settings.  Then I rebuilt everything using esri's documentation.

Next, my application was not working.  It uses dynamic token authentication via proxy as described here:

http://forums.esri.com/Thread.asp?c=158&f=2396&t=297001#926770

Each request to the server from the proxy page received a 401 Error-   I thought that it was a problem with the configuration of the token service, but eventually after trying out every possible IIS configuration, and confirming that the token was being received, I figured there was an issue with the code.  This was probably compounded by my authentication setting on the tokens virtual directory, which was set to integrated windows authentication (which was the only option that allowed the Rest directory and ArcCatalog to work from the local and remote machines).  

I added the code below to my proxy page code from this blog post.

webClient.Credentials=new NetworkCredential( "username","password","domain");

Which made the application work again.   Sounds easy, only took about 5 hours to figure out :)

Hopefully this can help someone out there!  The whole process starting with the server not working took about 24 working hours to resolve.   Soooo tired....