Event Source: W3SVC Event ID:1004 - IIS is NOT the issue

This took me about 3 hrs to troubleshoot so I hope this helps others - The error is on 2 Front End web server hosting SharePoint 2007 and it looks like an IIS 6.0 Issue - Nothing could be further from the truth.

Configuration: Server 1 & 2 have both SharePoint Portal Server Enterprise Edtion & Project Server 2007 Ent Edition installed as well. The Database Server is SQL Server 2005. *Note - as part of the configuration we configured one of the SSP Applications to use Port 8081. The client's environment had been working fine till one afternoon and the following errors occurred:

Event Type: Error
Event Source: W3SVC
Event Category: None
Event ID: 1004
Date: 3/25/2010
Time: 1:25:47 PM
User: N/A
Computer: VMPRJD02
Description:
Cannot register the URL prefix 'http://*:8081/' for site '517210341'. The site has been deactivated. The data field contains the error number.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

This is also accompanied by:

Event Type: Error
Event Source: HTTP
Event Category: None
Event ID: 15005
Date: 3/25/2010
Time: 1:25:47 PM
User: N/A
Computer: VMPRJD02
Description:
Unable to bind to the underlying transport for 0.0.0.0:8081. The IP Listen-Only list may contain a reference to an interface which may not exist on this machine. The data field contains the error number.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

There was no REAL way of figuring out what caused the error/issues to come up but after a bit of research this is what we found was causing the issue:

Long story short - I did alot of "Googling" but nothing came up that could help me figure out what was going on. SO...

1) I Isolated the problem by completely powering OFF one of the front end servers. This way I could narrow down the variables.
2) I needed to find out what was using the port 8081 and identify if it was IIS that was using the port or something else - here's how I isolated the issues further.
a. I located the site in IIS that was using port 8081 - in this case it was one of my SSP (Shared Service Providers). In IIS expand the websites - you'll probably notice that the "Default website and the website using port 8081" has been disabled or in a "STOPPED" state as per the error msgs above.
b. I just stopped IIS on the server - you can either do this via IIS MMC application or through Services on the server, eitherway just stop IIS.
c. I now needed to see if something was still using port 8081 so I opened a COMMAND window and typed (netstat -a) this will give you a display of all the connections and what ports they are using on the server.
c.1) I noticed that the port 8081 was still being used but it was listed as "listening "... - this meant that something other than IIS was using the port.
c.2) I ran a couple of cmds to clean out DNS (just in case) = ipconfig /flushdns and ipconfig /registerdns to renew things...
c.3) I re-ran the netstat -a and saw the same thing, this gave me an indicator that IIS was NOT using any ports...
d. So... That ruled out IIS as the issue - I then turned on IIS and ran the same tests again, this time I added the switch to netstat that will show you what .exe is using what port. *NOTE it will not show you what is in LISTENING mode, just active applications using current ports ( netstat -b )
here I ruled out that it wasnt SharePoint that was using the application as well by - (going to IIS and changing the website that was using port 8081 to use Port 8082) and the website comes up fine, NO errors in Log files so no reason to believe that SharePoint is the culprit either.
e. I decide to take a short cut and see what application was using the port 8081 by searching through the registry: Click Start, Run & in the dialog box type "regedit" - *Note do this at your own risk*
e.1) I click on "edit" and "find" at the top and in the search criteria I entered just the numbers - 8081 and clicked "search ".
f. THIS IS WHERE I FOUND THE CULPRIT - and wouldn't you know it - "The Mcafee antivirus agent" (WAS USING A DEDICATED PORT) of 8081 that sits in "Listening" mode… Now because it was configured to use that port "“ when the update was pushed to the server sometime during the week it sat "idle" the application experienced conflicts because one of the SharePoint applications we configured was already using port 8081.
g. The fix was simple: I contacted the IT Geeks and told them that their Antivirus program was using a dedicated port, I sent them the screenshot that shows in the registry the settings and gave them the plan of how I was going to restore the environment by Temporarily removing the AV Application then re-configure SharePoint to work correctly and RE-install the AV application on the server once I re-configured the SharePoint Application using Port 8081 to another port.

Realistically though - they should NOT have an AV Client or service use the same port ranges as normal web applications... I mean really??? pick some obscure port outside of range that SharePoint, ProjectServer 2007 & SQL 2005 uses..

Hope it help someone out there I'm glad this was just a Dev environment...

Work Smarter - Not Harder.. IISJUNKIE

 

I took a screenshot of where I found where the application was using the port in registry editor, is there any way I can add a picture to the post?