Server Running Slowly - High CPU Usage

For the past month or so our main server (running 2003 SBS) has been performing extremely slowly, with the CPU usage of the server often maxing out. I've spent a lot of time looking into it all, but I'm running out of ideas now of where to look and could do with some advise/a second opinion. I've written a mini report detailing the symptons and what I've done so far, which is all detailed below. I know there's quite a lot written, but I'd really appreciate whatever help others can give.


The server is a HP Proliant ML350 G3. It's running with 2.5GB of RAM and has an Intel Xeon CPU @ 3.06GHz CPU. It's got 6 SCSI drives running in RAID5, split into 2 partitions.

Whilst looking into it all the main programs I've been using have been Process Explorer and Process Monitor.

Symptoms and investigation
After looking into it I've managed to identify a number of processes running on the server which appear to be the main culprits behind the performance issues, with them often hogging the CPU for no apparent reason.

The main processes that are causing the problems are explorer.exe, winlogon.exe and taskmgr.exe. I've verified that they are all the legitimate versions of the programs and located in their proper locations.

I've looked into each program individually and come up with the following bits about them:
winlogon.exe
- It appears to be completely random when this starts hogging the CPU. I've checked the logs on the server and I can't see any users logging onto any systems anywhere which could cause it's usage to spike.
- If I log onto the server through another session, it doesn't appear to spike and neither does it if I log onto another system (with it trying to identify me against AD)
- Checked

explorer.exe
- The CPU usage for this appears to spike on the server whenever I'm browsing around the server (either just going through folders in Windows Explorer, or going through the start menu).
- I can make the CPU usage for this program shoot up just by browsing to an empty folder and then refreshing the screen. Whilst doing this I've monitored the theads in the app (using procexp) and found that there are two threads whos usage go up: BROWSEUI!DllCanUnloadNow+0x12b8 and ntdll.dll!RtlOpenCurrentUser+0x275. At all other times, these threads sit idle, with SHLWAPI.dll!SHCreateThread+0x158 being the only thread showing any usage (albeit pretty low, about 0.5% to 5% CPU)
- Browsing the server over the network or locally through the DOS prompt doesn't appear to increase CPU usage.
- I've checked the threads above when they have been spiking and I've not seen any unusual/unknown calls in their stack, although I could be missing something. An example of the BROWSEUI stack is:

Code:

ntoskrnl.exe+0x398bd
hal.dll+0x63d9
hal.dll+0x61ae
ntdll.dll!KiFastSystemCallRet
ADVAPI32.dll!RegDeleteValueW+0x5a
SHLWAPI.dll!SHDeleteValueW+0x2c
SHLWAPI.dll!Ordinal530+0x45
SHLWAPI.dll!SHDeleteValueW+0x14d
SHLWAPI.dll!Ordinal535+0x27
SHELL32.dll!SHFindFiles+0x27a4
SHELL32.dll!SHFindFiles+0x2757
SHELL32.dll!SHFindFiles+0x267a
SHELL32.dll!Ordinal719+0x686
SHELL32.dll!Ordinal719+0x5c8
SHELL32.dll!SHGetMalloc+0x9155
SHELL32.dll!Ordinal237+0x6ca
SHLWAPI.dll!Ordinal164+0x3e
SHDOCVW.dll!Ordinal232+0x7c3f
BROWSEUI.dll!Ordinal102+0x290a
BROWSEUI.dll!Ordinal135+0xe6d6
BROWSEUI.dll!Ordinal102+0xa17a
USER32.dll!LoadCursorW+0x4cf5
USER32.dll!LoadCursorW+0x4e86
USER32.dll!IsWindow+0x148
USER32.dll!SendMessageW+0x49
comctl32.dll!FlatSB_GetScrollProp+0xa172
comctl32.dll!FlatSB_GetScrollProp+0xad29
USER32.dll!LoadCursorW+0x4cf5
USER32.dll!LoadCursorW+0x4e86
USER32.dll!IsWindow+0x148
USER32.dll!SendMessageW+0x49
comctl32.dll!FlatSB_SetScrollProp+0x2f58d
comctl32.dll!CreateToolbar+0xb67
USER32.dll!LoadCursorW+0x4cf5
USER32.dll!LoadCursorW+0x4e86
USER32.dll!TranslateMessageEx+0x10d
USER32.dll!DispatchMessageW+0xf
BROWSEUI.dll!Ordinal102+0xb2e8
BROWSEUI.dll!Ordinal102+0xb3d4
BROWSEUI.dll!DllCanUnloadNow+0x12fc
kernel32.dll!GetModuleHandleA+0xdf

An example of the ntdll stack is:

Code:

ntoskrnl.exe+0x397ea
ntoskrnl.exe+0x3df9e
ntoskrnl.exe+0x50675
ntoskrnl.exe+0x40720
ntoskrnl.exe+0x3e0a2
ntoskrnl.exe+0x12af3b
ntoskrnl.exe+0x33bef
ntdll.dll!KiFastSystemCallRet
kernel32.dll!GetModuleHandleA+0xdf

taskmgr.exe
- As soon as the task manager is opened it's CPU usage rockets, with it taking 25% CPU at the lowest and peaking at whatever maxes the CPU usage when combined with the other apps (80%+)
- Slightly interesting: Process monitor's CPU usage is minimal 0.5% to 5% CPU usage normally.

Troubleshooting
I've used Process Monitor to look at everything going on in the server, to see whether I could notice any obvious issues (like lots of errors trying to access some file or registry key or something), but I couldn't see anything particularly out of the ordinary.

Last week I did notice that Internet Explorer was taking a huge amount of CPU usage up when launching and running. Knowing how deeply tied into Windows this is, I decided to try first looking into whether I could speed this up. This ended up being pretty easy, with the problem being a java addon. I simply uninstalled the Java SDK (along with all the other versions) as it's not needed on the server and IE has been fine since then. Unfortunatly this hasn't resolved any problems elsewhere.

I've tried disabling Kaspersky on the server completely (a number of times), but this has made no difference to the speed. After disabling it I've tried loading new sessions on the server, incase Kaspersky could still be linked into any of the already running apps, but again there was no change.

As explorer.exe is affected when browsing around the system, I thought it could possibly be a bad shell extension, so I used ShellExView to see all the installed extensions and disabled everything that wasn't microsoft, or that was only a few months old. This made no difference.

I used RegDLLView to see if there were any DLL's registered recently which could have effected it, but they are all several months old and I know what the most recent are and can't see any issues.

I've had the thought several times that it could be a fault with the RAID, but I've checked this using HP's Array diagnostic tools and it's all completely fine.

Other Info
I've checked the fragmentation of the partitions and they are unfortunatly heavily fragmented. I've attempted to defrag both the partitions over several weekends, but they aren't able to complete fully. To do them properly I'd need to take the server offline for at least a day (I'm guessing), which isn't currently feasible. I'm not convinced that this is the issue causing the huge CPU usage, but it's probably having an effect is worth noting.



I am currently starting to wonder whether there could be a virus on the server somewhere, which my AV (Kaspersky) has missed. I've tried running GMER on the server a couple of times to see if it can find any rootkits, but it keeps freezing up the server. The first time it locked up was whilst GMER was doing a scan. The second time it locked was I started GMER, but beforeit started doing a full scan. I'd had to deal with something else after initially loading GMER, and whilst I was looking at something else the server froze.
I tried using DDS on the server to get the required reports for starting a thread in "Malware and Virus Removal" but it won't run, stating "This tool does not your support your Operating System ".



Thanks once again, any help will be greatly appreciated as this is doing my head in!

 

Do you run scheduled virus scans on your system? This could cause performance issues. If you do, try disabling the A/V for a day or two and see if performance improves.

 

The server is set to do a quick scan every night of all updated/new files and then does a full scan over the weekend. I've tried disabling it a few times for a few hours, but it has never made any difference

I've come into the office over the weekend to do a defrag of the server to see if that makes any difference.