Windows 2003 Server Ent. Ed. repeatedly crashing for 5-10 hours every 20 days or so

We have an HP Proliant DL580 G2 as a DB server (Windows 2003 Enterprise Edition upgraded from Windows 2000 Ent. Ed., SLQ 2000). The server has mirrored memory (4GB) and has SAN connectivity for backend data storage utilizing dual HBA cards. We have seen an issue on three separate occasions where the server will give the BSOD and reboot constantly for 10 hours or so. Sometimes it stays up for 2 minutes sometimes for 48, but never longer than that. After about 10 hours the problem just goes away.

The issue seems to occur every 20 days or so, and seems to happen between 5pm and 8am. That may just be a coincidence. I have pulled both network cards and HBA cards (which are all of the PCI-connected boards on the box) and the issue seems to stick around. I have also consolidated the memory on a single memory riser board (thus negating the mirrored memory aspect of the server). None of these changes stopped the BSOD.

Below is the most recent dump file. Any insight on what may be going on would be helpful!

Opened log file 'c:\debuglog.txt'

Microsoft (R) Windows Debugger Version 6.4.0007.2
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Temp\DB46_Dmp_Files\Mini032905-22.dmp]
Mini Kernel Dump File: Only registers and stack trace are available

Symbol search path is: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: C:\WINNT;C:\WINNT\system32;C:\WINNT\system32\drivers
Windows Server 2003 Kernel Version 3790 MP (4 procs) Free x86 compatible
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Built by: 3790.srv03_gdr.040410-1234
Kernel base = 0x804de000 PsLoadedModuleList = 0x8057b6a8
Debug session time: Tue Mar 29 06:09:13.065 2005 (GMT-6)
System Uptime: 0 days 0:01:20.812
Loading Kernel Symbols
...............................................................................................................
Loading unloaded module list
..
Loading User Symbols
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck A, {98c, ff, 0, 804e61d0}

Probably caused by : ntkrnlmp.exe ( nt!KiIdleLoop+10 )

Followup: MachineOwner
---------

0: kd> !analyze -v;r;kv;lmtn;.logclose;q
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 0000098c, memory referenced
Arg2: 000000ff, IRQL
Arg3: 00000000, value 0 = read operation, 1 = write operation
Arg4: 804e61d0, address which referenced memory

Debugging Details:
------------------


READ_ADDRESS: 0000098c

CURRENT_IRQL: 2

FAULTING_IP:
nt!KiIdleLoop+10
804e61d0 8b838c090000 mov eax,[ebx+0x98c]

CUSTOMER_CRASH_COUNT: 22

DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP

BUGCHECK_STR: 0xA

LAST_CONTROL_TRANSFER: from 00000000 to 804e61d0

TRAP_FRAME: 8056f574 -- (.trap ffffffff8056f574)
.trap ffffffff8056f574
ErrCode = 00000000
eax=00000028 ebx=00000000 ecx=00000000 edx=80010031 esi=80579d00 edi=8a2324c0
eip=804e61d0 esp=8056f5e8 ebp=80579f60 iopl=0 nv up di pl nz ac pe cy
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010013
nt!KiIdleLoop+0x10:
804e61d0 8b838c090000 mov eax,[ebx+0x98c] ds:0023:0000098c=????????
.trap
Resetting default scope

STACK_TEXT:
80579f60 00000000 80579f68 80579f68 80579f70 nt!KiIdleLoop+0x10


FOLLOWUP_IP:
nt!KiIdleLoop+10
804e61d0 8b838c090000 mov eax,[ebx+0x98c]

SYMBOL_STACK_INDEX: 0

FOLLOWUP_NAME: MachineOwner

SYMBOL_NAME: nt!KiIdleLoop+10

MODULE_NAME: nt

IMAGE_NAME: ntkrnlmp.exe

DEBUG_FLR_IMAGE_TIMESTAMP: 40b53739

STACK_COMMAND: .trap ffffffff8056f574 ; kb

FAILURE_BUCKET_ID: 0xA_nt!KiIdleLoop+10

BUCKET_ID: 0xA_nt!KiIdleLoop+10

Followup: MachineOwner
---------

eax=ffdff13c ebx=0000000a ecx=80579d00 edx=80010031 esi=ffdff120 edi=0000098c
eip=80543ac9 esp=8056f540 ebp=8056f558 iopl=0 nv up ei ng nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00000286
nt!KeBugCheckEx+0x19:
80543ac9 5d pop ebp
ChildEBP RetAddr Args to Child
8056f558 804e2f58 0000000a 0000098c 000000ff nt!KeBugCheckEx+0x19 (FPO: [Non-Fpo])
8056f558 804e61d0 0000000a 0000098c 000000ff nt!KiTrap0E+0x224 (FPO: [0,0] TrapFrame @ 8056f574)
80579f60 00000000 80579f68 80579f68 80579f70 nt!KiIdleLoop+0x10 (FPO: [0,0,0])
start end module name
804de000 80745000 nt ntkrnlmp.exe Wed May 26 19:32:57 2004 (40B53739)
80745000 8076d000 hal halmacpi.dll Tue Mar 25 01:07:28 2003 (3E800030)
b798c000 b79ae000 RDPWD RDPWD.SYS Tue Mar 25 01:03:00 2003 (3E7FFF24)
b79d6000 b7a03000 Fastfat Fastfat.SYS Tue Mar 25 02:00:16 2003 (3E800C90)
b7acb000 b7b2a000 srv srv.sys Tue Mar 25 02:49:51 2003 (3E80182F)
b7d0a000 b7d47000 TmXPFlt TmXPFlt.sys Fri Feb 18 05:04:33 2005 (4215CBC1)
b7d47000 b7e31f60 VsapiNT VsapiNT.sys Fri Feb 18 04:43:07 2005 (4215C6BB)
b7e5a000 b7e89000 afd afd.sys Tue Mar 25 01:40:50 2003 (3E800802)
b7f61000 b7f6b000 TDTCP TDTCP.SYS Tue Mar 25 01:02:52 2003 (3E7FFF1C)
b7fc9000 b7fe1000 Cdfs Cdfs.SYS Tue Mar 25 02:17:19 2003 (3E80108F)
b8059000 b806c000 Fips Fips.SYS Tue Mar 25 02:54:59 2003 (3E801963)
b806c000 b80d8000 mrxsmb mrxsmb.sys Tue Jan 18 19:35:18 2005 (41EDB956)
b80d8000 b8103000 rdbss rdbss.sys Mon Oct 11 19:38:09 2004 (416B2771)
b8153000 b815f000 tmpreflt tmpreflt.sys Fri Feb 18 05:04:32 2005 (4215CBC0)
b81cb000 b8200000 netbt netbt.sys Fri Jul 18 12:16:03 2003 (3F182B53)
b8200000 b8262000 tcpip tcpip.sys Tue Mar 25 02:04:01 2003 (3E800D71)
b8262000 b827e000 ipsec ipsec.sys Tue Mar 25 01:55:45 2003 (3E800B81)
b82de000 b82f1000 usbhub usbhub.sys Tue Mar 25 01:10:46 2003 (3E8000F6)
ba579000 ba63c000 dmboot dmboot.sys Tue Mar 25 01:08:18 2003 (3E800062)
ba736000 ba769000 update update.sys Tue Mar 25 02:59:59 2003 (3E801A8F)
ba769000 ba79d000 rdpdr rdpdr.sys Tue Mar 25 01:09:30 2003 (3E8000AA)
ba79d000 ba7b2000 raspptp raspptp.sys Tue Mar 25 02:19:09 2003 (3E8010FD)
ba7b2000 ba7cd000 ndiswan ndiswan.sys Tue Mar 25 02:48:19 2003 (3E8017D3)
ba7cd000 ba7e4000 rasl2tp rasl2tp.sys Tue Mar 25 01:54:46 2003 (3E800B46)
ba7e4000 ba804a80 cpqteam cpqteam.sys Tue Oct 07 19:03:27 2003 (3F83544F)
ba805000 ba822c00 n1000325 n1000325.sys Mon Oct 13 15:26:48 2003 (3F8B0A88)
ba823000 ba844e80 USBPORT USBPORT.SYS Tue Mar 25 01:10:43 2003 (3E8000F3)
ba845000 ba86f000 ks ks.sys Tue Mar 25 02:47:36 2003 (3E8017A8)
ba86f000 ba883000 redbook redbook.sys Tue Mar 25 01:04:38 2003 (3E7FFF86)
ba883000 ba897000 cdrom cdrom.sys Tue Mar 25 01:05:18 2003 (3E7FFFAE)
ba897000 ba8af000 serial serial.sys Tue Mar 25 01:40:08 2003 (3E8007D8)
ba8af000 ba8c5000 i8042prt i8042prt.sys Tue Mar 25 03:01:43 2003 (3E801AF7)
ba8c5000 ba8de000 VIDEOPRT VIDEOPRT.SYS Tue Mar 25 01:08:02 2003 (3E800052)
ba8de000 ba931d80 ati2mpad ati2mpad.sys Thu Jul 18 20:13:20 2002 (3D3767B0)
baa85000 baacf000 cpqasm2 cpqasm2.sys Mon Oct 13 18:38:14 2003 (3F8B3766)
baacf000 baad8000 raspti raspti.sys Tue Mar 25 01:11:36 2003 (3E800128)
baadf000 baaea000 ptilink ptilink.sys Tue Mar 25 01:03:51 2003 (3E7FFF57)
baaef000 baafa000 TDI TDI.SYS Tue Mar 25 01:14:28 2003 (3E8001D4)
baaff000 bab0d000 raspppoe raspppoe.sys Tue Mar 25 01:11:37 2003 (3E800129)
bab0f000 bab18000 ndistapi ndistapi.sys Tue Mar 25 01:11:28 2003 (3E800120)
bab2f000 bab39000 Dxapi Dxapi.sys Tue Mar 25 01:06:01 2003 (3E7FFFD9)
bab3f000 bab4c580 CPQCISSE CPQCISSE.sys Thu Mar 11 15:50:26 2004 (4050DF22)
bab4f000 bab5a000 fdc fdc.sys Tue Mar 25 01:04:31 2003 (3E7FFF7F)
bab5f000 bab69000 serenum serenum.sys Tue Mar 25 01:04:01 2003 (3E7FFF61)
bf800000 bf9c7000 win32k win32k.sys Tue Dec 28 17:26:09 2004 (41D1EB91)
bf9c7000 bfa1d680 ati2drad ati2drad.dll Tue Mar 25 03:43:37 2003 (3E8024C9)
bff80000 bff96000 dxg dxg.sys Tue Mar 25 03:46:23 2003 (3E80256F)
f7151000 f716b000 CPQPHP CPQPHP.SYS Thu Aug 28 17:23:13 2003 (3F4E80D1)
f716b000 f718d000 Mup Mup.sys Tue Mar 25 02:55:58 2003 (3E80199E)
f718d000 f71ce000 NDIS NDIS.sys Tue Mar 25 02:45:35 2003 (3E80172F)
f71ce000 f726b000 Ntfs Ntfs.sys Tue Mar 25 01:40:05 2003 (3E8007D5)
f726b000 f728c000 KSecDD KSecDD.sys Tue Mar 25 01:05:39 2003 (3E7FFFC3)
f728c000 f72a2000 CLASSPNP CLASSPNP.SYS Tue Mar 25 01:38:14 2003 (3E800766)
f72a2000 f72e9a80 elxsli2 elxsli2.sys Tue Oct 19 13:16:30 2004 (417559FE)
f72ea000 f7310000 SCSIPORT SCSIPORT.SYS Tue Mar 25 02:01:25 2003 (3E800CD5)
f7310000 f7328000 lp6nds35 lp6nds35.sys Tue Mar 25 01:05:03 2003 (3E7FFF9F)
f7328000 f7344000 atapi atapi.sys Tue Mar 25 01:04:48 2003 (3E7FFF90)
f7344000 f7354f00 EmcpMPAP EmcpMPAP.sys Wed Nov 24 14:35:49 2004 (41A4F0A5)
f7355000 f736d480 EmcpSAPI EmcpSAPI.sys Wed Nov 24 14:36:04 2004 (41A4F0B4)
f736e000 f738f000 volsnap volsnap.sys Tue Mar 25 01:05:47 2003 (3E7FFFCB)
f738f000 f73d0b80 EmcpBase EmcpBase.sys Wed Nov 24 14:31:17 2004 (41A4EF95)
f73d1000 f73fb000 dmio dmio.sys Tue Mar 25 01:08:14 2003 (3E80005E)
f73fb000 f7420000 ftdisk ftdisk.sys Tue Mar 25 01:05:26 2003 (3E7FFFB6)
f7420000 f7435000 pci pci.sys Tue Mar 25 01:16:40 2003 (3E800258)
f7435000 f7466000 ACPI ACPI.sys Tue Mar 25 01:16:21 2003 (3E800245)
f7487000 f7490000 WMILIB WMILIB.SYS Tue Mar 25 01:13:00 2003 (3E80017C)
f7497000 f74a6000 isapnp isapnp.sys Tue Mar 25 01:16:35 2003 (3E800253)
f74a7000 f74b4000 PCIIDEX PCIIDEX.SYS Tue Mar 25 01:04:44 2003 (3E7FFF8C)
f74b7000 f74c6000 MountMgr MountMgr.sys Tue Mar 25 01:03:05 2003 (3E7FFF29)
f74c7000 f74d0380 EmcpMP EmcpMP.sys Wed Nov 24 14:35:31 2004 (41A4F093)
f74d7000 f74e6680 EmcpMPC EmcpMPC.sys Wed Nov 24 14:31:30 2004 (41A4EFA2)
f74e7000 f74f2280 EmcpMPAA EmcpMPAA.sys Wed Nov 24 14:35:40 2004 (41A4F09C)
f74f7000 f7505000 PartMgr PartMgr.sys Tue Mar 25 02:04:02 2003 (3E800D72)
f7507000 f7511000 cpqcissm cpqcissm.sys Wed Jul 21 11:49:40 2004 (40FE9EA4)
f7517000 f7526000 disk disk.sys Tue Mar 25 01:05:20 2003 (3E7FFFB0)
f7527000 f7533000 Dfs Dfs.sys Tue Mar 25 01:09:52 2003 (3E8000C0)
f7537000 f7540000 crcdisk crcdisk.sys Tue Mar 25 01:07:23 2003 (3E80002B)
f7547000 f7550000 watchdog watchdog.sys Tue Mar 25 01:09:01 2003 (3E80008D)
f7557000 f7561000 kbdclass kbdclass.sys Tue Mar 25 01:03:10 2003 (3E7FFF2E)
f7567000 f7571000 mouclass mouclass.sys Tue Mar 25 01:03:09 2003 (3E7FFF2D)
f7577000 f7586000 termdd termdd.sys Tue Mar 25 01:02:52 2003 (3E7FFF1C)
f7587000 f7595000 NDProxy NDProxy.SYS Tue Mar 25 01:11:30 2003 (3E800122)
f7607000 f7611000 flpydisk flpydisk.sys Tue Mar 25 01:04:32 2003 (3E7FFF80)
f7617000 f7623000 vga vga.sys Tue Mar 25 01:08:03 2003 (3E800053)
f7627000 f7631000 Msfs Msfs.SYS Tue Mar 25 01:08:56 2003 (3E800088)
f7637000 f7643000 Npfs Npfs.SYS Tue Mar 25 01:08:59 2003 (3E80008B)
f7647000 f7656000 msgpc msgpc.sys Tue Mar 25 01:10:12 2003 (3E8000D4)
f7657000 f7664000 wanarp wanarp.sys Tue Mar 25 01:11:22 2003 (3E80011A)
f7667000 f7674000 netbios netbios.sys Tue Mar 25 01:09:53 2003 (3E8000C1)
f7687000 f7690000 dump_scsiport dump_scsiport.sys Tue Mar 25 01:05:15 2003 (3E7FFFAB)
f7697000 f76a1000 dump_cpqcissm dump_cpqcissm.sys Wed Jul 21 11:49:40 2004 (40FE9EA4)
f7707000 f770f000 kdcom kdcom.dll Tue Mar 25 01:08:00 2003 (3E800050)
f770f000 f7717000 BOOTVID BOOTVID.dll Tue Mar 25 01:07:58 2003 (3E80004E)
f7717000 f771e000 pciide pciide.sys Tue Mar 25 01:04:46 2003 (3E7FFF8E)
f771f000 f7726000 dmload dmload.sys Tue Mar 25 01:08:08 2003 (3E800058)
f7727000 f772ca80 EmcpCG EmcpCG.sys Wed Nov 24 14:35:57 2004 (41A4F0AD)
f779f000 f77a7000 OPRGHDLR OPRGHDLR.SYS Tue Mar 25 01:16:25 2003 (3E800249)
f77af000 f77b3a20 CpqCiDrv CpqCiDrv.sys Fri Jul 11 10:32:03 2003 (3F0ED873)
f77b7000 f77bb200 usbohci usbohci.sys Tue Mar 25 01:10:41 2003 (3E8000F1)
f77bf000 f77c7000 audstub audstub.sys Tue Mar 25 01:09:12 2003 (3E800098)
f77e7000 f77ef000 Fs_Rec Fs_Rec.SYS Tue Mar 25 01:08:36 2003 (3E800074)
f77ef000 f77f6000 Null Null.SYS Tue Mar 25 01:03:05 2003 (3E7FFF29)
f77f7000 f77fe000 Beep Beep.SYS Tue Mar 25 01:03:04 2003 (3E7FFF28)
f77ff000 f7807000 mnmdd mnmdd.SYS Tue Mar 25 01:07:53 2003 (3E800049)
f7807000 f780f000 RDPCDD RDPCDD.sys Tue Mar 25 01:03:05 2003 (3E7FFF29)
f780f000 f7817000 rasacd rasacd.sys Tue Mar 25 01:11:50 2003 (3E800136)
f781f000 f7826000 dxgthk dxgthk.sys Tue Mar 25 01:05:52 2003 (3E7FFFD0)
f7897000 f7899200 EmcpHR EmcpHR.sys Wed Nov 24 14:36:10 2004 (41A4F0BA)
f7999000 f799a200 swenum swenum.sys Tue Mar 25 01:03:22 2003 (3E7FFF3A)
f79ad000 f79ae580 USBD USBD.SYS Tue Mar 25 01:10:39 2003 (3E8000EF)
f7a50000 f7a50a80 EmcpClass EmcpClass.sys Wed Nov 24 14:31:23 2004 (41A4EF9B)
f7a85000 f7a85b00 sysmgmt sysmgmt.sys Wed Jan 29 13:44:45 2003 (3E382F2D)

Unloaded modules:
f7677000 f7685000 imapi.sys
Timestamp: unavailable (00000000)
Checksum: 00000000
f77df000 f77e4000 Cdaudio.SYS
Timestamp: unavailable (00000000)
Checksum: 00000000
Closing open log file c:\debuglog.txt

 

Thats an unusual set of behavior. I'll have to wait until tomorrow when i can unassemble that wait loop to see what its trying to read thats not there...

I'll let you know tomorrow probably. There is a chance that this one will have to go to microsoft support to be debugged, depending on how my unassemble goes.

 

Have you updated the ROMPaq for that machine yet? There is a known issue thats been resolved with this update:

http://h18007.www1.hp.com/support/files/server/us/download/20831.html