Resolved DUMP DATA 2003 R2 cluster

Hi all,
Wanted to see what you think here. We are thinking it is a combination of snapshot and backups running into each other. This is a 2003R2 file server cluster running on Equallogic/iSCSI SAN that bluescreened on one node this morning.:

Opened log file 'c:debuglog.txt'

Microsoft (R) Windows Debugger Version 6.12.0002.633 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Temp\MEMORY.DMP]
Kernel Summary Dump File: Only kernel address space is available

Symbol search path is: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: C:\Windows;C:\Windows\system32;C:\Windows\system32\drivers
Windows Server 2003 Kernel Version 3790 (Service Pack 2) MP (4 procs) Free x86 compatible
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Built by: 3790.srv03_sp2_gdr.090319-1204
Machine Name:
Kernel base = 0x80800000 PsLoadedModuleList = 0x808a6ea8
Debug session time: Thu Nov 1 08:16:44.707 2012 (UTC - 4:00)
System Uptime: 19 days 8:38:41.177
Loading Kernel Symbols
...............................................................
..........................................................
Loading User Symbols

Loading unloaded module list
...........
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck A, {0, d0000002, 1, 80841d44}

Probably caused by : memory_corruption ( nt!MiSegmentDelete+2a )

Followup: MachineOwner
---------

0: kd> !analyze -v;r;kv;lmtn;.logclose;q
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 00000000, memory referenced
Arg2: d0000002, IRQL
Arg3: 00000001, bitfield :
bit 0 : value 0 = read operation, 1 = write operation
bit 3 : value 0 = not an execute operation, 1 = execute operation (only on chips which support this level of status)
Arg4: 80841d44, address which referenced memory

Debugging Details:
------------------


WRITE_ADDRESS: 00000000

CURRENT_IRQL: 2

FAULTING_IP:
nt!MiSegmentDelete+2a
80841d44 890a mov dword ptr [edx],ecx

DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0xA

PROCESS_NAME: System

TRAP_FRAME: f7912cf4 -- (.trap 0xfffffffff7912cf4)
.trap 0xfffffffff7912cf4
ErrCode = 00000002
eax=00000000 ebx=80a5e4d0 ecx=f8d30e40 edx=00000000 esi=808ab4a4 edi=808ab4a0
eip=80841d44 esp=f7912d68 ebp=f7912d8c iopl=0 nv up ei ng nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010282
nt!MiSegmentDelete+0x2a:
80841d44 890a mov dword ptr [edx],ecx ds:0023:00000000=????????
.trap
Resetting default scope

LAST_CONTROL_TRANSFER: from 80841d44 to 8088c993

STACK_TEXT:
f7912cf4 80841d44 badb0d00 00000000 f7912d20 nt!KiTrap0E+0x2a7
f7912d8c 80843ab4 f8d30e40 00000000 8b77b2d0 nt!MiSegmentDelete+0x2a
f7912dac 80949b7c 00000000 00000000 00000000 nt!MiDereferenceSegmentThread+0xa6
f7912ddc 8088e092 80843a0e 00000000 00000000 nt!PspSystemThreadStartup+0x2e
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16


STACK_COMMAND: kb

FOLLOWUP_IP:
nt!MiSegmentDelete+2a
80841d44 890a mov dword ptr [edx],ecx

SYMBOL_STACK_INDEX: 1

SYMBOL_NAME: nt!MiSegmentDelete+2a

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: nt

DEBUG_FLR_IMAGE_TIMESTAMP: 49c21e56

IMAGE_NAME: memory_corruption

FAILURE_BUCKET_ID: 0xA_nt!MiSegmentDelete+2a

BUCKET_ID: 0xA_nt!MiSegmentDelete+2a

Followup: MachineOwner
---------

eax=ffdff13c ebx=d0000002 ecx=00000001 edx=00000001 esi=ffdff120 edi=00000000
eip=8088c993 esp=f7912cdc ebp=f7912cf4 iopl=0 nv up ei ng nz na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00000286
nt!KiTrap0E+0x2a7:
8088c993 833da0628a8000 cmp dword ptr [nt!KiFreezeFlag (808a62a0)],0 ds:0023:808a62a0=00000000
ChildEBP RetAddr Args to Child
f7912cf4 80841d44 badb0d00 00000000 f7912d20 nt!KiTrap0E+0x2a7 (FPO: [0,0] TrapFrame @ f7912cf4)
f7912d8c 80843ab4 f8d30e40 00000000 8b77b2d0 nt!MiSegmentDelete+0x2a (FPO: [Non-Fpo])
f7912dac 80949b7c 00000000 00000000 00000000 nt!MiDereferenceSegmentThread+0xa6 (FPO: [Non-Fpo])
f7912ddc 8088e092 80843a0e 00000000 00000000 nt!PspSystemThreadStartup+0x2e (FPO: [Non-Fpo])
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16
start end module name
80800000 80a5a000 nt ntkrpamp.exe Thu Mar 19 06:28:38 2009 (49C21E56)
80a5a000 80a86000 hal halmacpi.dll Sat Feb 17 00:48:26 2007 (45D6972A)
b8147000 b8171000 Fastfat Fastfat.SYS Sat Feb 17 01:27:55 2007 (45D6A06B)
b8cbc000 b8ce7000 RDPWD RDPWD.SYS Sat Feb 17 00:44:38 2007 (45D69646)
b8ddf000 b8dea000 TDTCP TDTCP.SYS Sat Feb 17 00:44:32 2007 (45D69640)
b9557000 b95b5000 srv srv.sys Thu Dec 11 06:35:59 2008 (4940FB1F)
b962d000 b967d000 HTTP HTTP.sys Sat Feb 17 01:28:12 2007 (45D6A07C)
b9b2d000 b9b45000 clusnet clusnet.sys Sat Feb 17 01:02:57 2007 (45D69A91)
b9c0d000 b9c2a000 dump_atapi dump_atapi.sys Sat Feb 17 01:07:34 2007 (45D69BA6)
b9c2a000 b9c3f000 Cdfs Cdfs.SYS Sat Feb 17 01:27:08 2007 (45D6A03C)
b9c3f000 b9c5d000 EraserUtilRebootDrv EraserUtilRebootDrv.sys Tue Jul 31 19:33:30 2012 (50186B4A)
b9c5d000 b9cbc000 eeCtrl eeCtrl.sys Tue Jul 31 19:33:30 2012 (50186B4A)
b9cbc000 b9ccd000 Fips Fips.SYS Sat Feb 17 01:26:33 2007 (45D6A019)
b9ccd000 b9d43000 mrxsmb mrxsmb.sys Fri Sep 05 11:26:52 2008 (48C14FBC)
b9d43000 b9d73000 rdbss rdbss.sys Sat Feb 17 01:27:37 2007 (45D6A059)
b9d73000 b9dd5000 SPBBCDrv SPBBCDrv.sys Tue Apr 11 19:55:08 2006 (443C41DC)
b9dd5000 b9dff000 afd afd.sys Thu Aug 14 06:46:56 2008 (48A40D20)
b9dff000 b9e30000 netbt netbt.sys Sat Feb 17 01:28:57 2007 (45D6A0A9)
b9e30000 b9ec0000 tcpip tcpip.sys Fri Jun 20 10:20:25 2008 (485BBCA9)
b9ec0000 b9ed9000 ipsec ipsec.sys Sat Feb 17 01:29:28 2007 (45D6A0C8)
b9ef9000 b9f10000 ClusDisk ClusDisk.sys Sat Feb 17 01:02:55 2007 (45D69A8F)
ba0ab000 ba0bf000 Savrtpel Savrtpel.sys Wed Sep 06 17:26:26 2006 (44FF3D02)
ba0bf000 ba0e1000 SYMEVENT SYMEVENT.SYS Mon Sep 18 20:52:19 2006 (450F3F43)
ba0e1000 ba139000 savrt savrt.sys Wed Sep 06 17:26:23 2006 (44FF3CFF)
ba1d9000 ba1ee000 usbhub usbhub.sys Sat Feb 17 01:13:05 2007 (45D69CF1)
ba24e000 ba2ad000 update update.sys Mon May 28 08:15:16 2007 (465AC7D4)
ba2ad000 ba2c4000 iscsiprt iscsiprt.sys Fri Nov 23 01:03:54 2007 (47466D4A)
ba2c4000 ba2ec000 msiscsi msiscsi.sys Fri Nov 23 01:03:53 2007 (47466D49)
ba2fc000 ba305000 hidusb hidusb.sys Tue Mar 25 03:10:17 2003 (3E8000D9)
ba31c000 ba329000 wanarp wanarp.sys Sat Feb 17 00:59:17 2007 (45D699B5)
ba34c000 ba35a000 HIDCLASS HIDCLASS.SYS Tue Mar 25 03:10:17 2003 (3E8000D9)
ba38c000 ba3c3000 rdpdr rdpdr.sys Sat Feb 17 00:51:00 2007 (45D697C4)
ba3c3000 ba3d5000 raspptp raspptp.sys Sat Feb 17 01:29:20 2007 (45D6A0C0)
ba3d5000 ba3ee000 ndiswan ndiswan.sys Sat Feb 17 01:29:22 2007 (45D6A0C2)
ba3ee000 ba402000 rasl2tp rasl2tp.sys Sat Feb 17 01:29:02 2007 (45D6A0AE)
ba402000 ba429000 ks ks.sys Sat Feb 17 01:30:40 2007 (45D6A110)
ba429000 ba43d000 redbook redbook.sys Sat Feb 17 01:07:26 2007 (45D69B9E)
ba43d000 ba452000 cdrom cdrom.sys Sat Feb 17 01:07:48 2007 (45D69BB4)
ba452000 ba467000 serial serial.sys Sat Feb 17 01:06:46 2007 (45D69B76)
ba467000 ba483000 VIDEOPRT VIDEOPRT.SYS Sat Feb 17 01:10:30 2007 (45D69C56)
ba483000 ba5eb000 ati2mtag ati2mtag.sys Wed May 02 22:27:21 2007 (46394889)
ba5eb000 ba615000 USBPORT USBPORT.SYS Sat Feb 17 01:12:59 2007 (45D69CEB)
ba615000 ba641000 b57xp32 b57xp32.sys Thu Nov 29 20:35:42 2007 (474F68EE)
ba641000 ba67f000 e1e5132 e1e5132.sys Wed Feb 06 20:22:17 2008 (47AA5D49)
ba6af000 ba6bd000 NDProxy NDProxy.SYS Sat Feb 17 00:59:21 2007 (45D699B9)
ba6df000 ba6ea000 dcdbas32 dcdbas32.sys Fri Jan 04 02:14:15 2008 (477DDCC7)
ba6ef000 ba6f8000 mssmbios mssmbios.sys Sat Feb 17 00:59:12 2007 (45D699B0)
ba70f000 ba719000 mouclass mouclass.sys Tue Mar 25 03:03:09 2003 (3E7FFF2D)
ba71f000 ba72a000 fdc fdc.sys unavailable (FFFFFFFE)
ba72f000 ba738000 watchdog watchdog.sys Sat Feb 17 01:11:45 2007 (45D69CA1)
ba8cf000 ba8da000 kbdclass kbdclass.sys Sat Feb 17 01:05:39 2007 (45D69B33)
ba8ef000 ba8f8000 kbdhid kbdhid.sys Sat Feb 17 01:05:42 2007 (45D69B36)
ba8ff000 ba90a000 Msfs Msfs.SYS Sat Feb 17 00:50:33 2007 (45D697A9)
ba90f000 ba91e000 termdd termdd.sys Sat Feb 17 00:44:32 2007 (45D69640)
ba92f000 ba938000 raspti raspti.sys Sat Feb 17 00:59:23 2007 (45D699BB)
ba95f000 ba96c000 netbios netbios.sys Sat Feb 17 00:58:29 2007 (45D69985)
ba96f000 ba97a000 ptilink ptilink.sys Sat Feb 17 01:06:38 2007 (45D69B6E)
ba9c8000 ba9cf000 dxgthk dxgthk.sys Tue Mar 25 03:05:52 2003 (3E7FFFD0)
ba9f8000 ba9ff000 Null Null.SYS unavailable (FFFFFFFE)
baafa000 bab02000 rasacd rasacd.sys Tue Mar 25 03:11:50 2003 (3E800136)
bab12000 bab19d80 usbccgp usbccgp.sys Sat Feb 17 01:13:08 2007 (45D69CF4)
bab2a000 bab32000 mnmdd mnmdd.SYS Tue Mar 25 03:07:53 2003 (3E800049)
baef0000 baefe000 imapi imapi.sys Sat Feb 17 01:08:22 2007 (45D69BD6)
baf30000 baf3b000 TDI TDI.SYS Sat Feb 17 01:01:19 2007 (45D69A2F)
baf50000 baf5c000 vga vga.sys Sat Feb 17 01:10:30 2007 (45D69C56)
baf60000 baf6a000 ndisuio ndisuio.sys Sat Feb 17 00:58:25 2007 (45D69981)
bf800000 bf9d0000 win32k win32k.sys Mon Feb 09 06:30:06 2009 (499013BE)
bf9d0000 bf9e7000 dxg dxg.sys Sat Feb 17 01:14:39 2007 (45D69D4F)
bf9e7000 bfa2a000 ati2dvag ati2dvag.dll Wed May 02 22:27:36 2007 (46394898)
bfa2a000 bfa69000 ati2cqag ati2cqag.dll Wed May 02 21:31:46 2007 (46393B82)
bfa69000 bfa9f000 atikvmag atikvmag.dll Wed May 02 21:55:39 2007 (4639411B)
bff50000 bff58000 TSDDD TSDDD.dll Tue Mar 25 16:12:05 2003 (3E80B815)
bff60000 bff7e000 RDPDD RDPDD.dll Sat Feb 17 09:01:19 2007 (45D70AAF)
bffa0000 bffea000 ATMFD ATMFD.DLL Sat Feb 17 08:59:31 2007 (45D70A43)
f7208000 f722e000 KSecDD KSecDD.sys Sat Feb 17 00:46:32 2007 (45D696B8)
f722e000 f7253000 fltMgr fltMgr.sys Sat Feb 17 00:51:08 2007 (45D697CC)
f7253000 f7266000 CLASSPNP CLASSPNP.SYS Sat Feb 17 01:28:16 2007 (45D6A080)
f7266000 f7283000 atapi atapi.sys Sat Feb 17 01:07:34 2007 (45D69BA6)
f7283000 f72a6000 eqldsm eqldsm.sys Wed Jul 30 21:14:10 2008 (489111E2)
f72a6000 f72d0000 volsnap volsnap.sys Sat Feb 17 01:08:23 2007 (45D69BD7)
f72d0000 f72fc000 dmio dmio.sys Sat Feb 17 01:10:44 2007 (45D69C64)
f72fc000 f7323000 ftdisk ftdisk.sys Sat Feb 17 01:08:05 2007 (45D69BC5)
f7323000 f733c000 mpio mpio.sys Tue Nov 13 04:03:37 2007 (47396869)
f733c000 f7352000 pci pci.sys Sat Feb 17 00:59:03 2007 (45D699A7)
f7352000 f7386000 ACPI ACPI.sys Sat Feb 17 00:58:47 2007 (45D69997)
f7487000 f7490000 WMILIB WMILIB.SYS Tue Mar 25 03:13:00 2003 (3E80017C)
f7497000 f74a6000 isapnp isapnp.sys Sat Feb 17 00:58:57 2007 (45D699A1)
f74a7000 f74b4000 PCIIDEX PCIIDEX.SYS Sat Feb 17 01:07:32 2007 (45D69BA4)
f74b7000 f74c7000 MountMgr MountMgr.sys Sat Feb 17 01:05:35 2007 (45D69B2F)
f74c7000 f74d2000 PartMgr PartMgr.sys Sat Feb 17 01:29:25 2007 (45D6A0C5)
f74d7000 f74e3000 msiscdsm msiscdsm.sys Fri Nov 23 01:03:53 2007 (47466D49)
f74e7000 f74f7000 disk disk.sys Sat Feb 17 01:07:51 2007 (45D69BB7)
f74f7000 f7503000 Dfs Dfs.sys Sat Feb 17 00:51:17 2007 (45D697D5)
f7507000 f7510000 mpspfltr mpspfltr.sys Tue Nov 13 04:03:37 2007 (47396869)
f7517000 f7521000 crcdisk crcdisk.sys Sat Feb 17 01:09:50 2007 (45D69C2E)
f7587000 f7594000 Npfs Npfs.SYS Sat Feb 17 00:50:36 2007 (45D697AC)
f75c7000 f75d5000 msgpc msgpc.sys Sat Feb 17 00:58:37 2007 (45D6998D)
f75d7000 f75e0000 dump_WMILIB dump_WMILIB.SYS Tue Mar 25 03:13:00 2003 (3E80017C)
f75e7000 f75f1000 serenum serenum.sys Sat Feb 17 01:06:44 2007 (45D69B74)
f7647000 f7656000 raspppoe raspppoe.sys Sat Feb 17 00:59:23 2007 (45D699BB)
f7677000 f7686000 intelppm intelppm.sys Sat Feb 17 00:48:30 2007 (45D6972E)
f76a7000 f76b0000 ndistapi ndistapi.sys Sat Feb 17 00:59:19 2007 (45D699B7)
f76b7000 f76c1000 Dxapi Dxapi.sys Tue Mar 25 03:06:01 2003 (3E7FFFD9)
f76c8000 f7707000 NDIS NDIS.sys Sat Feb 17 01:28:49 2007 (45D6A0A1)
f7707000 f770f000 kdcom kdcom.dll Tue Mar 25 03:08:00 2003 (3E800050)
f770f000 f7717000 BOOTVID BOOTVID.dll Tue Mar 25 03:07:58 2003 (3E80004E)
f7717000 f771e000 pciide pciide.sys Tue Mar 25 03:04:46 2003 (3E7FFF8E)
f771f000 f7726000 dmload dmload.sys unavailable (FFFFFFFE)
f7727000 f772e000 mpdev mpdev.sys Tue Nov 13 04:03:37 2007 (47396869)
f7747000 f774f000 RDPCDD RDPCDD.sys Tue Mar 25 03:03:05 2003 (3E7FFF29)
f7777000 f777f000 Fs_Rec Fs_Rec.SYS Tue Mar 25 03:08:36 2003 (3E800074)
f77a7000 f77ae000 Beep Beep.SYS Tue Mar 25 03:03:04 2003 (3E7FFF28)
f77cf000 f77d7000 mouhid mouhid.sys Tue Mar 25 03:03:12 2003 (3E7FFF30)
f77e7000 f77ed300 HIDPARSE HIDPARSE.SYS Sat Feb 17 01:12:35 2007 (45D69CD3)
f77ef000 f77f4180 usbuhci usbuhci.sys Sat Feb 17 01:13:02 2007 (45D69CEE)
f77ff000 f7805b80 usbehci usbehci.sys Sat Feb 17 01:12:56 2007 (45D69CE8)
f7817000 f781f000 audstub audstub.sys Tue Mar 25 03:09:12 2003 (3E800098)
f7878000 f7897000 Mup Mup.sys Sat Feb 17 01:27:41 2007 (45D6A05D)
f79c1000 f79c24c0 BASFND BASFND.sys Thu Apr 24 19:16:49 2003 (3EA87061)
f79d7000 f79d8280 swenum swenum.sys Sat Feb 17 01:05:56 2007 (45D69B44)
f79e1000 f79e2580 USBD USBD.SYS Tue Mar 25 03:10:39 2003 (3E8000EF)
f7b4a000 f7bdf000 Ntfs Ntfs.sys Sat Feb 17 01:27:23 2007 (45D6A04B)

Unloaded modules:
b623e000 b63c4000 navex15.sys
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 00186000
b6229000 b623e000 naveng.sys
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 00015000
b6f1c000 b6f31000 naveng.sys
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 00015000
b6f31000 b70b7000 navex15.sys
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 00186000
b8a09000 b8a1e000 naveng.sys
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 00015000
b8a1e000 b8ba4000 navex15.sys
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 00186000
b9f10000 b9f25000 naveng.sys
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 00015000
b9f25000 ba0ab000 navex15.sys
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 00186000
b9ee6000 b9ef9000 i8042prt.SYS
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 00013000
f776f000 f7777000 Sfloppy.SYS
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 00008000
f7627000 f7631000 Flpydisk.SYS
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 0000A000
Closing open log file c:debuglog.txt

 

Hi,
OK we ran the Dell memory check and didn't come up with anything and there are no single bit errors. It looks like the backup/snapshot issue may have been it. I had them install all the cluster and storport patches which were out of date as well, rescheduled things a bit and have not had another issue. I appreciate the input.

Justen