Bugcheck - UNEXPECTED_KERNEL_MODE_TRAP (7f)

Can anyone please help with this? I have a Dell 1650 running Windows Server 2003 Standard Ed. and for the past 3 months we have been getting BSODs at different points in time - there seems to be no pattern. This is a PRODUCTION server that serves nearly 1000 users and when it crashes the "stuff hits the fan ". We believe it is a driver issue - but we are unable to determine which driver is causing it based on the bugcheck anaysis. PLEASE HELP! Thanks so much in advance!

P.S. I know that I'm having "SYMBOL" problems....not sure how to get the SYMBOLS??


2: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

UNEXPECTED_KERNEL_MODE_TRAP_M (1000007f)
This means a trap occurred in kernel mode, and it's a trap of a kind
that the kernel isn't allowed to have/catch (bound trap) or that
is always instant death (double fault). The first number in the
bugcheck parens is the number of the trap (8 = double fault, etc)
Consult an Intel x86 family manual to learn more about what these
traps are. Here is a *portion* of those codes:
If kv shows a taskGate
use .tss on the part before the colon, then kv.
Else if kv shows a trapframe
use .trap on that value
Else
.trap on the appropriate frame will show where the trap was taken
(on x86, this will be the ebp that goes with the procedure KiTrap)
Endif
kb will then show the corrected stack.
Arguments:
Arg1: 00000008, EXCEPTION_DOUBLE_FAULT
Arg2: f7737ef0
Arg3: 00000000
Arg4: 00000000

Debugging Details:
------------------

***** Kernel symbols are WRONG. Please fix symbols to do analysis.

*************************************************************************
*** ***
*** ***
*** Your debugger is not using the correct symbols ***
*** ***
*** In order for this command to work properly, your symbol path ***
*** must point to .pdb files that have full type information. ***
*** ***
*** Certain .pdb files (such as the public OS symbols) do not ***
*** contain the required information. Contact the group that ***
*** provided you with these symbols if you need this command to ***
*** work. ***
*** ***
*** Type referenced: nt!_KPRCB ***
*** ***
*************************************************************************

BUGCHECK_STR: 0x7f_8

CUSTOMER_CRASH_COUNT: 1

DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP

LAST_CONTROL_TRANSFER: from 804e3752 to 805070ec

STACK_TEXT:
WARNING: Stack unwind information not available. Following frames may be wrong.
b64f8038 804e3752 b64f80e8 00000000 b64f805c nt+0x290ec
b64f804c 804dfd24 b64f80e8 00000000 b64f8078 nt+0x5752
b64f805c b64f8078 00000000 00000000 00000000 nt+0x1d24
b64f8078 804e9f76 b64f83b4 b64f891c b64f8000 0xb64f8078
b64f809c 804e9f2b 0f000100 b64f8a2c b64f0030 nt+0xbf76
b64f8460 b6f87884 b64f8a2c b6f87884 00000000 nt+0xbf2b
b64f8488 b6f8798f b64f8a2c b64f84ac 00000000 SYMTDI+0x2e884
b64f84ac 804e9f76 b64f8948 b64f8a2c b64f8598 SYMTDI+0x2e98f
b64f84d0 804e9f2b b64f8948 b64f8a2c b64f8598 nt+0xbf76
b64f857c 8051866b b64f8948 b64f8598 0ffd0000 nt+0xbf2b
b64f892c 804e086a b64f8948 00000000 b64f899c nt+0x3a66b
b64f899c 33353631 000a027e 000e03a5 00000000 nt+0x286a
38383638 00000000 00000000 00000000 00000000 0x33353631


FOLLOWUP_IP:
SYMTDI+2e884
b6f87884 ?? ???

SYMBOL_STACK_INDEX: 6

FOLLOWUP_NAME: MachineOwner

SYMBOL_NAME: SYMTDI+2e884

MODULE_NAME: SYMTDI

IMAGE_NAME: SYMTDI.SYS

DEBUG_FLR_IMAGE_TIMESTAMP: 4050ed2d

STACK_COMMAND: kb

BUCKET_ID: WRONG_SYMBOLS

Followup: MachineOwner
---------

 

Follow up: I ran a bugcheck analysis against "memory.dmp" this time rather than "Mini110504-01.dmp" and I also modified my entry for Symbol path - so maybe this will provide some more light on my situation. Here is the analysis:

==========================================================
Microsoft (R) Windows Debugger Version 6.3.0017.0
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\WINDOWS\MEMORY.DMP]
Kernel Complete Dump File: Full address space is available

Symbol search path is: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols;C:\WINDOWS\Symbols
Executable search path is:
Windows Server 2003 Kernel Version 3790 MP (4 procs) Free x86 compatible
Product: Server, suite: TerminalServer SingleUserTS
Built by: 3790.srv03_gdr.040410-1234
Kernel base = 0x804de000 PsLoadedModuleList = 0x8057b6a8
Debug session time: Fri Nov 05 11:39:35 2004
System Uptime: 2 days 1:53:55.187
WARNING: Process directory table base 06C40000 doesn't match CR3 00039000
WARNING: Process directory table base 06C40000 doesn't match CR3 00039000
Loading Kernel Symbols
..............................................................................................................
Loading unloaded module list
........
Loading User Symbols
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 7F, {8, f7737ef0, 0, 0}

*** ERROR: Symbol file could not be found. Defaulted to export symbols for SYMTDI.SYS -
*** ERROR: Module load completed but symbols could not be loaded for kshlpr.SYS
*** ERROR: Symbol file could not be found. Defaulted to export symbols for ksocks.SYS -
*** ERROR: Module load completed but symbols could not be loaded for mcfsrdr.SYS
*** ERROR: Module load completed but symbols could not be loaded for savrt.sys
Probably caused by : SYMTDI.SYS ( SYMTDI!DebugBreakEx+d0 )

Followup: MachineOwner
---------

2: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

UNEXPECTED_KERNEL_MODE_TRAP (7f)
This means a trap occurred in kernel mode, and it's a trap of a kind
that the kernel isn't allowed to have/catch (bound trap) or that
is always instant death (double fault). The first number in the
bugcheck params is the number of the trap (8 = double fault, etc)
Consult an Intel x86 family manual to learn more about what these
traps are. Here is a *portion* of those codes:
If kv shows a taskGate
use .tss on the part before the colon, then kv.
Else if kv shows a trapframe
use .trap on that value
Else
.trap on the appropriate frame will show where the trap was taken
(on x86, this will be the ebp that goes with the procedure KiTrap)
Endif
kb will then show the corrected stack.
Arguments:
Arg1: 00000008, EXCEPTION_DOUBLE_FAULT
Arg2: f7737ef0
Arg3: 00000000
Arg4: 00000000

Debugging Details:
------------------


BUGCHECK_STR: 0x7f_8

TSS: 00000028 -- (.tss 28)
eax=b64f80e8 ebx=862bfb00 ecx=0000003d edx=00000001 esi=00000000 edi=b64f805c
eip=805070ec esp=b64f8000 ebp=b64f8038 iopl=0 nv up ei pl zr na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010246
nt!KiContinue+0x5d:
805070ec 50 push eax
Resetting default scope

DEFAULT_BUCKET_ID: DRIVER_FAULT

CURRENT_IRQL: 1

LAST_CONTROL_TRANSFER: from 804e3752 to 805070ec

STACK_TEXT:
b64f8038 804e3752 b64f80e8 00000000 b64f805c nt!KiContinue+0x5d
b64f804c 804dfd24 b64f80e8 00000000 b64f8078 nt!NtContinue+0x22
b64f804c 804ea971 b64f80e8 00000000 b64f8078 nt!KiSystemService+0xd0
b64f80cc 8050a270 b64f80e8 00000000 b64f8a2c nt!ZwContinue+0x11
b64f8460 b6f87884 b64f8a2c b6f87884 00000000 nt!RtlUnwind+0xa0
WARNING: Stack unwind information not available. Following frames may be wrong.
b64f8488 b6f8798f b64f8a2c b64f84ac 00000000 SYMTDI!DebugBreakEx+0xd0
b64f84ac 804e9f76 b64f8948 b64f8a2c b64f8598 SYMTDI!DebugBreakEx+0x1db
b64f84d0 804e9f2b b64f8948 b64f8a2c b64f8598 nt!ExecuteHandler2+0x26
b64f857c 8051866b b64f8948 b64f8598 0ffd0000 nt!ExecuteHandler+0x24
b64f892c 804e086a b64f8948 00000000 b64f899c nt!KiDispatchException+0xb1
b64f8994 804e0802 38383638 33353631 000a027e nt!CommonDispatchException+0x4a
b64f8a3c b6f71d8a 0ffd0000 00000040 b64f8e44 nt!Kei386EoiHelper+0x182
b64f8a9c b6f710a5 84633888 b64f8e44 81828008 SYMTDI!dereferenceModule+0x3f2e
b64f8be8 b6f70f7a 84258ce0 00000000 b64f8bfc SYMTDI!dereferenceModule+0x3249
b64f8c00 b6f6c495 84258ce0 b64f8e44 81828008 SYMTDI!dereferenceModule+0x311e
b64f8c44 b6f837aa b64f8e74 b6f82a01 b64f8c64 SYMTDI!ACMRegisterFilterModule+0x1d7d
b64f8c4c b6f82a01 b64f8c64 8182807b ff0f03a5 SYMTDI!rHeapFree+0x3e42
b64f8e74 b6f81177 85d84950 00000000 00000000 SYMTDI!rHeapFree+0x3099
b64f8e94 b6f810e5 85d84950 85d84950 b64f8ed4 SYMTDI!rHeapFree+0x180f
b64f8ea4 804f0154 86556b18 81828008 85d84950 SYMTDI!rHeapFree+0x177d
b64f8ed4 b6fa2e5b 8182809c 85e9d790 862c4268 nt!IopfCompleteRequest+0xa0
b64f8f04 804f0473 8609a460 81828008 818280c0 tcpip!TCPDispatch+0x11a
b64f8f14 b6f8122b 850fd770 85e88a20 81828078 nt!IofCallDriver+0x3f
b64f8f30 b6f80b13 85d84950 00000000 00000000 SYMTDI!rHeapFree+0x18c3
b64f8f58 b6f8278a 86556b18 81828008 86556b18 SYMTDI!rHeapFree+0x11ab
b64f8f6c 804f0473 86556b18 81828008 81828008 SYMTDI!rHeapFree+0x2e22
b64f8f7c 8058f898 8609a448 8605d7b4 b64f9130 nt!IofCallDriver+0x3f
b64f9074 8058e482 8609a460 00000000 8605d710 nt!IopParseDevice+0xab5
b64f90f0 8058dbb9 00000000 b64f9130 00000040 nt!ObpLookupObjectName+0x545
b64f9144 8058faa9 00000000 00000000 01003100 nt!ObOpenObjectByName+0xe8
b64f91c0 8058fb7d 863b4988 c0100000 b64f9330 nt!IopCreateFile+0x413
b64f920c 8058e28e 863b4988 c0100000 b64f9330 nt!IoCreateFile+0x3d
b64f924c 804dfd24 863b4988 c0100000 b64f9330 nt!NtCreateFile+0x2e
b64f924c 804eaa21 863b4988 c0100000 b64f9330 nt!KiSystemService+0xd0
b64f92f0 ba62a3fa 863b4988 c0100000 b64f9330 nt!ZwCreateFile+0x11
b64f9354 ba62b2ff 8600f930 00000070 855ab39c kshlpr+0x13fa
b64f936c ba62b271 8600f930 8600fa0c 855ab39c kshlpr+0x22ff
b64f9404 ba478abd 84e8f2d0 00012000 84c893f8 kshlpr+0x2271
b64f9470 ba4795c8 00001740 84f261b8 00000010 ksocks!bind+0x1cd
b64f9504 ba475a32 00001740 b64f9628 00000010 ksocks!connect+0x2c9
b64f9644 ba4760be 00000002 0000062c e61b07e0 ksocks+0xa32
b64f9690 b6b2e373 8443fc00 00000000 e348a000 ksocks!gethostbyname+0xdc
b64f97d8 b6b2e280 8443fc00 0000157d e52f44d0 mcfsrdr+0x2a373
b64f980c b6b2df5c 8443fc00 0000157d e52f44d0 mcfsrdr+0x2a280
b64f9834 b6b2dea3 8443fc00 0000157d e52f44d0 mcfsrdr+0x29f5c
b64f9860 b6b27a54 b64f9ae8 e52f44d0 e348a000 mcfsrdr+0x29ea3
b64f9ac0 b6b0c0a8 b64f9ae8 b64fa078 00000000 mcfsrdr+0x23a54
b64f9b34 b6b0be33 8441cdf0 85eda550 b64fa054 mcfsrdr+0x80a8
b64f9b98 b6b0abec 8441cdf0 85eda550 b64fa054 mcfsrdr+0x7e33
b64f9bec b6b0d2a7 8576a000 8441cdf0 85e0a398 mcfsrdr+0x6bec


FOLLOWUP_IP:
SYMTDI!DebugBreakEx+d0
b6f87884 5d pop ebp

SYMBOL_STACK_INDEX: 5

FOLLOWUP_NAME: MachineOwner

SYMBOL_NAME: SYMTDI!DebugBreakEx+d0

MODULE_NAME: SYMTDI

IMAGE_NAME: SYMTDI.SYS

DEBUG_FLR_IMAGE_TIMESTAMP: 4050ed2d

STACK_COMMAND: .tss 28 ; kb

BUCKET_ID: 0x7f_8_SYMTDI!DebugBreakEx+d0

Followup: MachineOwner
---------

 

Bugcheck dump

First you need to laod the correct symbols for windbg to verify against. You should check out http://www.microsoft.com/whdc/devtools/debugging/symbols.mspx
which tells you how to get the symbols from the microsoft site.

I also have a bugcheck problem but is different then yours and found many articles, but nothing definitive. I do remember seeing a few sites refrencing Double Faults and seems to point to driver conflicts. Don't quote me but from your dump it loos like SYMTDI.SYS may be the issue which is Norton. You can try Last Known Good Configuration. If that dosen't work you can try disabling antivirus and see if it happens again. (The second option is risky; no av). You should also see your recovery options not to automatacally reboot so you can get the BSOD message and see if it gives any more info.

Hope this helps

 

Yeah, I had just found this out before you posted that message. I got the symbols to work properly for my 2nd post (2nd bugcheck analysis). That would be incredibly interesting if Symantec was causing the problems. I would have to doubt that at this point just because we have a number of other Win2k3 servers running the same version of SAVCE - regardless, I will investigate that possibility. Thanks for your input! Anybody else have an idea???

 

SYMTDI.SYS threw an exception while the system was handling a previous exception(also thrown by symtdi). This is a BadThing and will cause a blue screen, as you have seen.

Your fastest path to stability is to disable the software associated with this driver, or preferably remove it. Note that since this is related to virus scanning, you may be able to setup for 'remote scanning' of this machine, by using another box to scan a network drive mapped to the server.

It appears this driver is dated Thu Mar 11 17:50:21 2004, is there an update availible? Thats certainly worth a shot.

You will need to contact symantec for further assistance. The nature of the fault is unable to be determined without symbols and source code (which your vendor has).

 

I got in touch with Symantec TechSupport and had them give me the latest and greatest version of SAVCE (9.0.2.1000). I was running version 9.0.0.388. So I'll uninstall the old and install the new and we'll see what happens. I'll keep the board posted for all those watching this thread. Thanks for the support guys!

 

Just to keep those watching this thread updated --

After MANY tech support calls to Symantec I finally got this problem kicked. The problem (simplified) was that there was a program or protocol hogging up all the kernel stack memory and when symantec would attempt to access some kernel memory to scan ahead of what you wanted to access it would blue screen. We believe the MAIN culprit was our SAN driver. This server is connected to a SAN by a Qlogic HBA with assorted software related to managing the LUNs on the SAN. So I uninstalled some unecessary SAN software and made a registry modification for Symantec which basically told Symantec that if there wasn't X amount of kernel memory available then DO NOT real-time scan. This leaves us a little open for virus vulnerability but so far not ONE crash since. We made some changes to our SAN setup so I will attempt removing the reg mod to see if it still remains stable. All-in-All - problem fixed*

 

Thanks for following up on that! I didn't think to check how far back that SANergy stack went. Curious that we didnt throw a stack overflow.. Ahh well. I'm glad you found a solution.