Dump Data: pool_corruption

Hi, I have a server that has bugchecked a couple of times and there isnt anything meaningful in the System or Application logs. I have the bugcheck report included in this post. If anyone has the time to look at it an offer up a suggestion, I would be most appreciative.

Thanks,

Desiree
-------------------------------------------------

Symbol search path is: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols

Microsoft (R) Windows Debugger Version 6.3.0017.0
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Data\athena\MEMORY.DMP]
Kernel Complete Dump File: Full address space is available

************************************************************
WARNING: Dump file has been truncated. Data may be missing.
************************************************************
Symbol search path is: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 2000 Kernel Version 2195 (Service Pack 4) MP (4 procs) Free x86 compatible
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Kernel base = 0x80400000 PsLoadedModuleList = 0x80484b80
Debug session time: Fri Jan 07 12:22:35 2005
System Uptime: 0 days 22:23:48.531
Loading Kernel Symbols
....................................................................................................
Loading unloaded module list
....
Loading User Symbols
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 24, {19025e, f5c5bb1c, f5c5b774, 8046d0f2}

Probably caused by : Pool_Corruption ( nt!ExFreePool+b )

Followup: Pool_corruption
---------

0: kd> !analyze -v;r;kv;lmtn
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

NTFS_FILE_SYSTEM (24)
If you see NtfsExceptionFilter on the stack then the 2nd and 3rd
parameters are the exception record and context record. Do a .cxr
on the 3rd parameter and then kb to obtain a more informative stack
trace.
Arguments:
Arg1: 0019025e
Arg2: f5c5bb1c
Arg3: f5c5b774
Arg4: 8046d0f2

Debugging Details:
------------------


EXCEPTION_RECORD: f5c5bb1c -- (.exr fffffffff5c5bb1c)
ExceptionAddress: 8046d0f2 (nt!ExFreePoolWithTag+0x00000162)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000000
Parameter[1]: 00000000
Attempt to read from address 00000000

CONTEXT: f5c5b774 -- (.cxr fffffffff5c5b774)
eax=00000000 ebx=00000000 ecx=00000001 edx=e8744250 esi=e9e2b708 edi=00000000
eip=8046d0f2 esp=f5c5bbe4 ebp=f5c5bc04 iopl=0 nv up ei pl zr na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010246
nt!ExFreePoolWithTag+0x162:
8046d0f2 f60701 test byte ptr [edi],0x1 ds:0023:00000000=??
Resetting default scope

DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0x24

LAST_CONTROL_TRANSFER: from 8046cf0d to 8046d0f2

STACK_TEXT:
f5c5bc04 8046cf0d e9e2b710 00000000 80416482 nt!ExFreePoolWithTag+0x162
f5c5bc10 80416482 e9e2b710 8046cf02 f5c5bd4c nt!ExFreePool+0xb
f5c5bc24 bfea3d7e bfe4d760 e9e2b710 e8744008 nt!ExFreeToPagedLookasideList+0x52
f5c5bc3c bfe52bb7 e8744008 f5c5bd4c f5c5bd58 Ntfs!NtfsDeleteCcb+0xae
f5c5bcc4 bfe55efa 8662ec48 e87440d8 e8744008 Ntfs!NtfsCommonClose+0x425
f5c5bd78 804176b5 00000000 00000000 00000000 Ntfs!NtfsFspClose+0x14c
f5c5bda8 804565fc 00000000 00000000 00000000 nt!ExpWorkerThread+0xaf
f5c5bddc 8046b6a6 80417606 00000000 00000000 nt!PspSystemThreadStartup+0x54
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16


FOLLOWUP_IP:
nt!ExFreePool+b
8046cf0d c20400 ret 0x4

SYMBOL_STACK_INDEX: 1

FOLLOWUP_NAME: Pool_corruption

SYMBOL_NAME: nt!ExFreePool+b

MODULE_NAME: Pool_Corruption

IMAGE_NAME: Pool_Corruption

DEBUG_FLR_IMAGE_TIMESTAMP: 0

STACK_COMMAND: .cxr fffffffff5c5b774 ; kb

BUCKET_ID: 0x24_nt!ExFreePool+b

Followup: Pool_corruption
---------

eax=ffdff13c ebx=00000024 ecx=c00000d8 edx=8046bd36 esi=8662ec48 edi=c0000005
eip=bfe42393 esp=f5c5b648 ebp=c0000188 iopl=0 nv up ei ng nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00000286
Ntfs!NtfsExceptionFilter+0xcf:
bfe42393 806605fe and byte ptr [esi+0x5],0xfe ds:0023:8662ec4d=00
ChildEBP RetAddr Args to Child
f5c5b668 bfe55f5c 8662ec48 f5c5b694 8046181b Ntfs!NtfsExceptionFilter+0xcf (FPO: [EBP 0xf5c5bd78] [2,0,4])
f5c5bd78 804176b5 00000000 00000000 00000000 Ntfs!NtfsFspClose+0x1ae (FPO: [Non-Fpo])
f5c5bda8 804565fc 00000000 00000000 00000000 nt!ExpWorkerThread+0xaf (FPO: [Non-Fpo])
f5c5bddc 8046b6a6 80417606 00000000 00000000 nt!PspSystemThreadStartup+0x54 (FPO: [Non-Fpo])
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16
start end module name
80062000 80076460 hal halmacpi.dll Thu Mar 20 21:04:42 2003 (3E7A733A)
80400000 805a0340 nt ntkrnlmp.exe Wed Oct 20 23:55:33 2004 (41773335)
a0000000 a018f000 win32k win32k.sys unavailable (FFFFFFFE)
a018f000 a01b0000 atidrab atidrab.dll unavailable (FFFFFFFE)
be42e000 be443f00 RDPWD RDPWD.SYS Wed Jan 21 14:50:25 2004 (400ED801)
be896000 be8a5a20 ipsec ipsec.sys Tue Apr 29 19:04:59 2003 (3EAF051B)
be94e000 be956a60 termdd termdd.sys Fri Mar 21 16:43:08 2003 (3E7B876C)
be9ee000 bea29bc0 srv srv.sys Tue Apr 29 19:05:07 2003 (3EAF0523)
bea7a000 bea9f920 sfmsrv sfmsrv.sys Mon Sep 09 20:17:41 2002 (3D7D3A25)
bed08000 bed10240 Fips Fips.SYS Tue May 09 11:28:29 2000 (39182E9D)
bed70000 bed8d4a0 afd afd.sys Wed Apr 30 04:45:29 2003 (3EAF8D29)
bedb6000 bedda2a0 sfmatalk sfmatalk.sys Fri Aug 16 08:28:12 2002 (3D5CEFDC)
bf743000 bf7a6e40 mrxsmb mrxsmb.sys Mon Nov 01 00:24:54 2004 (4185C8A6)
bf7e1000 bf80a760 rdbss rdbss.sys Fri Oct 15 17:03:40 2004 (41703B2C)
bf80b000 bf81b760 naveng naveng.sys Thu Sep 30 21:59:17 2004 (415CB9F5)
bf81c000 bf8b4680 navex15 navex15.sys Thu Sep 30 22:11:15 2004 (415CBCC3)
bf8b5000 bf8c7f00 SYMEVENT SYMEVENT.SYS Wed Jan 14 21:02:13 2004 (4005F4A5)
bf8c8000 bf917000 savrt savrt.sys Mon Feb 09 18:24:30 2004 (402816AE)
bf917000 bf93ee00 netbt netbt.sys Wed Jul 16 15:44:26 2003 (3F15AB1A)
bf93f000 bf990060 tcpip tcpip.sys Tue Apr 29 19:05:31 2003 (3EAF053B)
bf991000 bf9917c0 AlKernel AlKernel.sys Fri Aug 24 18:31:42 2001 (3B86D5CE)
bfa2f000 bfa32d40 dump_cpqcissm dump_cpqcissm.sys Mon Aug 18 14:54:24 2003 (3F4120E0)
bfba5000 bfbcf3a0 update update.sys Wed Apr 16 00:22:01 2003 (3E9CDA69)
bfbd0000 bfbefd00 ks ks.sys Wed Dec 04 12:09:38 2002 (3DEE36D2)
bfc02000 bfc25060 rdpdr rdpdr.sys Fri Mar 21 16:43:14 2003 (3E7B8772)
bfc26000 bfc3cba0 ndiswan ndiswan.sys Tue Apr 29 19:05:01 2003 (3EAF051D)
bfc3d000 bfc5d700 cpqteam cpqteam.sys Tue Aug 05 18:38:31 2003 (3F3031E7)
bfc5e000 bfc8c3a0 q57w2k q57w2k.sys Sat Jun 19 19:27:01 2004 (40D4CBC5)
bfc8d000 bfc9e6c0 atimpab atimpab.sys Wed Nov 10 18:34:06 1999 (382A00EE)
bfca7000 bfcaa580 vga vga.sys Sat Sep 25 14:37:40 1999 (37ED1674)
bfcc7000 bfd0b500 cpqasm2 cpqasm2.sys Mon Oct 13 19:25:56 2003 (3F8B3484)
bfd68000 bfd6b6c0 dump_diskdump dump_diskdump.sys Tue Feb 25 14:18:04 2003 (3E5BC16C)
bfd88000 bfd8b640 serenum serenum.sys Wed Jan 15 14:47:01 2003 (3E25BAB5)
bfda4000 bfda7e60 TDI TDI.SYS Wed Jan 15 14:56:26 2003 (3E25BCEA)
bfdb0000 bfdb22e0 ndistapi ndistapi.sys Wed Jan 15 14:54:15 2003 (3E25BC67)
bfde4000 bfdf9640 Mup Mup.sys Wed Jan 15 14:54:01 2003 (3E25BC59)
bfdfa000 bfe0e5a0 RAIDISK RAIDISK.sys Thu Mar 13 16:21:53 2003 (3E70F671)
bfe0f000 bfe38aa0 NDIS NDIS.sys Tue Apr 29 19:05:01 2003 (3EAF051D)
bfe39000 bfeb6800 Ntfs Ntfs.sys Wed Jun 04 18:11:33 2003 (3EDE6E95)
bfeb7000 bfec87c0 KSecDD KSecDD.sys Sat Sep 20 20:32:19 2003 (3F6CF193)
bfec9000 bfedb1c0 Dfs Dfs.sys Tue Feb 11 21:19:06 2003 (3E49AF1A)
bfedc000 bff47aa0 hp2300 hp2300.sys Mon Aug 04 20:01:26 2003 (3F2EF3D6)
bff48000 bff5a0c0 SCSIPORT SCSIPORT.SYS Fri May 16 21:11:02 2003 (3EC58C26)
bff5b000 bff82140 lsicsb6 lsicsb6.sys Thu Jul 31 16:38:06 2003 (3F297E2E)
bff83000 bff98180 atapi atapi.sys Tue Apr 01 13:08:25 2003 (3E89D599)
bff99000 bffba9c0 dmio dmio.sys Wed Jan 15 14:47:04 2003 (3E25BAB8)
bffbb000 bffd7220 ftdisk ftdisk.sys Mon Mar 31 17:21:58 2003 (3E88BF86)
bffd8000 bffffc20 ACPI ACPI.sys Wed Jan 15 14:44:22 2003 (3E25BA16)
f5800000 f580e6a0 pci pci.sys Wed Jan 15 14:44:07 2003 (3E25BA07)
f5810000 f581b680 isapnp isapnp.sys Wed Jan 15 14:43:47 2003 (3E25B9F3)
f5820000 f582faa0 adpu160m adpu160m.sys Wed Jan 15 14:42:27 2003 (3E25B9A3)
f5830000 f58390e0 symmpi symmpi.sys Tue Dec 10 12:31:28 2002 (3DF624F0)
f5840000 f5848700 CLASSPNP CLASSPNP.SYS Wed Jan 15 14:42:51 2003 (3E25B9BB)
f5900000 f590c4c0 VIDEOPRT VIDEOPRT.SYS Wed Jan 15 14:47:20 2003 (3E25BAC8)
f5910000 f591b680 i8042prt i8042prt.sys Wed Apr 16 00:00:59 2003 (3E9CD57B)
f5920000 f592ef80 CPQCISSE CPQCISSE.sys Mon Aug 04 15:57:42 2003 (3F2EBAB6)
f5930000 f593ca80 rasl2tp rasl2tp.sys Tue Apr 29 19:05:06 2003 (3EAF0522)
f5940000 f594bc40 raspptp raspptp.sys Wed May 14 19:47:00 2003 (3EC2D574)
f5950000 f595f400 serial serial.sys Wed Apr 16 00:19:39 2003 (3E9CD9DB)
f5970000 f5979be0 usbhub usbhub.sys Tue Mar 18 18:30:41 2003 (3E77AC21)
f59b0000 f59b9ce0 NDProxy NDProxy.SYS Thu Sep 30 19:25:35 1999 (37F3F16F)
f59c0000 f59ce180 Cdr4_2K Cdr4_2K.SYS Tue Sep 24 17:21:41 2002 (3D90D765)
f59d0000 f59d8fa0 Npfs Npfs.SYS Sat Oct 09 19:58:07 1999 (37FFD68F)
f59e0000 f59e8680 msgpc msgpc.sys Wed Jan 15 14:54:25 2003 (3E25BC71)
f59f0000 f59f81a0 netbios netbios.sys Tue Oct 12 15:34:19 1999 (38038D3B)
f5a00000 f5a10000 Savrtpel Savrtpel.sys Mon Feb 09 18:24:34 2004 (402816B2)
f5a80000 f5a85520 PCIIDEX PCIIDEX.SYS Tue Feb 25 13:31:08 2003 (3E5BB66C)
f5a88000 f5a8f4c0 MountMgr MountMgr.sys Tue Feb 10 14:47:53 2004 (40293569)
f5a90000 f5a96320 symc8xx symc8xx.sys Fri Mar 30 12:01:54 2001 (3AC4BC02)
f5a98000 f5a9d180 sym_hi sym_hi.sys Sat Sep 25 15:11:49 1999 (37ED1E75)
f5aa0000 f5aa7720 disk disk.sys Wed Jan 15 14:43:05 2003 (3E25B9C9)
f5ab0000 f5ab6a20 EFS EFS.SYS Wed Jan 15 14:46:55 2003 (3E25BAAF)
f5ac0000 f5ac48c0 TDTCP TDTCP.SYS Fri Mar 21 16:43:08 2003 (3E7B876C)
f5ac8000 f5acca20 CpqCiDrv CpqCiDrv.sys Fri Jul 11 11:32:03 2003 (3F0ED873)
f5ae0000 f5ae5ec0 kbdclass kbdclass.sys Thu Feb 20 11:37:30 2003 (3E55044A)
f5af0000 f5af5400 mouclass mouclass.sys Thu Feb 20 11:37:45 2003 (3E550459)
f5b00000 f5b06580 fdc fdc.sys Wed Jan 15 14:42:51 2003 (3E25B9BB)
f5b10000 f5b15fc0 openhci openhci.sys Fri Feb 28 19:28:59 2003 (3E5FFECB)
f5b28000 f5b2cfc0 USBD USBD.SYS Wed Jan 22 12:05:33 2003 (3E2ECF5D)
f5b30000 f5b35560 Cdralw2k Cdralw2k.SYS Tue Sep 24 17:20:49 2002 (3D90D731)
f5b40000 f5b45240 Msfs Msfs.SYS Tue Oct 26 19:21:32 1999 (3816377C)
f5b58000 f5b5fd00 wanarp wanarp.sys Fri Aug 16 08:25:01 2002 (3D5CEF1D)
f5b60000 f5b64400 ptilink ptilink.sys Wed Jan 15 14:47:15 2003 (3E25BAC3)
f5b70000 f5b740e0 raspti raspti.sys Fri Oct 08 16:45:10 1999 (37FE57D6)
f5c10000 f5c12a20 BOOTVID BOOTVID.dll Wed Nov 03 20:24:33 1999 (3820E051)
f5c14000 f5c16d00 PartMgr PartMgr.sys Wed Jan 15 14:43:07 2003 (3E25B9CB)
f5c18000 f5c1bfe0 symc810 symc810.sys Sat Sep 25 15:11:49 1999 (37ED1E75)
f5c1c000 f5c1fd40 cpqcissm cpqcissm.sys Mon Aug 18 14:54:24 2003 (3F4120E0)
f5c20000 f5c23360 cpqarry2 cpqarry2.sys Fri Oct 01 19:47:57 1999 (37F5482D)
f5d00000 f5d01d20 Diskperf Diskperf.sys Wed Feb 12 16:34:38 2003 (3E4ABDEE)
f5d02000 f5d03b80 dmload dmload.sys Wed Jan 15 14:47:06 2003 (3E25BABA)
f5d12000 f5d13ca0 Fs_Rec Fs_Rec.SYS Wed Jan 15 14:53:30 2003 (3E25BC3A)
f5d1a000 f5d1be40 rasacd rasacd.sys Sat Sep 25 14:41:23 1999 (37ED1753)
f5dc8000 f5dc8f80 WMILIB WMILIB.SYS Sat Sep 25 14:36:47 1999 (37ED163F)
f5dc9000 f5dc9b00 pciide pciide.sys Wed Jan 15 14:43:03 2003 (3E25B9C7)
f5df8000 f5df8a40 audstub audstub.sys Sat Sep 25 14:35:33 1999 (37ED15F5)
f5e02000 f5e03000 swenum swenum.sys Wed Dec 04 12:10:07 2002 (3DEE36EF)
f5e23000 f5e239e0 Null Null.SYS Sat Sep 25 14:34:58 1999 (37ED15D2)
f5e25000 f5e25ee0 Beep Beep.SYS Wed Oct 20 18:18:59 1999 (380E3FD3)
f5e27000 f5e27980 sysmgmt sysmgmt.sys Wed Jan 29 14:14:33 2003 (3E382819)
f5e28000 f5e28f80 mnmdd mnmdd.SYS Sat Sep 25 14:37:40 1999 (37ED1674)

Unloaded modules:
f5b08000 f5b0d000 Cdaudio.SYS
Timestamp: unavailable (00000000)
Checksum: 00000000
f5ae8000 f5aef000 Cdrom.SYS
Timestamp: unavailable (00000000)
Checksum: 00000000
f5ad0000 f5ad5000 Flpydisk.SYS
Timestamp: unavailable (00000000)
Checksum: 00000000
bf9d8000 bfa33000 dmboot.sys
Timestamp: unavailable (00000000)
Checksum: 00000000

 

I have posted how to do this 8 times, and have yet to see someone actually follow through on it.

You have pool corruption, a bad driver is corrupting memory. Windows has a thing built in to guard this behavior and catch the guy with his hand in the cookie jar.

Your next step is to enable special pool, wait for a new crash, get new crash analyzed.

Old post on this
http://www.windowsbbs.com/showpost.php?p=203512&postcount=7

Using the registry to enable special pool <---recommended proceedure using *
http://support.microsoft.com/kb/188831/EN-US/

How to enable driver verifier to guard special pool <--if you are chicken to use regkey
http://support.microsoft.com/kb/q244617/

More information, Look for 'Special Pool'
http://www.winntmag.com/Windows/Articles/ArticleID/7206/pg/2/2.html


Also, you have some old drivers on this box, are you current on your SSDs?
AlKernel.sys Fri Aug 24 18:31:42 2001
cpqarry2.sys Fri Oct 01 19:47:57 1999
symc810.sys Sat Sep 25 15:11:49 1999
sym_hi.sys Sat Sep 25 15:11:49 1999


PLEASE NOTE THIS WILL MAKE IT CRASH MORE, NOT LESS. IF this box does something important, i recommend you seek assistance from Microsoft, who can walk you through this process quicker than i intend to

 

laff @ Joe

 

I think they charge something like $35 and stay with you until it is cured.

 

4 way enterprise edition, thats why i suspect this box is more than someones desktop, we are talking professional support.. Currently 99$ for a web/email based incident. 245$ USD to talk to someone on the phone. 35$ gets you help with XP, and isnt going to get you to server support. Desktop support would just annihilate this install and get you back on your feet. Servers are more of a surgical process, it might take longer, but you barely notice the scar once your done. rofl.

 

<gulp> Ok, I'm going to do it. I will post back if anything interesting arises from it. The server is important, but easily recovered.

If you will allow me, I have one more question:

I understand the intent of this question above, I just can't quite figure out what "SSDs" stands for. Can you expand it for me?

Thank you for your response - I very much appreciate it.

My best,

Desiree

 

SSDs are what compaq used to call the system support disk carepackage CDs they sent out with the latest drivers every other month. I guess they dont use that term anymore or dont do that.

 

so whats the scoop?

 

alright, so maybe the customer is a tad bit more gun shy than I am Given the options, we decided to update the "SSDs" (thanks for the tip, by the way) and then move to the special pool tool as the next step if we get any more bugchecks on it. and wouldn't you know it, the darn server hasn't BSOD'd since. I missed out on a great opportunity to expand my working knowledge of deciphering hexidecimal data. By the way, I ordered the Windows Internals book like was recommended. Do you have any suggestions for study material on deciphering bugcheck errors? I would certainly appreciate any suggestions you can think of.

My best,

Desiree

 

Trust me, a shotgun driver update beats the stuffing out of crash and burn specialpool. Best possible outcome was the driver update magically made it go away. (and for those of you keeping score, 9 for 9 on not doing special pool )

You can query on joehobart and benmcdonald[ms] on this board, lots of writeups on dumps out there, some are annotated better than others. Read the debugger.chm file (theres a how to debug this stopcode for every possible bugcheck code in there). Go hang out on OSRs device driver site. Pick up one of those 'teach yourself C in 21 days' books and dabble in debugging your own programs using the ms debugger tools.
That Windows Internals book is probably your best bet for a bootstrap education first step. 90% of the dumps i read from here are just understanding that PAGE_FAULT_IN_NONPAGED_AREA actually means something in technical babble and knowing what the vectors of the data are. That book will teach you the foundational knowladge required to decipher what went wrong. Its probably a years worth of intense study. Has a lot of labs in there, debugging your own machine to see stacks, data structures, etc.

Beyond learning a little C, a touch of assembly(theres like 14 operators, its not hard) and windows architecture, the only thing left to get decent at it is experience and specilized component knowladge (ex: learning SCSI to debug scsi problems, learning networking/NDIS to debug nic problems, learning HTTP to debug wininet, etc..). Half the time i have no idea how the component works that im debugging, i just read the english in the stack and use MSDN to lookup stuff.

I could probably start annotating dumps with my thought process better if there were people interested in how to do it.