2003 Terminal Server reboots itself

I have read about similar problems to mine, like EssendonSTP (3 Jan 05), and Peli (26 April 05). Peli's is pretty much exactly the same as mine but has no responses to the post. Hopefully someone may have an answer or some pointers for me.

Problem
As with Peli and EssendonSTP, I am experiencing reboots at the end of the day which seem to coincide with users logging off. In fact one user said it always seems to happen right as she is logging off. This problem only started after I enabled Terminal Services in application mode and more than one user (myself) started using TS.
It occurs about once every two weeks and always has basically the same debug output.... see below

Environment
Windows 2003 server with sp1, terminal services enabled in application mode.
5 users with thin clients connecting to the Terminal Server.

Hardware
Dell SC1600 with 2GB RAM, dual Xeon 2.8 GHz proc's

In case you are wondering I actually have no idea what most of this output means, I just figured out how to view it recently with WinDBG. I picked up the extra's (;r;kv;lmtn) from this BBS and it seems to be what is wanted from those who are able to help.

Any help or at least direction is be greatly appreciated.

1: kd> !analyze -v;r;kv;lmtn;
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

SESSION_HAS_VALID_POOL_ON_EXIT (ab)
Caused by a session driver not freeing its pool allocations prior to a
session unload. This indicates a bug in win32k.sys, atmfd.dll,
rdpdd.dll or a video driver.
Arguments:
Arg1: 00000002, session ID
Arg2: 000109e8, number of paged pool bytes that are leaking
Arg3: 00000000, number of nonpaged pool bytes that are leaking
Arg4: 00000176, total number of paged and nonpaged allocations that are leaking.
nonpaged allocations are in the upper half of this word,
paged allocations are in the lower half of this word.

Debugging Details:
------------------


DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0xAB

CURRENT_IRQL: 0

LAST_CONTROL_TRANSFER: from 8098115a to 80873432

STACK_TEXT:
f30edc30 8098115a 000000ab 00000002 000109e8 nt!KeBugCheckEx+0x1b
f30edc68 809c420a 884f2990 884f2990 00000000 nt!MiCheckSessionPoolAllocations+0xe3
f30edce8 8083460e 884f2990 00000000 88e42020 nt!MiDereferenceSessionFinal+0x183
f30edd04 808fc993 884f2990 88e42020 88e42260 nt!MmCleanProcessAddressSpace+0x6b
f30edd8c 808f6d9d 00000000 00000000 88e42020 nt!PspExitThread+0x5f1
f30edda4 808fc3b2 88e42020 00000000 00000001 nt!PspTerminateThreadByPointer+0x4b
f30edddc 80832195 bf9299dc 88d7d328 00000000 nt!PspSystemThreadStartup+0x3c
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16


FOLLOWUP_IP:
nt!MiCheckSessionPoolAllocations+e3
8098115a cc int 3

SYMBOL_STACK_INDEX: 1

FOLLOWUP_NAME: MachineOwner

SYMBOL_NAME: nt!MiCheckSessionPoolAllocations+e3

MODULE_NAME: nt

DEBUG_FLR_IMAGE_TIMESTAMP: 4202de34

STACK_COMMAND: kb

IMAGE_NAME: memory_corruption

FAILURE_BUCKET_ID: 0xAB_nt!MiCheckSessionPoolAllocations+e3

BUCKET_ID: 0xAB_nt!MiCheckSessionPoolAllocations+e3

Followup: MachineOwner
---------

eax=f772713c ebx=000109e8 ecx=00000000 edx=00000000 esi=f7727120 edi=00000176
eip=80873432 esp=f30edc18 ebp=f30edc30 iopl=0 nv up ei ng nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00000286
nt!KeBugCheckEx+0x1b:
80873432 5d pop ebp
ChildEBP RetAddr Args to Child
f30edc30 8098115a 000000ab 00000002 000109e8 nt!KeBugCheckEx+0x1b (FPO: [Non-Fpo])
f30edc68 809c420a 884f2990 884f2990 00000000 nt!MiCheckSessionPoolAllocations+0xe3 (FPO: [Non-Fpo])

f30edce8 8083460e 884f2990 00000000 88e42020 nt!MiDereferenceSessionFinal+0x183 (FPO: [Non-Fpo])
f30edd04 808fc993 884f2990 88e42020 88e42260 nt!MmCleanProcessAddressSpace+0x6b (FPO: [Non-Fpo])
f30edd8c 808f6d9d 00000000 00000000 88e42020 nt!PspExitThread+0x5f1 (FPO: [Non-Fpo])
f30edda4 808fc3b2 88e42020 00000000 00000001 nt!PspTerminateThreadByPointer+0x4b (FPO: [Non-Fpo])
f30edddc 80832195 bf9299dc 88d7d328 00000000 nt!PspSystemThreadStartup+0x3c (FPO: [Non-Fpo])
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16
start end module name
80800000 80a6d000 nt ntkrnlmp.exe Thu Feb 03 18:30:12 2005 (4202DE34)
80a6d000 80a99000 hal halmacpi.dll Thu Feb 03 18:16:35 2005 (4202DB03)
bf800000 bf9ce000 win32k win32k.sys Thu Feb 03 18:28:27 2005 (4202DDCB)
bf9ce000 bf9e4000 dxg dxg.sys Thu Feb 03 18:15:29 2005 (4202DAC1)
bf9e4000 bfa3a680 ati2drad ati2drad.dll Thu Feb 03 20:17:35 2005 (4202F75F)
bff60000 bff7e000 RDPDD RDPDD.dll Thu Feb 03 20:17:43 2005 (4202F767)
f3156000 f3181000 RDPWD RDPWD.SYS Thu Feb 03 18:15:27 2005 (4202DABF)
f345a000 f3465000 TDTCP TDTCP.SYS Thu Feb 03 18:15:27 2005 (4202DABF)
f3832000 f3848f40 ESDPDX01 ESDPDX01.SYS Wed Dec 24 19:00:53 2003 (3FEA52E5)
f3989000 f39e8000 srv srv.sys Thu Feb 03 18:28:36 2005 (4202DDD4)
f3bb8000 f3bc2000 Dxapi Dxapi.sys Mon Mar 24 23:06:01 2003 (3E7FFFD9)
f3e48000 f3e5d000 Cdfs Cdfs.SYS Thu Feb 03 18:29:06 2005 (4202DDF2)
f3e5d000 f3e6e000 Fips Fips.SYS Thu Feb 03 18:28:47 2005 (4202DDDF)
f3e6e000 f3ee4000 mrxsmb mrxsmb.sys Thu Feb 03 18:29:23 2005 (4202DE03)
f3ee4000 f3f14000 rdbss rdbss.sys Thu Feb 03 18:29:01 2005 (4202DDED)
f3f14000 f3f3e000 afd afd.sys Thu Feb 03 18:28:59 2005 (4202DDEB)
f3f3e000 f3f6f000 netbt netbt.sys Thu Feb 03 18:28:47 2005 (4202DDDF)
f3f6f000 f3fae160 SYMTDI SYMTDI.SYS Fri Jun 11 18:20:13 2004 (40CA5A4D)
f3faf000 f4010000 tcpip tcpip.sys Thu Feb 03 18:28:49 2005 (4202DDE1)
f4010000 f4029000 ipsec ipsec.sys Thu Feb 03 18:29:05 2005 (4202DDF1)
f4049000 f4059980 naveng naveng.sys Tue Mar 15 12:35:44 2005 (42374720)
f405a000 f40f2e20 navex15 navex15.sys Tue Mar 15 12:51:30 2005 (42374AD2)
f4355000 f4360000 dump_dellcerc dump_dellcerc.sys Mon Dec 16 18:35:43 2002 (3DFE8D7F)
f4375000 f437f000 dump_diskdump dump_diskdump.sys Thu Feb 03 18:17:02 2005 (4202DB1E)
f4617000 f4624000 netbios netbios.sys Thu Feb 03 18:22:08 2005 (4202DC50)
f4757000 f4762000 Msfs Msfs.SYS Thu Feb 03 18:18:07 2005 (4202DB5F)
f4787000 f4793000 vga vga.sys Thu Feb 03 18:18:03 2005 (4202DB5B)
f47cd000 f47f7000 Fastfat Fastfat.SYS Thu Feb 03 18:28:38 2005 (4202DDD6)
f4f1e000 f4f2c000 msgpc msgpc.sys Thu Feb 03 18:22:17 2005 (4202DC59)
f50b6000 f50be000 rasacd rasacd.sys Mon Mar 24 23:11:50 2003 (3E800136)
f5306000 f5313000 wanarp wanarp.sys Thu Feb 03 18:22:57 2005 (4202DC81)
f539e000 f53a6000 mnmdd mnmdd.SYS Mon Mar 24 23:07:53 2003 (3E800049)
f5476000 f5488f00 SYMEVENT SYMEVENT.SYS Wed Jan 14 18:02:13 2004 (4005F4A5)
f5489000 f54d8000 savrt savrt.sys Mon Feb 09 15:24:30 2004 (402816AE)
f5538000 f5548000 Savrtpel Savrtpel.sys Mon Feb 09 15:24:34 2004 (402816B2)
f58b8000 f58c1000 ndisuio ndisuio.sys Thu Feb 03 18:22:02 2005 (4202DC4A)
f5c68000 f5c6f000 dxgthk dxgthk.sys Mon Mar 24 23:05:52 2003 (3E7FFFD0)
f5da0000 f5daa000 flpydisk flpydisk.sys Mon Mar 24 23:04:32 2003 (3E7FFF80)
f5e90000 f5e97000 Null Null.SYS Mon Mar 24 23:03:05 2003 (3E7FFF29)
f5ea0000 f5ea7000 parvdm parvdm.sys Mon Mar 24 23:03:49 2003 (3E7FFF55)
f5ee7000 f5efb000 usbhub usbhub.sys Thu Feb 03 18:18:58 2005 (4202DB92)
f60cb000 f60d9000 NDProxy NDProxy.SYS Thu Feb 03 18:23:01 2005 (4202DC85)
f61bb000 f61c8000 Npfs Npfs.SYS Thu Feb 03 18:18:08 2005 (4202DB60)
f626b000 f62ab000 update update.sys Thu Feb 03 18:28:43 2005 (4202DDDB)
f62ab000 f62e2000 rdpdr rdpdr.sys Thu Feb 03 18:18:24 2005 (4202DB70)
f62e2000 f62f5000 raspptp raspptp.sys Thu Feb 03 18:29:00 2005 (4202DDEC)
f62f5000 f630f000 ndiswan ndiswan.sys Thu Feb 03 18:29:03 2005 (4202DDEF)
f630f000 f6324000 rasl2tp rasl2tp.sys Thu Feb 03 18:28:46 2005 (4202DDDE)
f6324000 f634e000 USBPORT USBPORT.SYS Thu Feb 03 18:18:55 2005 (4202DB8F)
f634e000 f6376000 ks ks.sys Thu Feb 03 18:29:18 2005 (4202DDFE)
f6376000 f638a000 redbook redbook.sys Thu Feb 03 18:16:52 2005 (4202DB14)
f638a000 f639f000 cdrom cdrom.sys Thu Feb 03 18:17:03 2005 (4202DB1F)
f639f000 f63b7000 parport parport.sys Thu Feb 03 18:16:31 2005 (4202DAFF)
f63b7000 f63cc000 serial serial.sys Thu Feb 03 18:16:34 2005 (4202DB02)
f63cc000 f63df000 i8042prt i8042prt.sys Thu Feb 03 18:29:20 2005 (4202DE00)
f63df000 f63fa000 VIDEOPRT VIDEOPRT.SYS Thu Feb 03 18:18:02 2005 (4202DB5A)
f63fa000 f644dd80 ati2mpad ati2mpad.sys Thu Jul 18 18:13:20 2002 (3D3767B0)
f644e000 f6466c00 e1000325 e1000325.sys Mon Jan 13 11:12:12 2003 (3E230F8C)
f67d7000 f67df000 audstub audstub.sys Mon Mar 24 23:09:12 2003 (3E800098)
f67df000 f67e3400 usbohci usbohci.sys Thu Feb 03 18:18:57 2005 (4202DB91)
f6dfc000 f6e06000 mouclass mouclass.sys Mon Mar 24 23:03:09 2003 (3E7FFF2D)
f6e0c000 f6e1b000 intelppm intelppm.sys Thu Feb 03 18:16:36 2005 (4202DB04)
f6e1c000 f6e27000 fdc fdc.sys Thu Feb 03 18:16:50 2005 (4202DB12)
f6ff4000 f6ffe000 kbdclass kbdclass.sys Mon Mar 24 23:03:10 2003 (3E7FFF2E)
f7064000 f706d000 mssmbios mssmbios.sys Thu Feb 03 18:22:04 2005 (4202DC4C)
f7074000 f707f000 ptilink ptilink.sys Thu Feb 03 18:16:30 2005 (4202DAFE)
f7084000 f708d000 raspti raspti.sys Thu Feb 03 18:23:03 2005 (4202DC87)
f70e4000 f70f3000 termdd termdd.sys Thu Feb 03 18:15:27 2005 (4202DABF)
f70f4000 f70fd000 watchdog watchdog.sys Thu Feb 03 18:18:30 2005 (4202DB76)
f71f4000 f7213000 Mup Mup.sys Thu Feb 03 18:29:01 2005 (4202DDED)
f7213000 f7249000 NDIS NDIS.sys Thu Feb 03 18:28:43 2005 (4202DDDB)
f7249000 f72de000 Ntfs Ntfs.sys Thu Feb 03 18:28:43 2005 (4202DDDB)
f72de000 f7305000 KSecDD KSecDD.sys Thu Feb 03 18:16:45 2005 (4202DB0D)
f7305000 f731b000 sis sis.sys Thu Feb 03 18:18:24 2005 (4202DB70)
f731b000 f7340000 fltmgr fltmgr.sys Thu Feb 03 18:18:28 2005 (4202DB74)
f7340000 f7353000 CLASSPNP CLASSPNP.SYS Thu Feb 03 18:28:40 2005 (4202DDD8)
f7353000 f7364000 symmpi symmpi.sys Mon Dec 13 13:03:14 2004 (41BE0392)
f7364000 f7383000 SCSIPORT SCSIPORT.SYS Thu Feb 03 18:28:42 2005 (4202DDDA)
f7383000 f739f000 atapi atapi.sys Thu Feb 03 18:16:55 2005 (4202DB17)
f739f000 f73c9000 volsnap volsnap.sys Thu Feb 03 18:17:18 2005 (4202DB2E)
f73c9000 f73f5000 dmio dmio.sys Thu Feb 03 18:18:10 2005 (4202DB62)
f73f5000 f741c000 ftdisk ftdisk.sys Thu Feb 03 18:17:06 2005 (4202DB22)
f741c000 f7432000 pci pci.sys Thu Feb 03 18:22:03 2005 (4202DC4B)
f7432000 f7466000 ACPI ACPI.sys Thu Feb 03 18:21:58 2005 (4202DC46)
f7487000 f7490000 WMILIB WMILIB.SYS Mon Mar 24 23:13:00 2003 (3E80017C)
f7497000 f74a6000 isapnp isapnp.sys Mon Mar 24 23:16:35 2003 (3E800253)
f74a7000 f74b4000 PCIIDEX PCIIDEX.SYS Thu Feb 03 18:16:55 2005 (4202DB17)
f74b7000 f74c7000 MountMgr MountMgr.sys Thu Feb 03 18:15:23 2005 (4202DABB)
f74c7000 f74d2000 PartMgr PartMgr.sys Thu Feb 03 18:28:50 2005 (4202DDE2)
f74d7000 f74e2000 dellcerc dellcerc.sys Mon Dec 16 18:35:43 2002 (3DFE8D7F)
f74e7000 f74f7000 disk disk.sys Thu Feb 03 18:17:04 2005 (4202DB20)
f74f7000 f7503000 Dfs Dfs.sys Thu Feb 03 18:18:31 2005 (4202DB77)
f7507000 f7511000 crcdisk crcdisk.sys Thu Feb 03 18:17:51 2005 (4202DB4F)
f7597000 f75a1000 serenum serenum.sys Thu Feb 03 18:16:33 2005 (4202DB01)
f75b7000 f75c5000 imapi imapi.sys Thu Feb 03 18:17:18 2005 (4202DB2E)
f75d7000 f75e2000 TDI TDI.SYS Mon Mar 24 23:14:28 2003 (3E8001D4)
f7637000 f7640000 ndistapi ndistapi.sys Thu Feb 03 18:23:00 2005 (4202DC84)
f7697000 f76a6000 raspppoe raspppoe.sys Thu Feb 03 18:23:04 2005 (4202DC88)
f7707000 f770f000 kdcom kdcom.dll Mon Mar 24 23:08:00 2003 (3E800050)
f770f000 f7717000 BOOTVID BOOTVID.dll Mon Mar 24 23:07:58 2003 (3E80004E)
f7717000 f771e000 pciide pciide.sys Mon Mar 24 23:04:46 2003 (3E7FFF8E)
f771f000 f7726000 dmload dmload.sys Mon Mar 24 23:08:08 2003 (3E800058)
f77d7000 f77df000 RDPCDD RDPCDD.sys Mon Mar 24 23:03:05 2003 (3E7FFF29)
f780f000 f7817000 Fs_Rec Fs_Rec.SYS Mon Mar 24 23:08:36 2003 (3E800074)
f7817000 f781e000 Beep Beep.SYS Mon Mar 24 23:03:04 2003 (3E7FFF28)
f79e1000 f79e2280 swenum swenum.sys Thu Feb 03 18:15:43 2005 (4202DACF)
f79e5000 f79e6580 USBD USBD.SYS Mon Mar 24 23:10:39 2003 (3E8000EF)
f7a07000 f7a08a60 uphcleanhlp uphcleanhlp.sys Thu Mar 04 21:43:35 2004 (40481387)

Unloaded modules:
f5c50000 f5c58000 Sfloppy.SYS
Timestamp: unavailable (00000000)
Checksum: 00000000

 

Answer from Microsoft

I started a support case with MS and this is the answer I got. It might explain why the top 5 viewed issues in this forum are related to BSOD and reboots (same thing handled differently)

"According to the dump file, the issue occurred as some paged pools were not released properly in the current terminal session. The related pool tags included Ggls, Gffv, Gh2. This result indicates the issue is same with many other reports we received recently. As all the related pool tags are allocated from win32k.sys, we have forwarded the dump files to our development team and they are working on this issue. "
and
"So far, all of the reported problematic servers are with Windows 2003 Server and Service Pack 1 (SP1). Based on our experience, uninstalling SP1 can be considered as a temporary workaround before the fix is released. "

 

I got the same problem. After applying the SP1, I got Windows 2003 with SP1 and 20 users license. There is no problem for running the application but after logout. The server will blue screen and restart. Do you know when will issue the hotfix. Thanks.

Regards


Godwin

 

I have heard nothing. I no longer have the problem since un-installing sp1. I will not be re-installing it until this trouble has long been resolved and I no longer see it as the number 1 problem on this and other boards.

 

Thank you. Can you mind to share with me if you got any update from this. Thanks

Regards

 

A response from MS

I finally received a response from Microsoft.

"Our development team built a check build version of win32k.sys to trace pool allocation. I tested it in my test computer and it was able to save every pool allocation to stack trace record. As the issue is actually different with many different kinds of behaviors although the result is same, would you please help configure the check build version of win32k.sys in your terminal server so that we can get a dump file for analysis? "

My response, (read with Al's voice from Home Improvement) "I don't think so Tim!!"

I will install it on a test box, when I get time, and try and simulate the problem again. I am NOT going thru' that headache again on a production box. I'll let you know when I do so.

 

i've been involved in several of these ABs. They are slowly getting identified and fixed as people report dumps using the instrumented win32k. The causes have been all over the map, I would not count on someone else running into the same tag that you have.

 

Hmmmm,
We have several SP1 W3K server's doing the same thing, any progress??

 

I've got same problems in my servers now after we connect about 20 users via terminal. My environment is simular: W2003 Ent. with SP1.

I had searched simular topic in Internet but most vaulable information are just posted in this BBS.

Now, we will remove SP1 from these terminal servers and see what will happened.

Any updates will be posted in this topic

 

Good news to me. We removed the SP1 then our server runs very smooth. Still, we are watching the new release hotfix from MS for the final solution.

 

Hello,

since yesterday, when a new TerminalServer went productive, I encountered the same problem, though all other servers (in house and at customers) with W3K3 and SP1 don't show this error.

On this special server, I cannot uninstall the SP1, so I'll try not to log off the users, but only to disconnect their sessions. At night the server is rebooted anyway before backup, so it's fresh in the morning. Maybe this doesn't help you, but this is my personal workaround for the problem, as long as Microsoft hasn't a solution.

regards
Martin

 

Same boat as the rest of you

I am involved in a project where recently upgraded a Windows 2000/Citrix Metaframe XPe farm to Windows 2003 SP1 and Citrix Presentation Server 3.0. We are experiencing the same issues (BSOD-STOP 0x000000AB error).

We have started a case with Microsoft and they offered us a hotfix (Article ID : 901150) but this has not fixed our issue. MS support thinks it might be the driver ATI2MPAD.SYS (based on their debugging tools). They have recommended that we update the driver but that driver is native to Windows and HP (Proliant DL380 G3) doesn't have any newer drivers. We are still waiting on word from MS.

Since this is a financial institution we are hesitant to remove SP1 due to compliance issues with the auditors, but if there is no fix from MS shortly we will have to uninstall SP1 as our "fix" for now.

I will provide more details when we figure something out.

Mike

 

Update from our MS Technican

I just got a note from our MS technician.

"Thank you for contacting Microsoft Technical Support. I am the support professional working on your issue. As you server memory dump analyzed and find below the suggestions.

The server bug checked due to a user session having pool allocations that should have been freed before the session exited. The following tags were leaking.

Gffv, Ggls, Gh=<, Gpfe, and Gpff. Development is already aware of this issue and they are working on a fix. I do not have an estimated time line on when the fix will be ready. We do however have a private version of the fix available. Let me know if you wish to apply the private fix. Please for your information that a private fix does not have an installer and it will need to be manually copied onto the server from safe mode.

If you prefer to wait for a final fix to be available, then you can uninstall Service Pack 1 for W2K3 as a work-around."

I have requested the private fix and we are waiting for the developers to send it to us.

I will update when I have something more.

-Mike

 

mpagan

have you received this hotfix ?

 

If the private version is here, I think it will be released at least 1 month because they need make the package and do more test. Please keep to update. Great Job. Thanks.

 

Private hotfix aquired

Here is the last I got from MS. Its came with a 1MB zip file. Is there anywhere I can post it? For all to share?

admin note: see Joe Hobart's last post - sharing this is illegal and very possibly dangerous for others who have similar symptoms but not the exact same problem. I'm removing the remainder of this post and the others except for the last one.

 

mike- I reviewed your case and have sent you an email.

If you are experiencing this problem, you should contact PSS for further investigation. This symptom has quite a few causes, and you do *not* want to be running a private build without knowing for sure which tags are leaking.

 

guys, he cannot give out this fix without violating his license agreement and support contract with microsoft. If you want, or think you need this fix, you need to call microsoft.