RSS Feed Subscribe to the RSS feed Last updated:  15 Jul 2012

SysnativeBSODApps - additional check 'drivers found in stack' by mgrzeg
(x86) dps @esp-100 @esp +200
(x64) dps @rsp-200 @rsp+400

The Raw Truth - Using Raw Stacks by Vir Gnarus
Mark Russinovich's use of DPS (video):  http://channel9.msdn.com/Events/TechEd/Europe/2010/WCL301

Thanks to Niemiro:
itoldyouso - an undocumented command for the debugger
Raw Stack Analysis scripts  http://www.dumpanalysis.org/blog/index.php/raw-stack-analysis-scripts/
Stack Reconstruction:  http://blogs.msdn.com/b/ntdebugging/archive/2010/05/12/x64-manual-stack-reconstruction-and-stack-walking.aspx



15 Jul 2012
Thanks to niemiro:  
dps KiPreBugcheckStackSaveArea KiPreBugcheckStackSaveArea+3000

2 great commands from  jcgriff2 at http://www.sysnative.com
!for_each_module - Driver Version Info:  http://www.sysnative.com/forums/showthread.php/2480-!for_each_module-Driver-Version-Info
!for_each_module - Driver Location Info:  http://www.sysnative.com/forums/showthread.php/2479-!for_each_module-Driver-Location-Info

Application Hangs:
==============
Listing the lmvm for the executable (normally has version information that is useful)
vertarget
!exchain
!locks
listing the number of threads
!uniqstack
!address -summary
~*kvn250
!gflag
!runaway
!peb
!dh
!chkimg
if the module mscorwks is loaded:
!loadby sos mscorwkks
!clrstack
.load wow64exts
!analyze -v -hang
 
Application Crashes:
==============
Listing the lmvm for the executable (normally has version information that is useful)
vertarget
!exchain
!locks
listing the number of threads
!uniqstack
!address -summary
!vm
~*kvn250
!gflag
!runaway
!peb
!dh
!chkimg
if the module mscorwks is loaded:
!loadby sos mscorwkks
!clrstack
.load wow64exts
!analyze -v
 
System hangs:
============
!analyze -v -hang
!locks
!vm 4
!poolused
!stacks
!exqueue f
!irpfind
!process 0 ff or ListProcessStacks
!lpc message
!ntsdexts.locks
!session
!sprocess
!process 0 0
!running
!ready
!dpcs
!apc
 
 
BSOD:
======
!analyze -v
!pool
component timestamps
!vm 4
current threads on other processors
raw stack
bugcheck description (including ln exception address for corrupt or truncated dumps)
 
*****************************************************************
 30 Jun 2010

Interesting command that I found (works for NTStatus and WIN32 errors):

[code]
!error <error message here>
[/code]

For example:
[code]
!error 0xc0000005
[/code]
gives this output:
[code]
0: kd> !error 0xc0000005
Error code: (NTSTATUS) 0xc0000005 (3221225477) - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
[/code]

To freshen/speed up the symbol resolution:
.reload /f - plan to wait for a long time.  All the symbols will get pulled down over the network to a local on disk cache.
.reload /u (to unload the current symbols)
.reload /s (to reload the kernel symbols)


Some commands for my use:
.kframes 1000;!analyze -v;lmtsmn

Assorted Debugging Commands that have been found useful

 
0: kd> !devobj fffffa80077cb050 f
fffff80001fa7ec0: Unable to get value of ObpRootDirectoryObject
Device object (fffffa80077cb050) is for:
 <?} \Driver\SNP2UVC DriverObject fffffa800776c510
Current Irp 00000000 RefCount 0 Type 0000002f Flags 00002050
DevExt fffffa80077cb1a0 DevObjExt fffffa80077ce370 
AttachedDevice (Upper) fffffa8007737a30 \Driver\ksthunk
Device queue is not busy.

__________________

0: kd> !drvobj fffffa800776c510 f
fffff80001fa7ec0: Unable to get value of ObpRootDirectoryObject
fffff80001fa7ec0: Unable to get value of ObpRootDirectoryObject
Driver object (fffffa800776c510) is for:

 \Driver\SNP2UVC
Driver Extension List: (id , addr)

Device Object list:
fffffa80077cb050 

DriverEntry: fffffa600a5c7910 snp2uvc
DriverStartIo: 00000000 
DriverUnload: fffffa6000dd0d80 ks!KsNullDriverUnload
AddDevice: 00000000 

Dispatch routines:
[00] IRP_MJ_CREATE fffffa6000dc80d0 ks!DispatchCreate
[01] IRP_MJ_CREATE_NAMED_PIPE fffff80001e3c420 nt!IopInvalidDeviceRequest
[02] IRP_MJ_CLOSE fffffa6000dc1538 ks!DispatchClose
[03] IRP_MJ_READ fffff80001e3c420 nt!IopInvalidDeviceRequest
[04] IRP_MJ_WRITE fffff80001e3c420 nt!IopInvalidDeviceRequest
[05] IRP_MJ_QUERY_INFORMATION fffff80001e3c420 nt!IopInvalidDeviceRequest
[06] IRP_MJ_SET_INFORMATION fffff80001e3c420 nt!IopInvalidDeviceRequest
[07] IRP_MJ_QUERY_EA fffff80001e3c420 nt!IopInvalidDeviceRequest
[08] IRP_MJ_SET_EA fffff80001e3c420 nt!IopInvalidDeviceRequest
[09] IRP_MJ_FLUSH_BUFFERS fffffa600a5cae5c STREAM!StreamClassPassThroughIrp
[0a] IRP_MJ_QUERY_VOLUME_INFORMATION fffff80001e3c420 nt!IopInvalidDeviceRequest
[0b] IRP_MJ_SET_VOLUME_INFORMATION fffff80001e3c420 nt!IopInvalidDeviceRequest
[0c] IRP_MJ_DIRECTORY_CONTROL fffff80001e3c420 nt!IopInvalidDeviceRequest
[0d] IRP_MJ_FILE_SYSTEM_CONTROL fffff80001e3c420 nt!IopInvalidDeviceRequest
[0e] IRP_MJ_DEVICE_CONTROL fffffa600a5cae5c STREAM!StreamClassPassThroughIrp
[0f] IRP_MJ_INTERNAL_DEVICE_CONTROL fffff80001e3c420 nt!IopInvalidDeviceRequest
[10] IRP_MJ_SHUTDOWN fffff80001e3c420 nt!IopInvalidDeviceRequest
[11] IRP_MJ_LOCK_CONTROL fffff80001e3c420 nt!IopInvalidDeviceRequest
[12] IRP_MJ_CLEANUP fffffa600a5cb140 STREAM!StreamClassCleanup
[13] IRP_MJ_CREATE_MAILSLOT fffff80001e3c420 nt!IopInvalidDeviceRequest
[14] IRP_MJ_QUERY_SECURITY fffff80001e3c420 nt!IopInvalidDeviceRequest
[15] IRP_MJ_SET_SECURITY fffff80001e3c420 nt!IopInvalidDeviceRequest
[16] IRP_MJ_POWER fffffa600a5cb2cc STREAM!StreamClassPower
[17] IRP_MJ_SYSTEM_CONTROL fffffa600a5d4a6c STREAM!StreamClassForwardUnsupported
[18] IRP_MJ_DEVICE_CHANGE fffff80001e3c420 nt!IopInvalidDeviceRequest
[19] IRP_MJ_QUERY_QUOTA fffff80001e3c420 nt!IopInvalidDeviceRequest
[1a] IRP_MJ_SET_QUOTA fffff80001e3c420 nt!IopInvalidDeviceRequest
[1b] IRP_MJ_PNP fffffa600a5d2f40 STREAM!StreamClassPnP
Australia
 
  H2SO4 is offline Add to H2SO4's Reputation   Report Post  

Default
Quote   Quote: Originally Posted by usasma View Post
I'd toss in a memory diagnostic because the one non-124 error cites hardware during a memory access. Instructions here: Memory Diagnostics

This is the minidump file from 101909 (it's a STOP 0x3B) It's the only one that isn't a STOP 0x124...
FWIW, I'm virtually certain that one is also hardware. The RIP is misaligned - the machine is executing code which doesn't exist:

0: kd> .trap fffff880`0c4b88f0
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=fffff900c06b60ff rbx=0000000000000000 rcx=00000000fffffa80
rdx=fffff900c0200228 rsi=0000000000000000 rdi=0000000000000000
rip=fffff960000f5cf0 rsp=fffff8800c4b8a80 rbp=fffffffff60109f7
r8=0000000000000000 r9=0000000000000017 r10=fffffa80066585b0
r11=fffff900c06b6010 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na pe nc
win32k!XDCOBJ::bCleanDC+0x228:
fffff960`000f5cf0 a0b615000fb7c83bce mov al,byte ptr [CE3BC8B70F0015B6h] ds:0e30:ce3bc8b7`0f0015b6=??
0: kd> ub .
^ Unable to find valid previous instruction for 'ub .'
0: kd> u fffff960`000f5cc0 L29
win32k!XDCOBJ::bCleanDC+0x1f8:
...
fffff960`000f5cef e8a0b61500 call win32k!DEC_SHARE_REF_CNT (fffff960`00251394)
fffff960`000f5cf4 0fb7c8 movzx ecx,ax
fffff960`000f5cf7 3bce cmp ecx,esi
fffff960`000f5cf9 7529 jne win32k!XDCOBJ::bCleanDC+0x25c (fffff960`000f5d24)
fffff960`000f5cfb 488b03 mov rax,qword ptr [rbx]
fffff960`000f5cfe 488b88a0000000 mov rcx,qword ptr [rax+0A0h]
fffff960`000f5d05 0fb701 movzx eax,word ptr [rcx]
fffff960`000f5d08 488d1440 lea rdx,[rax+rax*2]

It has munged together portions of three adjacent instructions into a "frankenstruction" which doesn't exist - but causes a crash.

*************************************

kd> dqs rsp
fffff800`00b9c880 fffffa80`0251a2a0
fffff800`00b9c888 fffff880`00000000
fffff800`00b9c890 fffffa80`0251a2a0
fffff800`00b9c898 00000000`00000001
fffff800`00b9c8a0 00000000`00000001
fffff800`00b9c8a8 00000000`00000000
fffff800`00b9c8b0 fffffa80`038dd500
fffff800`00b9c8b8 fffff880`04982717 nvlddmkm+0x136717
fffff800`00b9c8c0 fffffa80`0251a2a0
fffff800`00b9c8c8 fffffa80`0251a2a0
fffff800`00b9c8d0 00000000`00000000
fffff800`00b9c8d8 00000000`00000000
fffff800`00b9c8e0 fffffa80`00000002
fffff800`00b9c8e8 fffffa80`03a1ac58
fffff800`00b9c8f0 00000000`00000000
fffff800`00b9c8f8 00000000`00000001
kd> lmvm nvl*
start end module name
fffff880`0484c000 fffff880`05353b00 nvlddmkm T (no symbols)
Loaded symbol image file: nvlddmkm.sys
Image path: \SystemRoot\system32\DRIVERS\nvlddmkm.sys
Image name: nvlddmkm.sys
Timestamp: Fri May 15 06:48:07 2009 (4A0C8387)
CheckSum: 00B1940A
ImageSize: 00B07B00
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
lmvm tunnel
start end module name
fffff880`040f3000 fffff880`04119000 tunnel (pdb symbols) C:\Program Files\Debugging Tools for Windows (x64)\sym\tunnel.pdb\A96FA718059643E58535C380B4E5BA741\tunnel.pdb
Loaded symbol image file: tunnel.sys
Mapped memory image file: C:\Program Files\Debugging Tools for Windows (x64)\sym\tunnel.sys\4A5BCCC126000\tunnel.sys
Image path: \SystemRoot\system32\DRIVERS\tunnel.sys
Image name: tunnel.sys
Timestamp: Mon Jul 13 20:09:37 2009 (4A5BCCC1)
CheckSum: 0002B032
ImageSize: 00026000
File version: 6.1.7600.16385
Product version: 6.1.7600.16385
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 3.6 Driver
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® Windows® Operating System
InternalName: tunnel.sys
OriginalFilename: tunnel.sys
ProductVersion: 6.1.7600.16385
FileVersion: 6.1.7600.16385 (win7_rtm.090713-1255)
FileDescription: Microsoft Tunnel Interface Driver
 LegalCopyright: © Microsoft Corporation. All rights reserved.

More from jcgriff2:
Code:

x *!; lmnt; lmntsm; .bugcheck
Code:

!for_each_module .echo @#ModuleName fver = @#FileVersion pver = @#ProductVersion
Code:

!for_each_module .echo @#ModuleIndex : @#Base @#End @#ModuleName @#ImageName @#LoadedImageName

Another link:  http://blogs.msdn.com/iliast/archive/2006/12/11/crash-dump-analysis.aspx

 
FROM: http://blogs.technet.com/brad_rutkowski/archive/2008/04/01/some-useful-debugging-commands.aspx

Vertarget:

Lists Version information for the machine/dump you're debugging.  You can also use "version" to tell you about the debugger bits.

1: kd> vertarget 
Windows Kernel Version 6001 (Service Pack 1) MP (4 procs) Free x64 
Product: LanManNt, suite: TerminalServer SingleUserTS 
Built by: 6001.18000.amd64fre.longhorn_rtm.080118-1840 
Kernel base = 0xfffff800`0160c000 PsLoadedModuleList = 0xfffff800`017d1db0 
Debug session time: Tue Apr  1 14:29:22.553 2008 (GMT-7) 
System Uptime: 0 days 0:03:14.328

!sysinfo

Good utility to check the CPU revs, BIOS revs, etc

1: kd> !sysinfo machineid 
Machine ID Information [From Smbios 2.3, DMIVersion 35, Size=3752] 
BiosVendor = American Megatrends Inc. 
BiosVersion = 080002 
BiosReleaseDate = 10/01/2007 
SystemManufacturer = Microsoft Corporation 
SystemProductName = Virtual Machine 
SystemVersion = 5.0 
BaseBoardManufacturer = Microsoft Corporation 
BaseBoardProduct = Virtual Machine 
BaseBoardVersion = 5.0

1: kd> !sysinfo cpuinfo 
[CPU Information] 
~MHz = REG_DWORD 2660 
Component Information = REG_BINARY 0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0 
Configuration Data = REG_FULL_RESOURCE_DESCRIPTOR ff,ff,ff,ff,ff,ff,ff,ff,0,0,0,0,0,0,0,0 
Identifier = REG_SZ Intel64 Family 6 Model 15 Stepping 6 
ProcessorNameString = REG_SZ Intel(R) Xeon(R) CPU            5150  @ 2.66GHz 
Update Signature = REG_BINARY 0,0,0,0,0,0,0,0 
Update Status = REG_DWORD 8 
VendorIdentifier = REG_SZ GenuineIntel 
MSR8B = REG_QWORD 0

Getting the server name from the dump:

It's quite a bit easier to do internally, but this will get it done too.  Good to know you're debugging the right server. :)

1: kd> x srv!SrvComputerName 
fffffa60`04024500 srv!SrvComputerName = 
1: kd> dq fffffa60`04024500 
fffffa60`04024500  00000000`00180018 fffff880`04ccd8c0 
fffffa60`04024510  00000000`00000000 00000000`00000000 
fffffa60`04024520  00000000`00000000 00000000`00000000 
fffffa60`04024530  00000000`000c000a fffff880`04a0fc60 
fffffa60`04024540  fffffa60`04024540 fffffa60`04024540 
fffffa60`04024550  00000000`00060001 fffffa60`04024558 
fffffa60`04024560  fffffa60`04024558 00000000`ffffffff 
fffffa60`04024570  00000000`00000000 00000000`00000000 
1: kd> du fffff880`04ccd8c0 
fffff880`04ccd8c0  "BRAD-LHDC-01?"

!running -ti

This will dump the stacks of each thread that is running on each processor

1: kd> !running -ti

System Processors f (affinity mask) 
  Idle Processors f 
All processors idle.

     Prcb              Current           Next 
  0  fffff80001780680  fffff80001785b80                    ................

Child-SP          RetAddr           Call Site 
fffff800`026bb8d0 fffffa60`00a066da nt!KeSetTimer+0x89 
fffff800`026bb920 fffffa60`00a06aca NETIO!WfpStartTimerForLeftTime+0x8a 
fffff800`026bb970 fffffa60`00a06585 NETIO!WfppLeastRecentlyUsedTimerRoutine+0x1aa 
fffff800`026bb9c0 fffffa60`00a067ff NETIO!WfpTimerWheelTimeoutHandler+0x175 
fffff800`026bba40 fffff800`016698b3 NETIO!WfpSysTimerNdisCallback+0x4f 
fffff800`026bba70 fffff800`0166a238 nt!KiTimerListExpire+0x333 
fffff800`026bbca0 fffff800`0166aa9f nt!KiTimerExpiration+0x1d8 
fffff800`026bbd10 fffff800`0166bb72 nt!KiRetireDpcList+0x1df 
fffff800`026bbd80 fffff800`018395c0 nt!KiIdleLoop+0x62 
fffff800`026bbdb0 00000000`fffff800 nt!zzz_AsmCodeRange_End+0x4 

  1  fffffa60005f3180  fffffa60005fcd40                    ................

Child-SP          RetAddr           Call Site 
fffffa60`0171bb08 fffff800`016b03d7 nt!RtlpBreakWithStatusInstruction 
fffffa60`0171bb10 fffff800`0165afef nt! ?? ::FNODOBFM::`string'+0x356a 
fffffa60`0171bb50 fffffa60`026867a2 nt!KiSecondaryClockInterrupt+0x11f 
fffffa60`0171bce8 fffffa60`02685685 intelppm!C1Halt+0x2 
fffffa60`0171bcf0 fffff800`0167c7c8 intelppm!C1Idle+0x9 
fffffa60`0171bd20 fffff800`0166bb31 nt!PoIdle+0x148 
fffffa60`0171bd80 fffff800`018395c0 nt!KiIdleLoop+0x21 
fffffa60`0171bdb0 00000000`fffffa60 nt!zzz_AsmCodeRange_End+0x4 

!stacks

This is a great utility to check what threads are waiting on for each process.  Find out more in the debuggers chm.

1: kd> !stacks 2 
Proc.Thread  .Thread  Ticks   ThreadState Blocker

Max cache size is       : 1048576 bytes (0x400 KB) 
Total memory in cache   : 0 bytes (0 KB) 
Number of regions cached: 0 
0 full reads broken into 0 partial reads 
    counts: 0 cached/0 uncached, 0.00% cached 
    bytes : 0 cached/0 uncached, 0.00% cached 
** Prototype PTEs are implicitly decoded 
                            [fffffa8000c77950 System] 
   4.000008  fffffa8000c774c0 ffffe94b GATEWAIT   nt!KiSwapContext+0x7f 
                                        nt!KiSwapThread+0x2fa 
                                        nt!KeWaitForGate+0x22a 
                                        nt!MmZeroPageThread+0x162 
                                        nt!Phase1Initialization+0xe 
                                        nt!PspSystemThreadStartup+0x57 
                                        nt!KiStartSystemThread+0x16 
   4.000010  fffffa8000ca0720 ffffff8c Blocked    nt!KiSwapContext+0x7f 
                                        nt!KiSwapThread+0x2fa 
                                        nt!KeWaitForSingleObject+0x2da 
                                        nt!PopIrpWorkerControl+0x22 
                                        nt!PspSystemThreadStartup+0x57 
                                        nt!KiStartSystemThread+0x16 
   4.000014  fffffa8000c78bb0 fffffcb0 Blocked    nt!KiSwapContext+0x7f 
                                        nt!KiSwapThread+0x2fa 
                                        nt!KeWaitForSingleObject+0x2da 
                                        nt!PopIrpWorker+0x164 
                                        nt!PspSystemThreadStartup+0x57 
                                        nt!KiStartSystemThread+0x16

!PCR

Command will show you some useful info from the processor control block.  Like the current thread, next, DPQ queues (Can run !dpcs).

1: kd> !pcr 
KPCR for Processor 1 at fffffa60005f3000: 
    Major 1 Minor 1 
        NtTib.ExceptionList: fffffa60005fd280 
            NtTib.StackBase: fffffa60005f6cc0 
           NtTib.StackLimit: 000000000554f578 
         NtTib.SubSystemTib: fffffa60005f3000 
              NtTib.Version: 00000000005f3180 
          NtTib.UserPointer: fffffa60005f37f0 
              NtTib.SelfTib: 000007fffff8a000

                    SelfPcr: 0000000000000000 
                       Prcb: fffffa60005f3180 
                       Irql: 0000000000000000 
                        IRR: 0000000000000000 
                        IDR: 0000000000000000 
              InterruptMode: 0000000000000000 
                        IDT: 0000000000000000 
                        GDT: 0000000000000000 
                        TSS: 0000000000000000

              CurrentThread: fffffa60005fcd40 
                 NextThread: 0000000000000000 
                 IdleThread: fffffa60005fcd40

                  DpcQueue:  0xfffffa800124dc70 0xfffffa6000e7abe0 [Normal] tcpip!TcpPeriodicTimeoutHandler

1: kd>

!LMI

When I want to find out ifno about a particular driver in the dump, i use "lm n t" to get all of them, but then !lmi to drill into one.  I use it quite often to see if I have the private or public symbol loaded

1: kd> !lmi srv.sys 
Loaded Module Info: [srv.sys] 
         Module: srv 
   Base Address: fffffa6004007000 
     Image Name: srv.sys 
   Machine Type: 34404 (X64) 
     Time Stamp: 47919135 Fri Jan 18 21:57:09 2008 
           Size: 94000 
       CheckSum: 70fe5 
Characteristics: 22  perf 
Debug Data Dirs: Type  Size     VA  Pointer 
             CODEVIEW    20, 142c8,   136c8 RSDS - GUID: {D3FD3BA3-615D-437E-83B9-D339ED15DEE3} 
               Age: 2, Pdb: srv.pdb 
                CLSID     4, 142c4,   136c4 [Data not mapped] 
     Image Type: MEMORY   - Image read successfully from loaded memory. 
    Symbol Type: PDB      - Symbols loaded successfully from symbol server. 
                 C:\Debugger_Public\sym\srv.pdb\D3FD3BA3615D437E83B9D339ED15DEE32\srv.pdb 
    Load Report: public symbols , not source indexed 
                 C:\Debugger_Public\sym\srv.pdb\D3FD3BA3615D437E83B9D339ED15DEE32\srv.pdb

Published Tuesday, April 01, 2008 10:47 PM by Brad Rutkowski