DmpExt: Windbg extension

As mentioned in my recent presentation at Brucon 2014, I’ve written a small Windbg extension for exploring basic information about the Windows crash dump stack.  You can download the source code here.  This is currently the only supported platform, but it’s only a few hours of work to add other platforms, which I hope to accomplish soon.  Of course, feel free to commit your own changes to the repo and I will check them out.

To use the extension, first, make sure you have all symbols loaded (“.reload”). The extension is more useful with symbol resolution.  You will want to attach x86 Windbg to a virtual machine running Windows 8.0 x86.

To load the extension:

0: kd> .load <path_to>\DmpExt.dll

To get extension usage:

0: kd> !dmpext

DmpExt 1.0
-----------
Target OS build: Win NT 6.2
Available commands:
-stack: Display the crash dump driver stack
-crashdmp: Display details about crashdmp.sys
-filters: Display details about crash dump filters

Or simply use !help:

0: kd> !help
Commands for C:\<path>\DmpExt.dll:
!dmpext - The !dmpext extension displays information about the crash dump
infrastructure.

The -stack command simply enumerates drivers in the crash dump stack:

0: kd> !dmpext -stack

Crash dump stack drivers:
dump_LSI_SAS
dump_dumpfve
dump_diskdump
dump_dmpflt

Here we see that the dump miniport (LSI_SAS.sys), dump port (diskdump.sys), and two third-party filter drivers (dumpfve.sys and dmpflt.sys) are all loaded into the stack.  These drivers make up the crash I/O path to disk.

The -crashdmp command will display information about the operating system’s book-keeping driver, crashdmp.sys, such as the image base and dump block pointer, which is the starting point for the bypass technique discussed on this website:

0: kd> !dmpext -crashdmp

Crashdmp.sys information:
	Image entry:  0x81a4dcb0
	Image base:  0x82fe5000
	Initialized:  1
	Crashdmp GUID:  0e1d2972-65af-4ac1-bfa3-cef4ab0c38fe
	Dump block:  0x82fee300
	Context:  0x82fee298
	Crashdmp device GUID:  d8e2592f-1aab-4d56-a746-1f7585df40f4
	Crashdmp call table (0x81a4e0c0):
		Version 1.4
		0 => crashdmp!CrashdmpInitialize
		1 => crashdmp!CrashdmpLoadDumpStack
		2 => crashdmp!CrashdmpInitDumpStack
		3 => crashdmp!CrashdmpFreeDumpStack
		4 => crashdmp!CrashdmpDisable
		5 => crashdmp!CrashdmpNotify
		6 => crashdmp!CrashdmpWrite
		7 => crashdmp!CrashdmpUpdatePhysicalRange
		8 => crashdmp!CrashdmpResumeCapable
		9 => crashdmp!CrashdmpGetTransferSizes
		10 => crashdmp!CrashdmpLogStatusData
		11 => crashdmp!CrashdmpReady

The -filters command will display information about 3rd party vendor crash dump filter drivers, such as whole disk encryption solutions that encrypt hibernation and crash dump file contents:

0: kd> !dmpext -filters

Installed dump filters:
	Listhead at 0x880b18ec
	Filter at 0x8809a798:
		Version: 2.0
		DumpStart: dump_dumpfve!DumpStart (9081a7cc)
		DumpFinish: dump_dumpfve!DumpFinish (9081a84e)
		DumpUnload: dump_dumpfve!DumpUnload (9081a912)
		DumpWrite: dump_dumpfve!DumpWrite (9081a858)
		DumpRead: dump_dumpfve!DumpRead (9081a8da)
		Dump data: 8809e000
	Filter at 0x88099318:
		Version: 2.0
*** ERROR: Module load completed but symbols could not be loaded for dump_dmpflt.sys
		DumpStart: dump_dmpflt (909ed490)
		DumpFinish: dump_dmpflt (909ed3e0)
		DumpUnload: dump_dmpflt (909ed4a0)
		DumpWrite: dump_dmpflt (909ed4d0)
		DumpRead: dump_dmpflt (909ed480)
		Dump data: 00000000

In the output above, Microsoft’s full volume encryption driver (dumpfve.sys) and my own dmpflt.sys filter driver are loaded in the stack. The filter block pointer provided in the output is the starting point for hijacking the crashdmp.sys book-keeping structure as outlined in the filter hijack technique.

Happy crash dumping!

Advertisements
Posted in blog, conference, source code, Windbg extensions

Brucon 2014

This Thursday at high noon, I will be presenting my crash dump research at Brucon 2014.  This presentation acts as sort of a chronology of my research, covering each major development since 2012, and is basically this entire website condensed into 57 slides.  I have produced a completely new slide deck with a few updates for Windows 8.1.  I’m also demoing several research tools I’ve discussed on this website, plus a new Windbg extension ‘dmpext’:

  • CrashDD – never before released tool which acts like the unix ‘dd’ tool, but for accessing disk through the crash path
  • LiveDump – user mode tool for creating a kernel crash dump
  • DmpExt – a new Windbg extension I will be revealing at Brucon for exploring crash dump stack information

Hope to see you there!

Posted in blog, conference

LiveDump 1.0 is available

To wrap up the recent news concerning live dumps, which I discussed here and here, I will cover the topic in a bit more detail and provide full source code to a tool you can use to explore the feature yourself.

Brief History of Live Dumps

Windows has always provided the ability to create non-invasive, live crash dumps through a kernel debugger using the “.dump” command (this does not crash the computer, see “.crash”).  Additionally, tools like LiveKD allow you to create “mirror dumps” of a live system (only kernel memory).  However, until these recent changes in Windows 8.1, these capabilities were not easily accessible to system administrators in most scenarios, due to the impractical necessity of attaching a kernel debugger to the target machine.  Note that invasive crash dumps (those that cause a bug check such as keyboard-initiated dumps) are not the same thing as live dumps.

NtSystemDebugControl

This undocumented function, previously the home of a serious security bug that allowed unrestricted access to all of physical memory, provides a host of debugging capabilities for kernel components (mostly meant for a kernel debugger).  The following sections will discuss how this function works with corresponding kernel functions to generate dump files.  Please take a look at kdtypes.h in the NDK for more information about data types discussed here.

Triage Dump – Parameter 29

Since at least Vista, NtSystemDebugControl has provided a means to create a triage dump via SYSDBG_COMMAND value 29 (SysDbgGetTriageDump).  In fact, this is the only control code you could issue to this API without having a kernel debugger attached and windows booted in debug mode.  Attempting to use this function without meeting those requirements would result in status code 0xc0000354 or STATUS_DEBUGGER_INACTIVE.  Parameter 29 will create a triage dump through an internal function, DbgkCaptureLiveDump, which uses a set of internal kernel functions that manipulate debugger “snapshot data” (eg, DbgkpTriageDumpSnapData).  The parameter that must accompany value 29 is SYSDBG_TRIAGE_DUMP, which specifies the thread handles to be dumped, along with various values that will be written in the triage dump header.  The caller must also supply an output buffer of at least 132k and at most 1MB for the dump data.  The structure of the input parameter from the NDK is shown below.

typedef struct _SYSDBG_TRIAGE_DUMP
{
ULONG Flags;
ULONG BugCheckCode;
ULONG_PTR BugCheckParam1;
ULONG_PTR BugCheckParam2;
ULONG_PTR BugCheckParam3;
ULONG_PTR BugCheckParam4;
ULONG ProcessHandles;
ULONG ThreadHandles;
PHANDLE Handles;
} SYSDBG_TRIAGE_DUMP, *PSYSDBG_TRIAGE_DUMP;

The ProcessHandles field must be 0 and the ThreadHandles field must be non-zero. The Handles field must be an array of opened thread handles. The remaining fields are optional.

As you can guess from this structure definition, triage dump generation as it is implemented in the kernel has the annoying requirement that you must specify one or more thread handles in the SYSDBG_TRIAGE_DUMP structure.  If you do not pass in the required thread handles, the function will return STATUS_INVALID_PARAMETER.  DbgkCaptureLiveDump will iterate over the thread handles you supply, take a reference on the associated PsThreadType object, and get “snapshot data” from the thread by queueing an APC to the thread.  This APC will execute a series of Dbgk* functions that collect thread stack information in cooperation with the thread.  All of this data is appended to the triage dump data buffer.

If you want to learn more about triage dumps, skywing has a good blog post here.

Kernel Dump – Parameter 37 (Windows 8.1+)

In the latest NDK, the highest value for SYSDBG_COMMAND is 36.  Windows 8.1 introduces a new value to SYSDBG_COMMAND, value 37, and NtSystemDebugControl has been updated to allow this command (along with 29, which operates the same as it has in the past) to execute with or without a debugger attached and regardless of debug mode.  Under the hood, NtSystemDebugControl calls DbgkCaptureLiveKernelDump, a new internal function that interacts with the I/O manager and memory manager to dump memory while the machine is still active.  Microsoft has introduced a set of 57 new internal functions (prepended with IopLiveDump) to support this mechanism, which I assume to be the “CrashAPI” I recently saw in Windbg output.

For parameter 37, NtSystemDebugControl takes as input the structure below (undocumented).

typedef struct _SYSDBG_LIVEDUMP_CONTROL
{
ULONG Version;
ULONG BugCheckCode;
ULONG BugCheckParam1;
ULONG BugCheckParam2;
ULONG BugCheckParam3;
ULONG BugCheckParam4;
PVOID DumpFileHandle;
PVOID CancelEventHandle;
SYSDBG_LIVEDUMP_CONTROL_FLAGS Flags;
SYSDBG_LIVEDUMP_CONTROL_ADDPAGES AddPagesControl;
} SYSDBG_LIVEDUMP_CONTROL, *PSYSDBG_LIVEDUMP_CONTROL;

This structure is similar to the SYSDBG_TRIAGE_DUMP structure with a few notable differences:

  1. You must specify an output file handle for the dump to be written.  The kernel takes care of filling it for you, instead of the old way of just giving you back a buffer.
  2. You have the option of canceling the dump process via an alertable object you can signal (more on that later)
  3. A placeholder flag field for fine-grained control over what memory pages are included in the dump
  4. A placeholder flag field for fine-grained control over various dump options

I call these placeholder flags, because they really don’t do much at the moment.  The values for the control flags currently supported are shown below.

typedef union _SYSDBG_LIVEDUMP_CONTROL_FLAGS
{
struct
{
ULONG UseDumpStorageStack: 1;
ULONG CompressMemoryPagesData: 1;
ULONG IncludeUserSpaceMemoryPages: 1;
ULONG Reserved: 29;
};
ULONG AsUlong;
} SYSDBG_LIVEDUMP_CONTROL_FLAGS;

Of particular interest to me (since it appears to directly relate to my research on this website), is the UseDumpStorageStack bit.  I don’t see this flag used anywhere in the LiveDump kernel functions (except for a single check in IopLiveDumpValidateParameters, which validates the control flag is not 3 – CompressMemoryPagesData + UseDumpStorageStack), so the feature appears to be unimplemented.  Given that the function that actually writes buffered dump data to the output dump file (IopLiveDumpWriteBuffer) simply calls ZwWriteFile, it’s pretty obvious the live dump feature exclusively uses the normal I/O stack and not the crash dump stack.  Just to be sure, I put a breakpoint on the dump port driver’s DiskDumpWrite function and issued a live dump request that set the UseDumpStorageStack bit to 1.  The breakpoint was not hit.

The flag values for controlling the pages included in the dump are shown below.

typedef union _SYSDBG_LIVEDUMP_CONTROL_ADDPAGES
{
struct
{
ULONG HypervisorPages: 1;
ULONG Reserved: 31;
};
ULONG AsUlong;
} SYSDBG_LIVEDUMP_CONTROL_ADDPAGES;

Currently the only support flag here is to add hypervisor pages to the generated dump (note that the control flag includes the ability to add user space pages to the dump).

The crash API generates a kernel dump as follows:

  1. Acquire an exclusive lock, IopLiveDumpLock
  2. Validate the SYSDBG_LIVEDUMP_CONTROL structure
    1. Control flags cannot be 3 (CompressMemoryPagesData + UseDumpStorageStack)
    2. The dump file handle must have been created for synchronous I/O
    3. If specified, the cancel object must be an event, process, thread or timer object
  3. Allocate a global structure (IopLiveDumpContext) to hold the settings passed in via SYSDBG_LIVEDUMP_CONTROL and other internal data
    1. Allocate per-processor memory page mappings
    2. Initialize processor “corral” – I haven’t delved into this new concept yet, but it appears to be the “right” way to do the old rootkit trick of queueing a DPC to each processor to cause it to “spin” while a single dedicated processor does some hacky work that must have exclusivity.  Surely this is how this new live dump feature is able to accomplish its goal while the system is still running (similar to the crash dump environment which is single thread/CPU, high IRQL, interrupts disabled, etc).
    3. Estimate dump buffer space requirements, discarding certain page ranges
  4. Capture calculated memory pages using page “mirroring” callbacks.  When mirroring is complete (IopLiveDumpEndMirroringCallback), add the standard dump stuff (KdDebuggerDataBlock, KiProcessorBlock, loaded module list, processor data, kernel data, and so on), fill the dump header, and populate the dump bitmap.
  5. Trigger a processor corral state change (4), which presumably simulates the restrictions of a single-processor crash environment while the dump data is written to disk.

I’m sure I didn’t get that exactly right, but I think it’s pretty close.  I’ll continue to update this blog post as I learn more.

LiveDump.exe

I wrote a small utility over the weekend called LiveDump that lets you create both a triage dump using parameter 29 and a complete/kernel dump (user + kernel space or a bitmap kernel dump) using the new parameter 37.

LiveDump must be run as administrator.  Its usage:

LiveDump.exe [type] [options] FileName

Type:
triage : create a triage dump (parameter 29)
kernel : create a kernel dump (parameter 37)

Options (triage dump only):
-p : PID to dump

Options (kernel dump only):
-c : compress memory pages in dump
-d : Use dump stack (currently not implemented in Windows 8.1, 9600.16404.x86fre.winblue_gdr.130913-2141)
-h : add hypervisor pages
-u : also dump user space memory

FileName is the full path to the dump file to create.

Download it here (source code here) – rename the extension to “.7z” and unzip with 7-zip.

The kernel dump capability has options that mirror the new structures and flags discussed earlier.

Since triage dumps are per-process, you will need to specify what process you want to dump. The tool will then create a triage dump of the first 16 threads in that process. The example below dumps the System process.

C:\Users\tester\Desktop>LiveDump.exe triage -p 4 dump.dmp
Attempting to create a triage dump...
Triage dump is for PID 4 with 16 threads.
Dump file 'dump.dmp' written successfully!

Note:  If you are seeing failure status 0xc0000022, you will need to launch the tool from an elevated cmd prompt by right-clicking and selecting “run as administrator”.

One final word:  this tool and research was done in a few hours over the weekend, so I haven’t done much in the way of testing or validating the resulting dumps (other than loading them in Windbg).  Please let me know if you run into any issues (@lilhoser).

Thanks Microsoft!

Ok, one last final word.  I might be just a touch flattered that Microsoft could have been watching my research about how to use the crash dump stack outside of its intended purpose.

My first inkling of big brother watching was with the first release of Windows 8, when Microsoft made extensive changes to the dump stack (incorporating it more tightly in hibernate/resume for the new hybrid boot) and introduced a new read capability in the dump port driver.  The lack of a read capability in the pre-Windows 8 dump stack was something I had to hack around in my original research in order to use the dump stack to read and write to disk.  Given that a read capability would be required for how they leveraged the dump stack in hybrid boot, it’s probably just coincidence.  Furthermore, the fact that there was such a short timeline between my research (April of 2012) and the initial RTM release of Windows 8 (late summer 2012) makes it unlikely my research really influenced their product roadmap.

Of course, this new UseDumpStorageStack bit is the second such occurrence of what I shall call “narcissistic projection of presumed hero worship”, as this represents the first time Microsoft has tried to leverage the dump stack during runtime for non-crashing/non-hibernation reasons.  They added a lot of code to support it too (“crash API”).  Maybe it will start showing up in other places, like a secret storage stack rootkit that NSA uses.

Greetz Alex!

Tagged with: , ,
Posted in blog, source code

Introducing LiveDump.exe

As mentioned in my previous blog post this afternoon, it appears Microsoft has added back the ability to dump physical memory to disk (in the form of a dump file) from user mode via NtSystemDebugControl.  I wrote a quick proof-of-concept tool and generated what appears to be a 250mb kernel bitmap dump:


Loading Dump File [C:\XXXXXX\dump.dmp]
Kernel Bitmap Dump File: Only kernel address space is available
...
Windows 8 Kernel Version 9600 UP Free x86 compatible
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 9600.16404.x86fre.winblue_gdr.130913-2141
Machine Name:
Kernel base = 0x81062000 PsLoadedModuleList = 0x8125b218
Debug session time: Fri Aug 1 18:40:22.806 2014 (UTC - 4:00)
System Uptime: 0 days 1:50:55.292
Loading Kernel Symbols
...
..
STACK_TEXT:
a489fa88 8129c557 842fa2b0 00000000 868d7e60 nt!MmDuplicateMemory+0x67a
a489fab0 8148d3b2 b4e31030 00000000 a489fb34 nt!IopLiveDumpCaptureMemoryPages+0x39
a489facc 811cb332 00000000 00000000 00000000 nt!IoCaptureLiveDump+0xc9
a489fb1c 814e9324 eb98f881 00000025 00c4fc74 nt!DbgkCaptureLiveKernelDump+0x2f4
a489fbf4 811736f7 00000025 00c4fd60 00000028 nt!NtSystemDebugControl+0x691
a489fbf4 7794f804 00000025 00c4fd60 00000028 nt!KiSystemServicePostCall
WARNING: Frame IP not in any known module. Following frames may be wrong.
00c4fdc0 00000000 00000000 00000000 00000000 0x7794f804

When I ran the tool (as an administrator), I saw this in the attached kernel debugger:

livedump
WER/CrashAPI:1959: ERROR ReadProcessMemory failed while trying to read PebBaseAddress
WER/CrashAPI:2068: ERROR Failed to read the peb from the process

 

Hmm, what is “CrashAPI”? Clearly there’s more going on here. Stay tuned!

Greetz to Alex, as usual, for helping me turn numbers into strings and correctly calculating powers of 2.

Posted in blog, conference, source code

Windows 8.1 update: “Live dump” capability

In preparation for refreshing some content in my slide deck for my presentation at Brucon 0x06, I happened upon some functions in the Windows 8.1 kernel I had never seen before.  They were all named with some variation of “LiveDump”.  Here are a few examples:

8129c9fd nt!IopLiveDumpInitiateCorralStateChange ()
8148dbe5 nt!IopLiveDumpWriteBuffer ()
8148d7f3 nt!IopLiveDumpAllocateMappingResources ()
8148d2e9 nt!IoCaptureLiveDump()

I was a little surprised, because I had peeked at 8.1 when it first came out and didn’t notice these.  What the heck is a live dump?  It appears to be a way to save a dump of physical memory (in dump file format) to a flat file using the normal I/O path (it appears the Microsoft developers have an un-implemented option to use the crash dump path – cool!).  Best of all, the functionality is exposed via our old friend NtSystemDebugControl (harkening back to the days of control code 10 that allowed a similar capability).  There are a few restrictions in place, but the good news is the generated dump file doesn’t have to live in the page file (though this would certainly not be the case if the crash path were used).

Stay tuned for more details on this discovery, as well as a tool to leverage the feature.  At Brucon, I will also be releasing a Windbg extension for all things crash dump related.  An upcoming blog post will reveal more details about my upcoming presentation.

Tagged with: ,
Posted in blog, conference

Site domain change

Just a quick note to anyone watching:  I have canceled my web hosting to support http://crashd.mp and moved the content to this free wordpress website.  Just tired of paying web hosting fees.  I might decide to renew the domain name just to hold it, but it won’t host any content.

Posted in blog, site administration

Crash dump stack in Windbg unloaded modules list

Just a quick note:  if you break into your kernel debugger after booting Windows and type ‘lm’, you will probably see some version of this at the bottom of the command output:

Unloaded modules:
fffff880`01846000 fffff880`01854000 crashdmp.sys
fffff880`01854000 fffff880`0185e000 dump_storport.sys
fffff880`0185e000 fffff880`0187b000 dump_LSI_SAS.sys
fffff880`0187b000 fffff880`0188e000 dump_dumpfve.sys

The unloaded modules list is used for debugging purposes and shows modules/drivers that were recently loaded and then unloaded (Volatility has a good post about the unloaded module list here).  But why do we see the crash dump stack drivers here?  Shouldn’t they be loaded in memory in case a crash occurs?  Well, they are, just scroll up a bit:

fffff880`04400000 fffff880`0440e000 crashdmp (deferred) 
fffff880`0440e000 fffff880`04418000 dump_diskdump (deferred) 
fffff880`04418000 fffff880`04435000 dump_LSI_SAS (deferred) 
fffff880`04435000 fffff880`04448000 dump_dumpfve (deferred)

So why do they appear in both the unloaded list and the current loaded list?  Since the crash dump stack always loads during phase 1 kernel initialization and then subsequently when a page file is created and hyber/crash file space is actually available, the stack is actually always loaded at least twice and it’s the second load that persists (when the boot device’s page file is created).  If you have the page file and hibernation features disabled, you should not see the crash dump stack in the loaded module list, but you would see it in the unloaded modules list (I have not tested this!).

Also note that “dump_storport.sys” was unloaded after the first stack load and “dump_diskdump” was loaded in its place on the second load.  These drivers are the same (note their image ranges show identical size) driver, diskdump.sys.  This renaming is part of the load process (for more details, explore the various sections on this website).

Tagged with: , ,
Posted in blog