Stealth File Patch

Earlier this year, I created a challenge for the Source Boston 2013 conference based on the dump filter hijack technique described on this website.  The challenge involved a series of file patches that the user must indirectly coerce by modifying the dump stack logging file.  The source code for the crash dump filter driver used in this challenge is hosted on google code.  You can also download copies of the dump files generated after key steps are completed ( post-stage 1 dump and post-stage 3 dump – please rename “.key” to “.dmp”).

The ultimate goal is for the user to coerce the crash dump filter driver to read the CTF key off disk using the crash dump I/O path.  As outlined in the dump filter hijack technique, this coercion is possible through manipulating the contents of the undocumented DumpStack.log.tmp file, which is exposed to the user by the challenge’s dump filter driver (disguised as storport_lsi.sys).  As part of solving the challenge, this manipulation causes the dump filter driver to leverage internal logging functions in crashdmp.sys to read and write arbitrary locations on disk (through the crash dump I/O path).

The contestant needed to research and understand various aspects of the crash dump mechanism in order to progress through the challenge.  The primary goals of the challenge were to:

  1. Educate the user on how the crash dump stack works
  2. Illustrate that the crash path represents a completely separate path to disk than normal system operation
  3. Demonstrate that the crash path can be used outside of the normal crash mechanism


Contestants were given a virtual machine pre-loaded with Windows 8 x86 Professional using a conference-provided BizSpark license.

The system will load storport_lsi.sys, a crash dump filter driver that handles all runtime aspects of the challenge.  It also bugchecks the system every 4 minutes.  This file is named “storport_lsi” to appear less suspicious.  The name comes from “storport.sys”, one of three storage port drivers provided by Microsoft, and “LSI_SAS.sys”, a popular storage miniport driver provided by LSI corporation.

A fake key file is used in the challenge (“??C:Userstesterdesktopkey.txt”).  The real challenge key is stored at offset 0x300 in the file c:WindowsSystem32Driversbeep.sys (the key is “key{theswordofathousandtruths}” which is stored on disk as base64-encoded blob “a2V5e3RoZXN3b3Jkb2ZhdGhvdXNhbmR0cnV0aHN9”).


The VM provided at the CTF was already configured.  If you wish to setup your own VM, follow these steps:

  1. Create an administrative scheduled task (or use this one:  disableautorepair-taskscheduler.xml – rename from “.key” to “.xml”) that runs on startup which executes various required BCD commands described in Table 1 below.  NB: If the login account has a blank password, you have to disable the Local Security policy “Accounts: Limit local account use of blank passwords…”
  2. Set the value names specified in Table 2 below in the registry key HKLMSystemCurrentControlSetCrashControl
  3. Determine the offset and size of beep.sys using the tool GetDiskRuns.exe (rename “.key” to “.exe”)

    1. If it differs from the preset values of offset=26808320 and size=6144, the storport_lsi.sys driver must be updated and rebuilt for that platform
    2. It is also possible to load a debug build of the storport_lsi.sys driver (rename “.key” to “.sys”) and view debug output to determine this information
  4. Copy storport_lsi.sys (rename “.key” to “.sys”) to C:\Windows\System32\Drivers
  5. Copy fake key.txt (rename “.key” to “.txt”) to the desktop
  6. Patch beep.sys to contain the challenge key, as described below, or just use this patched copy (rename “.key” to “.sys”)
  7. Reboot
  8. Take a snapshot!

To patch beep.sys to contain the key, do the following:

  • Enable dumpstack logging (see challenge Script section below)
  • Reboot
  • Modify c:dumpstack.log.tmp to contain parameters necessary to patch the key in:
??c:windowssystem32driversbeep.sys|0x300|0x61, 0x32, 0x56, 0x35, 0x65, 0x33, 0x52, 0x6F,0x5A, 0x58, 0x4E, 0x33, 0x62, 0x33, 0x4A, 0x6B, 0x62, 0x32, 0x5A, 0x68, 0x64, 0x47, 0x68, 0x76, 0x64, 0x58, 0x4E, 0x68, 0x62, 0x6D, 0x52, 0x30, 0x63, 0x6E, 0x56, 0x30, 0x61, 0x48, 0x4E, 0x39
  • Save the log; this will trigger a bugcheck
  • Reboot and verify the file has the key
  • Disable dump stack logging (reboot required), remove the log files and cleanup any other traces:
    • Remove the copy of storport_lsi.sys made in Via100d2.sys and create a new copy of storport_lsi.sys
    • Erase any minidump/dump files
    • Clear event logs
    • AppDataLocalTempWER*.xml

BCD Command


bcdedit /set {current} recoveryenabled No Disables auto repair (which will launch on boot after 2 successive crashes if not disabled!)
bcdedit /set {current} bootstatuspolicy IgnoreAllFailures Disables auto repair (which will launch on boot after 2 successive crashes if not disabled!)
bcdedit /set {current} testsigning On Disables kernel mode code signing (KMCS) policy and set the machine to boot in test signing mode

Table 1:  BCD commands

Registry Value Name Set Value To Purpose
CrashDumpEnabled 3 Sets the dump type to “small memory dump”, i.e., minidump
MinidumpDir %SystemRoot%Minidump Sets the minidump output folder
DumpFilters append “storport_lsi.sys” with no quotes or newline Enables our crash dump filter driver
Autoreboot 0 (OPTIONAL) disables auto reboot after crash

Table 2:  Registry values

Challenge Script

–==[ STAGE 1:  Enable dump stack logging ]==–

To get to stage 2 of the challenge, the user must enable dump stack logging (c:DumpStack.log.tmp) via the registry:

HKLMSystemCurrentControlSetControlCrashControlEnableLogFile = 1

HKLMSystemCurrentControlSetControlCrashControlLogLevel = 0xfffffffe

After the system bugchecks, if logging has been enabled, the message “No file specified yet!!” will appear and Stage 2 begins after reboot.  If logging has not been enabled, the message “U MAD YET BRAH?  Dump stack logging is disabled” will appear and Stage 1 repeats after reboot.

Note:  There is a bug somewhere in my code that prevents me from modifying the bugcheck message at this stage in the challenge.  I wasn’t able to isolate where in the code it happens, so instead of writing an obvious message to the screen, the bug check code and parameters are used to write a hint in ASCII hex digits (“dumpstacklogging”):  BugCheck 0x64756d70 {0x73746163, 0x6b2e6c6f, 0x672e746d,  0x7000000}.  Note that the blue screen in Windows 8 will only show the bugcheck code – to see the rest of the string stored in the bugcheck parameters, the user will need to load the dump file in Windbg.

–==[ STAGE 2:  Manipulate log contents such that a file is read through crash I/O path ]==–

On system startup, storport_lsi.sys will adjust the handle and security settings on the dump stack log file so that the user can edit it with administrator privileges (the exclusive handle normally held by crashdmp.sys is also removed).   To proceed through this step of the challenge, the user must realize they can edit the file and that if a full file path is specified in the contents of the log file, the contents of that requested file will be read using the crash I/O path and copied into the dump stack log file whenever the crash path is used next (ie, when a crash occurs or the system hibernates).  In order to do this, the user will need to edit the log file as follows:

  1. Run Notepad as Administrator
  2. Open C:DumpStack.log.tmp
  3. Clear the contents
  4. Type a valid path in the format ??C:<full_path_to_file>
    1. A “valid path” is defined as any file on the boot volume that is not one of {the dump file, the dump stack log file, the hiber file, the page file} and is at least 512 bytes in size (required for paging I/O)
  5. Save over the log file, ignoring any warnings that the file is open in another process.

Storport_lsi.sys will periodically check the contents of the dump stack log file during normal system operation.  When a valid file path is saved to the file, storport_lsi.sys will “stage” this file to be copied into C:DumpStack.log.tmp and immediately bugcheck the system (so the user knows they did something right).

If a valid file is staged, or the bugcheck interval expires (whichever is first), storport_lsi.sys will bugcheck the system.  If no valid path is found in the log file, storport_lsi.sys will display the bugcheck message “No file specified yet!!” and upon reboot, Stage 2 will begin again.

If the user specifies any valid file other than the key.txt file on the desktop, its contents (up to 1mb) will be copied to DumpStack.log.tmp and will appear there on system restart.  The bugcheck message will read “File copied.  Cool story, bro” and Stage 2 will begin again on restart.

If the user specifies the fake key file, the bugcheck message will read “Key file is empty, nice try!  [path]|[0xoffset]|[0xbyte1],[0xbyte2]…” and stage 3 begins on restart.

–==[ Stage 3:  Manipulate log contents to cause an arbitrary file write through the crash I/O path ]==–

At this point the user should realize they were able to copy a file’s contents into DumpStack.log.tmp by manipulating the dump stack log and that they need to now fill it with a special command to cause a write somewhere (they aren’t sure what to write or where).  The point of the final step in the challenge is to get them to WRITE something to any file on disk through the crash I/O path.  This is done by supplying a command in the format [path]|[0xoffset]|[0xbytes]:

[path] – full path to write to, same format as in read operation
[offset] - offset into the file to write (hex number)
[bytes] - hex bytes to write at that offset (comma-separated hex bytes)

In order to complete the challenge, the user must cause a write to any arbitrary file on disk.  Storport_lsi.sys will know it’s a unique write because as part of being a crash filter driver, its DumpWrite callback will be notified whenever the crash dump path is used to write to disk.  Inside its DumpWrite, storport_lsi.sys can check to see if the write is destined for a file other than the dump file or the dump log file, and if so, assume the user successfully figured out this stage of the challenge.

If the user triggers the write, then the DumpWrite callback will append the key to the generated dump file.  It will also disable the storport_lsi.sys driver by simply renaming it to “Via100d2.sys”.  The final bugcheck message will be “Troll complete.  We got a badass here.”

Note: Users are likely to completely hose the VM here, if they overwrite the boot sector or a driver!  Be sure to have a snapshot ready.

–==[ Stage 4:  Key dumped to crash dump file ]==–

Upon reboot after completing stage 3, the challenge is complete and the key is stored in the minidump file generated in the Windowsminidump folder.  The produced minidump is most likely corrupt and might not be loaded by Windbg or parsed successfully by dump parsing tools (such as dumpchk).  This is half-way intentional in order to make it a little more laborious to extract the key, but also because I’m too lazy to devise a strategy of what memory blobs I should not be overwriting in my dump callback when I’m appending the key to the dump file (there is no easy way to know if a particular MDL passed to my driver’s callback is a portion of the loaded module list, the KRPC, thread stack, or any other piece of debug data being appended to the dump).  But the key will be peppered all over the place in the dump file, so an astute observer should notice it.

Conclusions:  Design Considerations

When designing this challenge, I wanted to make it hard enough such that live forensic techniques to retrieve the key would be unsuccessful, but also intuitive enough such that the user would naturally be steered to the solution (to use the crash I/O path).  How to protect the key is dependent on where it is stored.

The first option, which is ideal, is to show the user the key file on the desktop, but not allow them to access it through the normal path.  This setup would make it clear to the user that they need to open that file, but that normal methods aren’t working.  This blockade would naturally drive them to consider other ways to access the disk (ergo, the crash path).  There are three obvious ways to prevent trivial live disk access to the file:

  1. Hook the disk port driver in the normal I/O path to deny request to physical sectors
  2. Install a file system filter and reject access at the FS layer
  3. Hold an exclusive lock on the key file in our driver

Options 2 and 3 are susceptible to live raw disk access and scsiport pass-through requests.  Other remote disk forensics like F-Response (over iSCSI) might also defeat options 2 and 3.  Option 1 is a difficult task, both for stability and completeness reasons, and it is made even worse by the fact that it must work on Windows 8, which has moved exclusively to using the storport.sys driver.  Note that we assume trivial offline attacks (eg, access to the vmdk file) are not allowed, since the VM is in the cloud.

The other option is to embed the key somewhere on disk where the user won’t know to look.  This option is acceptable from a protection standpoint, because it makes live forensics moot (yes they can read all the sectors, but are they going to grep for base64 strings??) and provides a reasonable level of protection for the key.  The primary exposure is in the storport_lsi.sys driver itself, which can be dumped from memory and reversed.  However, the user would have to be a solid reverser AND have deep windows internals knowledge to understand some of the structures they would need to reverse out (primary disk run mapping pairs).  While it’s certainly doable, this option gives us the best of both worlds – protecting the key without severely destabilizing the system.  There are two primary drawbacks however:

  • The static offset stored in storport_lsi.sys must be retrieved when the challenge is setup and it’s possible for the disk runs for beep.sys to change
  • How do we make the discovery of this aspect of the challenge intuitive to the user?

If the disk runs change for beep.sys (e.g. through defrag), though unlikely, this would cause the challenge to fail completely, because storport_lsi.sys won’t be able to retrieve the key properly.  This is an acceptable risk since it is a very unlikely scenario.  To solve the second issue, I decided not to force the user to have to gain any knowledge or awareness of the keys location.  It’s just a hiding place and the storport_lsi.sys driver knows how to extract it through the crash I/O path automatically once the user has completed the challenge steps.

While the normal I/O path could be used to retrieve the key from beep.sys, there is no realistic way the user would know where to look, since storport_lsi.sys does not use file names to pull the key out of beep.sys – it uses a special I/O control code issued to the file system driver, which stores the file offset in special structures called disk runs.  The user could reverse storport_lsi.sys, locate the code that queries the file system driver for FSCTL_QUERY_RETRIEVAL_POINTERS, and reverse the structures, but this requires advanced skills and non-trivial amount of time.  The key is stored in the beep.sys driver binary using the crash dump I/O path, so no MAC time modifications or other forensic evidence exists.

Another consideration for this challenge was whether to use crash or hibernation, since the crash dump stack operates both.  Originally I wanted to use hibernation events instead of bugcheck/crashing, for numerous reasons:

  • Easier and cleaner to code
  • Smoother experience for the user (though not as fun)
  • Hibernation would allow for more/varied challenge features
  • Easier to maintain challenge state
  • More options for hiding the key in memory

However, it is simply not possible to use guest hibernation in VMWare.  This is a known issue.  VMWare workstation actually installs a service through VMWare Tools that disables guest hibernation (via ZwPowerInformation call to tell the OS not to reserve space for the hiberfil.sys), presumably because their own ‘Suspend’ feature would conflict.  Sure enough, re-enabling guest hibernation (by removing the service) causes VMWare to choke on resume from hibernation.


Things to try first:

  • Reboot and retry whatever operation caused an issue
  • See the known issues below

Note that if anything goes wrong in the challenge, storport_lsi.sys can be disabled by simply deleting it from the registry key or removing the file from disk in recovery console by hitting SHIFT+F8 on startup (must hit fast and repeatedly — there is about a 10 second delay before the machine might bugcheck!).  From the startup boot menu, select:

  • Choose Defaults or Other Options
  • Other Options
  • Troubleshoot
  • Advanced Options
  • Command Prompt

Now enter from the prompt: ‘del d:windowssystem32driversstorport_lsi.sys’ (repair mode mounts the boot drive as “D:”).

Rather than deleting the driver, first try deleting the offending ‘d:dumpstack.log.tmp’ file in the same manner as described above.  If a troublesome file is staged in the dump stack log, it will remain there and cause an endless cycle of reboots.

Other more far-fetched things that “can possibly”/”most certainly will” break the challenge:

  • Beep.sys must exist at offset 26808320 and be of size 6144 – this should have been taken care of during setup, but it’s possible the offset changed.  See the Considerations section.
  • Static function signatures for internal functions in crashdmp.sys and ntoskrnl are extremely fragile and are likely to break if the version of the OS and patch level do not exactly match what the driver was coded for
  • If the internal context structure used by crashdmp.sys changes at all, the static offsets used by the technique must be updated
  • If the behavior of any of the internal crashdmp/ntoskrnl functions changes (unlikely but possible), everything will break
  • The code will only work on x86 (this is mostly an artificial limitation)
  • Enabling a whole-disk encryption solution will encrypt the dump file contents and its interaction with other dump filters has not been tested.
  • ELAM (Early Load Anti-Malware) could possibly cause problems, but this is just a hunch
  • Windows auto repair causes problems with hanging after a reboot from a crash, so it is disabled

 Known Issues

  1. Due to an unknown bug, the kernel paint/draw functions used to write the BSOD messages are unstable, particularly during stage 1.  Therefore, instead of displaying the BSOD message in Stage 1, the message is stored as bugcheck parameters and must be retrieved manually by the user in Windbg.  Please see stage 1 of the Challenge Script below for details.  Stage 2 and 3 still attempt to write the messages but they could also be unstable.  If the user sees a BSOD with no message, tell them to just reboot and repeat whatever action they performed.  If this fails, just give the messages out as hints.
  2. The file specified by the user in Stage 2 must be at least 512 bytes large or I won’t stage the file.  This is a limitation in paging I/O I don’t fully understand. Base system
%d bloggers like this: