Debugging Stop 0x119 - Invalid Fence IDs

This topic was discussed briefly on Sysnative, but it never seemed to come to any conclusive answer about the mystery of Fence IDs and Stop 0x119's. Stop 0x119's are quite rare, so you probably won't debug one of these bugchecks, but if you do find yourself in this situation, then you've come to the right place because in this post I'll be talking about Fence IDs. Just as a pre-warning, I will not completely discuss the internals of DMA (Direct Memory Access). That will need to wait for a separate topic. I have yet to explain the Heap fully too.

Stop 0x119 - General Overview

As you can see, the bugcheck is already inconclusive in terms of parameters. It only provides the type of violation detected by the GPU Scheduler, but doesn't provide an answer as to how or why it happened. The second parameter is supposedly the received fence ID, personally I believe the third parameter is the prior Fence ID, since Fence ID can be updated after a Fence instruction. However, it could also be the expected Fence ID.

Before, we go into the exact details, let's take a quick look at the call stack.

The DxgNotifyInterruptCB routine is used to notify the DirectX Graphics Kernel of a ISR interrupt (Graphics Hardware), when a DMA buffer has completed. This where the problem begins, once a DMA buffer has completed, the driver will need to provide the Fence ID associated with that DMA buffer. If the Fence ID is not the same as originally associated with the DMA buffer, from the DxgkDdiSubmitCommand routine, then the system will bugcheck with Stop 0x119.

Now, let's take a more depth look at Fence IDs and DMA buffers. Direct Memory Access (DMA) is used to enable a hardware device (in this case a GPU) to access memory without using the CPU. There's a few different methods of DMA, but for now if you don't understand DMA, then just remember that buffers are locations in memory described in a buffer descriptor list (uses MDLs).

Creating DMA Buffers

A DMA buffer is created with the DxgkDdiBuildPagingBuffer routine, which enables allocations to moved to and from the GPU memory. The paging buffer is described by the DXGKARG_BUILDPAGINGBUFFER data structure, the DxgkDdiBuildPagingBuffer routine takes a pointer to the data structure.

Paging Buffers are built for memory operations, with the most common with being memory transfers. The memory transfer operation simply moves data from one memory location to another memory location. Paging Buffers additionally allow more video resources to be allocated, if physical GPU memory limitations do not allow this. The video or graphics resources will be paged into RAM.

The next second step is submit this paging DMA buffer to the GPU, with the use of the
DxgkDdiSubmitCommand routine. The DxgkDdiSubmitCommand routine takes a pointer to the DXGKARG_SUBMITCOMMAND data structure. It also associates a Fence ID with the DMA buffer, if the driver is going to write into the DMA buffer or it wishes to queue the DMA buffer into the ring buffer. 

A Command Buffer is the User-Mode version of a DMA Buffer, and therefore a Command Buffer will need to be converted to a DMA Buffer for the use in Kernel-Mode. The differences between the two buffer types are explained in this article from Microsoft - Introduction to Command Buffer and DMA Buffer

The third step is to call the DxgkDdiPatch routine to assign physical addresses to the video resources used by the DMA buffer. The DXGKARG_PATCH data structure is used to describe the DMA buffer to be patched or assigned with the physical addresses. These physical addresses could be related to GPU memory, RAM or PCI memory.

The DxgkDdiSubmitCommand is then called again, to queue the DMA buffer for the GPU, the Fence ID is used to identify the DMA buffer and will be checked again when the DMA buffer has completed. This seen with the use of the an interrupt, and the DxgkDdiInterruptRoutine which handles hardware based interrupts from the display adapter. With this function or routine (in driver terms), then there is some control logic to determine if the display adapter generated the interrupt. The Fence ID will be read by the GPU here.

If the display adapter, did generate the interrupt, then the request will be completed immediately or queued as a DPC to be processed later.

I've explained the use of DPCs in previous blog posts, but just for convenience, you can view the queued DPCs with the !dpcs extension. The DxgkCbQueueDpc is used to queue the DPC.

The DxgkCbNotifyInterrupt routine is then called to notify the GPU scheduler, that the DMA buffer has completed. The previously reported Fence ID is compared with the current Fence ID, and if the current Fence ID is newer, then the Fence ID will be updated.

Looking back at the call stack, we can see the video scheduler then checks if the graphics driver has reported the Fence ID. If the driver hasn't reported a Fence ID after the last the hardware interrupt, then the DxgkDdiQueryCurrentFence routine is called to query any pending fence instructions.

Fences

Going back to DMA and GPUs, the graphics driver can use a Fence which is a instruction of 64 bits of data and a fence address, and insert it into DMA buffers, this causes the GPU to read the fence address and then write the fence data at the fence address. There are two types of fences: Regular (used in User-Mode) and Privileged (Kernel-Mode). It's important to remember that if a fence address was accessible in User-Mode, and is used with a Privileged Fence instruction, then malicious software may change the data the Kernel was expecting to receive.

Additional Reading:

DxgkDdiQueryCurrentFence

Supplying Fence Identifiers 

Windows Vista and Later Display Driver Model Operation Flow

Submitting a Command Buffer