Debugging Stop 0x128 - Worker Routines and I/O Priorities

This bugcheck isn't documented with much information in the BSOD Index, so I've decided to give it some investigation, and upon viewing a Minidump, it seems it may be one of those bugchecks which needs a Kernel Memory dump.

Firstly, let's do what WinDbg suggests and run the ln (list nearest) on the address, to find the worker routine or function call which has caused the problem.
 

Just for demonstration purposes, I've used a different dump file, to help you understand the key concepts of this bugcheck.

Windows has threads which are called System Worker Threads, these threads are used for completing work for other threads, when their IRQL Level does not permit them to complete a certain task, therefore a driver can use a System Worker Thread to complete the work for them. For example, a driver thread running at IRQL Level 2, may need to access paged pool, however, this is inaccessible at such a IRQL Level.

When a driver requires such as service from the System, a driver can call IoQueueWorkItem to place work item on the queue dispatcher object, the System Worker Threads then view these queue to see which tasks need to be completed. This routine is called at IRQL Level 2 or below.

We can view the work items added to the queue in WinDbg, by using the !exqueue extension. There are three different types of work queues and system Worker threads. 

• • Critical Worker Threads - have a thread priority of 13, and will process important work items and data, they usually have their stacks in physical memory, especially on Server systems. 
  • Delayed Worker Threads - have a thread priority of 12, and can have their stack paged out to the disk in a paging file. You can use the !stacks extension to see which Kernel Stacks have been paged out.
  • Hypercritical Worker Thread - have a thread priority of 15, and the stack is also maintained in physical memory. These queue is used for freeing terminated threads.
    

There is also a few flags which you can use with the !exqueue to view additional information. Check the WinDbg documentation for more details.

Before, work items can even be added to the queue, there is a few other things which must happen in order to create a work item for use.
The system must create a structure called IO_WORKITEM, by using the IoAllocateWorkItem or IoIntializeWorkItem. IoAllocateWorkItem take one parameter, which is usually a pointer to the device object of the caller, so the Work Item can later be added to the queue.

The Work Item is then associated with a routine, added to the queue with IoQueueWorkItem. After the Work Item has been used, and is no longer required it is freed. The routine for freeing the Work Item depends on how it was created; IoFreeWorkItem is used for IoAllocateWorkItem, and IoUnintializeWorkItem is used for IoInitializeWorkItem. These freeing routines can only be called when the Work Item isn't in a queue.
 
If a routine takes too long to process, it can cause a deadlock, therefore some drivers may need to create their own System Thread with PsCreateSystemThread. Only one thread can be released by a queue dispatcher object becomes signaled.

nt!IopProcessWorkItem is used when a System Worker Thread processes a work item, the work item's associated routine is called at this point.

We can use the !thread extension to gather the thread address and IO Priority level to match it to the work item stored in the queue.
There are five different I/O Priority Levels, and a each IRP is placed on a certain queue depending upon it's priority.
The I/O Priority names to Priority numbers seen in the thread, can be found in a enumeration called _IO_PRIORITY_HINT:

 Note these IO Priority levels are just a hint, and this does not mean they will be followed. Drivers can set the IO Priority on a IRP with IoSetIoPriorityHint. Higher priority IRPs are generally processed before lower priority IRPs.

 The priority levels are as follows:

• Critical - Memory Manager
• High -File Systems
• Normal - General Programs
• Low - Prefetcher
• Very Low - Background Services/Activities

To conclude, this problem is most likely caused by drivers, and therefore I recommend the use of Driver Verifier.

References:

WDK Documentation
I/O Manager and Vista