Debugging and Understanding Stack Overflows

Another blog post for my readers to enjoy (hopefully), tonight I'm going to explain how stack overflows are caused and how to debug a stack overflow. They are usually always result in a Stop 0x7F with a Double Fault (8) as the first parameter.

Firstly, I think it's important to understand what is a Stack Overflow and how it's produced. It's a common problem when using Recursive functions, that is to say, functions which call themselves. The stack is a area of memory reserved for a process, which is used to store the local variables and function return addresses. A Stack Overflow, typically happens when this area of memory is completely exhausted (used up), and then the function calls return into areas of memory which do not belong to the process. It can also be caused by variables which are too large.

It's important to understand, that the current stack provided may be corrupt or completely useless, and this will explain the reasoning for the debugger to use the .tss command on the TSS (Task State Segment).

A TSS is a structure used by the Windows Kernel to keep track of Task Management, and any running tasks. The TSS will contain the stack pointers, which is important in relation to this type of problem. I won't explain completely the internals of TSS.

Anyhow, when a interrupt occurs (exception in this case), the kernel looks at the TSS and loads a stack frame pointer field into the Stack Segment, which enables the Kernel to have a different stack to the program, therefore preserving the stack if anything should go wrong. These stacks used by the TSS are stored in a Interrupt Stack Table, which will allow the Kernel to load the other stack, and allows the Kernel to use good stacks in case of a serious error like the Double Fault we are facing.

References:

Kernel Stacks (x86/x64) 

Task State Segment Wiki 

Modern Memory Management Part 2 

Difference between a Stack Overflow and Segmentation Fault

Debugging a Stack Overflow