Virtual to Physical Address Translation (Part 1)

This is going to be one of the most fundamental blog posts for debugging. The importance of virtual to physical memory address translation, will improve your understanding of much of the other aspects of Windows Memory Management. We need to understand PDEs, PTEs, TLB, PFNs and Look Aside Lists.

x86 Kernel Address Space Layout

To begin, lets take a look at the the layout of the Kernel Virtual Address Space, and the difference between x86 and x64. The Kernel Virtual Address Space is divided into sections for different aspects of the operating system.

Looking at the above diagram, we can see that the Kernel Address Space and the User Address Space have been divided equally, this is normal for a x86 system, whereby 4GB of addressable memory is divided into two 2GB parts. However, an exception to this rule, is user processes can be large address space aware.

The address space would then appear to something like the above in the diagram. For a process to be large address space aware, the IMAGE_FILE_LARGE_ADDRESS_AWARE flag must be set in the image header for the executable file of the process.

We can see using the WinDbg !dh extension, that the image header of the NT Kernel Module is able to support addresses beyond the 2GB address space.

Now, let's look at what the Virtual Memory Address Space consists of, and how it different on x64 systems. On x86, the Kernel Address Space runs from FFFFFFF to 80000000, and consists of:

FFFFFFF

• HAL
• Crash Dump Drivers
• Nonpaged Pool System space
• System PTE
• Paged Pool System space
• System Cache
• System Cache Structures
• Process Working Set Lists
•  Page Tables
• Session Space
• System Mapped Views
• Special Pool
• Initial loading of boot drivers and HAL by the NTLDR 

  80000000

x64 Kernel Address Layout

Since, the address space is much larger on x64 systems, the address space isn't divided into two equal parts like x86, instead memory regions are mapped according to their size.

On x64 systems, 32-bit programs will be able to use 4GB of addressable space, due to the much larger addressable space for x64 (16 EB). However, there has been limitations set to the current addressable space of x64, due to hardware limitations and the lack of need for the amount of memory. These limitations have lead to the address space being limited to 48-bit, which means the other 16-bits have been used to create canonical addresses.

Please note, that non-canonical addresses will result in a access violation exception. Furthermore, it's important to understand that the addressing is based upon the CPU Architecture and the MMU.

The operating system licensing restrictions can also prove to be a limiting factor.

Dynamic Virtual Address Type Allocation and Management

As a result of the Dynamic Virtual Address Space management used for the Kernel Address Space on x86. To achieve, this MiInitializeDynamicVa is called to create the dynamic address ranges for the supported virtual address types listed below, in the enumeration called _MI_SYSTEM_VA_TYPE:

After everything has been loaded and initialized, virtual address space can be dynamically allocated with MiObtainSystemVa and subsequently released with MiReturnSystemVa. Dynamic Virtual Address Management brings many advantages to x86 and x64 systems , such as smaller memory reservation and consumption, since page tables do not have to allocated with unused memory addresses, these memory addresses can be allocated on demand when they are used. Furthermore, memory can be dynamically reserved leading to better memory management.