Repository files navigation

hvpp is a lightweight Intel x64/VT-x hypervisor written in C++ focused primarily on virtualization of already runningoperating system.

Althoughseveralopen-sourceresearchhypervisorsaimedatsimplicityalreadyexist, in my opinion this field isstill somewhat unexplored and needs more open-source projects. This can especially help those who have juststarted exploring virtualization technologies and are looking for small/reference projects. If you're one of them, mybets are that you're really disappointed right now, because all you've got are barely dozen of (great!) projects andhuge pile of Intel Manual pages.

C++ has been chosen as a language for this project because of two reasons:

• The virtualization architecture can be better expressed in OOP concepts (with such objects as VCPU, EPT).
• I didn't find other projects which would use modern C++17 features, except forbareflank. Althoughbareflank is compilable under Windows, Linux and UEFI, on Windows, it uses cygwin to cross-compile the hypervisor. Since the hypervisor is a self contained ELF binary, the Windows kernel is missing the debug symbols for the hypervisor, which prevents easy debugging with WinDbg.

Even though this project is primarily developed for Windows, I've decided to not use traditionalWindows Driver namingconvention (akaDrvCamelCase). Instead, traditional C++snake_case is used. The reason is that hypervisor is very"stand-alone" and doesn't depend on many OS functions. Therefore I decided to treat it as a regular C++ project.

If you want to challenge yourself in low-level programming, my advice would be to try and write a simple hypervisor.During the process you'll get invaluable knowledge and experience and you'll probably discover many new things.For instance here's a selection of some things I've learned thanks to writing this project:

• what areMTRR good for
• NT APIs like
  • MmGetPhysicalMemoryRanges() (and\REGISTRY\MACHINE\HARDWARE\RESOURCEMAP)
  • KeIpiGenericCall - which can be used instead of undocumentedKeGenericCallDpc (but at the cost of higher IRQL)
• thatTraceLogging API (built uponETW) exist
• how Meltdown vulnerability has been mitigated in Windows
• that VMWare uses not only I/O port 0x5658 for backdoor,but also port 0x5659 and it uses almost all GP registersfor both input and output parameters
• thatbugsinotherhypervisors exist (and found away how to fix them)
• that VirtualBoxuses ring 1 for software-based virtualization (i.e. when VT-x/AMD-Visn't available)
• how guest memoryhas been virtualized before EPT (by so called Shadow-paging, trappingpage-faults andmov cr3, <source> instructions in the guest)

Also - as obvious as it might sound - I'd like to point out that if you decide to write your own VT-x hypervisor, you'llNEEDIntel® 64 and IA-32 architectures software developer’s manual combined volumes: 1, 2A, 2B, 2C, 2D, 3A, 3B, 3C, 3D, and 4.So download the PDF - together withAdobe Acrobat Reader - because trust me, you don't want toread and navigate through 5000 pages with browser's built-in PDF reader.

• EPT with identity mappingwith usage of 2MB pages for the first 512GB of physical memory (seeept.cpp).This results in faster translations of the memory. It also means splitting particular 2MB pages into 4kb pages mightbe desired if EPT hooking is required. This process is actually not complicated at all and this repository includesexample on how to achieve that.
• Simple pass-through VM-exit handler, which can handle:
  • exceptions orNMIs
  • CPUID,(WB)INVD,INVLPG,RDTSC(P),MOV CR,MOV DR,IN/OUT,RDMSR,WRMSR,SGDT,SIDT,LGDT,LIDT,SLDT,STR,LLDT,LTR,XSETBV andINVPCID instructions
  • VMCALL instruction (used for termination ofhvpp)
  • VMCLEAR,VMLAUNCH,VMRESUME,VMPTRLD,VMPTRST,VMREAD,VMWRITE,VMFUNC,VMXOFF,VMXON,INVEPTandINVVPID instructions raise#UD (invalid opcode exception)
• Ability to run in VMWare (tested even with KVM - I advise to turn offHyper-V Enlightenments,as it cancause conflicts). VMWare's nested virtualization makes development and debugging ofhypervisors much easier.
• Simple custom memory manager (see mm.cpp). The need for custom memory manager emerges from thefact that you should think twice before calling any kernel function from VM-exit handler, because many of them can becalled at IRQL <= DISPATCH_LEVEL (such asExAllocatePoolWithTag). But in VM-exit handler, interrupts are disabledand yourIRQL is effectively HIGH_LEVEL.
• Detailed code comments, which should explain what the code does and why - sometimes with direct references toIntel Manual for further reading.
• TraceLogging API (which builds on ETW) - the main benefit is it can be used forreally highfrequency logging (10'000+ per second) and it can be used from any IRQL - which makes it a perfect candidate evenfor logging in VM-exit handlers.
  • hvpp already includes tracing VM-exit handler (seevmexit_stats.cpp).
• Various reimplemented classes and functions - such as bitmaps and spinlocks - to avoid calling kernel functions.
• Included simple application (hvppctrl) which should demonstrateCPUID instruction interception, hiding hooksin user-mode applications via EPT and communication withhvpp viaVMCALL

Note:hvpp is compiled as a static library, which is linked with thehvppdrv project.

• Bootstrap of the driver (hvpp,driver.cpp)
  • preallocate enough memory and initialize thehvpp memory manager
  • initialize the logger
• Bootstrap of the hypervisor (hvppdrv,main.cpp)
  • createvmexit_handler instance
• Start the hypervisor with provided VM-exit handler:hypervisor::start(vmexit_handler& handler)
  • initialize virtual cpu (VCPU) for each logical processor
  • assign providedvmexit_handler instance to each VCPU
  • launch all VCPUs via IPI (inter-processor interrupt):vcpu_t::start()
    • setup VMXON region and VMCS:vcpu_t::vmx_enter()
    • vmexit_handler::setup() is called, which allows anyone to initialize the VM-exit handler and/or modify the VMCSbefore the launch (seevmexit_custom_handler::setup() in hvppdrv,vmexit_custom.cpp)
• Hypervisor is now running and handling VM-exits via provided VM-exit handler
• Stop the hypervisor:hypervisor::stop()
  • destroy each VCPU via IPI:vcpu_t::stop()
    • vmexit_handler::teardown() is called and switches into VMX mode (vmexit_passthrough_handler::teardown() doesit byVMCALL instruction)
    • in VMX mode,vcpu_t::vmx_leave() is called - it leaves VMX mode withVMXOFF instruction

Compilehvpp using Visual Studio 2017. Solution file is included. The only required dependency isWDK.

You can runhvpp on Windows 7 or higher. Windows 10 is recommended though, because it supportsTraceLogging.

EnableTest-Signing boot configuration option (note that you'll need administrative privileges to usebcdedit andsc commands):

bcdedit /set testsigning on

Register driver with Service Control Manager (yes, it's important to leave these spaces):

sc create hvpp type= kernel binPath= "C:\full\path\to\hvppdrv.sys"

Now you should restart your computer fortestsigning to take effect, otherwise you'll be unable to start the driver.But before you do, you might want to prepareDebugView from SysInternals andtraceview.exe tool from the WDK (note thattraceview will work properly only on Windows 10).

After restart, launchDebugView andTraceView. InTraceView:

• go toFile -> Create New Log Session, click onAdd Provider
  • pickManually Entered Control GUID or Hashed Name
  • paste916fcd3e-673b-4610-aaba-0b71e28acd40 (arbitrarily chosen, seelib/win32/tracelog.cpp)
  • clickOK
• in the next dialog, leave theSource Of WPP Format Information set toAuto
  • clickOK
• after that, clickNext, which will bring you toLog Session Options dialog
  • inLog Session Name editbox you can give this logging session any name you like, e.g.HvppSession or you canleave it as it is
  • if you desire to analyze this logging session when it's stopped (e.g. withWindows Performance Analyzer)you can optionally enableLog Trace Event Data To File, which saves whole logging session into an.ETL file
  • clickFinish

TraceView is now set-up and ready to showtracelogs fromhvpp. You can launchhvppnow:

sc start hvpp

hvpp now performs various checks and enters VMX mode if they pass. In case of success you should see messagehvpp started in theDebugView.

Runhvppctrl:

hvppctrl.exe

• hvppctrl performsCPUID instruction withEAX = 0x70707668 ('hvpp') whichhvpp should intercept and returnstringhello from hvpp in EAX, EBX, ECX and EDX registers (seevmexit_custom.cpp).hvppctrl should print this string.

• hvppctrl tries to "stealthily" hookntdll!ZwClose function using EPT. The exact process is describedfurther below.

• hvppctrl performs IOCTL, which should instructhvpp to set one-time breakpoint whenIN/OUT instructionmanipulating with port0x64 (keyboard) is executed.

• locatesZwClose function inntdll.dll
  • disassembles first 16 bytes of this function and prints them
    • printed instructions should indicate that this function isNOT hooked yet
  • calls this function (withNULL parameter, this function call will most likely fail with some NTSTATUS error code,which it ignores)
  • prints value ofHookCallCount and it's expected value (explained below)
• hooksntdll!ZwClose fuction usingDetours
  • disassembles first 16 bytes of this function and prints them
    • printed instructions should now indicate that the functionIS hooked (byjmp being first instruction)
  • calls this function (withNULL parameter)
    • instead of original function, the hook function will be called - on each call, it increments variableHookCallCount
  • prints value ofHookCallCount and it's expected value - it should be 1 now, as the hooked function has been calledfor the first time now
• callshvpp byVMCALL instruction andRCX = 0xc1 (arbitrarily chosen),RDX = AddressOfReadPage andR8 = AddressOfExecutePage - this instructshvpp to hide the hook
  • disassembles first 16 bytes of this function and prints them
    • printed instructions should now indicate that the function hook isHIDDEN (by showing original content ofmemory - nojmp)
  • calls this function (withNULL parameter)
    • despite what the memory returned when we read it, the hook function will be called again and theHookCallCountwill be incremented again
  • prints value ofHookCallCount and it's expected value - it should be 2
• callshvpp byVMCALL instruction andRCX = 0xc2 (arbitrarily chosen) - this instructshvpp to unhide thehook
  • disassembles first 16 bytes of this function and prints them
    • printed instructions should now indicate that the function hook isNOT HIDDEN and it should showjmp as afirst instruction again
  • calls this function (withNULL parameter)
    • because the function is still hooked, the hook function will be called andHookCallCount will be incrementedagain
  • prints value ofHookCallCount and it's expected value - it should be 3
• unhooksntdll!ZwClose function
  • disassembles first 16 bytes of this function and prints them
    • printed instructions should now indicate that the function isNOT hooked - it should show the same contentas when this function wasn't hooked
  • calls this function (withNULL parameter)
    • original function will be called, therefore theHookCallCount should not be incremented now
  • prints value ofHookCallCount and it's expected value - it should be still 3

At the same time you should see tracelog messages in theTraceView - they are generated on eachVMCALL and on eachEPT Violation.

When you decide you want to turn off thehvpp, just execute:

sc stop hvpp

• hvpp is designed to virtualize already running OS - i.e. it's not capable of running multiple guests like VMWareor VirtualBox. It also lacks support for any nested VMX operations.
• hvpp is designed to run only on 64bit Intel processors, which support VT-x and EPT. This makes the code moresimple and less bloated.
• hvpp is designed to run only on Windows - future work might focus on Linux.
• hvpp currently doesn't exit VMX mode on sleep or hibernate (S3 and S4 power states) - IntelManual says we should do so - this is known limitation.

This software is open-source under the MIT license. See the LICENSE.txt file in this repository.

Detours is licensed under MIT license (a copy of the license is includedhere).

udis86 is licensed under the terms of the 2-clause "Simplified BSD License" (a copy of the license is includedhere).

SimpleVisor:https://github.com/ionescu007/SimpleVisor

HyperPlatform:https://github.com/tandasat/HyperPlatform

HyperBone:https://github.com/DarthTon/HyperBone

Bareflank:https://github.com/Bareflank/hypervisor

ksm:https://github.com/asamy/ksm

MoRE:https://github.com/ainfosec/MoRE

hyperdbg:https://github.com/rmusser01/hyperdbg

virtdbg:https://github.com/upring/virtdbg

BluePill:https://invisiblethingslab.com/resources/bh07/nbp-0.32-public.zip

Phrack #69:http://www.phrack.org/issues/69/15.html

NOVA Microhypervisor:https://github.com/udosteinberg/NOVA

Finally, I'd especially like to suggest reading interesting writings from Satoshi Tanda (github,twitter):

• http://tandasat.github.io/HyperPlatform/
• http://tandasat.github.io/HyperPlatform/userdocument/index.html
• http://standa-note.blogspot.com/2015/08/writing-hypervisor-for-kernel-mode-code.html

And notes from LordNoteworthy (github,twitter):

• https://github.com/LordNoteworthy/cpu-internals
• https://github.com/LordNoteworthy/windows-internals

If you find this project interesting, you can buy me a coffee

  BTC 3GwZMNGvLCZMi7mjL8K6iyj6qGbhkVMNMF  LTC MQn5YC7bZd4KSsaj8snSg4TetmdKDkeCYk