Introduction

VirtualBox is one of the most widely used virtualization platforms, with support for multiple host operating systems, including Windows and Linux. There is, however, a notable difference between the two: VNC support is available out of the box on Linux, while Windows users have no equivalent option.

Rather than relying on third-party remote desktop software or the standard VRDP implementation, I decided to build a custom Windows version of VirtualBox with native VNC support. What appeared to be a straightforward task turned out to be a considerably more demanding project, requiring approximately two and a half days of work.

Why VNC?

VNC is a widely adopted remote desktop protocol, supported by many clients such as TigerVNC, RealVNC and TightVNC. My goal was to replicate the Linux experience on Windows: connecting directly to a virtual machine with a standard VNC client, without installing additional desktop-sharing software on the host.

VirtualBox already offers a modular remote desktop architecture through VRDE, and its source code includes a VNC implementation. The challenge was making it work reliably on Windows.

The Build Environment

Compiling VirtualBox on Windows requires an extensive toolchain. The environment I prepared included:

  • Microsoft Visual Studio 2022 Build Tools
  • Git
  • Windows 10 and Windows 11 SDKs
  • Windows Driver Kit (WDK)
  • Yasm and Dasm
  • Open Watcom
  • Qt 6.12.0
  • WiX Toolset
  • Flex and Bison

In addition, I built libvncserver separately using MSVC and integrated the resulting library into the VirtualBox source tree.

Why Building the VNC Extension Pack Alone Was Not Enough

A fundamental obstacle was the fact that simply compiling the VNC Extension Pack was not enough. The official VirtualBox distribution provided by Oracle does not load custom extension packs that have been compiled independently and lack a digital signature. Consequently, it was not possible to create the VNC component and install it alongside the official version.

Instead, it was necessary to recompile VirtualBox itself, integrating VNC support directly into the build. This required adjusting the build configuration and making several modifications to the KMK override files. The compilation process did not succeed on the first attempt; various issues had to be identified and resolved over the course of several iterations.

Specifically, an official VirtualBox release, such as the one distributed by Oracle, is not compiled with the following flag:

VBOX_WITHOUT_HARDENING = 1

This means that the official build runs with Windows hardening enabled. In this mode, VirtualBox verifies the integrity and digital signature of the executables and modules loaded by its processes, and refuses to load anything that does not meet these requirements. An independently compiled Extension Pack, which is not signed by Oracle, therefore cannot be loaded.

Building VirtualBox with this flag set disables these checks, so that custom, unsigned components can be loaded. This is the reason why a full rebuild was unavoidable. It is also worth noting that the flag only affects user-mode binaries: it does not remove the requirement for signed kernel-mode drivers, which is addressed separately in the section on driver signing.

Packaging the Installer

Once compilation succeeded, I generated a Windows MSI installer with the WiX Toolset. I intentionally excluded the following components, which were not needed for my use case:

  • VirtualBox Web Service
  • Guest Additions ISO

Rebuilding the Guest Additions ISO was unnecessary, as the official ISO distributed by Oracle is fully suitable for this setup. The outcome was a custom MSI installer containing my modified VirtualBox build.

Tip

The ISO files for the Guest Additions are not included in the source code or in the precompiled versions; however, they can be downloaded via VirtualBox itself when you choose to install the VirtualBox Guest Additions from within a VM.

Driver Signing

Driver signing was another significant challenge. VirtualBox depends on kernel-mode drivers, and my custom build produces drivers that are not signed with Oracle's certificates. To load them on my Windows host, I used my own test-signing certificate, which involved:

  • Signing the generated drivers with the custom certificate
  • Installing the corresponding public certificate in the appropriate Windows certificate store
  • Enabling Windows Test Signing Mode

This is an important limitation of the current build. Windows Test Signing Mode must remain enabled for these test-signed drivers to load. Installing the certificate alone is not enough to satisfy Windows' standard driver-signing policies.

Anyone who wishes to use the precompiled version will therefore need the MSI installer, the public certificate, and the appropriate test-signing configuration. The private signing key will not be distributed.

Results

After approximately two and a half days of troubleshooting, configuration changes and rebuilds, the project achieved its goal. VNC works correctly on my custom Windows build of VirtualBox. I can connect to a running virtual machine with a standard VNC client by specifying the host address, port and password configured through VBoxManage. No external VNC server is required on the Windows desktop.

The result is native VNC access to VirtualBox virtual machines through the VRDE architecture, which is exactly what I set out to build.

Next Steps

I plan to publish the project on GitHub, including the source modifications, build configuration and documentation. The repository will provide detailed build instructions so that other developers can reproduce the process without starting from scratch.

The aim is not to replace the official Oracle distribution, but to offer a community-built alternative for Windows users who need VNC support.

Conclusion

This project proved considerably more complex than anticipated. The VNC integration itself was only one part of the work: preparing the Windows build environment, integrating libvncserver, modifying the KMK configuration, producing the MSI installer and handling kernel driver signing were all essential steps.

I am pleased with the outcome. When a feature is missing on your platform, sometimes the most effective solution is to build it yourself.