• Overview: Where we were, where we’re going.
• Old Code, Old Problems: ‘fun’ code issues found while scaling out.
• Rethinking Compiler Tooling: How can the compiler rise to the scaling problems?
• A New Approach to Referencing an IFC.
• Playing Nice With Precompiled Headers (PCH).
• Selecting a Launch Project: REDACTED.
• Looking Ahead: Throughput.

Overview

Last time we talked about how and why header units can be integrated into a large cross-platform codebase like Office. We discussedhow header units helped surface conformance issues (good) and expose and fix compiler bugs (good-ish).

We talked about how we went about taking “baby steps” to integrate header units into smaller liblets—we’re talkingsomething on the order of 100s of header units. This blog entry is all about scale and how we move from 100s of headerunits to 1000s of header units, including playing nicely with precompiled headers!

Office’s header unit experiments continued by following the charge that we left you with in the last blog post: “HeaderUnit All the Things!”. From that perspective we were wildly successful! By the last count we were able to successfullycreate over 5000 header units from liblet public headers. The road to reach that milestone wasn’t always smooth, andwe’ll cover some of the challenges.

We’d like to highlight that the recent release ofMSVC 17.6.6 makesthis one of the best times to get started with header units! The release contains the full set of fixes that werediscovered in cooperation with Office. The full set of fixes will also be available in 17.7.5 and 17.8 preview 2.

Old Code, Old Problems

While scaling out we encountered quite a number of complications. Some fixes involved updating Office code, whileothers needed to be solved on the compiler side. We’ll present just a few examples from each bucket.

Symbolic Links

The way Office sources coalesce is a mix between sources populated by git and libraries populated by NuGetpackages which are then linked into a build source tree via symbolic links. From a build perspective, this is extremelyconvenient because you can decouple library updates from the sources and you can have one copy of library code sharedacross multiple copies of the git sources.

Symbolic links are interesting from a compiler perspective for two reasons: diagnostics and#pragma once. For sourcelocations, there are two options: use the symbolic link as the file name or use the physical file the link resolves to.When issuing diagnostics, the compiler tries to use the symbolic link location because that is what was provided on the/Iline, but there are cases where you might want the physical file location.

#pragma once is a whole other beast with respect to symbolic links. In the beginning, C-style include guardswere created as a method of preventing repeated filecontent.#pragma once came about as a method ofpreventing inclusion of thesame file. The distinction between file and content is part of the reason why#pragmaonce is difficult to standardize due to its reliance on filesystem vagaries to identify what it means to point to the“same file”.

Consider a small example:

• Real file:C:\inc\a.h which contains a single#pragma once
• Symlink 1:C:\syms\inc\lib1\a.h -> C:\inc\a.h
• Symlink 2:C:\syms\inc\lib2\a.h -> C:\inc\a.h

Further consider a header inC:\syms\inc\lib2\b.h:

#pragma once#include "a.h"

Then in our sources we have:

#include "lib1/a.h"#include "lib2/b.h"

Let’s dive into what should happen here:

• The compiler sees symlink #1 through"lib1/a.h", sees#pragma once in the file content and records that fileC:\syms\inc\lib1\a.h is associated with a pragma once.
• The compiler reads"lib2/b.h" and sees an inclusion of"lib2/a.h".
• Symlink #2 is then read and the compiler observes that#pragma once is in the file content and records that fileC:\syms\inc\lib2\a.h is associated with a pragma once.

See a problem?

The fact that the compiler read the file content from symlink #2 is where things start to go wrong. What shouldhave happened is that the compiler unwraps the symbolic link to discover what the underlying file is and records thereal file as the owner of the pragma once, which is exactly what we did to solve the problem in normal compilationscenarios.

How does the situation above play into header units? Header units need to work like a PCH where they recordmacros and pragma state, which includes#pramga once. This means that the IFC needs to record symboliclinks and their “unwrapped” files so that the compiler can properly enforce#pragma once. After this workwas done, many more scenarios were unblocked in the Office build.

As a side note: it is always better to rely on standard C++ features to prevent repeated file contentinclusion and doing so side-steps the symbolic link problem entirely.

Inconsistent Conditional Compilation

We called out the most critical source of issues in the first blog post: inconsistent conditional compilation. Asthe experiment added projects we ran into an increasing number of conflicts. Individual projects, or sometimesisolated build nodes, couldn’t agree if thedefault chartype is unsigned,if RTTIshould be enabled, or ifUNICODE supportshould be set, and that’s only in command line options!

Across liblet headers there were also code level macros to detangle: conditional selection of a memory allocator,masks for disallowed Windows SDK functions, plus theASSUME macro example still hadn’t been resolved! Solving suchproblems could require touching nearly every project in Office.

Internal Linkage

Declarations at namespace, including global, scope markedstatic have internal linkage. With textual includesthis isn’t an issue for declarations such asstatic const float pi = 3.14 because the contents of the header file becomepart of the translation unit. The source used to compile a header unit is a translation unit unto itself and thus thedefinition ofpi will not be visible when it is imported. Fortunately, the fix is simple. Such data declarationsshould be marked asinline constexpr. However, after converting the constant declarations to beinline constexpr wewere still seeing many unresolved symbols during linking. These issues ultimately required a compiler change so thatIFC could contain the full initialization information for the data.

Naturally, the compiler had to account for the common case where global variables marked asinline constexprshould have their values encoded into the actual IFC. In general, the compiler already does this for simpler valuessuch as integral types, but more complex user-defined types or arrays of objects had to be accounted for. As a compiler optimization, the variable is only instantiated when it is referenced.More specifically, the compiler will only materialize the initializer if the variable is odr-used.

Mismatched Include Paths

Office has continued to utilize the/translateIncludeflag to consume header units without rewriting source code. For this to operate as expected the#includedirectives in source must match what is specified by a/headerUnitswitch. In Office the most common issue was a reference that used the internal path to a header instead of the externalone.

In the above example the incorrect path will not match theheader-filename portion of the/headerUnit switch and thus will be textually included. Finding incorrect include paths is tricky becausethe code will compile correctly. The best way to discover any issues is to examine the output provided by/sourceDependenciesfor unexpected textual includes.

Updating the IFC Specification

There’s been a perennial feedback bug since the original modules implementation in the compiler:`using namespace`declaration ignored when compiling as a header unit module. The problem was that the IFC had no representation forusing-directives at namespace scope (e.g.using namespace std;). It turns out that Office also ran into this issue so to continue the experiment,we had to fix it. After some thinking through the problem, we came up withIFC-76 which describes an encoding methodfor persisting directives in a translation unit.

Breaking Historical Assumptions

Before modules existed, the compiler had been around for nearly 25 years. This was enough time for the toolchainto develop several assumptions about how the front-end conveys data to the back-end. One such coupling wasencoding type information directly into compiled functions. Handles to the compiler-generated type information are 0-index-based and the underlying data is generated along with each handle. There was one case where this type index wasemitted directly into a tree for the purposes of annotating type information withnew expressions for thedebugger:

int* make_int() {  return new int{}; // <- generated a direct encoding of the type index for 'new'}

Why does this encoding cause a problem for modules? Since the compiler is now persisting compiled inlinefunctions into the IFC the compiler also persists this type index from one compilation to the next and would, sometimes,cause a linker crash as it goes to lookup the type index from the PDB but crashes because the index for that particulartranslation unit is based on an array generated in a completely different translation unit. Office was able to revealthis issue very quickly as many of the link steps involved lots of translation units from which this bug wouldsurface.

Windows SDK Woes

There is a, quite an old now, bug out:Visual Studio can’t find time() function using modules and std.core. Theroot cause here is that the UCRT contains a definition of thetime() function from C where it is defined as astaticinline function within the SDK header. Thesestatic inline functions stem from C not having the C++ meaning ofinline butstatic inline on a function declaration allows the function to behave as if it had C++inline-likesemantics. Note that defining the C standard functiontime() asstatic inline is in direct violation of the Cstandard which explicitly says that C standard library functions have external linkage. The Windows SDK team is hard atwork fixing the issue above along with some other SDK issues that have plagued C++ modules interactions in the past, sostay tuned for fixes soon!

Rethinking Compiler Tooling

As we scaled the number of projects being built by the compiler using the header unit technology it becameimmediately evident that we quickly needed to rethink how to debug compiler problems. The traditional loop involvedeither reducing the failure to a simple two or three file repro or attaching to a remote debugging instance where thecompiler was running on the build machine. It’s easy to see how and why these approaches do not scale. Here are theconcrete problems we needed to solve:

• Reproduction data collection should be asynchronous. We did not want the process of capturing a repro to be a blocking task.
• The data emitted by the compiler should be rich enough to reproduce the failure without debugging a remote compiler.
• The process of emitting data should be completely opt-in and not have a performance impact if you did not request it.
• The tool should offer a powerful visualization of the data such that we can easily navigate it and identify the underlying problem quickly.

With the requirements outlined we designed a system inside the compiler which would act as a trace logging systemfor any modules-related functionality. If you would find value in using these types of tools, please let usknow!

A New Approach to Referencing an IFC

Before the Office header unit experiments, the compiler relied on a pair of command line switches to specifyindividual IFC files:/referencefor named modules, and/headerUnit forheader units. It turns out that when thousands of header units or named modules are involved the command line growsquite long and unwieldy! An enormous list of flags is difficult to work with if there are compiler bugs to investigate,as you cannot ‘comment’ out a header unit reference easily. We solved this problem by implementing a newway of conveying IFC dependencies to the compiler:/ifcMap. The/ifcMap allows the user to provide an IFC reference mapfile, which is a subset of theTOML file format, to the compiler which details a mapping from named module name orheader-name to its respective IFC which should be loaded. Here’s a quick example of a valid .toml file for the switch:

# Header Units[[header-unit]]name = ['quote', 'm1.h']ifc = 'm1.h.ifc'[[header-unit]]name = ['quote', 'm2.h']ifc = 'm2.h.ifc'# Modules[[module]]name = 'm1'ifc = 'm1-renamed.ifc'[[module]]name = 'm2'ifc = 'm2-renamed.ifc'

/ifcMap allowed office to scale the number of header units painlessly and offer a solution to easily manage lotsof header unit references beyond having them splat on one giant command line. The IFC map also enables a tightiterative approach when considering factors like debugging needs, both for the developer and the compiler team.

Playing Nice with Precompiled Headers (PCH)

Part of scaling out for the compiler is that large projects often usePCH as a way ofachieving build speed. PCH is a reliable technology and has had the benefit of over 30 years of hardening andoptimization. Header units as the standardized replacement for PCH still need to integrate seamlessly into these olderbuild environments still using tried and true PCH technology. Furthermore, Office needs to maintain compatibility withthe non-Windows platforms that aren’t ready to adopt header units yet.

The approach mentioned last time to force include the Office shared precompiled header file into each header unitresulted in a lot of duplicated parsing in the compiler. To that end we added support to consume the binary PCHdirectly when creating a header unit. This resulted in a nice performance win when compiling header units!Unfortunately, this resulted in a large build throughput degradation when consuming header units. As much as possiblewe would need to get precompiled headers out of the picture.

The first, naïve, tactic was to ensure that each public header was truly self-contained. Eliminating invisibledependencies felt like a virtuous task, even without considering the benefits to the header unit experiment! Thousandsof#include <windows.h> and#include <stl.h> additions later, we were ready totest performance again… and it was barely any better. The issue is that precompiled headers and IFC are fundamentallydifferent technologies. Requiring the compiler to reconcile data from both sources is incredibly wasteful. The headersbeing compiled into header units were free of binary PCH dependencies but a majority of cpp files were still using bothtechnologies.

At this point it’s worth noting that we measured a significant build throughput improvement in projects thatconsumed header units but did not utilize a precompiled header. Individual compilands that switched from traditionalPCH consumption to PCH-as-header unit import, saw build time improvements well above 50%. This is one of the keybenefits that modules promised!

The idea tocreate a header unit out of the existing PCH was the flash of insight we needed to keep the experimentmoving forward! Although the compiler allows mixing PCH and header units it’s much more efficient to “pick a lane” inyour build system. By not presenting duplicate information to the compiler, it’s easier to create build throughputwins.

Selecting a Launch Project: Microsoft Word

Although we were able to scale the experiment up and generate over 5000 header units from our liblet publicheaders, some low-level blocks needed to be cleared before we could utilize header units in our production buildenvironment. We looked for a sizable project that could avoid the inconsistent conditional compilation issues thatneeded more time to clean up. Luckily, we found a great candidate in Microsoft Word.

Word has utilized MSVC’sC++ BuildInsights to craft optimal precompiled headers. Specifically, the techniques presented inFaster builds with PCHsuggestions from C++ Build Insights – C++ Team Blog were used to measure the performance benefit foreach individual file included in their main PCH. Word was to be our first test converting existing precompiled headersdirectly to header units. At a high level the steps required were:

1. Create a header unit instead of a pch.  Switch/Yc to/exportHeader
2. Replace the use PCH flag with a standard header unit reference:/Yu to/headerUnit:quote word_shared.h=path/to/word_shared.ifc
3. Profit?

The code changes required in Word after adjusting the build flags were similar to what was described above. Constants needed to be madeinlineconstexpr, missing includes or forward declarations added, and a handful of function definitions moved out-of-line. Intotal only 2 dozen C++ files in Word required code changes to compile successfully after the switch! The most unexpected ofthese changes were to standardize on quotes instead of angle brackets when writing the PCH’s#include for the sake of consistency withthe/headerUnit switch.

Along with the conversion from Word’s PCH to header units we created a header unit from the standard library.In the C++23 world, C++ projects can utilize the standard library module that ships alongside the compiler viaimport std; orimportstd.compat;. Unfortunately, Office makes edits to the standard library and thus must create its own header unit ornamed module.

With this set of changes, we proved it possible to compile, link and launch Microsoft Word with header units!

Looking Ahead: Throughput

The next step was to demonstrate the advantages that header units would bring to the Word engineering team.Fortunately, we were able to show a build performance improvement great enough that the team agreed to adopt headerunits into the Office production build system alongside msvc 17.6.6! In our next installment we’ll go over our performance findings indepth.

Closing

As always, we welcome your feedback. Feel free to send any comments through e-mail atvisualcpp@microsoft.com or throughTwitter @visualc. Also, feel free to follow Cameron DaCamara on Twitter@starfreakclone.

If you encounter other problems with MSVC in VS 2019/2022 please let us know via theReporta Problem option, either from the installer or the Visual Studio IDE itself. For suggestions or bug reports, let usknow throughDevComm.