Repository files navigation

Whole Program LLVM

This project, WLLVM, provides tools for building whole-program (orwhole-library) LLVM bitcode files from an unmodified C or C++source package. It currently runs on*nix platforms such as Linux,FreeBSD, and Mac OS X.

WLLVM provides python-based compiler wrappers that work in twosteps. The wrappers first invoke the compiler as normal. Then, foreach object file, they call a bitcode compiler to produce LLVMbitcode. The wrappers also store the location of the generated bitcodefile in a dedicated section of the object file. When object files arelinked together, the contents of the dedicated sections areconcatenated (so we don't lose the locations of any of the constituentbitcode files). After the build completes, one can use a WLLVMutility to read the contents of the dedicated section and link all ofthe bitcode into a single whole-program bitcode file. This utilityworks for both executable and native libraries.

This two-phase build process is necessary to be a drop-in replacementfor gcc or g++ in any build system. Using the LTO framework in gccand the gold linker plugin works in many cases, but fails in thepresence of static libraries in builds. WLLVM's approach has thedistinct advantage of generating working binaries, in case some partof a build process requires that.

WLLVM works with either clang or the gcc dragonegg plugin. If you are not interested in dragonegg support,and speed is an issue for you, you may want to try outgllvm.

As of August 2016 WLLVM is now a pip package. You can just do:

pip install wllvm

or

sudo pip install wllvm

depending on your machine's permissions.

If you want to develop or use the development version:

git clone https://github.com/travitch/whole-program-llvmcd whole-program-llvm

Now you need to install WLLVM. You can either installglobally on your system in develop mode:

sudo pip install -e .

or install WLLVM into a virtual python environmentin develop mode to avoid installing globally:

virtualenv venvsource venv/bin/activatepip install -e .

WLLVM includes four python executables:wllvm for compiling C codeandwllvm++ for compiling C++, an auxiliary toolextract-bc forextracting the bitcode from a build product (object file, executable, libraryor archive), and a sanity checker,wllvm-sanity-checker for detectingconfiguration oversights.

Three environment variables must be set to use these wrappers:

• LLVM_COMPILER should be set to eitherdragonegg orclang.
• LLVM_GCC_PREFIX should be set to the prefix for the version of gcc that shouldbe used with dragonegg. This can be empty if there is no prefix. This variable isnot used if$LLVM_COMPILER == clang.
• LLVM_DRAGONEGG_PLUGIN should be the full path to the dragonegg plugin. Thisvariable is not used if$LLVM_COMPILER == clang.

Once the environment is set up, just usewllvm andwllvm++ as your Cand C++ compilers, respectively.

In addition to the above environment variables the following can be optionally used:

• LLVM_CC_NAME can be set if your clang compiler is not calledclang butsomething likeclang-3.7. SimilarlyLLVM_CXX_NAME can be used to describewhat the C++ compiler is called. Note that in these sorts of cases, the environmentvariableLLVM_COMPILER should still be set toclang notclang-3.7 etc.We also pay attention to the environment variablesLLVM_LINK_NAME andLLVM_AR_NAME in ananalagous way, since they too get adorned with suffixes in various Linux distributions.

• LLVM_COMPILER_PATH can be set to the absolute path to the folder thatcontains the compiler and other LLVM tools such asllvm-link to be used.This prevents searching for the compiler in your PATH environment variable.This can be useful if you have different versions of clang on your systemand you want to easily switch compilers without tinkering with your PATHvariable.ExampleLLVM_COMPILER_PATH=/home/user/llvm_and_clang/Debug+Asserts/bin.

• WLLVM_CONFIGURE_ONLY can be set to anything. If it is set,wllvmandwllvm++ behave like a normal C or C++ compiler. They do notproduce bitcode. SettingWLLVM_CONFIGURE_ONLY may preventconfiguration errors caused by the unexpected production of hiddenbitcode files. It is sometimes required when configuring a build.

export LLVM_COMPILER=clangtar xf pkg-config-0.26.tar.gzcd pkg-config-0.26CC=wllvm ./configuremake

This should produce the executablepkg-config. To extract the bitcode:

extract-bc pkg-config

which will produce the bitcode modulepkg-config.bc.

A gentler set of instructions on building apache in a vagrant Ubuntu 14.04 can be foundhere, and for Ubuntu 16.04here.

export LLVM_COMPILER=dragoneggexport LLVM_GCC_PREFIX=llvm-export LLVM_DRAGONEGG_PLUGIN=/unsup/llvm-2.9/lib/dragonegg.sotar xf pkg-config-0.26.tar.gzcd pkg-config-0.26CC=wllvm ./configuremake

Again, this should produce the executablepkg-config. To extract the bitcode:

extract-bc pkg-config

which will produce the bitcode modulepkg-config.bc.

export LLVM_COMPILER=clangtar -xvf bullet-2.81-rev2613.tgzmkdir bullet-bincd bullet-binCC=wllvm CXX=wllvm++ cmake ../bullet-2.81-rev2613/make# Produces src/LinearMath/libLinearMath.bcaextract-bc src/LinearMath/libLinearMath.a

Note that by default extracting bitcode from an archive producesan archive of bitcode. You can also extract the bitcode directly into a module.

extract-bc -b src/LinearMath/libLinearMath.a

producessrc/LinearMath/libLinearMath.a.bc.

To see how to build freeBSD 10.0 from scratch check out thisguide.

Sometimes it is necessary to disable the production of bitcode.Typically this is during configuration, where the productionof unexpected files can confuse the configure script. For thiswe have a flagWLLVM_CONFIGURE_ONLY which can be used asfollows:

WLLVM_CONFIGURE_ONLY=1 CC=wllvm ./configureCC=wllvm make

export LLVM_COMPILER=clangtar xvfz jansson-2.7.tar.gzcd jansson-2.7CC=wllvm ./configuremakemkdir bitcodecp src/.libs/libjansson.a bitcodecd bitcodeextract-bc libjansson.allvm-ar x libjansson.bcals -la

Sometimes it can be useful to preserve the bitcode files produced in abuild, either to prevent deletion or to retrieve it later. If theenvironment variableWLLVM_BC_STORE is set to the absolute path ofan existing directory,then WLLVM will copy the produced bitcode file into that directory.The name of the copied bitcode file is the hash of the path to theoriginal bitcode file. For convenience, when using both the manifestfeature ofextract-bc and the store, the manifest will contain boththe original path, and the store path.

To support cross-compilation WLLVM supports the-target triple used by clang.More information can be foundhere.

Additionally, WLLVM leveragesobjcopy for some of its heavy lifting. Whencross-compiling you must ensure to use the appropriateobjcopy for the targetarchitecture. TheBINUTILS_TARGET_PREFIX environment variable can be used toset the objcopy of choice, for example,arm-linux-gnueabihf.

In some situations it is desirable to pass certain flags to clang in the step thatproduces the bitcode. This can be fulfilled by setting theLLVM_BITCODE_GENERATION_FLAGS environment variable to the desiredflags, for example"-flto -fwhole-program-vtables".

The WLLVM tools can show various levels of output to aid with debugging.To show this output set theWLLVM_OUTPUT_LEVEL environmentvariable to one of the following levels:

• ERROR
• WARNING
• INFO
• DEBUG

For example:

    export WLLVM_OUTPUT_LEVEL=DEBUG

Output will be directed to the standard error stream, unless you specify thepath of a logfile via theWLLVM_OUTPUT_FILE environment variable.

For example:

    export WLLVM_OUTPUT_FILE=/tmp/wllvm.log

Too many environment variables? Try doing a sanity check:

wllvm-sanity-checker

it might point out what is wrong.

WLLVM is released under the MIT license. See the fileLICENSE fordetails.