Repository files navigation

A low-level build system.

llbuild is a set of libraries for building build systems. Unlike most buildsystem projects which focus on the syntax for describing the build, llbuild isdesigned around a reusable, flexible, and scalable general purposebuildengine capable of solving many "build system"-like problems. The project alsoincludes additional libraries on top of that engine which provide support forconstructingbespoke build systems (likeswift build) or for building fromNinja manifests.

llbuild currently includes:

• A flexible core engine capable of discovering new work on the fly.

• Scalability for dependency graphs reaching millions of nodes.

• Support for building Ninja manifests (e.g., for building LLVM, Clang, andSwift).

• An llbuild-native build description format designed for extensibility.

• Library-based design intended to support embedding and reuse.

The project currently produces three top-level products;llbuild,swift-build-tool,andlibllbuild /llbuild.framework.

Thellbuild tool provides a command line interface to various feature of thellbuild libraries. It has several subtools available, seellbuild --help formore information. The most important subtool is the Ninja build support:

You can usellbuild to build Ninja-based projects using:

$ llbuild ninja build

This tool supports a subset of the command line arguments supported by Ninjaitself, to allow it to be used as a compatible replacement, even by tools likeCMake that depend on particular Ninja command line flags during theirconfiguration process.

As a convenience, if you invokellbuild via a symlink calledninja then itwill automatically use this subtool. This supports installing llbuild asninjainto yourPATH and then using it as an alternative to Ninja for buildingarbitrary projects (like LLVM, Clang, and Swift). This is also how we self-hostllbuild (via the CMake Ninja generator).

Thellbuild ninja subtool also provides additional commands which areprimarily only useful for developers interested in working on the Ninjasupport. These commands allow testing the lexer, parser, and manifest loadingcomponents independently and are used as part of the test suite.

If you want to rep your use of llbuild in public, you can proudly leverage ourconveniently provided sticker (PSD version adjacent). 😁

Inspired by Buck,llbuild ninja supports a--profile PATH option to generateaChromium trace forvisualizing where time is spent during a build. For example, the following graphis for a build of llbuild itself:

Theswift-build-tool product is the command line interface to the build systemused by theSwift Package Manager. It isbuilt as part of theSwift project build and incorporatedinto the Swift language snapshots.

This tool is built on top of theBuildSystem library.

Thelibllbuild library exposes a C API for the llbuild libraries, which can beused directly by third-parties or to build additional language bindings. Seebindings for example Swift and Python bindings that use thislibrary.

This API is what is used, for example, in Xcode as the basis for the new buildsystem introduced in Xcode 9.

The design of llbuild is a continuation of the LLVM philosophy of applyinglibrary-based design to traditional developer tools.Clang has followed this approach to deliver a highperformance compiler and assembler while also enabling new tools likeclang-format or the libclang interfaces for code completion andindexing. However, the rigid command line interface between traditional buildsystems and the compiler still limits the optimizations and features which canbe implemented in Clang.

llbuild is designed to allow construction of more feature rich buildenvironments which integrate external tools -- like the compiler -- using APIsinstead of command line interfaces. By allowing the build system and tools tocommunicate directly and to be co-designed, we believe we can unlock additionaloptimization opportunities and create more robust, easy-to-use build systems.

For more information, seeA New Architecture for Building Softwarefrom the 2016 LLVM Developer's Conference.

In the abstract, build systems are used to perform a task while also being:

• Incremental: Outputs should be efficiently rebuilt given a small change to theinputs, by leveraging the ability to save partial outputs from a prior build.

• Consistent: Equivalent inputs should always produce the same result as building from clean.

• Persistent: Results should be stored so that builds can be interrupted andresumed after failure without needing to redo the full computation.

• Parallel & Efficient: It must be possible to perform independent elements ofthe computation in parallel, in order to compute the result as efficiently aspossible.

When viewed in this light, it is clear that the core technology of a buildsystem is applicable to any complex, long-running computation in which it iscommon for the user to only modify a small portion of the input before wantingthe recompute the result. For example, a movie editor application will commonlyneed to rerender small portions of the overall movie in response to interactiveedits in order to support preview of the final result. However, suchapplications frequently do not take full advantage of the ability to store andpartially recompute the results because of the complexity of correctly managingthe dependencies between parts of the computation.

Part of the goal in designing llbuild around a general purpose build engine isto allow its use in contexts which are not traditionally thought of as requiringa "build system".

Technical documentation is available atllbuild.readthedocs.io.

Contributions to /swift-llbuild are welcomed and encouraged! Please see theContributing to Swift guide.

Before submitting the pull request, please make sure you havetested yourchangesand that they follow the Swift projectguidelines for contributingcode. Bug reports should befiled inthe issue tracker ofswift-llbuild repository on GitHub.

To be a truly great community,Swift.org needs to welcomedevelopers from all walks of life, with different backgrounds, and with a widerange of experience. A diverse and friendly community will have more greatideas, more unique perspectives, and produce more great code. We will workdiligently to make the Swift community welcoming to everyone.

To give clarity of what is expected of our members, Swift has adopted thecode of conduct defined by the Contributor Covenant. This document is usedacross many open source communities, and we think it articulates our valueswell. For more, see theCode of Conduct.

llbuild is a work in progress. Some of the more significant open projects whichwe hope to tackle are:

• Support for using file signatures instead of timestamps for change detection.

• Support richer data types for communication between tasks.

  Tasks currently only compute a single scalar value as their result. Wewould like to support richer data types for tasks results, for example tasksshould be able to compute sets of results, and have the engine automaticallycommunicate the addition or removal of individual items in the set todownstream consumers.

• Support a more sophisticated database implementation.

  The current implementation uses a SQLite3 database for storing buildresults. This was a pragmatic choice for bring up, but it can be a performancebottleneck for some applications, and we do not need the flexibility of a fullSQL database. We would like to evaluate the tradeoffs of designing a customsolution for llbuild.

• Support transparent distributed builds.

  We would like llbuild to have facilities for transparently distributing abuild across an array of worker machines.

• Support automatic auditing of build consistency.

  Few build systems diagnose problems effectively. Frequently, undeclared inputsor misbehaving tools can cause inconsistent build results. We would likellbuild to automatically diagnose these problems, for example by periodically orspeculatively rebuilding items which are not expected to have changed andcomparing the results.

• Performance tuning of core engine queues.

  The core build engine does its work using a number of queues of work items,and locking for the subset which support concurrent manipulation. We wouldlike to investigate moving the shared queues to using a lock-free datastructure and to micro-optimize the queues in general, in order to supportvery fine-grained task subdivisions without negatively impacting performance.

Q. Why does llbuild include some parts of LLVM?

A. As a low-level, embeddable component, we want llbuild itself to have asimple build process without any significant build time dependencies. However,we also wanted to take advantage of some of the data structures and supportfacilities that have been developed for LLVM. For now, our solution is toincorporate some parts of LLVM's Support libraries into the repository, with thehope that over time LLVM will either factor out those libraries in a way thatmakes it easier to reuse them, or that we will develop our own exclusive set ofsupport data structures and utilities and drop use of the LLVM ones.

Q. Why does llbuild includeNinja support?

A. llbuild includes a Ninja compatibility layer which allows building projectswhich use Ninja manifests using the llbuild core engine. We developed thissupport as a proof of concept for the core engine, and as a way to bootstrapourselves (we develop llbuild using the CMake Ninja generator and llbuild tobuild itself). This support is also valuable for allowing direct benchmarkingcomparisons of llbuild.

Our implementation of Ninja support also includes a separate library forprogrammatically loading Ninja manifests, which may prove useful to otherprojects wishing to use or manipulate Ninja files.

We intend to continue to maintain the Ninja support to keep compatibility withthe main project.

llbuild is heavily influenced by modern build systems likeShake,Buck, andNinja. We would particularly like to thank NeilMitchell for his work describing the Shake algorithm which provided theinspiration for the mechanism llbuild uses to allow additional work to bediscovered on the fly.

For similar projects, seeAdaptonandSalsa.

Seehttps://swift.org/LICENSE.txt for license information.