Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

[DTLTO][Clang] Add support for Integrated Distributed ThinLTO#147265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
bd1976bris merged 16 commits intollvm:mainfrombd1976bris:dtlto_clang
Jul 15, 2025

Conversation

bd1976bris
Copy link
Collaborator

This patch introduces support for Integrated Distributed ThinLTO (DTLTO) in Clang.

DTLTO enables the distribution of ThinLTO backend compilations via external distribution systems, such as Incredibuild, during the traditional link step:https://llvm.org/docs/DTLTO.html.

Testing:

  • lit test coverage has been added to Clang's Driver tests.
  • The DTLTO cross-project tests will use this Clang support.

For the design discussion of the DTLTO feature, see:#126654

Note that I have removed the forwarding of -mllvm options to the backend compilations which was discussed in the design review from this patch. LTO configuration for DTLTO will be addressed in a follow-up patch. In the meantime -mllvm options can be forwarded manually if required.

This patch introduces support for Integrated Distributed ThinLTO(DTLTO) in Clang.DTLTO enables the distribution of ThinLTO backend compilations viaexternal distribution systems, such as Incredibuild, during thetraditional link step:https://llvm.org/docs/DTLTO.html.Testing:- `lit` test coverage has been added to Clang's Driver tests.- The DTLTO cross-project tests will use this Clang support.For the design discussion of the DTLTO feature, see:llvm#126654Note that I have removed the forwarding of -mllvm options to thebackend compilations which was discussed in the design review fromthis patch. LTO configuration for DTLTO will be addressed in afollow-up patch. In the meantime -mllvm options can be forwardedmanually if required.
@llvmbotllvmbot added clangClang issues not falling into any other category clang:driver'clang' and 'clang++' user-facing binaries. Not 'clang-cl' labelsJul 7, 2025
@llvmbot
Copy link
Member

llvmbot commentedJul 7, 2025•
edited
Loading

@llvm/pr-subscribers-clang

@llvm/pr-subscribers-clang-driver

Author: bd1976bris (bd1976bris)

Changes

This patch introduces support for Integrated Distributed ThinLTO (DTLTO) in Clang.

DTLTO enables the distribution of ThinLTO backend compilations via external distribution systems, such as Incredibuild, during the traditional link step:https://llvm.org/docs/DTLTO.html.

Testing:

  • lit test coverage has been added to Clang's Driver tests.
  • The DTLTO cross-project tests will use this Clang support.

For the design discussion of the DTLTO feature, see:#126654

Note that I have removed the forwarding of -mllvm options to the backend compilations which was discussed in the design review from this patch. LTO configuration for DTLTO will be addressed in a follow-up patch. In the meantime -mllvm options can be forwarded manually if required.

Full diff:https://github.com/llvm/llvm-project/pull/147265.diff

5 Files Affected:

  • (modified) clang/docs/ThinLTO.rst (+32)
  • (modified) clang/include/clang/Driver/Options.td (+13-1)
  • (modified) clang/lib/Driver/ToolChains/Gnu.cpp (+15)
  • (added) clang/test/Driver/DTLTO/dtlto.c (+43)
  • (modified) cross-project-tests/dtlto/ld-dtlto.c (+13-18)
diff --git a/clang/docs/ThinLTO.rst b/clang/docs/ThinLTO.rstindex c042547678919..687795ac655a7 100644--- a/clang/docs/ThinLTO.rst+++ b/clang/docs/ThinLTO.rst@@ -240,6 +240,38 @@ The ``BOOTSTRAP_LLVM_ENABLE_LTO=Thin`` will enable ThinLTO for stage 2 and stage 3 in case the compiler used for stage 1 does not support the ThinLTO option.+Integrated Distributed ThinLTO (DTLTO)+--------------------------------------++Integrated Distributed ThinLTO (DTLTO) enables the distribution of backend+ThinLTO compilations via external distribution systems, such as Incredibuild,+during the traditional link step.++The implementation is documented here: https://llvm.org/docs/DTLTO.html.++DTLTO requires the LLD linker (``-fuse-ld=lld``).++``-fthinlto-distributor=<path>``+   - Specifies the ``<path>`` to the distributor process executable for DTLTO.+   - If specified, ThinLTO backend compilations will be distributed by LLD.++``-Xthinlto-distributor=<arg>``+   - Passes ``<arg>`` to the distributor process (see ``-fthinlto-distributor=``).+   - Can be specified multiple times to pass multiple options.+   - Multiple options can also be specified by separating them with commas.++Examples:+   - ``clang -flto=thin -fthinlto-distributor=incredibuild.exe -Xthinlto-distributor=--verbose,--j10 -fuse-ld=lld``+   - ``clang -flto=thin -fthinlto-distributor=$(which python) -Xthinlto-distributor=incredibuild.py -fuse-ld=lld``++If ``-fthinlto-distributor=`` is specified, Clang supplies the path to a+compiler to be executed remotely to perform the ThinLTO backend+compilations. Currently, this is Clang itself.++Note that currently, DTLTO is only supported in some LLD flavors. Support will+be added to other LLD flavours in the future.+See `DTLTO <https://lld.llvm.org/dtlto.html>`_ for more information.+ More Information ================diff --git a/clang/include/clang/Driver/Options.td b/clang/include/clang/Driver/Options.tdindex 0c8a219b19bf4..9c6f77af97be0 100644--- a/clang/include/clang/Driver/Options.td+++ b/clang/include/clang/Driver/Options.td@@ -990,6 +990,13 @@ def Xlinker : Separate<["-"], "Xlinker">, Flags<[LinkerInput, RenderAsInput]>,   Visibility<[ClangOption, CLOption, FlangOption]>,   HelpText<"Pass <arg> to the linker">, MetaVarName<"<arg>">,   Group<Link_Group>;+def Xthinlto_distributor_EQ : CommaJoined<["-"], "Xthinlto-distributor=">,+  Flags<[LinkOption]>,+  Visibility<[ClangOption, CLOption]>,+  HelpText<"Pass <arg> to the ThinLTO distributor process. Can be specified "+           "multiple times or with comma-separated values.">,+  MetaVarName<"<arg>">,+  Group<Link_Group>; def Xoffload_linker : JoinedAndSeparate<["-"], "Xoffload-linker">,   Visibility<[ClangOption, FlangOption]>,   HelpText<"Pass <arg> to the offload linkers or the ones identified by -<triple>">,@@ -4233,7 +4240,12 @@ def ffinite_loops: Flag<["-"],  "ffinite-loops">, Group<f_Group>, def fno_finite_loops: Flag<["-"], "fno-finite-loops">, Group<f_Group>,   HelpText<"Do not assume that any loop is finite.">,   Visibility<[ClangOption, CC1Option]>;-+def fthinlto_distributor_EQ : Joined<["-"], "fthinlto-distributor=">,+  Group<f_Group>,+  HelpText<"Path to the ThinLTO distributor process. If specified, "+           "ThinLTO backend compilations will be distributed by LLD">,+  MetaVarName<"<path>">,+  Visibility<[ClangOption, CLOption]>; def ftrigraphs : Flag<["-"], "ftrigraphs">, Group<f_Group>,   HelpText<"Process trigraph sequences">, Visibility<[ClangOption, CC1Option]>; def fno_trigraphs : Flag<["-"], "fno-trigraphs">, Group<f_Group>,diff --git a/clang/lib/Driver/ToolChains/Gnu.cpp b/clang/lib/Driver/ToolChains/Gnu.cppindex f5e2655857432..3e9ca8f79d160 100644--- a/clang/lib/Driver/ToolChains/Gnu.cpp+++ b/clang/lib/Driver/ToolChains/Gnu.cpp@@ -455,6 +455,21 @@ void tools::gnutools::Linker::ConstructJob(Compilation &C, const JobAction &JA,     addLTOOptions(ToolChain, Args, CmdArgs, Output, Inputs,                   D.getLTOMode() == LTOK_Thin);+  // Forward the DTLTO options to the linker. We add these unconditionally,+  // rather than in addLTOOptions() as it is the linker that decides whether to+  // do LTO or not dependent upon whether there are any bitcode input files in+  // the link.+  if (Arg *A = Args.getLastArg(options::OPT_fthinlto_distributor_EQ)) {+    CmdArgs.push_back(+        Args.MakeArgString("--thinlto-distributor=" + Twine(A->getValue())));+    CmdArgs.push_back(+        Args.MakeArgString("--thinlto-remote-compiler=" ++                           Twine(ToolChain.getDriver().getClangProgramPath())));++    for (auto A : Args.getAllArgValues(options::OPT_Xthinlto_distributor_EQ))+      CmdArgs.push_back(Args.MakeArgString("--thinlto-distributor-arg=" + A));+  }+   if (Args.hasArg(options::OPT_Z_Xlinker__no_demangle))     CmdArgs.push_back("--no-demangle");diff --git a/clang/test/Driver/DTLTO/dtlto.c b/clang/test/Driver/DTLTO/dtlto.cnew file mode 100644index 0000000000000..a23a10fdfb055--- /dev/null+++ b/clang/test/Driver/DTLTO/dtlto.c@@ -0,0 +1,43 @@+// REQUIRES: lld++/// Check DTLTO options are forwarded to the linker.++// RUN: echo "--target=x86_64-linux-gnu \+// RUN:   -Xthinlto-distributor=distarg1 \+// RUN:   -Xthinlto-distributor=distarg2,distarg3 \+// RUN:   -fuse-ld=lld" > %t.rsp++/// Check that options are forwarded as expected with --thinlto-distributor=.+// RUN: %clang -### @%t.rsp -fthinlto-distributor=dist.exe %s 2>&1 | \+// RUN:   FileCheck %s --implicit-check-not=warning++// CHECK: ld.lld+// CHECK-SAME: "--thinlto-distributor=dist.exe"+// CHECK-SAME: "--thinlto-remote-compiler={{.*}}clang+// CHECK-SAME: "--thinlto-distributor-arg=distarg1"+// CHECK-SAME: "--thinlto-distributor-arg=distarg2"+// CHECK-SAME: "--thinlto-distributor-arg=distarg3"+++/// Check that options are not added without --thinlto-distributor= and+/// that there is an unused option warning issued for -Xthinlto-distributor=+/// options. We specify -flto here as these options should be unaffected by it.+// RUN: %clang -### @%t.rsp -flto=thin %s 2>&1 | \+// RUN:   FileCheck %s --check-prefixes=NONE,NOMORE --implicit-check-not=warning++// NONE: warning: argument unused during compilation: '-Xthinlto-distributor=distarg1'+// NONE: warning: argument unused during compilation: '-Xthinlto-distributor=distarg2,distarg3'+// NONE:     ld.lld+// NOMORE-NOT: distributor+// NOMORE-NOT: remote-compiler+++/// Check the expected arguments are forwarded by default with only+/// --thinlto-distributor=.+// RUN: %clang --target=x86_64-linux-gnu -fthinlto-distributor=dist.exe \+// RUN:   -fuse-ld=lld -Werror -### %s 2>&1 | \+// RUN:   FileCheck %s --check-prefixes=DEFAULT,NOMORE++// DEFAULT: ld.lld+// DEFAULT-SAME: "--thinlto-distributor=dist.exe"+// DEFAULT-SAME: "--thinlto-remote-compiler={{.*}}clangdiff --git a/cross-project-tests/dtlto/ld-dtlto.c b/cross-project-tests/dtlto/ld-dtlto.cindex 3ee962346bd4a..7dffe2e015bcb 100644--- a/cross-project-tests/dtlto/ld-dtlto.c+++ b/cross-project-tests/dtlto/ld-dtlto.c@@ -5,13 +5,11 @@  // RUN: rm -rf %t && mkdir %t && cd %t-// RUN: %clang --target=x86_64-linux-gnu -c -flto=thin %s -o dtlto.o--// RUN: ld.lld dtlto.o \-// RUN:   --thinlto-distributor=%python \-// RUN:   --thinlto-distributor-arg=%llvm_src_root/utils/dtlto/local.py \-// RUN:   --thinlto-remote-compiler=%clang \-// RUN:   --thinlto-remote-compiler-arg=--save-temps+// RUN: %clang --target=x86_64-linux-gnu %s -flto=thin -fuse-ld=lld \+// RUN:   -fthinlto-distributor=%python \+// RUN:   -Xthinlto-distributor=%llvm_src_root/utils/dtlto/local.py \+// RUN:   -Wl,--thinlto-remote-compiler-arg=--save-temps \+// RUN:   -nostdlib -Werror  /// Check that the required output files have been created. // RUN: ls | sort | FileCheck %s@@ -22,18 +20,15 @@ /// Linked ELF. // CHECK: {{^}}a.out{{$}}-/// Produced by the bitcode compilation.-// CHECK-NEXT: {{^}}dtlto.o{{$}}- /// --save-temps output for the backend compilation.-// CHECK-NEXT: {{^}}dtlto.s{{$}}-// CHECK-NEXT: {{^}}dtlto.s.0.preopt.bc{{$}}-// CHECK-NEXT: {{^}}dtlto.s.1.promote.bc{{$}}-// CHECK-NEXT: {{^}}dtlto.s.2.internalize.bc{{$}}-// CHECK-NEXT: {{^}}dtlto.s.3.import.bc{{$}}-// CHECK-NEXT: {{^}}dtlto.s.4.opt.bc{{$}}-// CHECK-NEXT: {{^}}dtlto.s.5.precodegen.bc{{$}}-// CHECK-NEXT: {{^}}dtlto.s.resolution.txt{{$}}+// CHECK-NEXT: {{^}}ld-dtlto-[[TMP:[a-zA-Z0-9_]+]].s{{$}}+// CHECK-NEXT: {{^}}ld-dtlto-[[TMP]].s.0.preopt.bc{{$}}+// CHECK-NEXT: {{^}}ld-dtlto-[[TMP]].s.1.promote.bc{{$}}+// CHECK-NEXT: {{^}}ld-dtlto-[[TMP]].s.2.internalize.bc{{$}}+// CHECK-NEXT: {{^}}ld-dtlto-[[TMP]].s.3.import.bc{{$}}+// CHECK-NEXT: {{^}}ld-dtlto-[[TMP]].s.4.opt.bc{{$}}+// CHECK-NEXT: {{^}}ld-dtlto-[[TMP]].s.5.precodegen.bc{{$}}+// CHECK-NEXT: {{^}}ld-dtlto-[[TMP]].s.resolution.txt{{$}}  /// No files are expected after. // CHECK-NOT: {{.}}

Copy link
Contributor

@teresajohnsonteresajohnson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

A few minor comments/questions

@@ -455,6 +455,21 @@ void tools::gnutools::Linker::ConstructJob(Compilation &C, const JobAction &JA,
addLTOOptions(ToolChain, Args, CmdArgs, Output, Inputs,
D.getLTOMode() == LTOK_Thin);

// Forward the DTLTO options to the linker. We add these unconditionally,
// rather than in addLTOOptions() as it is the linker that decides whether to
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Why not be consistent with the other LTO options though, which are added in addLTOOptions? The same argument could be made for those (and maybe they should be sent unconditionally, but that seems like a separate discussion).

Copy link
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Apologies - we’ve recently started setting LTO options unconditionally in the PS5 driver, and I carried that pattern over here without fully reconsidering it. As you pointed out, the issue is broader than just the DTLTO options and deserves separate handling.

It’s also unclear whether setting options unconditionally is the right long-term approach. An alternative could be to pass a flag to the linker, allowing it to emit a warning if LTO is being performed but the Clang driver wasn't explicitly invoked in LTO mode.

In any case, I’ve moved the DTLTO option handling intoaddLTOOptions as you suggested. Thanks for the guidance.

/// that there is an unused option warning issued for -Xthinlto-distributor=
/// options. We specify -flto here as these options should be unaffected by it.
// RUN: %clang -### @%t.rsp -flto=thin %s 2>&1 | \
// RUN: FileCheck %s --check-prefixes=NONE,NOMORE --implicit-check-not=warning
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Confused about the--implicit-check-not=warning since there are some warnings below. Although I guess this is checking that there are none other than those being explicitly checked below - but is that intended?

Copy link
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Yes that was the intent. I have now used a common response file for allFileCheck invocations in this test and added a comment to explain the intention behind the use ofimplicit-check-notarguments.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

For clang driver, prefer-Werror to implicit-check-not=warning: .
A warning would makeclang -### -Werror exit with code 1.

bd1976bris reacted with thumbs up emoji
Copy link
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

@MaskRay done. Thanks.

// NONE: warning: argument unused during compilation: '-Xthinlto-distributor=distarg1'
// NONE: warning: argument unused during compilation: '-Xthinlto-distributor=distarg2,distarg3'
// NONE: ld.lld
// NOMORE-NOT: distributor
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Maybe better to put these in--implicit-check-not= since this would not catch any before the above warnings.

Copy link
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Done. I have also used the same set of implicit-check-not arguments in each of the FileCheck invocations (via a response file) to reduce the size of the test and improve readability.

Copy link
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

@teresajohnson I have now removed the use of response files and implemented (judiciously)@MaskRay's suggestion of using-Werror.

Copy link
Contributor

@teresajohnsonteresajohnson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

lgtm with comments from other reviewers resolved.

@bd1976brisbd1976bris merged commit5004c59 intollvm:mainJul 15, 2025
17 checks passed
@Prabhuk
Copy link
Contributor

Clang :: Driver/DTLTO/dtlto.c test is failing in our toolchain builders.

See full log here:https://logs.chromium.org/logs/fuchsia/buildbucket/cr-buildbucket/8709270904931005265/+/u/clang/tests/stdout

Exit Code: 1Command Output (stderr):--/b/s/w/ir/x/w/llvm_build/bin/clang -flto=thin /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/dtlto.c -### -fuse-ld=lld --target=x86_64-linux-gnu    -Xthinlto-distributor=a1 -Xthinlto-distributor=a2,a3    -fthinlto-distributor=d.exe -Werror 2>&1 |    /b/s/w/ir/x/w/llvm_build/bin/FileCheck /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/dtlto.c --check-prefix=FORWARD # RUN: at line 6+ /b/s/w/ir/x/w/llvm_build/bin/FileCheck /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/dtlto.c --check-prefix=FORWARD+ /b/s/w/ir/x/w/llvm_build/bin/clang -flto=thin /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/dtlto.c -### -fuse-ld=lld --target=x86_64-linux-gnu -Xthinlto-distributor=a1 -Xthinlto-distributor=a2,a3 -fthinlto-distributor=d.exe -Werror/b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/dtlto.c:14:18: error: FORWARD-SAME: expected string not found in input// FORWARD-SAME: "--thinlto-distributor-arg=a1"                 ^<stdin>:6:875: note: scanning from here "/b/s/w/ir/x/w/llvm_build/bin/ld.lld" "--hash-style=gnu" "--build-id" "--eh-frame-hdr" "-m" "elf_x86_64" "-pie" "-dynamic-linker" "/lib64/ld-linux-x86-64.so.2" "-o" "a.out" "/lib/x86_64-linux-gnu/Scrt1.o" "/lib/x86_64-linux-gnu/crti.o" "crtbeginS.o" "-L/lib/x86_64-linux-gnu" "-L/lib/../lib64" "-L/usr/lib/x86_64-linux-gnu" "-L/usr/lib64" "-L/lib" "-L/usr/lib" "-plugin-opt=mcpu=x86-64" "-plugin-opt=thinlto" "--thinlto-distributor=d.exe" "--thinlto-remote-compiler=/b/s/w/ir/x/w/llvm_build/bin/llvm" "--thinlto-distributor-arg=a1" "--thinlto-distributor-arg=a2" "--thinlto-distributor-arg=a3" "/b/s/w/ir/x/t/lit-tmp-7dk9usqs/dtlto-d79331.o" "/b/s/w/ir/x/w/llvm_build/lib/clang/22/lib/x86_64-unknown-linux-gnu/libclang_rt.builtins.a" "--as-needed" "-lunwind" "--no-as-needed" "-lc" "/b/s/w/ir/x/w/llvm_build/lib/clang/22/lib/x86_64-unknown-linux-gnu/libclang_rt.builtins.a" "--as-needed" "-lunwind" "--no-as-needed" "crtendS.o" "/lib/x86_64-linux-gnu/crtn.o"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          ^<stdin>:6:940: note: possible intended match here "/b/s/w/ir/x/w/llvm_build/bin/ld.lld" "--hash-style=gnu" "--build-id" "--eh-frame-hdr" "-m" "elf_x86_64" "-pie" "-dynamic-linker" "/lib64/ld-linux-x86-64.so.2" "-o" "a.out" "/lib/x86_64-linux-gnu/Scrt1.o" "/lib/x86_64-linux-gnu/crti.o" "crtbeginS.o" "-L/lib/x86_64-linux-gnu" "-L/lib/../lib64" "-L/usr/lib/x86_64-linux-gnu" "-L/usr/lib64" "-L/lib" "-L/usr/lib" "-plugin-opt=mcpu=x86-64" "-plugin-opt=thinlto" "--thinlto-distributor=d.exe" "--thinlto-remote-compiler=/b/s/w/ir/x/w/llvm_build/bin/llvm" "--thinlto-distributor-arg=a1" "--thinlto-distributor-arg=a2" "--thinlto-distributor-arg=a3" "/b/s/w/ir/x/t/lit-tmp-7dk9usqs/dtlto-d79331.o" "/b/s/w/ir/x/w/llvm_build/lib/clang/22/lib/x86_64-unknown-linux-gnu/libclang_rt.builtins.a" "--as-needed" "-lunwind" "--no-as-needed" "-lc" "/b/s/w/ir/x/w/llvm_build/lib/clang/22/lib/x86_64-unknown-linux-gnu/libclang_rt.builtins.a" "--as-needed" "-lunwind" "--no-as-needed" "crtendS.o" "/lib/x86_64-linux-gnu/crtn.o"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ^Input file: <stdin>Check file: /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/dtlto.c-dump-input=help explains the following input dump.Input was:<<<<<<           1: Fuchsia clang version 22.0.0git (https://llvm.googlesource.com/llvm-project 20b47bfdb5aee6e66d06342c275d56477371dc4e)            2: Target: x86_64-unknown-linux-gnu            3: Thread model: posix            4: InstalledDir: /b/s/w/ir/x/w/llvm_build/bin            5:  "/b/s/w/ir/x/w/llvm_build/bin/llvm" "clang" "-cc1" "-triple" "x86_64-unknown-linux-gnu" "-emit-llvm-bc" "-flto=thin" "-flto-unit" "-dumpdir" "a-" "-disable-free" "-clear-ast-before-backend" "-disable-llvm-verifier" "-discard-value-names" "-main-file-name" "dtlto.c" "-mrelocation-model" "pic" "-pic-level" "2" "-pic-is-pie" "-mframe-pointer=all" "-fmath-errno" "-ffp-contract=on" "-fno-rounding-math" "-mconstructor-aliases" "-funwind-tables=2" "-target-cpu" "x86-64" "-tune-cpu" "generic" "-debugger-tuning=gdb" "-fdebug-compilation-dir=/b/s/w/ir/x/w/llvm_build/tools/clang/test/Driver/DTLTO" "-fcoverage-compilation-dir=/b/s/w/ir/x/w/llvm_build/tools/clang/test/Driver/DTLTO" "-resource-dir" "/b/s/w/ir/x/w/llvm_build/lib/clang/22" "-internal-isystem" "/b/s/w/ir/x/w/llvm_build/lib/clang/22/include" "-internal-isystem" "/usr/local/include" "-internal-externc-isystem" "/usr/include/x86_64-linux-gnu" "-internal-externc-isystem" "/include" "-internal-externc-isystem" "/usr/include" "-Werror" "-ferror-limit" "19" "-fgnuc-version=4.2.1" "-fskip-odr-check-in-gmf" "-faddrsig" "-D__GCC_HAVE_DWARF2_CFI_ASM=1" "-o" "/b/s/w/ir/x/t/lit-tmp-7dk9usqs/dtlto-d79331.o" "-x" "c" "/b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/dtlto.c"            6:  "/b/s/w/ir/x/w/llvm_build/bin/ld.lld" "--hash-style=gnu" "--build-id" "--eh-frame-hdr" "-m" "elf_x86_64" "-pie" "-dynamic-linker" "/lib64/ld-linux-x86-64.so.2" "-o" "a.out" "/lib/x86_64-linux-gnu/Scrt1.o" "/lib/x86_64-linux-gnu/crti.o" "crtbeginS.o" "-L/lib/x86_64-linux-gnu" "-L/lib/../lib64" "-L/usr/lib/x86_64-linux-gnu" "-L/usr/lib64" "-L/lib" "-L/usr/lib" "-plugin-opt=mcpu=x86-64" "-plugin-opt=thinlto" "--thinlto-distributor=d.exe" "--thinlto-remote-compiler=/b/s/w/ir/x/w/llvm_build/bin/llvm" "--thinlto-distributor-arg=a1" "--thinlto-distributor-arg=a2" "--thinlto-distributor-arg=a3" "/b/s/w/ir/x/t/lit-tmp-7dk9usqs/dtlto-d79331.o" "/b/s/w/ir/x/w/llvm_build/lib/clang/22/lib/x86_64-unknown-linux-gnu/libclang_rt.builtins.a" "--as-needed" "-lunwind" "--no-as-needed" "-lc" "/b/s/w/ir/x/w/llvm_build/lib/clang/22/lib/x86_64-unknown-linux-gnu/libclang_rt.builtins.a" "--as-needed" "-lunwind" "--no-as-needed" "crtendS.o" "/lib/x86_64-linux-gnu/crtn.o" same:14'0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match foundsame:14'1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                ?                     possible intended match>>>>>>--

@bd1976bris
Copy link
CollaboratorAuthor

bd1976bris commentedJul 15, 2025•
edited
Loading

@Prabhuk I might need some help here. Looking at the output for theClang :: Driver/DTLTO/dtlto.c test on your buildbot LLD is being invoked with the argument"--thinlto-remote-compiler=/b/s/w/ir/x/w/llvm_build/bin/llvm". That is clearly not the location of a Clang compiler (/b/s/w/ir/x/w/llvm_build/bin/clang) ! This is produced by callingToolChain.getDriver().getClangProgramPath() in the Clang driver (see:5004c59#diff-c89ebf1fc950679f63c19739a35aa966a12674f7d61af6fb382ab1fa0ee21dd7R1328). Is there something strange about the buildbot or perhaps thefuchsia target that might lead to this?

@ilovepi
Copy link
Contributor

I see this patch also failing its own tests in our ci

******************** TEST 'Clang :: Driver/DTLTO/dtlto.c' FAILED ********************Exit Code: 1Command Output (stderr):--/b/s/w/ir/x/w/llvm_build/bin/clang -flto=thin /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/dtlto.c -### -fuse-ld=lld --target=x86_64-linux-gnu    -Xthinlto-distributor=a1 -Xthinlto-distributor=a2,a3    -fthinlto-distributor=d.exe -Werror 2>&1 |    /b/s/w/ir/x/w/llvm_build/bin/FileCheck /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/dtlto.c --check-prefix=FORWARD # RUN: at line 6+ /b/s/w/ir/x/w/llvm_build/bin/FileCheck /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/dtlto.c --check-prefix=FORWARD+ /b/s/w/ir/x/w/llvm_build/bin/clang -flto=thin /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/dtlto.c -### -fuse-ld=lld --target=x86_64-linux-gnu -Xthinlto-distributor=a1 -Xthinlto-distributor=a2,a3 -fthinlto-distributor=d.exe -Werror/b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/dtlto.c:14:18: error: FORWARD-SAME: expected string not found in input// FORWARD-SAME: "--thinlto-distributor-arg=a1"                 ^<stdin>:6:875: note: scanning from here "/b/s/w/ir/x/w/llvm_build/bin/ld.lld" "--hash-style=gnu" "--build-id" "--eh-frame-hdr" "-m" "elf_x86_64" "-pie" "-dynamic-linker" "/lib64/ld-linux-x86-64.so.2" "-o" "a.out" "/lib/x86_64-linux-gnu/Scrt1.o" "/lib/x86_64-linux-gnu/crti.o" "crtbeginS.o" "-L/lib/x86_64-linux-gnu" "-L/lib/../lib64" "-L/usr/lib/x86_64-linux-gnu" "-L/usr/lib64" "-L/lib" "-L/usr/lib" "-plugin-opt=mcpu=x86-64" "-plugin-opt=thinlto" "--thinlto-distributor=d.exe" "--thinlto-remote-compiler=/b/s/w/ir/x/w/llvm_build/bin/llvm" "--thinlto-distributor-arg=a1" "--thinlto-distributor-arg=a2" "--thinlto-distributor-arg=a3" "/b/s/w/ir/x/t/lit-tmp-7dk9usqs/dtlto-d79331.o" "/b/s/w/ir/x/w/llvm_build/lib/clang/22/lib/x86_64-unknown-linux-gnu/libclang_rt.builtins.a" "--as-needed" "-lunwind" "--no-as-needed" "-lc" "/b/s/w/ir/x/w/llvm_build/lib/clang/22/lib/x86_64-unknown-linux-gnu/libclang_rt.builtins.a" "--as-needed" "-lunwind" "--no-as-needed" "crtendS.o" "/lib/x86_64-linux-gnu/crtn.o"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          ^<stdin>:6:940: note: possible intended match here "/b/s/w/ir/x/w/llvm_build/bin/ld.lld" "--hash-style=gnu" "--build-id" "--eh-frame-hdr" "-m" "elf_x86_64" "-pie" "-dynamic-linker" "/lib64/ld-linux-x86-64.so.2" "-o" "a.out" "/lib/x86_64-linux-gnu/Scrt1.o" "/lib/x86_64-linux-gnu/crti.o" "crtbeginS.o" "-L/lib/x86_64-linux-gnu" "-L/lib/../lib64" "-L/usr/lib/x86_64-linux-gnu" "-L/usr/lib64" "-L/lib" "-L/usr/lib" "-plugin-opt=mcpu=x86-64" "-plugin-opt=thinlto" "--thinlto-distributor=d.exe" "--thinlto-remote-compiler=/b/s/w/ir/x/w/llvm_build/bin/llvm" "--thinlto-distributor-arg=a1" "--thinlto-distributor-arg=a2" "--thinlto-distributor-arg=a3" "/b/s/w/ir/x/t/lit-tmp-7dk9usqs/dtlto-d79331.o" "/b/s/w/ir/x/w/llvm_build/lib/clang/22/lib/x86_64-unknown-linux-gnu/libclang_rt.builtins.a" "--as-needed" "-lunwind" "--no-as-needed" "-lc" "/b/s/w/ir/x/w/llvm_build/lib/clang/22/lib/x86_64-unknown-linux-gnu/libclang_rt.builtins.a" "--as-needed" "-lunwind" "--no-as-needed" "crtendS.o" "/lib/x86_64-linux-gnu/crtn.o"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ^Input file: <stdin>Check file: /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/dtlto.c-dump-input=help explains the following input dump.Input was:<<<<<<           1: Fuchsia clang version 22.0.0git (https://llvm.googlesource.com/llvm-project 20b47bfdb5aee6e66d06342c275d56477371dc4e)            2: Target: x86_64-unknown-linux-gnu            3: Thread model: posix            4: InstalledDir: /b/s/w/ir/x/w/llvm_build/bin            5:  "/b/s/w/ir/x/w/llvm_build/bin/llvm" "clang" "-cc1" "-triple" "x86_64-unknown-linux-gnu" "-emit-llvm-bc" "-flto=thin" "-flto-unit" "-dumpdir" "a-" "-disable-free" "-clear-ast-before-backend" "-disable-llvm-verifier" "-discard-value-names" "-main-file-name" "dtlto.c" "-mrelocation-model" "pic" "-pic-level" "2" "-pic-is-pie" "-mframe-pointer=all" "-fmath-errno" "-ffp-contract=on" "-fno-rounding-math" "-mconstructor-aliases" "-funwind-tables=2" "-target-cpu" "x86-64" "-tune-cpu" "generic" "-debugger-tuning=gdb" "-fdebug-compilation-dir=/b/s/w/ir/x/w/llvm_build/tools/clang/test/Driver/DTLTO" "-fcoverage-compilation-dir=/b/s/w/ir/x/w/llvm_build/tools/clang/test/Driver/DTLTO" "-resource-dir" "/b/s/w/ir/x/w/llvm_build/lib/clang/22" "-internal-isystem" "/b/s/w/ir/x/w/llvm_build/lib/clang/22/include" "-internal-isystem" "/usr/local/include" "-internal-externc-isystem" "/usr/include/x86_64-linux-gnu" "-internal-externc-isystem" "/include" "-internal-externc-isystem" "/usr/include" "-Werror" "-ferror-limit" "19" "-fgnuc-version=4.2.1" "-fskip-odr-check-in-gmf" "-faddrsig" "-D__GCC_HAVE_DWARF2_CFI_ASM=1" "-o" "/b/s/w/ir/x/t/lit-tmp-7dk9usqs/dtlto-d79331.o" "-x" "c" "/b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/dtlto.c"            6:  "/b/s/w/ir/x/w/llvm_build/bin/ld.lld" "--hash-style=gnu" "--build-id" "--eh-frame-hdr" "-m" "elf_x86_64" "-pie" "-dynamic-linker" "/lib64/ld-linux-x86-64.so.2" "-o" "a.out" "/lib/x86_64-linux-gnu/Scrt1.o" "/lib/x86_64-linux-gnu/crti.o" "crtbeginS.o" "-L/lib/x86_64-linux-gnu" "-L/lib/../lib64" "-L/usr/lib/x86_64-linux-gnu" "-L/usr/lib64" "-L/lib" "-L/usr/lib" "-plugin-opt=mcpu=x86-64" "-plugin-opt=thinlto" "--thinlto-distributor=d.exe" "--thinlto-remote-compiler=/b/s/w/ir/x/w/llvm_build/bin/llvm" "--thinlto-distributor-arg=a1" "--thinlto-distributor-arg=a2" "--thinlto-distributor-arg=a3" "/b/s/w/ir/x/t/lit-tmp-7dk9usqs/dtlto-d79331.o" "/b/s/w/ir/x/w/llvm_build/lib/clang/22/lib/x86_64-unknown-linux-gnu/libclang_rt.builtins.a" "--as-needed" "-lunwind" "--no-as-needed" "-lc" "/b/s/w/ir/x/w/llvm_build/lib/clang/22/lib/x86_64-unknown-linux-gnu/libclang_rt.builtins.a" "--as-needed" "-lunwind" "--no-as-needed" "crtendS.o" "/lib/x86_64-linux-gnu/crtn.o" same:14'0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match foundsame:14'1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                ?                     possible intended match>>>>>>--

Bot:https://ci.chromium.org/ui/p/fuchsia/builders/toolchain.ci/clang-host-linux-x64/b8709270904931005265/overview

Logs:https://logs.chromium.org/logs/fuchsia/buildbucket/cr-buildbucket/8709270904931005265/+/u/clang/tests/stdout

If this isn't trivial to fix, I'd suggest revert and reland once you've addressed it.

@ilovepi
Copy link
Contributor

ilovepi commentedJul 15, 2025•
edited
Loading

@Prabhuk I might need some help here. Looking at the output for theClang :: Driver/DTLTO/dtlto.c test on your buildbot LLD is being invoked with the argument"--thinlto-remote-compiler=/b/s/w/ir/x/w/llvm_build/bin/llvm". That is clearly not the location of a Clang compiler (/b/s/w/ir/x/w/llvm_build/bin/clang) ! This is produced by callingToolChain.getDriver().getClangProgramPath() in the Clang driver (see:5004c59#diff-c89ebf1fc950679f63c19739a35aa966a12674f7d61af6fb382ab1fa0ee21dd7R1328). Is there something strange about the buildbot or perhaps thefuchsia target that might lead to this?

We use the multicall build, so that is in fact our clang and lld (multicall is basically like a busybox config). clang and lld exist in the build and install directories as symlinks to the llvm binary. It's well supported in the CMake, and has been used by us and others for several years now.

The toolchain we're building and testing is just a normal linux/windows/mac toolchain. Its configured to compile Fuchsia, but we don't enable anything too special, just some standard cmake options.

@bd1976bris
Copy link
CollaboratorAuthor

We use the multicall library. that is in fact our clang and lld (multicall is basically like a busybox config). clang and lld exist in the build directory and install as symlinks to the llvm binary. It's well supported in the CMake, and has been used by us and others for several years now.

@ilovepi - thanks very much! I think that it is impossible in general for me to predict the name of the tool that will be returned byToolChain.getDriver().getClangProgramPath() so I think that the fix to the test is trivial - I'll modify the Clang Driver test to simply check that there is a non-empty path supplied for the "--thinlto-remote-compiler" argument. We have the cross-project test which will check that the path is actually valid.

@Kewen12
Copy link
Contributor

Hi there, this PR failed the test on our bot breaking the builds. If it takes some time to land the fix, maybe consider revert and reland it? Thanks!

bot:https://lab.llvm.org/buildbot/#/builders/10/builds/9468

bd1976bris added a commit to bd1976bris/llvm-project that referenced this pull requestJul 15, 2025
Not all builds name the compiler executable `clang`. For example,the Fuchsia build bots use `llvm` as the executable name, as theycombine everything together in a busybot-style binary.Update the Clang driver test to simply check that a non-empty pathis provided for the `--thinlto-remote-compiler` argument, ratherthan hardcoding the executable name. The cross-project test willverify that the path is valid later.Shouldfixllvm#147265.
bd1976bris added a commit that referenced this pull requestJul 15, 2025
Not all builds name the compiler executable `clang`. For example, theFuchsia build bots use `llvm` as their single toolchain executable name,as they combine everything together in a busybot-style binary.Update the Clang driver test to simply check that a non-empty path isprovided for the `--thinlto-remote-compiler` argument, rather thanhardcoding the executable name. The cross-project test will verify thatthe path is valid later.Shouldfix#147265.
@bd1976bris
Copy link
CollaboratorAuthor

bd1976bris commentedJul 15, 2025•
edited
Loading

@Kewen12,@ilovepi and@Prabhuk - should be fixed by:edfec9c. I'm running out of dev time today so if there are any other problems I will revert.

Kewen12 reacted with heart emoji

@Kewen12
Copy link
Contributor

Thanks! I appreciated the quick fix!

@bd1976bris
Copy link
CollaboratorAuthor

@tru - would you be happy for me to backport this to the LLVM 21 release branch to complete the DTLTO feature?

@tru
Copy link
Collaborator

tru commentedJul 17, 2025

@tru - would you be happy for me to backport this to the LLVM 21 release branch to complete the DTLTO feature?

Yes!

bd1976bris reacted with thumbs up emoji

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@ilovepiilovepiilovepi left review comments

@teresajohnsonteresajohnsonteresajohnson approved these changes

@MaskRayMaskRayAwaiting requested review from MaskRay

@trutruAwaiting requested review from tru

@justincadyjustincadyAwaiting requested review from justincady

Assignees
No one assigned
Labels
clang:driver'clang' and 'clang++' user-facing binaries. Not 'clang-cl'clangClang issues not falling into any other category
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

8 participants
@bd1976bris@llvmbot@Prabhuk@ilovepi@Kewen12@tru@MaskRay@teresajohnson
[8]ページ先頭
©2009-2025 Movatter.jp