Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit162120b

Browse files
authored
Rollup merge of#126318 - Kobzol:bootstrap-perf, r=onur-ozkan
Add a `x perf` command for integrating bootstrap with `rustc-perf`This PR adds a new `x perf` command to bootstrap. The idea is to let rustc developers profile (`profile_local`) and benchmark (`bench_local`) a stage1/stage2 compiler directly from within `rust`.Before, if you wanted to use `rustc-perf`, you had to clone it, set it up, copy the `rustc` sysroot after every change to `rust` etc. This is an attempt to automate that.I opened this PR mostly for discussion. My idea is to offer an interface that looks something like this (a random sample of commands):```bashx perf --stage 2 profile eprintlnx perf --stage1 profile cachegrindx perf benchmark --id baselinex perf benchmark --id after-editx perf cmp baseline after-edit```In this PR, I'd like to only implement the simplest case (`profile_local (eprintln)`), because that only requires a single sysroot (you don't compare anything), and it's relatively easy to set up. Also, I'd like to avoid forcing developers to deal with the rustc-perf UI, so more complex use-cases (like benchmarking two sysroots and comparing the results) should probably wait forrust-lang/rustc-perf#1734 (which is hopefully coming along soon-ish).I'm not sure if it's better to do this in bootstrap directly, or if I should create some shim tool that will receive a `rustc` sysroot, and offer a simplified CLI on top of `rustc-perf`.## Why is a separate CLI needed?We definitely need to add some support to bootstrap to automate preparing `rustc-perf` and the `rustc` sysroot, but in theory after that we could just let people invoke `rustc-perf` manually. While that is definitely possible, you'd need to manually figure out where is your sysroot located, which seems annoying to me. The `rustc-perf` CLI is also relatively complex, and for this use-case it makes sense to only use a subset of it. So I thought that it would be better to offer a simplified interface on top of it that would make life easier for contributors. But maybe it's not worth it.CC `@onur-ozkan`
2 parentsf3ced9d +0bd58d8 commit162120b

File tree

11 files changed

+380
-8
lines changed

11 files changed

+380
-8
lines changed

‎src/bootstrap/src/core/build_steps/mod.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ pub(crate) mod doc;
77
pub(crate)mod format;
88
pub(crate)mod install;
99
pub(crate)mod llvm;
10+
pub(crate)mod perf;
1011
pub(crate)mod run;
1112
pub(crate)mod setup;
1213
pub(crate)mod suggest;
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
use std::process::Command;
2+
3+
usecrate::core::build_steps::compile::{Std,Sysroot};
4+
usecrate::core::build_steps::tool::RustcPerf;
5+
usecrate::core::builder::Builder;
6+
usecrate::core::config::DebuginfoLevel;
7+
8+
/// Performs profiling using `rustc-perf` on a built version of the compiler.
9+
pubfnperf(builder:&Builder<'_>){
10+
let collector = builder.ensure(RustcPerf{
11+
compiler: builder.compiler(0, builder.config.build),
12+
target: builder.config.build,
13+
});
14+
15+
if builder.build.config.rust_debuginfo_level_rustc ==DebuginfoLevel::None{
16+
builder.info(r#"WARNING: You are compiling rustc without debuginfo, this will make profiling less useful.
17+
Consider setting `rust.debuginfo-level = 1` in `config.toml`."#);
18+
}
19+
20+
let compiler = builder.compiler(builder.top_stage, builder.config.build);
21+
builder.ensure(Std::new(compiler, builder.config.build));
22+
let sysroot = builder.ensure(Sysroot::new(compiler));
23+
let rustc = sysroot.join("bin/rustc");
24+
25+
let results_dir = builder.build.tempdir().join("rustc-perf");
26+
27+
letmut cmd =Command::new(collector);
28+
let cmd = cmd
29+
.arg("profile_local")
30+
.arg("eprintln")
31+
.arg("--out-dir")
32+
.arg(&results_dir)
33+
.arg("--include")
34+
.arg("helloworld")
35+
.arg(&rustc);
36+
37+
builder.info(&format!("Running `rustc-perf` using `{}`", rustc.display()));
38+
39+
// We need to set the working directory to `src/tools/perf`, so that it can find the directory
40+
// with compile-time benchmarks.
41+
let cmd = cmd.current_dir(builder.src.join("src/tools/rustc-perf"));
42+
builder.build.run(cmd);
43+
44+
builder.info(&format!("You can find the results at `{}`", results_dir.display()));
45+
}

‎src/bootstrap/src/core/build_steps/tool.rs‎

Lines changed: 77 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,8 @@ struct ToolBuild {
3232
extra_features:Vec<String>,
3333
/// Nightly-only features that are allowed (comma-separated list).
3434
allow_features:&'staticstr,
35+
/// Additional arguments to pass to the `cargo` invocation.
36+
cargo_args:Vec<String>,
3537
}
3638

3739
implBuilder<'_>{
@@ -100,6 +102,7 @@ impl Step for ToolBuild {
100102
if !self.allow_features.is_empty(){
101103
cargo.allow_features(self.allow_features);
102104
}
105+
cargo.args(self.cargo_args);
103106
let _guard = builder.msg_tool(
104107
Kind::Build,
105108
self.mode,
@@ -126,10 +129,7 @@ impl Step for ToolBuild {
126129
if tool =="tidy"{
127130
tool ="rust-tidy";
128131
}
129-
let cargo_out = builder.cargo_out(compiler,self.mode, target).join(exe(tool, target));
130-
let bin = builder.tools_dir(compiler).join(exe(tool, target));
131-
builder.copy_link(&cargo_out,&bin);
132-
bin
132+
copy_link_tool_bin(builder,self.compiler,self.target,self.mode, tool)
133133
}
134134
}
135135
}
@@ -214,6 +214,21 @@ pub fn prepare_tool_cargo(
214214
cargo
215215
}
216216

217+
/// Links a built tool binary with the given `name` from the build directory to the
218+
/// tools directory.
219+
fncopy_link_tool_bin(
220+
builder:&Builder<'_>,
221+
compiler:Compiler,
222+
target:TargetSelection,
223+
mode:Mode,
224+
name:&str,
225+
) ->PathBuf{
226+
let cargo_out = builder.cargo_out(compiler, mode, target).join(exe(name, target));
227+
let bin = builder.tools_dir(compiler).join(exe(name, target));
228+
builder.copy_link(&cargo_out,&bin);
229+
bin
230+
}
231+
217232
macro_rules! bootstrap_tool{
218233
($(
219234
$name:ident, $path:expr, $tool_name:expr
@@ -283,6 +298,7 @@ macro_rules! bootstrap_tool {
283298
},
284299
extra_features: vec![],
285300
allow_features: concat!($($allow_features)*),
301+
cargo_args: vec![]
286302
})
287303
}
288304
}
@@ -349,10 +365,60 @@ impl Step for OptimizedDist {
349365
source_type:SourceType::InTree,
350366
extra_features:Vec::new(),
351367
allow_features:"",
368+
cargo_args:Vec::new(),
352369
})
353370
}
354371
}
355372

373+
/// The [rustc-perf](https://github.com/rust-lang/rustc-perf) benchmark suite, which is added
374+
/// as a submodule at `src/tools/rustc-perf`.
375+
#[derive(Debug,Clone,Hash,PartialEq,Eq)]
376+
pubstructRustcPerf{
377+
pubcompiler:Compiler,
378+
pubtarget:TargetSelection,
379+
}
380+
381+
implStepforRustcPerf{
382+
/// Path to the built `collector` binary.
383+
typeOutput =PathBuf;
384+
385+
fnshould_run(run:ShouldRun<'_>) ->ShouldRun<'_>{
386+
run.path("src/tools/rustc-perf")
387+
}
388+
389+
fnmake_run(run:RunConfig<'_>){
390+
run.builder.ensure(RustcPerf{
391+
compiler: run.builder.compiler(0, run.builder.config.build),
392+
target: run.target,
393+
});
394+
}
395+
396+
fnrun(self,builder:&Builder<'_>) ->PathBuf{
397+
// We need to ensure the rustc-perf submodule is initialized.
398+
builder.update_submodule(Path::new("src/tools/rustc-perf"));
399+
400+
let tool =ToolBuild{
401+
compiler:self.compiler,
402+
target:self.target,
403+
tool:"collector",
404+
mode:Mode::ToolBootstrap,
405+
path:"src/tools/rustc-perf",
406+
source_type:SourceType::Submodule,
407+
extra_features:Vec::new(),
408+
allow_features:"",
409+
// Only build the collector package, which is used for benchmarking through
410+
// a CLI.
411+
cargo_args:vec!["-p".to_string(),"collector".to_string()],
412+
};
413+
let collector_bin = builder.ensure(tool.clone());
414+
// We also need to symlink the `rustc-fake` binary to the corresponding directory,
415+
// because `collector` expects it in the same directory.
416+
copy_link_tool_bin(builder, tool.compiler, tool.target, tool.mode,"rustc-fake");
417+
418+
collector_bin
419+
}
420+
}
421+
356422
#[derive(Debug,Clone,Hash,PartialEq,Eq,Ord,PartialOrd)]
357423
pubstructErrorIndex{
358424
pubcompiler:Compiler,
@@ -403,6 +469,7 @@ impl Step for ErrorIndex {
403469
source_type:SourceType::InTree,
404470
extra_features:Vec::new(),
405471
allow_features:"",
472+
cargo_args:Vec::new(),
406473
})
407474
}
408475
}
@@ -437,6 +504,7 @@ impl Step for RemoteTestServer {
437504
source_type:SourceType::InTree,
438505
extra_features:Vec::new(),
439506
allow_features:"",
507+
cargo_args:Vec::new(),
440508
})
441509
}
442510
}
@@ -595,6 +663,7 @@ impl Step for Cargo {
595663
source_type:SourceType::Submodule,
596664
extra_features:Vec::new(),
597665
allow_features:"",
666+
cargo_args:Vec::new(),
598667
})
599668
}
600669
}
@@ -622,6 +691,7 @@ impl Step for LldWrapper {
622691
source_type:SourceType::InTree,
623692
extra_features:Vec::new(),
624693
allow_features:"",
694+
cargo_args:Vec::new(),
625695
})
626696
}
627697
}
@@ -670,6 +740,7 @@ impl Step for RustAnalyzer {
670740
extra_features:vec!["in-rust-tree".to_owned()],
671741
source_type:SourceType::InTree,
672742
allow_features:RustAnalyzer::ALLOW_FEATURES,
743+
cargo_args:Vec::new(),
673744
})
674745
}
675746
}
@@ -717,6 +788,7 @@ impl Step for RustAnalyzerProcMacroSrv {
717788
extra_features:vec!["in-rust-tree".to_owned()],
718789
source_type:SourceType::InTree,
719790
allow_features:RustAnalyzer::ALLOW_FEATURES,
791+
cargo_args:Vec::new(),
720792
});
721793

722794
// Copy `rust-analyzer-proc-macro-srv` to `<sysroot>/libexec/`
@@ -923,6 +995,7 @@ macro_rules! tool_extended {
923995
extra_features: $sel.extra_features,
924996
source_type:SourceType::InTree,
925997
allow_features: concat!($($allow_features)*),
998+
cargo_args: vec![]
926999
});
9271000

9281001
if(false $(|| !$add_bins_to_sysroot.is_empty())?) && $sel.compiler.stage >0{

‎src/bootstrap/src/core/builder.rs‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -666,6 +666,7 @@ pub enum Kind {
666666
Setup,
667667
Suggest,
668668
Vendor,
669+
Perf,
669670
}
670671

671672
implKind{
@@ -687,6 +688,7 @@ impl Kind {
687688
Kind::Setup =>"setup",
688689
Kind::Suggest =>"suggest",
689690
Kind::Vendor =>"vendor",
691+
Kind::Perf =>"perf",
690692
}
691693
}
692694

@@ -698,6 +700,7 @@ impl Kind {
698700
Kind::Run =>"Running",
699701
Kind::Suggest =>"Suggesting",
700702
Kind::Clippy =>"Linting",
703+
Kind::Perf =>"Profiling & benchmarking",
701704
_ =>{
702705
let title_letter =self.as_str()[0..1].to_ascii_uppercase();
703706
returnformat!("{title_letter}{}ing",&self.as_str()[1..]);
@@ -749,7 +752,8 @@ impl<'a> Builder<'a> {
749752
tool::RustdocGUITest,
750753
tool::OptimizedDist,
751754
tool::CoverageDump,
752-
tool::LlvmBitcodeLinker
755+
tool::LlvmBitcodeLinker,
756+
tool::RustcPerf,
753757
),
754758
Kind::Clippy =>describe!(
755759
clippy::Std,
@@ -945,7 +949,7 @@ impl<'a> Builder<'a> {
945949
Kind::Clean =>describe!(clean::CleanAll, clean::Rustc, clean::Std),
946950
Kind::Vendor =>describe!(vendor::Vendor),
947951
// special-cased in Build::build()
948-
Kind::Format |Kind::Suggest =>vec![],
952+
Kind::Format |Kind::Suggest|Kind::Perf=>vec![],
949953
}
950954
}
951955

@@ -1017,6 +1021,7 @@ impl<'a> Builder<'a> {
10171021
path.as_ref().map_or([].as_slice(), |path| std::slice::from_ref(path)),
10181022
),
10191023
Subcommand::Vendor{ ..} =>(Kind::Vendor,&paths[..]),
1024+
Subcommand::Perf{ ..} =>(Kind::Perf,&paths[..]),
10201025
};
10211026

10221027
Self::new_internal(build, kind, paths.to_owned())

‎src/bootstrap/src/core/config/config.rs‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2043,6 +2043,7 @@ impl Config {
20432043
Subcommand::Bench{ ..} => flags.stage.or(bench_stage).unwrap_or(2),
20442044
Subcommand::Dist{ ..} => flags.stage.or(dist_stage).unwrap_or(2),
20452045
Subcommand::Install{ ..} => flags.stage.or(install_stage).unwrap_or(2),
2046+
Subcommand::Perf{ ..} => flags.stage.unwrap_or(1),
20462047
// These are all bootstrap tools, which don't depend on the compiler.
20472048
// The stage we pass shouldn't matter, but use 0 just in case.
20482049
Subcommand::Clean{ ..}
@@ -2080,7 +2081,8 @@ impl Config {
20802081
|Subcommand::Setup{ ..}
20812082
|Subcommand::Format{ ..}
20822083
|Subcommand::Suggest{ ..}
2083-
|Subcommand::Vendor{ ..} =>{}
2084+
|Subcommand::Vendor{ ..}
2085+
|Subcommand::Perf{ ..} =>{}
20842086
}
20852087
}
20862088

‎src/bootstrap/src/core/config/flags.rs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -469,6 +469,9 @@ Arguments:
469469
#[arg(long)]
470470
versioned_dirs:bool,
471471
},
472+
/// Perform profiling and benchmarking of the compiler using the
473+
/// `rustc-perf` benchmark suite.
474+
Perf{},
472475
}
473476

474477
implSubcommand{
@@ -490,6 +493,7 @@ impl Subcommand {
490493
Subcommand::Setup{ ..} =>Kind::Setup,
491494
Subcommand::Suggest{ ..} =>Kind::Suggest,
492495
Subcommand::Vendor{ ..} =>Kind::Vendor,
496+
Subcommand::Perf{ ..} =>Kind::Perf,
493497
}
494498
}
495499

‎src/bootstrap/src/lib.rs‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -659,6 +659,9 @@ impl Build {
659659
Subcommand::Suggest{ run} =>{
660660
return core::build_steps::suggest::suggest(&builder::Builder::new(self),*run);
661661
}
662+
Subcommand::Perf{ ..} =>{
663+
return core::build_steps::perf::perf(&builder::Builder::new(self));
664+
}
662665
_ =>(),
663666
}
664667

‎src/etc/completions/x.py.fish‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ complete -c x.py -n "__fish_use_subcommand" -f -a "run" -d 'Run tools contained
4848
complete-c x.py-n"__fish_use_subcommand"-f-a"setup"-d'Set up the environment for development'
4949
complete-c x.py-n"__fish_use_subcommand"-f-a"suggest"-d'Suggest a subset of tests to run, based on modified files'
5050
complete-c x.py-n"__fish_use_subcommand"-f-a"vendor"-d'Vendor dependencies'
51+
complete-c x.py-n"__fish_use_subcommand"-f-a"perf"-d'Perform profiling and benchmarking of the compiler using the `rustc-perf` benchmark suite'
5152
complete-c x.py-n"__fish_seen_subcommand_from build"-l config-d'TOML configuration file for build'-r-F
5253
complete-c x.py-n"__fish_seen_subcommand_from build"-l build-dir-d'Build directory, overrides `build.build-dir` in `config.toml`'-r-f-a"(__fish_complete_directories)"
5354
complete-c x.py-n"__fish_seen_subcommand_from build"-l build-d'build target of the stage0 compiler'-r-f
@@ -628,3 +629,37 @@ complete -c x.py -n "__fish_seen_subcommand_from vendor" -l llvm-profile-generat
628629
complete-c x.py-n"__fish_seen_subcommand_from vendor"-l enable-bolt-settings-d'Enable BOLT link flags'
629630
complete-c x.py-n"__fish_seen_subcommand_from vendor"-l skip-stage0-validation-d'Skip stage0 compiler validation'
630631
complete-c x.py-n"__fish_seen_subcommand_from vendor"-s h-lhelp-d'Print help (see more with\'--help\')'
632+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l config-d'TOML configuration file for build'-r-F
633+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l build-dir-d'Build directory, overrides `build.build-dir` in `config.toml`'-r-f-a"(__fish_complete_directories)"
634+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l build-d'build target of the stage0 compiler'-r-f
635+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l host-d'host targets to build'-r-f
636+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l target-d'target targets to build'-r-f
637+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l exclude-d'build paths to exclude'-r-F
638+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l skip-d'build paths to skip'-r-F
639+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l rustc-error-format-r-f
640+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l on-fail-d'command to run on failure'-r-f-a"(__fish_complete_command)"
641+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l stage-d'stage to build (indicates compiler to use/test, e.g., stage 0 uses the bootstrap compiler, stage 1 the stage 0 rustc artifacts, etc.)'-r-f
642+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l keep-stage-d'stage(s) to keep without recompiling (pass multiple times to keep e.g., both stages 0 and 1)'-r-f
643+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l keep-stage-std-d'stage(s) of the standard library to keep without recompiling (pass multiple times to keep e.g., both stages 0 and 1)'-r-f
644+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l src-d'path to the root of the rust checkout'-r-f-a"(__fish_complete_directories)"
645+
complete-c x.py-n"__fish_seen_subcommand_from perf"-s j-ljobs-d'number of jobs to run in parallel'-r-f
646+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l warnings-d'if value is deny, will deny warnings if value is warn, will emit warnings otherwise, use the default configured behaviour'-r-f-a"{deny'',warn'',default''}"
647+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l error-format-d'rustc error format'-r-f
648+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l color-d'whether to use color in cargo and rustc output'-r-f-a"{always'',never'',auto''}"
649+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l llvm-skip-rebuild-d'whether rebuilding llvm should be skipped, overriding `skip-rebuld` in config.toml'-r-f-a"{true'',false''}"
650+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l rust-profile-generate-d'generate PGO profile with rustc build'-r-F
651+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l rust-profile-use-d'use PGO profile for rustc build'-r-F
652+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l llvm-profile-use-d'use PGO profile for LLVM build'-r-F
653+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l reproducible-artifact-d'Additional reproducible artifacts that should be added to the reproducible artifacts archive'-r
654+
complete-c x.py-n"__fish_seen_subcommand_from perf"-lset-d'override options in config.toml'-r-f
655+
complete-c x.py-n"__fish_seen_subcommand_from perf"-s v-l verbose-d'use verbose output (-vv for very verbose)'
656+
complete-c x.py-n"__fish_seen_subcommand_from perf"-s i-l incremental-d'use incremental compilation'
657+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l include-default-paths-d'include default paths in addition to the provided ones'
658+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l dry-run-d'dry run; don\'t build anything'
659+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l dump-bootstrap-shims-d'Indicates whether to dump the work done from bootstrap shims'
660+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l json-output-d'use message-format=json'
661+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l bypass-bootstrap-lock-d'Bootstrap uses this value to decide whether it should bypass locking the build process. This is rarely needed (e.g., compiling the std library for different targets in parallel)'
662+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l llvm-profile-generate-d'generate PGO profile with llvm built for rustc'
663+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l enable-bolt-settings-d'Enable BOLT link flags'
664+
complete-c x.py-n"__fish_seen_subcommand_from perf"-l skip-stage0-validation-d'Skip stage0 compiler validation'
665+
complete-c x.py-n"__fish_seen_subcommand_from perf"-s h-lhelp-d'Print help (see more with\'--help\')'

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp