Enable Cargo's new build-dir layout by ranger-ross · Pull Request #155439 · rust-lang/rust

ranger-ross · 2026-04-17T13:50:00Z

This PR enables the new Cargo build-dir layout in boostrap builds with -Zbuild-dir-new-layout.

See: #t-infra/bootstrap > Has anyone tested `./x` with the new build-dir layout?

bjorn3 · 2026-04-17T13:52:09Z

    let target_root_dir = stamp.path().parent().unwrap();
-    // `target_deps_dir` looks like $dir/$target/release/deps
-    let target_deps_dir = target_root_dir.join("deps");
+    // `target_deps_dir` looks like $dir/$target/release/build


Suggested change

// `target_deps_dir` looks like $dir/$target/release/build

// `target_build_dir` looks like $dir/$target/release/build

View changes since the review

weihanglo · 2026-04-17T13:59:14Z

    // got a list of prefix/extensions and we basically just need to find the
-    // most recent file in the `deps` folder corresponding to each one.
-    let contents = target_deps_dir
+    // most recent file in the `build` folder corresponding to each one.


Not yet read the code by myself. Just wonder why bootstrap are doing this? And if bootstrap needs to do it, it should be a feature in Cargo, at least a (perma-)unstable feature.

View changes since the review

See the commit block around line 2374. Basically we need the non-uplifted name that contains the unique hash. The artifact notifications provide the uplifted name where possible.

ranger-ross · 2026-04-17T15:28:43Z

+    //
+    // Cargo's build folder is structured as `build/<pkg>/<hash>/out/<artifacts>` so
+    // we need to traverse multiple directory layers to get to actual files.
+    let read_dir = |path: &Path| path.read_dir().ok().into_iter().flatten().filter_map(Result::ok);


checks failed because if there are any lingering files/dirs in <build-dir>/<target>/build that are not using the new layout it panics.

Update the original code to not panic if we call .read_dir() on a non-directory. Lets see if CI is happy with this

View changes since the review

ranger-ross · 2026-04-19T12:01:59Z

Looks like the miri job timed out.
Out of curiosity, how stable is this job?

(Sorry, haven't touch the rl/r repo much so I wondering if the failure is related to the new layout or if its normal for spurious failures)

mati865 · 2026-04-19T12:48:31Z

x86_64 Linux jobs rarely hang, I've retried it to be sure.

ranger-ross · 2026-04-19T15:34:48Z

Okay it's hanged on both runs so it's probably the changes.

Is there a way for me to run this check on my machine? I was trying to parse through the GitHub actions to see what command is being run but I was struggling to find it.

weihanglo · 2026-04-19T15:46:14Z

@ranger-ross see https://rustc-dev-guide.rust-lang.org/tests/docker.html#testing-with-docker

cargo run --manifest-path src/ci/citool/Cargo.toml run-local x86_64-gnu-miri

This is the easiest way to run it locally.

ranger-ross · 2026-04-20T14:09:41Z

Okay, so I did some testing and locally it runs fine. (takes around ~20 mins on my machine)

After looking a bit closer at the job logs, it actually never got stuck.
It just take WAY longer than normal....
According to the github stats x86_64-gnu-miri normally takes ~58 mins to run, but we took over 6 hrs and hit the timeout.

Looking at the other jobs, most of them did not have a large perf regression (except x86_64-gnu-tools which was also pretty bad)

That is concerning... 😅

rustbot · 2026-04-27T15:15:47Z

Some changes occurred in compiler/rustc_codegen_cranelift

cc @bjorn3

The Miri subtree was changed

cc @rust-lang/miri

Some changes occurred in src/tools/rustfmt

cc @rust-lang/rustfmt

Some changes occurred in src/tools/compiletest

cc @jieyouxu

rustbot · 2026-04-27T15:15:50Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

RalfJung · 2026-04-27T15:25:18Z

        fail(f"`{target_dir}` contains unexpected files")
    # Ensure something exists inside that target dir.
-    os.access(os.path.join(target_dir, "miri", "debug", "deps"), os.F_OK)
+    os.access(os.path.join(target_dir, "miri", "debug", "build"), os.F_OK)


Any idea why this change was not required when we enabled the new build dir layout in Miri's CI?

View changes since the review

I am not sure, perhaps it existed from a previous run or was created from some non-cargo script?
Cargo should not create deps when -Zbuild-dir-new-layout is enabled.

https://github.com/rust-lang/cargo/blob/3c4678d6a93a072f0a6cf379ea673e35abbe415f/src/cargo/core/compiler/layout.rs#L425-L426

I don't think we cache these folders so I can't explain how they could be left over from a previous run. And nothing else should be touching those folders.

Suggested change

os.access(os.path.join(target_dir, "miri", "debug", "build"), os.F_OK)

os.access(os.path.join(target_dir, "miri", "debug"), os.F_OK)

That should work with old and new layout so we avoid surprising failures when people run this script locally in the Miri subtree.

RalfJung · 2026-04-27T15:26:50Z

    }
 }
+
+/// Gets the to all `out` dirs in a given Cargo `build-dir/<profile>/build` dir.


I cannot make sense of this comment, it seems to have some extra words that shouldn't be there.^^

View changes since the review

Sorry, English is hard. Will fix.

Should be

Gets all of the out dirs in a given Cargo build-dir/<profile>/build dir

Note that the same comment occurs twice in this PR. :)

ranger-ross · 2026-04-27T15:27:49Z

I realized the issue is not with the speed of compilation, but rather that for some reason in CI the compiler (and many other units) are being re-built multiple times which slows everything down and times out.

I haven't figured out why yet. While testing the CI on my fork, I've noticed its inconsistent and sometimes everything only builds once. I'm still investigating why this could be happening. rust-lang/cargo#16345 comes to mind as something that might change this behavior, but I am still unsure whats happening.

I went ahead and pushed some new changes with more fixes for tests and cranelift.

I'll continue investigating this week, but any rl/r experts see any obvious issues with my changes lmk
@rustbot author

bjorn3 · 2026-04-27T17:44:45Z

I have already changed cg_clif to use -Zbuild-dir-new-layout in https://github.com/rust-lang/rustc_codegen_cranelift/ It just hasn't been synced back yet.

View changes since the review

ahh okay, I'll drop that commit from this PR. I think it shouldn't impact the bootstrap changes. (correct me if I am wrong)

ytmimi · 2026-04-27T21:40:03Z

-    // Chop off `deps`.
+    // Chop off `out`.
+    me.pop();
+    // Chop off `<hash>`.
+    me.pop();
+    // Chop off `<pkgname>`.
+    me.pop();
+    // Chop off `build`.


Is this going to break things in rustfmt's CI when we do a subtree push?

View changes since the review

ahh yeah it would probably break, maybe better for me to remove that from this PR and raise a separate PR in the rustfmt repo?

I'm still trying to understand what changes we need in this PR verse what changes can be split out into smaller more targeted changes

In that case, I'd prefer we open up a PR to add the new cargo build dir layout support to rustfmt directly and verify that things are working before making any changes to the rustfmt subtree here in r-l/rust

fair enough. I tested on my fork and I think rust-lang/rust CI fails with out rustfmt changes.

I went ahead and opened rust-lang/rustfmt#6879

rust-bors · 2026-04-28T22:41:43Z

☔ The latest upstream changes (presumably #155941) made this pull request unmergeable. Please resolve the merge conflicts.

rustbot assigned bjorn3 Apr 17, 2026

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Apr 17, 2026

ranger-ross mentioned this pull request Apr 17, 2026

Stabilize build-dir layout v2 rust-lang/cargo#16807

Open

3 tasks

bjorn3 reviewed Apr 17, 2026

View reviewed changes

weihanglo reviewed Apr 17, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

ranger-ross force-pushed the cargo-new-build-dir-layout branch from bf07fb8 to eae2cbf Compare April 17, 2026 15:27

ranger-ross commented Apr 17, 2026

View reviewed changes

This comment was marked as outdated.

Sign in to view

ranger-ross added 3 commits April 28, 2026 00:14

Enable Cargo's new build-dir layout in boostrap

6670955

Enable Cargo's new layout in rustc-codegen-cranelift

a1cfec4

Enable Cargo's new layout in tests

ad47c85

ranger-ross force-pushed the cargo-new-build-dir-layout branch from eae2cbf to ad47c85 Compare April 27, 2026 15:15

rustbot added A-compiletest Area: The compiletest test runner A-testsuite Area: The testsuite used to check the correctness of rustc T-rustfmt Relevant to the rustfmt team, which will review and decide on the PR/issue. labels Apr 27, 2026

ranger-ross changed the title ~~Enable Cargo's new build-dir layout in boostrap~~ Enable Cargo's new build-dir layout Apr 27, 2026

RalfJung reviewed Apr 27, 2026

View reviewed changes

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 27, 2026

bjorn3 reviewed Apr 27, 2026

View reviewed changes

ytmimi reviewed Apr 27, 2026

View reviewed changes

ranger-ross mentioned this pull request Apr 29, 2026

Enable Cargo's new build-dir layout in tests+ci rust-lang/rustfmt#6879

Open

	// `target_deps_dir` looks like $dir/$target/release/build
	// `target_build_dir` looks like $dir/$target/release/build

	os.access(os.path.join(target_dir, "miri", "debug", "build"), os.F_OK)
	os.access(os.path.join(target_dir, "miri", "debug"), os.F_OK)

Uh oh!

Conversation

ranger-ross commented Apr 17, 2026

Uh oh!

bjorn3 Apr 17, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

weihanglo Apr 17, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

ranger-ross Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ranger-ross commented Apr 19, 2026

Uh oh!

mati865 commented Apr 19, 2026

Uh oh!

ranger-ross commented Apr 19, 2026

Uh oh!

weihanglo commented Apr 19, 2026

Uh oh!

ranger-ross commented Apr 20, 2026

Uh oh!

This comment was marked as outdated.

rustbot commented Apr 27, 2026

Uh oh!

rustbot commented Apr 27, 2026

Uh oh!

RalfJung Apr 27, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RalfJung Apr 27, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ranger-ross commented Apr 27, 2026

Uh oh!

bjorn3 Apr 27, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ytmimi Apr 27, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rust-bors Bot commented Apr 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

bjorn3 Apr 17, 2026 •

edited by rustbot

Loading

weihanglo Apr 17, 2026 •

edited by rustbot

Loading

ranger-ross Apr 17, 2026 •

edited

Loading

RalfJung Apr 27, 2026 •

edited by rustbot

Loading

RalfJung Apr 27, 2026 •

edited by rustbot

Loading

bjorn3 Apr 27, 2026 •

edited by rustbot

Loading

ytmimi Apr 27, 2026 •

edited by rustbot

Loading