BUG: make build_script_run write to BUILD_OUT_DIR instead of OUT_DIR

[qknight]

Problem

The problem discussed here is a serious problem for compiling crates with nix.

I want to contribute this as a RFC so that upstream cargo changes the standard.

Main required difference would be:

• build.rs must write to BUILD_OUT_DIR instead of OUT_DIR

Technically this means we will have to update a few build.rs scripts in some crates on crates.io

Luckily this specification 'error' has been fixed for the upcoming unstable feature multiple-build-scripts (already in nightly).

However, we need to fix it for the 'simple' build.rs specification also!

problematic crates

This is a collection of build.rs examples which don't work with the nix backend while they work with the legacy backend.

The main cause is that cargo creates 4 units to compile:

1. coreutils-0.5.0-script_build-f1108c19c7d0dd7a.nix
2. coreutils-0.5.0-script_build_run-7d8760345f435e2a.nix
3. coreutils-0.5.0-a1a4167b4152c2f4.nix
4. coreutils-0.5.0-bin-f0d6fd778fb8e17e.nix

In the legacy backend all 4 build steps share the same working directory. In contrast the nix-backend: we have 4 different OUT_DIR folders, one per build, so (2.) coreutils-0.5.0-script_build_run-7d8760345f435e2a.nix writes to /nix/store/path2 (OUT_DIR) and (3.) coreutils-0.5.0-a1a4167b4152c2f4.nix assumes files in OUT_DIR which is now /nix/store/path3.
Step (2.), in this coreutils example, the generated uutils_map.rs is used from (4.) coreutils-0.5.0-bin-f0d6fd778fb8e17e.nix which has OUT_DIR set to /nix/store/path4.

This does not even cover the experimental feature of multiple-build-scripts but it comes with a nice feature:

Accessing Output Directories: Output directory of each build script can be accessed by using _OUT_DIR where the is the file-stem of the build script, exactly as-is. For example, bar_OUT_DIR for script at foo/bar.rs. (Only set during compilation, can be accessed via env! macro)

https://doc.rust-lang.org/cargo/reference/unstable.html#multiple-build-scripts

coreutils

https://github.com/uutils/coreutils/blob/5fc9f8e7e0e47c9d7a6e29a63f5c6bbd3e5cf558/build.rs#L50

In nix, the OUT_DIR of the last stage, coreutils.rs, which wants to use uutils_map.rs will not find it in OUT_DIR but rather in the current directory.

diff --git a/src/bin/coreutils.rs b/src/bin/coreutils.rs
index 6a9141936..59b9a37e5 100644
--- a/src/bin/coreutils.rs
+++ b/src/bin/coreutils.rs
@@ -12,7 +12,7 @@ use std::process;
 
 const VERSION: &str = env!("CARGO_PKG_VERSION");
 
-include!(concat!(env!("OUT_DIR"), "/uutils_map.rs"));
+include!(concat!(env!("BUILD_OUT_DIR"), "/uutils_map.rs"));

/home/nixos/tests/coreutils/target/debug/nix/derivations/coreutils-0.5.0-a1a4167b4152c2f4.nix
+    BUILD_OUT_DIR="${coreutils-0_5_0-script_build_run-7d8760345f435e2a}";

/home/nixos/tests/coreutils/target/debug/nix/derivations/coreutils-0.5.0-bin-f0d6fd778fb8e17e.nix
+    BUILD_OUT_DIR="${coreutils-0_5_0-script_build_run-7d8760345f435e2a}";

alternative nix implementations to cargo

let's see how others solve the build.rs issue

crane

crane reimplements cargo and grok says this about it:

Standard Handling of build.rs in Crane

Integration into Build Phases: Crane doesn't require manually splitting the build script compilation, execution, and crate compilation into separate derivations (as you're doing). Instead, it typically handles everything in a unified way via functions like buildPackage or mkCargoDerivation. During the build:
Cargo compiles the build.rs script into an executable.
It then runs the executable, which can generate files (e.g., your uutils_map.rs) and write them to OUT_DIR (a Cargo-set environment variable pointing to a unique, writable directory like target//build/-/out within the build environment).
Finally, Cargo compiles the main crate, setting the same OUT_DIR value for the rustc invocation. This allows macros like include!(concat!(env!("OUT_DIR"), "/uutils_map.rs")) to resolve correctly at compile time, as env!("OUT_DIR") is available and points to the location where the build script wrote the files.

Environment and Hooks: Crane uses hooks (e.g., configureCargoCommonVarsHook, inheritCargoArtifactsHook) to set up environment variables like CARGO_TARGET_DIR (often ./target) and ensure OUT_DIR is writable. Generated files are created in the build directory (not directly in $out), incorporated into the compilation, and not persisted as separate outputs unless needed (since they're baked into the resulting binary or library).
Dependency Separation for Caching: For better incremental builds, Crane separates dependency builds (via buildDepsOnly) from the main crate build. In buildDepsOnly:
Dependencies' build.rs scripts are run, and their outputs are captured as artifacts.
These artifacts (including any generated files or metadata) are inherited into the main buildPackage derivation using cargoArtifacts.
For your main crate's build.rs, it's executed during buildPackage, not in the deps phase.

https://crane.dev/faq/sandbox-unfriendly-build-scripts.html