client: implement client-side compilation mode

.github/workflows/ci.yml

@@ -44,6 +44,8 @@ jobs:
       - name: Clone repository
         uses: actions/checkout@v5
 
+      # TODO: restore `cargo test` once per-feature tests are compatible with
+      # both server-side and client-side compilation modes.
       - name: Check feature ${{ matrix.feature }}
         run: cargo check --no-default-features --features ${{ matrix.feature }}
 
@@ -78,6 +80,10 @@ jobs:
             extra_args: --features=dist-server
           - os: ubuntu-22.04
             rustc: stable
+          - os: ubuntu-22.04
+            rustc: stable
+            extra_desc: client-side-mode
+            test_client_side: true
           - os: ubuntu-22.04
             rustc: beta
           - os: ubuntu-22.04
@@ -98,7 +104,14 @@ jobs:
             extra_desc: cuda12.8
           # # M1 CPU
           - os: macos-14
+          - os: macos-14
+            extra_desc: client-side-mode
+            test_client_side: true
           - os: macos-15-intel
+          - os: windows-2022
+            extra_desc: client-side-mode
+            test_client_side: true
+            no_coverage: true
           - os: windows-2022
             cuda: "12.8"
             # Oldest supported version, keep in sync with README.md
@@ -209,6 +222,7 @@ jobs:
           CARGO_INCREMENTAL: "0"
           RUSTC_WRAPPER: ""
           RUSTFLAGS: "-Cinstrument-coverage -Ccodegen-units=1 -Copt-level=0 -Coverflow-checks=off"
+          SCCACHE_CLIENT_SIDE_COMPILE: ${{ matrix.test_client_side && '1' || '0' }}
 
       - name: Upload failure
         if: failure()

.pre-commit-config.yaml

@@ -11,7 +11,7 @@ repos:
     -   id: rust-clippy
         name: Rust clippy
         description: Run cargo clippy on files included in the commit.
-        entry: cargo +nightly clippy --workspace --all-targets --all-features --
+        entry: cargo +nightly clippy --workspace --all-targets --features all --
         pass_filenames: false
         types: [file, rust]
         language: system

README.md

@@ -165,6 +165,13 @@ If you don't [specify otherwise](#storage-options), sccache will use a local dis
 
 sccache works using a client-server model, where the server runs locally on the same machine as the client. The client-server model allows the server to be more efficient by keeping some state in memory. The sccache command will spawn a server process if one is not already running, or you can run `sccache --start-server` to start the background server process without performing any compilation.
 
+### Compilation Modes
+
+sccache supports two compilation modes:
+
+- **Server-side compilation (default)**: The server performs compiler detection, preprocessing, and compilation. This is the traditional mode.
+- **Client-side compilation (experimental)**: The client performs all compilation work, and the server acts as a pure cache storage service. Enable this mode by setting `SCCACHE_CLIENT_SIDE_COMPILE=1`. This mode provides better scalability and reduced server load but is currently experimental. When compilation is not cacheable, the client runs the compiler directly. See [Architecture.md](docs/Architecture.md) for detailed comparison.
+
 By default sccache server will listen on `127.0.0.1:4226`, you can specify environment variable `SCCACHE_SERVER_PORT` to use a different port or `SCCACHE_SERVER_UDS` to listen on unix domain socket. Abstract unix socket is also supported as long as the path is escaped following the [format](https://doc.rust-lang.org/std/ascii/fn.escape_default.html). For example:
 
 ```

docs/Architecture.md

@@ -1,21 +1,166 @@
 # Sccache high level architecture
 
-This schema shows at high level how sccache works.
+Sccache supports two compilation modes: **server-side compilation** (legacy) and **client-side compilation** (new). The mode is controlled by the `SCCACHE_CLIENT_SIDE_COMPILE` environment variable.
 
+## Server-Side Compilation (Legacy Mode)
+
+This is the default mode when `SCCACHE_CLIENT_SIDE_COMPILE` is unset or set to `0`.
+
+In this mode, the server performs all compilation work:
+
+```mermaid
+  sequenceDiagram
+      participant Client as Client Process
+      participant Server as Sccache Server
+      participant Storage as Cache Storage
+
+      Client->>Server: 1. Compile Request (exe, args, cwd, env)
+      Server->>Server: 2. Detect Compiler
+      Server->>Server: 3. Preprocess & Hash
+      Server->>Storage: 4. Check Cache
+      alt Cache Lookup Result: Hit
+          rect rgba(0, 128, 0, 0.1)
+              Storage-->>Server: Cached Entry
+              Server-->>Client: 5a. Return Cached Result
+          end
+      else Hit: No - Cache Miss
+          rect rgba(200, 0, 0, 0.1)
+              Storage-->>Server: Miss
+              Server->>Server: 5b. Compile Locally
+              Server->>Storage: 6. Store Result
+              Server-->>Client: 7. Return Result
+          end
+      end
+```
+
+**Characteristics**:
+- Server performs compiler detection, preprocessing, hash generation, and compilation
+- All work happens on the server machine
+- Server can become a bottleneck with many parallel clients
+- Higher server CPU and memory usage
+
+## Client-Side Compilation (New Mode)
+
+Enabled by setting `SCCACHE_CLIENT_SIDE_COMPILE=1`.
+
+In this mode, the client performs compilation work and the server acts as pure storage:
+
+```mermaid
+  sequenceDiagram
+      participant Client as Client Process
+      participant Server as Sccache Server (Storage Only)
+      participant Storage as Cache Storage
+
+      Client->>Client: 1. Detect Compiler
+      Client->>Client: 2. Preprocess & Hash
+      Client->>Server: 3. CacheGet Request (cache_key)
+      Server->>Storage: 4. Query Storage
+      alt Cache Lookup Result: Hit
+          rect rgba(0, 128, 0, 0.1)
+              Storage-->>Server: Cached Entry
+              Server-->>Client: 5a. Return Cache Entry
+              Client->>Client: Use Cached Result
+          end
+      else Hit: No - Cache Miss
+          rect rgba(200, 0, 0, 0.1)
+              Storage-->>Server: Miss
+              Server-->>Client: 5b. Cache Miss
+              Client->>Client: 6. Compile Locally
+              Client->>Server: 7. CachePut Request (cache_key, entry)
+              Server->>Storage: 8. Store in Cache
+          end
+      end
+```
+
+**Characteristics**:
+- Client performs compiler detection, preprocessing, hash generation, and compilation
+- Server only handles cache storage operations (get/put)
+- Work is distributed across all clients (better scalability)
+- Lower server CPU and memory usage
+- Reduced network latency (single request instead of multiple round trips)
+
+**Why this is fast**: preprocessing in client-side mode is cheap — it only concatenates source files rather than running the full C/C++ preprocessor. This avoids the expensive `#include` expansion and macro evaluation that dominates traditional preprocessing time, making it practical to move this work to the client without a performance penalty.
+
+**Note**: Client-side compilation is functional but considered experimental. Enable it by setting `SCCACHE_CLIENT_SIDE_COMPILE=1`.
+
+## Comparison
+
+| Aspect | Server-Side (Legacy) | Client-Side (New) |
+|--------|---------------------|-------------------|
+| Compiler Detection | Server | Client (with caching) |
+| Preprocessing | Server | Client |
+| Hash Generation | Server | Client |
+| Compilation | Server | Client |
+| Server Role | Full compilation service | Pure storage service |
+| Server CPU Usage | High | Low |
+| Server Memory Usage | Moderate | Low |
+| Client Overhead | Low | Moderate |
+| Scalability | Limited by server | Excellent |
+| Network Requests | Multiple round trips | Single request |
+
+## Cache Key Generation
+
+Regardless of the mode, cache keys are generated from:
 
 ```mermaid
   flowchart LR
       id1[[Environment variables]] --> hash
       id2[[Compiler binary]] --> hash
       id3[[Compiler arguments]] --> hash
-      id5[[Files]] --> |  | hash
-      Compile --> Upload
-      Storage[(Storage)] --> | yes | Download
-      hash([hash]) --> | exists? | Storage
-      Storage --> | no | Compile
-      Upload --> Storage
+      id5[[Preprocessed Files]] --> hash
+      hash([BLAKE3 Hash]) --> key[Cache Key]
+
+      style id1 fill:#e8f4fd,stroke:#5dade2,color:#333
+      style id2 fill:#e8f4fd,stroke:#5dade2,color:#333
+      style id3 fill:#e8f4fd,stroke:#5dade2,color:#333
+      style id5 fill:#e8f4fd,stroke:#5dade2,color:#333
+      style hash fill:#eafaf1,stroke:#58d68d,color:#333
+      style key fill:#fef9e7,stroke:#f4d03f,color:#333
 ```
 
+### C/C++ vs Rust
+
+The "preprocessing" step differs significantly between languages:
+
+- **C/C++**: runs the compiler's preprocessor (`gcc -E` / `clang -E`) to expand all `#include` directives and macros, producing a single translation unit. The preprocessed output is then hashed. This is the expensive part — include expansion can pull in thousands of header files.
+
+- **Rust**: there is no preprocessor. Instead, sccache runs `rustc --emit dep-info`, a lightweight invocation that outputs a `.d` file listing all source files and environment variables the crate depends on — without compiling anything. Sccache then hashes each source file individually, along with extern crate `.rlib` files, static libraries, and any target JSON file. This dependency discovery step is fast compared to full compilation.
+
+In client-side mode, this work moves to the client. For Rust, the cost is minimal since `--emit dep-info` is cheap. For C/C++, preprocessing is replaced by simple file concatenation, avoiding the expensive include expansion entirely.
+
+For more details about how hash generation works, see [the caching documentation](Caching.md).
+
+## Protocol
+
+### Server-Side Mode Protocol
+
+- **Request**: `Compile(Compile)` - Contains executable path, arguments, working directory, environment variables
+- **Response**: `CompileFinished(CompileFinished)` - Contains exit code, stdout, stderr, and output file paths
+
+### Client-Side Mode Protocol
+
+- **Request**: `CacheGet(CacheGetRequest)` - Contains cache key
+- **Response**: `CacheGetResponse::Hit(Vec<u8>)` - Cache entry as bytes
+- **Response**: `CacheGetResponse::Miss` - Cache miss
+- **Request**: `CachePut(CachePutRequest)` - Contains cache key and entry bytes
+- **Response**: `CachePutResponse(Duration)` - Storage duration
+
+The protocol supports version negotiation to maintain backward compatibility during migration from server-side to client-side mode.
+
+## Storage Backends
+
+Both modes use the same cache storage backends:
+
+- **Local Disk** (`SCCACHE_DIR`)
+- **S3 Compatible** (`SCCACHE_BUCKET`, `SCCACHE_ENDPOINT`)
+- **Redis** (`SCCACHE_REDIS_ENDPOINT`)
+- **Memcached** (`SCCACHE_MEMCACHED_ENDPOINT`)
+- **Google Cloud Storage** (`SCCACHE_GCS_BUCKET`)
+- **Azure Blob Storage** (`SCCACHE_AZURE_CONNECTION_STRING`)
+- **GitHub Actions Cache** (`SCCACHE_GHA_CACHE_URL`)
+- **WebDAV** (`SCCACHE_WEBDAV_ENDPOINT`)
+- **Alibaba Cloud OSS** (`SCCACHE_OSS_BUCKET`)
+- **Tencent Cloud COS** (`SCCACHE_COS_BUCKET`)
 
-For more details about hash generation works, see [the caching documentation](Caching.md).
+See [Configuration.md](Configuration.md) for storage backend configuration details.
 

docs/Configuration.md

@@ -165,6 +165,7 @@ Note that some env variables may need sccache server restart to take effect.
 ### misc
 
 * `SCCACHE_ALLOW_CORE_DUMPS` to enable core dumps by the server
+* `SCCACHE_CLIENT_SIDE_COMPILE` when set to `1`, enables client-side compilation mode where the client performs compiler detection, preprocessing, and compilation work, with the server acting as a pure cache storage service. When set to `0` or unset (default), uses the legacy server-side compilation mode. See [Architecture.md](Architecture.md) for more details.
 * `SCCACHE_CONF` configuration file path
 * `SCCACHE_BASEDIRS` base directory (or directories) to strip from paths for cache key computation. This is similar to ccache's `CCACHE_BASEDIR` and enables cache hits across different absolute paths when compiling the same source code. Multiple directories can be separated by `;` on Windows hosts and by `:` on any other operating system. When multiple directories are specified, the longest matching prefix is used. Path matching is **case-insensitive** on Windows and **case-sensitive** on other operating systems. Environment variable takes precedence over file configuration. Only absolute paths are supported; relative paths will cause an error and prevent the server from start.
 * `SCCACHE_CACHED_CONF`

src/cache/cache_io.rs

@@ -13,6 +13,7 @@
 use super::utils::{get_file_mode, set_file_mode};
 use crate::errors::*;
 use fs_err as fs;
+use serde::{Deserialize, Serialize};
 use std::fmt;
 use std::io::{Cursor, Read, Seek, Write};
 use std::path::PathBuf;
@@ -21,7 +22,7 @@ use zip::write::FileOptions;
 use zip::{CompressionMethod, ZipArchive, ZipWriter};
 
 /// Cache object sourced by a file.
-#[derive(Clone)]
+#[derive(Clone, Debug, Serialize, Deserialize)]
 pub struct FileObjectSource {
     /// Identifier for this object. Should be unique within a compilation unit.
     /// Note that a compilation unit is a single source file in C/C++ and a crate in Rust.