Skip to content

Add support for ROCm / Strix Halo#771

Open
keithmattix wants to merge 7 commits intokaito-project:mainfrom
keithmattix:add-strix-halo-support
Open

Add support for ROCm / Strix Halo#771
keithmattix wants to merge 7 commits intokaito-project:mainfrom
keithmattix:add-strix-halo-support

Conversation

@keithmattix
Copy link
Copy Markdown

What this PR does / why we need it:
AMD is one of the leaders in AI hardware, especially for individual consumers. LocalAI supports their ROCm acceleration features and AIKit should be able to as well.

Which issue(s) this PR fixes (optional, using fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when the PR gets merged):

Special notes for your reviewer:
I tested this PR with my personal Framework Desktop (containing a Strix Halo processor). It depends on mudler/LocalAI#9410 to work end to end.

Signed-off-by: Keith Mattix <keithmattix2@gmail.com>
Signed-off-by: Keith Mattix <keithmattix2@gmail.com>
Signed-off-by: Keith Mattix <keithmattix2@gmail.com>
@keithmattix keithmattix requested a review from sozercan as a code owner April 18, 2026 17:46
Copilot AI review requested due to automatic review settings April 18, 2026 17:46
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds AMD ROCm (incl. Strix Halo) as a supported GPU runtime path in AIKit by wiring a new rocm runtime through validation, image/build generation, backend selection, and docs/examples.

Changes:

  • Introduces runtime: rocm with ROCm 7.2 installation during image builds and ROCm-specific env configuration.
  • Adds ROCm llama-cpp backend tag/name selection (plus CPU fallback install) and a new ROCm test aikitfile.
  • Updates docs and developer tooling (Makefile target + contributing note) to document/run ROCm images.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
website/docs/llama-cpp.md Documents ROCm as a supported llama.cpp runtime and links a ROCm example aikitfile.
website/docs/gpu.md Adds a ROCm (experimental) guide, setup, run instructions, and troubleshooting.
test/aikitfile-llama-rocm.yaml Adds a ROCm test model build configuration for llama-cpp.
pkg/utils/const.go Adds RuntimeROCm and Ubuntu 24.04 base constant for ROCm images.
pkg/build/build.go Extends runtime validation to allow rocm and restricts ROCm to llama-cpp backend.
pkg/aikit2llb/inference/image.go Adds ROCm environment variables to generated image config.
pkg/aikit2llb/inference/convert.go Switches ROCm builds to Ubuntu 24.04 and installs ROCm apt repo + packages.
pkg/aikit2llb/inference/backend_test.go Adds a unit test for ROCm backend tag selection.
pkg/aikit2llb/inference/backend.go Adds ROCm backend tag/name mapping and CPU fallback installation for ROCm.
Makefile Adds run-test-model-rocm helper target for local ROCm runs.
CONTRIBUTING.md Adds buildx/platform tip to avoid multi-platform pitfalls in local development.

Comment thread pkg/build/build.go
Comment thread pkg/aikit2llb/inference/image.go Outdated
Comment thread website/docs/gpu.md Outdated
keithmattix and others added 4 commits April 18, 2026 13:15
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Keith Mattix II <keithmattix2@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Keith Mattix II <keithmattix2@gmail.com>
Signed-off-by: Keith Mattix <keithmattix2@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants