Add support for ROCm / Strix Halo#771
Open
keithmattix wants to merge 7 commits intokaito-project:mainfrom
Open
Add support for ROCm / Strix Halo#771keithmattix wants to merge 7 commits intokaito-project:mainfrom
keithmattix wants to merge 7 commits intokaito-project:mainfrom
Conversation
Signed-off-by: Keith Mattix <keithmattix2@gmail.com>
Signed-off-by: Keith Mattix <keithmattix2@gmail.com>
Signed-off-by: Keith Mattix <keithmattix2@gmail.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds AMD ROCm (incl. Strix Halo) as a supported GPU runtime path in AIKit by wiring a new rocm runtime through validation, image/build generation, backend selection, and docs/examples.
Changes:
- Introduces
runtime: rocmwith ROCm 7.2 installation during image builds and ROCm-specific env configuration. - Adds ROCm llama-cpp backend tag/name selection (plus CPU fallback install) and a new ROCm test aikitfile.
- Updates docs and developer tooling (Makefile target + contributing note) to document/run ROCm images.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| website/docs/llama-cpp.md | Documents ROCm as a supported llama.cpp runtime and links a ROCm example aikitfile. |
| website/docs/gpu.md | Adds a ROCm (experimental) guide, setup, run instructions, and troubleshooting. |
| test/aikitfile-llama-rocm.yaml | Adds a ROCm test model build configuration for llama-cpp. |
| pkg/utils/const.go | Adds RuntimeROCm and Ubuntu 24.04 base constant for ROCm images. |
| pkg/build/build.go | Extends runtime validation to allow rocm and restricts ROCm to llama-cpp backend. |
| pkg/aikit2llb/inference/image.go | Adds ROCm environment variables to generated image config. |
| pkg/aikit2llb/inference/convert.go | Switches ROCm builds to Ubuntu 24.04 and installs ROCm apt repo + packages. |
| pkg/aikit2llb/inference/backend_test.go | Adds a unit test for ROCm backend tag selection. |
| pkg/aikit2llb/inference/backend.go | Adds ROCm backend tag/name mapping and CPU fallback installation for ROCm. |
| Makefile | Adds run-test-model-rocm helper target for local ROCm runs. |
| CONTRIBUTING.md | Adds buildx/platform tip to avoid multi-platform pitfalls in local development. |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Keith Mattix II <keithmattix2@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Keith Mattix II <keithmattix2@gmail.com>
Signed-off-by: Keith Mattix <keithmattix2@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
AMD is one of the leaders in AI hardware, especially for individual consumers. LocalAI supports their ROCm acceleration features and AIKit should be able to as well.
Which issue(s) this PR fixes (optional, using
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when the PR gets merged):Special notes for your reviewer:
I tested this PR with my personal Framework Desktop (containing a Strix Halo processor). It depends on mudler/LocalAI#9410 to work end to end.