[Core] CI step added to check external github url by msarfraz · Pull Request #9831 · Azure/azure-cli-extensions

msarfraz · 2026-04-29T03:11:13Z

Description
This PR removes the dependency on GitHub (raw.githubusercontent.com) VM image aliases, replacing it with Azure Blob Storage (azcliprod.blob.core.windows.net). This change enables Azure CLI to work properly in network isolated environments where GitHub access is blocked.

In addition, new validation added in CI pipeline to flag if any raw.githubusercontent.com URL is used in the code.

Background
In enterprise environments with strict network isolation policies, access to raw.githubusercontent.com is not allowed.

Changes

VM Image Alias Migration

Before:
https://raw.githubusercontent.com/Azure/azure-rest-api-specs/master/arm-compute/quickstart-templates/aliases.json
After:
https://azcliprod.blob.core.windows.net/cli/vm/aliases_master.json

Before:
https://raw.githubusercontent.com/Azure/azure-rest-api-specs/main/arm-compute/quickstart-templates/aliases.json
After:
https://azcliprod.blob.core.windows.net/cli/vm/aliases.json

This checklist is used to make sure that common guidelines for a pull request are followed.

Related command

General Guidelines

Have you run azdev style <YOUR_EXT> locally? (pip install azdev required)
Have you run python scripts/ci/test_index.py -q locally? (pip install wheel==0.30.0 required)
My extension version conforms to the Extension version schema

For new extensions:

My extension description/summary conforms to the Extension Summary Guidelines.

About Extension Publish

There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update src/index.json automatically.
You only need to update the version information in file setup.py and historical information in file HISTORY.rst in your PR but do not modify src/index.json.

azure-client-tools-bot-prd · 2026-04-29T03:11:20Z

️✔️Azure CLI Extensions Breaking Change Test

️✔️Non Breaking Changes

azure-client-tools-bot-prd · 2026-04-29T03:11:22Z

Hi @msarfraz,
Please write the description of changes which can be perceived by customers into HISTORY.rst.
If you want to release a new extension version, please update the version in setup.py as well.

yonzhan · 2026-04-29T03:11:25Z

Thank you for your contribution! We will review the pull request and get back to you soon.

github-actions · 2026-04-29T03:11:29Z

The git hooks are available for azure-cli and azure-cli-extensions repos. They could help you run required checks before creating the PR.

Please sync the latest code with latest dev branch (for azure-cli) or main branch (for azure-cli-extensions).
After that please run the following commands to enable git hooks:

pip install azdev --upgrade
azdev setup -c <your azure-cli repo path> -r <your azure-cli-extensions repo path>

github-actions · 2026-04-29T03:12:25Z

For more info about extension versioning, please refer to Extension version schema

Copilot

Pull request overview

Adds a CI guardrail to prevent introducing new raw.githubusercontent.com dependencies (problematic in network-isolated environments) by scanning PR diffs for forbidden external GitHub raw URLs and maintaining an exclusions list for known/intentional occurrences.

Changes:

Added scripts/ci/validate_external_source_urls.py to scan added diff lines for forbidden raw.githubusercontent.com URLs.
Added scripts/ci/external_url_exclusions.json to configure path-based exclusions for the validator.
Added a new CheckExternalUrls Azure Pipelines job to run the validator in PR builds.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File	Description
scripts/ci/validate_external_source_urls.py	New diff-based validator that fails CI when forbidden raw GitHub URLs are added.
scripts/ci/external_url_exclusions.json	Exclusions configuration consumed by the validator to skip docs/tests/etc.
azure-pipelines.yml	New pipeline job to execute the validator during CI.

Copilot · 2026-04-29T03:16:48Z

+        "scripts/*"
+      ],
+      "_justification": "CI and tooling scripts are maintained separately from extension source content and are excluded to avoid self-flagging the validator configuration."


The exclusion rule "scripts/*" effectively disables validation for all scripts (including CI tooling). That makes it easy for new raw.githubusercontent.com usages to be introduced in scripts without being flagged. Consider narrowing this exclusion to only the specific files that need to self-contain the checker (e.g., the validator script/config), rather than excluding the entire scripts/ tree.

Suggested change

"scripts/*"

],

"_justification": "CI and tooling scripts are maintained separately from extension source content and are excluded to avoid self-flagging the validator configuration."

"scripts/ci/external_url_exclusions.json"

],

"_justification": "The external URL validator configuration is excluded so it can self-contain the allowlist and justification text without self-flagging."

Copilot · 2026-04-29T03:16:48Z

+def _run_diff(src: str, tgt: str, cached: bool = False) -> str:
+    cmd = ["git", "diff", "--unified=0", "--no-color"]
+    if cached:
+        cmd.append("--cached")
+    else:
+        cmd.append(f"{tgt}...{src}")
+
+    proc = subprocess.run(
+        cmd,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+        text=True,
+        check=False,
+    )
+    if proc.returncode != 0:
+        raise RuntimeError(proc.stderr.strip() or "git diff failed")
+    return proc.stdout


git diff {tgt}...{src} relies on computing a merge-base. In CI with shallow fetches, the merge-base often isn't present and this form can fail (or produce an incomplete diff). Either fetch sufficient history/unshallow so the merge-base is available, or switch to a diff form that doesn't require merge-base (e.g., tgt..src) depending on the intended semantics.

Copilot · 2026-04-29T03:16:49Z

+- job: CheckExternalUrls
+  displayName: "Check External Source URLs"
+  pool:


This job runs for non-PR builds as well, and then exits early in the script. To avoid consuming an extra agent/job on every branch build, add a job-level condition similar to other PR-only jobs (e.g. eq(variables['Build.Reason'], 'PullRequest')) instead of relying on an in-script check.

Copilot · 2026-04-29T03:16:49Z

+          # If CI is set to shallow fetch, target branch should be explicitly fetched.
+          # External URL exclusions are maintained in scripts/ci/external_url_exclusions.json.
+          git fetch origin --depth=1 $(System.PullRequest.TargetBranch)


The pipeline fetches the target branch with --depth=1, but the validator uses a git diff form that needs merge-base history. With shallow history this can cause the validation step to fail unexpectedly. Consider increasing fetch depth (or unshallow) enough to compute the merge base, or adjust the validator to avoid relying on merge-base diffs.

Suggested change

# If CI is set to shallow fetch, target branch should be explicitly fetched.

# External URL exclusions are maintained in scripts/ci/external_url_exclusions.json.

git fetch origin --depth=1 $(System.PullRequest.TargetBranch)

# If CI is set to shallow fetch, ensure enough history is available

# for validators that may need to compute a merge base.

if [[ "$(git rev-parse --is-shallow-repository)" == "true" ]]; then

git fetch origin --unshallow

fi

# External URL exclusions are maintained in scripts/ci/external_url_exclusions.json.

git fetch origin "$(System.PullRequest.TargetBranch):refs/remotes/origin/$(System.PullRequest.TargetBranch)"

Copilot · 2026-04-29T03:16:49Z

+    parser = argparse.ArgumentParser(description="Check diff for forbidden raw github URL usage.")
+    parser.add_argument("--src", default="HEAD", help="Source ref/commit for git diff.")
+    parser.add_argument("--tgt", default="HEAD~1", help="Target ref/commit for git diff.")
+    parser.add_argument("--cached", action="store_true", help="Check staged changes in git index.")
+    args = parser.parse_args()
+
+    try:
+        _get_excluded_path_patterns()
+        diff_text = _run_diff(src=args.src, tgt=args.tgt, cached=args.cached)
+    except Exception as ex:  # pylint: disable=broad-except
+        if args.cached:
+            print(f"Unable to evaluate staged diff: {ex}", file=sys.stderr)
+        else:
+            print(f"Unable to evaluate diff between '{args.tgt}' and '{args.src}': {ex}", file=sys.stderr)
+        return 1
+
+    violations = _find_violations(diff_text)
+    if not violations:
+        print("No forbidden external github URL found in added lines.")
+        return 0


User-facing output and argparse description use inconsistent capitalization ("github" vs "GitHub"). Since this is a CI validation message, it’s worth standardizing to "GitHub" throughout for clarity/professionalism.

Copilot · 2026-04-29T03:16:49Z

+- job: CheckExternalUrls
+  displayName: "Check External Source URLs"


PR description mentions migrating VM image alias URLs from raw.githubusercontent.com to Azure Blob Storage, but the changes in this PR appear to only add the CI validator + exclusions/pipeline job. Either update the PR description to match the actual scope, or include the alias migration changes here if they’re intended to be part of this PR.

github-actions · 2026-04-30T01:38:46Z

CodeGen Tools Feedback Collection

Thank you for using our CodeGen tool. We value your feedback, and we would like to know how we can improve our product. Please take a few minutes to fill our codegen survey

This reverts commit cdc1547.

CI step added to check external github url

f245264

Copilot AI review requested due to automatic review settings April 29, 2026 03:11

azure-client-tools-bot-prd Bot added this to the June 2026 (2026-06-02) milestone Apr 29, 2026

microsoft-github-policy-service Bot requested review from Pan-Qi, bebound, jiasli, jsntcy, kairu-ms, wangzelin007 and yonzhan April 29, 2026 03:11

microsoft-github-policy-service Bot added Auto-Assign Auto assign by bot CI labels Apr 29, 2026

microsoft-github-policy-service Bot assigned wangzelin007 Apr 29, 2026

Copilot started reviewing on behalf of msarfraz April 29, 2026 03:12 View session

Copilot AI reviewed Apr 29, 2026

View reviewed changes

updated githubusercontent main aliases.json to azure blob

cdc1547

msarfraz requested a review from calvinhzy as a code owner April 30, 2026 01:37

Revert "updated githubusercontent main aliases.json to azure blob"

ce5b820

This reverts commit cdc1547.

-          # If CI is set to shallow fetch, target branch should be explicitly fetched.
-          # External URL exclusions are maintained in scripts/ci/external_url_exclusions.json.
-          git fetch origin --depth=1 $(System.PullRequest.TargetBranch)
+          # If CI is set to shallow fetch, ensure enough history is available
+          # for validators that may need to compute a merge base.
+          if [[ "$(git rev-parse --is-shallow-repository)" == "true" ]]; then
+            git fetch origin --unshallow
+          fi
+          # External URL exclusions are maintained in scripts/ci/external_url_exclusions.json.
+          git fetch origin "$(System.PullRequest.TargetBranch):refs/remotes/origin/$(System.PullRequest.TargetBranch)"

		- job: CheckExternalUrls
		displayName: "Check External Source URLs"

Conversation

msarfraz commented Apr 29, 2026

Before: https://raw.githubusercontent.com/Azure/azure-rest-api-specs/main/arm-compute/quickstart-templates/aliases.json After: https://azcliprod.blob.core.windows.net/cli/vm/aliases.json

Related command

General Guidelines

About Extension Publish

Uh oh!

azure-client-tools-bot-prd Bot commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

azure-client-tools-bot-prd Bot commented Apr 29, 2026

Uh oh!

yonzhan commented Apr 29, 2026

Uh oh!

github-actions Bot commented Apr 29, 2026

Uh oh!

github-actions Bot commented Apr 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 30, 2026

CodeGen Tools Feedback Collection

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Before:
https://raw.githubusercontent.com/Azure/azure-rest-api-specs/main/arm-compute/quickstart-templates/aliases.json
After:
https://azcliprod.blob.core.windows.net/cli/vm/aliases.json

azure-client-tools-bot-prd Bot commented Apr 29, 2026 •

edited

Loading