[CI] Add ListPackages and support nested package directories in dev tools#18543
Merged
mrodm merged 13 commits intoelastic:mainfrom Apr 21, 2026
Merged
[CI] Add ListPackages and support nested package directories in dev tools#18543mrodm merged 13 commits intoelastic:mainfrom
mrodm merged 13 commits intoelastic:mainfrom
Conversation
- Add ListPackages function to dev/citools/packages.go that walks a directory and returns sorted paths of all valid packages found. - Extend packageManifest struct with FormatVersion, Type and Version fields, and add IsValid() method mirroring dev/packagenames logic. - Refactor dev/packagenames to delegate package discovery to citools.ListPackages, removing the now-redundant walkPackagePaths. Generated with Claude (claude-sonnet-4-6)
- Rename readPackageManifest to ReadPackageManifest so it can be used from other packages. - Remove dev/packagenames/manifest.go and its local packageManifest type, replacing all usages with citools.ReadPackageManifest. - Update all internal callers (kibana.go, logsdb.go, subscription.go, packages.go) to use the renamed function. Generated with Claude (claude-sonnet-4-6)
- Replace the inline WalkDir loop in validatePackages with a call to citools.ListPackages, removing duplicated package-discovery logic. - Update testdata manifests (devexp, package_1) with the required format_version, name, type and version fields so they are recognised as valid packages by citools.ListPackages/IsValid. Generated with Claude (claude-sonnet-4-6)
- Add nested_packages testdata with packages at two directory levels: one at the top level and two under a category/ subdirectory, one of which has data streams. - Add five new TestValidatePackages cases covering: all packages with explicit owners, a missing owner for a nested package, category-level ownership inherited by nested packages, valid per-stream ownership in a nested package, and missing stream owner in a nested package. Generated with Claude (claude-sonnet-4-6)
- Export manifestFileName as ManifestFileName in dev/citools/packages.go so it can be reused across packages. - Remove the duplicate manifestFileName constant from dev/packagenames and replace its usage with citools.ManifestFileName. - Replace the hardcoded "manifest.yml" string in dev/codeowners with citools.ManifestFileName. Generated with Claude (claude-sonnet-4-6)
Add a termination check for ownerDir == "/" to prevent an infinite loop
when filepath.Dir reaches the filesystem root on non-Windows systems,
where filepath.Dir("/") returns "/" instead of ".".
Generated with the assistance of Claude Sonnet 4.6
Add a ListPackages target that calls citools.ListPackages to print all package paths found under the packages directory, one per line. Generated with Claude (claude-sonnet-4-6)
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
Drop the hardcoded validTypes slice (integration, input, content) and the slices.Contains check from packageManifest.IsValid(). Validity now only requires that format_version, name, type, and version are all non-empty. This allows new package types to be supported in the future without requiring changes to this validation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
3 tasks
🚀 Benchmarks reportPackage
|
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
vpn |
6666.67 | 4329 | -2337.67 (-35.07%) | 💔 |
To see the full report comment with /test benchmark fullreport
Collaborator
Author
|
Merging since CI failures are unrelated to this change. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed commit message
Centralize package discovery logic in
dev/citoolsand extend the codeowners and packagenames dev scripts to support packages stored in nested directory structures (e.g.packages/category/package-name/).Important
Still pending to update the Buildkite scripts to allow testing packages located in nested directories.
WHAT:
dev/citools/packages.go(new): addsListPackages(dir string)whichwalks a directory tree, identifies valid packages by their
manifest.yml, and returns their sorted paths. A package is consideredvalid when its manifest contains a non-empty
format_version,name,type, andversion, withtypebeing one ofintegration,input,or
content. Discovery stops descending once a package root is found,so packages are never double-counted.
dev/citools/packagemanifest.go: exportsReadPackageManifest(previously private), extends
packageManifestwithFormatVersion,Type, andVersionfields, and adds anIsValid()method.dev/codeowners: refactorsvalidatePackagesto delegate discovery tocitools.ListPackages, and introducesfindOwnerForFilewhich walksup the directory tree to resolve ownership for packages nested more
than one level deep. Fixes an infinite loop that occurred when an
absolute path was passed to the owner lookup.
dev/packagenames: removes the local duplicatemanifest.goandwalkPackagePathsimplementation, replacing both with calls tocitools.ListPackagesandcitools.ReadPackageManifest.magefile.go: adds aListPackagesmage target for inspection anddebugging.
WHY:
The repository is adopting a two-level package layout (
packages/<technology>/<package>) alongside the existing flat layout (packages/<package>). The previous implementations incodeownersandpackagenamesassumed a single, flat level and each maintained its own copy of manifest-reading logic. This change provides a single, tooling works correctly with either layout.Checklist
changelog.ymlfile.guidelines.
Author's Checklist
dev/citools,dev/codeowners, anddev/packagenamesunit tests pass locally.b077ed3("Test a subset of the packages - to be removed") is dropped before merge.How to test this PR locally
For failures:
Related issues