[feat/update-documentation] Add latest doc generator changes by mjwolf · Pull Request #3271 · elastic/elastic-package

mjwolf · 2026-02-11T18:24:34Z

This PR adds a new elastic-package update documentation command that uses LLM (Large Language Model) agents to automatically generate and update integration package documentation.

Key Features

Multi-agent workflow for documentation generation with critic review and iterative refinement
Section-by-section generation for better control and consistency
Google ADK integration with Gemini models for LLM-powered content generation
Phoenix tracing support for debugging and monitoring LLM interactions
Example-based learning using category-specific documentation examples (PostgreSQL, SentinelOne, Fortinet, etc.)
Interactive TUI for reviewing and approving documentation changes

Components Added

internal/llmagent/docagent/ - Core documentation agent with prompts, executor, and file operations
internal/llmagent/tools/ - Package tools and example retrieval for context-aware generation
internal/llmagent/tracing/ - OpenTelemetry tracing with Phoenix integration
cmd/update_documentation.go - CLI command implementation

Changes

Updated go.mod with new dependencies for Google ADK, Gemini, and tracing
Added comprehensive unit tests for all new components
Updated package README template structure

Now cost is working and an entire execution is contained in a session. Project name, root span and showing all traces in the span are not working.

* Use the multi-agentic workflow by default, and take feedback from validators to generator. * Update tracing attributes and hierarchy * Move tracing config into elastic-package config * Add a tool which can fetch telemetry from Phoenix, and save it as single json file: `scripts/fetch_phoenix_traces.py`

- Implement parallel processing for section generation with sectionResult type - Reduce tool calls by embedding more information into initial prompts - Refactor workflow for better caching and input handling - Update critic and validator agents to accept content directly from user messages - Enhance output format specifications for validation and criticism results 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

…oc-agents

- Add --debug-critic-only, --debug-validator-only, --debug-generator-only CLI flags - Add parallel/sequential section generation config option (ELASTIC_PACKAGE_LLM_PARALLEL_SECTIONS) - Refactor section generation into reusable generateSingleSection method - Add section context (title, level) and build info to tracing spans - Add RunCriticOnContent/RunValidatorOnContent for individual agent execution - Remove unused fetch_phoenix_traces.py script 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

…ing relevant versioning from original readme

…cate sections

…oc-agents

changes focus on improving documentation reliability through three key areas: Troubleshooting completeness - The generator and validators now enforce input-specific troubleshooting tables (Symptom | Cause | Solution format) for each input type the integration uses (httpjson, tcp, udp, filestream, etc.). This was previously missing, resulting in thin troubleshooting sections. Severity upgraded from Minor → Major. Validation section consistency - Added standardized verification steps using data_stream.dataset:<integration>* pattern to check ALL datastreams, instead of hardcoding specific ones like citrix_adc.log. This makes validation guidance work across any integration. Structure guardrails - Added explicit generator instructions preventing common LLM mistakes: multiple H1 headings, heading level skips (H1→H4), and uncontextualized placeholder values (example.com without "replace with your value"). These changes address the top failure patterns observed in the semantic analysis: inconsistent troubleshooting depth, unreliable validation steps, and structural issues that triggered false validation failures.

Doc agents

It looks the section ranking and picking the best one works nicely I also update the validators to only allow actionable suggestions I fixed a bunch of validator false positives And I did some deduping of document links Also fixed phoenix tracing for iteration spans and grouping off all traces

- removed all knowledge of the test harness implementation that was used earlier to test llm generation - fixed phoenix tracing logging and fetching results into the output folder - fix some minor doc prompting and validation

Merge branch 'doc-agent-qcorp-sections' bringing in: - Per-section validation loops with best-iteration tracking - Convergence detection for bonus iterations - Full document generation workflow (GenerateFullDocumentWithWorkflow) - Generation result types (GenerationConfig, GenerationResult, SectionGenerationResult) - Staged validators moved to specialists/validators package - SectionContext moved from specialists to validators package Preserved from doc-agents: - Debug flags (--debug-critic-only, --debug-validator-only, --debug-generator-only) - generateSingleSection and generateSectionsSequential helper functions Documentation updated to reflect current implementation. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

…ccheck - Rename docagent/specialists to docagent/agents - Move section parser tests to docagent/parsing - Inline section_generator into docagent, simplify docagent and postprocessor - Shorten README and tools/readme template; update internal llmagent README - Fix SA4006: remove unused currentSections assignments in runInteractiveSectionReview 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

- Merge UpdateDocumentation and UpdateDocumentationWithConfig into single UpdateDocumentation (default config only); remove second function - Tracing: remove env vars (LLM_TRACING_*), ConfigFromEnv, and Init; use profile config (llm.tracing.*) only in getTracingConfig - Docs: update READMEs for config-only tracing and Workflow Config comments 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Programatically verify, and if needed fix, the agentless deployment section in the postprocessing step.

…stic#3294) Bumps [github.com/elastic/go-ucfg](https://github.com/elastic/go-ucfg) from 0.8.8 to 0.9.0. - [Release notes](https://github.com/elastic/go-ucfg/releases) - [Changelog](https://github.com/elastic/go-ucfg/blob/main/CHANGELOG.md) - [Commits](elastic/go-ucfg@v0.8.8...v0.9.0) --- updated-dependencies: - dependency-name: github.com/elastic/go-ucfg dependency-version: 0.9.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

jsoriano

Took a quick look. This is becoming too big to review at once. Would it make sense to split in different PRs?

jsoriano · 2026-02-19T10:33:09Z

-   - Use --doc-file to specify which markdown file to update (defaults to README.md)
-   - In interactive mode, you'll be prompted to select from available files
-   - Supports packages with multiple documentation files (e.g., README.md, vpc.md, etc.)
+   - For packages with multiple documentation files, the user can specify which file to update in interactive mode, or use the --doc-file flag to specify the file to update in non-interactive mode.


Nit. No need to use a list here now.

I've changed this to a single sentence

jsoriano · 2026-02-19T11:35:51Z

+			return fmt.Errorf("documentation update failed: %w", err)
+		}
 	}
+	cmd.Println("Done")


It would be better to leave this message to the user in the main command method.

Removed this from here

jsoriano · 2026-02-19T11:37:44Z

-			useModifyMode = true
-		}
+	if thinkingBudget != nil {
+		cmd.Printf("Using Gemini provider with model: %s (thinking budget: %d)\n", modelID, *thinkingBudget)


Only Gemini has thinking budget? Otherwise this message should be more generic.

I change Gemini to LLM

jsoriano · 2026-02-19T11:38:54Z

 	} else {
-		cmd.Println("Running in non-interactive mode - proceeding automatically.")
-		useModifyMode = modifyPrompt != ""
+		cmd.Printf("Using Gemini provider with model: %s\n", modelID)


Ideally it will be possible to use other providers apart from Geminy, right? Should this message be more generic?

Right now, Gemini is the only supported LLM. It should be possible to support others in the future with smaller changes to add them as a provider. So I've changed "Gemini" to "LLM" wherever appropriate, to make it more ready to support others in the future.

jsoriano · 2026-02-19T12:24:14Z

-Combine with --modify-prompt "instructions" for targeted non-interactive changes.
+  # Batch evaluation of multiple packages
+  elastic-package update documentation --evaluate \
+    --evaluate-batch citrix_adc,nginx,apache \


We recently added a foreach command (#3027), that is intended to be used to run commands on several packages. Maybe we don't need command-specific batch options.

Yeah, that'll be better to use. So I took out this batch command, and for-each can be used instead.

jsoriano · 2026-02-19T13:01:27Z

+const structureValidatorInstruction = `You are a documentation structure validator for Elastic integration packages.
+Your task is to validate that the README follows the expected structure and format per the official template.
+
+## Expected Section Structure (from package-docs-readme.md.tmpl)


We should have a way to keep this file synced with the recommendations here.

I've changed this to generate the structure from the readme.tmpl file, so it will stay in sync in the future

jsoriano · 2026-02-19T13:07:31Z

+// - VendorSetup: vendor setup instructions accuracy against docs and LLM knowledge
+// - AdvancedSettings: advanced settings gotchas from manifest.yml are documented
+// - Scaling: input-specific scaling and fault tolerance guidance


The last three validators are not listed bellow, are they going to be added later?

No, these were combined with other validators or prompts, so they're not needed, and I don't plan to add them back.

jsoriano · 2026-02-19T13:11:22Z

+// Returns the content with the correct title.
+func EnsureDocumentTitle(content, packageTitle string) string {
+	expectedTitle := fmt.Sprintf("# %s Integration for Elastic", packageTitle)
+	aiNotice := "> **Note**: This AI-assisted guide was validated by our engineers. You may need to adjust the steps to match your environment."


Is this notice going to be added to all integration readmes?

Not necessarily all readmes. This should be in all AI-assisted readmes, but we still have non-AI-assisted documentation, and could add more in the future, so it doesn't need to be in everything.

jsoriano · 2026-02-19T13:16:03Z

+
+These are the foundational principles of Elastic documentation. Internalize them.
+
+### Principle 1: Voice and tone


Some of these principles are redundant with the ones found in the prompts of the validators. Is this intentional?

We want the validators to verify that the generator prompts are being followed correctly, so the prompts are pretty similar in both places. They're different enough though that I think it would be messier to try to reduce the duplication while still giving the correct prompts to each.

jsoriano · 2026-02-19T13:23:13Z

+}
+
+// NewPhoenixClient creates a new Phoenix client
+func NewPhoenixClient(baseURL string) *PhoenixClient {


Does it need an additional API key?

teresaromero · 2026-02-19T13:54:13Z

i agree on separating the PR into single smaller pieces. recently the vale framework has been introduced in some repos to review the style on docs. instead of having all the rules locall, should the llm be able to use the rules used by elastic through this framework? https://github.com/elastic/vale-rules

vinit-chauhan · 2026-02-19T19:24:21Z

Should we add .cursor to gitignore?

I've added this to the .gitignore file

* chore: [updatecli] Update latest snapshot to 8.19.12-f02dfb3e Made with ❤️️ by updatecli * chore: [updatecli] Update latest snapshot to 8.19.12-d9a5f69d Made with ❤️️ by updatecli --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

…lastic#3299) As skip configuration is set with tags config, those settings were ignored when reading the policy test configuration. This caused that policy tests could not be skipped. This PR updates the method to read the configuration to use go-ucfg and go-ucfg/yaml libs. This is the same approach done for pipeline and system tests.

- Add LLM_PROVIDER / llm.provider (default: gemini); keep GOOGLE_API_KEY as default - Add model factory in executor (gemini only for now); executor stores provider, exposes Provider() - Add provider param to tracing StartSessionSpan, StartAgentSpan, StartLLMSpan - Thread provider through docagent, workflow config, and doceval batch - Only set ThinkingConfig for Gemini in executor - Update config.yml.example, readme.md.tmpl for provider and provider-specific keys Co-authored-by: Cursor <cursoragent@cursor.com>

- Drop --evaluate-batch, --evaluate-integrations-path, --evaluate-parallel flags - Remove runBatchEvaluation, printBatchEvaluationSummary, and doceval/batch.go - Document using elastic-package for-each to run evaluation per package - Add 'update' to foreach allowed subcommands for update documentation --evaluate Co-authored-by: Cursor <cursoragent@cursor.com>

- Remove validators.DataStreamManifest; use packages.DataStreamManifest from internal/packages - loadDataStreams now uses packages.ReadDataStreamManifestFromPackageRoot and builds DataStreamInfo from result Co-authored-by: Cursor <cursoragent@cursor.com>

Rename files in cmd, internal/llmagent to use no underscores except meaningful suffixes (_test.go). Update internal/llmagent/README.md to reference new filenames. Co-authored-by: Cursor <cursoragent@cursor.com>

Make package-docs-readme.md.tmpl the single source of truth for expected section structure: - Add convertParsedToRequired() to build RequiredSection slice from parsing.ParseSections(template); exclude optional subsections (Agentless deployment, API usage, Vendor resources) and normalize templated 'Set up steps in {[.Manifest.Title]}' to 'Set up steps in'. - Initialize RequiredSections at package init from the template via initRequiredSections(), with fallback if parse is empty. - Build structureValidatorInstruction at init using formatExpectedStructureBlock() so the LLM 'Expected Section Structure' block stays in sync with the template. - Add TestRequiredSectionsDerivedFromTemplate to assert section count and deployment subsection match the template and instruction references it. Co-authored-by: Cursor <cursoragent@cursor.com>

elasticmachine · 2026-02-24T01:10:30Z

💔 Build Failed

Buildkite Build
Commit: 46c9cae

Failed CI Steps

:go: Run check-static

History

💚 Build #7227 succeeded 741ec46
💔 Build #7226 failed bd819b3
💔 Build #7223 failed acba65a
💔 Build #7222 failed 17017f6
💔 Build #7194 failed b3ceffc

cc @mjwolf

mjwolf · 2026-02-24T01:27:05Z

@jsoriano @teresaromero Thanks for looking at this!

I will try to break this up into a few smaller PRs to make it a bit easier to review. Unfortunately, with how the text generator and validators feed back into each other to make improvements, it will probably still be pretty big in the first PR. I can take out some of the other features not directly related to generation, to make things smaller. So I'll close this PR, and open new ones soon.

@jsoriano,I've addressed most of your comments in this branch. There's a few I'm not sure we should do right now. For integrating with the package-spec validators, I think that would be a larger change that needs some coordination with package-spec work, so I think that could be done in another task.

For integrating with Vale, I only learned about vale usage recently, so I'm not sure how to integrate them. It might need a lot of changes. It's probably a good idea, but probably also could be done separately.

mjwolf · 2026-02-24T02:17:31Z

Closing in favour of #3302. This has only the core generation functionality. Unfortunately, it couldn't be any smaller without breaking the functionality or quality of the generated documentation.

mjwolf and others added 30 commits December 18, 2025 09:30

Update tracing attributes

3662bfe

Now cost is working and an entire execution is contained in a session. Project name, root span and showing all traces in the span are not working.

Update set span status in tracing

4ec3a5b

Disallow Transfers in critic and validator

2b3c388

update validator with more text

e6b3b01

Merge branch 'doc-agents' of github.com:mjwolf/elastic-package into d…

3ac5007

…oc-agents

make i going in the right direction?

fbbecda

some revisions to the overlapping logic

f296f7f

final

0b902e1

advanced settings, advanced settings validator, fixed bug about remov…

048529e

…ing relevant versioning from original readme

remove parallel section generation, add structure validator for dupli…

e210dd9

…cate sections

scaling validator, llm generator notes

2ae4511

add scaling generator and validation, added convergence bonus iteration

987308e

make vendor setup better again!

cea2469

consolidated docagent and harness into a new context builder

bd39850

added phoenix tracing into the output

ec3009f

batch parallel processing, validator work for llm validation

d8f49ae

default 4 parallel versus 1

ef9ee0c

Merge branch 'doc-agents' of github.com:mjwolf/elastic-package into d…

e935adb

…oc-agents

make sure the traces get updated

1b4144e

Merge pull request #6 from jrmolin/doc-agents

975904c

Doc agents

clean things up just a bit

43b8eb0

add more consistency across runs

8d103ed

phoenix and test harness

3f77ae6

- removed all knowledge of the test harness implementation that was used earlier to test llm generation - fixed phoenix tracing logging and fetching results into the output folder - fix some minor doc prompting and validation

Add back section-based generation

8eea7b3

mjwolf and others added 8 commits February 13, 2026 14:40

Refactor update documentation cmd

b50d248

Remove dead code and unneeded tests

41e85a7

refactor evaluation specific files into a doceval package

6f490c2

Remove duplicate disclaimers from generated docs

2c8e68e

Add programatic validation of agentless deployment text

17017f6

Programatically verify, and if needed fix, the agentless deployment section in the postprocessing step.

Merge remote-tracking branch 'upstream/main'

acba65a

mjwolf requested a review from a team as a code owner February 18, 2026 19:26

mjwolf removed the request for review from a team February 18, 2026 19:26

mjwolf and others added 4 commits February 18, 2026 11:38

Merge remote-tracking branch 'upstream/feat/update-documentation'

2ac84ce

Remove unneeded disclaimer validation

bd819b3

remove external prompt override

741ec46

jsoriano reviewed Feb 19, 2026

View reviewed changes

vinit-chauhan reviewed Feb 19, 2026

View reviewed changes

github-actions bot and others added 9 commits February 23, 2026 09:00

Merge remote-tracking branch 'upstream/main'

29ec790

Rename Go files to avoid underscores (Go naming convention)

c2a2edb

Rename files in cmd, internal/llmagent to use no underscores except meaningful suffixes (_test.go). Update internal/llmagent/README.md to reference new filenames. Co-authored-by: Cursor <cursoragent@cursor.com>

Change empty content placeholder

d67698a

mjwolf closed this Feb 24, 2026


		These are the foundational principles of Elastic documentation. Internalize them.

		### Principle 1: Voice and tone

Conversation

mjwolf commented Feb 11, 2026

Key Features

Components Added

Changes

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

teresaromero commented Feb 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mjwolf Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

mjwolf commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mjwolf commented Feb 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

mjwolf Feb 23, 2026 •

edited

Loading

elasticmachine commented Feb 24, 2026 •

edited

Loading

mjwolf commented Feb 24, 2026 •

edited

Loading