fix: split token-aware into default exclusions + optional --economic mode

[omerarslan0]

Summary

Splits the token-aware codebase analysis changes into two tiers based on maintainer feedback:

Default mode (always active):

• Binary application detection (.app, .exe, .AppImage, .msi, .deb, .rpm, .flatpak) with abort message
• Binary file exclusion (.o, .so, .dylib, .a, .exe, .dll, .wasm, .pyc, .class, .jar)
• Build artifact exclusion (build/, dist/, __pycache__/, node_modules/, .git/, target/, cmake-build*/)
• Generated code exclusion (*_generated.*, *.pb.go, *.pb.cc)
• Media/assets exclusion (images, videos, fonts, archives)
• Vendored/third-party code exclusion (vendor/, third_party/)
• "Scan before reading" principle

--economic mode (opt-in):

• ~50,000 token read budget (~200KB)
• Skip files > 100KB, read first 200 lines for important large files
• Priority reading order (README → CLI entry points → API headers → core modules)
• "Summarize don't dump" and "incremental deepening" strategies

For users on smaller context windows or who want faster/cheaper generation runs.

Changes

• HARNESS.md: Split Phase 1 into default exclusions + new --economic section
• cli-anything.md: Remove token budget numbers from default Phase 1, add --economic reference
• refine.md: Remove token budget numbers from Step 2, add --economic reference

Test plan

• Verify HARNESS.md Phase 1 default rules contain only exclusion/detection logic
• Verify --economic section contains all token budget constraints
• Verify cli-anything.md and refine.md reference --economic flag
• Confirm no token budget numbers remain in default mode sections

Closes #105

[yuh-yang]

Looks nice! Can we make token-aware a new mode? Also, the added content about binary contents made me think if we can have a hacker mode for reverse engineering binary softwares? We can tag that as experimental if we plan to implement that.

[omerarslan0]

I'm working on it.

[omerarslan0]

Thanks for the feedback!

Token-aware as a mode: I'd lean towards keeping token budget as the default behavior rather than a separate mode — the whole point of this fix (#105) is that without it, large codebases overflow the context window silently. If we make it opt-in, agents will hit the same issue unless they know to enable it.

That said, I can add a --no-token-limit flag (or a token_budget: unlimited config option) so users with small codebases can opt out. This way the safe behavior is default but it's not a hard constraint. Would that work?

Hacker mode for binary RE: Love the idea — decompiler output analysis (Ghidra/IDA/radare2) could open up closed-source software for harness generation. I think this deserves its own PR though, since the implementation is substantially different from source code analysis. I'll open an issue tagged experimental to track it. For now, this PR just gives a clear error message so users know to provide source code instead.

[yuh-yang]

For Token-aware as a mode:

Thanks! I see your point. I do agree that large codebases overflow the context window silently, but to me it's more like an economical/model capability issue rather than the design of CLI-Anything. Generally when I'm thinking of this, I break things into several points:

• We want to build CLI-Anything as a general harness rather than a specific workflow. That being said, we still introduced the 7 phases as a guideline for the agents. But I explicitly tried to avoid very certain and specific guidelines that would be deterministic to the agents' behaviors.
• The new contents added towards token-budget-aware mode introduced some strong hypothesis and rules that would be as constrains to the agents' behaviors. For example, >100KiB files are skipped, and file content read should be under 50K tokens (which is a bit too less for modern models with 1M context window). I generally suggest deleting this part, or move this to the economic mode as a separate mode (reasons for this mode illustrated below).
• I really agree with the design that binary files and build caches should be skipped -- they should not be part of the harness, nor would they contribute anything to the CLI building process. I generally suggest keeping this in whatever modes.
• I would still agree on moving this token-aware thing to a separate economic mode, where we can have more constrains. But in order we can keep up with the model training updates and make sure CLI-Anything can make full use of the long-term growing model capabilities, I'd vote for a more general harness design, where we could even try to simplify the 7-phase design when a stronger coding agent needs more space.

Let me know if these make sense and how you think!

[omerarslan0]

That makes sense — I think you're right that the token budget rules are too prescriptive for a general-purpose harness. The 50K limit and 100KB file cap made sense as a safety net, but they'd age badly as context windows grow.

Here's what I'll do:

Keep in default mode (all modes):

• Binary application detection (abort with clear message)
• Binary file exclusion (.o, .so, .pyc, .dll, etc.)
• Build artifact exclusion (build/, dist/, pycache/, node_modules/, .git/, etc.)
• Media/generated/vendored code exclusion
• "Scan before reading" principle (list tree first, then prioritize)

Move to a separate --economic mode:

• 50K token budget cap
• 100KB individual file size limit
• "Summarize don't dump" strategy
• Strict priority ordering

This way the default mode still skips files that are genuinely useless (binaries, build cache, media) but doesn't artificially cap how much source code the agent can read. The economic mode is there for users on smaller context windows or who want faster/cheaper runs.

I'll update the PR with this split. Does that align with what you had in mind? @yuh-yang

[yuh-yang]

That'd make a lot of sense to me! Thanks for the discussion.

[omerarslan0]

Updated.

Default mode: binary detection, binary/build/media/vendor exclusion, scan-before-read principle.

--economic mode: 50K token budget, 100KB file limit, priority reading order, summarize-don't-dump, incremental deepening.

Ready for review.

[omerarslan0]

@yuh-yang check this.

[yuh-yang]

hmmm I only see the criterion for token-aware mode? And they're directly overwriting on the current file.

[omerarslan0]

Updated — split is done now. Default Phase 1 only has binary detection and exclusion rules (binaries, build artifacts, media, vendored, generated). Token budget stuff (50K cap, 100KB file limit, priority ordering, summarize-don't-dump) moved to a separate --economic section after Phase 7. cli-anything.md and refine.md cleaned up too, just reference --economic flag instead of hardcoding numbers.