Bug: parallel runs via orch serve — successful runs marked as failed

[oxgeneral]

Summary

When orch serve runs multiple agents in parallel, tasks that complete successfully (subtype: success in logs) are incorrectly marked as failed. The run JSON also retains status: running instead of being updated to completed.

Steps to Reproduce

1. Create 3+ agents and assign them independent tasks (no dependencies)
2. Start orch serve --verbose
3. All agents start in parallel (confirmed by orch task list showing multiple in_progress)
4. Wait for agents to complete their work

Expected Behavior

• Run JSON: status: completed
• Task status: review or done
• Agent output captured in task result

Actual Behavior

• Run JSONL log shows: {"type":"done","data":"{\"type\":\"result\",\"subtype\":\"success\",...}"}
• Run JSON stays: status: running, exit_code: null
• Task status: failed (no failure_reason)
• Task result: empty

Evidence

Three parallel runs — all show success in logs but fail in state:

# JSONL (last line of each run) — all SUCCESS
run_CA9htig: type=done, data.subtype=success
run_caCyEJC: type=done, data.subtype=success  
run_BbF5_Nn: type=done, data.subtype=success

# Run JSON — all still RUNNING
run_CA9htig: status=running, exit_code=null
run_caCyEJC: status=running, exit_code=null
run_BbF5_Nn: status=running, exit_code=null

# Tasks — all FAILED
tsk_t56iAPO: status=failed, failure_reason=none
tsk_AomcObW: status=failed, failure_reason=none
tsk_AVsqW5_: status=failed, failure_reason=none

Hypothesis

The orch serve tick loop may have a race condition when processing multiple concurrent run completions. Possibly:

• The completion handler doesn't update run JSON when multiple runs finish in the same tick
• The stale status: running in run JSON causes the orchestrator to treat the run as timed out / crashed, marking the task as failed

Workaround

Run tasks sequentially with orch run <task-id> one at a time.

Impact

Critical — this breaks the core value proposition of parallel agent orchestration. Multi-agent teams cannot function via orch serve.

Environment

• orch: latest (npm @oxgeneral/orch)
• OS: macOS Darwin 25.2.0
• Node: v20.19.5
• Agents: 4 concurrent (claude adapter, claude-opus-4-6)

[m13v]

the race condition hypothesis sounds right. we hit almost the same bug running parallel Claude agents where completions arriving in the same event loop tick got dropped.

the pattern that fixed it for us: process completions through a queue instead of handling them inline in the tick loop. each completion writes to an append-only log, and a separate consumer processes them one at a time. this avoids the case where two completions arrive simultaneously and the second overwrites the first's state before it is persisted.

for the specific bug where run JSON stays status:running, check if the file write for the run JSON uses a read-modify-write pattern. if two completions both read the file, both see status:running, both write their update, the last write wins and the first completion's state change is lost. atomic writes (write to temp file, then rename) fix this.

also worth checking: is the JSONL log flushed synchronously? if the log shows success but the state update happens asynchronously, a crash between log-write and state-write would produce exactly the symptoms you describe.

[m13v]

our locking mechanism for parallel agent coordination that prevents the exact race condition you describe (atomic mkdir mutex + TTL-based lock with crash recovery): https://github.com/m13v/browser-lock/blob/main/playwright-lock.sh

[oxgeneral]

Fixed in v1.0.21 🎉

Root cause confirmed: race condition between reconcile() and collectEvents(). When a process exited cleanly, there was a window where reconcile detected the dead PID before handleRunSuccess acquired the mutex — falsely marking successful runs as "crashed".

What was done:

• Added activeCollectors Set — reconcile now skips PID-liveness and stall checks for tasks with an active event collector
• Fixed orphaned runs stuck in status: running when the running entry was already cleaned up by reconcile
• All completion handling remains serialized via the existing promise-chain mutex

Result: parallel orch serve runs now complete correctly — no more false failed status on successful agents.

15 new tests added, E2E verified with 3 parallel shell agents via orch serve --once.

Thanks for the excellent bug report — the evidence section with JSONL vs run JSON vs task status comparison was incredibly helpful for pinpointing the exact race window! Your hypothesis about the tick loop race condition was spot on. 🙏

Also thanks to @m13v for the comment about queue-based completion handling — we opted for the activeCollectors guard approach since it's a minimal change that eliminates the race without restructuring the tick loop, and the existing mutex already serializes all state mutations.

[m13v]

nice detective work on the race condition. the activeCollectors Set approach is clean, avoids adding heavy synchronization while still protecting the happy path. the JSONL flush fix is a good catch too, partial writes are hard to debug in production.