Herdr-Orchestrated AI Agent Workflow

This project implements an AI agent workflow that runs inside a Herdr workspace. Herdr supplies the terminal tab, panes, and agent sessions. An external shell controller coordinates implementation, review, validation, repair, and human approval. A separate Python runtime owns durable lifecycle state, checkpoints, leases, and artifact integrity.

Parallel review is one stage in that system. It is not the system’s central abstraction. The larger problem is how to let several agent sessions perform different kinds of work without using terminal activity as proof that the work is complete.

The implementation therefore separates three kinds of information:

• Execution answers which pane or agent is running now.
• Coordination answers which role should work next and which handoff the controller is waiting for.
• Durable state answers which stage has passed its gates and whether the run may continue.

This distinction is the basis of the workflow. A pane can be alive while its output is missing, stale, malformed, or from the wrong review cycle. The monitor may show that pane as working or idle, but only the runtime can accept a lifecycle transition.

Launcher and pane topology

The normal entrypoint is new-ticket-orchestrator.sh. It must run from a Herdr-managed pane. The launcher requires Herdr’s caller context and an explicit workspace identifier; it does not use whichever terminal happens to have focus.

The launcher creates one tab and five panes:

Ticket workflow tab
├── Orchestrator — receives the human task and controls the workflow
├── Agent 1 — implementation worker and finding repair
├── Agent 2 — structured reviewer
├── Agent 3 — independent reviewer with an additional architecture pass
└── Workflow Monitor — renders pane observations and durable status

The four agent panes start Claudex without forcing a model argument. The additional architecture review is a pass inside Agent 3, not a sixth pane. The monitor is a shell process rather than an AI agent.

Pane creation is ordered. The script creates the tab, splits the required panes, names them, waits until every shell is ready, starts the agent processes, and waits until each agent reports an idle state. Only then does it send the initial orchestration prompt.

If an initial prompt was pasted but not submitted, the launcher checks that the orchestrator is still idle, sends Enter to that existing pane, and waits for a state change. It does not create a replacement tab for this condition.

When a durable run ID is supplied, the launcher asks the runtime for status before creating any pane. The returned run ID, request ID, and ACTIVE status must match. A mismatch stops the launcher before the Herdr layout exists. The launcher passes the verified handoff to the orchestrator, but it never starts, resumes, approves, or cancels the durable run on the owner’s behalf.

An interactive flow adapter can collect the request, repository, specification, acceptance command, base reference, and task, create the run, and then invoke the same launcher. The adapter does not become another state machine; lifecycle ownership remains in the runtime.

Controller outside the agent sessions

The workflow controller is an external shell process. It is not hidden inside one model conversation. It knows the pane IDs, task context, comparison base, repository, specification, runtime run ID, and paths for controller handoffs.

The controller performs four kinds of coordination:

• It waits for the required agent state before sending work.
• It gives each role a bounded prompt and an exact output location.
• It checks completion markers and cycle identity before consuming a handoff.
• It calls the runtime only at stage boundaries where required evidence exists.

This boundary avoids giving an agent direct authority over canonical state. An agent produces work or a report. The controller validates the handoff shape and submits the relevant evidence. The runtime decides whether the state transition is legal.

The controller also keeps a local operational phase for conditions such as worker activity, reviewer activity, waiting for human input, and waiting for approval. These phase files help the monitor and restart logic, but they are not accepted as durable checkpoints.

If Agent 1 asks a question, the controller enters a local WAITING_FOR_HUMAN_INPUT phase and remains attached to the same worker. This does not convert the durable run to BLOCKED. The worker can continue after the answer without being replaced merely because its turn exceeded a reviewer-oriented timeout.

At the approval gate, the controller stays alive while waiting for an explicit owner decision. If the orchestrator cannot persist its human-facing approval report, the controller records a warning but keeps the durable approval state and continues waiting. Reporting failure is not allowed to detach the process that owns the wait.

Controller-runtime action boundary

The runtime uses one explicit CLI invocation for each action. It has no daemon and no event stream. Successful actions return a sanitized JSON status object; failures return a non-zero exit with a human-readable error.

The action set has two layers.

Owner and lifecycle actions include:

• start
• status
• pause
• resume
• retry
• request-changes
• approve
• cancel

Controller boundary actions include:

• stage-start
• acceptance
• checkpoint
• review
• resolve-review
• provider
• block

The distinction is practical. The controller can mark an implementation stage active, run the configured acceptance command, submit a checkpoint, record normalized review evidence, and submit validation and repair artifacts. It cannot bypass the lifecycle graph or silently approve the run.

Hooks follow the same rule. The SessionStart hook only reports an unfinished-run hint. It cannot start or resume work.

Canonical JSON state

Each run has one canonical workflow.json. The repository includes a Draft 2020-12 JSON Schema for this record. The runtime also validates required fields, types, enums, and bounds before every read and write.

The schema rejects unknown top-level properties and defines the lifecycle status enum. The record includes:

• run, request, owner, and machine identity;
• exact runtime release and workflow schema version;
• repository root, base commit, and current commit;
• lifecycle status, phase, active stage, and pause request;
• review cycle, blocking-finding count, and maximum cycles;
• acceptance command, specification reference, and stage order;
• last completed stage and checkpoint time;
• bounded retry policy;
• artifact manifest and current observations; and
• creation and update timestamps.

The current runtime pins active runs to runtime release 0.1.0 and workflow schema version 1. Resume does not select a “latest” version. A run created under one contract must be resumed under that exact pair.

State is local to the consumer repository and excluded from Git. Run directories use mode 0700; JSON files and lock files use mode 0600. State writes go to a temporary file, flush and fsync its content, apply private permissions, and replace the destination atomically.

The runtime allows only one active run for the same request identifier. Run creation is serialized by a request-scoped file lock, so two start commands cannot both pass an earlier read and create competing active runs.

Artifact contract and integrity

The canonical state contains an artifact manifest. Each reference has exactly four fields:

{
  "path": "artifacts/example.json",
  "sha256": "<64 lowercase hexadecimal characters>",
  "type": "acceptance",
  "producer": "runtime"
}

The published schema defines exactly those four fields. Runtime-generated references use that shape. The path must be a non-empty relative POSIX path inside the run directory. Absolute paths, backslashes, dot segments, missing files, malformed digests, empty types, and empty producers are rejected.

Runtime-generated evidence uses a JSON envelope with kind, recorded_at, and a typed payload. External controller files can also be registered, but the runtime hashes the exact bytes before adding the reference. Later integrity checks resolve the path inside the run directory and recompute SHA-256. A changed or missing file invalidates the manifest.

Artifact paths are immutable identifiers within a run. The current implementation rejects a second manifest entry for the same path. The controller also creates cycle-specific worker artifact names and refuses to overwrite an existing runtime artifact. A later review cycle therefore cannot replace the bytes that justified an earlier checkpoint while retaining the old manifest identity.

Not every JSON payload is governed by one large universal schema. The top-level workflow and artifact-reference shape follow the published schema. Individual evidence types are validated by their owning runtime action. Review, acceptance, checkpoint, lifecycle decision, provider failure, finding validation, and worker resolution each have different required semantics.

Normal status output does not return artifact payloads. It reports canonical state, checkpoint health, artifact health, lease state, block reason, and sanitized pane or agent observations. Secret-like values in observations and acceptance output are redacted before normal display or evidence persistence.

Workflow from task to approval

The full loop is:

Create a durable run
    -> launch the Herdr topology
    -> start implementation
    -> run configured acceptance
    -> record an implementation checkpoint
    -> run two review lanes
    -> normalize and record findings
    -> validate blocking findings without editing
    -> repair only validated findings
    -> run the next bounded review cycle
    -> wait for explicit human approval or requested changes

Implementation stage

The controller calls stage-start for implementation before prompting Agent 1. The runtime requires an ACTIVE run, no other active stage, no pending pause request, and the expected next stage from the configured stage order.

Agent 1 receives the task, base reference, repository, and specification path. Its work log is not a checkpoint by itself. The controller must first cross the acceptance and artifact gates.

Acceptance gate

The acceptance command is required in configuration. The runtime does not choose a test command from the repository. It also refuses a command that differs from the configured value.

The Python runtime parses the configured command into an argument vector and executes it from the primary repository without a shell. It captures the exit code and bounded stdout and stderr, records the Python runtime version, and redacts common secret assignments.

A successful result updates last_acceptance_at and writes an acceptance artifact. A timeout, process-start failure, or non-zero exit writes a validation-failure artifact and moves the run to BLOCKED.

Checkpoint gate

Only complete_stage can create a checkpoint. Direct checkpoint mutation is rejected.

The gate checks that:

• acceptance and validation were reported as successful;
• the run is not terminal;
• the stage exists in the configured order and is the expected next stage;
• successful acceptance evidence exists;
• acceptance ran after the previous checkpoint;
• at least one required stage artifact exists; and
• every artifact digest remains valid.

After those checks, the runtime records the repository’s current commit, clears the active stage, and stores the checkpoint time. A checkpoint is therefore a relationship between repository state, a configured acceptance result, and immutable evidence. It is not a timestamp copied from a pane.

Parallel review as one stage

After the implementation checkpoint, the controller starts two review lanes against the same task context and code generation.

The structured lane owns task and specification alignment, coding standards, deterministic local correctness, direct validation, error handling, schema or protocol behavior, and behavior tests.

The independent lane owns concurrency, lifecycle transitions, state, race conditions, leases, checkpoints, resume behavior, provider failure, artifact integrity, controller-runtime integration, and recovery. Its additional architecture pass covers maintainability, cross-cutting risk, and edge cases.

The controller waits for complete output from both lanes. It rejects mixed review generations and does not allow the worker to consume one current report together with one stale report. Reviewer session rotation may continue independently, but the worker starts only after the paired results needed for its handoff are complete.

Reviewer output uses machine-readable finding lines or a JSON document. Each finding has an exact severity from Critical, Important, Minor, or Nit. The report may declare BLOCKING_FINDINGS, but the runtime derives the count from the individual findings and rejects a mismatch. Critical and Important block; Minor and Nit remain recorded without forcing repair.

The controller’s reviewer protocol also requires stable finding IDs across cycles. The runtime itself validates the findings list, severity values, derived blocking count, and reviewer-handoff result. This distinction matters: prompt-level report quality and runtime-level safety checks are related, but they are not the same enforcement mechanism.

Finding validation and repair loop

A reviewer finding does not go directly to Agent 1 as an instruction to edit code. The controller starts a separate read-only validation step. The validator reads both complete review reports, tests each blocking claim against the current source, and writes a finding-validation handoff.

Only after that handoff is complete does Agent 1 receive the validated set. The worker writes a separate resolution artifact that describes the changes and test evidence. The runtime’s resolve-review action requires both validation and worker-resolution files, a positive validation result, intact artifact hashes, and remaining review-cycle capacity.

On success, the runtime records both files, increments the cycle, clears the pending blocking count, and returns to ACTIVE. If evidence is missing, malformed at the controller boundary, or no longer intact, the run becomes BLOCKED.

This creates a three-part chain:

review claim -> independent validation -> targeted worker resolution

The separation prevents the worker from deciding that a report about its own work is valid simply by fixing it. It also preserves invalidated or unresolved review evidence instead of replacing the earlier report with a cleaner narrative.

When the maximum cycle is reached with blocking findings, the runtime blocks the run. It does not open an untracked cycle or create a replacement run.

Human decision and lifecycle

The durable lifecycle is:

ACTIVE
├── PAUSED
├── BLOCKED
├── WAITING_FOR_HUMAN_APPROVAL
├── FAILED
├── CANCELLED
└── COMPLETED

COMPLETED, FAILED, and CANCELLED are terminal and immutable.

The approval gate requires review evidence, zero pending blocking findings, successful acceptance, and intact artifacts. Reaching that gate changes the status to WAITING_FOR_HUMAN_APPROVAL; it does not complete the run. approve is valid only from that state.

The owner can instead request changes. That transition keeps the same run, increments the review cycle, clears the prior implementation checkpoint and acceptance time, and reopens the first configured stage. This makes additional work part of the recorded history rather than a new session with no connection to the earlier decision.

Cancellation requires a reason. Ordinary questions do not change lifecycle state. The launcher, controller, monitor, and plugin hook cannot infer approval from silence, pane closure, or clean terminal output.

Pause, lease, and cooperative stop

The runtime lease identifies one local mutation owner with a token, process ID, host, machine identifier, heartbeat, and expiry. A file lock serializes lease updates. A second live controller cannot acquire the same run.

Pause is aligned with checkpoint safety. If no stage is active and a completed checkpoint exists, the run can enter PAUSED immediately. If a stage is active, the runtime records a pause request and changes the phase to PAUSE_REQUESTED while the durable status remains ACTIVE.

The controller must not start another stage. It finishes or safely stops the current work at the next verified checkpoint. That checkpoint performs the transition to PAUSED and releases the lease. The design does not label a mid-stage process stop as a safe checkpoint.

An expired lease is not silently stolen. Reclaim is explicit and permitted only when the lease is stale. Retry and resume remain owner actions.

Resume and recovery

Resume is a verification operation, not a process restart. Before returning a paused or blocked run to active work, the runtime checks:

• owner identity;
• the same machine boundary;
• exact runtime release and workflow schema version;
• repository root;
• availability of the recorded base commit;
• ancestry of the current commit from that base;
• current repository state against the checkpoint; and
• every artifact reference and SHA-256 digest.

A live lease blocks resume without mutating the run. A stale lease can be reclaimed only through the defined path. Repository drift, a missing base, an owner mismatch, a release mismatch, or a damaged artifact blocks continuation instead of selecting a convenient stage.

Pane state is reconciled after durable verification. Missing panes are replaceable observations, not missing proof. A pane may be recreated and linked to the verified run, but its previous activity cannot manufacture a checkpoint.

Older controller phase, PID, marker, pane, and log files can be imported only as blocked evidence. The legacy importer does not infer completed stages from them or rewrite those files.

The agent-session rotation path uses another explicit handoff protocol. It waits for an idle session, writes a nonce-bound handoff, validates that exact handoff, clears the old context, and requires the next session to acknowledge it. Protocol runs are excluded from rotation so the handoff cannot recursively rotate itself.

Provider calls use bounded retries with recorded attempts and exponential backoff. Exhausting the configured attempts writes a provider-failure artifact and blocks the run.

Monitor semantics

The Workflow Monitor reads two sources: Herdr pane observations and the durable workflow record. Pane status can be working, idle, waiting, blocked, or unknown. The monitor also shows workflow duration, human-input waits, the approval gate, and durable block reasons.

The monitor does not write checkpoints or decide transitions. If the controller disappears while the runtime remains active, the monitor keeps that mismatch visible. If the runtime is blocked, it shows the durable reason even when every agent pane appears idle.

This is why the monitor can be useful without becoming authoritative. It combines operational state for a human, while the runtime remains the only source for lifecycle and artifact integrity.

Evidence exercised in the current source

The runtime test suite covers the state machine and its integration boundaries. In the inspected worktree, 47 Python tests pass. They cover:

• required configuration and installer behavior;
• one active run per request;
• state and artifact validation, digest changes, and duplicate artifact paths;
• stage order, acceptance freshness, checkpoint authorization, and repository commit recording;
• lifecycle transitions, cooperative pause, request changes, approval, and cancellation;
• review parsing, severity gates, blocking-count derivation, maximum cycles, validation, and worker resolution;
• lease exclusion, stale recovery, owner checks, repository drift, and unavailable base commits;
• provider retries and failure artifacts;
• launcher-to-runtime handoff and rejection before pane creation;
• plugin actions, read-only session hooks, local-state Git exclusion, and legacy import boundaries; and
• interactive start, listing, details, cancellation, and stored UI defaults.

The shell and protocol tests also pass for the current controller and launcher. They exercise pane readiness, stalled-prompt recovery, runtime checkpoint calls, normalized paired reviews, malformed evidence, mixed-generation rejection, concurrent reviewer rotation, maximum-cycle behavior, owner fencing, scoped recovery handoffs, child-process cleanup, long-running worker waits, controller attachment at the approval gate, monitor termination, and monitor display of durable block reasons.

Separate session-cycle tests cover the nonce handoff, context clear, bootstrap acknowledgement, and the rule that an orchestrator handoff must match the controller request.

These tests are evidence for the implemented contracts and simulated process boundaries. They do not measure model quality, review accuracy, throughput, latency, or productivity.

Current limits

The runtime currently follows one primary Git repository and one explicit specification path. Other repositories may exist in the surrounding workspace, but their commits are not automatically incorporated into one run record.

The published JSON Schema sets additionalProperties: false, but the handwritten runtime validator focuses on required fields, value types, enums, and bounds. It does not currently contain a general check that rejects every unknown top-level or artifact-reference property. Generated records follow the schema, but full schema equivalence remains a validation gap for externally modified state.

The workflow is local and same-machine. It does not provide cross-machine resume, cross-member handoff, a shared remote state store, export/import of trusted active state, a daemon, or event replay.

The acceptance record is still blocked on a security decision for local state that can contain sensitive evidence. Real multi-pane stress validation is also open for provider failure, pane replacement, and machine restart behavior. Runtime 0.1.0 is therefore a technical candidate, not an accepted production release.

The monitor observes Herdr and local files; it is not a distributed scheduler. The workflow also depends on explicit project configuration, including a correct acceptance command. A successful configured command is evidence that command passed, not proof that the implementation is correct in every dimension.

The result is a concrete workflow contract: Herdr hosts the agents, the external controller coordinates their stages, strict JSON state and hashed artifacts preserve what crossed each boundary, and the owner remains responsible for pause, recovery, requested changes, approval, and cancellation.