Overview
Use this sequence when an agent performs poorly:
- Observe the failure precisely. Determine whether the agent misunderstood the goal, lacked access,
could not find the right code, encountered an unclear interface, missed a test, or made a weak judgment.
- Classify the cause. Separate model limitations from harness limitations. Do not assume that every
failure requires a stronger model.
- Improve the environment first when possible. Add a focused instruction, clarify the task
boundary, improve a test, simplify a module, or add a pointer to the correct location.
- Re-run the same class of task. Check whether the change reduced retries, confusion, and token
use.
- Evaluate model upgrades on operational criteria. Compare the quality gain with cost, latency, and
availability rather than adopting a new model only because it is new.
- Avoid model-specific overfitting. Prefer improvements that would help any competent agent work in
the repository.
One conservative adoption policy is to let a new model settle before making it the default. This creates time to see how its capability, price, latency, and availability behave in practice. That is an operating preference, not a requirement; the important point is to evaluate the whole system rather than the model in isolation.