Hard pipelines on agents — durable JSON state

For multi-step pipelines that must be deterministic despite using stochastic agents, persist explicit state in a JSON file (or a GitHub Issue's comments — see #127). Every step reads the JSON, does its work, writes its outputs back, advances the state. The skeleton is the JSON; the agents are workers.

Most "the agent forgot mid-pipeline" bugs disappear once state is external and addressable. The pattern lets you swap models per step (cheap model for filtering, expensive for synthesis), retry just the failed step, and audit any pipeline by reading the JSON. A real example: a blog pipeline producing ~3,000 visitors/month, running for months without re-deploys.

1. Decompose the pipeline into named stages with clear contracts: input fields, output fields, retry policy.
2. Define a JSON schema for the run state. Minimal:

{
"run_id": "...",
"stage": "research",
"current_step": 3,
"artifacts": {"research_notes": "..."},
"history": [{"stage": "...", "ts": "...", "status": "..."}]
}

3. Each agent step reads the JSON, performs its work, updates the JSON, commits or pushes a record.
4. Decide the substrate: a file in the repo (simplest), a GitHub Issue's comments (durable + queryable), or a real DB (Postgres on Railway / Supabase).
5. Add a top-level controller (cron / webhook / human trigger) that advances stages.
6. Each stage logs to a runs/ folder for replay & debugging.

• Don't bury state in a vector DB. State should be human-readable and human-editable.
• Don't conflate "state JSON" with "all data the pipeline produces." State holds pointers; bulky artifacts live in files.
• Two writers to the same JSON without locking will corrupt it. Either single-writer or use a real DB.