Skip to content

Fan-Out — Parallelizing at Scale

For large migrations, analyses, or batch operations, distribute work across many parallel Claude invocations:

# Step 1: Generate task list
claude -p "list all Python files that need migrating to async. output one path per line" \
  > files_to_migrate.txt

# Step 2: Fan out across files
while IFS= read -r file; do
  claude -p "Migrate $file to async/await. Preserve all behavior. Return OK or FAIL." \
    --permission-mode auto \
    --allowedTools "Read,Edit,Bash(git commit *)" \
    --max-turns 15 &
done < files_to_migrate.txt
wait

Test on 2-3 files first. Refine the prompt on small batch failures, then run the full set. --allowedTools only skips the permission prompt for the tools it names — it does not bound which tools the session may call (the CLI reference says so outright: "To restrict which tools are available, use --tools instead"). Under --permission-mode auto a background classifier, not that list, decides what else may run, so this recipe alone does not keep the fan-out inside Read,Edit,Bash(git commit *). For a fan-out that must not stop for input and must stay close to an exact list, use --permission-mode dontAsk: it auto-denies anything that would otherwise prompt, running only actions matching your permissions.allow rules, read-only Bash commands, and calls a PreToolUse hook approves.

Scope-Verified Parallelism

When multiple sessions fan out and write to the same codebase, file collisions cause corrupt merges and lost work. Prevent this at the hook layer rather than through manual file-locking discipline:

// .claude/settings.json
{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Edit",
        "hooks": [
          {
            "type": "command",
            "command": "scripts/check-file-lock.sh"
          }
        ]
      }
    ]
  }
}

The hook command is a write-claim check: before an Edit lands it asks whether another live session already owns that file, and exits 2 (deny) if so. In Kanevry/session-orchestrator that claim is held by a JS lock-module family (scripts/lib/session-lock.mjs, scripts/lib/locks/state-md-lock.mjs) which grants ownership only after verifying a second identity factor — pid, host and start time — because a semantic session label is not globally unique. (That repo contains no check-file-lock.sh, and its STATE.md schema has no acting_on field; treat the shell name above as a placeholder for your own hook script.)

This enables safe parallel fan-out without coordination overhead: each agent checks at the point of write, not at the point of task assignment.

(session-orchestrator — Kanevry/session-orchestrator, Jun 2026.)

A complementary check earlier in the pipeline: rather than only catching a collision at the point of write, a classifier-gated pre-claim check can mark candidate parallel work green/yellow/red/blocked before dispatch, based on file overlap, shared dependencies, and shared infrastructure — reducing how often the write-time lock above actually has to deny anything. See Task-Shaped DAG Orchestration for the full pattern; the two checks are complementary layers, not substitutes for each other.

A remote lock for coordination that must survive across clones and worktrees, not just within one working copy. Kanevry's write-claim above protects agents sharing one working copy; a git-ref-based lock extends the same idea to agents in separate worktrees or clones by publishing the claim as a ref on the remote: refs/loop-lock/<feature>.<epoch>. Race resolution is deterministic without a coordinator: "after publishing, the acquirer re-lists the lock refs and keeps the lock only if its own ref sorts first — two racing acquirers resolve deterministically, the loser backs off." A lock older than a stale-hours threshold (2h by default) is treated as abandoned and reclaimable. It degrades gracefully when there is no shared remote to publish to — falling back to a local-only lock file rather than failing closed. (muzafferkadir/nextjs-loop-engineering-starter, Sep 2026.)

Multi-Loop Coordination

For repos running multiple distinct loops (CI Sweeper + PR Babysitter + Dependency Sweeper, etc.), fan-out coordination extends to cross-loop ownership:

  1. Each action loop writes acting_on: <branch-or-pr-id> in its STATE.md before starting any work on that item
  2. All loops read all STATE.md files before claiming a new item — skip items already claimed by any other loop
  3. Priority ordering determines who acts first when two loops find the same work
  4. Cross-loop conflicts that cannot be auto-resolved go to a human inbox in STATE.md

See Loop Patterns for the full priority ordering and eight named loop patterns, most with defined STATE.md coordination (Thin Loop keeps none).

Agentic MapReduce

For codebase-wide analysis tasks (security audits, dependency sweeps, dead-code detection), structure the fan-out as an explicit MapReduce rather than an unstructured swarm:

  1. Map — a coordinator agent scans the codebase and maps out signal locations (files, patterns, or regions likely to contain what you're looking for)
  2. Fan out — each mapped region is assigned to its own bounded agent invocation, scoped to only that region (same scope-verified parallelism as above)
  3. Reduce — a synthesis pass collects all per-region findings into a single deduplicated list
  4. Verify — each candidate finding is independently confirmed in an isolated sandbox before being reported, so the reduce step doesn't just aggregate unverified guesses

The map stage is what separates this from plain fan-out: instead of dividing work mechanically (one agent per file), a signal-aware pass decides where to look first, so bounded-context agents spend their budget on high-likelihood regions rather than scanning uniformly.

(Devin, "Security Swarm", Jul 2026.)

Pipe Claude into your existing pipelines

cat error.log | claude -p "identify the root cause" --output-format json | jq '.result'