I Don't Ship Agent Writes Without Shadow Mode

I watched a production agent pass schema validation, clear the budget check, and still corrupt live data. It called a CRM update tool with well-formed arguments. The tool contract said yes. The agent wrote to the wrong customer record and kept going. Nobody caught it until the customer complained.

The team had schemas, allowlists, and token budgets. What they didn't have was a dry-run path. The agent's first attempt hit production. When it picked the wrong row, there was no diff review, no promotion gate, no second look. The write went through because the arguments were valid, not because the side effect was safe.

Validation proves a tool call is well-formed. It doesn't prove the mutation is correct. That's what shadow mode is for: run the write against a parallel executor that shows you what would change, compare the diff, and promote to live only if it passes. Reads hit production. Writes hit shadow first.

Why schema validation isn't enough

Tool contracts are essential. I wrote about them in my last post. They enforce argument schemas, check allowlists, track budgets, and validate return types. But a contract that says "this call is allowed" doesn't know if updating customer ID 7392 instead of 7329 is what the user actually wanted.

I've debugged agents that:

• Updated the correct table with the wrong row because the LLM confused two similar IDs in context
• Sent an email to the right address with a message that referenced the wrong account
• Wrote a file with valid JSON that contained data from a different conversation
• Deleted a test record that happened to have the same name as a production record

Every one of those calls passed schema validation. The arguments were well-typed. The tool was in the allowlist. The budget was fine. The failure was in the side effect itself, and the agent didn't see the problem because it never saw a diff of what was about to change.

You can't encode "is this the right customer?" in a JSON schema. You need to show the agent—or a human reviewer—what the write will do before it touches live state.

What shadow mode is

Shadow mode is a parallel executor for writes. It runs the same tool interface, talks to the same data layer, but instead of mutating production state, it returns what would change. The agent sees a structured diff: the old value, the new value, the affected rows, the outbound message body, the API payload that would be sent.

Reads don't go through shadow. They hit live data or a read replica. There's no side effect to preview. But writes—database upserts, file mutations, emails, external API calls, anything that changes state—route through shadow first.

The architecture splits at the mode router. The tool contract has already admitted the call. Now the router decides which executor runs it:

Shadow executor: Takes write tools, runs them against a shadow store or a dry-run adapter, returns a diff report. Logs everything. Doesn't touch production.

Live executor: Takes read tools and—after promotion—approved writes. Runs against production. Returns real data.

The diff report is structured: entity type, operation (insert/update/delete), old state, new state, affected IDs. For an email tool, it's the recipient list, subject, body, and headers. For a file write, it's the path, old content, new content. For an API call, it's the endpoint, method, and payload.

The agent receives the diff report as the tool result. Depending on your risk tolerance, one of three things happens next:

1. Auto-promote: Low-risk writes (idempotent upserts, append-only logs) promote automatically if the diff looks sane—no unexpected fields, row count within bounds, entity ID matches the request context. The live executor runs the same mutation.

2. Agent review: The agent inspects the diff and explicitly confirms promotion. You extend the tool contract with a promote(idempotency_key) function. The agent calls it if the diff matches intent. If not, it returns an error to the user or retries with corrected arguments.

3. Human approval: High-risk writes (deletes, irreversible external actions, financial transactions) gate on human review. The shadow result is logged, a notification fires, and the write doesn't run until someone clicks approve.

Same tool interface. Same schema. Different executor.

The decision tree

Not every tool needs shadow mode. The routing logic depends on side effects.

Read-only tools (search, fetch, query): Execute live. No shadow needed. Return data immediately.

Soft writes (idempotent database upserts, file writes, internal API calls with rollback): Shadow → compare → promote. The shadow executor runs a dry-run, returns the diff, and the agent or an automatic gate decides whether to promote. Promotion runs the same mutation against live state, tagged with an idempotency key so retries don't duplicate.

Irreversible / external side effects (send email, charge card, delete without backup, call third-party webhook): Shadow → human approval. The shadow executor shows what would happen. A human reviews the diff. Only on approval does the live executor run. No auto-promote path.

The mode router uses tool metadata—a side_effect enum or a decorator that marks the tool as read, soft_write, or irreversible—to pick the path. The agent doesn't control routing. The tool's declared risk profile does.

Implementation sketch

This isn't a drop-in starter kit. It's the structure I've built multiple times when production writes matter. Your language, data layer, and agent framework will differ. The concepts stay the same.

ModeRouter: Receives the tool name, arguments, and context. Checks the tool's side-effect type. Routes reads to LiveExecutor, writes to ShadowExecutor.

type SideEffect = 'read' | 'soft_write' | 'irreversible';

interface ToolMetadata {
  name: string;
  sideEffect: SideEffect;
  schema: JSONSchema;
}

class ModeRouter {
  private shadowExecutor: ShadowExecutor;
  private liveExecutor: LiveExecutor;
  private toolRegistry: Map<string, ToolMetadata>;

  async route(
    toolName: string,
    args: unknown,
    context: RequestContext
  ): Promise<ToolResult> {
    const tool = this.toolRegistry.get(toolName);
    if (!tool) {
      return { ok: false, error: 'Unknown tool' };
    }

    if (tool.sideEffect === 'read') {
      return this.liveExecutor.execute(tool, args, context);
    }

    const shadowResult = await this.shadowExecutor.execute(tool, args, context);
    if (!shadowResult.ok) {
      return shadowResult;
    }

    const diff = shadowResult.diff;
    const gate = this.checkGate(tool, diff, context);

    if (gate === 'reject') {
      return { ok: false, error: 'Diff gate failed', diff };
    }

    if (gate === 'auto_promote' || gate === 'agent_approved') {
      return this.liveExecutor.promote(tool, args, context, shadowResult.idempotencyKey);
    }

    return { ok: true, pendingApproval: true, diff, approvalId: shadowResult.approvalId };
  }

  private checkGate(tool: ToolMetadata, diff: DiffReport, context: RequestContext): GateDecision {
    // Implement your diff checks here: row count, entity ID match, field allowlist
    // Return 'auto_promote', 'agent_review', 'human_required', or 'reject'
  }
}

ShadowExecutor: Runs the tool against a shadow data layer. For database writes, that's a transaction you roll back, a scratch schema, or a diff query that shows what would change without committing. For external APIs, it's a mock adapter that logs the payload. Returns a DiffReport and an idempotency key.

interface DiffReport {
  operation: 'insert' | 'update' | 'delete' | 'send' | 'call';
  entityType: string;
  oldState?: Record<string, unknown>;
  newState?: Record<string, unknown>;
  affectedIds: string[];
  metadata: Record<string, unknown>;
}

class ShadowExecutor {
  async execute(
    tool: ToolMetadata,
    args: unknown,
    context: RequestContext
  ): Promise<{ ok: true; diff: DiffReport; idempotencyKey: string } | { ok: false; error: string }> {
    const idempotencyKey = generateIdempotencyKey(tool.name, args, context);

    // Run the tool's logic against a shadow store or dry-run adapter
    // Capture what would change without committing
    const diff = await this.runShadow(tool, args, context);

    // Log for audit
    await this.logShadowRun(tool, args, diff, idempotencyKey, context);

    return { ok: true, diff, idempotencyKey };
  }

  private async runShadow(
    tool: ToolMetadata,
    args: unknown,
    context: RequestContext
  ): Promise<DiffReport> {
    // Your shadow logic: START TRANSACTION, run mutation, SELECT old/new state, ROLLBACK
    // Or call a dry-run endpoint that returns what would happen without side effects
  }
}

LiveExecutor: Runs the actual mutation. For writes, it checks the idempotency key to prevent duplicate promotes. For reads, it just executes.

class LiveExecutor {
  async execute(
    tool: ToolMetadata,
    args: unknown,
    context: RequestContext
  ): Promise<ToolResult> {
    // Execute read tools directly
    return this.runLive(tool, args, context);
  }

  async promote(
    tool: ToolMetadata,
    args: unknown,
    context: RequestContext,
    idempotencyKey: string
  ): Promise<ToolResult> {
    // Check if this key was already promoted
    const alreadyRan = await this.checkIdempotency(idempotencyKey);
    if (alreadyRan) {
      return { ok: false, error: 'Already promoted' };
    }

    // Execute the write
    const result = await this.runLive(tool, args, context);

    // Mark the key as used
    await this.recordIdempotency(idempotencyKey, result);

    return result;
  }
}

DiffReport: Structured output that the agent can parse. Includes enough context to decide if the mutation matches intent. Different tools return different shapes, but the structure is consistent: operation, entity, old, new, IDs.

Failure modes

I've shipped this pattern enough times to know where it breaks.

Shadow drift from live. If your shadow store is a separate database or a stale replica, the diff might not match what would actually happen in production. Solution: run shadow writes in a transaction against the live database and roll back, or use a read-after-write pattern where the shadow executor queries live state before computing the diff.

Non-idempotent tools. If the tool's logic isn't idempotent—running it twice produces different results—the shadow run and the live promote might diverge. Solution: make your tools idempotent. If you can't, track shadow state and replay the exact parameters on promote, or use a two-phase commit where the shadow run reserves the mutation.

Partial promotes. The agent approves a batch of writes. Some promote successfully, some fail. Now you have partial state and the agent needs to know what succeeded. Solution: return structured results per mutation, log every promote attempt, and give the agent a rollback or retry tool.

Agents that retry after promote. The agent calls promote, the live executor runs, the agent doesn't parse the success response correctly and retries the same mutation. Solution: idempotency keys. The second promote call is a no-op.

"Shadow" that still hits a real webhook. Someone wraps an external API call with a shadow executor that calls the API with a dry_run=true flag. The API ignores the flag and runs the side effect anyway. Solution: don't trust external dry-run modes you haven't verified. Use a mock adapter for external calls in shadow. Test it.

Close

If your agent can mutate production on the first try, you don't have a control plane—you have a demo with credentials. Tool contracts validate the call structure. Shadow mode validates the side effect. Both are necessary. Neither is sufficient alone.

Schemas keep the agent from calling tools with garbage arguments. Shadow mode keeps the agent from executing well-formed garbage. Reads hit live. Writes hit shadow first. Diffs gate promotion. You get structured logs, rollback paths, and a second look before state changes.

I've watched agents make expensive mistakes because the team assumed prompt instructions and schema validation were enough. They weren't. The agent passed every check and still wrote to the wrong row, sent the wrong email, or deleted the wrong file. The fix wasn't a better prompt. It was a shadow executor that showed what was about to happen and a promotion gate that only let safe writes through.

If you're building production agents that touch state—CRM writes, file systems, emails, financial transactions, anything that matters—you need this architecture. Tool contracts admit the call. Shadow mode admits the mutation.

I work with teams shipping production AI systems where writes have consequences. If you're working on agent infrastructure, orchestration systems, or anything in this space, reach out: bilaltehseen@gmail.com or check out my hire page.