Rolling Out Agent Behavior Changes Gradually with Feature Flags

Assign buckets deterministically

If you decide rollout with a plain random draw, every regeneration of the same article flips between old and new, and the observation turns to mud. The premise that makes the comparison valid is that the same job always falls into the same group.

So take a key that uniquely identifies the run (site name plus slug, or a job ID), hash it, and map it onto buckets 0–99. Mixing the flag key into the hash keeps multiple flags from repeatedly landing on the same set of runs and skewing everything.

import { createHash } from "node:crypto";
 
// Derive a stable 0..99 bucket from run key + flag key
function bucketOf(runKey: string, flagKey: string): number {
  const h = createHash("sha256").update(`${flagKey}:${runKey}`).digest();
  // Map the first two bytes onto 0..99 (distribution is close enough to uniform)
  return ((h[0] << 8) | h[1]) % 100;
}
 
type Flag = {
  enabled: boolean;
  rollout: number;
  killed: boolean;
  scope?: string[];
  owner?: string;
  createdAt?: string;
};
 
// Should this run get the new behavior? Side-effect free and fully deterministic
export function isCanary(
  flag: Flag | undefined,
  runKey: string,
  flagKey: string,
  siteScope: string,
): boolean {
  if (!flag || !flag.enabled) return false;       // undefined/disabled -> old behavior
  if (flag.killed) return false;                   // emergency stop wins over everything
  if (flag.scope?.length && !flag.scope.includes(siteScope)) return false;
  return bucketOf(runKey, flagKey) < flag.rollout; // 25 -> buckets 0..24 are canary
}

Evaluating killed before rollout is the crucial part. When something goes wrong at night, lowering the rollout number and snapping everything back to the old behavior are different in kind. The former is "adjust the fraction"; the latter is "retreat unconditionally." So the retreat switch sits ahead of every other condition, so there is nothing to hesitate over in an emergency.

Switch behavior at a single seam

Once the decision exists, the branch itself stays thin. Swap only the prompt variable and leave the downstream generation logic shared.

import Anthropic from "@anthropic-ai/sdk";
 
const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
 
const INTRO_CONTROL =
  "Begin the article with a concrete description of the problem the reader is facing.";
const INTRO_CANARY =
  "Begin the article from the moment the author first ran into the problem, in a quiet first person.";
 
export async function generateArticle(opts: {
  runKey: string;      // e.g. "claudelab/agent-feature-flag-rollout"
  site: string;        // e.g. "claudelab"
  flags: Record<string, Flag>;
  topic: string;
}) {
  const useCanary = isCanary(
    opts.flags["gentler-intro-phrasing"],
    opts.runKey,
    "gentler-intro-phrasing",
    opts.site,
  );
  const variant = useCanary ? "canary" : "control";
  const introRule = useCanary ? INTRO_CANARY : INTRO_CONTROL;
 
  const msg = await client.messages.create({
    model: "claude-sonnet-5",
    max_tokens: 4096,
    system: `You are a technical writer.\n${introRule}`,
    messages: [{ role: "user", content: opts.topic }],
  });
 
  // Always record which group generated this — the lifeline of the later comparison
  return { variant, text: msg.content, runKey: opts.runKey };
}

Do not forget to return variant alongside the result. If you fail to record which group a run belonged to, you cannot line up control against canary afterward, and the whole point of the flag evaporates.

Compare canary against control

Whether the new behavior "seems good" is a judgment you make by placing the same metric from both groups side by side, not by impression. In my case, the first thing I look at is the quality gate pass rate. It carries no subjectivity and can be tallied the same day.

type RunLog = {
  runKey: string;
  variant: "control" | "canary";
  gatePassed: boolean;   // passed the quality gate on the first try
  charCount: number;     // character count of the generated body
  ts: string;
};
 
// Compute each group's pass rate from recent logs
function compare(logs: RunLog[]) {
  const rate = (v: string) => {
    const g = logs.filter((l) => l.variant === v);
    const pass = g.filter((l) => l.gatePassed).length;
    return { n: g.length, passRate: g.length ? pass / g.length : 1 };
  };
  return { control: rate("control"), canary: rate("canary") };
}

For the observation window, if you generate only one article a day, watch it for about three days. That keeps you from reacting to noise while the canary sample is still tiny. In my own setup one article runs per night, so a 25% canary is actually drawn only once every few days. Given that, I have decided to hold judgment until at least three canary runs have accumulated.

When you detect a regression, close the flag on the next run

If the comparison shows the canary pass rate clearly below the control group, retreat without waiting for a human to wake up. Here again, set only killed and touch none of the other values, because I want to preserve when and at what numbers it was closed for the investigation later.

import { readFileSync, writeFileSync } from "node:fs";
 
// Kill if canary is at least REGRESSION_PT below control
const REGRESSION_PT = 0.05; // 5 points
const MIN_CANARY = 3;       // minimum sample size
 
export function autoRollback(flagKey: string, logs: RunLog[], path: string) {
  const { control, canary } = compare(logs);
  if (canary.n < MIN_CANARY) return { acted: false, reason: "insufficient sample" };
 
  const gap = control.passRate - canary.passRate;
  if (gap < REGRESSION_PT) return { acted: false, reason: "no regression" };
 
  const flags = JSON.parse(readFileSync(path, "utf8"));
  flags[flagKey].killed = true;
  flags[flagKey].killedAt = new Date().toISOString();
  flags[flagKey].killedReason =
    `canary passRate ${canary.passRate.toFixed(2)} < control ${control.passRate.toFixed(2)}`;
  writeFileSync(path, JSON.stringify(flags, null, 2));
  return { acted: true, gap };
}

The important thing is to run this function right after generation, before the next run begins. Slot the evaluation at the end of the nightly batch and the regression is contained to at most one night's worth. It never becomes that morning again — the one where four sites went down together and cost me a whole day.

Field notes the docs do not cover

Do not break bucket stability. Change how you build the run key midway and jobs that were control migrate to canary, severing the continuity of the comparison. Fix the key definition once you set it; if you must change it, re-cut it as a new flag.

Treat kill as distinct from adjusting the fraction. rollout: 0 and killed: true look similar in outcome but mean different things. The former is "not widening today"; the latter is "I have judged this behavior dangerous." During an audit, that difference decides how fast you trace the cause.

Flags have a lifespan. This is the one I overlooked most. Once a new behavior is proven good, delete the flag and promote the behavior to the default. Leave it and branches pile up until nobody can tell which path is alive. Once a month I sweep flags whose createdAt is more than thirty days old, and I set aside time to settle each one as promote, remove, or hold.

The smaller you start with a reversible change, the better, and I recommend it strongly.

Closing

When you apply a change to a system that runs unattended, before making it "take effect everywhere," prepare a path to "confirm it on a fraction and retreat quietly if it turns out badly." A feature flag is a very plain tool for exactly that.

As a next step, try just one, in this order:

1. Pick one change you currently apply to every run immediately
2. Re-place it as a branch with a single scope and a 25% rollout
3. Once three canary runs accumulate, compare against control and, if good, widen to 50 then 100

The view the next morning will be a little calmer.

These are only humble operational notes, but if they make even one of your nightly batches a little quieter, I will be glad. Thank you for reading.