Tightening Tool Schemas From the Arguments You See in Production

Deriving constraints from the distribution

Once you have days or weeks of JSONL, look at the value distribution per field. There are three things to decide: is it a closed set (→ enum), is the format uniform (→ pattern), and does the numeric range converge (→ minimum / maximum)?

import { readFileSync } from "node:fs";
 
type FieldStats = {
  count: number;
  values: Map<string, number>;
  numeric: number[];
};
 
function analyze(path: string, tool: string) {
  const stats = new Map<string, FieldStats>();
  for (const line of readFileSync(path, "utf8").split("\n")) {
    if (!line) continue;
    const rec = JSON.parse(line);
    if (rec.tool !== tool) continue;
    for (const [k, v] of Object.entries(rec.input)) {
      const s = stats.get(k) ?? { count: 0, values: new Map(), numeric: [] };
      s.count++;
      const key = String(v);
      s.values.set(key, (s.values.get(key) ?? 0) + 1);
      if (typeof v === "number") s.numeric.push(v);
      stats.set(k, s);
    }
  }
 
  for (const [field, s] of stats) {
    const distinct = s.values.size;
    const suggestion: string[] = [];
    // A small, converged value set is an enum candidate
    if (distinct <= 8 && s.numeric.length === 0) {
      suggestion.push(`enum: [${[...s.values.keys()].map((x) => `"${x}"`).join(", ")}]`);
    }
    if (s.numeric.length > 0) {
      suggestion.push(`min=${Math.min(...s.numeric)} max=${Math.max(...s.numeric)}`);
    }
    console.log(`${field}: distinct=${distinct} count=${s.count} => ${suggestion.join(" / ") || "no constraint"}`);
  }
}
 
analyze("tool_calls.jsonl", "send_notification");

A sample of the output:

status: distinct=3 count=1840 => enum: ["done", "完了", "failed"]
priority: distinct=4 count=1840 => min=1 max=5
channel: distinct=2 count=1840 => enum: ["push", "email"]

Here is where the Japanese value in status becomes a number you can see. In my case roughly 6% of calls used "完了" and the rest "done" — a UI string leaking straight into the argument.

Tighten in stages

Seeing the distribution is not a license to slam a hard enum on immediately. A legitimate value may simply not have appeared during your observation window, and over-tightening pushes the model into empty tool calls or refusals because it cannot pick a value. Go in three stages.

1. Warn only. Leave the schema untouched; when a value outside the expected set arrives, just log it as schema_candidate_violation. Watch for one or two weeks to see whether false positives show up.
2. Soft constraint. Spell it out in the description: "always use one of: done / failed." It has no enforcement, but this alone collapses most of the spelling variance. Keep the warn log running here too.
3. Hard constraint. Once the warnings are quiet enough, add the enum. At the same time, keep one escape-hatch value (say "unknown") for the unexpected, and route it to a default branch in the receiving code.

The tightened tool definition looks like this:

const sendNotification = {
  name: "send_notification",
  description: "Send a notification. status must be one of the enum; use unknown if undecidable.",
  input_schema: {
    type: "object" as const,
    properties: {
      status: { type: "string", enum: ["done", "failed", "unknown"] },
      priority: { type: "integer", minimum: 1, maximum: 5 },
      channel: { type: "string", enum: ["push", "email"] },
    },
    required: ["status", "channel"],
  },
};

"完了" disappears here structurally. You do not have to ask the model in the prompt to "write in English." The value outside the set simply cannot be generated.

Before and after

Here are the results of observing the same notification tool for 2,000 calls each, once with the loose schema and once with the hard constraint. "Invalid argument rate" is the share of arguments that fell outside the receiving code's expected set; "retry rate" is the share that triggered an on-the-spot validation retry.

The token count going down is easy to miss. Declaring the set with an enum is shorter, and more reliable, than piling long caveats into the description to coax consistent spelling. Keeping an escape-hatch value meant empty tool calls barely moved.

What the docs don't tell you

An enum works twice: as a constraint and as a short instruction. The enumerated values double as a hint to the model, so you can often delete a paragraph of prose caveats. Tightening frequently makes the definition lighter.

Over-tightening turns into refusals and empty calls. Exclude a legitimate value that never appeared in your window and the model is cornered into "cannot choose." Always keep one escape-hatch value and route it to a default branch.

UI and locale-derived values will leak in. Running a Japanese app, display strings like "完了" seep into arguments. Without telemetry, that 6% stays invisible forever.

Distributions shift with model updates. Switch models and the argument habits change with them. Treat tightening as a recurring quarterly review driven by ongoing records, not a one-off.

Recommendations by situation

A correct type does not protect meaning. Measure the values that arrive, declare the set, and leave one way out. Those three steps catch most of the branches that fall through in silence, right at the entrance.

I hope this helps with your own implementation. Thank you for reading.