⬡ API & SDK/2026-07-14Advanced

A Two-Stage Pre-Publish Gate for User-Facing AI Text in Consumer Apps

Design a two-stage pre-publish gate for short AI-generated text you ship to end users: a deterministic rule layer plus a Claude classifier, with fail-closed handling, generation-time vetting, and a cost model. Full implementation code included.

Claude⁴⁶ API²⁷ Moderation Indie Dev²² App Development⁵

✦ Premium Article

When I first tried to add a daily one-line message to one of my calming apps, the thing that scared me was not whether the model could write a nice sentence. It was the one sentence in a thousand — in ten thousand — that slips through and lands, unedited, on a user's morning screen.

A chatbot can recover. If it says something odd, the conversation itself corrects it. A single line dropped into an app screen is different. The user reads it as the app talking, and there is no back-and-forth to walk it back. We tend to judge generation quality on the average, but for user-facing copy the question is not the average. It is how you stop the single worst case.

As an indie developer running several apps, what I eventually settled on was this: separate from any effort to improve generation quality, place a gate right before delivery whose only job is that no unvetted character ever reaches a user. This article walks through implementing that pre-publish gate as two stages — a deterministic rule layer and a Claude classifier.

Put the gate on the generation path, not the request path

The first decision is when the vetting runs. Calling the API the moment a user opens the app looks straightforward but carries three costs: it makes users wait, it bills you on every view, and when the API is down it forces the worst dilemma — block delivery, or ship unvetted.

I push vetting to the generation side. Copy is produced ahead of time, vetted then, and only what passes is stored as approved. The user's screen simply pulls an already-cleared line. The user experience stays fully synchronous, while the weight of judgment hides in an asynchronous background batch.

Aspect	Vet at request time	Vet at generation time (this article)
User wait	Adds judgment latency	Zero — just a lookup
Cost	Billed per view	Only per generated candidate
On API outage	Halt delivery or ship unvetted	Keep running from the vetted pool
Human review	Only possible after the fact	Possible before delivery

Nail this down first and every later layer can be designed on the assumption that it runs offline, in a batch, and is allowed to be a little slow.

The deterministic rule layer — all items, in milliseconds

The first stage calls no model at all. Whatever you can reject here is faster, cheaper, and more reproducible done in code than handed to a probabilistic model. Its job is to knock out the "obviously not allowed" across every item in milliseconds.

import re
import unicodedata
 
# Vocabulary tuned to your delivery policy; keep it in an external file in production
BANNED_LEXICON = {"scam", "guaranteed profit", "kill yourself"}
URL_RE = re.compile(r"https?://|www\.", re.IGNORECASE)
EMAIL_RE = re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+")
EXCESS_SYMBOL_RE = re.compile(r"[!?]{3,}")
 
def deterministic_gate(text: str, min_len: int = 8, max_len: int = 90) -> dict:
    """First-pass filter over every item. If it passes, reasons is empty."""
    normalized = unicodedata.normalize("NFKC", text).strip()
    reasons = []
 
    length = len(normalized)
    if length < min_len:
        reasons.append("too_short")
    if length > max_len:
        reasons.append("too_long")
 
    lowered = normalized.lower()
    for word in BANNED_LEXICON:
        if word in lowered:
            reasons.append(f"banned:{word}")
 
    if URL_RE.search(lowered) or EMAIL_RE.search(normalized):
        reasons.append("contains_contact")
    if EXCESS_SYMBOL_RE.search(normalized):
        reasons.append("excess_symbols")
 
    return {"passed": len(reasons) == 0, "reasons": reasons, "text": normalized}
 
 
# Expected behavior
print(deterministic_gate("Take one quiet step forward today."))
# => {'passed': True, 'reasons': [], 'text': 'Take one quiet step forward today.'}
print(deterministic_gate("Guaranteed profit, learn how!!!"))
# => {'passed': False, 'reasons': ['banned:guaranteed profit', 'excess_symbols'], ...}

The important discipline here is not to adjudicate gray areas. Anything subtle or context-dependent should pass this layer and be handed to Claude in the second stage. The rule layer sticks to rejecting the indefensible. You grow the lexicon from the human review described later, so do not chase perfection on day one.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦You can stop worrying that the one bad line a model occasionally produces will land straight on a user's screen

✦You'll be able to combine a deterministic rule layer with a Claude classifier and fail-closed logic so unvetted text is never shipped

✦By vetting on the generation path instead of the request path, you raise safety without adding latency or per-view API cost

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

The Claude classifier — where ambiguity gets absorbed

The second stage judges the context-dependent appropriateness of lines that survived the rule layer. Free-text opinions are useless here. You must receive the verdict as structure. Fix the output schema with Claude's tool use and pull out only the boolean and reason codes. A cheap Haiku-class model handles this comfortably.

import json
from anthropic import Anthropic
 
client = Anthropic()  # reads ANTHROPIC_API_KEY from the environment
 
CLASSIFY_TOOL = {
    "name": "record_verdict",
    "description": "Judge and record the safety and quality of a delivery line",
    "input_schema": {
        "type": "object",
        "properties": {
            "safe": {"type": "boolean", "description": "OK to deliver"},
            "categories": {
                "type": "array",
                "items": {"type": "string"},
                "description": "Concern categories: self_harm, medical_claim, off_brand, etc.",
            },
            "quality": {"type": "integer", "description": "Writing quality 1-5"},
            "brand_fit": {"type": "integer", "description": "Fit with the app tone 1-5"},
        },
        "required": ["safe", "categories", "quality", "brand_fit"],
    },
}
 
SYSTEM = (
    "You review delivery copy for a calming, wellbeing app. "
    "Flag medical or financial assertions, phrasing that evokes self-harm, "
    "and aggressive or sensational tone that clashes with the app's gentle voice."
)
 
def classify(text: str, model: str = "claude-haiku-4-5-20251001") -> dict:
    resp = client.messages.create(
        model=model,
        max_tokens=256,
        system=SYSTEM,
        tools=[CLASSIFY_TOOL],
        tool_choice={"type": "tool", "name": "record_verdict"},
        messages=[{"role": "user", "content": f"Judge this delivery line:\n{text}"}],
    )
    for block in resp.content:
        if block.type == "tool_use" and block.name == "record_verdict":
            return block.input
    # If no tool output comes back, treat it as undecided
    return {"safe": False, "categories": ["undecided"], "quality": 0, "brand_fit": 0}
 
 
# Expected output shape
# {'safe': True, 'categories': [], 'quality': 4, 'brand_fit': 5}

Forcing the call with tool_choice is the point. It removes any room for "Yes, I think this line is safe because..." prose and gives you the same dictionary shape every time. Fixing output by type is the same instinct as structuring output with schema validation and a repair loop.

Do not collapse the verdict to a single boolean. Receiving quality and brand_fit as numbers lets you route "safe but low quality" and "safe but off-brand" to a hold. Safety and quality are different axes.

Compose the verdicts — make fail-closed the spine

Now wire the two layers together. The single most important property is that when in doubt, you never tip toward "ship it" — fail-closed, all the way through. If the classifier times out, blows the budget, or throws, that line does not go to delivery; it is held. When the pool runs dry, what you serve is not a generated line but a safe default written by hand in advance.

def vet_snippet(text: str, budget_ok: bool) -> dict:
    """Vet one line and return a verdict. Unvetted text is never approved."""
    rule = deterministic_gate(text)
    if not rule["passed"]:
        return {"status": "rejected", "stage": "rule", "reasons": rule["reasons"]}
 
    if not budget_ok:
        # Out of budget: skip judgment this round and defer (lean on the vetted pool)
        return {"status": "deferred", "stage": "budget"}
 
    try:
        v = classify(rule["text"])
    except Exception as e:  # network, rate limit, outage all tip to hold
        return {"status": "deferred", "stage": "classify_error", "error": str(e)}
 
    if v["safe"] and v["quality"] >= 3 and v["brand_fit"] >= 3:
        return {"status": "approved", "text": rule["text"], "verdict": v}
    if v["safe"]:
        return {"status": "review", "text": rule["text"], "verdict": v}  # safe but weak
    return {"status": "rejected", "stage": "classify", "verdict": v}

Splitting the state into approved / review / rejected / deferred matters because each is handled differently downstream. Only approved enters the delivery pool. review goes to human eyes, rejected feeds the lexicon and prompt improvements, and deferred retries in the next batch. Not conflating "unsafe" with "couldn't decide" is what keeps fail-closed from being a slogan.

Vet at generation time, and just fetch at delivery time

Store each vetted line together with its verdict. The app's delivery path becomes nothing more than pulling one approved item from this vetted pool. The Claude call disappears entirely from the user's request path, and neither latency nor cost scales with delivery volume.

Keep the pool stocked with a scheduled generation batch. If inventory drops below a threshold, generate more; if there is headroom, stop. That simple refill logic is enough. When you batch the classification, leaning on Message Batches rather than the synchronous API keeps cost down — the async design in designing async cost with Message Batches applies directly.

Cost scales with generation count, not delivery count

The cost of this design is set by how many candidates you generate and vet, not by how often users open the app. That is the decisive difference from request-time vetting. A rough monthly estimate takes this shape:

Item	Formula
Generated items vetted / month	N = target inventory ÷ average pass rate × refills
Classification cost / month	N × (input tokens × input rate + output tokens × output rate)
Effective rate reduction	Roughly halved via Message Batches

Classification runs on a cheap Haiku-class model with a few hundred tokens each way. However far delivery grows, classification cost is capped by generation count, so per-delivery vetting cost falls as users increase. Model rates change, so rather than hardcoding a figure, confirm current pricing on the Anthropic pricing page.

Build human sampling into the design

Left alone, this two-stage gate drifts from reality. New phrasings, seasonal topics, a shifting user base — both the rule-layer lexicon and the classifier prompt only keep working if you keep updating them. So route a fixed fraction of approved items, plus every item that fell to review, into a periodic human review queue.

Whenever review turns up an approved line you wish had not shipped, decide each time whether it belongs in the deterministic lexicon or the system prompt. Context-independent and always disallowed? The lexicon. Context-dependent? The prompt's criteria. That round trip keeps the gate a living mechanism. Pairing this with an eye on adversarial input — see prompt injection defense patterns — thickens the guard around delivery copy.

Common mistakes and pitfalls

First, trying to replace the deterministic layer with Claude. Handing length overruns and contact-info leaks to a probabilistic model is slower, pricier, and leakier. What is mechanically decidable should be rejected mechanically.

Second, receiving the classifier's output as free text. The moment you start parsing "I think it's safe" with regex later, the pipeline turns brittle. Always pin it to a type with tool use.

Third, failing open. The single line "if the API is down, just ship the generated text as-is" is exactly the bypass that lets the worst case through. Hold on failure, fall back to defaults when inventory runs out — keep that order.

Fourth, never re-examining approved lines. Tone and users both move. Treat the approved pool as having a shelf life and keep the sampling running.

One step to try this week

Grab just ten delivery lines — ones you ship, or plan to ship — and run them through deterministic_gate above. Simply seeing how many get knocked out mechanically, and which reason codes line up, will reveal the first rules your own app actually needs. The second-stage Claude classifier can come after that.

Safety for delivery copy is not a flashy feature. But knowing there is quietly one gate behind the single line a user receives each morning — how you build that one gate is, I've come to feel, the kind of thing that pays off the longer you keep an indie app alive.

Thank You for Reading

Claude Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.