Before You Send Reviews and Crash Logs to the Claude API: A Reversible PII Masking Design

Designing reversible masking: choosing the token format

The idea is simple. Replace each detected piece of personal data with a one-off unique token, and hold the token-to-value mapping in memory for that request only. Claude receives the tokenized text; if the same tokens survive in the returned output, I look them up in the mapping and restore the original values.

The quietly important decision is what string the token should be. I first reached for «EMAIL_1» with guillemets, for looks. That backfired in production. While formatting its output, the model would normalize those unusual marks into different quote characters, or sweep them into a translated phrase when I asked for English. Any deformation of the token breaks the exact match on restore.

# Before: looks nice, but the model easily breaks it during normalization or translation
token = f"«{kind}_{n}»"        # « » can morph into other quote marks
 
# After: ASCII letters, digits, underscores only. The model passes it through verbatim
token = f"__PII_{kind}_{n}__"  # no symbol delimiters, so the original form survives

Once I switched to ASCII letters and underscores wrapped in double underscores, token corruption all but disappeared. Assigning the same token to the same value also preserves the consistency of mentions within the body. Here is the core masker.

import re
from dataclasses import dataclass, field
 
@dataclass
class Masker:
    detectors: dict[str, re.Pattern]                       # kind -> regex
    _forward: dict[str, str] = field(default_factory=dict) # value -> token
    _reverse: dict[str, str] = field(default_factory=dict) # token -> value
    _counts: dict[str, int] = field(default_factory=dict)
 
    def _token_for(self, kind: str, value: str) -> str:
        if value in self._forward:
            return self._forward[value]            # same value -> same token
        n = self._counts.get(kind, 0) + 1
        self._counts[kind] = n
        token = f"__PII_{kind}_{n}__"
        self._forward[value] = token
        self._reverse[token] = value
        return token
 
    def mask(self, text: str) -> str:
        for kind, pattern in self.detectors.items():
            text = pattern.sub(lambda m: self._token_for(kind, m.group(0)), text)
        return text
 
    def restore(self, text: str) -> str:
        # Only restore tokens I issued. Leave unknown tokens untouched
        for token, value in self._reverse.items():
            text = text.replace(token, value)
        return text

Building the mapping per request and discarding it when done is the crux. A persistent mapping reused across the whole process becomes its own accumulation of personal data, which defeats the point of masking.

Implementing the detector: regex and NER, used for different jobs

Detection is two layers. Things with a fixed format I pick up with high precision via regex; things without a fixed format, like names, I supplement with named entity recognition (NER) only when needed.

Fixed formats go to regex

Email addresses, phone numbers, URLs, and my app's own identifiers are most reliable with regex. Since I define the identifier format myself, I can drive its false positives to nearly zero.

DETECTORS = {
    "EMAIL": re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}"),
    "PHONE": re.compile(r"\+?\d[\d\-\s]{7,}\d"),
    "URL":   re.compile(r"https?://[^\s]+"),
    # Identifier my app issues. Fixed format, so false positives are rare
    "UID":   re.compile(r"\bUSR-[0-9A-F]{12}\b"),
}

Fuzzy things like names go to NER, but carefully

Names can't be caught by regex. That is where NER comes in, with a caveat. NER trades recall against precision, and lowering the threshold starts flagging common nouns as names until the body is full of holes. For review triage I narrowed it to "person-label, high-confidence only," and tipped every borderline candidate to the not-masked side. For sorting reviews, I weigh the risk of a broken body and worse labels more heavily than the risk of one name slipping through occasionally.

Detection order has a pitfall too. When an email-like string sits inside a URL, the detectors compete, so applying the longer, more specific format first is safer. I run URL first, then email, phone, and identifier.

Sending to Claude and restoring the result

Wiring the parts together gives a round trip: hand the masked body to Claude, take back the label, and restore any tokens left in the output. I want to run a large volume of reviews cheaply, so I use Haiku.

import anthropic
 
client = anthropic.Anthropic(api_key="YOUR_ANTHROPIC_API_KEY")
 
def triage_review(raw_text: str) -> dict:
    masker = Masker(detectors=DETECTORS)
    masked = masker.mask(raw_text)            # mask before sending
 
    resp = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=300,
        system=(
            "You sort app reviews. Strings like __PII_...__ in the body are "
            "already-masked personal data. Do not alter or translate them; "
            "quote them verbatim. Return JSON with a kind "
            "(bug/request/praise/other) and a short summary."
        ),
        messages=[{"role": "user", "content": masked}],
    )
    out = resp.content[0].text
    restored = masker.restore(out)            # restore only where needed
    return {"masked_input": masked, "restored_output": restored}

The key was spelling out token handling in the system prompt. Just adding "do not alter or translate them; quote them verbatim" noticeably raised how often tokens survive intact in the output. Even so, the model can paraphrase a token or mangle it during translation; the odds never hit zero. That is exactly why the restore side only puts back tokens I issued. If the model invents a token, anything not in the mapping is ignored, and no original data is restored by accident.

Pitfalls I hit in production

Working as designed is not the end; operations surface their own problems. Three that actually bit me.

The first was leakage through logs. The preprocessing is safe, but when an exception fires, the stack trace and debug log can carry the unmasked raw body verbatim. The instant I raised the log level to investigate an error, data I thought I had hidden sat there in plaintext — exactly backwards. The fix was to standardize on functions that take a raw body catching the exception, recording only the category, and never logging the body itself. Masking has to be conscious not just at the entrance but at the log exit too.

The second was double restoration. Calling restore twice on the same Masker, or feeding an already-restored string through another mapping, breaks things when it tries to restore tokens that are already gone. I enforce "restore the output exactly once" at the code level.

The third is the rare case where a token changes the meaning of the sentence. When a token reads grammatically out of place — "my name is PII_NAME_1" — the model sometimes decides "this is a redaction" and helpfully rewrites it into another phrasing. Spelling it out in the system prompt and moving to ASCII tokens dropped the frequency a lot, but not to zero. For important runs, I verify that every issued token survives in the output and route just that review to a human if one is missing.

How far to take it: a realistic answer for solo devs

If you try to do everything perfectly as a solo developer, preprocessing alone eats your time. I recommend protecting your sources in priority order. Concretely: first reliably seal email addresses, phone numbers, and your own identifiers — the things you can catch mechanically with high precision and that hurt most if they leak — and keep fuzzy things like names to high-confidence detection only.

By the numbers, in my setup regex-based detection alone captures the large majority of the meaningful personal data, and the false-positive rate on my own identifiers stays under 0.5%. Dropping NER on every record in favor of high-priority regex cut preprocessing time to less than half — roughly a 2x or better speedup. It is the same lesson as reading my own revenue reports: rather than chasing everything perfectly, securing the high-impact parts solidly is what keeps a solo operation sustainable.

Even then, the act of handing data to an external API always carries responsibility. Stating honestly in your terms and privacy policy how you handle collected data matters just as much as the technical preprocessing.

Measuring what slips through: a golden corpus against regressions

The scariest thing in preprocessing is fixing one thing and creating another leak. Adjusting a single detector regex routinely drops a pattern you used to catch. I keep a small golden corpus: a handful of reviews that contained personal data, hand-marked with "email here," "identifier here." Rather than holding real bodies, which only grows what I have to manage, I replace them with synthetic data that keeps the structure.

# Cases with expectations. The text is synthetic; only the marks are ground truth
CASES = [
    {"text": "refund please user@example.com device USR-0A1B2C3D4E5F",
     "expect_kinds": {"EMAIL": 1, "UID": 1}},
    {"text": "Great app, gave it 5 stars",
     "expect_kinds": {}},   # no personal data: nothing must be hidden
]
 
def test_detection():
    for c in CASES:
        m = Masker(detectors=DETECTORS)
        m.mask(c["text"])
        got = {k: v for k, v in m._counts.items() if v}
        assert got == c["expect_kinds"], f"mismatch: {got} != {c['expect_kinds']}"
    print("detection OK")

The second case quietly earns its keep. It checks, every time, that the detector is not wrongly masking an ordinary review with no personal data. Preprocessing quality runs on two wheels — not leaking and not over-hiding — and without a way to measure the latter, the body silently fills with holes and the model's accuracy slips. Since I started running this on every change, I stopped being afraid to tune detectors. It is a small safety valve before shipping, but for a process that handles entrusted data, this is one I refuse to skip.

Mask Before Claude, Because the Leak Is on Your Side

Where you put preprocessing is a design decision to settle before you ever tune accuracy. Anthropic's policy is explicit that Claude API traffic is not used for training by default. Even so, the path your request body travels runs deeper into your own system than most people expect.

Three routes have made me nervous in production:

• Replay into staging: Sampling production traffic to reproduce regressions is genuinely useful, but it hands raw personal data to whoever runs those tests. That can breach your own handling policy even when nothing leaves your VPC.
• Exception stack traces: In both Python and TypeScript, it is alarmingly easy for an exception to call __repr__ on the whole messages array and spill it into your error log.
• Cache keys: If a prompt or response cache keys on the raw body, that PII now lives in Redis slow logs, snapshot dumps, and every backup you have taken.

All three are problems in your system, not Claude's. That is why masking belongs before the API call — ideally as deep inside your SDK wrapper as you can push it. Filtering the response is fine as a last resort, but it should never be your first line of defense.

Your next step

Start by writing the body you are sending to Claude to standard output, exactly once, in your own pipeline. Seeing with your own eyes how much personal data is mixed into your real reviews and crash logs is what makes it concrete which detectors to prioritize. Once you have checked, delete that output without fail. That is the first step.

Thank you for reading this far. If it helps the design of anyone else holding users' words in trust, I would be glad.