When Whisper Wrote 'Thanks for Watching' Into the Silence — Measuring and Killing Japanese Transcription Hallucinations

Filter by threshold — but protect real speech

Once measured, drop segments by threshold. The point is to carefully do the opposite of "when in doubt, keep it": drop only what is both silent and low-confidence, and keep anything that might contain real speech.

def filter_hallucinations(result):
    kept, removed = [], []
    for seg in result.segments:
        is_silence = seg.no_speech_prob > 0.6
        is_low_conf = seg.avg_logprob < -1.0
        # drop only when silence probability is high AND confidence is low
        if is_silence and is_low_conf:
            removed.append(seg.text.strip())
            continue
        kept.append(seg.text.strip())
 
    cleaned = " ".join(t for t in kept if t)
    return cleaned, removed

Thresholds need tuning per source. A quiet meeting room is fine at no_speech_prob > 0.6, but outdoor or phone audio lowers no_speech_prob via background noise and lets hallucinations slip through. I first dropped it to 0.5 and cut real backchannels, then went back to 0.6. Do not set a threshold once and forget it; hold it on the assumption you will re-measure when the source changes.

Strip silence with VAD before Whisper sees it

Threshold filtering is after the fact. One layer more effective is preprocessing that never hands silence to Whisper at all — VAD (Voice Activity Detection).

Cut the silence first, then transcribe, and you reduce Whisper's very opportunity to fill gaps.

import webrtcvad
import wave
import contextlib
 
def has_speech(frame_bytes, vad, sample_rate=16000):
    return vad.is_speech(frame_bytes, sample_rate)
 
def strip_silence(pcm_path, aggressiveness=2):
    # aggressiveness 0 (lax) to 3 (strict). 2 is easy to work with for meetings
    vad = webrtcvad.Vad(aggressiveness)
    speech_frames = []
    with contextlib.closing(wave.open(pcm_path, "rb")) as wf:
        sample_rate = wf.getframerate()
        frame_ms = 30
        frame_bytes = int(sample_rate * frame_ms / 1000) * 2  # 16bit = 2 bytes
        while True:
            frame = wf.readframes(frame_bytes // 2)
            if len(frame) < frame_bytes:
                break
            if has_speech(frame, vad, sample_rate):
                speech_frames.append(frame)
 
    return b"".join(speech_frames)

VAD is not a cure-all. Cutting breaths and short pauses joins speech unnaturally, so start with a middling aggressiveness of 2. I run both VAD and the threshold filter: preprocessing drops most silence, and the after-the-fact confidence filter catches what remains. A two-stage setup lets each cover the other's misses. The overall pipeline design overlaps with voice agent production architecture.

Have Claude validate what is left

Even through thresholds and VAD, a contextually odd sentence rarely survives. This is where Claude comes in. Whisper's job is "sound to text"; Claude's job is judging whether the "text" fits the context.

Pass the surrounding segments and have it point out insertions that clearly do not fit. Receiving the result via a tool (structured output) lets you process it mechanically downstream.

import anthropic, json
 
client = anthropic.Anthropic(api_key="YOUR_API_KEY")
 
def validate_transcript(segments):
    # segments: list of {"start": 12.4, "text": "..."}
    numbered = "\n".join(f"[{i}] {s['text']}" for i, s in enumerate(segments))
 
    resp = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1024,
        tools=[{
            "name": "report_hallucinations",
            "description": "return indices of insertions that do not fit context",
            "input_schema": {
                "type": "object",
                "properties": {
                    "suspect_indices": {
                        "type": "array",
                        "items": {"type": "integer"},
                        "description": "segments that do not connect, or are pure boilerplate",
                    }
                },
                "required": ["suspect_indices"],
            },
        }],
        tool_choice={"type": "tool", "name": "report_hallucinations"},
        messages=[{
            "role": "user",
            "content": (
                "The following is a transcript of meeting audio. Detect segments that "
                "do not connect with the surrounding context and consist only of streaming "
                "boilerplate like 'Thanks for watching.' Keep genuine speech.\n\n" + numbered
            ),
        }],
    )
 
    for block in resp.content:
        if block.type == "tool_use":
            return block.input["suspect_indices"]
    return []

This stage catches "normal confidence but clearly out of context" insertions that the confidence signals miss. Still, I do not hand everything to Claude; the final call always pairs it with the confidence signals. The structured-output design benefits from the thinking in structured responses with schema validation.

Keep the boilerplate blocklist as a last resort

It is tempting to just string-match "Thanks for watching" and delete them all. I did that first. But it should stay a last resort.

The reason is simple: some meetings genuinely say it. In a video-production planning call or a stream retrospective, someone means it. Delete unconditionally by string match and you cut real utterances.

BLOCKLIST = ["ご視聴ありがとうございました", "チャンネル登録"]
 
def blocklist_filter(segments):
    result = []
    for s in segments:
        text = s["text"].strip()
        # drop only if it is boilerplate ONLY and silence probability is high
        only_boilerplate = any(text == b for b in BLOCKLIST)
        if only_boilerplate and s.get("no_speech_prob", 0) > 0.6:
            continue
        result.append(s)
    return result

If you use a blocklist, always pair it with the confidence signals and drop only when both conditions align: boilerplate-only and high silence probability. Deleting on a bare string match will someday cut something real and cost you trust.

The numbers after rolling it out (field notes)

I put this into one minutes pipeline and ran it for about two weeks as an indie developer. I make every change myself and check the answers with my own ears.

Before the fix, one hour of meeting audio averaged about 12 hallucinations — streaming boilerplate most, then repeats of prior speech.

After VAD preprocessing and the confidence filter (no_speech_prob > 0.6 and avg_logprob < -1.0), insertions fell to about 4 on average. Adding Claude's context validation brought it to about 2. It does not reach zero. Outdoor and phone audio inevitably leave misses as background noise pushes confidence up.

What matters is measuring, at the same time, that you are not over-cutting. I compared body character counts before and after to confirm real speech was not excessively dropped. Fewer hallucinations but a thinner transcript is no win. Not relaxing just because a number improved is, I feel, a shared trap in this kind of quality work.

Turn it into a weekly check

Finally, the minimal routine so this does not stay a one-off. Keep it coarse enough to run quickly.

Transcript quality is decided not by the number of hallucinations removed but by whether the remaining body is correct. Killing hallucinations while measuring them is a quiet but effective way to hold that state. I prefer this "count before you cut" ordering — being able to explain it with numbers is what lets me tune it safely later.

Enable verbose_json in your own pipeline and count the high-no_speech_prob segments across one hour. The moment the real count is visible, the countermeasures become concrete. I hope this helps your implementation. Thank you for reading.