The Connection That Dies Mid-Thought Is Being Killed by Your Relay, Not the Upstream

Six relay behaviors, side by side

Writing mitigations from guesswork leaves you with nothing to learn from when they do not work. So I built a minimal rig with the upstream, the relay, and the client separated, and swapped out only the relay implementation.

Real-world seconds make this tedious to test, so the timings are scaled down while preserving the ratio. The upstream thinks for 6 seconds; the client's silence limit is 4 seconds. That is roughly one-fiftieth of the production 300-second scale. A 1.5-second ping interval against a 4-second silence limit leaves about 2.7x of headroom; scaled back up, that means a ping interval under 110 seconds is enough in production.

The upstream sends one frame, then falls quiet and emits nothing but comment lines at a set interval.

// origin.mjs — the simulated upstream: one frame, then a long think with pings only
import http from 'node:http';
 
const SILENCE = Number(process.env.SILENCE ?? 6000); // length of the thinking pause
const PING = Number(process.env.PING_MS ?? 0);       // 0 disables pings
 
http.createServer((req, res) => {
  res.writeHead(200, {
    'Content-Type': 'text/event-stream',
    'Cache-Control': 'no-cache',
    Connection: 'keep-alive',
  });
  res.write('event: message_start\ndata: {"type":"message_start"}\n\n');
 
  // The thinking pause. No content, only comment lines.
  const iv = PING ? setInterval(() => res.write(': ping\n\n'), PING) : null;
 
  setTimeout(() => {
    if (iv) clearInterval(iv);
    res.write('event: content_block_delta\ndata: {"text":"done"}\n\n');
    res.write('event: message_stop\ndata: {}\n\n');
    res.end();
  }, SILENCE);
}).listen(4001);

The client is a watchdog that does nothing but rearm a timer whenever bytes arrive. Its simplicity is the whole explanation of the behavior.

// client.mjs — any byte at all rewinds the clock
const IDLE = Number(process.env.IDLE ?? 4000);
const t0 = Date.now();
const ctrl = new AbortController();
let timer = setTimeout(() => ctrl.abort(), IDLE);
let bytes = 0;
let wakeups = 0;
 
try {
  const r = await fetch('http://127.0.0.1:4002/v1/messages', { signal: ctrl.signal });
  const reader = r.body.getReader();
  for (;;) {
    const { done, value } = await reader.read();
    if (done) break;
    bytes += value.length;
    wakeups++;
    clearTimeout(timer);                                  // it never inspects the content
    timer = setTimeout(() => ctrl.abort(), IDLE);
  }
  clearTimeout(timer);
  console.log(`OK    elapsed=${Date.now() - t0}ms bytes=${bytes} wakeups=${wakeups}`);
} catch (e) {
  clearTimeout(timer);
  console.log(`ABORT elapsed=${Date.now() - t0}ms bytes=${bytes} wakeups=${wakeups}`);
}

The relay can switch between five behaviors: stream passes bytes through, buffer reads everything before replying, strip removes whole ping frames, strip-loose removes only the ping lines and leaves the blank lines, and aggregate flushes in five-second batches.

// proxy.mjs — swap MODE to compare behaviors
import http from 'node:http';
 
const MODE = process.env.MODE ?? 'stream';
const FLUSH = Number(process.env.FLUSH ?? 5000);
 
http.createServer(async (req, res) => {
  const up = await fetch('http://127.0.0.1:4001/v1/messages');
  res.writeHead(200, { 'Content-Type': 'text/event-stream' });
 
  // The one line everybody writes. Everything piles up here.
  if (MODE === 'buffer') {
    res.end(await up.text());
    return;
  }
 
  const reader = up.body.getReader();
  const dec = new TextDecoder();
  let pending = '';
  const iv = MODE === 'aggregate'
    ? setInterval(() => { if (pending) { res.write(pending); pending = ''; } }, FLUSH)
    : null;
 
  for (;;) {
    const { done, value } = await reader.read();
    if (done) break;
    let chunk = dec.decode(value, { stream: true });
 
    if (MODE === 'strip')       chunk = chunk.replace(/^:[^\n]*\n\n/gm, ''); // whole frames
    if (MODE === 'strip-loose') chunk = chunk.split('\n').filter((l) => !l.startsWith(':')).join('\n');
 
    if (!chunk.length) continue;
    if (MODE === 'aggregate') pending += chunk;
    else res.write(chunk);
  }
 
  if (iv) clearInterval(iv);
  if (pending) res.write(pending);
  res.end();
}).listen(4002);

Same upstream, same client, relay swapped:

Row five is the one that went against my expectation. It survived while deleting the pings.

The reason is unglamorous. A line-based filter removes the : ping line but not the blank line that terminates the frame, so two newline bytes keep trickling through. The watchdog only counts bytes, so that is enough. The ping is semantically gone; the monitoring is satisfied anyway.

Together these rows show that "forward the pings and you're safe" is wrong in two directions at once. There are paths that survive without pings, and paths that die with them. What matters is not whether pings exist but whether a silent interval was created.

Note also that two of the three aborted cases delivered zero bytes to the client, because the relay was holding all of them. The dangerous part is what that does to your logs: no error upstream, no exception in the relay, and the fact of the disconnect recorded only on the client. That is a textbook recipe for a bug you cannot find by cross-referencing three log streams.

The one line that breaks it at the edge

As an indie developer I run several sites on Cloudflare Workers, so having a Worker relay a response is an everyday shape for me. What that taught me is how naturally the stream-breaking version writes itself.

// Before — works fine, until the day someone asks a hard question
export default {
  async fetch(request, env) {
    const upstream = await fetch(env.UPSTREAM, request);
    const body = await upstream.text();          // waits for all of it right here
    return new Response(body, {
      status: upstream.status,
      headers: upstream.headers,
    });
  },
};

await upstream.text() is exactly the line your hand reaches for when you want to add logging, or branch on something in the payload. And the response still comes back correctly, so short prompts reveal nothing.

// After — never touch the body, just hand the stream along
export default {
  async fetch(request, env) {
    const upstream = await fetch(env.UPSTREAM, request);
 
    // Do not consume the body. Pass the ReadableStream straight through.
    const headers = new Headers(upstream.headers);
    headers.set('Cache-Control', 'no-cache, no-transform');
    headers.set('X-Accel-Buffering', 'no');       // ask upstream layers not to buffer
    headers.delete('Content-Length');             // a stream has no length
 
    return new Response(upstream.body, {
      status: upstream.status,
      headers,
    });
  },
};

no-transform and X-Accel-Buffering: no are statements of intent aimed at whatever sits in front of your Worker. Fixing your own relay does nothing if an nginx ahead of it is buffering under the default proxy_buffering on. If you own that nginx, set proxy_buffering off; and proxy_cache off; on the relevant location, and raise proxy_read_timeout above your longest expected thinking pause.

For deciding what to do in your own code, this order keeps it simple:

1. Ask whether you genuinely need to read the body. For most relays the answer is no, so default to passing upstream.body through untouched
2. If you truly must inspect it, use a TransformStream to peek while it flows. Do not accumulate
3. If any layer compresses, reformats, or inspects the response, confirm it emits frame by frame

The third is the one people miss. A compressing middlebox coalesces small writes because that is the efficient thing to do for content delivery. It is only harmful for streaming.

Do not break what happens after the cut

One more thing the test rig cannot show you, but production will.

Claude Code retries automatically after certain upstream rejections and disables the rejected capability for the rest of the conversation to recover. That retry decision is made by matching on the wording of the upstream's error body.

Which means a relay that helpfully wraps errors in its own envelope breaks the recovery path even when it faithfully preserves the status code. I understand the urge to normalize error shapes for monitoring, and in other contexts I would be the one recommending it. On this path, the right answer was to return error responses exactly as received.

The same logic applies to relays that rewrite request bodies for inspection. Headers and body fields travel as pairs, so dropping half of a pair produces a hard 400. If you must inspect, read without modifying.

This principle — that a relay must not dilute what the upstream is telling you — is the same one behind reading rate-limit headers and throttling ahead of exhaustion, which I wrote up in field notes on measuring rate-limit headroom from headers.

Verifying before it ships

Once you have changed your relay, you do not need to wait for a long thinking pause to test it. The rig above transfers to your own setup by adjusting SILENCE and IDLE to match your real ratio.

There is only one thing to confirm: while the upstream is quiet, are bytes still reaching the client? If yes, you are safe. If no, you will fail on the day that interval grows past the limit.

If you want a single action today, open your relay's source and search for a .text() or .json() after an await. If one is there, that is where your silent interval is being manufactured.

It took me several rounds of blaming the upstream and the model before I found that line. I hope this saves you the detour.