Isolating Poison Messages in a Claude Async Pipeline: A Dead-Letter Queue Implementation Note

Configuring a DLQ on Cloudflare Queues

Since I run all four sites on Cloudflare Workers, my async work also runs on Cloudflare Queues. The nice part is that a DLQ is a first-class infrastructure feature — you do not need to maintain your own retry-count column in the app.

The configuration is just two fields on the consumer. max_retries is the maximum retries per message (defaults to 3), and dead_letter_queue is the queue to send a message to once that cap is reached. If the target queue does not exist, it is created automatically.

# wrangler.toml
name = "wallpaper-tagger"
main = "src/index.ts"
 
[[queues.producers]]
queue = "tag-jobs"
binding = "TAG_QUEUE"
 
[[queues.consumers]]
queue = "tag-jobs"
max_retries = 3            # after 3 failures, go to the DLQ
max_batch_size = 10
dead_letter_queue = "tag-jobs-dlq"
 
# Attach a consumer to the DLQ too, so you can observe and redrive
[[queues.consumers]]
queue = "tag-jobs-dlq"
max_batch_size = 5

If you do not define dead_letter_queue, messages past the cap are silently discarded. This is easy to miss on first encounter; I spent a few hours confused about why "I set a retry cap, yet failed jobs left no trace anywhere." If you do not want to lose failures, the DLQ declaration is mandatory.

The Consumer Implementation — Before / After

My first consumer called retryAll() on any failure — a textbook poison-message factory.

// Before — one failure drags down the whole batch
export default {
  async queue(batch: MessageBatch, env: Env) {
    try {
      for (const msg of batch.messages) {
        await tagOneImage(msg.body, env); // if any one throws...
      }
      batch.ackAll();
    } catch (e) {
      batch.retryAll(); // ...the nine healthy records get resent too
    }
  },
};

This has two sins: it resends healthy messages on a single failure, and it does not distinguish a transient failure from a poison message.

The fixed version acks/retries each message independently and immediately acks poison messages after recording them. On Cloudflare Queues, acking a message without calling message.retry() marks it processed; calling retry() redelivers it, and hitting max_retries routes it to the DLQ automatically.

// After — process each message independently and isolate only the poison
import { classifyError } from "./classify";
 
export default {
  async queue(batch: MessageBatch, env: Env) {
    for (const msg of batch.messages) {
      try {
        await tagOneImage(msg.body, env);
        msg.ack(); // only this one is marked successful
      } catch (e) {
        const verdict = classifyError(e);
        if (verdict === "poison") {
          // Poison: ack immediately to remove it from the normal flow,
          // and record it with its failure reason in a DLQ-equivalent log
          await env.DLQ_LOG.put(
            `poison:${msg.id}`,
            JSON.stringify({ body: msg.body, reason: String(e), at: Date.now() }),
            { expirationTtl: 60 * 60 * 24 * 30 }
          );
          msg.ack();
        } else {
          // Transient: send to retry. Hitting max_retries routes it to the DLQ
          msg.retry({ delaySeconds: backoff(msg.attempts) });
        }
      }
    }
  },
};
 
// Exponential backoff (attempts is 1-based) + jitter
function backoff(attempts: number): number {
  const base = Math.min(2 ** attempts, 60);
  return base + Math.floor(Math.random() * base * 0.3);
}

Being able to pass delaySeconds to msg.retry() matters more than it looks. Without it, you resend immediately after a rate limit and hit another 429 — a wasteful retry loop. The jitter on the backoff prevents multiple messages in the same batch from being resent at the exact same instant and re-congesting.

Observing the DLQ and Redriving Safely

A DLQ is not "where failures go to die" — it is "where failures are parked until you understand the cause and send them back." In my DLQ consumer I classify incoming messages by cause before deciding what to do next.

// dlq-consumer.ts — classify dead-lettered poison messages for triage
export async function handleDlq(batch: MessageBatch, env: Env) {
  for (const msg of batch.messages) {
    const job = msg.body as TagJob;
    const reason = await diagnose(job); // re-assess the cause
    await env.DLQ_INDEX.put(`${reason}:${job.id}`, JSON.stringify(job));
    msg.ack(); // ack once pulled from the DLQ (don't keep piling up)
  }
}
 
// Redrive: send only the repairable ones back to the main queue
export async function redrive(env: Env, reason: string) {
  const list = await env.DLQ_INDEX.list({ prefix: `${reason}:` });
  for (const key of list.keys) {
    const job = JSON.parse((await env.DLQ_INDEX.get(key.name))!);
    const repaired = repairJob(job); // e.g. strip invalid bytes, trim description
    if (!repaired) continue;          // don't send back what you can't fix
    await env.TAG_QUEUE.send({ ...repaired, idemKey: `redrive:${job.id}` });
    await env.DLQ_INDEX.delete(key.name);
  }
}

The most important thing in a redrive is idempotency. Some jobs in the DLQ may have "succeeded in calling Claude but failed while saving the result." Reprocessing them unconditionally calls Claude twice for the same image and double-bills you. I attach an idemKey to each job and guard the result-save step with "if already saved under this key, skip the API call." With an idempotency key in place, you can rerun a redrive as many times as you like without fear.

Poison-Message Patterns Specific to the Claude API

On top of generic queue design, a Claude pipeline has its own sources of poison messages. When I took inventory of what actually accumulated in my DLQ, most fell into one of these.

Inputs that exceed the max_tokens ceiling and cannot fit into a single request no matter how you slice them. In my wallpaper metadata, a handful of images had enormous EXIF comments embedded — exactly this case. The fix is to measure input length before enqueueing and route over-limit items to a separate lane (summarize first, then send).

Structured outputs via tool_use where Claude's returned JSON does not match your schema. The same input tends to produce the same structure on retry, so treating it as poison is reasonable. I send schema violations to the DLQ while flagging the prompt instructions for review.

Responses with stop_reason: refusal, where Claude declined to generate for content-policy reasons. Retrying does not solve this, so it goes straight to the DLQ for a human to check. In my use case (wallpaper tagging), art-style image descriptions occasionally tripped this.

Given all this, one discipline pays off: always inspect stop_reason, and never treat anything but end_turn as success. Many incidents start by assuming "a 200 came back, so it succeeded" and then saving an incomplete response that was truncated by max_tokens.

// Judge success by stop_reason, not just status
const res = await client.messages.create({ /* ... */ });
if (res.stop_reason !== "end_turn" && res.stop_reason !== "tool_use") {
  // max_tokens / refusal is not a "successful 200"
  throw new IncompleteResponseError(res.stop_reason);
}

What Worked in Production

After running this setup for about half a year across content generation for four sites and my wallpaper app backend, a few conclusions of my own emerged.

Don't be greedy with max_retries — 3 is practical. Most rate limits and overloads recover within three backoffs, and going higher only let poison messages linger longer. I started at 5; after dropping to 3, the throughput dips became noticeably shallower.

Treat the DLQ as a monitored signal. I send an alert when the DLQ message count crosses a threshold. A quietly growing DLQ was almost always the sign of a new kind of breakage appearing in the input data. Isolating failures and then ignoring them means missing the very failures you worked to eject.

And ultimately, handling poison messages is the design of "how you treat broken inputs" itself. Both of my grandfathers were temple carpenters, and I'm told they inspected each piece of timber before joining it. Just as you don't quietly mix rotted or knotted wood into a structure, not mixing one broken record into a healthy flow is what extends the life of the whole pipeline. Running an app portfolio with 50 million cumulative downloads on my own since 2014, I've felt the value of "moving broken things aside quickly, quietly, and reliably" again and again.

If you want to try this, I'd suggest starting by adding a classifyError-style split to a synchronous loop you already have. Just seeing failures as "healing vs. not" changes the stability of a pipeline more than you'd expect. Thank you for reading.