Adding Priority and Fairness to a Claude API Job Queue — Backpressure Patterns from a 50M-Download Indie App

Implementing the three-tier priority queue — critical / standard / batch

My worker keeps three logical queues in Redis, each a separate LIST, distinguished only by priority label. The enqueuing side decides which queue a request belongs in.

# enqueue.py — classification on the producer side
def classify_priority(req) -> str:
    # 1. user-action-triggered  -> critical
    if req.origin == "user_action":
        return "critical"
    # 2. anything Claude can answer within 24h -> batch
    if req.sla_seconds is None or req.sla_seconds >= 86400:
        return "batch"
    # 3. everything else (review replies, periodic summaries) -> standard
    return "standard"
 
def enqueue(redis, req):
    priority = classify_priority(req)
    payload = json.dumps(req.to_dict())
    redis.lpush(f"queue:{priority}:{req.tenant_id}", payload)
    redis.zincrby("tenant:enqueued_count", 1, req.tenant_id)

The worker, in its main loop, picks which queue to read from. The naive rule is "critical if any, else standard, else batch," but that starves batch whenever standard keeps refilling. To prevent starvation I added a minimum guarantee.

# worker.py — picking a priority on each loop iteration
import random
 
# weights: critical 70%, standard 25%, batch 5%
PRIORITY_WEIGHTS = [("critical", 70), ("standard", 25), ("batch", 5)]
 
def pick_priority(redis) -> str | None:
    # critical is absolutely preferred when non-empty
    if has_any(redis, "critical"):
        return "critical"
    # weighted random between standard and batch
    r = random.random() * 100
    cumulative = 0
    for name, w in PRIORITY_WEIGHTS:
        cumulative += w
        if r < cumulative:
            if has_any(redis, name):
                return name
    return None

The detail that matters here: critical is hard-preferred when non-empty, but standard vs batch is probabilistic. If you write "always pick standard before batch," batch starves indefinitely under any steady standard inflow — I made that mistake and froze a nightly bulk job for hours. Probabilistic share guarantees batch at least 5% of throughput in the worst case.

DRR (Deficit Round Robin) for tenant-level fairness

Splitting traffic by priority doesn't solve fairness within a priority. Inside the critical queue, if tenant A has 100 enqueued jobs and tenant B has 1, naive FIFO puts B at position 101. DRR (Deficit Round Robin) is the classic fix — give each tenant a quota that refills every round, and let them dequeue up to their accumulated quota.

# drr.py — Deficit Round Robin scheduler
class DRRScheduler:
    def __init__(self, redis, quantum: int = 5):
        self.redis = redis
        self.quantum = quantum  # quota refilled per round
        self.deficit: dict[str, int] = {}
 
    def pick(self, priority: str) -> tuple[str, bytes] | None:
        active = self._active_tenants(priority)
        if not active:
            return None
 
        for tenant_id in active:
            self.deficit.setdefault(tenant_id, 0)
            self.deficit[tenant_id] += self.quantum
 
            while self.deficit[tenant_id] >= 1:
                payload = self.redis.rpop(f"queue:{priority}:{tenant_id}")
                if payload is None:
                    break  # tenant out of work
                cost = self._estimate_cost(payload)
                if cost > self.deficit[tenant_id]:
                    # not enough quota; carry over to next round
                    self.redis.rpush(f"queue:{priority}:{tenant_id}", payload)
                    break
                self.deficit[tenant_id] -= cost
                return tenant_id, payload
        return None
 
    def _estimate_cost(self, payload: bytes) -> int:
        # cheap proxy: input_chars / 4000 ≈ estimated token cost
        req = json.loads(payload)
        return max(1, len(req["input"]) // 4000)

With this scheduler, tenant A's queue head stops dequeuing once their quota runs out, and tenant B picks up on the next round. Measured: in a scenario where tenant A dumped 100 jobs into critical, tenant B's single job went from p99 9.8s → 1.7s.

The quantum value needs production tuning. I started with quantum=1 and saw long-input jobs get stuck (deficit never grew enough to dequeue them). I settled on quantum=5 (about 20,000 input tokens per round) after watching real distributions.

Backpressure and 429 hygiene — respect Retry-After

Once priority and fairness are in place, you still have to deal with Anthropic-side rate limits. On my tier I'm capped at 50 requests/min, 40,000 input tokens/min, and 8,000 output tokens/min — exceed any of those and you get 429.

Naive immediate retries make 429 storms worse. What stabilized production for me was always reading Retry-After and sleeping 1.2× that value with ±20% jitter.

# call_with_backpressure.py
import asyncio, random
 
async def call_claude_with_backpressure(client, scheduler, payload, max_retry=4):
    for attempt in range(max_retry + 1):
        try:
            return await client.messages.create(**payload)
        except APIStatusError as e:
            if e.status_code != 429:
                raise
            retry_after = float(e.response.headers.get("retry-after", "1.0"))
            sleep_for = retry_after * 1.2 * (1.0 + (random.random() - 0.5) * 0.4)
            scheduler.record_throttle(payload["tenant_id"], retry_after)
            await asyncio.sleep(sleep_for)
    raise RuntimeError("max retry exceeded")

record_throttle is the part that matters. I log per-tenant 429 counts in a Redis sorted set keyed to a 60-second sliding window, and use it as one of the inputs to the auto-demotion logic in the next section. Attributing 429s to a specific tenant (rather than "the worker") breaks the cycle where one runaway tenant punishes everyone else.

Measured: switching from naive exponential backoff to jittered backoff cut my 429 re-occurrence rate from 38% to 7%. Jitter isn't cosmetic — without it, multiple workers retry on the same millisecond and trigger thundering herds against Anthropic.

Auto-demoting tenants who exceed budget

DRR keeps fairness in steady state, but if one tenant sustains over-budget input forever, they still erode standard's overall throughput. My rule: any tenant that triggers 5+ 429s in the last 60 seconds, OR exceeds 150% of their daily input-token budget, gets auto-demoted from standard to batch for five minutes.

# tenant_demotion.py
DEMOTION_THRESHOLDS = {
    "throttle_60s": 5,
    "daily_budget_ratio": 1.5,
}
 
def check_and_demote(redis, tenant_id: str) -> bool:
    now = time.time()
    redis.zremrangebyscore(f"throttle:{tenant_id}", 0, now - 60)
    throttle_count = redis.zcard(f"throttle:{tenant_id}")
    daily_tokens = int(redis.get(f"tokens:{tenant_id}:{today_key()}") or 0)
    budget = int(redis.get(f"budget:{tenant_id}") or 100000)
 
    if (throttle_count >= DEMOTION_THRESHOLDS["throttle_60s"] or
            daily_tokens > budget * DEMOTION_THRESHOLDS["daily_budget_ratio"]):
        redis.setex(f"demoted:{tenant_id}", 300, "1")  # 5-min batch demotion
        return True
    return False

When a tenant is in demoted state, classify_priority on the enqueue side rewrites their priority to batch regardless of what they requested. Demotion auto-expires after five minutes. It's less "punishment" and more "congestion control" — system self-defense, not tenant accountability.

A real-world example from my apps: a third-party review-aggregation service I had integrated started hammering the API far above their contract, and they (as a tenant) were starving review replies for my other apps. After auto-demotion shipped, any runaway tenant's impact on neighbors converges within five minutes. AdMob eCPM impact during such events stays inside about a 12% throughput dip rather than going to zero — comfortably inside what my revenue can absorb.

Operational metrics and SLOs — measure the tail

Once the scheduler and backpressure are running, you need to confirm they're actually working. The six metrics I now export to Prometheus / Grafana:

• claude_queue_depth{priority} — backlog size per queue (gauge, every 30s)
• claude_request_latency_seconds{priority} — end-to-end latency (histogram)
• claude_throttle_count{tenant_id} — per-tenant 429 count (rolling 60s)
• claude_demoted_tenants — current count of demoted tenants
• claude_drr_fairness_index — Jain's fairness index over the DRR scheduler
• claude_input_tokens_total{tenant_id} — cumulative input tokens per tenant

The two that matter most are p99 latency and Jain's fairness index. p50 always looks great — it's the tail where the actual user pain hides. My SLOs are: critical p99 ≤ 2.0s, standard p99 ≤ 6.0s, batch p99 ≤ 120s.

Jain's fairness index is computed as (Σx_i)² / (n × Σx_i²). It's 1.0 when every tenant gets identical service and approaches 1/n when one tenant dominates. Before my scheduler shipped, the index hovered at 0.42 with 8 tenants (one tenant was eating almost everything). After DRR it sits between 0.86–0.91. The metric is comparable across tenant-count changes, which makes it nicer for ongoing health checks than raw per-tenant share.

There's a risk in being "too fair," though. If every tenant gets exactly equal share, paying customers stop hitting the latency they're paying for, and the pricing tiers lose their meaning. I give paid tenants (Pro plan and up) a 3× DRR quantum, which intentionally keeps the fairness index closer to 0.85 than to 1.00.

Four gotchas surfaced over a year in production

That's the design. To close, four things I only learned by running this in front of real users:

1. Absolute priority on critical causes priority inversion

Hard-preferring critical makes workers sit idle waiting for the next critical job when one is "about to arrive," and total standard throughput drops. I switched from "critical always wins" to "critical wins with probability 0.9" per worker. That alone dropped standard p50 from 4.1s to 2.6s.

2. Synchronized quantum refills cause API microbursts

A naive DRR implementation refills every tenant's quantum at the same instant, and every tenant simultaneously dequeues a job — producing a microburst against Claude. I jitter the refill timestamp per tenant (hash-based, 0–1000ms spread). The standard deviation of per-second requests to the Anthropic endpoint dropped from 3.4 to 1.1.

3. Never trust the caller to label their own priority

The first design let callers pass priority="critical". As the codebase grew, "I'll just send it as critical" became the default style. Now callers only specify origin (user_action / scheduled / batch_job), and the scheduler classifies priority centrally. Put that responsibility in one place.

4. Model upgrades shift queue parameters

When Claude Haiku 4.5 shipped, long-job processing time dropped by 54% (median). Suddenly work I was demoting to batch could clear standard just fine, and my demotion thresholds were over-aggressive. Model changes shift the optimal queue parameters. I re-tune quantum and demotion thresholds once a quarter against a 30-day latency distribution.

A year in, the same Claude tier and the same worker count feel like a completely different product after the priority queue and DRR landed. The most concrete change isn't a graph — it's that I get paged from 3×/week to 1×/month. Design isn't only about user experience; it also affects how much its operator sleeps.

If you want to start, copy the code above and run with quantum=5 and a 5% batch guarantee for a week. Watch p99 and Jain's fairness index, and you'll quickly develop intuition for the right quantum against your own workload. If this saves you the same week-long detour I went through, that's exactly the audience I had in mind when writing it.