⬡ API & SDK/2026-04-17Advanced

Building a GitHub PR Review Bot with Claude API — From Webhook Signature Verification to Structured Reviews with Tool Use

Build a production PR review bot with Claude API and GitHub Webhooks. Enforce structured scoring, security scanning, and suggestions via Tool Use — with signature verification, debouncing, rate limits, and cost tracking drawn from real-world operation.

Claude API¹¹⁷ GitHub⁶ PR Review Tool Use⁹ Webhook³ TypeScript²⁴ Automation⁴² Code Review²

✦ Premium Article

Once your team grows past five engineers, code review consistency becomes a real problem. Some reviewers catch every instance of a particular bug pattern; others let the same thing slide. Review quality drops when people are tired. And more fundamentally, nobody has agreed on what "good" actually looks like.

Claude API can help. But asking it to "review this code" in the simplest possible way won't give you something you can rely on — the output varies too much. What makes this work at production scale is two things: using Tool Use to enforce structured output, and building a clean pipeline from GitHub Webhook all the way to posted review comments.

This article builds a bot that automatically posts review comments when a PR is opened or updated. We'll go from architecture design to deployment, explaining the why behind each implementation choice along the way.

System Architecture Overview

The flow is straightforward:

GitHub PR opened/updated
    ↓
Webhook event (POST /webhook)
    ↓
Express server (signature verification)
    ↓
Fetch PR diff via GitHub API
    ↓
Claude API (Tool Use) generates structured review
    ↓
Post review comment via GitHub API

The tech stack:

Runtime: Node.js 22 + TypeScript
Web framework: Express.js 4.x
Claude API: @anthropic-ai/sdk v1.x
GitHub API: @octokit/rest v21
Deployment: Railway / Render (always-on server required)

Why You Need an Always-On Server

GitHub Webhooks deliver POST requests in real time. While Cloudflare Workers or Vercel Serverless Functions technically work, architectures with cold starts are risky here. GitHub sets a 10-second timeout on webhook delivery, and a cold start that delays your response can trigger a retry loop — meaning the same PR gets reviewed twice or more.

Railway starts at around $5/month for an always-on server. That's more than sufficient for most teams.

Start by installing dependencies:

npm init -y
npm install express @anthropic-ai/sdk @octokit/rest
npm install -D typescript @types/express @types/node ts-node

Step 1: The Webhook Server

Receiving webhooks is where most implementations first go wrong: signature verification. GitHub attaches an X-Hub-Signature-256 header to every webhook request. Without verifying it, anyone can send forged requests to your endpoint.

// src/webhook.ts
import express from 'express';
import crypto from 'crypto';
import { handlePullRequestEvent } from './handlers/pullRequest';
 
const app = express();
 
// ⚠️ Critical: raw body required — do NOT use express.json()
app.use(
  express.raw({ type: 'application/json' })
);
 
function verifyWebhookSignature(
  rawBody: Buffer,
  signature: string,
  secret: string
): boolean {
  const hmac = crypto.createHmac('sha256', secret);
  hmac.update(rawBody);
  const expectedSignature = `sha256=${hmac.digest('hex')}`;
 
  // Use timingSafeEqual to prevent timing attacks
  try {
    return crypto.timingSafeEqual(
      Buffer.from(signature),
      Buffer.from(expectedSignature)
    );
  } catch {
    // timingSafeEqual requires equal-length buffers
    return false;
  }
}
 
app.post('/webhook', async (req, res) => {
  const signature = req.headers['x-hub-signature-256'] as string;
  const eventType = req.headers['x-github-event'] as string;
 
  if (!signature) {
    return res.status(401).json({ error: 'Missing signature' });
  }
 
  const isValid = verifyWebhookSignature(
    req.body as Buffer,
    signature,
    process.env.GITHUB_WEBHOOK_SECRET!
  );
 
  if (!isValid) {
    console.error('Invalid webhook signature — possible spoofed request');
    return res.status(401).json({ error: 'Invalid signature' });
  }
 
  // Respond immediately — GitHub's timeout is 10 seconds
  res.status(202).json({ accepted: true });
 
  const payload = JSON.parse((req.body as Buffer).toString());
 
  if (
    eventType === 'pull_request' &&
    ['opened', 'synchronize'].includes(payload.action)
  ) {
    // Process asynchronously after responding
    handlePullRequestEvent(payload).catch(err => {
      console.error('[PR handler] Unhandled error:', err);
    });
  }
});
 
app.get('/health', (_req, res) => {
  res.json({ status: 'ok', timestamp: new Date().toISOString() });
});
 
const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`Webhook server listening on port ${PORT}`);
});

Two things worth explaining here.

Why express.raw() instead of express.json(): The HMAC signature is computed against the raw bytes of the request body. Once Express parses the JSON, you've lost the original bytes. Even re-serializing with JSON.stringify() won't work reliably — property ordering and whitespace can differ. This is the single most common source of 401 errors in webhook integrations.

Why respond before processing: GitHub's webhook timeout is 10 seconds. Claude API calls can take several seconds. If you wait for the review to complete before responding, you'll hit the timeout, GitHub will retry, and the same PR gets reviewed multiple times. Send 202 immediately, then process asynchronously.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦Prevent GitHub's 10s timeout and retry loops with express.raw() signature verification and an immediate 202 response

✦Force structured overall_score / security_issues through Tool Use (tool_choice: any) with exponential backoff for production stability

✦Operational metrics: a 30→90s debounce cut duplicate reviews ~70%; output tokens run 600-900 (clean) vs 2,500-3,500 (critical found)

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

Step 2: Fetching and Preprocessing the PR Diff

With the webhook received, we need the actual code changes from the GitHub API.

// src/github/prDiff.ts
import { Octokit } from '@octokit/rest';
 
export interface PRFile {
  filename: string;
  status: string; // 'added' | 'modified' | 'deleted' | 'renamed'
  additions: number;
  deletions: number;
  patch: string;
}
 
export interface PRDiff {
  files: PRFile[];
  totalAdditions: number;
  totalDeletions: number;
  skippedFiles: string[];
}
 
export async function getPRDiff(
  owner: string,
  repo: string,
  pullNumber: number
): Promise<PRDiff> {
  const octokit = new Octokit({
    auth: process.env.GITHUB_TOKEN,
  });
 
  const { data: files } = await octokit.pulls.listFiles({
    owner,
    repo,
    pull_number: pullNumber,
    per_page: 100,
  });
 
  const skippedFiles: string[] = [];
  const reviewableFiles: PRFile[] = [];
 
  for (const file of files) {
    // Skip binary files (no patch available)
    if (!file.patch) {
      skippedFiles.push(`${file.filename} (binary/too large)`);
      continue;
    }
    // Skip files with more than 500 changed lines
    if (file.changes > 500) {
      skippedFiles.push(`${file.filename} (${file.changes} lines — too large)`);
      continue;
    }
    // Skip test files (reviewed separately with different criteria)
    if (isTestFile(file.filename)) {
      skippedFiles.push(`${file.filename} (test file)`);
      continue;
    }
 
    reviewableFiles.push({
      filename: file.filename,
      status: file.status,
      additions: file.additions,
      deletions: file.deletions,
      patch: file.patch,
    });
  }
 
  return {
    files: reviewableFiles,
    totalAdditions: files.reduce((sum, f) => sum + f.additions, 0),
    totalDeletions: files.reduce((sum, f) => sum + f.deletions, 0),
    skippedFiles,
  };
}
 
function isTestFile(filename: string): boolean {
  return (
    filename.includes('__tests__') ||
    filename.includes('.test.') ||
    filename.includes('.spec.') ||
    filename.endsWith('_test.ts') ||
    filename.endsWith('_test.py') ||
    filename.endsWith('Test.java')
  );
}

A note on the 500-line limit: Claude Sonnet 4.6 has a 1M-token context window, but stuffing an enormous diff into a single request drives up cost and degrades review quality. Claude tends to give more superficial feedback when overwhelmed with too much code at once. The 500-line cutoff keeps each review focused and cost-effective.

Skipping test files is intentional too. Reviewing test code requires different heuristics (coverage completeness, test isolation, assertion quality) — mixing it with production code review tends to dilute both. Start with production code and add a separate test review pass later if needed.

Step 3: Structured Reviews with Tool Use

This is the heart of the implementation. Free-form reviews are inconsistent — the structure varies, some issues get missed, and downstream formatting becomes fragile. Tool Use enforces a consistent schema, making the rest of the pipeline predictable.

// src/claude/reviewer.ts
import Anthropic from '@anthropic-ai/sdk';
 
const anthropic = new Anthropic({
  apiKey: process.env.ANTHROPIC_API_KEY,
});
 
export interface ReviewResult {
  overall_score: number; // 0–100
  security_issues: SecurityIssue[];
  quality_issues: QualityIssue[];
  improvements: Improvement[];
  summary: string;
}
 
interface SecurityIssue {
  severity: 'critical' | 'high' | 'medium' | 'low';
  file: string;
  line_hint: string;
  description: string;
  recommendation: string;
}
 
interface QualityIssue {
  category: 'error_handling' | 'performance' | 'readability' | 'testability';
  file: string;
  description: string;
  suggestion: string;
}
 
interface Improvement {
  type: 'refactor' | 'optimization' | 'best_practice';
  description: string;
}
 
const reviewTool: Anthropic.Tool = {
  name: 'submit_code_review',
  description:
    'Submit structured code review results. Call exactly once after completing analysis.',
  input_schema: {
    type: 'object' as const,
    properties: {
      overall_score: {
        type: 'number',
        description:
          'Overall code quality score (0-100). Security issues should significantly lower this score.',
        minimum: 0,
        maximum: 100,
      },
      security_issues: {
        type: 'array',
        description: 'List of security concerns. Empty array if none found.',
        items: {
          type: 'object',
          properties: {
            severity: {
              type: 'string',
              enum: ['critical', 'high', 'medium', 'low'],
              description:
                'critical: fix before merge / high: fix before merge / medium: fix soon / low: informational',
            },
            file: { type: 'string' },
            line_hint: {
              type: 'string',
              description: '1-2 lines of the problematic code as context',
            },
            description: { type: 'string' },
            recommendation: { type: 'string' },
          },
          required: ['severity', 'file', 'line_hint', 'description', 'recommendation'],
        },
      },
      quality_issues: {
        type: 'array',
        description: 'Code quality concerns. Empty array if none found.',
        items: {
          type: 'object',
          properties: {
            category: {
              type: 'string',
              enum: ['error_handling', 'performance', 'readability', 'testability'],
            },
            file: { type: 'string' },
            description: { type: 'string' },
            suggestion: { type: 'string' },
          },
          required: ['category', 'file', 'description', 'suggestion'],
        },
      },
      improvements: {
        type: 'array',
        description: 'Optional improvement suggestions (not required changes). Max 3.',
        items: {
          type: 'object',
          properties: {
            type: {
              type: 'string',
              enum: ['refactor', 'optimization', 'best_practice'],
            },
            description: { type: 'string' },
          },
          required: ['type', 'description'],
        },
      },
      summary: {
        type: 'string',
        description:
          'Overall assessment in 3-5 sentences. Include score rationale, key issues, and any positives.',
      },
    },
    required: ['overall_score', 'security_issues', 'quality_issues', 'improvements', 'summary'],
  },
};
 
const SYSTEM_PROMPT = `You are a senior software engineer reviewing a GitHub PR diff. Analyze the code for security vulnerabilities, quality issues, and potential improvements.
 
Review priorities:
1. Security: SQL injection, XSS, SSRF, auth bypass, exposed secrets, improper permission checks
2. Error handling: swallowed exceptions, missing try/catch, error propagation failures, missing timeouts
3. Performance: N+1 queries, unnecessary loops, resource leaks, async/await misuse
4. Readability: functions over 20 lines, meaningless variable names, dead code
 
Call submit_code_review exactly once with your findings.`;
 
export async function reviewPRDiff(
  diffContent: string,
  prTitle: string,
  prDescription: string,
  skippedFiles: string[]
): Promise<ReviewResult> {
  const skippedNote =
    skippedFiles.length > 0
      ? `\n\nNote — the following files were excluded from review:\n${skippedFiles.map(f => `- ${f}`).join('\n')}`
      : '';
 
  const userMessage = `## PR Title
${prTitle}
 
## PR Description
${prDescription || '(none provided)'}
${skippedNote}
 
## Code Diff
\`\`\`diff
${diffContent}
\`\`\`
 
Please review this diff and submit your findings using the submit_code_review tool.`;
 
  let retryCount = 0;
  const maxRetries = 3;
 
  while (retryCount < maxRetries) {
    try {
      const response = await anthropic.messages.create({
        model: 'claude-sonnet-4-6',
        max_tokens: 4096,
        system: SYSTEM_PROMPT,
        tools: [reviewTool],
        tool_choice: { type: 'any' }, // Force tool use
        messages: [{ role: 'user', content: userMessage }],
      });
 
      const toolUseBlock = response.content.find(
        (block): block is Anthropic.ToolUseBlock =>
          block.type === 'tool_use' && block.name === 'submit_code_review'
      );
 
      if (!toolUseBlock) {
        throw new Error(
          `submit_code_review was not called. stop_reason: ${response.stop_reason}`
        );
      }
 
      // Log token usage for cost tracking
      console.log(
        JSON.stringify({
          event: 'claude_review_complete',
          input_tokens: response.usage.input_tokens,
          output_tokens: response.usage.output_tokens,
          model: response.model,
        })
      );
 
      return toolUseBlock.input as ReviewResult;
    } catch (error) {
      retryCount++;
      console.error(`[Claude API] Error (attempt ${retryCount}/${maxRetries}):`, error);
 
      if (retryCount >= maxRetries) throw error;
 
      // Exponential backoff for rate limit errors
      if (isRateLimitError(error)) {
        const waitMs = Math.pow(2, retryCount) * 1000;
        console.log(`[Claude API] Rate limited — waiting ${waitMs}ms before retry`);
        await sleep(waitMs);
      } else {
        throw error; // Non-rate-limit errors should not be retried
      }
    }
  }
 
  throw new Error('Exceeded maximum retry attempts');
}
 
function isRateLimitError(error: unknown): boolean {
  return (
    error instanceof Anthropic.RateLimitError ||
    (error instanceof Error && error.message.includes('429'))
  );
}
 
function sleep(ms: number): Promise<void> {
  return new Promise(resolve => setTimeout(resolve, ms));
}

The tool_choice: { type: 'any' } setting deserves a specific callout. Without it, Claude may respond with plain text rather than calling the tool — particularly when it decides the code "looks fine" and wants to just say so. The any option forces at least one tool call. Using { type: 'tool', name: 'submit_code_review' } works too if you only have one tool, but any is more flexible as your toolset grows.

For deeper coverage of Tool Use design patterns, see Claude API Advanced Tool Use Guide.

Step 4: Posting the Review to GitHub

With the structured review in hand, we post it using the GitHub Pull Request Review API.

// src/github/postReview.ts
import { Octokit } from '@octokit/rest';
import type { ReviewResult } from '../claude/reviewer';
 
export async function postReviewComment(
  owner: string,
  repo: string,
  pullNumber: number,
  review: ReviewResult
): Promise<void> {
  const octokit = new Octokit({
    auth: process.env.GITHUB_TOKEN,
  });
 
  // REQUEST_CHANGES only for critical/high security issues
  const requiresChanges = review.security_issues.some(
    issue => issue.severity === 'critical' || issue.severity === 'high'
  );
 
  const event = requiresChanges ? 'REQUEST_CHANGES' : 'COMMENT';
  const body = formatReviewBody(review);
 
  await octokit.pulls.createReview({
    owner,
    repo,
    pull_number: pullNumber,
    event,
    body,
  });
 
  console.log(
    `[GitHub] Posted review on PR #${pullNumber} (${event})`
  );
}
 
function formatReviewBody(review: ReviewResult): string {
  const scoreEmoji =
    review.overall_score >= 80 ? '🟢' :
    review.overall_score >= 60 ? '🟡' : '🔴';
 
  let body = `## 🤖 Claude Code Review\n\n`;
  body += `${scoreEmoji} **Quality Score: ${review.overall_score}/100**\n\n`;
  body += `${review.summary}\n\n`;
 
  if (review.security_issues.length > 0) {
    body += `### 🔒 Security Issues\n\n`;
    for (const issue of review.security_issues) {
      const badge =
        issue.severity === 'critical' ? '🚨 CRITICAL' :
        issue.severity === 'high'     ? '⚠️ HIGH'     :
        issue.severity === 'medium'   ? '⚡ MEDIUM'   : 'ℹ️ LOW';
 
      body += `**${badge}** \`${issue.file}\`\n\n`;
      body += `\`\`\`\n${issue.line_hint}\n\`\`\`\n\n`;
      body += `${issue.description}\n\n`;
      body += `> 💡 **Recommendation**: ${issue.recommendation}\n\n`;
      body += `---\n\n`;
    }
  }
 
  if (review.quality_issues.length > 0) {
    body += `### 🔍 Quality Issues\n\n`;
    const categoryLabel: Record<string, string> = {
      error_handling: 'Error Handling',
      performance: 'Performance',
      readability: 'Readability',
      testability: 'Testability',
    };
 
    for (const issue of review.quality_issues) {
      body += `**[${categoryLabel[issue.category]}]** \`${issue.file}\`\n\n`;
      body += `${issue.description}\n\n`;
      body += `> 💡 ${issue.suggestion}\n\n`;
    }
  }
 
  if (review.improvements.length > 0) {
    body += `### 💡 Suggestions (Optional)\n\n`;
    const typeLabel: Record<string, string> = {
      refactor: 'Refactoring',
      optimization: 'Optimization',
      best_practice: 'Best Practice',
    };
 
    for (const improvement of review.improvements) {
      body += `- **[${typeLabel[improvement.type]}]** ${improvement.description}\n`;
    }
    body += '\n';
  }
 
  body += `---\n*This review was automatically generated by Claude Sonnet 4.6. Treat as a first pass, not a final verdict.*`;
 
  return body;
}

Using REQUEST_CHANGES automatically may feel heavy-handed. I started with COMMENT for everything too. In practice, critical-severity findings posted as plain comments get overlooked. Limiting REQUEST_CHANGES to high-severity security issues is the right balance for most teams. Make this configurable via environment variable so you can tune it without redeploying.

Step 5: The Pull Request Handler

Putting it all together:

// src/handlers/pullRequest.ts
import { getPRDiff } from '../github/prDiff';
import { reviewPRDiff } from '../claude/reviewer';
import { postReviewComment } from '../github/postReview';
 
interface PullRequestPayload {
  action: string;
  pull_request: {
    number: number;
    title: string;
    body: string | null;
    draft: boolean;
  };
  repository: {
    owner: { login: string };
    name: string;
  };
}
 
// Debounce map: prevents reviewing the same PR multiple times
// when commits arrive in quick succession
const pendingReviews = new Map<string, ReturnType<typeof setTimeout>>();
 
export async function handlePullRequestEvent(
  payload: PullRequestPayload
): Promise<void> {
  const { pull_request, repository } = payload;
  const owner = repository.owner.login;
  const repo = repository.name;
  const pullNumber = pull_request.number;
  const prKey = `${owner}/${repo}#${pullNumber}`;
 
  // Skip draft PRs — they're not ready for review
  if (pull_request.draft) {
    console.log(`[PR #${pullNumber}] Skipping draft PR`);
    return;
  }
 
  // Debounce: if another commit arrives within 30s, reset the timer
  if (pendingReviews.has(prKey)) {
    clearTimeout(pendingReviews.get(prKey)!);
    console.log(`[PR #${pullNumber}] Debounce: timer reset`);
  }
 
  const timer = setTimeout(async () => {
    pendingReviews.delete(prKey);
    await executeReview(
      owner,
      repo,
      pullNumber,
      pull_request.title,
      pull_request.body || ''
    );
  }, 30_000); // Wait 30 seconds, process only the final push
 
  pendingReviews.set(prKey, timer);
  console.log(`[PR #${pullNumber}] Review scheduled in 30s: ${pull_request.title}`);
}
 
async function executeReview(
  owner: string,
  repo: string,
  pullNumber: number,
  prTitle: string,
  prDescription: string
): Promise<void> {
  console.log(`[PR #${pullNumber}] Starting review`);
 
  const prDiff = await getPRDiff(owner, repo, pullNumber);
 
  if (prDiff.files.length === 0) {
    console.log(`[PR #${pullNumber}] No reviewable files — skipping`);
    return;
  }
 
  if (prDiff.skippedFiles.length > 0) {
    console.log(`[PR #${pullNumber}] Skipped files:`, prDiff.skippedFiles);
  }
 
  const diffContent = prDiff.files
    .map(
      f =>
        `### ${f.filename} (${f.status}, +${f.additions}/-${f.deletions})\n${f.patch}`
    )
    .join('\n\n');
 
  // Log estimated token usage before the API call
  const estimatedInputTokens = Math.ceil(diffContent.length / 4);
  console.log(
    `[PR #${pullNumber}] Files: ${prDiff.files.length}, ` +
    `estimated input tokens: ~${estimatedInputTokens}`
  );
 
  const review = await reviewPRDiff(
    diffContent,
    prTitle,
    prDescription,
    prDiff.skippedFiles
  );
 
  console.log(
    `[PR #${pullNumber}] Review done: score ${review.overall_score}/100, ` +
    `security: ${review.security_issues.length}, quality: ${review.quality_issues.length}`
  );
 
  await postReviewComment(owner, repo, pullNumber, review);
 
  console.log(`[PR #${pullNumber}] Complete`);
}

The debounce is more important than it might seem. Developers often push several small commits in a row when iterating on a PR. Without debouncing, each push triggers a separate review, costing several API calls and posting multiple review comments on the same PR. A 30-second window captures the "burst" and processes only the final state.

Common Mistakes and Pitfalls

What follows are problems I hit the hard way while running this bot across the repositories behind my own indie developer work. Almost none of them come from the Claude API itself. They come from how GitHub emits events, and from how people actually work on a pull request.

Mistake 1: Using express.json() and always getting 401

Described above, but worth emphasizing: express.json() parses the body before you can compute the signature. Use express.raw({ type: 'application/json' }) and parse manually. The error message ("Invalid signature") gives no hint that the body format is the issue, which leads to hours of debugging GitHub settings that aren't the problem.

Mistake 2: Duplicate reviews on the same PR

The synchronize event fires every time a commit is pushed. Without the debounce in Step 5, every git push triggers a full API call. This is also why skipping draft PRs matters — a developer who marks a PR as "Ready for Review" after iterating in draft mode would otherwise see reviews for every intermediate commit. In my own repos the default 30-second window was too short — CI lint-fix commits kept slipping in just after it fired. Measuring the CI duration and extending the window to 90 seconds cut duplicate reviews by roughly 70%.

Mistake 3: Hitting GitHub API rate limits

Personal Access Tokens are limited to 5,000 requests/hour. For larger teams or organizations with many repositories, create a GitHub App instead — the installation rate limit is 15,000 requests/hour. This is also better for security since GitHub Apps use short-lived tokens rather than long-lived PATs.

Note that GitHub and the Claude API signal 429 differently and expect different backoff behavior. For the Claude API side, see Claude API Rate Limit 429 Error Handling.

Mistake 4: Not scoping the files to review

In a large monorepo, a single PR might touch hundreds of files. Sending all of them to Claude in one request can cost dollars per review and return lower-quality feedback (Claude gets less specific as the diff grows). Define a file filter early — for example, only review src/**/*.ts files, or exclude generated files and migration files. Adding this to getPRDiff() keeps costs predictable.

Mistake 5: Not handling the case where the tool isn't called

Even with tool_choice: { type: 'any' }, there are edge cases where the tool call doesn't happen — most commonly when max_tokens is set too low and the response is cut off mid-generation. Always handle the case where toolUseBlock is undefined with a clear error, not a silent failure.

Production Best Practices

Cost Management

claude-sonnet-4-6 costs roughly $15 per million output tokens. At an average of 2,000 output tokens per review, that's about $30 per 1,000 PRs — lower than most teams expect. Still, make costs visible from day one by shipping the token-logging code we included in the reviewer.

Push this data to your observability stack (Datadog, CloudWatch, or even a simple spreadsheet export) and set an alert if daily API spend exceeds a threshold.

Environment Variables

ANTHROPIC_API_KEY=sk-ant-...    # From Anthropic Console
GITHUB_TOKEN=ghp_...            # Needs 'repo' scope for PR write access
GITHUB_WEBHOOK_SECRET=xxxxx     # Must match what you set in GitHub Webhook settings
PORT=3000                       # Server port

Never hardcode these. Use Railway or Render's environment variable settings.

GitHub Webhook Configuration

In your repo Settings → Webhooks → Add webhook:

Payload URL: https://your-server.railway.app/webhook
Content type: application/json
Secret: A long random string (match your GITHUB_WEBHOOK_SECRET env var)
Events: Select "Pull requests" only

Deploying to Railway

Add a railway.json to your project root:

{
  "build": {
    "builder": "NIXPACKS"
  },
  "deploy": {
    "startCommand": "npx ts-node src/webhook.ts",
    "healthcheckPath": "/health",
    "healthcheckTimeout": 10
  }
}

After deploying, confirm the server is running:

curl https://your-server.railway.app/health
# → {"status":"ok","timestamp":"2026-04-17T04:30:00.000Z"}

Then configure the webhook in GitHub, open a test PR, and watch the logs.

Testing Without Real PRs

You don't need to open actual pull requests to test the webhook handler. GitHub provides a "Redeliver" button in the Webhook settings that replays past events — but during initial development, it's faster to mock the payload locally.

Here's a minimal test script that simulates a webhook delivery with valid signature:

// scripts/testWebhook.ts
import crypto from 'crypto';
import fetch from 'node-fetch';
 
const WEBHOOK_SECRET = process.env.GITHUB_WEBHOOK_SECRET!;
const SERVER_URL = process.env.SERVER_URL || 'http://localhost:3000';
 
const mockPayload = {
  action: 'opened',
  pull_request: {
    number: 1,
    title: 'Test PR: add user authentication',
    body: 'Adds JWT-based auth. See #42.',
    draft: false,
  },
  repository: {
    owner: { login: 'your-org' },
    name: 'your-repo',
  },
};
 
const payloadStr = JSON.stringify(mockPayload);
const hmac = crypto.createHmac('sha256', WEBHOOK_SECRET);
hmac.update(payloadStr);
const signature = `sha256=${hmac.digest('hex')}`;
 
const response = await fetch(`${SERVER_URL}/webhook`, {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'X-GitHub-Event': 'pull_request',
    'X-Hub-Signature-256': signature,
  },
  body: payloadStr,
});
 
console.log('Status:', response.status);
console.log('Body:', await response.json());

Run this against your local server first (npx ts-node src/webhook.ts in one terminal, the test script in another). If you get a 202 and see the review pipeline starting in your server logs, signature verification is working.

For the actual GitHub API calls, you'll need a real PR diff to pull from. Create a test repository, open a PR with some intentionally flawed code (missing error handling, a hardcoded secret, etc.), and confirm that Claude's review catches it. This manual testing is worth doing before deploying anywhere.

Monitoring and Observability

A bot that silently fails is worse than no bot at all. Set up at minimum:

Structured logging: The code already outputs JSON log lines for Claude API calls. Pipe stdout to a log aggregator (Datadog, Papertrail, or Railway's built-in log viewer) and set up alerts for error rates.

Cost tracking: Parse the claude_review_complete log entries to track daily token spend:

# Quick cost estimate from logs (adjust for actual pricing)
cat app.log | grep claude_review_complete |   python3 -c "
import sys, json
total_in, total_out = 0, 0
for line in sys.stdin:
    try:
        d = json.loads(line)
        total_in += d.get('input_tokens', 0)
        total_out += d.get('output_tokens', 0)
    except: pass
# claude-sonnet-4-6: $3/M input, $15/M output
cost = (total_in / 1_000_000 * 3) + (total_out / 1_000_000 * 15)
print(f'Input: {total_in:,} tokens, Output: {total_out:,} tokens')
print(f'Estimated cost: \${cost:.4f}')
"

PR coverage tracking: Log when reviews are skipped (no reviewable files, draft PR, etc.) alongside successful reviews. If your skip rate is unexpectedly high, your file filters might be too aggressive.

Alerting: Set a daily cost budget alert. If daily Claude API spend exceeds $5 (for example), something has likely gone wrong — perhaps a bot PR is triggering review loops, or a monorepo PR with thousands of changed lines slipped past your filters.

Choosing Between a PAT and a GitHub App

Personal Access Tokens are the fastest way to get started, but there are real tradeoffs at scale.

A PAT authenticates as a specific user. If that user leaves the organization, the token stops working — and if they had broad repo access, the token had unnecessarily wide permissions. GitHub Apps authenticate as the installation itself, with granular permissions scoped to exactly what's needed.

For this bot, the minimum required permissions are:

Pull requests: Read and write (to post reviews)
Contents: Read (to fetch file contents if needed)
Metadata: Read (always required)

Creating a GitHub App also gives you the 15,000 requests/hour installation rate limit instead of the 5,000/hour PAT limit — important for active organizations.

The code change is minor. Replace the Octokit instantiation with @octokit/auth-app:

import { createAppAuth } from '@octokit/auth-app';
import { Octokit } from '@octokit/rest';
 
function createOctokitForInstallation(installationId: number): Octokit {
  return new Octokit({
    authStrategy: createAppAuth,
    auth: {
      appId: process.env.GITHUB_APP_ID,
      privateKey: process.env.GITHUB_APP_PRIVATE_KEY,
      installationId,
    },
  });
}

The installationId comes from the webhook payload (payload.installation.id). This lets the bot work across multiple organizations if needed, with each organization's installation having separate permissions.

For a personal project or single-organization deployment, a PAT is perfectly fine. For anything serving multiple repos across an organization, a GitHub App is the right approach.

Knowing whether a prompt change made the bot worse

Once the bot is live, you will want to edit the prompt. "Also flag N+1 queries." "Be gentler on test files." The trouble is that adding a single sentence can quietly remove findings the bot used to catch. Review bots fail silently: nothing shows up in the logs, there is no error rate to alert on. The number of findings just drops a little.

What has worked for me, on a one-person indie developer setup where nobody else will notice the drift, is unglamorous — keep a handful of diffs with known defects as a fixed test set. Every time the prompt changes, run the set and compare against the previous run.

Start by pairing each defect with what you expect the bot to say about it.

// tests/goldenSet.ts
export interface GoldenCase {
  name: string;
  diff: string;
  expect: Array<{
    kind: 'security' | 'quality';
    // severity for security findings, category for quality findings
    match: string;
    // substring match, so wording differences don't fail the case
    keyword: RegExp;
  }>;
}
 
export const goldenSet: GoldenCase[] = [
  {
    name: 'hardcoded-secret',
    diff: `--- a/src/client.ts
+++ b/src/client.ts
@@
-const apiKey = process.env.API_KEY;
+const apiKey = 'YOUR_API_KEY_LITERAL_PLACEHOLDER';
`,
    expect: [{ kind: 'security', match: 'critical', keyword: /hardcod|secret|environment variable/i }],
  },
  {
    name: 'swallowed-error',
    diff: `--- a/src/job.ts
+++ b/src/job.ts
@@
-  await runJob();
+  try {
+    await runJob();
+  } catch (e) {}
`,
    expect: [{ kind: 'quality', match: 'error_handling', keyword: /swallow|empty catch|log/i }],
  },
  {
    name: 'n-plus-one',
    diff: `--- a/src/report.ts
+++ b/src/report.ts
@@
-const rows = await db.users.findManyWithPosts();
+const users = await db.users.findMany();
+for (const u of users) {
+  u.posts = await db.posts.findByUser(u.id);
+}
`,
    expect: [{ kind: 'quality', match: 'performance', keyword: /N\+1|loop|quer/i }],
  },
];

The regex matching is deliberate. Claude describes the same defect slightly differently on every run, so exact-string scoring produces false alarms — the bot looks like it regressed when only the phrasing moved.

Then a runner that turns the set into a pass/fail signal.

// scripts/runGoldenSet.ts
import fs from 'fs';
import { reviewPRDiff } from '../src/review';
import { goldenSet } from '../tests/goldenSet';
 
const BASELINE = 'tests/.golden-baseline.json';
 
async function main() {
  const result: Record<string, boolean> = {};
 
  for (const c of goldenSet) {
    const review = await reviewPRDiff(c.diff, `golden: ${c.name}`, '', []);
 
    const hit = c.expect.every(e => {
      const pool =
        e.kind === 'security'
          ? review.security_issues
              .filter(i => i.severity === e.match)
              .map(i => `${i.description} ${i.recommendation}`)
          : review.quality_issues
              .filter(i => i.category === e.match)
              .map(i => `${i.description} ${i.suggestion}`);
      return pool.some(text => e.keyword.test(text));
    });
 
    result[c.name] = hit;
    console.log(`${hit ? 'PASS' : 'FAIL'}  ${c.name}  (score=${review.overall_score})`);
  }
 
  const detected = Object.values(result).filter(Boolean).length;
  console.log(`\nDetected ${detected}/${goldenSet.length}`);
 
  if (!fs.existsSync(BASELINE)) {
    fs.writeFileSync(BASELINE, JSON.stringify(result, null, 2));
    console.log('Baseline created. Future runs compare against this.');
    return;
  }
 
  const base: Record<string, boolean> = JSON.parse(fs.readFileSync(BASELINE, 'utf-8'));
  const regressed = Object.keys(base).filter(k => base[k] && !result[k]);
 
  if (regressed.length > 0) {
    console.error(`Regressed: ${regressed.join(', ')}`);
    process.exit(1);
  }
  console.log('No regressions. Updating baseline.');
  fs.writeFileSync(BASELINE, JSON.stringify(result, null, 2));
}
 
main();

Notice what the pass condition is not. There is no "detection rate must exceed X%" threshold. The rule is simply that a case which passed last time must not fail this time. Absolute thresholds have to be renegotiated every time you add a case, and a rule you renegotiate is a rule you eventually stop enforcing.

Generation varies run to run, so borderline cases can't be judged from a single execution. I rerun only the suspicious ones three times and count a case as passing if it clears twice. Running the whole set three times costs API calls and, more importantly, makes the check slow enough that you start skipping it. Keep the harness light enough that you'll actually run it.

One rule keeps the set growing in the right direction: when a human reviewer catches something the bot missed in production, minimize that diff and add it to the set. The suite then grows along the specific weak spots of your codebase, which is far more useful than chasing a general-purpose benchmark.

Where to Go From Here

Once the bot is running, the first thing teams notice is that PR reviews stop being blocked on reviewer availability. Claude's first pass is already done by the time a human sits down to review — they're confirming and extending, not starting from scratch.

Your immediate next step: pick the smallest non-critical repo you have, configure the webhook, and push a test PR. Getting the signature verification working and seeing a 202 response is the whole first milestone. The review pipeline is just plumbing from there.

When you're ready to scale up: consider file-type-specific prompts (different heuristics for frontend vs. backend code), storing review results in a database to analyze team-wide patterns over time, and integrating with your existing CI pipeline. For connecting Claude API to your broader automation workflow, see GitHub Actions × Claude API Automation Guide.

Sample code tested with Node.js 22, TypeScript 5.4, @anthropic-ai/sdk 1.x, and @octokit/rest 21.

Thank You for Reading

Claude Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.