Webhooks power real-time integrations across the modern web—from payment notifications and GitHub deployments to IoT device updates and SaaS platform events. This complete guide covers everything you need to build reliable, secure, and scalable webhook systems, whether you're consuming webhooks from providers or building your own webhook infrastructure.
Webhook Architecture Fundamentals
Webhooks invert the traditional request-response model. Instead of polling for changes, events are pushed to your server in real-time when they occur.
┌─────────────────────────────────────────────────────────────────────┐
│ Webhook vs Polling Architecture │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ POLLING (Traditional) │
│ ═════════════════════ │
│ │
│ Your Server Provider API │
│ │ │ │
│ │──── GET /events? ─────▶│ │
│ │◀─── No new events ─────│ │
│ │ │ │
│ │──── GET /events? ─────▶│ (repeat every N seconds) │
│ │◀─── No new events ─────│ │
│ │ │ │
│ │──── GET /events? ─────▶│ │
│ │◀─── [Event data!] ─────│ (finally something) │
│ │ │ │
│ │
│ Problems: Wasted requests, delayed detection, server load │
│ │
│ ═══════════════════════════════════════════════════════════════ │
│ │
│ WEBHOOKS (Event-Driven) │
│ ═══════════════════════ │
│ │
│ Your Server Provider System │
│ │ │ │
│ │ │──── Event occurs! │
│ │◀── POST /webhook ──────│ │
│ │─── 200 OK ────────────▶│ │
│ │ │ │
│ │ │──── Another event! │
│ │◀── POST /webhook ──────│ │
│ │─── 200 OK ────────────▶│ │
│ │ │ │
│ │
│ Benefits: Real-time, efficient, scalable, lower latency │
│ │
└─────────────────────────────────────────────────────────────────────┘
Core Webhook Components
┌─────────────────────────────────────────────────────────────────────┐
│ Webhook System Components │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ WEBHOOK PROVIDER │ │
│ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────┐ │ │
│ │ │ Event │──▶│ Delivery │──▶│ HTTP Client │ │ │
│ │ │ Source │ │ Queue │ │ (with retries) │ │ │
│ │ └─────────────┘ └─────────────┘ └────────┬────────┘ │ │
│ │ │ │ │
│ └───────────────────────────────────────────────┼──────────────┘ │
│ │ │
│ ┌────────────────────────────┘ │
│ │ HTTPS POST + Signature │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ WEBHOOK CONSUMER │ │
│ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────┐ │ │
│ │ │ Endpoint │──▶│ Signature │──▶│ Event Queue │ │ │
│ │ │ Handler │ │ Validator │ │ (async proc) │ │ │
│ │ └─────────────┘ └─────────────┘ └────────┬────────┘ │ │
│ │ │ │ │ │
│ │ │ Return 200 │ │ │
│ │ ▼ ▼ │ │
│ │ ┌─────────────┐ ┌─────────────────┐ │ │
│ │ │ Response │ │ Background │ │ │
│ │ │ (fast!) │ │ Workers │ │ │
│ │ └─────────────┘ └─────────────────┘ │ │
│ │ │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
Guide Directory
This hub connects to detailed guides covering every aspect of webhook development.
Fundamentals & Concepts
| Guide | Description |
|---|---|
| Webhooks Explained | Core concepts, terminology, use cases |
| Webhook Best Practices | Production-ready patterns and anti-patterns |
Security
| Guide | Description |
|---|---|
| Webhook Security Guide | Comprehensive security model |
| Signature Verification | HMAC signing and verification |
| Security Implementation | Step-by-step security workflow |
Reliability & Operations
| Guide | Description |
|---|---|
| Retry Logic | Exponential backoff and reliability |
| Testing & Debugging | Local testing, debugging strategies |
| Error Handling | Dead letter queues, recovery patterns |
Advanced Topics
| Guide | Description |
|---|---|
| Scaling & Performance | High-volume webhook processing |
| Platform Integrations | Stripe, GitHub, Slack patterns |
| Building Providers | Design your own webhook system |
Quick Start: Consuming Webhooks
Get started quickly with a production-ready webhook handler pattern.
// Express.js webhook handler with best practices
import express from 'express';
import crypto from 'crypto';
const app = express();
// Use raw body for signature verification
app.use('/webhooks', express.raw({ type: 'application/json' }));
// Webhook endpoint
app.post('/webhooks/stripe', async (req, res) => {
const startTime = Date.now();
const requestId = crypto.randomUUID();
try {
// 1. Verify signature FIRST
const signature = req.headers['stripe-signature'];
const webhookSecret = process.env.STRIPE_WEBHOOK_SECRET!;
const event = verifyStripeSignature(req.body, signature, webhookSecret);
// 2. Check for duplicates (idempotency)
if (await isDuplicateEvent(event.id)) {
console.log(`Duplicate event ${event.id}, skipping`);
return res.status(200).json({ received: true, duplicate: true });
}
// 3. Mark event as received
await markEventReceived(event.id);
// 4. Queue for async processing
await eventQueue.add('stripe-webhook', {
eventId: event.id,
eventType: event.type,
payload: event,
receivedAt: new Date().toISOString(),
requestId
});
// 5. Log and respond quickly
console.log({
requestId,
eventId: event.id,
eventType: event.type,
processingTime: Date.now() - startTime
});
return res.status(200).json({ received: true });
} catch (error) {
if (error.message === 'Invalid signature') {
console.error({ requestId, error: 'Invalid signature', ip: req.ip });
return res.status(401).json({ error: 'Invalid signature' });
}
console.error({ requestId, error: error.message });
return res.status(500).json({ error: 'Internal error' });
}
});
// Signature verification
function verifyStripeSignature(
payload: Buffer,
signatureHeader: string,
secret: string
): any {
const parts = signatureHeader.split(',');
const timestamp = parts.find(p => p.startsWith('t='))?.split('=')[1];
const signature = parts.find(p => p.startsWith('v1='))?.split('=')[1];
if (!timestamp || !signature) {
throw new Error('Invalid signature');
}
// Prevent replay attacks - reject old events
const currentTime = Math.floor(Date.now() / 1000);
if (currentTime - parseInt(timestamp) > 300) { // 5 minutes
throw new Error('Invalid signature'); // Timestamp too old
}
// Compute expected signature
const signedPayload = `${timestamp}.${payload.toString()}`;
const expectedSignature = crypto
.createHmac('sha256', secret)
.update(signedPayload)
.digest('hex');
// Timing-safe comparison
if (!crypto.timingSafeEqual(
Buffer.from(signature),
Buffer.from(expectedSignature)
)) {
throw new Error('Invalid signature');
}
return JSON.parse(payload.toString());
}
// Idempotency check
async function isDuplicateEvent(eventId: string): Promise<boolean> {
// Check Redis/database for existing event
const exists = await redis.exists(`webhook:event:${eventId}`);
return exists === 1;
}
async function markEventReceived(eventId: string): Promise<void> {
// Store with 7-day TTL to handle late retries
await redis.setex(`webhook:event:${eventId}`, 7 * 24 * 60 * 60, 'received');
}
Async Processing Pattern
The recommended pattern separates receipt from processing for reliability and scale.
// Worker that processes queued webhook events
import { Worker } from 'bullmq';
const worker = new Worker('stripe-webhook', async (job) => {
const { eventId, eventType, payload, requestId } = job.data;
console.log({
worker: 'stripe-webhook',
eventId,
eventType,
requestId,
attempt: job.attemptsMade + 1
});
try {
// Route to appropriate handler based on event type
switch (eventType) {
case 'payment_intent.succeeded':
await handlePaymentSuccess(payload);
break;
case 'payment_intent.payment_failed':
await handlePaymentFailure(payload);
break;
case 'customer.subscription.created':
await handleSubscriptionCreated(payload);
break;
case 'customer.subscription.deleted':
await handleSubscriptionCanceled(payload);
break;
case 'invoice.paid':
await handleInvoicePaid(payload);
break;
case 'invoice.payment_failed':
await handleInvoicePaymentFailed(payload);
break;
default:
console.log({ eventType, message: 'Unhandled event type' });
}
// Mark as fully processed
await markEventProcessed(eventId);
} catch (error) {
console.error({
eventId,
eventType,
error: error.message,
attempt: job.attemptsMade + 1
});
throw error; // Triggers retry
}
}, {
connection: redisConnection,
concurrency: 10, // Process 10 events concurrently
// Retry configuration
settings: {
backoffStrategy: (attemptsMade) => {
// Exponential backoff: 1m, 5m, 15m, 30m, 1h
const delays = [60000, 300000, 900000, 1800000, 3600000];
return delays[Math.min(attemptsMade, delays.length - 1)];
}
}
});
// Handle permanently failed jobs
worker.on('failed', async (job, error) => {
if (job.attemptsMade >= 5) {
// Move to dead letter queue
await deadLetterQueue.add('failed-webhook', {
originalJob: job.data,
error: error.message,
attempts: job.attemptsMade,
failedAt: new Date().toISOString()
});
// Alert operations team
await alerting.send({
severity: 'high',
message: `Webhook permanently failed after ${job.attemptsMade} attempts`,
eventId: job.data.eventId,
eventType: job.data.eventType,
error: error.message
});
}
});
// Handlers implement idempotent operations
async function handlePaymentSuccess(event: any): Promise<void> {
const paymentIntent = event.data.object;
const orderId = paymentIntent.metadata.order_id;
// Idempotent: only update if not already processed
const order = await db.order.findUnique({ where: { id: orderId } });
if (order.status === 'paid') {
console.log({ orderId, message: 'Order already paid, skipping' });
return;
}
await db.$transaction([
db.order.update({
where: { id: orderId },
data: {
status: 'paid',
paidAt: new Date(),
paymentIntentId: paymentIntent.id
}
}),
db.payment.create({
data: {
orderId,
amount: paymentIntent.amount,
currency: paymentIntent.currency,
paymentIntentId: paymentIntent.id
}
})
]);
// Trigger downstream actions
await emailService.sendOrderConfirmation(order);
await inventoryService.reserveItems(order);
}
Webhook Security Model
┌─────────────────────────────────────────────────────────────────────┐
│ Webhook Security Layers │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Layer 1: TRANSPORT SECURITY │
│ ════════════════════════════ │
│ • HTTPS only (TLS 1.2+) │
│ • Valid SSL certificate │
│ • No self-signed certs in production │
│ │
│ Layer 2: AUTHENTICATION (Signature Verification) │
│ ═══════════════════════════════════════════════ │
│ • HMAC-SHA256 signature in header │
│ • Shared secret between provider and consumer │
│ • Timing-safe comparison to prevent timing attacks │
│ │
│ Layer 3: REPLAY PROTECTION │
│ ══════════════════════════ │
│ • Timestamp in signature (reject >5 min old) │
│ • Idempotency keys (track processed events) │
│ • Nonce validation (if supported) │
│ │
│ Layer 4: NETWORK CONTROLS (Optional) │
│ ════════════════════════════════════ │
│ • IP allowlisting (if provider publishes IPs) │
│ • VPN/private endpoints for sensitive data │
│ • Rate limiting on webhook endpoint │
│ │
│ Layer 5: DATA VALIDATION │
│ ═════════════════════════ │
│ • Schema validation of payload │
│ • Business logic validation │
│ • Sanitize before use (treat as untrusted input) │
│ │
└─────────────────────────────────────────────────────────────────────┘
Common Signature Patterns
Different providers use different signature schemes:
// Provider-specific signature verification
// Stripe: HMAC-SHA256 with timestamp
function verifyStripeSignature(payload: string, header: string, secret: string): boolean {
const elements = header.split(',');
const timestamp = elements.find(e => e.startsWith('t='))?.slice(2);
const signature = elements.find(e => e.startsWith('v1='))?.slice(3);
const signedPayload = `${timestamp}.${payload}`;
const expected = crypto.createHmac('sha256', secret).update(signedPayload).digest('hex');
return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected));
}
// GitHub: HMAC-SHA256
function verifyGitHubSignature(payload: string, header: string, secret: string): boolean {
const signature = header.replace('sha256=', '');
const expected = crypto.createHmac('sha256', secret).update(payload).digest('hex');
return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected));
}
// Slack: HMAC-SHA256 with timestamp
function verifySlackSignature(
payload: string,
signature: string,
timestamp: string,
secret: string
): boolean {
const baseString = `v0:${timestamp}:${payload}`;
const expected = 'v0=' + crypto.createHmac('sha256', secret).update(baseString).digest('hex');
return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected));
}
// Shopify: HMAC-SHA256 base64
function verifyShopifySignature(payload: string, header: string, secret: string): boolean {
const expected = crypto.createHmac('sha256', secret).update(payload).digest('base64');
return crypto.timingSafeEqual(Buffer.from(header), Buffer.from(expected));
}
// Generic: HMAC with configurable algorithm
function verifyHMACSignature(
payload: string,
signature: string,
secret: string,
algorithm: 'sha256' | 'sha1' = 'sha256',
encoding: 'hex' | 'base64' = 'hex'
): boolean {
const expected = crypto
.createHmac(algorithm, secret)
.update(payload)
.digest(encoding);
return crypto.timingSafeEqual(
Buffer.from(signature),
Buffer.from(expected)
);
}
Event Type Routing
Organize webhook handlers by event type for maintainability.
// Event handler registry pattern
type WebhookHandler = (event: WebhookEvent) => Promise<void>;
interface HandlerRegistry {
[eventType: string]: WebhookHandler;
}
class WebhookProcessor {
private handlers: HandlerRegistry = {};
private defaultHandler?: WebhookHandler;
// Register handlers for specific event types
on(eventType: string, handler: WebhookHandler): this {
this.handlers[eventType] = handler;
return this;
}
// Register handler for unhandled events
onDefault(handler: WebhookHandler): this {
this.defaultHandler = handler;
return this;
}
// Process an event
async process(event: WebhookEvent): Promise<void> {
const handler = this.handlers[event.type] || this.defaultHandler;
if (!handler) {
console.log({ eventType: event.type, message: 'No handler registered' });
return;
}
await handler(event);
}
}
// Usage: Build a processor for Stripe webhooks
const stripeProcessor = new WebhookProcessor()
// Payment events
.on('payment_intent.succeeded', handlePaymentSuccess)
.on('payment_intent.payment_failed', handlePaymentFailed)
.on('charge.refunded', handleRefund)
.on('charge.dispute.created', handleDispute)
// Subscription events
.on('customer.subscription.created', handleSubscriptionCreated)
.on('customer.subscription.updated', handleSubscriptionUpdated)
.on('customer.subscription.deleted', handleSubscriptionDeleted)
.on('customer.subscription.trial_will_end', handleTrialEnding)
// Invoice events
.on('invoice.paid', handleInvoicePaid)
.on('invoice.payment_failed', handleInvoiceFailed)
.on('invoice.upcoming', handleUpcomingInvoice)
// Customer events
.on('customer.created', handleCustomerCreated)
.on('customer.updated', handleCustomerUpdated)
.on('customer.deleted', handleCustomerDeleted)
// Default handler for unregistered events
.onDefault(async (event) => {
console.log({ eventType: event.type, message: 'Unhandled event type' });
// Optionally store for later analysis
await storeUnhandledEvent(event);
});
// In your worker
worker.on('stripe-webhook', async (job) => {
await stripeProcessor.process(job.data.payload);
});
Monitoring and Observability
// Webhook metrics and monitoring
import { Counter, Histogram, Gauge } from 'prom-client';
// Metrics
const webhooksReceived = new Counter({
name: 'webhooks_received_total',
help: 'Total webhooks received',
labelNames: ['provider', 'event_type', 'status']
});
const webhookProcessingDuration = new Histogram({
name: 'webhook_processing_duration_seconds',
help: 'Webhook processing duration',
labelNames: ['provider', 'event_type'],
buckets: [0.1, 0.5, 1, 2, 5, 10, 30]
});
const webhookQueueDepth = new Gauge({
name: 'webhook_queue_depth',
help: 'Current webhook queue depth',
labelNames: ['provider']
});
const webhookRetries = new Counter({
name: 'webhook_retries_total',
help: 'Total webhook processing retries',
labelNames: ['provider', 'event_type']
});
// Instrumented handler
async function handleWebhook(req: Request, res: Response) {
const startTime = Date.now();
const provider = req.params.provider;
let eventType = 'unknown';
let status = 'success';
try {
const event = await validateAndParse(req);
eventType = event.type;
await processWebhook(event);
} catch (error) {
status = error.message === 'Invalid signature' ? 'invalid_signature' : 'error';
throw error;
} finally {
// Record metrics
webhooksReceived.inc({ provider, event_type: eventType, status });
webhookProcessingDuration.observe(
{ provider, event_type: eventType },
(Date.now() - startTime) / 1000
);
}
}
// Dashboard queries (Prometheus/Grafana)
const dashboardQueries = {
// Success rate
successRate: `
sum(rate(webhooks_received_total{status="success"}[5m]))
/ sum(rate(webhooks_received_total[5m])) * 100
`,
// Average processing time
avgProcessingTime: `
histogram_quantile(0.95,
rate(webhook_processing_duration_seconds_bucket[5m])
)
`,
// Error rate by type
errorsByType: `
sum by (event_type) (
rate(webhooks_received_total{status!="success"}[5m])
)
`,
// Queue depth
queueDepth: `webhook_queue_depth`,
// Retry rate
retryRate: `
sum(rate(webhook_retries_total[5m]))
/ sum(rate(webhooks_received_total[5m])) * 100
`
};
// Alerting rules
const alertRules = `
groups:
- name: webhook_alerts
rules:
- alert: WebhookHighErrorRate
expr: |
sum(rate(webhooks_received_total{status!="success"}[5m]))
/ sum(rate(webhooks_received_total[5m])) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: Webhook error rate above 5%
- alert: WebhookQueueBacklog
expr: webhook_queue_depth > 1000
for: 5m
labels:
severity: warning
annotations:
summary: Webhook queue depth exceeding 1000
- alert: WebhookSlowProcessing
expr: |
histogram_quantile(0.95,
rate(webhook_processing_duration_seconds_bucket[5m])
) > 30
for: 5m
labels:
severity: warning
annotations:
summary: Webhook p95 processing time exceeding 30s
- alert: WebhookSignatureFailures
expr: rate(webhooks_received_total{status="invalid_signature"}[5m]) > 0.1
for: 5m
labels:
severity: critical
annotations:
summary: High rate of webhook signature failures
`;
Learning Path
Follow this progression to master webhook development:
┌─────────────────────────────────────────────────────────────────────┐
│ Webhook Development Learning Path │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ BEGINNER │
│ ════════ │
│ 1. Webhooks Explained - Core concepts and terminology │
│ 2. Webhook Best Practices - Essential patterns │
│ 3. Set up a simple webhook endpoint │
│ 4. Use ngrok for local testing │
│ │
│ INTERMEDIATE │
│ ════════════ │
│ 5. Signature Verification Guide - Implement security │
│ 6. Retry Logic Guide - Handle failures gracefully │
│ 7. Testing & Debugging Guide - Build test harnesses │
│ 8. Implement async processing with queues │
│ │
│ ADVANCED │
│ ════════ │
│ 9. Scaling & Performance Guide - Handle high volume │
│ 10. Error Handling Guide - Dead letter queues, recovery │
│ 11. Platform Integrations - Provider-specific patterns │
│ 12. Build comprehensive monitoring │
│ │
│ EXPERT │
│ ══════ │
│ 13. Building Webhook Providers - Design your own system │
│ 14. Multi-tenant webhook infrastructure │
│ 15. Webhook security hardening │
│ 16. Distributed webhook processing │
│ │
└─────────────────────────────────────────────────────────────────────┘
Quick Reference
Webhook Endpoint Checklist
□ HTTPS only (valid certificate)
□ Signature verification implemented
□ Timestamp validation (reject old requests)
□ Idempotency handling (track event IDs)
□ Async processing (queue events)
□ Fast response (<5 seconds)
□ Return 200 on success
□ Return 401 on invalid signature
□ Return 5xx only for retryable errors
□ Comprehensive logging
□ Monitoring and alerting
□ Dead letter queue for failures
Common HTTP Response Codes
| Code | Meaning | Provider Behavior |
|---|---|---|
| 200 | Success | No retry |
| 201 | Created | No retry |
| 202 | Accepted | No retry |
| 400 | Bad request | Usually no retry |
| 401 | Unauthorized | Usually no retry |
| 404 | Not found | Usually no retry |
| 410 | Gone | Disable webhook |
| 429 | Too many requests | Retry with backoff |
| 500 | Server error | Retry |
| 502 | Bad gateway | Retry |
| 503 | Service unavailable | Retry |
| 504 | Gateway timeout | Retry |
Event Processing States
Received → Validated → Queued → Processing → Completed
↓
Failed → Retry (up to N times)
↓
Dead Letter Queue
Conclusion
Webhooks enable powerful real-time integrations, but building reliable webhook systems requires attention to security, idempotency, async processing, and error handling. Key principles:
- Verify signatures - Never trust unverified webhook data
- Process async - Queue events and respond immediately
- Be idempotent - Handle duplicate deliveries gracefully
- Retry intelligently - Exponential backoff with limits
- Monitor everything - Success rates, latency, queue depth
Start with the fundamentals, implement security correctly, and progressively add reliability features as your webhook volume grows.
For detailed guidance on specific topics, explore the guides linked throughout this hub or start with our Webhooks Explained article.