Reliable webhook receiver: signatures, idempotency, queue and retries

Prompt

Act as a backend engineer experienced in integrations and distributed systems.

I need to receive webhooks reliably:
- Provider and how it signs requests: {{provider}}
- Events I care about and what I do with each: {{events}}
- Stack: {{stack}}
- Available queue system: {{queue}}

Deliverables:
1. A text diagram of the flow: receive, verify, persist, enqueue, process, retry and dead-letter.
2. The receiving endpoint:
   - Read the raw body before parsing and verify the HMAC signature with a constant-time comparison.
   - Reject requests whose timestamp is outside a short window to prevent replay attacks.
   - Store the event and return 2xx within milliseconds; no heavy logic in the request.
3. Idempotency: an events table with the provider's event id as a unique key, states (received, processing, done, failed) and what to do when the same event arrives twice.
4. Worker: processing with exponential backoff and jitter, a maximum attempt count, a distinction between transient and permanent errors, and a dead-letter queue with a way to replay it.
5. Ordering: how to handle out-of-order events (for example by checking the resource version or timestamp, or re-fetching from the provider's API).
6. Logging and alerting: what to log without storing sensitive data and which metrics to watch.
7. Tests: valid and invalid signature, duplicate event, stale timestamp, transient failure that recovers and permanent failure that lands in the dead-letter queue.
8. How to test locally (tunnel or event forwarding) without exposing the secret.

Constraints:
- Do not guess the provider's exact signature format or header names: if I do not give them, mark them as [FILL IN] and tell me where to look in their docs.
- Read the secret from environment variables or a secrets manager, never from code.
- If key information is missing, ask me before writing code.

How to use it

Usage example

Sample output

Gateway → POST /webhooks/payments → verify signature + timestamp → INSERT event (unique)
        → commit → 200 → Celery: process → retries with backoff → dead-letter

@router.post("/webhooks/payments")
async def receive(request: Request, db: AsyncSession = Depends(get_db)) -> dict:
    raw = await request.body()                        # raw body, before parsing
    ts = request.headers.get("[FILL IN: timestamp header]", "")
    sig = request.headers.get("[FILL IN: signature header]", "")
    if not ts.isdigit() or abs(time.time() - int(ts)) > 300:
        raise HTTPException(400, "missing or stale timestamp")

    # Signed message format per the provider's docs: [FILL IN]
    expected = hmac.new(settings.webhook_secret.encode(),
                        f"{ts}.".encode() + raw, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, sig):        # constant-time comparison
        raise HTTPException(400, "invalid signature")

    event = json.loads(raw)
    stmt = (insert(WebhookEvent)
            .values(provider_event_id=event["id"], type=event["type"],
                    payload=event, status="received")
            .on_conflict_do_nothing(index_elements=["provider_event_id"])
            .returning(WebhookEvent.id))
    new_id = (await db.execute(stmt)).scalar_one_or_none()
    await db.commit()
    if new_id is not None:                            # duplicate → not enqueued again
        process_event.delay(new_id)
    return {"ok": True}

Key facts

Category

Programming & development · APIs

Output type

Code

Level

Advanced

Designed for

ChatGPT, Claude, Gemini and Microsoft Copilot

Verified in

Pending editorial verification

Variables

Provider and signature, Events and actions, Stack and Queue system

Language

English

Price

Free, no sign-up

Updated

Nov 7, 2026

Frequently asked questions