Building a Self-Healing Retry Pipeline with Azure Service Bus

Retry logic is one of those things every distributed system has, and most of them get wrong in the same ways. The failure mode isn’t usually “we have no retry” — it’s “we have retry that silently stops working and nobody notices for weeks.”

This post is about a specific class of retry failure: an event-driven campaign trigger pipeline that was losing events on failure with no visibility, no recovery, and no alarm. I’ll walk through what was broken, why the fix required rethinking the retry infrastructure entirely, and the architectural decisions behind the replacement.

The Problem: Silence on Failure

Our event processing pipeline consumed scheduled campaign trigger events from Azure Event Hub. When processing succeeded, we advanced the checkpoint and moved on. When it failed, we were supposed to retry — but the retry path was a dead end.

The retry call went to a gRPC service that, on paper, handled re-queuing. In practice:

The gRPC path was not connected to any live retry infrastructure
The error was caught and swallowed — the Event Hub checkpoint was still advanced
A failed event that triggered a retry call that also failed was permanently lost
No status update was written to the database, no error logged at the right level

The result: from the outside, the system looked healthy. Metrics were green. Events were being processed. But a percentage of scheduled campaigns were silently never delivered, with no trace of why.

This is the worst kind of failure — invisible, intermittent, and impossible to reproduce in staging.

Why Not Just Fix the gRPC Path?

The first instinct was to repair the existing retry path. We decided against it for three reasons:

1. The retry mechanism had no scheduling capability. A failed event at 3:47 AM needs to retry at 3:57 AM, not immediately. Immediate retries on a transient failure (database overload, downstream service restart) create thundering herd problems. The gRPC service had no concept of scheduled re-delivery.

2. There was no deduplication. If the retry call was somehow made twice for the same event, two retry jobs would be scheduled. At scale, this compounds — the 5th retry attempt might actually be the 5th and 6th attempt running concurrently.

3. It produced no observable state. After a failure, there was no way to answer “is this event going to be retried, and when?” — a question that customer support needs to be able to answer when a campaign doesn’t fire.

The replacement needed to solve all three: delayed delivery, deduplication, and observable state.

The Architecture: Azure Service Bus Delayed Messages

Azure Service Bus has a feature that turns out to be exactly what retry pipelines need: scheduled message delivery. You can enqueue a message and specify that it should not be delivered until a future timestamp. The message sits in the queue, invisible to consumers, until its scheduled delivery time arrives.

This gives us retry scheduling as infrastructure rather than application code — we don’t need a cron job, a separate scheduling service, or an in-memory timer. The Service Bus queue is the retry queue.

The full retry flow:

flowchart TD
    EH[Azure Event Hub] -->|consume event| EP[EventProcessor]
    EP -->|success| CK[Advance Checkpoint DB status = DELIVERED]
    EP -->|failure| PUB[publishRetryToServiceBus attempt n]
    PUB -->|schedules at now + Fib*baseTtl| SBQ[(Service Bus Queue scheduled message)]
    SBQ -->|delay expires| SBC[ServiceBusConsumer]
    SBC --> EP2[EventProcessor retry]
    EP2 -->|success| CK2[completeMessage DB status = DELIVERED]
    EP2 -->|failure and attempt lte 29| PUB2[publishRetryToServiceBus attempt n+1]
    EP2 -->|attempt > 29| FAIL[completeMessage DB status = FAILED]
    PUB2 --> SBQ

    style CK fill:#0d3026,color:#64ffda,stroke:#64ffda
    style CK2 fill:#0d3026,color:#64ffda,stroke:#64ffda
    style FAIL fill:#3d1515,color:#ff6464,stroke:#ff6464
    style SBQ fill:#1a2744,color:#8892b0,stroke:#8892b0

The module structure in NestJS:

ServiceBusModule
  ├── ServiceBusProvider   (AMQP connection lifecycle)
  ├── ServiceBusService    (send + reconnect logic)
  └── ServiceBusConsumer   (receive + watchdog recovery)
       └── ServiceBusListener (wires consumer to event processor)

Each layer has a single responsibility. ServiceBusModule is self-contained — any module that imports it gets the full stack without wiring individual providers.

Fibonacci Backoff: Why Not Exponential?

The delay between retries uses Fibonacci numbers rather than the more common exponential backoff. The reason is ceiling behavior.

Exponential backoff at base 2 with a 30-second seed:

Attempt 10: ~8.5 hours
Attempt 15: ~11 days
Attempt 20: reaches months

Fibonacci with a 10-second base:

Attempt 10: ~55 × 10s = ~9 minutes
Attempt 20: ~6,765 × 10s = ~18 hours
Attempt 29: ~832,040 × 10s = ~96 days ← this is the problem

Without a ceiling, Fibonacci also diverges past what Azure Service Bus supports (14-day maximum message TTL). So getThresholdTime caps every computed delay at 7 days:

export const SB_MAX_DELAY_SECONDS = 604800; // 7 days

export function getThresholdTime(attempt: number, baseTtl = 10): number {
  const fib = fibonacci(attempt);
  const delay = fib * baseTtl;
  return Math.max(10, Math.min(delay, SB_MAX_DELAY_SECONDS));
}

The floor (10 seconds) prevents attempt 1 from scheduling a near-immediate retry that races with the original failure recovery. The ceiling prevents messages from exceeding the SB TTL and expiring before delivery.

The resulting schedule for the first 10 attempts:

Attempt	Fibonacci	Delay
1	1	10s
2	1	10s
3	2	20s
4	3	30s
5	5	50s
6	8	1m 20s
7	13	2m 10s
8	21	3m 30s
9	34	5m 40s
10	55	9m 10s

flowchart LR
    A1["Attempt 1 10s"] --> A2["Attempt 2 10s"] --> A3["Attempt 3 20s"] --> A4["Attempt 4 30s"] --> A5["Attempt 5 50s"]
    A5 --> A6["Attempt 6 1m 20s"] --> A7["Attempt 7 2m 10s"] --> A8["Attempt 8 3m 30s"] --> A9["Attempt 9 5m 40s"] --> A10["Attempt 10 9m 10s"]
    A10 --> DOTS["...grows to 7 days max"]

    style A1 fill:#112240,color:#8892b0,stroke:#233554
    style A2 fill:#112240,color:#8892b0,stroke:#233554
    style A3 fill:#0d2a1e,color:#64ffda,stroke:#64ffda
    style A4 fill:#0d2a1e,color:#64ffda,stroke:#64ffda
    style A5 fill:#0d2a1e,color:#64ffda,stroke:#64ffda
    style A6 fill:#0d3026,color:#64ffda,stroke:#64ffda
    style A7 fill:#0d3026,color:#64ffda,stroke:#64ffda
    style A8 fill:#0d3026,color:#64ffda,stroke:#64ffda
    style A9 fill:#0d3026,color:#64ffda,stroke:#64ffda
    style A10 fill:#0d3026,color:#64ffda,stroke:#64ffda
    style DOTS fill:#112240,color:#495670,stroke:#233554

The gradual ramp gives transient failures (network blip, cold start) time to resolve before aggressive retries compound the load.

Deduplication: Making Retries Idempotent

Azure Service Bus supports duplicate detection via messageId. If two messages with the same messageId arrive within the duplicate detection window, the second is silently dropped.

We set messageId to {idempotency_key}-attempt-{n}:

const message = {
  body: { ...body, attempts, correlation_id },
  messageId: `${body.idempotency_key}-attempt-${attempts}`,
  scheduledEnqueueTimeUtc: deliveryTime,
};

This means:

If publishRetryToServiceBus is called twice for the same event at the same attempt number, only one message is enqueued
Different attempt numbers get different messageId values, so legitimate retry progression isn’t blocked
The idempotency guarantee is at the Service Bus layer, not in application code

Checkpoint Management: Complete, Don’t Abandon

This is the detail most retry implementations get wrong.

When an event processor fails and we publish a retry to Service Bus, the original Service Bus message should be completed (acknowledged), not abandoned.

Abandoning a message tells Service Bus: “I couldn’t process this, please redeliver.” But we’ve already scheduled a retry via publishRetryToServiceBus. If we also abandon the message, Service Bus redelivers the original, and now we have two retry messages in-flight for the same event — the one we scheduled and the one Service Bus redelivered.

// After handler fails and retry is published:
await receiver.completeMessage(message); // ✅ correct
// NOT: await receiver.abandonMessage(message); // ❌ creates duplicates

The invariant: exactly one retry message per failure. completeMessage removes the current message from the queue. The retry message we published handles the next attempt.

There’s one exception: if publishRetryToServiceBus itself fails (Service Bus is unavailable), we don’t swallow that error. The outer event processor re-throws, preventing the Event Hub checkpoint from advancing. The original event remains available for replay from Event Hub — a second layer of durability.

Connection Resilience: The Watchdog Pattern

AMQP connections to Azure Service Bus are long-lived TCP connections. They break — network partitions, idle timeouts, Azure maintenance windows, pod restarts. A consumer that doesn’t recover from a broken connection stops processing silently. From the outside: metrics show no errors (because errors require processing), queue depth grows.

ServiceBusConsumer has three layers of recovery:

flowchart TD
    ERR[processError fired] --> L1{Sender healthy?}
    L1 -->|yes| RESUB[Re-subscribe with original handlers]
    L1 -->|no| REINIT[reinitialize rebuild AMQP client]
    REINIT --> LOOP[Retry loop 3 attempts - 5s, 10s, 15s]
    LOOP -->|success| RESUB
    LOOP -->|all 3 fail| WD{Watchdog already running?}
    WD -->|yes| NOOP[No-op one watchdog max]
    WD -->|no| START[startWatchdog setInterval 60s]
    START --> TICK[Every 60s: reinitialize + startListening]
    TICK -->|success| CLEAR[clearInterval normal operation]
    TICK -->|fail| TICK

    style CLEAR fill:#0d3026,color:#64ffda,stroke:#64ffda
    style NOOP fill:#1a2744,color:#8892b0,stroke:#8892b0
    style START fill:#2a1a00,color:#ffd700,stroke:#ffd700

Layer 1: Pre-send health check Before every send, isSenderHealthy() checks sender.isClosed. If the sender is stale, reinitialize() rebuilds the AMQP client and sender before sending.

Layer 2: Immediate reconnect loop On processError, the consumer attempts up to 3 reconnects with linear backoff (5s, 10s, 15s). Each attempt calls provider.reinitialize() first — getting a fresh ServiceBusClient — then re-subscribes with the original handlers.

for (let i = 0; i < 3; i++) {
  await sleep(RECONNECT_DELAYS[i]);
  try {
    await this.provider.reinitialize();
    await this.startListening();
    return; // recovered
  } catch (err) {
    this.logger.warn(`Reconnect attempt ${i + 1} failed`);
  }
}
// All 3 failed — start watchdog
this.startWatchdog();

Layer 3: Watchdog timer If all immediate reconnects fail (extended outage, sustained partition), a setInterval watchdog fires every 60 seconds to keep attempting recovery:

private startWatchdog(): void {
  if (this.watchdogTimer) return; // already running, don't duplicate
  this.watchdogTimer = setInterval(async () => {
    try {
      await this.provider.reinitialize();
      await this.startListening();
      clearInterval(this.watchdogTimer);
      this.watchdogTimer = null;
    } catch {
      // keep trying
    }
  }, 60_000);
}

The watchdog deduplicates itself — a second processError while the watchdog is running does not start a second interval. onModuleDestroy clears the timer for clean graceful shutdown.

The key fix from the previous implementation: the old code reused the broken ServiceBusClient for reconnect attempts, which meant every reconnect attempt failed against the same dead connection. reinitialize() tears down and rebuilds from scratch.

Distributed Tracing: Correlation IDs Across Retries

Every retry attempt carries a correlation_id that is seeded on first failure and preserved unchanged across all subsequent retries:

const correlation_id = body.correlation_id ?? body.idempotency_key;

On first failure, correlation_id doesn’t exist in the body yet, so it’s seeded from idempotency_key. On all subsequent retries, the existing correlation_id is preserved.

This means you can filter logs across all retry attempts for a single event with one query:

correlation_id = "campaign-trigger-abc123"
→ attempt 1: failed at 03:47:23 (DB timeout)
→ attempt 2: failed at 03:47:33 (DB timeout)
→ attempt 3: succeeded at 03:48:03

Without a stable correlation ID, tracing a campaign delivery failure through 5 retry attempts across distributed logs requires reconstructing the chain from timestamps and idempotency keys — possible, but slow. With a correlation ID, it’s a single log query.

Test Strategy

The retry infrastructure has three critical behavioral properties that are easy to break silently:

Watchdog deduplication — a second processError must not start a second watchdog
Complete-not-abandon — handler failure must call completeMessage, not abandonMessage
Reconnect ordering — reinitialize() must be called before startListening() on reconnect

All three were tested with Jest fake timers to avoid real-time waits:

jest.useFakeTimers();

it('does not start a second watchdog if one is already running', async () => {
  // First processError starts watchdog
  await consumer.processError(mockError);
  expect(clearIntervalSpy).not.toHaveBeenCalled();

  // Second processError while watchdog is running
  await consumer.processError(mockError);

  // Only one interval should exist
  expect(setIntervalSpy).toHaveBeenCalledTimes(1);
});

Fibonacci behavior was unit-tested at the boundary cases:

it('floors at 10 seconds for early attempts', () => {
  expect(getThresholdTime(1)).toBe(10);
  expect(getThresholdTime(2)).toBe(10);
});

it('caps at SB_MAX_DELAY_SECONDS for large attempts', () => {
  expect(getThresholdTime(29)).toBe(SB_MAX_DELAY_SECONDS);
  expect(getThresholdTime(50)).toBe(SB_MAX_DELAY_SECONDS);
});

Known Limitations

The attempts counter is not database-persisted. The retry count travels in the Service Bus message body. If a pod restarts while a message is in-flight, the message is redelivered with its original body.attempts value intact — so the Fibonacci backoff is preserved. However, if a message is lost before delivery (unlikely but possible on SB outage during scheduled delivery window), the attempts counter resets to 1 and the backoff restarts from the beginning. A production hardening would persist attempts to the database on each retry cycle.

The hard failure at attempt 29 is not configurable. The ceiling is hardcoded. A more flexible design would make it configurable per event type, since some campaigns have tighter SLA requirements than others.

Both are tracked for follow-up. Neither blocks the primary correctness guarantee: events no longer silently vanish.

When This Pattern Applies

Use Azure Service Bus delayed messages for retry scheduling when:

You need scheduled retries (not immediate re-processing)
You need deduplication guarantees at the infrastructure level
The failure rate is low enough that retry queue depth stays manageable
You need observable retry state (what’s queued, when it will redeliver)

Don’t use it when:

You need sub-second retry intervals (SB scheduled delivery has ~1s precision)
Your retry volume is high enough that SB queue depth becomes a cost concern
You’re retrying stateless HTTP calls where the built-in axios-retry or similar is sufficient

The pattern is most valuable when the thing you’re retrying has side effects (database writes, downstream API calls, message sends) that make duplicate execution dangerous — which is exactly where deduplication-by-messageId earns its keep.