Atul Singh — Architecture & Systems

Building a Self-Healing Retry Pipeline with Azure Service Bus

Tue, 24 Mar 2026 00:00:00 GMT

Retry logic is one of those things every distributed system has, and most of them get wrong in the same ways. The failure mode isn't usually "we have no retry" — it's "we have retry that silently stops working and nobody notices for weeks."

This post is about a specific class of retry failure: an event-driven campaign trigger pipeline that was losing events on failure with no visibility, no recovery, and no alarm. I'll walk through what was broken, why the fix required rethinking the retry infrastructure entirely, and the architectural decisions behind the replacement.

The Problem: Silence on Failure

Our event processing pipeline consumed scheduled campaign trigger events from Azure Event Hub. When processing succeeded, we advanced the checkpoint and moved on. When it failed, we were supposed to retry — but the retry path was a dead end.

The retry call went to a gRPC service that, on paper, handled re-queuing. In practice:

The gRPC path was not connected to any live retry infrastructure
The error was caught and swallowed — the Event Hub checkpoint was still advanced
A failed event that triggered a retry call that also failed was permanently lost
No status update was written to the database, no error logged at the right level

The result: from the outside, the system looked healthy. Metrics were green. Events were being processed. But a percentage of scheduled campaigns were silently never delivered, with no trace of why.

This is the worst kind of failure — invisible, intermittent, and impossible to reproduce in staging.

Why Not Just Fix the gRPC Path?

The first instinct was to repair the existing retry path. We decided against it for three reasons:

1. The retry mechanism had no scheduling capability. A failed event at 3:47 AM needs to retry at 3:57 AM, not immediately. Immediate retries on a transient failure (database overload, downstream service restart) create thundering herd problems. The gRPC service had no concept of scheduled re-delivery.

2. There was no deduplication. If the retry call was somehow made twice for the same event, two retry jobs would be scheduled. At scale, this compounds — the 5th retry attempt might actually be the 5th and 6th attempt running concurrently.

3. It produced no observable state. After a failure, there was no way to answer "is this event going to be retried, and when?" — a question that customer support needs to be able to answer when a campaign doesn't fire.

The replacement needed to solve all three: delayed delivery, deduplication, and observable state.

The Architecture: Azure Service Bus Delayed Messages

Azure Service Bus has a feature that turns out to be exactly what retry pipelines need: scheduled message delivery. You can enqueue a message and specify that it should not be delivered until a future timestamp. The message sits in the queue, invisible to consumers, until its scheduled delivery time arrives.

This gives us retry scheduling as infrastructure rather than application code — we don't need a cron job, a separate scheduling service, or an in-memory timer. The Service Bus queue is the retry queue.

The full retry flow:

flowchart TD
    EH[Azure Event Hub] -->|consume event| EP[EventProcessor]
    EP -->|success| CK[Advance Checkpoint DB status = DELIVERED]
    EP -->|failure| PUB[publishRetryToServiceBus attempt n]
    PUB -->|schedules at now + Fib*baseTtl| SBQ[(Service Bus Queue scheduled message)]
    SBQ -->|delay expires| SBC[ServiceBusConsumer]
    SBC --> EP2[EventProcessor retry]
    EP2 -->|success| CK2[completeMessage DB status = DELIVERED]
    EP2 -->|failure and attempt lte 29| PUB2[publishRetryToServiceBus attempt n+1]
    EP2 -->|attempt > 29| FAIL[completeMessage DB status = FAILED]
    PUB2 --> SBQ

    style CK fill:#0d3026,color:#64ffda,stroke:#64ffda
    style CK2 fill:#0d3026,color:#64ffda,stroke:#64ffda
    style FAIL fill:#3d1515,color:#ff6464,stroke:#ff6464
    style SBQ fill:#1a2744,color:#8892b0,stroke:#8892b0

The module structure in NestJS:

ServiceBusModule
  ├── ServiceBusProvider   (AMQP connection lifecycle)
  ├── ServiceBusService    (send + reconnect logic)
  └── ServiceBusConsumer   (receive + watchdog recovery)
       └── ServiceBusListener (wires consumer to event processor)

Each layer has a single responsibility. ServiceBusModule is self-contained — any module that imports it gets the full stack without wiring individual providers.

Fibonacci Backoff: Why Not Exponential?

The delay between retries uses Fibonacci numbers rather than the more common exponential backoff. The reason is ceiling behavior.

Exponential backoff at base 2 with a 30-second seed:

Attempt 10: ~8.5 hours
Attempt 15: ~11 days
Attempt 20: reaches months

Fibonacci with a 10-second base:

Attempt 10: ~55 × 10s = ~9 minutes
Attempt 20: ~6,765 × 10s = ~18 hours
Attempt 29: ~832,040 × 10s = ~96 days ← this is the problem

Without a ceiling, Fibonacci also diverges past what Azure Service Bus supports (14-day maximum message TTL). So getThresholdTime caps every computed delay at 7 days:

export const SB_MAX_DELAY_SECONDS = 604800; // 7 days

export function getThresholdTime(attempt: number, baseTtl = 10): number {
  const fib = fibonacci(attempt);
  const delay = fib * baseTtl;
  return Math.max(10, Math.min(delay, SB_MAX_DELAY_SECONDS));
}

The floor (10 seconds) prevents attempt 1 from scheduling a near-immediate retry that races with the original failure recovery. The ceiling prevents messages from exceeding the SB TTL and expiring before delivery.

The resulting schedule for the first 10 attempts:

Attempt	Fibonacci	Delay
1	1	10s
2	1	10s
3	2	20s
4	3	30s
5	5	50s
6	8	1m 20s
7	13	2m 10s
8	21	3m 30s
9	34	5m 40s
10	55	9m 10s

flowchart LR
    A1["Attempt 1 10s"] --> A2["Attempt 2 10s"] --> A3["Attempt 3 20s"] --> A4["Attempt 4 30s"] --> A5["Attempt 5 50s"]
    A5 --> A6["Attempt 6 1m 20s"] --> A7["Attempt 7 2m 10s"] --> A8["Attempt 8 3m 30s"] --> A9["Attempt 9 5m 40s"] --> A10["Attempt 10 9m 10s"]
    A10 --> DOTS["...grows to 7 days max"]

    style A1 fill:#112240,color:#8892b0,stroke:#233554
    style A2 fill:#112240,color:#8892b0,stroke:#233554
    style A3 fill:#0d2a1e,color:#64ffda,stroke:#64ffda
    style A4 fill:#0d2a1e,color:#64ffda,stroke:#64ffda
    style A5 fill:#0d2a1e,color:#64ffda,stroke:#64ffda
    style A6 fill:#0d3026,color:#64ffda,stroke:#64ffda
    style A7 fill:#0d3026,color:#64ffda,stroke:#64ffda
    style A8 fill:#0d3026,color:#64ffda,stroke:#64ffda
    style A9 fill:#0d3026,color:#64ffda,stroke:#64ffda
    style A10 fill:#0d3026,color:#64ffda,stroke:#64ffda
    style DOTS fill:#112240,color:#495670,stroke:#233554

The gradual ramp gives transient failures (network blip, cold start) time to resolve before aggressive retries compound the load.

Deduplication: Making Retries Idempotent

Azure Service Bus supports duplicate detection via messageId. If two messages with the same messageId arrive within the duplicate detection window, the second is silently dropped.

We set messageId to {idempotency_key}-attempt-{n}:

const message = {
  body: { ...body, attempts, correlation_id },
  messageId: `${body.idempotency_key}-attempt-${attempts}`,
  scheduledEnqueueTimeUtc: deliveryTime,
};

This means:

If publishRetryToServiceBus is called twice for the same event at the same attempt number, only one message is enqueued
Different attempt numbers get different messageId values, so legitimate retry progression isn't blocked
The idempotency guarantee is at the Service Bus layer, not in application code

Checkpoint Management: Complete, Don't Abandon

This is the detail most retry implementations get wrong.

When an event processor fails and we publish a retry to Service Bus, the original Service Bus message should be completed (acknowledged), not abandoned.

Abandoning a message tells Service Bus: "I couldn't process this, please redeliver." But we've already scheduled a retry via publishRetryToServiceBus. If we also abandon the message, Service Bus redelivers the original, and now we have two retry messages in-flight for the same event — the one we scheduled and the one Service Bus redelivered.

// After handler fails and retry is published:
await receiver.completeMessage(message); // ✅ correct
// NOT: await receiver.abandonMessage(message); // ❌ creates duplicates

The invariant: exactly one retry message per failure. completeMessage removes the current message from the queue. The retry message we published handles the next attempt.

There's one exception: if publishRetryToServiceBus itself fails (Service Bus is unavailable), we don't swallow that error. The outer event processor re-throws, preventing the Event Hub checkpoint from advancing. The original event remains available for replay from Event Hub — a second layer of durability.

Connection Resilience: The Watchdog Pattern

AMQP connections to Azure Service Bus are long-lived TCP connections. They break — network partitions, idle timeouts, Azure maintenance windows, pod restarts. A consumer that doesn't recover from a broken connection stops processing silently. From the outside: metrics show no errors (because errors require processing), queue depth grows.

ServiceBusConsumer has three layers of recovery:

flowchart TD
    ERR[processError fired] --> L1{Sender healthy?}
    L1 -->|yes| RESUB[Re-subscribe with original handlers]
    L1 -->|no| REINIT[reinitialize rebuild AMQP client]
    REINIT --> LOOP[Retry loop 3 attempts - 5s, 10s, 15s]
    LOOP -->|success| RESUB
    LOOP -->|all 3 fail| WD{Watchdog already running?}
    WD -->|yes| NOOP[No-op one watchdog max]
    WD -->|no| START[startWatchdog setInterval 60s]
    START --> TICK[Every 60s: reinitialize + startListening]
    TICK -->|success| CLEAR[clearInterval normal operation]
    TICK -->|fail| TICK

    style CLEAR fill:#0d3026,color:#64ffda,stroke:#64ffda
    style NOOP fill:#1a2744,color:#8892b0,stroke:#8892b0
    style START fill:#2a1a00,color:#ffd700,stroke:#ffd700

Layer 1: Pre-send health check Before every send, isSenderHealthy() checks sender.isClosed. If the sender is stale, reinitialize() rebuilds the AMQP client and sender before sending.

Layer 2: Immediate reconnect loop On processError, the consumer attempts up to 3 reconnects with linear backoff (5s, 10s, 15s). Each attempt calls provider.reinitialize() first — getting a fresh ServiceBusClient — then re-subscribes with the original handlers.

for (let i = 0; i < 3; i++) {
  await sleep(RECONNECT_DELAYS[i]);
  try {
    await this.provider.reinitialize();
    await this.startListening();
    return; // recovered
  } catch (err) {
    this.logger.warn(`Reconnect attempt ${i + 1} failed`);
  }
}
// All 3 failed — start watchdog
this.startWatchdog();

Layer 3: Watchdog timer If all immediate reconnects fail (extended outage, sustained partition), a setInterval watchdog fires every 60 seconds to keep attempting recovery:

private startWatchdog(): void {
  if (this.watchdogTimer) return; // already running, don't duplicate
  this.watchdogTimer = setInterval(async () => {
    try {
      await this.provider.reinitialize();
      await this.startListening();
      clearInterval(this.watchdogTimer);
      this.watchdogTimer = null;
    } catch {
      // keep trying
    }
  }, 60_000);
}

The watchdog deduplicates itself — a second processError while the watchdog is running does not start a second interval. onModuleDestroy clears the timer for clean graceful shutdown.

The key fix from the previous implementation: the old code reused the broken ServiceBusClient for reconnect attempts, which meant every reconnect attempt failed against the same dead connection. reinitialize() tears down and rebuilds from scratch.

Distributed Tracing: Correlation IDs Across Retries

Every retry attempt carries a correlation_id that is seeded on first failure and preserved unchanged across all subsequent retries:

const correlation_id = body.correlation_id ?? body.idempotency_key;

On first failure, correlation_id doesn't exist in the body yet, so it's seeded from idempotency_key. On all subsequent retries, the existing correlation_id is preserved.

This means you can filter logs across all retry attempts for a single event with one query:

correlation_id = "campaign-trigger-abc123"
→ attempt 1: failed at 03:47:23 (DB timeout)
→ attempt 2: failed at 03:47:33 (DB timeout)
→ attempt 3: succeeded at 03:48:03

Without a stable correlation ID, tracing a campaign delivery failure through 5 retry attempts across distributed logs requires reconstructing the chain from timestamps and idempotency keys — possible, but slow. With a correlation ID, it's a single log query.

Test Strategy

The retry infrastructure has three critical behavioral properties that are easy to break silently:

Watchdog deduplication — a second processError must not start a second watchdog
Complete-not-abandon — handler failure must call completeMessage, not abandonMessage
Reconnect ordering — reinitialize() must be called before startListening() on reconnect

All three were tested with Jest fake timers to avoid real-time waits:

jest.useFakeTimers();

it('does not start a second watchdog if one is already running', async () => {
  // First processError starts watchdog
  await consumer.processError(mockError);
  expect(clearIntervalSpy).not.toHaveBeenCalled();

  // Second processError while watchdog is running
  await consumer.processError(mockError);

  // Only one interval should exist
  expect(setIntervalSpy).toHaveBeenCalledTimes(1);
});

Fibonacci behavior was unit-tested at the boundary cases:

it('floors at 10 seconds for early attempts', () => {
  expect(getThresholdTime(1)).toBe(10);
  expect(getThresholdTime(2)).toBe(10);
});

it('caps at SB_MAX_DELAY_SECONDS for large attempts', () => {
  expect(getThresholdTime(29)).toBe(SB_MAX_DELAY_SECONDS);
  expect(getThresholdTime(50)).toBe(SB_MAX_DELAY_SECONDS);
});

Known Limitations

The attempts counter is not database-persisted. The retry count travels in the Service Bus message body. If a pod restarts while a message is in-flight, the message is redelivered with its original body.attempts value intact — so the Fibonacci backoff is preserved. However, if a message is lost before delivery (unlikely but possible on SB outage during scheduled delivery window), the attempts counter resets to 1 and the backoff restarts from the beginning. A production hardening would persist attempts to the database on each retry cycle.

The hard failure at attempt 29 is not configurable. The ceiling is hardcoded. A more flexible design would make it configurable per event type, since some campaigns have tighter SLA requirements than others.

Both are tracked for follow-up. Neither blocks the primary correctness guarantee: events no longer silently vanish.

When This Pattern Applies

Use Azure Service Bus delayed messages for retry scheduling when:

You need scheduled retries (not immediate re-processing)
You need deduplication guarantees at the infrastructure level
The failure rate is low enough that retry queue depth stays manageable
You need observable retry state (what's queued, when it will redeliver)

Don't use it when:

You need sub-second retry intervals (SB scheduled delivery has ~1s precision)
Your retry volume is high enough that SB queue depth becomes a cost concern
You're retrying stateless HTTP calls where the built-in axios-retry or similar is sufficient

The pattern is most valuable when the thing you're retrying has side effects (database writes, downstream API calls, message sends) that make duplicate execution dangerous — which is exactly where deduplication-by-messageId earns its keep.

CQRS in Practice: Where the Pattern Earns Its Complexity

Tue, 10 Feb 2026 00:00:00 GMT

CQRS (Command Query Responsibility Segregation) is one of those patterns that gets recommended far more often than it's warranted. It shows up in every system design interview, every architecture whiteboard — and most of the time, it's the wrong answer. But when it's the right answer, the improvement is dramatic.

This post is about when it was the right answer.

The Contention Problem CQRS Actually Solves

CQRS solves a specific problem: high-write systems where reads compete with writes for the same data model.

In a traditional architecture, your write path and your read path share the same schema, the same indexes, and the same connection pool. At low scale, this is fine. At high scale, writes hold locks that block reads, reads create query plans that conflict with write-optimised indexes, and your connection pool becomes a battleground.

The key insight is not "separate your reads and writes" — it's "your read model and your write model have different requirements, and forcing them into the same schema creates contention that grows superlinearly with load."

flowchart LR
    subgraph Before["Shared Model - Contention"]
        W1[Write Path] --> DB1[(Single Schema)]
        R1[Read Path] --> DB1
    end

    subgraph After["CQRS - Separated Concerns"]
        W2[Write Path] --> WDB[(Write Model)]
        WDB -->|async refresh| RDB[(Read Model MV)]
        R2[Read Path] --> RDB
    end

    style DB1 fill:#3d1515,color:#ff8080,stroke:#ff6464
    style WDB fill:#0d3026,color:#64ffda,stroke:#64ffda
    style RDB fill:#0d3026,color:#64ffda,stroke:#64ffda

What CQRS Is NOT

Before going further, let's clear up common misconceptions:

CQRS is not Event Sourcing. Event sourcing stores state as a sequence of events. CQRS separates read and write models. You can use either without the other. Conflating them is the #1 source of over-engineering.

CQRS is not a microservices requirement. You can implement CQRS within a single service, a single database, even a single repository. It's a data model pattern, not a deployment pattern.

CQRS is not a database pattern. It doesn't require separate databases for reads and writes (though it can benefit from them). The separation can be as simple as a materialized view.

Where We Applied It: The Segmentation Engine

Our multi-tenant CRM platform ingests 1M+ events per 5 minutes from service operations across hundreds of locations. Marketing teams build customer segments using complex filters — "active subscribers at Location A with 3+ transactions in the last 30 days and wallet credit > $50."

The Write Path (Commands)

Events arrive via Azure Event Hub across 32 partitions. The ingestion pipeline processes them into normalised tables:

Event Hub → Consumer Pods → subscription_account
                          → transaction_log
                          → wallet_account
                          → package_account

These tables are write-optimised: normalised schema, minimal indexes, append-heavy. The write path doesn't care about query patterns.

The Read Path (Queries)

The segmentation API needs to answer questions like "how many customers match these 5 filters?" in under 200ms. Running this query against the write-optimised tables required 7-table joins with subqueries. Response time: 30+ seconds.

The Separation

Instead of fighting the write schema, we built a separate read model:

-- Read model: pre-computed, denormalised, query-optimised
CREATE MATERIALIZED VIEW customer_attributes_mv AS
SELECT
    c.id,
    COALESCE(sub.is_active_subscriber, false) as is_active_subscriber,
    sub.location_codes,       -- ARRAY type for GIN index
    sub.package_ids,          -- ARRAY type for GIN index
    COALESCE(wal.total_balance, 0) as wallet_balance
FROM customer c
LEFT JOIN sub_stats sub ON c.customer_uuid = sub.customer_id
LEFT JOIN wal_stats wal ON c.customer_uuid = wal.customer_id;

The read model is a PostgreSQL Materialized View with GIN-indexed arrays. It's refreshed every 45 seconds using REFRESH MATERIALIZED VIEW CONCURRENTLY — zero query downtime during refresh.

The Full Data Flow

flowchart TB
    EH["Event Hub 32 partitions"] --> Pods["Consumer Pods"]

    Pods --> T1[("subscription_account")]
    Pods --> T2[("transaction_log")]
    Pods --> T3[("wallet_account")]
    Pods --> T4[("package_account")]

    T1 --> |"REFRESH every 45s"| MV1[("customer_attributes_mv")]
    T3 --> |"REFRESH every 45s"| MV1
    T4 --> |"REFRESH every 45s"| MV1
    T2 --> |"REFRESH every 5min"| MV2[("customer_transactions_mv")]

    MV1 --> API["Segmentation API"]
    MV2 --> API
    API --> |"under 200ms"| Result["Segment Count"]

    style MV1 fill:#0d3026,color:#64ffda,stroke:#64ffda
    style MV2 fill:#0d3026,color:#64ffda,stroke:#64ffda
    style Result fill:#0d3026,color:#64ffda,stroke:#64ffda

The Result

Query	Before (shared model)	After (CQRS)	Improvement
Is Active Subscriber	30s	50ms	600x
Subscriber + Balance > $50	35s	80ms	437x
Complex 5-filter segment	60s	200ms	300x

The 30s → 100ms improvement came directly from this separation. Not from better indexes on the write tables. Not from query optimisation. From acknowledging that the read model and write model had fundamentally different requirements and should not share a schema.

The Real Cost

CQRS is not free. Here's what you pay:

1. Eventual Consistency Windows

The read model is 45–90 seconds behind the write model. Every product decision must account for this.

sequenceDiagram
    participant W as Write Path
    participant DB as Source Tables
    participant MV as Materialized View
    participant API as Segmentation API

    Note over W,DB: T=0 — new customer created
    W->>DB: INSERT into subscription_account

    Note over DB,MV: T=0 -> T=45s — MV is stale
    API->>MV: Query segment count
    Note over MV: Returns count WITHOUT new customer

    Note over DB,MV: T=45s — REFRESH CONCURRENTLY runs
    DB->>MV: Atomic swap with fresh data

    API->>MV: Query segment count
    Note over MV: Returns count WITH new customer

This is acceptable for marketing segmentation (not time-critical). It would be unacceptable for an order status page.

2. Debugging Difficulty

When a customer "doesn't appear" in a segment, is it because:

The event hasn't been ingested yet?
The MV hasn't refreshed yet?
The filter logic is wrong?
The MV definition doesn't include the right join?

Debugging CQRS requires tracing a data point across the write path, through the refresh cycle, and into the read model. This is substantially harder than debugging a single query against a shared database.

3. Operational Overhead

Materialized Views need monitoring:

Refresh time — grows as data grows. A 45s refresh cycle that takes 5s today may take 50s next quarter.
Staleness tracking — you need a mv_refresh_status table to know when data was last synced.
Index count — 30+ GIN indexes slow down refresh significantly. Every read optimisation has a write cost.
Per-tenant management — with 100+ tenant databases, each MV refresh is a separate operation.

4. Schema Drift Risk

The write schema evolves independently. If you add a column to subscription_account, you must remember to update the MV definition. There's no compiler error, no type check — just a missing column that won't surface until someone tries to filter by it.

Decision Framework: 3 Questions Before Using CQRS

Before reaching for CQRS, ask:

1. "Do my reads and writes have fundamentally different data shapes?"

If your reads can be answered by the same schema your writes use, with appropriate indexes, CQRS adds complexity without benefit. Most CRUD applications fall into this category.

Our case: Writes are normalised inserts across 7 tables. Reads need a single denormalised row with pre-computed aggregates and array columns. These shapes are incompatible.

2. "Is the consistency window acceptable for every consumer of the read model?"

List every system that reads from your read model. For each one: can it tolerate N seconds of staleness? If any critical consumer needs real-time consistency, CQRS creates more problems than it solves.

Our case: Marketing segmentation, not real-time operations. 90 seconds of staleness is invisible to users.

3. "Have I exhausted simpler alternatives?"

Before CQRS:

Can you add composite indexes?
Can you use a query cache?
Can you denormalise a single column?
Can you pre-compute at write time?

Our case: We tried composite indexes first. The fundamental problem was 7-table joins with subqueries across millions of rows — no index strategy solves that at scale.

The Pattern, Applied Honestly

CQRS gave us a 300x performance improvement on the most critical user-facing feature of the platform. It turned an unusable 30-second wait into a responsive 100ms experience.

But it also gave us an eventual consistency model we must design around, a debugging surface area we must train for, and operational complexity we must monitor. These are not abstractions — they are real engineering costs that show up at 3 AM when an MV refresh starts timing out.

The pattern earns its complexity when the alternative — trying to serve both paths from a single model — is provably worse. In our case, it was. In most cases I've evaluated since, it isn't.

Use CQRS when no simpler alternative works. Not because it's architecturally elegant, but because it's architecturally necessary.

This is part of a series on building high-performance systems. See also: PostgreSQL Materialized Views for Segmentation (the read model deep dive), LRU Connection Pooling (the infrastructure beneath), and Azure Event Hub at Scale (the write pipeline).

Building Scalable Event Processing with Azure Event Hub

Sat, 24 Jan 2026 00:00:00 GMT

When your system needs to handle millions of events per minute, choosing the right architecture becomes critical. In this post, I'll share how we built a high-throughput event processing system using Azure Event Hub that handles 1M+ events per 5 minutes with room to spare.

The Challenge

Our multi-tenant SaaS platform needed to:

Process 3,333 events/second sustained (with 5,000-10,000 peak bursts)
Scale horizontally across Kubernetes pods
Maintain exactly-once processing guarantees
Handle pod failures gracefully without losing events

Architecture Overview

flowchart TB
    subgraph EH["Azure Event Hub (32 partitions)"]
        P1[Partition 0-10]
        P2[Partition 11-21]
        P3[Partition 22-31]
    end
    
    subgraph Pods["Kubernetes Pods"]
        Pod1[Pod 1]
        Pod2[Pod 2]
        Pod3[Pod 3]
    end
    
    subgraph Data["Data Layer"]
        PG[(PostgreSQL)]
        MY[(MySQL POS)]
        RD[(Redis)]
    end
    
    P1 --> Pod1
    P2 --> Pod2
    P3 --> Pod3
    
    Pod1 --> PG
    Pod2 --> MY
    Pod3 --> RD
    
    style EH fill:#112240,stroke:#64ffda,stroke-width:2px
    style Pods fill:#112240,stroke:#64ffda,stroke-width:2px

Key Design Decisions

32 Partitions: Each partition handles ~1MB/sec, giving us 32,000 events/sec capacity — 10x our target load.

Consumer Groups: All pods share the same consumer group, allowing Event Hub to automatically distribute partitions.

Horizontal Pod Autoscaling: Kubernetes HPA scales from 10-20 pods based on CPU (70%) and memory (80%) thresholds.

The Critical Piece: BlobCheckpointStore

Here's what many developers miss — without proper checkpoint management, your Event Hub consumers will fail at scale.

The Problem

// ❌ WRONG - No checkpoint store
this.consumer = new EventHubConsumerClient(
  consumerGroup,
  connectionString,
  eventHubName
  // Missing checkpointStore!
);

Without BlobCheckpointStore, you get:

No partition ownership coordination — Pods don't know who owns which partition
In-memory checkpoints lost — On pod restart, events are missed or reprocessed
No rebalancing — When pods scale, partitions don't redistribute
Starvation — Some pods may never receive events

The Solution

import { BlobCheckpointStore } from '@azure/eventhubs-checkpointstore-blob';
import { ContainerClient } from '@azure/storage-blob';

private async initializeConsumer() {
  // Create checkpoint store backed by Azure Blob Storage
  const containerClient = new ContainerClient(
    process.env.AZURE_STORAGE_CONNECTION_STRING,
    'eventhub-checkpoints'
  );
  const checkpointStore = new BlobCheckpointStore(containerClient);

  // Create consumer WITH checkpoint store
  this.consumer = new EventHubConsumerClient(
    consumerGroup,
    connectionString,
    eventHubName,
    checkpointStore  // ✅ This enables partition management
  );

  this.subscription = this.consumer.subscribe({
    processEvents: this.processEvents.bind(this),
    processError: this.processError.bind(this),
  });
}

How BlobCheckpointStore Works

It stores two critical pieces of information in Azure Blob Storage:

Azure Blob Storage
└── eventhub-checkpoints/
    ├── ownership/
    │   ├── partition-0 → "pod-1-uuid" (expires in 60s)
    │   ├── partition-1 → "pod-2-uuid"
    │   └── partition-2 → "pod-3-uuid"
    └── checkpoint/
        ├── partition-0 → {offset: 1234, sequenceNumber: 567}
        ├── partition-1 → {offset: 2345, sequenceNumber: 678}
        └── partition-2 → {offset: 3456, sequenceNumber: 789}

Automatic Rebalancing:

Pod 3 dies → ownership expires in 60s → Pod 1 or 2 claims partition-2
Pod 4 starts → steals a partition from the busiest pod
Scale down → remaining pods claim orphaned partitions

Connection Management: The 94.7% Hit Rate Secret

With multi-tenant architecture, each tenant has their own database. Creating a new connection for every event would be catastrophic at scale.

LRU-Cached Connection Pooling

class TenantConnectionManager {
  private cache: LRUCache<string, DataSource>;
  private refCounts: Map<string, number>;

  constructor() {
    this.cache = new LRUCache({
      max: 20,  // Keep 20 most recently used connections
      ttl: 600000,  // 10-minute idle timeout
      dispose: (ds) => ds.destroy()  // Clean up on eviction
    });
  }

  async getConnection(tenantId: string): Promise<DataSource> {
    const cached = this.cache.get(tenantId);
    if (cached) {
      this.refCounts.set(tenantId, (this.refCounts.get(tenantId) || 0) + 1);
      return cached;  // ✅ Cache hit
    }

    // Create new connection
    const dataSource = await this.createDataSource(tenantId);
    this.cache.set(tenantId, dataSource);
    return dataSource;
  }

  releaseConnection(tenantId: string) {
    const refs = (this.refCounts.get(tenantId) || 1) - 1;
    this.refCounts.set(tenantId, refs);
    // Connection stays in cache for future requests
  }
}

Results

Metric	Before	After
Cache Hit Rate	0%	94.7%
Memory per Pod	320 MB	110 MB
Connection Setup Time	Every request	Only on cache miss

Eliminating N+1 Queries: 100x Performance Boost

One of the biggest performance wins came from fixing N+1 query patterns.

The Anti-Pattern

// ❌ SLOW - N+1 queries
for (const customer of customers) {
  const pastTrigger = await checkIfSent(customer.id);  // 1000 queries!
}

The Fix

// ✅ FAST - Single batch query
const pastTriggers = await queryRunner.manager.find(TriggerLogEntity, {
  where: { 
    customer_id: In(customerIds), 
    trigger_id 
  }
});

const sentSet = new Set(pastTriggers.map(t => t.customer_id));

for (const customer of customers) {
  if (sentSet.has(customer.id)) continue;  // O(1) lookup
  await sendNotification(customer);
}

Impact

1,000 customers: 1,000 queries → 1 query
Latency: 3,000ms → 30ms (100x faster)

Resilience Patterns

1. Duplicate Detection with Redis

async processEvent(event: EventData) {
  const lockKey = `event:${event.systemProperties.sequenceNumber}`;
  
  // Try to acquire lock (NX = only if not exists)
  const acquired = await redis.set(lockKey, '1', 'NX', 'EX', 300);
  
  if (!acquired) {
    logger.debug('Event already being processed, skipping');
    return;  // Another pod is handling this
  }

  try {
    await this.handleEvent(event);
  } finally {
    // Lock expires automatically after 5 minutes
  }
}

2. Health Check with Auto-Reconnect

private lastEventTime: Date;

@Cron('*/2 * * * *')  // Every 2 minutes
async healthCheck() {
  const minutesSinceLastEvent = 
    (Date.now() - this.lastEventTime.getTime()) / 60000;

  if (minutesSinceLastEvent > 5) {
    logger.warn('Consumer stale, reconnecting...');
    await this.consumer.close();
    await this.initializeConsumer();
  }
}

3. Exponential Backoff Retries

async processWithRetry(event: any, attempt = 0) {
  const DELAYS = [1000, 5000, 15000];  // 1s, 5s, 15s
  
  try {
    await this.processEvent(event);
  } catch (error) {
    if (attempt < 3 && this.isRetryable(error)) {
      await this.delay(DELAYS[attempt]);
      return this.processWithRetry(event, attempt + 1);
    }
    
    // Send to Dead Letter Queue after 3 failures
    await this.sendToDeadLetterQueue(event, error);
  }
}

Kubernetes HPA Configuration

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: event-processor
spec:
  minReplicas: 10
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
      - type: Pods
        value: 2
        periodSeconds: 60

Final Architecture

flowchart TB
    subgraph Ingestion["INGESTION LAYER"]
        EH["Azure Event Hub<br/>32 Partitions | 32K evt/sec"]
        BCS["BlobCheckpointStore<br/>Partition ownership + Checkpoints"]
    end
    
    subgraph Processing["PROCESSING LAYER"]
        P1["Pod 1<br/>333 e/s"]
        P2["Pod 2<br/>333 e/s"]
        P3["Pod 3<br/>333 e/s"]
        PN["Pod N<br/>333 e/s"]
        CM["Connection Manager<br/>LRU Cache | 94.7% hit rate"]
    end
    
    subgraph DataLayer["DATA LAYER"]
        PG[(PostgreSQL<br/>Primary)]
        MY[(MySQL<br/>POS Tenants)]
        RD[(Redis<br/>Cache/DLQ)]
    end
    
    EH --> BCS
    BCS --> P1
    BCS --> P2
    BCS --> P3
    BCS --> PN
    
    P1 --> CM
    P2 --> CM
    P3 --> CM
    PN --> CM
    
    CM --> PG
    CM --> MY
    CM --> RD
    
    style EH fill:#4ecdc4,stroke:#333,stroke-width:2px
    style CM fill:#64ffda,stroke:#333,stroke-width:2px

Key Takeaways

BlobCheckpointStore is essential — Without it, scaling is impossible
Cache your connections — 94.7% hit rate saves massive resources
Batch your queries — N+1 patterns kill performance
Plan for failures — Retries, DLQ, and health checks are non-negotiable
Monitor everything — You can't optimize what you can't measure

Performance Results

Metric	Target	Achieved
Throughput	3,333 evt/s	✅ 3,333 evt/s
Peak Capacity	10,000 evt/s	✅ 32,000 evt/s
Latency (p50)	<100ms	✅ 50ms
Latency (p99)	<1s	✅ 500ms
Error Rate	<1%	✅ <0.1%
Availability	99.9%	✅ 99.9%

Building high-scale systems requires careful attention to these details. The difference between a system that works and one that works at scale is often in the implementation patterns you choose. See also: PostgreSQL Materialized Views for Segmentation, LRU Connection Pooling, and CQRS in Practice.

Questions? Connect with me on LinkedIn.

Achieving 94.7% Cache Hit Rate with LRU Connection Pooling

Tue, 20 Jan 2026 00:00:00 GMT

In a multi-tenant SaaS architecture, database connection management can make or break your application's performance. Here's how we achieved a 94.7% cache hit rate and 65% memory reduction using LRU-cached connection pooling.

The Multi-Tenant Challenge

Our platform serves 100+ tenants, each with their own isolated database. The naive approach:

// ❌ Creating a new connection per request
async function processEvent(tenantId: string, event: Event) {
  const connection = await createConnection({
    host: tenantConfig[tenantId].host,
    database: tenantConfig[tenantId].database,
    // ... other config
  });
  
  try {
    await processWithConnection(connection, event);
  } finally {
    await connection.close();
  }
}

Why This Fails at Scale

Events/sec	Connections Created	Memory Usage	Connection Time
100	100/sec	~500MB	3,000ms overhead
1,000	1,000/sec	~2GB	30,000ms overhead
3,333	3,333/sec	❌ OOM	❌ System crash

At 3,333 events/second, creating new connections for each request is simply not viable.

The Solution: LRU-Cached Connection Manager

flowchart TB
    subgraph CM["Connection Manager"]
        direction TB
        subgraph LRU["LRU Cache (max: 20)"]
            T1["Tenant A<br/>refs: 5"]
            T2["Tenant B<br/>refs: 2"]
            T3["Tenant C<br/>refs: 0"]
        end
        
        T1 --> D1[DataSource A]
        T2 --> D2[DataSource B]
        T3 --> D3[DataSource C]
        
        style D3 stroke-dasharray: 5 5
        Note["(idle, evictable)"] -.-> D3
    end
    
    CM --> DBs[(PostgreSQL / MySQL<br/>Tenant Databases)]

    style CM fill:#112240,stroke:#64ffda,stroke-width:2px
    style LRU fill:#233554,stroke:#64ffda
    style DBs fill:#112240,stroke:#64ffda,stroke-width:2px

Implementation

import { LRUCache } from 'lru-cache';
import { DataSource } from 'typeorm';

class TenantConnectionManager {
  private cache: LRUCache<string, DataSource>;
  private refCounts: Map<string, number> = new Map();
  private connectionConfigs: Map<string, TenantConfig> = new Map();

  constructor() {
    this.cache = new LRUCache({
      max: 20,  // Maximum 20 concurrent tenant connections
      ttl: 10 * 60 * 1000,  // 10-minute idle timeout
      
      // Clean up when evicting
      dispose: async (dataSource, key) => {
        const refs = this.refCounts.get(key) || 0;
        if (refs === 0) {
          await dataSource.destroy();
          console.log(`Evicted idle connection for tenant: ${key}`);
        }
      },
      
      // Don't evict if actively in use
      disposeAfter: async (dataSource, key) => {
        if ((this.refCounts.get(key) || 0) > 0) {
          return false;  // Prevent eviction
        }
        return true;
      }
    });
  }

  async getConnection(tenantId: string): Promise<DataSource> {
    // Check cache first
    let dataSource = this.cache.get(tenantId);
    
    if (dataSource) {
      // ✅ Cache HIT - increment reference count
      this.incrementRef(tenantId);
      return dataSource;
    }

    // Cache MISS - create new connection
    dataSource = await this.createDataSource(tenantId);
    this.cache.set(tenantId, dataSource);
    this.incrementRef(tenantId);
    
    return dataSource;
  }

  releaseConnection(tenantId: string): void {
    this.decrementRef(tenantId);
    // Connection stays in cache for future requests
  }

  private incrementRef(tenantId: string): void {
    const current = this.refCounts.get(tenantId) || 0;
    this.refCounts.set(tenantId, current + 1);
  }

  private decrementRef(tenantId: string): void {
    const current = this.refCounts.get(tenantId) || 1;
    this.refCounts.set(tenantId, Math.max(0, current - 1));
  }

  private async createDataSource(tenantId: string): Promise<DataSource> {
    const config = await this.getTenantConfig(tenantId);
    
    const dataSource = new DataSource({
      type: 'postgres',
      host: config.host,
      port: config.port,
      database: config.database,
      username: config.username,
      password: config.password,
      
      // Connection pool settings
      poolSize: 5,
      extra: {
        idleTimeoutMillis: 30000,
        connectionTimeoutMillis: 5000,
      }
    });

    await dataSource.initialize();
    return dataSource;
  }
}

Using the Connection Manager

class EventProcessor {
  constructor(
    private connectionManager: TenantConnectionManager
  ) {}

  async processEvent(tenantId: string, event: Event) {
    const dataSource = await this.connectionManager.getConnection(tenantId);
    
    try {
      const queryRunner = dataSource.createQueryRunner();
      await queryRunner.connect();
      
      try {
        await this.handleEvent(queryRunner, event);
      } finally {
        await queryRunner.release();
      }
    } finally {
      // Always release the connection reference
      this.connectionManager.releaseConnection(tenantId);
    }
  }
}

Reference Counting: The Secret Sauce

The key to safe connection management is reference counting:

Event 1 arrives for Tenant A
  → getConnection('A') → refs: 1
  
Event 2 arrives for Tenant A (before Event 1 completes)
  → getConnection('A') → refs: 2 (cache hit!)
  
Event 1 completes
  → releaseConnection('A') → refs: 1
  
Event 2 completes
  → releaseConnection('A') → refs: 0
  
After 10 minutes idle
  → Connection eligible for eviction → refs: 0 ✓

This prevents the catastrophic scenario of closing a connection while it's still in use.

Handling Connection Health

Connections can go stale. Here's how we handle health checks:

async getConnection(tenantId: string): Promise<DataSource> {
  let dataSource = this.cache.get(tenantId);
  
  if (dataSource) {
    // Verify connection is still healthy
    if (await this.isHealthy(dataSource)) {
      this.incrementRef(tenantId);
      return dataSource;
    }
    
    // Connection is stale - remove and recreate
    this.cache.delete(tenantId);
    await dataSource.destroy();
  }

  // Create fresh connection
  dataSource = await this.createDataSource(tenantId);
  this.cache.set(tenantId, dataSource);
  this.incrementRef(tenantId);
  
  return dataSource;
}

private async isHealthy(dataSource: DataSource): Promise<boolean> {
  try {
    await dataSource.query('SELECT 1', [], { timeout: 5000 });
    return true;
  } catch (error) {
    console.warn('Connection health check failed:', error.message);
    return false;
  }
}

Metrics and Monitoring

Track your cache performance:

class TenantConnectionManager {
  private metrics = {
    hits: 0,
    misses: 0,
    evictions: 0,
    healthCheckFailures: 0,
  };

  async getConnection(tenantId: string): Promise<DataSource> {
    const cached = this.cache.get(tenantId);
    
    if (cached) {
      this.metrics.hits++;
    } else {
      this.metrics.misses++;
    }
    
    // ... rest of implementation
  }

  getStats() {
    const total = this.metrics.hits + this.metrics.misses;
    return {
      hitRate: total > 0 ? (this.metrics.hits / total * 100).toFixed(2) : 0,
      totalRequests: total,
      activeConnections: this.cache.size,
      ...this.metrics
    };
  }
}

Our Production Metrics

After deploying this solution:

%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#64ffda', 'mainBkg': '#112240'}}}%%
pie title Connection Status
    "Cache Hits (94.7%)" : 94.7
    "Cache Misses (5.3%)" : 5.3

[!NOTE] Production Metrics Dashboard

Memory per Pod: 110 MB (was 320 MB)

Active Connections: 15-18 (of 20 max)

Avg Get Time (Hit): 0.3ms

Avg Get Time (Miss): 45ms

Eviction Rate: ~2/hour

Common Pitfalls to Avoid

1. Forgetting to Release Connections

// ❌ BAD - Connection leak
async processEvent(tenantId: string, event: Event) {
  const ds = await this.connectionManager.getConnection(tenantId);
  await this.handleEvent(ds, event);
  // Forgot to release! Refs will never reach 0
}

// ✅ GOOD - Always use try/finally
async processEvent(tenantId: string, event: Event) {
  const ds = await this.connectionManager.getConnection(tenantId);
  try {
    await this.handleEvent(ds, event);
  } finally {
    this.connectionManager.releaseConnection(tenantId);
  }
}

2. Not Handling Pool Exhaustion

async createDataSource(tenantId: string): Promise<DataSource> {
  const config = await this.getTenantConfig(tenantId);
  
  return new DataSource({
    // ...
    poolSize: 5,
    extra: {
      // ✅ Add timeouts to prevent hanging
      connectionTimeoutMillis: 5000,
      idleTimeoutMillis: 30000,
      
      // ✅ Queue connections if pool is exhausted
      max: 5,
      waitForConnections: true,
    }
  });
}

3. Ignoring Connection Errors

async getConnection(tenantId: string): Promise<DataSource> {
  try {
    // ... normal flow
  } catch (error) {
    // ✅ Log and handle gracefully
    console.error(`Failed to get connection for tenant ${tenantId}:`, error);
    
    // Remove bad entry from cache
    this.cache.delete(tenantId);
    
    // Throw specific error for retry logic
    throw new ConnectionError(tenantId, error);
  }
}

Results Summary

Metric	Before	After	Improvement
Memory per Pod	320 MB	110 MB	65% reduction
Cache Hit Rate	0%	94.7%	∞
Connection Setup	Every request	~2/hour	99.9% reduction
P99 Latency	150ms	50ms	3x faster
Max Throughput	~500 evt/s	~3,500 evt/s	7x increase

Key Takeaways

LRU caching is essential for multi-tenant connection management
Reference counting prevents premature connection closure
Health checks ensure you don't return stale connections
Monitoring helps you tune cache size and TTL parameters
Always use try/finally to prevent connection leaks

This pattern has been running in production for months, handling 1M+ events per 5 minutes with rock-solid stability.

For more details on the full architecture, check out my posts on Building Scalable Event Processing with Azure Event Hub, PostgreSQL Materialized Views for Segmentation, and CQRS in Practice.

From 30s to 100ms: PostgreSQL Materialized Views for Customer Segmentation

Thu, 15 Jan 2026 00:00:00 GMT

A deep dive into PostgreSQL Materialized Views for high-performance customer segmentation

The Problem

Our multi-tenant CRM platform allows marketing teams to create customer segments with complex filters like:

Customers with active subscriptions at Location A
Who made 3+ transactions in the last 30 days
With wallet credit > $50

Users could combine 65+ filter attributes across 7 tables with AND/OR conditions. The UI needed to show a live count as users built segments.

The pain point? Response times of 30+ seconds made the feature unusable.

graph LR
    subgraph "Current Performance"
        A[User Adds Filter] --> B[API Call]
        B --> C[Complex SQL Query]
        C --> D[30s Wait...]
        D --> E[Count Displayed]
    end

    style D fill:#ff6b6b,stroke:#333,stroke-width:2px

Root Cause Analysis

The Query Pattern

Each filter generated IN (SELECT...) subqueries:

SELECT COUNT(DISTINCT customer.id)
FROM customer
WHERE
  customer.id IN (
    SELECT customer_id FROM subscription_account
    WHERE status = 'active'
  )
  AND customer.id IN (
    SELECT customer_id FROM transaction_log
    WHERE created_at > NOW() - INTERVAL '30 days'
    GROUP BY customer_id HAVING COUNT(*) >= 3
  )
  AND customer.id IN (
    SELECT customer_id FROM wallet_account
    WHERE balance > 50
  )

The Performance Killer

Issue	Impact
Each subquery = full table scan	O(n) per filter
No covering indexes	Index-only scans impossible
COUNT DISTINCT on millions	Memory pressure
7 potential JOINs	Cartesian explosion risk

With millions of rows in transaction tables and 100+ tenants, this was a recipe for disaster.

The Solution: Materialized Views

Instead of computing aggregates at query time, we pre-compute them and store in materialized views.

Architecture Overview

flowchart TB
    subgraph Source["Source Tables"]
        SA[subscription_account]
        TL[transaction_log]
        WA[wallet_account]
        PA[package_account]
    end

    subgraph MVs["Materialized Views"]
        MV1["customer_attributes_mv<br/>Subscription, Wallet, Package aggregates"]
        MV2["customer_transactions_mv<br/>Transaction aggregates"]
    end

    subgraph API["API Layer"]
        SAPI[Segment Filter API]
    end

    SA --> MV1
    WA --> MV1
    PA --> MV1
    TL --> MV2

    MV1 --> SAPI
    MV2 --> SAPI

    style MV1 fill:#0d3026,color:#64ffda,stroke:#64ffda
    style MV2 fill:#0d3026,color:#64ffda,stroke:#64ffda

Two-View Strategy

We split into two MVs based on refresh frequency needs:

View	Data	Refresh Interval	Reason
`customer_attributes_mv`	Subscriptions, Wallets, Packages	45 seconds	High-value, frequently filtered
`customer_transactions_mv`	Transactions, Visits, Purchases	5 minutes	Large volume, less time-sensitive

MV Design: Pre-computed Aggregates

customer_attributes_mv

CREATE MATERIALIZED VIEW customer_attributes_mv AS
WITH
sub_stats AS (
    SELECT
        customer_id,
        COUNT(*) as subscription_count,
        BOOL_OR(status = 1) as is_active_subscriber,
        ARRAY_AGG(DISTINCT location_code) as location_codes,
        ARRAY_AGG(DISTINCT package_id) as package_ids,
        MAX(next_bill_date) as next_bill_date
    FROM subscription_account
    GROUP BY customer_id
),
wal_stats AS (
    SELECT
        customer_id,
        COUNT(*) as wallet_count,
        SUM(balance) as total_balance
    FROM wallet_account
    GROUP BY customer_id
)
SELECT
    c.id,
    c.customer_uuid,
    c.first_name,
    c.last_name,
    c.email,
    c.phone,
    -- Pre-computed flags (no subqueries needed at query time!)
    COALESCE(sub.is_active_subscriber, false) as is_active_subscriber,
    COALESCE(sub.subscription_count, 0) as subscription_count,
    sub.location_codes,
    sub.package_ids,
    sub.next_bill_date,
    -- Wallet aggregates
    COALESCE(wal.wallet_count, 0) as wallet_count,
    COALESCE(wal.total_balance, 0) as wallet_balance
FROM customer c
LEFT JOIN sub_stats sub ON c.customer_uuid = sub.customer_id
LEFT JOIN wal_stats wal ON c.customer_uuid = wal.customer_id;

GIN Indexes for Array Queries

The magic for multi-select filters:

-- Fast "customer has package X" lookups
CREATE INDEX idx_attr_packages_gin
  ON customer_attributes_mv USING GIN (package_ids);

-- Fast "customer visited location Y" lookups
CREATE INDEX idx_attr_locations_gin
  ON customer_attributes_mv USING GIN (location_codes);

-- Unique index required for CONCURRENTLY refresh
CREATE UNIQUE INDEX idx_attr_mv_id
  ON customer_attributes_mv (id);

Query Transformation

Before: Subquery Hell (30+ seconds)

SELECT COUNT(DISTINCT id) FROM customer
WHERE id IN (
  SELECT customer_id FROM subscription_account WHERE status = 1
)
AND id IN (
  SELECT customer_id FROM wallet_account WHERE balance > 50
);

After: Direct MV Query (<100ms)

SELECT COUNT(*) FROM customer_attributes_mv
WHERE is_active_subscriber = true
  AND wallet_balance > 50;

Array Contains with GIN Index

-- "Customer has package 123 OR 456"
SELECT COUNT(*) FROM customer_attributes_mv
WHERE package_ids && ARRAY[123, 456];

-- "Customer visited location 'LOC_A' AND 'LOC_B'"
SELECT COUNT(*) FROM customer_attributes_mv
WHERE location_codes @> ARRAY['LOC_A', 'LOC_B'];

Operators:

&& = overlaps (ANY)
@> = contains (ALL)

Refresh Strategy

The Challenge

Multi-tenant: 100+ separate databases
Data freshness: <2 minute staleness acceptable
Zero downtime: Queries must work during refresh

Solution: Status Table + Concurrent Refresh

-- Track when data changed vs when MV was refreshed
CREATE TABLE mv_refresh_status (
    mv_name VARCHAR(100) PRIMARY KEY,
    last_refresh_at TIMESTAMP,
    last_data_change_at TIMESTAMP,
    is_refreshing BOOLEAN DEFAULT false
);

The Critical Keyword: CONCURRENTLY

-- ❌ BLOCKS all queries during refresh (bad!)
REFRESH MATERIALIZED VIEW customer_attributes_mv;

-- ✅ Allows concurrent reads (good!)
REFRESH MATERIALIZED VIEW CONCURRENTLY customer_attributes_mv;

Requirement: Unique index on MV for CONCURRENTLY to work.

How CONCURRENTLY Works Internally

sequenceDiagram
    participant Writer as Data Writer
    participant Source as Source Tables
    participant Refresh as REFRESH Process
    participant MV as Materialized View
    participant Reader as Query Reader

    Note over Refresh: Takes snapshot at T0
    Refresh->>Source: Read snapshot T0

    Writer->>Source: INSERT new row T1
    Note over Source: Row committed

    Reader->>MV: SELECT COUNT
    Note over MV: Returns old data (T0)

    Refresh->>MV: Atomic swap T2
    Note over MV: New data includes T0 NOT T1

    Note over Reader: Next query sees T0 data
    Note over Writer: T1 appears in next refresh

Key insight: Writes during refresh appear in the next refresh cycle, not the current one. This is why we accept 1-2 minute staleness.

TypeScript Implementation

interface RefreshConfig {
  mvName: string;
  intervalMs: number;
  debounceMs: number;
}

class MVRefreshService {
  private readonly configs: RefreshConfig[] = [
    { mvName: 'customer_attributes_mv', intervalMs: 45000, debounceMs: 10000 },
    { mvName: 'customer_transactions_mv', intervalMs: 300000, debounceMs: 60000 },
  ];

  @Cron('*/30 * * * * *') // Every 30 seconds
  async checkAndRefresh() {
    for (const tenant of await this.getTenants()) {
      const connection = await this.getConnection(tenant.id);

      for (const config of this.configs) {
        await this.refreshIfNeeded(connection, config);
      }
    }
  }

  private async refreshIfNeeded(
    connection: DataSource,
    config: RefreshConfig
  ): Promise<void> {
    const status = await this.getRefreshStatus(connection, config.mvName);

    // Skip if already refreshing
    if (status?.is_refreshing) return;

    // Skip if no data changes since last refresh
    if (status?.last_data_change_at <= status?.last_refresh_at) return;

    // Debounce: wait for writes to settle
    const timeSinceChange = Date.now() - status.last_data_change_at.getTime();
    if (timeSinceChange < config.debounceMs) return;

    // Perform refresh
    await this.executeRefresh(connection, config.mvName);
  }

  private async executeRefresh(
    connection: DataSource,
    mvName: string
  ): Promise<void> {
    const queryRunner = connection.createQueryRunner();

    try {
      // Mark as refreshing
      await queryRunner.query(
        `UPDATE mv_refresh_status SET is_refreshing = true WHERE mv_name = $1`,
        [mvName]
      );

      // Increase work_mem for complex aggregations
      await queryRunner.query(`SET work_mem = '128MB'`);

      // The magic: CONCURRENTLY allows reads during refresh
      await queryRunner.query(
        `REFRESH MATERIALIZED VIEW CONCURRENTLY ${mvName}`
      );

      // Update status
      await queryRunner.query(
        `UPDATE mv_refresh_status
         SET last_refresh_at = NOW(), is_refreshing = false
         WHERE mv_name = $1`,
        [mvName]
      );

      this.logger.log(`Refreshed ${mvName} successfully`);
    } catch (error) {
      await queryRunner.query(
        `UPDATE mv_refresh_status SET is_refreshing = false WHERE mv_name = $1`,
        [mvName]
      );
      throw error;
    } finally {
      await queryRunner.release();
    }
  }
}

Performance Results

Before vs After

Scenario	Before	After	Improvement
Is Active Subscriber	30s	50ms	600x faster
Subscriber + Wallet > $50	35s	80ms	437x faster
3+ Transactions in 30 days	45s	120ms	375x faster
Complex 5-filter segment	60s	200ms	300x faster

Production Metrics

┌────────────────────────────────────────────────────────┐
│         Segment Filter API - Production Stats          │
├────────────────────────────────────────────────────────┤
│                                                        │
│  Response Time (p50):    45ms   ████░░░░░░░░          │
│  Response Time (p95):    120ms  █████████░░░          │
│  Response Time (p99):    250ms  ████████████          │
│                                                        │
│  MV Refresh Time:        2-5s   ██░░░░░░░░░░          │
│  Data Staleness:         <90s   ████░░░░░░░░          │
│                                                        │
│  Daily Queries:          50,000+                       │
│  Error Rate:             <0.01%                        │
└────────────────────────────────────────────────────────┘

Handling Edge Cases

1. High-Volume Data Sync Window

Our system receives bulk operational data during a fixed daily window. We extend the debounce to avoid refresh storms:

private getDebounceMs(config: RefreshConfig): number {
  const currentHour = new Date().getHours();
  const isDuringBulkWindow = currentHour >= 13 && currentHour < 14;

  // 5 min debounce during bulk sync, normal otherwise
  return isDuringBulkWindow ? 300000 : config.debounceMs;
}

2. MV Doesn't Exist Yet (New Tenant)

Graceful fallback to legacy query:

async getSegmentCount(tenantId: string, filters: Filter[]): Promise<number> {
  const connection = await this.getConnection(tenantId);
  const mvExists = await this.checkMVExists(connection, 'customer_attributes_mv');

  if (mvExists && this.canUseMV(filters)) {
    return this.queryMV(connection, filters);  // Fast path
  } else {
    return this.legacyQuery(connection, filters);  // Slow fallback
  }
}

3. Unsupported Filters

Some filters can't be pre-computed (e.g., free-text search). We detect and route:

private canUseMV(filters: Filter[]): boolean {
  const MV_SUPPORTED = ['is_active_subscriber', 'wallet_balance', 'package_ids', 'location_codes'];
  return filters.every(f => MV_SUPPORTED.includes(f.attribute));
}

Lessons Learned

✅ What Worked

Separate MVs by refresh needs — Attributes (45s) vs Transactions (5min)
GIN indexes for arrays — Perfect for "any of these values" queries
CONCURRENTLY refresh — Zero query downtime
Status table pattern — Only refresh when data actually changed
Debouncing — Prevents refresh storms during bulk writes

⚠️ Gotchas

Index count matters — 30+ GIN indexes slow down refresh significantly
ARRAY_AGG ordering — Add ORDER BY for deterministic results
Multi-tenant complexity — Each DB needs separate MV management
Unique index required — CONCURRENTLY fails without it

Key Takeaways

Pre-compute what you filter on — Don't make PostgreSQL work hard at query time
Array columns + GIN indexes — Perfect for multi-select filters
CONCURRENTLY is non-negotiable — Never block production reads
Staleness is acceptable — Most UIs can tolerate 1-2 minute delays
Monitor refresh times — They grow as data grows

For our use case, trading 1-2 minute data staleness for 300-600x performance improvement was an easy decision. Users went from abandoning the feature to actively using it for complex targeting campaigns.

This is part of a series on building high-performance systems. Check out my other posts on Azure Event Hub Architecture, LRU Connection Pooling, and CQRS in Practice.