Achieving 94.7% Cache Hit Rate with LRU Connection Pooling

In a multi-tenant SaaS architecture, database connection management can make or break your application’s performance. Here’s how we achieved a 94.7% cache hit rate and 65% memory reduction using LRU-cached connection pooling.

The Multi-Tenant Challenge

Our platform serves 100+ tenants, each with their own isolated database. The naive approach:

// ❌ Creating a new connection per request
async function processEvent(tenantId: string, event: Event) {
  const connection = await createConnection({
    host: tenantConfig[tenantId].host,
    database: tenantConfig[tenantId].database,
    // ... other config
  });
  
  try {
    await processWithConnection(connection, event);
  } finally {
    await connection.close();
  }
}

Why This Fails at Scale

Events/sec	Connections Created	Memory Usage	Connection Time
100	100/sec	~500MB	3,000ms overhead
1,000	1,000/sec	~2GB	30,000ms overhead
3,333	3,333/sec	❌ OOM	❌ System crash

At 3,333 events/second, creating new connections for each request is simply not viable.

---

The Solution: LRU-Cached Connection Manager

flowchart TB
    subgraph CM["Connection Manager"]
        direction TB
        subgraph LRU["LRU Cache (max: 20)"]
            T1["Tenant A<br/>refs: 5"]
            T2["Tenant B<br/>refs: 2"]
            T3["Tenant C<br/>refs: 0"]
        end
        
        T1 --> D1[DataSource A]
        T2 --> D2[DataSource B]
        T3 --> D3[DataSource C]
        
        style D3 stroke-dasharray: 5 5
        Note["(idle, evictable)"] -.-> D3
    end
    
    CM --> DBs[(PostgreSQL / MySQL<br/>Tenant Databases)]

    style CM fill:#112240,stroke:#64ffda,stroke-width:2px
    style LRU fill:#233554,stroke:#64ffda
    style DBs fill:#112240,stroke:#64ffda,stroke-width:2px

Implementation

import { LRUCache } from 'lru-cache';
import { DataSource } from 'typeorm';

class TenantConnectionManager {
  private cache: LRUCache<string, DataSource>;
  private refCounts: Map<string, number> = new Map();
  private connectionConfigs: Map<string, TenantConfig> = new Map();

  constructor() {
    this.cache = new LRUCache({
      max: 20,  // Maximum 20 concurrent tenant connections
      ttl: 10 * 60 * 1000,  // 10-minute idle timeout
      
      // Clean up when evicting
      dispose: async (dataSource, key) => {
        const refs = this.refCounts.get(key) || 0;
        if (refs === 0) {
          await dataSource.destroy();
          console.log(`Evicted idle connection for tenant: ${key}`);
        }
      },
      
      // Don't evict if actively in use
      disposeAfter: async (dataSource, key) => {
        if ((this.refCounts.get(key) || 0) > 0) {
          return false;  // Prevent eviction
        }
        return true;
      }
    });
  }

  async getConnection(tenantId: string): Promise<DataSource> {
    // Check cache first
    let dataSource = this.cache.get(tenantId);
    
    if (dataSource) {
      // ✅ Cache HIT - increment reference count
      this.incrementRef(tenantId);
      return dataSource;
    }

    // Cache MISS - create new connection
    dataSource = await this.createDataSource(tenantId);
    this.cache.set(tenantId, dataSource);
    this.incrementRef(tenantId);
    
    return dataSource;
  }

  releaseConnection(tenantId: string): void {
    this.decrementRef(tenantId);
    // Connection stays in cache for future requests
  }

  private incrementRef(tenantId: string): void {
    const current = this.refCounts.get(tenantId) || 0;
    this.refCounts.set(tenantId, current + 1);
  }

  private decrementRef(tenantId: string): void {
    const current = this.refCounts.get(tenantId) || 1;
    this.refCounts.set(tenantId, Math.max(0, current - 1));
  }

  private async createDataSource(tenantId: string): Promise<DataSource> {
    const config = await this.getTenantConfig(tenantId);
    
    const dataSource = new DataSource({
      type: 'postgres',
      host: config.host,
      port: config.port,
      database: config.database,
      username: config.username,
      password: config.password,
      
      // Connection pool settings
      poolSize: 5,
      extra: {
        idleTimeoutMillis: 30000,
        connectionTimeoutMillis: 5000,
      }
    });

    await dataSource.initialize();
    return dataSource;
  }
}

Using the Connection Manager

class EventProcessor {
  constructor(
    private connectionManager: TenantConnectionManager
  ) {}

  async processEvent(tenantId: string, event: Event) {
    const dataSource = await this.connectionManager.getConnection(tenantId);
    
    try {
      const queryRunner = dataSource.createQueryRunner();
      await queryRunner.connect();
      
      try {
        await this.handleEvent(queryRunner, event);
      } finally {
        await queryRunner.release();
      }
    } finally {
      // Always release the connection reference
      this.connectionManager.releaseConnection(tenantId);
    }
  }
}

---

Reference Counting: The Secret Sauce

The key to safe connection management is reference counting:

Event 1 arrives for Tenant A
  → getConnection('A') → refs: 1
  
Event 2 arrives for Tenant A (before Event 1 completes)
  → getConnection('A') → refs: 2 (cache hit!)
  
Event 1 completes
  → releaseConnection('A') → refs: 1
  
Event 2 completes
  → releaseConnection('A') → refs: 0
  
After 10 minutes idle
  → Connection eligible for eviction → refs: 0 ✓

This prevents the catastrophic scenario of closing a connection while it’s still in use.

---

Handling Connection Health

Connections can go stale. Here’s how we handle health checks:

async getConnection(tenantId: string): Promise<DataSource> {
  let dataSource = this.cache.get(tenantId);
  
  if (dataSource) {
    // Verify connection is still healthy
    if (await this.isHealthy(dataSource)) {
      this.incrementRef(tenantId);
      return dataSource;
    }
    
    // Connection is stale - remove and recreate
    this.cache.delete(tenantId);
    await dataSource.destroy();
  }

  // Create fresh connection
  dataSource = await this.createDataSource(tenantId);
  this.cache.set(tenantId, dataSource);
  this.incrementRef(tenantId);
  
  return dataSource;
}

private async isHealthy(dataSource: DataSource): Promise<boolean> {
  try {
    await dataSource.query('SELECT 1', [], { timeout: 5000 });
    return true;
  } catch (error) {
    console.warn('Connection health check failed:', error.message);
    return false;
  }
}

---

Metrics and Monitoring

Track your cache performance:

class TenantConnectionManager {
  private metrics = {
    hits: 0,
    misses: 0,
    evictions: 0,
    healthCheckFailures: 0,
  };

  async getConnection(tenantId: string): Promise<DataSource> {
    const cached = this.cache.get(tenantId);
    
    if (cached) {
      this.metrics.hits++;
    } else {
      this.metrics.misses++;
    }
    
    // ... rest of implementation
  }

  getStats() {
    const total = this.metrics.hits + this.metrics.misses;
    return {
      hitRate: total > 0 ? (this.metrics.hits / total * 100).toFixed(2) : 0,
      totalRequests: total,
      activeConnections: this.cache.size,
      ...this.metrics
    };
  }
}

Our Production Metrics

After deploying this solution:

%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#64ffda', 'mainBkg': '#112240'}}}%%
pie title Connection Status
    "Cache Hits (94.7%)" : 94.7
    "Cache Misses (5.3%)" : 5.3

[!NOTE] Production Metrics Dashboard

• Memory per Pod: 110 MB (was 320 MB)
• Active Connections: 15-18 (of 20 max)
• Avg Get Time (Hit): 0.3ms
• Avg Get Time (Miss): 45ms
• Eviction Rate: ~2/hour

---

Common Pitfalls to Avoid

1. Forgetting to Release Connections

// ❌ BAD - Connection leak
async processEvent(tenantId: string, event: Event) {
  const ds = await this.connectionManager.getConnection(tenantId);
  await this.handleEvent(ds, event);
  // Forgot to release! Refs will never reach 0
}

// ✅ GOOD - Always use try/finally
async processEvent(tenantId: string, event: Event) {
  const ds = await this.connectionManager.getConnection(tenantId);
  try {
    await this.handleEvent(ds, event);
  } finally {
    this.connectionManager.releaseConnection(tenantId);
  }
}

2. Not Handling Pool Exhaustion

async createDataSource(tenantId: string): Promise<DataSource> {
  const config = await this.getTenantConfig(tenantId);
  
  return new DataSource({
    // ...
    poolSize: 5,
    extra: {
      // ✅ Add timeouts to prevent hanging
      connectionTimeoutMillis: 5000,
      idleTimeoutMillis: 30000,
      
      // ✅ Queue connections if pool is exhausted
      max: 5,
      waitForConnections: true,
    }
  });
}

3. Ignoring Connection Errors

async getConnection(tenantId: string): Promise<DataSource> {
  try {
    // ... normal flow
  } catch (error) {
    // ✅ Log and handle gracefully
    console.error(`Failed to get connection for tenant ${tenantId}:`, error);
    
    // Remove bad entry from cache
    this.cache.delete(tenantId);
    
    // Throw specific error for retry logic
    throw new ConnectionError(tenantId, error);
  }
}

---

Results Summary

Metric	Before	After	Improvement
Memory per Pod	320 MB	110 MB	65% reduction
Cache Hit Rate	0%	94.7%	∞
Connection Setup	Every request	~2/hour	99.9% reduction
P99 Latency	150ms	50ms	3x faster
Max Throughput	~500 evt/s	~3,500 evt/s	7x increase

---

Key Takeaways

1. LRU caching is essential for multi-tenant connection management
2. Reference counting prevents premature connection closure
3. Health checks ensure you don’t return stale connections
4. Monitoring helps you tune cache size and TTL parameters
5. Always use try/finally to prevent connection leaks

This pattern has been running in production for months, handling 1M+ events per 5 minutes with rock-solid stability.

---

For more details on the full architecture, check out my posts on Building Scalable Event Processing with Azure Event Hub, PostgreSQL Materialized Views for Segmentation, and CQRS in Practice.