Module P-7 — Connecting External Services and Caching

Redis and Caching with ioredis

Redis is an in-memory data store. The two primary use cases in a Node.js API are caching (store the result of an expensive query, serve it for subsequent requests) and session/token storage (store refresh tokens so they can be revoked).

bash
npm install ioredis

Redis client singleton

typescript
// src/db/redis.ts
import Redis from 'ioredis';
import { env } from '../config/env.js';

const redis = new Redis(env.REDIS_URL, {
  maxRetriesPerRequest: 3,
  retryStrategy: (times) => Math.min(times * 50, 2000), // exponential backoff, max 2s
  enableReadyCheck: true,
  lazyConnect: false,
});

redis.on('connect', () => console.log('[Redis] Connected'));
redis.on('error', (err) => console.error('[Redis] Error:', err.message));

export default redis;

The Cache-Aside Pattern

The most common caching strategy: check the cache first, hit the database only on a miss, then populate the cache for next time.

typescript
// src/services/posts.service.ts
import redis from '../db/redis.js';
import * as postsRepo from '../repositories/posts.repository.js';
import { NotFoundError } from '../errors/AppError.js';

const CACHE_TTL = 300; // 5 minutes in seconds

export async function getPostById(id: number) {
  const cacheKey = `post:${id}`;

  // 1. Check cache
  const cached = await redis.get(cacheKey);
  if (cached) {
    return JSON.parse(cached);
  }

  // 2. Cache miss — query the database
  const post = await postsRepo.findById(id);
  if (!post) throw new NotFoundError('Post');

  // 3. Populate cache with TTL
  await redis.setex(cacheKey, CACHE_TTL, JSON.stringify(post));

  return post;
}

Cache invalidation

The cache must be invalidated when data changes — stale cache is worse than no cache:

typescript
export async function updatePost(id: number, data: UpdatePostInput, userId: number) {
  const post = await postsRepo.findById(id);
  if (!post) throw new NotFoundError('Post');
  if (post.authorId !== userId) throw new ForbiddenError();

  const updated = await postsRepo.update(id, data);

  // Invalidate after successful write
  await redis.del(`post:${id}`);

  return updated;
}

export async function deletePost(id: number, userId: number) {
  const post = await postsRepo.findById(id);
  if (!post) throw new NotFoundError('Post');
  if (post.authorId !== userId) throw new ForbiddenError();

  await postsRepo.delete(id);
  await redis.del(`post:${id}`);
}

Caching list queries

Lists are harder — when a post is created, which lists are stale? The simplest approach: use short TTLs for lists and longer TTLs for individual records.

typescript
export async function listPublishedPosts(page: number, limit: number) {
  const cacheKey = `posts:published:${page}:${limit}`;
  const cached = await redis.get(cacheKey);
  if (cached) return JSON.parse(cached);

  const posts = await postsRepo.findPublished({ page, limit });

  // Short TTL for lists — 60 seconds is fine for most feeds
  await redis.setex(cacheKey, 60, JSON.stringify(posts));

  return posts;
}

Redis for refresh token storage

Using Redis instead of a Postgres table for refresh tokens — faster lookups, automatic expiry:

typescript
// src/repositories/tokens.repository.ts
import redis from '../db/redis.js';

const TOKEN_TTL = 7 * 24 * 60 * 60; // 7 days in seconds

export async function create(userId: number, token: string): Promise<void> {
  // Store with user ID for revoke-all-sessions capability
  await redis.setex(`refresh:${token}`, TOKEN_TTL, String(userId));
}

export async function findByToken(token: string): Promise<number | null> {
  const userId = await redis.get(`refresh:${token}`);
  return userId ? parseInt(userId) : null;
}

export async function deleteByToken(token: string): Promise<void> {
  await redis.del(`refresh:${token}`);
}

// Revoke all sessions for a user (requires a different data structure)
// See the Set-based approach in the Architect phase for multi-device logout

Sending Email with Nodemailer

Nodemailer is the standard Node.js library for sending email. In production you connect it to a transactional email provider (SendGrid, Resend, Postmark, SES) — never your own SMTP server.

bash
npm install nodemailer
npm install -D @types/nodemailer

Email transport singleton

typescript
// src/services/email.service.ts
import nodemailer from 'nodemailer';
import { env } from '../config/env.js';

// Transactional email via SMTP (works with SendGrid, Resend, Mailgun, SES)
const transporter = nodemailer.createTransport({
  host: env.SMTP_HOST,
  port: env.SMTP_PORT,
  secure: env.SMTP_PORT === 465, // true for port 465, false for 587
  auth: {
    user: env.SMTP_USER,
    pass: env.SMTP_PASS,
  },
});

// For development — log emails to console instead of sending
const devTransport = nodemailer.createTransport({
  jsonTransport: true, // logs to console
});

const transport = env.NODE_ENV === 'production' ? transporter : devTransport;

Sending emails

typescript
interface OrderConfirmationData {
  to: string;
  orderNumber: string;
  items: Array<{ name: string; quantity: number; price: number }>;
  total: number;
}

export async function sendOrderConfirmation(data: OrderConfirmationData): Promise<void> {
  const itemsList = data.items
    .map(item => `${item.name} × ${item.quantity} — $${item.price.toFixed(2)}`)
    .join('\n');

  await transport.sendMail({
    from: `"My Shop" <noreply@myshop.com>`,
    to: data.to,
    subject: `Order #${data.orderNumber} confirmed`,
    text: `
Your order has been confirmed.

Items:
${itemsList}

Total: $${data.total.toFixed(2)}

Thank you for your order!
    `.trim(),
    html: `
<h2>Order #${data.orderNumber} Confirmed</h2>
<table>
  ${data.items.map(item => `
    <tr>
      <td>${item.name}</td>
      <td>× ${item.quantity}</td>
      <td>$${item.price.toFixed(2)}</td>
    </tr>
  `).join('')}
</table>
<p><strong>Total: $${data.total.toFixed(2)}</strong></p>
    `,
  });
}

export async function sendPasswordReset(to: string, resetToken: string): Promise<void> {
  const resetUrl = `${env.APP_URL}/reset-password?token=${resetToken}`;

  await transport.sendMail({
    from: `"My Shop" <noreply@myshop.com>`,
    to,
    subject: 'Password reset request',
    text: `Click the link to reset your password: ${resetUrl}\n\nThis link expires in 1 hour.`,
    html: `<p>Click <a href="${resetUrl}">here</a> to reset your password.</p><p>This link expires in 1 hour.</p>`,
  });
}

Don't block the request on email

Email sending should never make the response wait:

typescript
// src/services/orders.service.ts
export async function createOrder(input) {
  const order = await ordersRepo.create(input);

  // Fire-and-forget — the response goes out immediately
  // If the email fails, we log it and move on. The order was placed.
  sendOrderConfirmation({
    to: user.email,
    orderNumber: order.id.toString(),
    items: order.items,
    total: order.total,
  }).catch(err => {
    console.error(`[Email] Failed to send order confirmation for order ${order.id}:`, err.message);
  });

  return order;
}

For reliable email delivery in high-volume apps, put emails on a job queue (covered in P-12, BullMQ) instead of sending inline.

File Uploads to Object Storage (S3/R2)

Never store uploaded files on the API server's disk. Servers are ephemeral in cloud deployments, and local disk doesn't scale horizontally. Store files in object storage: AWS S3, Cloudflare R2, DigitalOcean Spaces (all S3-compatible).

bash
npm install @aws-sdk/client-s3 @aws-sdk/s3-request-presigner multer
npm install -D @types/multer

Multer for parsing multipart uploads

typescript
// src/middleware/upload.ts
import multer from 'multer';

export const upload = multer({
  storage: multer.memoryStorage(), // hold in memory, we'll stream to S3
  limits: {
    fileSize: 5 * 1024 * 1024, // 5 MB max
    files: 1,
  },
  fileFilter: (req, file, cb) => {
    const allowed = ['image/jpeg', 'image/png', 'image/webp'];
    if (allowed.includes(file.mimetype)) {
      cb(null, true);
    } else {
      cb(new Error('Only JPEG, PNG, and WebP images are allowed'));
    }
  },
});

S3 client singleton

typescript
// src/db/s3.ts
import { S3Client } from '@aws-sdk/client-s3';
import { env } from '../config/env.js';

export const s3 = new S3Client({
  region: env.AWS_REGION,
  credentials: {
    accessKeyId: env.AWS_ACCESS_KEY_ID,
    secretAccessKey: env.AWS_SECRET_ACCESS_KEY,
  },
  // For Cloudflare R2 or other S3-compatible services:
  // endpoint: env.S3_ENDPOINT,
  // forcePathStyle: true,
});

Upload service

typescript
// src/services/storage.service.ts
import { PutObjectCommand, DeleteObjectCommand } from '@aws-sdk/client-s3';
import { s3 } from '../db/s3.js';
import { env } from '../config/env.js';
import { randomUUID } from 'crypto';
import path from 'path';

export async function uploadFile(
  file: Express.Multer.File,
  folder = 'uploads',
): Promise<string> {
  const ext = path.extname(file.originalname).toLowerCase();
  const key = `${folder}/${randomUUID()}${ext}`;

  await s3.send(new PutObjectCommand({
    Bucket: env.S3_BUCKET,
    Key: key,
    Body: file.buffer,
    ContentType: file.mimetype,
    ContentLength: file.size,
    // ACL: 'public-read',  // only if bucket is public
  }));

  // Return the public URL (for public buckets)
  return `https://${env.S3_BUCKET}.s3.${env.AWS_REGION}.amazonaws.com/${key}`;
}

export async function deleteFile(url: string): Promise<void> {
  // Extract key from URL
  const key = new URL(url).pathname.slice(1); // remove leading /
  await s3.send(new DeleteObjectCommand({ Bucket: env.S3_BUCKET, Key: key }));
}

Upload route

typescript
// src/routes/users.routes.ts
router.patch(
  '/:id/avatar',
  authenticate,
  upload.single('avatar'),       // multer middleware — expects field named "avatar"
  validate(idParamSchema, 'params'),
  usersController.updateAvatar,
);

// src/controllers/users.controller.ts
export const updateAvatar = asyncHandler(async (req, res) => {
  if (!req.file) throw new ValidationError('Avatar file is required');

  const avatarUrl = await storageService.uploadFile(req.file, 'avatars');
  const user = await usersService.updateAvatar(req.params.id, avatarUrl, req.user!.id);
  res.json(user);
});

Pre-signed URLs for client-side uploads

For large files, upload directly from the client to S3 — skip your API server entirely:

typescript
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { PutObjectCommand } from '@aws-sdk/client-s3';

export async function getPresignedUploadUrl(
  filename: string,
  contentType: string,
): Promise<{ uploadUrl: string; key: string }> {
  const key = `uploads/${randomUUID()}-${filename}`;

  const command = new PutObjectCommand({
    Bucket: env.S3_BUCKET,
    Key: key,
    ContentType: contentType,
  });

  const uploadUrl = await getSignedUrl(s3, command, { expiresIn: 300 }); // 5 min
  return { uploadUrl, key };
}

The client receives uploadUrl, PUTs the file directly to S3, then sends just the key to your API to record the uploaded file.

Outbound HTTP with fetch

Node.js 18+ ships a global fetch — no extra libraries needed for most use cases.

typescript
// Calling a third-party API
const response = await fetch('https://api.stripe.com/v1/charges', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${env.STRIPE_SECRET_KEY}`,
    'Content-Type': 'application/x-www-form-urlencoded',
  },
  body: new URLSearchParams({
    amount: '5000',
    currency: 'usd',
    source: 'tok_visa',
  }),
});

if (!response.ok) {
  const error = await response.json();
  throw new AppError(`Stripe error: ${error.error?.message}`, response.status);
}

const charge = await response.json();

Wrapper with timeout and error handling

typescript
// src/utils/httpClient.ts
interface FetchOptions extends RequestInit {
  timeoutMs?: number;
}

export async function httpFetch<T>(url: string, options: FetchOptions = {}): Promise<T> {
  const { timeoutMs = 10_000, ...fetchOptions } = options;

  const controller = new AbortController();
  const timeoutId = setTimeout(() => controller.abort(), timeoutMs);

  try {
    const response = await fetch(url, {
      ...fetchOptions,
      signal: controller.signal,
    });

    clearTimeout(timeoutId);

    if (!response.ok) {
      const body = await response.text();
      throw new AppError(
        `HTTP ${response.status} from ${new URL(url).hostname}: ${body.slice(0, 200)}`,
        response.status >= 500 ? 502 : response.status,
      );
    }

    return response.json() as Promise<T>;
  } catch (err) {
    clearTimeout(timeoutId);
    if (err instanceof Error && err.name === 'AbortError') {
      throw new AppError(`Request to ${new URL(url).hostname} timed out after ${timeoutMs}ms`, 504);
    }
    throw err;
  }
}

Idempotent Webhook Processing

Webhooks are HTTP requests sent by third-party services (Stripe, GitHub, Twilio) to notify your API of events. Two rules for production webhook handling:

1. Verify the signature — don't trust the payload without checking it came from the real sender.
2. Process idempotently — webhook delivery is at-least-once. The same event may arrive twice.

typescript
// src/controllers/webhooks.controller.ts
import crypto from 'crypto';
import { asyncHandler } from '../utils/asyncHandler.js';
import redis from '../db/redis.js';

export const stripeWebhook = asyncHandler(async (req, res) => {
  // 1. Verify signature (Stripe-specific — each provider has its own approach)
  const signature = req.headers['stripe-signature'] as string;
  const payload = req.body; // must be raw buffer — don't use express.json() on this route

  const expectedSig = crypto
    .createHmac('sha256', env.STRIPE_WEBHOOK_SECRET)
    .update(payload)
    .digest('hex');

  if (!crypto.timingSafeEqual(
    Buffer.from(signature.split(',').find(s => s.startsWith('v1='))!.slice(3)),
    Buffer.from(expectedSig),
  )) {
    return res.status(401).json({ error: 'Invalid signature' });
  }

  const event = JSON.parse(payload.toString());

  // 2. Idempotency check — have we processed this event before?
  const idempotencyKey = `webhook:processed:${event.id}`;
  const alreadyProcessed = await redis.get(idempotencyKey);

  if (alreadyProcessed) {
    return res.status(200).json({ received: true }); // acknowledge without reprocessing
  }

  // 3. Process the event
  switch (event.type) {
    case 'payment_intent.succeeded':
      await ordersService.confirmPayment(event.data.object.metadata.orderId);
      break;
    case 'customer.subscription.deleted':
      await subscriptionsService.cancelSubscription(event.data.object.id);
      break;
    default:
      // Unhandled event type — that's fine, acknowledge and move on
  }

  // 4. Mark as processed (TTL of 7 days covers retry windows)
  await redis.setex(idempotencyKey, 7 * 24 * 60 * 60, '1');

  res.status(200).json({ received: true });
});

The webhook route needs raw body access — register it before express.json():

typescript
// src/app.ts
// Raw body for webhooks — before express.json()
app.post(
  '/webhooks/stripe',
  express.raw({ type: 'application/json' }),
  webhooksController.stripeWebhook,
);

// JSON body for everything else
app.use(express.json({ limit: '10kb' }));

Summary

• Redis caching: cache-aside pattern — check cache, query DB on miss, populate cache, invalidate on write. Use short TTLs (60s) for lists, longer (5 min) for individual records.
• Refresh token storage in Redis: setex with matching TTL — tokens auto-expire without cleanup jobs.
• Email with Nodemailer: connect to a transactional provider (Resend, SendGrid, SES). Fire-and-forget with .catch() for non-critical emails; use a job queue for critical ones.
• File uploads: multer parses multipart form data, stream to S3 from memory. Pre-signed URLs let large files bypass your server entirely.
• Outbound HTTP: native fetch with AbortController for timeouts. Centralise in a wrapper that handles non-OK responses uniformly.
• Webhook idempotency: verify signature + deduplicate with Redis before processing. Acknowledge with 200 immediately to prevent provider retries.

Next: structured logging with Pino, correlation IDs that make logs searchable, multi-stage Docker builds, PM2 for production process management, and deploying via GitHub Actions CI/CD.