Audit Logging produces a dedicated, structured, durable, tamper-evident record of security- and business-relevant actions: who did what to which resource, when, from where, with what outcome — and often why.

The problem

Audit requirements come from many directions:

• Regulation and standards — financial regulators, PCI DSS, SOX, HIPAA, ISO 27001, data-protection laws that require logging access to personal data.
• Security — detecting and investigating account takeover, insider misuse, privilege escalation.
• Disputes and support — “I never changed my nominee”, “the price was different when I ordered”.
• Operations — who changed the production configuration before the incident?

Application logs fail these needs because they’re optimized for debugging, not evidence.

How it works

flowchart TD
    U[User / service / admin] --> S[Service performs action]
    S -->|same transaction| OB[(Outbox: AuditEvent)]
    OB --> RL[Relay]
    RL --> T[(Audit topic<br/>restricted ACLs)]
    T --> AS[Audit service<br/>validate, enrich, hash-chain]
    AS --> STORE[(Append-only store<br/>WORM / Object Lock)]
    AS --> SIEM[SIEM<br/>Splunk / Sentinel / Elastic Security]
    STORE --> Q[Audit search UI<br/>for compliance & support]

Core properties of a good audit trail:

What to capture

{
  "auditId": "7d3c8e5a-4f21-4b9e-9a0d-61f2b5c0e8a4",
  "occurredAt": "2026-10-17T09:53:41.218Z",
  "actor": {
    "type": "USER",
    "id": "emp-20931",
    "displayName": "R. Iyer",
    "roles": ["CREDIT_OFFICER"],
    "authMethod": "SSO+MFA",
    "onBehalfOf": null
  },
  "action": "credit-limit.increase",
  "resource": { "type": "CardAccount", "id": "CA-0042117" },
  "outcome": "SUCCESS",
  "reason": "Customer request ref SR-88123; income proof verified",
  "changes": [
    { "field": "creditLimit", "from": "150000.00", "to": "250000.00" }
  ],
  "context": {
    "sourceIp": "10.20.4.18",
    "userAgent": "BranchPortal/5.4",
    "channel": "branch",
    "service": "card-limits",
    "serviceVersion": "2026.10.17",
    "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
    "approvalId": "APR-2026-55102"
  },
  "schemaVersion": 2
}

Typical auditable actions:

• Authentication — logins, failures, MFA changes, password resets, session revocations.
• Authorization changes — role grants, permission changes, admin elevation.
• Sensitive reads — viewing customer PII, exporting data, downloading statements.
• Business-critical writes — limit changes, payments, approvals, overrides, refunds, price changes.
• Configuration and operations — production config changes, feature-flag flips, deployments, key rotations.

Capture failures and denials too — a run of ACCESS_DENIED events is often the first sign of an attack.

Capturing reliably

Audit events must not be lost when the action succeeded, nor recorded when it rolled back.

• Outbox — write the audit event in the same transaction as the business change (Transactional Outbox). The most dependable approach for writes.
• Event sourcing — if the domain is event-sourced, the event store already is an audit trail of state changes (Event Sourcing); enrich events with actor and reason metadata.
• Gateway / middleware — capture sensitive reads and access decisions at a central policy point (API gateway, authorization service) so every endpoint is covered.
• Platform logs — cloud control-plane logs (AWS CloudTrail, Azure Activity Log, GCP Audit Logs) and Kubernetes audit logs cover infrastructure actions.

@Transactional
public void increaseLimit(CardAccountId id, Money newLimit, String reason, Actor actor) {
  CardAccount acct = accounts.load(id);
  Money old = acct.creditLimit();
  acct.changeLimit(newLimit, actor);                     // domain rules + approvals
  accounts.save(acct);

  outbox.save(AuditEvent.builder()                       // same transaction as the change
      .actor(actor)
      .action("credit-limit.increase")
      .resource("CardAccount", id.value())
      .outcome(Outcome.SUCCESS)
      .reason(reason)
      .change("creditLimit", old.toPlainString(), newLimit.toPlainString())
      .traceId(Tracing.currentTraceId())
      .build());
}

Tamper evidence

Make undetected modification impractical:

• Append-only storage with retention locks — S3 Object Lock (compliance mode), Azure immutable blob storage, WORM-capable archives.
• Hash chaining — each record includes the hash of the previous record; altering one breaks the chain from that point on.
• Periodic anchoring — sign or publish the latest chain hash (daily digest) to a separate system so the chain itself can’t be rewritten wholesale.
• Separation of duties — application teams can write audit events but not modify or delete them; a separate team administers the audit store.

import hashlib, json

def chain(prev_hash: str, record: dict) -> dict:
    body = json.dumps(record, sort_keys=True, separators=(",", ":"))
    record["prevHash"] = prev_hash
    record["hash"] = hashlib.sha256((prev_hash + body).encode()).hexdigest()
    return record

def verify(records: list[dict]) -> bool:
    prev = "GENESIS"
    for r in records:
        body = {k: v for k, v in r.items() if k not in ("prevHash", "hash")}
        expected = hashlib.sha256((prev + json.dumps(body, sort_keys=True, separators=(",", ":"))).encode()).hexdigest()
        if r["prevHash"] != prev or r["hash"] != expected:
            return False
        prev = r["hash"]
    return True

(Per-stream chains — per tenant or per day — keep this scalable; a single global chain becomes a serialization bottleneck.)

Audit logs vs. application logs

Privacy

Audit logs often contain personal data and must themselves comply with data-protection rules:

• Minimize — record identifiers and changed fields, not entire records.
• Mask sensitive values (card numbers, national IDs) in changes.
• Legal basis and retention — keep for the legally required period, then delete or anonymize.
• Erasure conflicts — where audit retention obligations override erasure requests, document the basis; consider pseudonymous actor IDs with a separately managed mapping.

Trade-offs

Common pitfalls

Warning: auditing via application log.info(). When log sampling, level changes, or rotation drop the line, the audit record is gone — and engineers can edit or delete it. Audit events need their own reliable path and storage.

• Recording the attempt, not the outcome — “user requested limit increase” without whether it succeeded.
• Losing the real actor behind service accounts — propagate the end-user identity (and any impersonation) through service calls.
• No audit of reads for sensitive data, which many regulations explicitly require.
• Free-text actions ("changed stuff") instead of a controlled vocabulary.

Tools and real-world usage

• Capture — outbox + Kafka, event stores, API gateway and policy-engine logs (OPA decision logs), Kubernetes audit logs, AWS CloudTrail / Azure Activity Logs / GCP Cloud Audit Logs.
• Storage — S3 Object Lock, Azure immutable storage, append-only databases or ledger tables (e.g., SQL Server ledger tables), dedicated audit services.
• Analysis — SIEM platforms such as Splunk, Microsoft Sentinel, Elastic Security, Google Security Operations.

In banking environments I’ve worked in, audit trails for document access and customer-data changes were as important as the features themselves — designed in from the first sprint and verified in every compliance review.

Related patterns

• Event Sourcing — state changes as a built-in audit trail.
• Transactional Outbox — reliable audit event capture.
• Log Aggregation — operational logs, kept separate from audit.
• Correlation ID and Distributed Tracing — linking audit events to the requests that caused them.

Key takeaways

1. Audit logs answer who did what to which resource, when, from where, with what outcome — and why.
2. Capture them reliably (outbox, event store, central policy point), including failures and sensitive reads.
3. Store them append-only and tamper-evident, with separation of duties.
4. Treat the audit trail as regulated data: strict schema, restricted access, defined retention.

The problem

Without metrics:

• you learn about outages from customers,
• you can’t tell whether a deploy made things worse,
• capacity planning is guesswork,
• and “is it slow?” is a matter of opinion.

Logs can’t replace metrics at scale: counting events by scanning logs is slow and expensive, and log-based alerts are brittle.

What to measure

The RED method (for request-driven services)

The USE method (for resources)

Google’s four golden signals — latency, traffic, errors, saturation — combine both views.

Business metrics

Technical metrics can look fine while the business is broken. Add a few business KPIs: orders placed per minute, payment success ratio, sign-ups, statement generation completed. A sudden drop in orders is often the fastest outage detector.

Metric types

Rule of thumb: use histograms for latency. You can compute percentiles across all instances (histogram_quantile), whereas averaging per-instance p99s is mathematically meaningless.

Example: Micrometer (Spring Boot)

Spring Boot Actuator with Micrometer exposes HTTP, JVM, pool, and Kafka metrics automatically at /actuator/prometheus. Add domain metrics explicitly:

@Service
public class PaymentService {

  private final Counter captured;
  private final Counter declined;
  private final Timer gatewayLatency;

  public PaymentService(MeterRegistry registry) {
    this.captured = Counter.builder("payments_captured_total")
        .description("Payments successfully captured").register(registry);
    this.declined = Counter.builder("payments_declined_total")
        .tag("reason", "unknown").register(registry);        // see note on tags below
    this.gatewayLatency = Timer.builder("payment_gateway_duration")
        .publishPercentileHistogram()                         // export histogram buckets
        .serviceLevelObjectives(Duration.ofMillis(300), Duration.ofSeconds(1))
        .tag("gateway", "primary")
        .register(registry);
  }

  public PaymentResult capture(Payment p) {
    PaymentResult r = gatewayLatency.record(() -> gateway.capture(p));
    if (r.isSuccess()) captured.increment(); else declined.increment();
    return r;
  }
}

management:
  endpoints.web.exposure.include: health,prometheus
  metrics:
    tags:                                  # common tags on every metric
      service: payments
      env: prod
    distribution:
      percentiles-histogram:
        http.server.requests: true

Example: Prometheus queries

# Request rate per route (RED: Rate)
sum by (uri) (rate(http_server_requests_seconds_count{service="payments"}[5m]))

# Error ratio (RED: Errors)
sum(rate(http_server_requests_seconds_count{service="payments", status=~"5.."}[5m]))
/
sum(rate(http_server_requests_seconds_count{service="payments"}[5m]))

# p99 latency across all instances (RED: Duration)
histogram_quantile(0.99,
  sum by (le) (rate(http_server_requests_seconds_bucket{service="payments"}[5m])))

# Saturation: Kafka consumer lag for the ledger group
sum by (topic) (kafka_consumergroup_lag{consumergroup="ledger"})

SLOs and alerting

Alert on symptoms users feel, not on every cause. Define Service Level Objectives and alert when you’re burning the error budget too fast.

flowchart TD
    SLI["SLI: % of checkout requests<br/>successful and < 800 ms"] --> SLO["SLO: 99.9% over 30 days"]
    SLO --> EB["Error budget: 0.1%<br/>≈ 43 min of 'bad' per month"]
    EB --> FAST["Fast burn alert<br/>budget burning 14× → page now"]
    EB --> SLOW["Slow burn alert<br/>budget burning 3× over 6 h → ticket"]

# Prometheus alerting rule: multi-window burn rate (page)
groups:
  - name: checkout-slo
    rules:
      - alert: CheckoutErrorBudgetFastBurn
        expr: |
          (
            sum(rate(http_server_requests_seconds_count{service="checkout",status=~"5.."}[5m]))
            / sum(rate(http_server_requests_seconds_count{service="checkout"}[5m]))
          ) > (14.4 * 0.001)
          and
          (
            sum(rate(http_server_requests_seconds_count{service="checkout",status=~"5.."}[1h]))
            / sum(rate(http_server_requests_seconds_count{service="checkout"}[1h]))
          ) > (14.4 * 0.001)
        for: 2m
        labels: { severity: page }
        annotations:
          summary: "Checkout is burning its 30-day error budget ~14x too fast"
          runbook: "https://runbooks.example.com/checkout/error-budget"

The two windows (5 m and 1 h) make the alert both fast and resistant to blips — the approach described in Google’s SRE Workbook.

Cardinality: the silent killer

Every unique combination of label values is a separate time series. Labels like user_id, order_id, email, or raw URL paths create millions of series and can take down your metrics backend.

Put unbounded identifiers in logs and traces; use exemplars to link a metric data point to a sample trace.

Dashboards that help

• One service overview per service: RED panels, saturation, dependencies, deploy markers.
• Top-down — start from user journeys (checkout, login) and drill into services.
• Consistent layouts and units across teams.
• Annotations for deployments and incidents, so cause and effect are visible.

Trade-offs

Common pitfalls

Pitfall: alerting on causes (CPU > 80%, one pod restarted) instead of symptoms (error budget burning). You get paged constantly for things users never notice, and alert fatigue hides the real incidents.

• Averages for latency — hide the tail; use percentiles from histograms.
• Counters reset on restart — always use rate()/increase(), never raw counter values.
• Missing saturation metrics — pools, queues, and lags warn you before errors start.
• Dashboards nobody owns — review and prune them.

Tools and real-world usage

• Prometheus (+ Alertmanager), Grafana, Thanos / Cortex / Mimir / VictoriaMetrics for long-term, multi-cluster storage.
• Instrumentation — Micrometer (Java), OpenTelemetry Metrics (all languages), prometheus_client libraries, .NET System.Diagnostics.Metrics.
• SaaS — Datadog, New Relic, Dynatrace, Grafana Cloud, cloud-native monitoring (CloudWatch, Azure Monitor, Cloud Monitoring).

Related patterns

• Distributed Tracing — exemplars link metrics to traces.
• Log Aggregation — investigate after a metric alert fires.
• Health Check — binary health vs. rich metrics.
• Dead Letter Queue and Circuit Breaker — DLQ depth and breaker state are key metrics.

Key takeaways

1. Measure RED for services, USE for resources, plus a few business KPIs.
2. Use counters, gauges, and histograms; compute percentiles from histograms.
3. Alert on SLO burn rates, not on every cause.
4. Keep label cardinality bounded; put IDs in logs and traces.

The problem

• Container logs are ephemeral — a crashed pod takes its logs with it.
• One request’s story spans many services (Correlation ID only helps if logs are searchable together).
• Unstructured text logs are slow and unreliable to search.
• Compliance and security teams need retained, tamper-resistant logs.

How it works

flowchart TD
    subgraph Node["Each Kubernetes node"]
      P1[Pod A<br/>JSON to stdout] --> F[/var/log/containers/*.log/]
      P2[Pod B<br/>JSON to stdout] --> F
      F --> AG[Agent DaemonSet<br/>Fluent Bit / OTel Collector / Vector]
    end
    AG -->|"enrich: k8s labels, namespace, node"| BUF[(Buffer<br/>Kafka — optional)]
    BUF --> PROC[Processing<br/>parse, redact, route, drop noise]
    PROC --> HOT[(Hot store<br/>Elasticsearch / Loki / Splunk)]
    PROC --> COLD[(Archive<br/>object storage)]
    HOT --> UI[Kibana / Grafana / Splunk search]

1. Emit — applications write structured logs (JSON) to stdout/stderr.
2. Collect — a node-level agent tails container log files (or receives OTLP logs), adding metadata: service, namespace, pod, node, version.
3. Process — parse, normalize fields, redact sensitive data, drop noise, route by type.
4. Store — index in a hot store for fast search; archive to cheap object storage for long retention.
5. Search, alert, and correlate — query by fields, pivot to traces via trace_id.

Structured logging

Structured logs are the single biggest improvement you can make.

{
  "ts": "2026-10-17T09:51:12.384Z",
  "level": "WARN",
  "service": "payments",
  "version": "2026.10.17",
  "env": "prod",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "correlation_id": "checkout-7f3a",
  "event": "payment.gateway.retry",
  "message": "Gateway timeout, retrying",
  "attempt": 2,
  "gateway": "primary",
  "duration_ms": 2003,
  "order_id": "ord-55120"
}

Guidelines:

• Consistent field names across services (adopt OpenTelemetry semantic conventions or ECS — Elastic Common Schema).
• An event name for each log type (payment.gateway.retry) — searchable and countable.
• IDs as fields — trace_id, correlation_id, business IDs (order, account) where policy allows.
• Levels used deliberately — ERROR means someone should look; INFO for business-significant events; DEBUG off in production by default.
• No secrets, tokens, passwords, card numbers, or unnecessary personal data.

# Python: structlog with JSON output and trace correlation
import structlog
from opentelemetry import trace

def add_trace(_, __, event_dict):
    ctx = trace.get_current_span().get_span_context()
    if ctx.is_valid:
        event_dict["trace_id"] = format(ctx.trace_id, "032x")
        event_dict["span_id"] = format(ctx.span_id, "016x")
    return event_dict

structlog.configure(processors=[
    structlog.contextvars.merge_contextvars,
    structlog.processors.add_log_level,
    structlog.processors.TimeStamper(fmt="iso", utc=True, key="ts"),
    add_trace,
    structlog.processors.JSONRenderer(),
])

log = structlog.get_logger(service="payments")
log.warning("payment.gateway.retry", attempt=2, gateway="primary", duration_ms=2003, order_id="ord-55120")

Collection: Fluent Bit example

[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    multiline.parser  docker, cri
    Tag               kube.*
    Mem_Buf_Limit     50MB
    Skip_Long_Lines   On

[FILTER]
    Name              kubernetes
    Match             kube.*
    Merge_Log         On          # parse JSON app logs into fields
    Keep_Log          Off
    Labels            On

[FILTER]
    Name              modify      # redact obvious secrets defensively
    Match             kube.*
    Remove            authorization
    Remove            password

[FILTER]
    Name              grep        # drop noisy health-check access logs
    Match             kube.*
    Exclude           path ^/actuator/health

[OUTPUT]
    Name              loki
    Match             kube.*
    Host              loki-gateway.observability
    Labels            job=fluent-bit, namespace=$kubernetes['namespace_name'], app=$kubernetes['labels']['app']

Choosing a backend

In practice: Loki labels should be low-cardinality (namespace, app, level). Putting order_id or user_id in labels explodes the index; keep those as fields in the log line.

Retention, cost, and noise

Logs are often the largest observability bill. Control it deliberately:

• Tiered retention — e.g., 7–14 days hot, 90 days warm, 1–7 years archived where regulation requires.
• Drop or sample noise — health checks, successful debug chatter, repetitive warnings.
• Log levels per environment and dynamic level changes for incidents.
• Prefer metrics for counts — don’t log every request just to count them; that’s what Application Metrics are for.
• Budget by service — show teams their log volume and cost.

Security and compliance

• Redact at the source first; filters are a safety net.
• Access control — logs often contain personal data; restrict by team and environment.
• Integrity — security-relevant logs may need tamper-evident storage (WORM object storage) and separation from application teams. For “who did what” records, see Audit Logging.
• Data residency — where logs are stored matters under data-protection laws.

Trade-offs

Common pitfalls

Pitfall: logging as the primary monitoring tool. Teams build alerts on regexes over log text — fragile, slow, and expensive. Use metrics for alerting; use logs for investigation.

• Unstructured, multi-line logs (stack traces split across events). Configure multiline parsing or log exceptions as single JSON fields.
• Writing logs to files inside containers instead of stdout, so agents never see them.
• No backpressure handling — a backend outage makes agents drop logs or exhaust node memory. Use buffers and limits.
• Different field names per team (traceId, trace_id, TraceID) — cross-service queries break.

Tools and real-world usage

• Collectors — Fluent Bit, Fluentd, Vector, OpenTelemetry Collector, Logstash.
• Backends — Elasticsearch/OpenSearch + Kibana/Dashboards, Grafana Loki, Splunk, Datadog, Elastic Cloud, cloud-native options (CloudWatch Logs, Azure Monitor Logs, Google Cloud Logging).
• Buffers — Kafka between collectors and backends for high volume and resilience.

Related patterns

• Correlation ID — makes aggregated logs joinable.
• Distributed Tracing — pivot from a log line to its trace.
• Application Metrics — for counting and alerting.
• Audit Logging — durable, tamper-evident records of user actions.

Key takeaways

1. Emit structured JSON logs to stdout with consistent fields, including trace and correlation IDs.
2. Collect with node agents, enrich with platform metadata, redact, and route.
3. Choose a backend by query needs and cost profile; keep index labels low-cardinality.
4. Manage retention and noise deliberately — and use metrics, not logs, for alerting.

The problem

In a distributed system:

• latency is the sum and overlap of many calls — which one is slow?
• errors surface far from their cause — a 500 at the gateway may originate in an inventory query three hops away;
• asynchronous hops (queues, topics) break the obvious request/response chain;
• each team sees only its own service’s logs and metrics.

How it works

A trace represents one end-to-end operation. It’s composed of spans — named, timed operations with attributes, events, and status — linked by parent/child relationships.

gantt
    title Trace 4bf92f35… — POST /checkout (612 ms)
    dateFormat  x
    axisFormat  %L ms
    section Gateway
    POST /checkout                 :0, 612
    section Checkout svc
    PlaceOrder                     :8, 600
    validate cart                  :10, 40
    section Pricing svc
    GET /quote                     :42, 120
    section Inventory svc
    POST /reservations             :125, 455
    SELECT stock FOR UPDATE (slow) :130, 440
    section Payments svc
    POST /payments                 :458, 590

Reading the waterfall: the inventory SELECT … FOR UPDATE dominates the request — a lock-contention problem, not a CPU one.

Key concepts:

OpenTelemetry: the standard

OpenTelemetry (OTel) — a CNCF project — provides vendor-neutral APIs, SDKs, auto-instrumentation, a wire protocol (OTLP), and the Collector. Instrument once; send to Jaeger, Tempo, X-Ray, Honeycomb, Datadog, or anything else.

flowchart TD
    A[Service A<br/>OTel SDK / agent] -->|OTLP| COL
    B[Service B<br/>OTel SDK / agent] -->|OTLP| COL
    M[Mesh proxies] -->|OTLP / Zipkin| COL
    subgraph COL["OpenTelemetry Collector"]
      R[receivers] --> P[processors<br/>batch, tail sampling,<br/>attribute scrubbing] --> E[exporters]
    end
    E --> J[(Jaeger / Tempo)]
    E --> V[(Vendor backend)]

Auto-instrumentation (Java)

The Java agent instruments common frameworks (Spring, JDBC, Kafka clients, HTTP clients) with no code changes:

java -javaagent:/otel/opentelemetry-javaagent.jar \
     -Dotel.service.name=checkout \
     -Dotel.resource.attributes=deployment.environment=prod,service.version=2026.10.17 \
     -Dotel.exporter.otlp.endpoint=http://otel-collector:4317 \
     -Dotel.traces.sampler=parentbased_traceidratio \
     -Dotel.traces.sampler.arg=0.10 \
     -jar checkout.jar

Manual spans for business operations

Auto-instrumentation covers I/O; add spans and attributes for business meaning:

import { trace, SpanStatusCode } from "@opentelemetry/api";

const tracer = trace.getTracer("checkout");

export async function placeOrder(cart: Cart) {
  return tracer.startActiveSpan("PlaceOrder", async (span) => {
    span.setAttributes({
      "order.line_count": cart.lines.length,
      "order.total": cart.total,            // business context, no PII
      "customer.tier": cart.customerTier,
    });
    try {
      const quote = await pricing.quote(cart);        // child CLIENT span via auto-instrumentation
      const reservation = await inventory.reserve(cart);
      span.addEvent("stock.reserved", { "reservation.id": reservation.id });
      return await payments.capture(cart, quote);
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR, message: (err as Error).message });
      throw err;
    } finally {
      span.end();
    }
  });
}

Tracing across messaging

Asynchronous hops need explicit propagation in message headers. OTel’s Kafka, RabbitMQ, and SQS instrumentations inject traceparent on produce and extract it on consume, creating PRODUCER and CONSUMER spans.

• For one message → one handler, the consumer span is a child of the producer span.
• For batch consumers, use span links to each message’s context rather than a single parent.
• With an outbox, store the trace context in the outbox row so the relay’s publish span links back to the original request (Transactional Outbox).

Sampling

Tracing every request at high volume is expensive. Choose a sampling strategy:

# OpenTelemetry Collector: tail sampling — keep all errors and slow traces, 5% of the rest
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }
  attributes/scrub:
    actions:
      - { key: http.request.header.authorization, action: delete }
      - { key: user.email, action: delete }

Using traces well

• Start from symptoms — jump from a slow-latency alert or error log line to exemplar traces (metrics with trace exemplars, logs with trace_id).
• Look for patterns, not just single traces: sequential calls that could be parallel, N+1 queries, retries, lock waits.
• Correlate the three signals — traces, metrics, logs — via trace IDs and shared resource attributes (service.name, deployment.environment).
• Service graphs derived from spans show real dependencies — often surprising ones.

Trade-offs

Common pitfalls

Pitfall: broken context. One service uses an HTTP client that isn’t instrumented, or a thread pool drops context, and the trace splits into disconnected fragments. Standardize on OTel auto-instrumentation and test propagation end-to-end in CI.

• Head sampling at 1% and then wondering why the failing request has no trace — use tail sampling for errors.
• High-cardinality span names (GET /orders/8812) instead of route templates (GET /orders/{id}).
• PII in attributes — scrub at the SDK or Collector.
• Tracing only the backend — include gateways, meshes, and (where useful) the browser/mobile client.

Tools and real-world usage

• OpenTelemetry SDKs, auto-instrumentation agents, and the Collector.
• Backends — Jaeger, Zipkin, Grafana Tempo (open source); AWS X-Ray, Google Cloud Trace, Azure Monitor; Honeycomb, Datadog, New Relic, Dynatrace, Elastic APM.
• Meshes (Istio, Linkerd) emit spans for every hop; application spans connect them internally.

Related patterns

• Correlation ID — the trace ID doubles as one.
• Log Aggregation — logs stamped with trace IDs.
• Application Metrics — metrics with exemplars that link to traces.
• Service Mesh — automatic spans at the network layer.

Key takeaways

1. A trace is a tree of timed spans for one request, stitched together by propagated context.
2. Use OpenTelemetry: auto-instrument I/O, add business spans and attributes.
3. Propagate context through messages, outboxes, and thread pools.
4. Use tail sampling to keep errors and slow traces, and scrub sensitive data.

The problem

• Instance addresses change with every deploy and scale event.
• Some instances are unhealthy, starting up, or draining.
• Hard-coded addresses or manually updated load-balancer pools break, lag, or route to dead instances.
• Different environments (dev, staging, prod, regions) need different addresses for the same logical service.

How it works

Every discovery system has a service registry — a database of service name → healthy instances — plus a way for instances to register and for callers to look up.

Registration

Third-party registration is generally more reliable: the platform knows when it starts and stops instances, and services don’t need registry code.

Lookup

flowchart TD
    subgraph CS["Client-side discovery"]
      C1[Caller with discovery client] -->|1 · query 'payments'| R1[(Registry)]
      R1 -->|2 · instances list| C1
      C1 -->|3 · pick one, call directly| I1[payments instance]
    end
    subgraph SS["Server-side discovery"]
      C2[Caller] -->|1 · call payments.svc| LB[Load balancer / proxy]
      LB -->|2 · looks up| R2[(Registry)]
      LB -->|3 · forwards| I2[payments instance]
    end

Service meshes blur the line: a local sidecar proxy does client-side-style load balancing on behalf of the app, configured via the mesh’s discovery (Envoy’s xDS APIs).

Kubernetes: discovery built in

In Kubernetes, discovery is a platform feature:

apiVersion: v1
kind: Service
metadata: { name: payments, namespace: shop }
spec:
  selector: { app: payments }       # pods with this label are endpoints
  ports: [{ name: http, port: 80, targetPort: 8080 }]
---
# Only pods passing their readinessProbe appear in the EndpointSlice.
# Callers simply use DNS:
#   http://payments                       (same namespace)
#   http://payments.shop.svc.cluster.local (fully qualified)

• Endpoints/EndpointSlices are maintained by the control plane from pod labels and readiness (Health Check).
• CoreDNS resolves service names to a stable ClusterIP; kube-proxy (or eBPF in Cilium) load-balances to endpoints.
• Headless Services (clusterIP: None) return pod IPs directly — useful for client-side load balancing (e.g., gRPC) and stateful sets.

Consul: discovery across platforms

Consul spans VMs, Kubernetes, and multiple data centers, with health checks and DNS/HTTP interfaces.

# Service definition registered with the local Consul agent
service {
  name = "payments"
  port = 8080
  tags = ["v2", "primary"]
  meta = { version = "2026.10.17" }

  check {
    id       = "payments-http"
    http     = "http://localhost:8080/actuator/health/readiness"
    interval = "10s"
    timeout  = "2s"
    deregister_critical_service_after = "1m"
  }
}

# DNS interface: only healthy instances are returned
dig @127.0.0.1 -p 8600 payments.service.consul SRV

# HTTP API with health filtering
curl -s "http://127.0.0.1:8500/v1/health/service/payments?passing=true"

Eureka + Spring Cloud: client-side discovery

# Service registers itself and fetches the registry
spring:
  application:
    name: checkout
eureka:
  client:
    service-url: { defaultZone: "http://eureka-1:8761/eureka/,http://eureka-2:8761/eureka/" }
    registry-fetch-interval-seconds: 10
  instance:
    prefer-ip-address: true
    lease-renewal-interval-in-seconds: 10
    lease-expiration-duration-in-seconds: 30

@Bean
@LoadBalanced                       // resolves "http://payments" via the registry, client-side
RestClient.Builder restClientBuilder() { return RestClient.builder(); }

// Usage
restClientBuilder.build().post().uri("http://payments/v1/payments").body(req).retrieve();

Design considerations

• Health integration — the registry must reflect readiness, not just “process exists”. Stale entries route traffic to dead instances.
• Caching and TTLs — clients cache lookups. DNS TTLs, JVM DNS caching (networkaddress.cache.ttl), and registry fetch intervals determine how fast callers notice changes. Long caches → traffic to terminated instances; very short → registry load.
• Graceful deregistration — on shutdown, deregister/fail readiness before stopping, and let in-flight requests drain.
• Registry availability — the registry is critical infrastructure; run it highly available. Good clients keep using their last-known instance list if the registry is briefly unreachable (Eureka’s self-preservation mode is an example).
• Locality — prefer instances in the same zone/region to cut latency and cross-zone costs (topology-aware routing).
• Environment naming — use namespaces, data centers, or tags rather than embedding environment names in service names.

Trade-offs

Common pitfalls

Pitfall: JVM DNS caching. Some JVM configurations cache successful DNS lookups indefinitely (notably when a security manager is installed), so a client keeps calling an IP that no longer exists. Set networkaddress.cache.ttl to a small value (e.g., 30–60 s) in environments where IPs change.

• Long-lived connections (HTTP/2, gRPC) pinned to old instances — new instances get no traffic. Use client-side balancing with periodic re-resolution or connection max-age.
• Self-registration without deregistration — crashed instances linger until their lease expires.
• Discovery that ignores readiness — new pods receive traffic before they’re warm.
• Hard-coding cluster-internal DNS names into clients outside the cluster.

Tools and real-world usage

• Kubernetes Services, EndpointSlices, CoreDNS; headless Services for client-side load balancing.
• HashiCorp Consul — multi-platform registry with health checks, DNS, and service mesh.
• Netflix Eureka with Spring Cloud — client-side discovery popular in Spring estates.
• AWS Cloud Map, ECS Service Connect, Azure Container Apps built-in discovery.
• etcd / ZooKeeper — underlying registries for many systems; Envoy xDS — dynamic endpoint discovery for proxies.

Related patterns

• Health Check — determines which instances are discoverable.
• Service Mesh — builds on discovery to route and secure traffic.
• API Gateway — uses discovery to reach upstreams.
• Sidecar — local proxies that perform discovery-driven load balancing.

Key takeaways

1. A registry maps service names to healthy instances; instances register themselves or are registered by the platform.
2. Client-side discovery gives smarter balancing; server-side is language-agnostic.
3. On Kubernetes, use Services and DNS — readiness controls membership.
4. Watch caching, long-lived connections, and graceful deregistration.

The problem

Without a mesh, each service team must implement — in each language — mutual TLS, retries, timeouts, circuit breaking, traffic splitting, and per-call metrics. In practice:

• internal traffic is often unencrypted and unauthenticated (“it’s inside the cluster”);
• resilience policies are inconsistent across teams;
• there’s no uniform view of service-to-service health;
• canary releases and traffic shifting require custom code or load-balancer tickets.

How it works

flowchart TD
    subgraph CP["Control plane (istiod / linkerd control plane)"]
      CA[Certificate authority<br/>workload identities]
      CFG[Config & policy<br/>routes, retries, authz]
      DS[Service discovery]
    end
    subgraph P1["Pod: checkout"]
      A1[checkout app] --> X1[proxy]
    end
    subgraph P2["Pod: payments"]
      X2[proxy] --> A2[payments app]
    end
    X1 == "mTLS (SPIFFE identities)" ==> X2
    CP -. "xDS config + certs" .-> X1
    CP -. "xDS config + certs" .-> X2
    X1 -. "metrics & traces" .-> OBS[(Prometheus / tracing backend)]
    X2 -. "metrics & traces" .-> OBS

• Data plane — proxies (Envoy in Istio, linkerd2-proxy in Linkerd) that intercept all inbound and outbound traffic of each workload, as sidecars or as node-level/ambient components.
• Control plane — distributes configuration and certificates to proxies, and watches the platform’s service registry.

The application keeps making plain HTTP/gRPC calls; the mesh transparently adds security, policy, and telemetry.

What a mesh provides

1. Security: mTLS and identity

Each workload gets a cryptographic identity (typically a SPIFFE ID derived from its Kubernetes service account) with automatically rotated certificates. Traffic between proxies is mutually authenticated and encrypted, and you can write authorization policies by identity, not IP address.

# Istio: require mTLS for every workload in the namespace
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata: { name: default, namespace: payments }
spec:
  mtls: { mode: STRICT }
---
# Only the checkout service account may call payments' capture endpoint
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: { name: payments-capture, namespace: payments }
spec:
  selector: { matchLabels: { app: payments } }
  action: ALLOW
  rules:
    - from:
        - source: { principals: ["cluster.local/ns/checkout/sa/checkout"] }
      to:
        - operation: { methods: ["POST"], paths: ["/v1/payments/*/capture"] }

2. Traffic management

Weighted routing, header-based routing, mirroring, and fault injection — the tools for canaries, blue/green, and Strangler Fig migrations.

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata: { name: payments }
spec:
  hosts: [payments]
  http:
    - match: [{ headers: { x-canary: { exact: "true" } } }]
      route: [{ destination: { host: payments, subset: v2 } }]
    - route:
        - { destination: { host: payments, subset: v1 }, weight: 95 }
        - { destination: { host: payments, subset: v2 }, weight: 5 }
      timeout: 2s
      retries: { attempts: 2, perTryTimeout: 800ms, retryOn: "connect-failure,refused-stream,unavailable" }
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata: { name: payments }
spec:
  host: payments
  subsets:
    - { name: v1, labels: { version: v1 } }
    - { name: v2, labels: { version: v2 } }
  trafficPolicy:
    outlierDetection: { consecutive5xxErrors: 5, interval: 10s, baseEjectionTime: 30s }

3. Resilience

Timeouts, retries with budgets, connection-pool limits (Bulkhead), and outlier ejection (a form of Circuit Breaker) — configured consistently, per destination.

4. Observability

Every proxy emits golden signals — request rate, error rate, latency — per source and destination, plus access logs and trace spans. You get a service graph without instrumenting code (though application-level tracing is still needed to connect spans inside a service; see Distributed Tracing).

Data-plane models

Is a mesh worth it?

A mesh is significant operational investment. Rough guidance:

Linkerd’s design emphasizes simplicity and low overhead; Istio offers the broadest feature set; Consul spans Kubernetes and VMs. Many teams start with mTLS and telemetry only, then adopt traffic management later.

Trade-offs

Common pitfalls

Pitfall: retries configured in the mesh and in application clients and at the gateway. A single failing call becomes dozens. Decide which layer owns retries for each path — see Retry with Backoff.

• Turning on STRICT mTLS cluster-wide on day one — break non-meshed clients; roll out with PERMISSIVE first and watch telemetry.
• Treating the mesh as authorization for business rules — it knows which service calls; it doesn’t know which customer may refund which order.
• Ignoring upgrades — control-plane and proxy versions must be upgraded regularly and carefully.
• No owner — a mesh needs a platform team.

Tools and real-world usage

• Istio — feature-rich; sidecar and ambient modes; Envoy data plane.
• Linkerd — lightweight, Rust-based proxy, focused on simplicity and security.
• Consul service mesh — multi-platform (Kubernetes and VMs).
• Cilium — eBPF networking with mesh capabilities; Kuma / Kong Mesh — Envoy-based, multi-zone.

Related patterns

• Sidecar — the classic data-plane deployment.
• Service Discovery — the mesh builds on the platform’s registry.
• API Gateway — north-south counterpart to the mesh’s east-west traffic.
• Circuit Breaker, Timeout, Retry with Backoff — policies the mesh can enforce.

Key takeaways

1. A mesh = data-plane proxies handling all service traffic + a control plane configuring them.
2. Its biggest wins are mTLS identity, consistent resilience policy, and uniform telemetry.
3. Choose sidecar, ambient, or eBPF models based on overhead and maturity needs.
4. Adopt incrementally, own it with a platform team, and avoid layering retries.

Sidecars are how service meshes inject mTLS and traffic control, how many teams ship logs and metrics, and how runtimes like Dapr give every language the same building blocks.

The problem

Every service needs the same plumbing: TLS, retries and timeouts, metrics and traces, log shipping, secret fetching, configuration reload, auth token validation. Implementing these in every service:

• duplicates effort across teams and languages,
• drifts — each team’s version behaves slightly differently,
• couples upgrades — patching a TLS library means rebuilding and redeploying every service,
• is hard for polyglot estates — the Java library doesn’t help the Python or Go teams.

How it works

flowchart TD
    subgraph Pod["Kubernetes Pod (shared network namespace & volumes)"]
      APP[Application container<br/>business logic only]
      PX[Proxy sidecar<br/>Envoy: mTLS, retries, metrics]
      LOG[Log shipper sidecar<br/>Fluent Bit]
      SEC[Secrets sidecar<br/>Vault Agent]
      APP -- "localhost:15001" --> PX
      APP -- "writes /var/log/app" --> LOG
      SEC -- "renders /vault/secrets/db" --> APP
    end
    PX -- "mTLS" --> OTHER[Other services]
    LOG --> SINK[(Log platform)]
    SEC --> VAULT[(Vault)]

Because sidecars share the pod’s network (localhost) and can share volumes, they can intercept traffic, read files the app writes, or provide files the app reads — without the application knowing much about them.

Common sidecar roles

Example: native sidecars in Kubernetes

Kubernetes 1.29+ supports native sidecars: init containers with restartPolicy: Always. They start before the app, keep running alongside it, and shut down after it — fixing long-standing ordering problems (e.g., the app starting before the proxy is ready, or Jobs never completing because the sidecar keeps running).

apiVersion: apps/v1
kind: Deployment
metadata: { name: statements }
spec:
  template:
    spec:
      initContainers:
        - name: otel-collector                  # native sidecar
          image: otel/opentelemetry-collector-contrib:0.110.0
          restartPolicy: Always
          args: ["--config=/conf/collector.yaml"]
          ports: [{ containerPort: 4317 }]       # app sends OTLP to localhost:4317
          resources:
            requests: { cpu: 50m, memory: 64Mi }
            limits:   { memory: 128Mi }
          volumeMounts: [{ name: otel-conf, mountPath: /conf }]

        - name: log-shipper
          image: fluent/fluent-bit:3.1
          restartPolicy: Always
          volumeMounts: [{ name: app-logs, mountPath: /var/log/app, readOnly: true }]

      containers:
        - name: app
          image: registry.example.com/statements:2026.10.17
          env:
            - { name: OTEL_EXPORTER_OTLP_ENDPOINT, value: "http://localhost:4317" }
          volumeMounts: [{ name: app-logs, mountPath: /var/log/app }]

      volumes:
        - { name: app-logs, emptyDir: {} }
        - { name: otel-conf, configMap: { name: statements-otel } }

Example: Dapr sidecar

Dapr injects a daprd sidecar that exposes building blocks over localhost. The app publishes an event without a Kafka client library:

import requests

# Dapr sidecar listens on localhost:3500; "pubsub" component maps to Kafka, Service Bus, etc.
requests.post(
    "http://localhost:3500/v1.0/publish/pubsub/statements.generated",
    json={"statementId": "st-2026-10-0042", "accountId": "AC-1842"},
    headers={"traceparent": current_traceparent()},
    timeout=2,
)

apiVersion: dapr.io/v1alpha1
kind: Component
metadata: { name: pubsub }
spec:
  type: pubsub.kafka
  version: v1
  metadata:
    - { name: brokers, value: "kafka:9092" }
    - { name: consumerGroup, value: "statements" }
    - { name: authType, value: "none" }   # use mTLS/SASL in real environments

Switching from Kafka to Azure Service Bus becomes a component change, not a code change.

Sidecar vs. alternatives

Design considerations

• Resource requests and limits for every sidecar — they count toward pod scheduling and can be OOM-killed.
• Startup and shutdown ordering — use native sidecars, or hold the app’s start until the proxy is ready.
• Version management — sidecars are often injected by a platform (mutating webhooks); coordinate upgrades and roll them like any release.
• Security — sidecars share the pod’s network and often its service account; give them least privilege.
• Observability of the sidecar itself — its health, latency, and errors are now part of your service’s behavior.

Trade-offs

Common pitfalls

Pitfall: Jobs that never finish. A classic (pre-native-sidecar) bug: a Kubernetes Job’s main container exits, but the proxy or log-shipper sidecar keeps running, so the Job never completes. Native sidecars (restartPolicy: Always init containers) fix this.

• No resource limits — a misbehaving log shipper starves the application.
• App starts before the proxy — the first requests fail with connection errors.
• Too many sidecars — five helpers per pod multiplies resource overhead across thousands of pods.
• Hidden coupling — the app depends on sidecar behavior (retries, auth) that isn’t documented or tested.

Tools and real-world usage

• Envoy (Istio, Consul), linkerd2-proxy — mesh data-plane sidecars.
• Dapr — portable building blocks via sidecar.
• OpenTelemetry Collector, Fluent Bit, Vector — telemetry and logs.
• Vault Agent Injector, Secrets Store CSI Driver — secrets delivery.
• OAuth2 Proxy, Open Policy Agent — authentication and authorization sidecars.

Related patterns

• Service Mesh — a fleet of proxy sidecars plus a control plane.
• Distributed Tracing and Log Aggregation — commonly delivered via sidecars.
• Anti-Corruption Layer — an adapter sidecar can translate a legacy protocol.
• Health Check — sidecar readiness affects pod readiness.

Key takeaways

1. A sidecar is a helper container sharing the service’s pod, network, and lifecycle.
2. It delivers cross-cutting capabilities to any language without code changes.
3. Budget resources, manage ordering (native sidecars), and secure each sidecar.
4. Weigh it against libraries, node agents, and sidecar-less meshes.

It’s the antidote to the big-bang rewrite — the multi-year project that tries to replace everything at once and too often fails to ship.

The problem

Big-bang rewrites fail for predictable reasons:

• The old system keeps changing while you rebuild it — you’re chasing a moving target.
• Hidden behavior (undocumented rules, batch jobs, edge cases) is discovered at cut-over.
• No value reaches users until the very end.
• Cut-over day is a single, enormous, high-risk event.

How it works

flowchart TD
    U[Users / clients] --> P[["Routing facade<br/>(gateway / proxy)"]]
    P -->|"/accounts/statements → new"| N1[New Statements service]
    P -->|"/accounts/profile → new"| N2[New Profile service]
    P -->|"everything else → legacy"| L[Legacy monolith]
    N1 -. "ACL" .-> L
    L -- "CDC / events" --> N2

1. Intercept — put a routing layer (API gateway, reverse proxy, or a facade in the monolith) in front of the legacy system. Initially it routes 100% to legacy.
2. Pick a slice — a capability with clear boundaries and real value.
3. Build it new — in the target architecture, behind an Anti-Corruption Layer where it must talk to legacy.
4. Divert traffic — route that slice to the new system, gradually (by user, percentage, or tenant).
5. Remove the legacy code for that slice once it’s no longer used.
6. Repeat until the legacy system is empty — then switch it off.

Example: routing with NGINX

upstream legacy  { server legacy-app.internal:8080; }
upstream stmts   { server statements.svc.internal:8080; }

# Canary by cookie/header first, then by percentage
split_clients "${remote_addr}${http_user_agent}" $statements_backend {
    20%   stmts;          # 20% of anonymous traffic to the new service
    *     legacy;
}

map $http_x_canary $statements_target {
    "true"   stmts;       # internal testers always hit the new path
    default  $statements_backend;
}

server {
  listen 443 ssl;

  location /accounts/statements {
    proxy_pass http://$statements_target;
    proxy_set_header X-Correlation-ID $request_id;
  }

  location / {
    proxy_pass http://legacy;     # everything not yet migrated
  }
}

Gateways (Kong, Envoy, the Kubernetes Gateway API) offer the same with weighted routes and header matches, and feature flags can route by customer segment inside the application.

Choosing the first slice

Rule of thumb: stop adding features to the legacy system first. Every new capability goes into the new architecture — this alone starts the strangling.

The data problem

Routing requests is the easy part; data is the hard part. Options, roughly in order of maturity:

flowchart TD
    A["Phase 1 · New service reads legacy data<br/>via ACL / API (legacy is source of truth)"] --> B["Phase 2 · New service keeps its own copy<br/>synced via CDC from legacy tables"]
    B --> C["Phase 3 · New service becomes source of truth<br/>legacy reads from it (reverse sync)"]
    C --> D["Phase 4 · Legacy dependency removed<br/>legacy tables archived"]

• Change Data Capture from legacy tables keeps new stores in sync without modifying legacy code.
• Dual writes are dangerous — prefer one source of truth per data set at each phase, with one-directional sync.
• Reconcile — run automated comparisons between old and new data during migration and alert on drift.

Reducing risk: shadow traffic and parallel runs

Before sending real users to the new slice, compare behaviors:

• Shadow (dark) traffic — mirror production requests to the new service, discard its responses, and compare them with legacy’s. Envoy/Istio support request mirroring.
• Parallel run — for calculations (interest, fees, eligibility), run both and alert on differences; the legacy result remains authoritative until confidence is high.
• Gradual rollout — internal users → 1% → 10% → 50% → 100%, with a one-switch rollback.

# Istio: mirror 100% of statement traffic to the new service while legacy still serves
http:
  - match: [{ uri: { prefix: /accounts/statements } }]
    route: [{ destination: { host: legacy } }]
    mirror: { host: statements }
    mirrorPercentage: { value: 100.0 }

Warning: shadow traffic must not cause side effects. Mirrored writes to a new service that sends emails or posts to a ledger will double-act. Mirror reads freely; mirror writes only to sandboxed or no-op paths.

Finishing the job

Strangler migrations often stall at 80%. The last 20% is the hardest code with the least glamour — and keeping two systems running indefinitely is the worst outcome. Keep momentum with:

• A visible migration map of slices: legacy, in progress, migrated, removed.
• Decommissioning as a goal with dates and budget, not an afterthought.
• Deleting legacy code for each migrated slice (not just routing around it).
• Measuring legacy traffic and legacy change requests trending to zero.

Trade-offs

Common pitfalls

Pitfall: recreating the monolith’s structure in microservices — same data model, same boundaries, same coupling — just over HTTP. Use the migration to redraw boundaries around business capabilities and aggregates.

• No routing layer — every client must be changed to call the new service.
• Bidirectional sync between old and new databases, causing loops and conflicts.
• Migrating the easiest technical layer first (e.g., “the UI”) with no capability moving.
• Never deleting legacy code, so the old system keeps being maintained.

Tools and real-world usage

• Routing — API gateways (Kong, Apigee, AWS API Gateway), reverse proxies (NGINX, Envoy, HAProxy), service meshes (traffic splitting and mirroring), feature-flag platforms.
• Data — Debezium, Oracle GoldenGate, AWS DMS for CDC; reconciliation jobs.
• Legacy integration — ACL services, MQ bridges, z/OS Connect for mainframes.

Strangler-style migrations are the norm for banks moving off monolithic platforms: digital channels, statements, notifications, and document services move first, while the core ledger is strangled last, behind ACLs and CDC — the same approach I’ve seen work when re-architecting a legacy content-management platform into microservices that served both old SOAP consumers and new REST APIs during the transition.

Related patterns

• Anti-Corruption Layer — protects new services from the legacy model.
• API Gateway — the usual routing facade.
• Change Data Capture — data sync during migration.
• Shared Database — the starting point you’re often migrating from.

Key takeaways

1. Put a routing facade in front of legacy, then move one capability at a time.
2. Stop building new features in legacy; start with valuable, well-bounded slices.
3. Keep one source of truth per data set per phase; sync one way with CDC; reconcile.
4. De-risk with shadow traffic, parallel runs, and gradual rollout — and finish by deleting the old system.

The Anti-Corruption Layer (ACL), from Eric Evans’s Domain-Driven Design, is a translation boundary: it talks to the other system in its language and presents a clean interface in yours.

The problem

Integrating directly with a foreign model causes:

• Semantic leakage — CUST_STAT_CD = '9' means “deceased” in one system and “dormant” in another; your code fills up with magic values.
• Structural coupling — your entities mirror their tables and fields.
• Change amplification — when the legacy system changes, every consumer changes.
• Migration paralysis — you can’t replace the legacy system because everything depends on its model.

How it works

flowchart TD
    subgraph New["Your bounded context — clean domain model"]
      D[Account Opening service<br/>Customer, Account, KycStatus]
    end
    subgraph ACL["Anti-Corruption Layer"]
      F[Facade<br/>domain-shaped interface]
      T[Translator<br/>model ↔ model mapping]
      A[Adapter<br/>protocol: SOAP / MQ / fixed-width files]
      F --> T --> A
    end
    subgraph Legacy["Legacy core banking"]
      L[(CIF records, status codes,<br/>EBCDIC copybooks)]
    end
    D --> F
    A --> L

An ACL typically has three parts:

The ACL can be a module inside your service or a separate translation service shared by several consumers.

Example: translating a core-banking customer

The legacy system returns:

{
  "CIF_NO": "0004418823",
  "CUST_NM": "RAO/ANANYA",
  "CUST_TYP": "I",
  "STAT_CD": "A",
  "KYC_FLG": "Y",
  "KYC_DT": "20240311",
  "RISK_CAT": "02",
  "BR_CD": "0412"
}

Your domain wants:

data class Customer(
    val id: CustomerId,
    val name: PersonName,
    val type: CustomerType,
    val status: CustomerStatus,
    val kyc: KycStatus,
    val riskRating: RiskRating,
    val homeBranch: BranchId,
)

The ACL:

// Facade — the only thing the domain sees
interface CustomerDirectory {
    fun find(id: CustomerId): Customer?
}

// Implementation lives in the ACL module
class CoreBankingCustomerDirectory(
    private val adapter: CoreBankingSoapAdapter,
    private val translator: CustomerTranslator,
) : CustomerDirectory {
    override fun find(id: CustomerId): Customer? =
        adapter.inquireCif(id.value.padStart(10, '0'))      // legacy quirks stay here
            ?.let(translator::toDomain)
}

class CustomerTranslator(private val clock: Clock) {
    fun toDomain(r: CifRecord): Customer = Customer(
        id = CustomerId(r.cifNo.trimStart('0')),
        name = r.custNm.split("/").let { (last, first) -> PersonName(first.titleCase(), last.titleCase()) },
        type = when (r.custTyp) {
            "I" -> CustomerType.INDIVIDUAL
            "C" -> CustomerType.CORPORATE
            else -> throw UnknownLegacyValue("CUST_TYP", r.custTyp)
        },
        status = when (r.statCd) {
            "A" -> CustomerStatus.ACTIVE
            "D" -> CustomerStatus.DORMANT
            "C" -> CustomerStatus.CLOSED
            "9" -> CustomerStatus.DECEASED
            else -> throw UnknownLegacyValue("STAT_CD", r.statCd)
        },
        kyc = if (r.kycFlg == "Y")
            KycStatus.Verified(LocalDate.parse(r.kycDt, DateTimeFormatter.BASIC_ISO_DATE))
        else KycStatus.Pending,
        riskRating = RiskRating.fromScore(r.riskCat.toInt()),
        homeBranch = BranchId(r.brCd),
    )
}

All the legacy weirdness — zero-padding, slash-separated names, single-letter codes, yyyyMMdd dates — is confined to one place. The domain never sees STAT_CD.

Two directions

ACLs translate both ways:

• Inbound — legacy data and events → your domain objects/events (e.g., translating CDC records from legacy tables into clean Domain Events).
• Outbound — your commands → legacy transactions (e.g., OpenAccount → a sequence of mainframe calls with the right codes).

flowchart TD
    E[Legacy table change<br/>via CDC] --> ACL1[ACL translator]
    ACL1 --> DE([CustomerKycVerified domain event])
    CMD([OpenAccount command]) --> ACL2[ACL translator]
    ACL2 --> MQ[Legacy MQ transaction<br/>ACCT-OPN-01]

Testing the ACL

The ACL is where integration bugs hide, so test it heavily:

• Translation unit tests — table-driven tests for every code and edge case (blank fields, unknown codes, invalid dates).
• Contract tests — record real legacy responses (sanitized) and replay them; alert when the legacy system starts returning unexpected values.
• Fail loudly on unknown values — an unrecognized status code should raise an error and an alert, not silently map to a default.

When is an ACL worth it?

Trade-offs

Common pitfalls

Pitfall: a “pass-through” ACL that just renames fields (CUST_NM → custName) and passes legacy codes straight through. The legacy concepts still leak. Translate meaning, not just spelling.

• Leaky abstractions — facade methods that mirror legacy transactions (inquireCif) rather than domain needs.
• Silent defaults for unknown codes, hiding data-quality problems.
• Business logic in the ACL — it should translate, not decide.
• Bypassing the ACL “just this once” for a quick feature.

Tools and real-world usage

• Adapter services written in the team’s main language, often with mapping libraries (MapStruct, AutoMapper).
• Integration frameworks — Apache Camel, Spring Integration, MuleSoft and other iPaaS tools for protocol and format conversion.
• Mainframe integration — IBM MQ, CICS web services, z/OS Connect APIs wrapped behind ACLs.

In banking modernization programmes, ACLs around core banking, card management, and document management platforms are what let new digital channels move fast without inheriting decades of legacy semantics.

Related patterns

• Strangler Fig — the ACL keeps the new system clean while the old one is replaced.
• Change Data Capture — feed legacy changes through an ACL into domain events.
• Shared Database — what ACLs help you avoid depending on.
• Backend for Frontend — a client-facing adapter, often in front of ACL-protected services.

Key takeaways

1. An ACL translates between a foreign model and your domain, in both directions.
2. Structure it as facade (your language), translator (meaning), adapter (protocol).
3. Translate concepts, not just field names, and fail loudly on unknown values.
4. It’s the enabler for replacing legacy systems incrementally.

A Correlation ID is a unique identifier assigned when a request (or business transaction) enters the system and propagated to every service, message, and log line it causes. Search for it once and you see the whole story.

The problem

In a monolith, one request is one thread with one log context. In a distributed system:

• one user action fans out into many synchronous calls,
• asynchronous messages trigger work minutes later in other services,
• retries and redeliveries repeat work,
• and logs are spread across services, pods, and time.

How it works

sequenceDiagram
    participant C as Client
    participant G as API Gateway
    participant A as Checkout
    participant P as Payments
    participant K as Kafka
    participant L as Ledger consumer
    C->>G: POST /checkout
    Note over G: no incoming ID → generate<br/>traceparent 00-4bf9…-01
    G->>A: traceparent: 00-4bf9…
    A->>P: traceparent: 00-4bf9…
    P->>K: message headers: traceparent 00-4bf9…
    K->>L: (later) same header
    Note over A,L: every log line includes trace_id=4bf9…
    G-->>C: 200 + header X-Request-ID / traceresponse

Four rules:

1. Generate at the edge if the incoming request has none (and decide whether to trust IDs from external clients).
2. Propagate on every outbound call — HTTP headers, gRPC metadata, message headers.
3. Put it in every log line via the logging context (MDC, scopes, structured fields).
4. Return it to the caller so users and support can quote it.

Which header?

With Distributed Tracing via OpenTelemetry, the trace ID is your correlation ID — you get propagation for free, plus timing. Many teams still keep a separate business correlation ID (e.g., orderId or checkoutSessionId) that spans multiple traces over hours or days.

Event-driven systems often carry all three in message metadata: correlationId, causationId, and messageId.

Example: Spring Boot filter + MDC

With OpenTelemetry’s Java agent or Micrometer Tracing, traceId and spanId are put in the MDC automatically. Here’s the manual version, which also shows the moving parts:

@Component
@Order(Ordered.HIGHEST_PRECEDENCE)
public class CorrelationIdFilter extends OncePerRequestFilter {

  public static final String HEADER = "X-Correlation-ID";

  @Override
  protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res, FilterChain chain)
      throws ServletException, IOException {
    String incoming = req.getHeader(HEADER);
    String id = isValid(incoming) ? incoming : UUID.randomUUID().toString();
    MDC.put("correlationId", id);
    res.setHeader(HEADER, id);                 // return it to the caller
    try {
      chain.doFilter(req, res);
    } finally {
      MDC.remove("correlationId");             // threads are pooled: always clean up
    }
  }

  private boolean isValid(String v) {          // don't trust arbitrary input into logs
    return v != null && v.length() <= 64 && v.matches("[A-Za-z0-9-]+");
  }
}

// Propagate on outbound HTTP calls
@Bean
RestClient restClient(RestClient.Builder b) {
  return b.requestInterceptor((request, body, exec) -> {
    String id = MDC.get("correlationId");
    if (id != null) request.getHeaders().set(CorrelationIdFilter.HEADER, id);
    return exec.execute(request, body);
  }).build();
}

<!-- logback-spring.xml: JSON logs with correlation fields -->
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <includeMdcKeyName>correlationId</includeMdcKeyName>
  <includeMdcKeyName>traceId</includeMdcKeyName>
  <includeMdcKeyName>spanId</includeMdcKeyName>
</encoder>

Across asynchronous hops

Propagation breaks most often at message boundaries and thread pools.

// Producer: copy context into message headers (OpenTelemetry instrumentation does this automatically)
ProducerRecord<String, OrderPlaced> rec = new ProducerRecord<>("orders.events", order.id(), event);
rec.headers().add("correlationId", MDC.get("correlationId").getBytes(UTF_8));
rec.headers().add("causationId", currentMessageIdOrRequestId().getBytes(UTF_8));
kafka.send(rec);

// Consumer: restore context before processing
@KafkaListener(topics = "orders.events")
public void on(ConsumerRecord<String, OrderPlaced> rec) {
  Header h = rec.headers().lastHeader("correlationId");
  try (MDC.MDCCloseable c = MDC.putCloseable("correlationId",
          h != null ? new String(h.value(), UTF_8) : UUID.randomUUID().toString())) {
    handle(rec.value());
  }
}

Also:

• Thread pools / CompletableFuture — wrap executors so context is copied (OpenTelemetry’s Context.taskWrapping, Micrometer’s context propagation).
• Outbox rows — store correlation and trace headers in the outbox so the relay can attach them (Transactional Outbox).
• Batch jobs — assign a correlation ID per run and per item.

Security and privacy

• Validate incoming IDs (length, character set) to prevent log injection.
• Don’t encode personal data in correlation IDs (no emails, phone numbers, account numbers).
• Decide trust boundaries — you may accept traceparent from internal callers but start a new trace (linking the external one) at the public edge.

Trade-offs

Common pitfalls

Pitfall: generating a new ID in every service. Each service logs its own UUID, and nothing connects them. Generate once at the edge; everyone else propagates.

• Losing the ID at the message broker — producers don’t set headers, consumers don’t restore them.
• MDC leaks — not clearing context on pooled threads, so the next request logs the previous ID.
• Unstructured logs — the ID is in the text but not a field, making search slow and unreliable.
• Not returning the ID to clients, so support can’t ask for it.

Tools and real-world usage

• OpenTelemetry SDKs and auto-instrumentation — W3C Trace Context propagation across HTTP, gRPC, and messaging.
• Micrometer Tracing (Spring Boot 3), Serilog enrichers and Activity in .NET, pino/winston with AsyncLocalStorage in Node.js.
• API gateways (Kong correlation-id plugin, NGINX $request_id, Envoy x-request-id) — generate at the edge.
• Log platforms (ELK, Loki, Splunk, Datadog) — pivot from a correlation ID to all related logs and traces.

Related patterns

• Distributed Tracing — correlation plus timing and causality.
• Log Aggregation — where correlation IDs become searchable.
• Async Request-Reply — correlation IDs match replies to requests.
• Saga — business correlation IDs follow long-running processes.

Key takeaways

1. Generate once at the edge, propagate everywhere, log it in every line, return it to the caller.
2. Prefer W3C traceparent; add a business correlation ID for long processes.
3. Carry IDs through message headers, outbox rows, and thread pools.
4. Validate incoming IDs and keep personal data out of them.