The Certificate Problem

§ 01 — THE QUESTIONThe padlock is a ceremony

A lock icon looks like a feature. It is, in fact, a chain of cryptographic signatures, a trust hierarchy burned into every operating system worldwide, a real-time revocation check, and a key-agreement protocol — all resolved in under 50 milliseconds before a single byte of HTTP is exchanged.

The question catches most candidates off-guard because PKI sits at an uncomfortable intersection: it's security infrastructure, but the interesting engineering is data modeling and distributed systems. A weak answer draws a box labeled "Certificate Authority" and calls it done. A strong answer names the four forces that make this hard, then shows how every later decision exists to survive one of them.

There are exactly four forces, and they are not the usual suspects:

THE TRUST ANCHOR

Root CA keys are the crown jewels. Every certificate ever issued traces back to a root key that must never touch an online system. If the root is compromised, every browser in the world must remove it — and millions of sites go dark. The design has to protect the root unconditionally, which means an offline HSM, air-gapped signing, and an intermediate CA layer that carries all operational risk.

THE REVOCATION PROBLEM

You cannot un-issue a certificate. A cert is valid on its face — any browser can verify the signature without calling anyone. Revocation is a second, out-of-band signal (CRL or OCSP) that the CA must serve globally at low latency, or browsers soft-fail and accept the cert anyway. The PKI is only as strong as its revocation infrastructure.

THE 90-DAY WALL

Short lifetimes mean automation is not optional. The CA/Browser Forum is shortening max TLS cert lifetimes — 90 days today, 47 days by 2027. No human renews every 47 days at scale. Every architectural choice flows from "the client must automate renewal, period."

THE TRANSPARENCY REQUIREMENT

Every cert must be publicly logged before it's accepted. Chrome has required Certificate Transparency (CT) since 2018 — a cert not in 2+ public append-only Merkle logs is rejected outright. This means the issuance pipeline has a mandatory, latency-adding step (get SCTs from independent logs) before signing.

Scoping out loud

In scope: certificate issuance (DV, OV, EV), the chain of trust, TLS handshake mechanics, CRL/OCSP/CRLite revocation, domain control validation (DCV), Certificate Transparency, and the Kubernetes-native CA (cert-manager, SPIFFE/SPIRE, Istio mTLS). Out of scope (modeled-for, not designed in detail): email signing (S/MIME), code signing, document signing, IoT device enrollment internals, and HSM hardware procurement. Two tensions I name up front: (1) issuance must be fast (sub-300ms for DV-auto) but also auditable forever (7-year retention per CA/B Forum); (2) revocation must be real-time but also work offline (CRLite).

Envelope math, volunteered:

The number unlike the others: 8–10 billion OCSP checks per day. That is not a database problem. It is a CDN/Anycast problem — pre-signed responses, Redis-cached, refreshed before TTL expires. Every other number follows from resolving that one deliberately.

The accessible primer — HTTP vs HTTPS, padlock states, chain of trust, TLS handshake, and failure stories — lives in the article: Designing DigiCert →

§ 02 — THE CERTIFICATEWhat's actually in an X.509 cert

A certificate is a signed data structure that binds a public key to an identity. The diagram below shows every field your browser validates. See the article for the plain-language walkthrough.

Quick reference: SAN (not CN) is canonical for hostnames. Extended Key Usage restricts misuse. OCSP URL (in AIA) is where clients check revocation. CT SCT List embeds 2+ Signed Certificate Timestamps — Chrome requires them. Signature covers all fields above; verify against the Issuer's public key.

That's all HTTPS is: HTTP (the language websites speak) wrapped inside TLS (the sealed envelope). The certificate is the proof that the envelope was sealed by the real sender — not someone pretending to be your friend.

What the padlock means — and what it doesn't

The red X — "This root certificate is not trusted"

When macOS shows ⊗ This root certificate is not trusted, it means: your computer doesn't recognise the organisation that signed this certificate as a trustworthy authority. Think of it like a passport stamped by a country your government doesn't recognise. The passport might be real — but nobody here will accept it.

1. REMOVED FROM ROOT PROGRAM

The CA was caught misbehaving — issuing certs without proper verification, or issuing certs for domains they don't control. Apple/Mozilla/Google removed it from their trusted root list. TrustCor was removed in 2022 after researchers found links to a surveillance company. The cert is technically valid but no browser or OS will vouch for the organisation that signed it.

2. SELF-SIGNED

The cert was signed by itself — no third party vouches for it. Common in development. Your computer has never heard of "Bob's Test CA" and has no reason to trust it.

3. EXPIRED OR REVOKED

The cert passed its notAfter date, or the CA published a revocation notice. Even if the CA is trusted, this specific cert is no longer valid.

HTTP vs HTTPS — for your CEO

HTTP sends data in plain text across the internet — like writing your credit card number on a postcard. Any router, ISP, or coffee shop Wi-Fi between you and the website can see it. HTTPS encrypts everything using TLS, so even if someone intercepts the traffic, they see random bytes. The certificate establishes the encryption keys AND verifies you're talking to the real site, not an impersonator. Companies that still run HTTP for anything involving user data are exposing their customers and violating most data privacy regulations (GDPR, CCPA, PCI-DSS).

The chain of trust

§ 03 — TLS 1.3The handshake, message by message

TLS 1.3 reduced the handshake to one round-trip. Here is every message exchanged and what the client verifies before accepting the cert. See the article for the plain-language walkthrough.

CLIENT                                          SERVER
──────                                          ──────
ClientHello ──────────────────────────────────►
  - TLS version: 1.3
  - key_share: client's ephemeral X25519 pub key
  - supported cipher suites

                     ◄─────────────────────────── ServerHello
                                                    - selected cipher: TLS_AES_256_GCM_SHA384
                                                    - key_share: server's X25519 pub key
                                                    ← BOTH SIDES DERIVE session keys now →
                                                    (all subsequent messages are encrypted)

                     ◄───────────────────────────  EncryptedExtensions
                     ◄───────────────────────────  Certificate (leaf + intermediates)
                     ◄───────────────────────────  CertificateVerify (server signs handshake hash
                                                    with its private key — proves key possession)
                     ◄───────────────────────────  Finished (HMAC over entire handshake)

CLIENT verifies the cert:
  1. Chain: leaf → intermediate → trusted root in local trust store
  2. SANs include the hostname being connected to
  3. Not Before ≤ now ≤ Not After
  4. CertificateVerify signature validates against cert's public key
  5. OCSP staple (or online OCSP call) → status = GOOD
  6. CT SCTs present (Chrome requires 2+ from independent logs)

Finished ─────────────────────────────────────►  (confirms receipt)

═══════════════════════════════════════════════  Application Data (HTTP/2, encrypted)
Total added RTTs: 1   (0-RTT resumption possible on reconnect)

§ 04 — THE SCHEMAThe issuance database

The interesting design is the OCSP cache as a separate hot read-path table and the serial number generation constraint. Every table below serves a distinct compliance or operational requirement.

-- Certificate Authorities (roots + intermediates)
CREATE TABLE certificate_authority (
    ca_id           BIGSERIAL PRIMARY KEY,
    ca_type         TEXT NOT NULL CHECK (ca_type IN ('root','intermediate','issuing')),
    parent_ca_id    BIGINT REFERENCES certificate_authority(ca_id),
    common_name     TEXT NOT NULL,
    subject_dn      TEXT NOT NULL,
    public_key_sha256 BYTEA NOT NULL UNIQUE,   -- key fingerprint
    cert_pem        TEXT NOT NULL,
    valid_from      TIMESTAMPTZ NOT NULL,
    valid_until     TIMESTAMPTZ NOT NULL,
    hsm_slot_id     TEXT,                      -- HSM partition reference (never the key itself)
    is_active       BOOLEAN NOT NULL DEFAULT TRUE
);

-- Every issued certificate
CREATE TABLE certificate (
    cert_id         BIGSERIAL PRIMARY KEY,
    serial_number   BYTEA NOT NULL,            -- 20 bytes CSPRNG entropy (RFC 5280 / SC63)
    issuing_ca_id   BIGINT NOT NULL REFERENCES certificate_authority(ca_id),
    subject_cn      TEXT NOT NULL,
    san_list        TEXT[] NOT NULL,           -- DNS names, IPs — indexed via GIN
    cert_type       TEXT NOT NULL CHECK (cert_type IN ('dv','ov','ev','code_sign','s_mime','client')),
    public_key_algo TEXT NOT NULL,             -- RSA, EC, Ed25519
    public_key_sha256 BYTEA NOT NULL,
    cert_pem        TEXT NOT NULL,
    valid_from      TIMESTAMPTZ NOT NULL,
    valid_until     TIMESTAMPTZ NOT NULL,
    issued_at       TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    status          TEXT NOT NULL DEFAULT 'active'
                    CHECK (status IN ('active','revoked','expired','hold')),
    customer_id     BIGINT NOT NULL,
    order_id        BIGINT NOT NULL,
    ct_log_ids      TEXT[] NOT NULL DEFAULT '{}',  -- SCT log IDs (must have 2+)
    UNIQUE (issuing_ca_id, serial_number)
);

-- Revocation (append-only; never delete)
CREATE TABLE revocation (
    revocation_id   BIGSERIAL PRIMARY KEY,
    cert_id         BIGINT NOT NULL REFERENCES certificate(cert_id),
    revoked_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    reason_code     INT NOT NULL,              -- RFC 5280 CRLReason (1=keyCompromise, 4=superseded, etc.)
    reason_text     TEXT,
    requested_by    TEXT NOT NULL,             -- customer / admin / automated
    crl_published_at TIMESTAMPTZ              -- null until included in next CRL
);

-- Domain Control Validation events
CREATE TABLE dcv_event (
    dcv_id          BIGSERIAL PRIMARY KEY,
    domain          TEXT NOT NULL,
    method          TEXT NOT NULL CHECK (method IN ('http-01','dns-01','tls-alpn-01','email')),
    challenge_token TEXT NOT NULL,
    challenge_value TEXT NOT NULL,
    validated_at    TIMESTAMPTZ,
    expires_at      TIMESTAMPTZ NOT NULL,      -- reuse window (was 825d → now 90d)
    cert_id         BIGINT REFERENCES certificate(cert_id),
    ip_logged       INET                       -- requester IP for audit trail
);

-- OCSP response cache — the hot read path (8B req/day)
-- Populated at issuance and on revocation; refreshed before next_update
CREATE TABLE ocsp_response_cache (
    cert_id         BIGINT PRIMARY KEY REFERENCES certificate(cert_id),
    ca_id           BIGINT NOT NULL REFERENCES certificate_authority(ca_id),
    this_update     TIMESTAMPTZ NOT NULL,
    next_update     TIMESTAMPTZ NOT NULL,      -- cache TTL
    ocsp_status     TEXT NOT NULL CHECK (ocsp_status IN ('good','revoked','unknown')),
    signed_response BYTEA NOT NULL             -- DER-encoded OCSPResponse; serve directly
);

-- Indexes
CREATE INDEX idx_cert_san       ON certificate USING GIN (san_list);
CREATE INDEX idx_cert_serial    ON certificate (issuing_ca_id, serial_number);
CREATE INDEX idx_cert_expiring  ON certificate (valid_until) WHERE status = 'active';
CREATE INDEX idx_cert_customer  ON certificate (customer_id, status, valid_until);
CREATE INDEX idx_rev_crl        ON revocation (crl_published_at) WHERE crl_published_at IS NULL;
CREATE INDEX idx_ocsp_refresh   ON ocsp_response_cache (next_update);

Serial number design: never use a sequence. CA/B Forum Ballot SC63 requires at least 64 bits of CSPRNG entropy in every serial. Use os.urandom(20) (20 bytes = 160 bits), store as BYTEA. The UNIQUE(issuing_ca_id, serial_number) constraint catches the astronomically-rare collision and lets the application retry. Sequential serials would leak issuance volume to anyone watching CT logs.

§ 05 — ISSUANCE PIPELINEFrom CSR to signed certificate

Every certificate starts as a PKCS#10 CSR the applicant generates from their private key. The CA validates it, an HSM-held key signs and returns the cert. For DV-auto: under 300 ms end-to-end.

Applicant (customer / cert-manager operator)
  │
  │  1. Generate key pair locally — private key NEVER leaves applicant
  │  2. Create CSR: openssl req -new -key key.pem -out csr.pem
  │  3. POST /certificate  { csr_pem, order_id, dcv_method }
  │
  ▼
DigiCert RA — Registration Authority (validates domain control)
  │
  ├─ DV (Domain Validation — automated):
  │   http-01:      GET /.well-known/acme-challenge/{token}
  │                 → must return "{token}.{account_thumbprint}"
  │   dns-01:       TXT _acme-challenge.example.com
  │                 = base64url(sha256("{token}.{account_thumbprint}"))
  │   tls-alpn-01:  TLS handshake on :443 with acmeValidation-v1 OID in SAN
  │
  ├─ OV/EV: org checks (WHOIS, phone call, Dun & Bradstreet / GLEIF / LEI)
  │
  ├─ Policy linting:
  │   - Key size ≥ RSA 2048 or EC P-256+; SHA-256+ only; SAN required
  │   - Not validity > 90 days; no wildcards on EV; EKU must match cert type
  │   - Run pkilint + zlint — any WARN/ERR blocks issuance
  │
  ├─ CT pre-certificate submission (in parallel to 2+ independent logs):
  │   POST https://ct.googleapis.com/logs/argon2025h2/ct/v1/add-pre-chain
  │   POST https://ct.cloudflare.com/logs/nimbus2025/ct/v1/add-pre-chain
  │   ← receive Signed Certificate Timestamps (SCTs, 104 bytes each)
  │
  └─ HSM sign (PKCS#11 call; private key never leaves the HSM):
       Build final TBSCertificate with SCTs embedded
       Sign with Intermediate CA private key
       Return: { cert_pem, chain_pem }

Write to DB:
  INSERT certificate(..., ct_log_ids, status='active')
  INSERT ocsp_response_cache(cert_id, status='good', next_update=NOW()+48h, signed_response=...)
  INSERT dcv_event(validated_at=NOW(), ...)
  Publish to Kafka topic cert.issued → audit log → S3/Iceberg

§ 06 — REVOCATIONCRL, OCSP, and CRLite

Revocation must reach every browser within 24 hours (CA/B Forum §4.9.1.1). Three mechanisms, each with different trade-offs:

§ 07 — KUBERNETESPKI at cloud-native scale

In Kubernetes, three tools compose a complete PKI: cert-manager (ACME/DigiCert lifecycle), SPIFFE/SPIRE (cryptographic workload identity), and Istio/Linkerd (transparent mTLS between every pod — no application code changes required).

cert-manager (the ACME operator)

---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: digicert-acme
spec:
  acme:
    server: https://acme.digicert.com/v2/acme/directory
    email: ops@example.com
    privateKeySecretRef:
      name: digicert-acme-account-key
    solvers:
      - dns01:
          route53:
            region: us-east-1
            accessKeyIDSecretRef:
              name: route53-creds
              key: access-key-id
            secretAccessKeySecretRef:
              name: route53-creds
              key: secret-access-key

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example-com-tls
  namespace: default
spec:
  secretName: example-com-tls-secret
  issuerRef:
    name: digicert-acme
    kind: ClusterIssuer
  dnsNames:
    - example.com
    - "*.example.com"
  duration: 2160h          # 90 days
  renewBefore: 720h        # renew 30 days before expiry
  privateKey:
    algorithm: ECDSA
    size: 256              # P-256

SPIFFE / SPIRE (workload identity)

# Registration: bind k8s service account → SPIFFE ID
# spiffe://cluster.local/ns/payments/sa/checkout-api
#   selector: k8s:ns:payments + k8s:sa:checkout-api
#   ttl: 3600  (1-hour SVIDs, auto-rotated by SPIRE agent)

# Each pod gets its SVID at /run/spire/sockets/agent.sock
# Istio or app reads the SVID and uses it for mTLS

# AuthorizationPolicy: only checkout-api may call fraud-scorer
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: fraud-scorer-allow
  namespace: payments
spec:
  selector:
    matchLabels:
      app: fraud-scorer
  rules:
    - from:
        - source:
            principals:
              - "cluster.local/ns/payments/sa/checkout-api"

The hot-reload problem

cert-manager writes a renewed cert into the Kubernetes Secret, but running pods don't reload automatically. Three failure modes: env var mount (never updates — requires pod restart); volume mount (kubelet syncs in 60 s but app must re-read; most TLS libraries don't); cached TLS context (Go's tls.Config / Java's SSLContext loaded at startup). Fix: use Go's tls.Config.GetCertificate callback, or let Istio/Envoy handle TLS termination (xDS reloads with zero downtime).

§ 08 — DIGICERT AT SCALEThe architecture decisions

Four simultaneous system-design problems: low-latency issuance, globally-distributed revocation, 7-year audit retention, and Kubernetes automation.

§ 09 — NUANCED Q&ATwenty-two questions the loop actually asks

These questions separate the staff-level answer from the senior one — curl behavior, REST API flows, lifecycle triggers, and how certs accumulate inside a real machine.

CLI & curl

REST API & the ACME Protocol

Lifecycle — When Does DigiCert Know to Issue a New Cert?

How Certs Grow & Sink on a Machine

Edge Cases & Gotchas

§ 10 — THE ECOSYSTEMCLM, TLM, competitors, Cloudflare, and what startups actually do

CLM vs TLM — what's the difference?

CLM (Certificate Lifecycle Management) is the generic industry term for the practice of tracking, issuing, renewing, and revoking certificates across an organization — knowing where every cert is, who owns it, and when it expires. Without CLM, enterprises discover expired certs when production goes down at 2 AM.

TLM (Trust Lifecycle Manager) is DigiCert's commercial product name for their CLM platform — it's their branded answer to the question "how does a Fortune 500 manage 50,000 certificates across 200 systems?" TLM adds discovery (scan your network for certs you didn't know existed), policy enforcement (alert if someone installs a self-signed cert), automation hooks (integrate with Venafi, ServiceNow, HashiCorp Vault), and executive dashboards showing cert risk posture.

The CA competitive landscape

How Cloudflare and hosting providers choose a CA

When you buy a domain on GoDaddy or Squarespace and HTTPS "just works," someone made a CA partnership decision on your behalf. Here's how it works:

CLOUDFLARE

Cloudflare issues its own certs via Universal SSL — every site behind Cloudflare gets a free TLS cert automatically. For the Cloudflare-to-origin connection, you can use a DigiCert-issued cert, a Cloudflare-issued cert, or a self-signed cert (Cloudflare verifies it via its own CA). Cloudflare historically used DigiCert for its edge certs; in 2023 it announced a shift toward issuing certs via its own CA (Cloudflare PKI) and Let's Encrypt for DV, reducing DigiCert dependency. For enterprise customers, Cloudflare Advanced Certificate Manager lets you specify DigiCert, Let's Encrypt, or Google as the CA.

AWS

AWS Certificate Manager (ACM) issues free TLS certs for use with ALB, CloudFront, and API Gateway. ACM certs are issued from Amazon's own CA (Amazon Trust Services — root: Amazon Root CA 1/2/3/4). You cannot export the private key — it's HSM-backed in AWS. For custom CAs or exportable certs, use ACM Private CA (paid). For non-AWS resources, you must use a commercial CA like DigiCert or Let's Encrypt.

GoDaddy / Namecheap / web hosts

These resell DigiCert, Sectigo, or SSL.com certs at markup. "Free SSL" on cheap hosts is typically Let's Encrypt. Premium "business SSL" is typically Sectigo or DigiCert OV. The host automates issuance and renewal via their control panel (cPanel, Plesk) — the customer never sees the ACME protocol.

HOW THE CHOICE IS MADE

Factors: (1) price — Let's Encrypt is free, DigiCert is expensive; (2) cert type needed — Let's Encrypt is DV only, so OV/EV requires a commercial CA; (3) automation support — all modern CAs support ACME; (4) SLA and support — enterprise hosting requires guaranteed uptime on OCSP/issuance and 24/7 human support; (5) compliance — PCI-DSS, HIPAA, and government contracts may specify DigiCert, Entrust, or Sectigo by name.

What startups do for inexpensive (or free) certificates

§ 11 — REAL-WORLD SCENARIOSCertificates in the wild

How DigiCert saves the day: the Canvas school hack scenario

Canvas (Instructure) is the LMS used by millions of students and teachers. In a real-world breach scenario, here's where certificates are both the attack surface and the defense:

THE ATTACK SURFACE

A school district's Canvas instance runs on a subdomain like mydistrict.instructure.com. If an attacker can get a TLS certificate for that subdomain — even fraudulently — they can stand up a convincing phishing site that shows a valid padlock, harvest student login credentials, and access gradebooks, 504 plans, and IEP documents. Before Certificate Transparency, this was possible without detection. A rogue CA could issue a cert for mydistrict.instructure.com and nobody would know.

HOW CT STOPS IT

Certificate Transparency means any cert issued for *.instructure.com appears in public logs within minutes. Instructure (and DigiCert's monitoring service) watches CT logs for their domains. An unauthorized cert triggers an immediate alert — revocation request filed within the hour. The attack window shrinks from "months before discovery" (DigiNotar 2011) to "minutes." Google's Safe Browsing also ingests CT alerts and can push warnings to Chrome users visiting a phishing site using a recently-logged suspicious cert.

MKCERT FOR DEV SAFETY

Teachers and IT staff sometimes run local Canvas instances for development. mkcert is a tool that creates a locally-trusted CA and issues certs for localhost and *.test domains — the cert is trusted only on the machine where mkcert installed its CA root. This avoids the red X in local dev without adding self-signed exceptions to the system trust store, and the locally-trusted CA is never exposed to the internet.

STUDENT DATA & COMPLIANCE

FERPA (US education privacy law) requires that student data be protected in transit — HTTPS is mandatory, not optional. A school district whose Canvas instance serves HTTP is in FERPA violation. DigiCert and Instructure's SLA guarantees that OCSP responders and the certificate chain are available 99.99% of the time — because a cert outage on Canvas during finals week is a compliance event, not just a downtime event.

Cloud provider certificate management

Certificates in SaaS applications — Salesforce as the example

SaaS applications like Salesforce use certificates at three distinct layers, each with different ownership and lifecycle:

LAYER 1 — SALESFORCE'S OWN TLS (you don't control this)

login.salesforce.com, yourorg.my.salesforce.com — Salesforce issues and manages these certs (DigiCert or GTS). They renew automatically. If they expire, it's Salesforce's outage, not yours. Zero customer action required.

LAYER 2 — CUSTOM DOMAIN TLS (you control this)

If you've set up a custom domain like crm.example.com pointing to Salesforce, you own and upload that cert. Salesforce's My Domain feature accepts a PEM cert + private key from DigiCert, Sectigo, or any CA. You are responsible for renewal. This is the #1 source of Salesforce cert expiry outages in enterprise — a domain team uploads a 1-year cert, the Salesforce admin who set it up leaves, and 12 months later CRM goes dark.

LAYER 3 — API INTEGRATION / mTLS (you control this)

Salesforce Connected Apps can require client certificates for API authentication (mutual TLS). The client cert proves to Salesforce that the calling system is who it claims — no password needed. You generate the key pair, upload the public cert to Salesforce's Connected App config, and use the private key in your integration service. These certs also expire and must be rotated without interrupting the integration.

Certificates in cloud-native infrastructure

A typical modern cloud-native stack uses certificates at every layer simultaneously — and they're all managed differently:

The operational challenge: these six layers each have different renewal schedules, different owners, and different failure modes. A mature org has a cert inventory (from CLM/TLM or a Prometheus exporter) that surfaces all of them in one dashboard — so the 2 AM alert is "cert expires in 13 days" not "site is down."

§ 12 — SUMMARYWhat the strong answer looks like

← Back to Design Scenarios