Skip to main content

ADR 0007: Keycloak HA with two replicas and jdbc-ping

  • Status: accepted
  • Date: 2026-11-07 (production cutover observed the same day)
  • Decision makers: the project owner, as recorded in the Keycloak fix plan

Context and problem statement​

Keycloak ran as one pod. Any restart signed everyone out, and with enforcement on, Keycloak became a hard dependency of every protected path. How should it be made highly available?

Decision drivers​

  • Zero failed requests during a pod restart or rolling update.
  • Fewer moving parts in a cluster that already depends on one database.
  • A session created on one pod must be valid on the other.

Considered options​

  • Keep one replica and add an alert.
  • Two replicas clustered with the kubernetes stack, which needs a headless Service and DNS discovery.
  • Two replicas clustered with jdbc-ping, which discovers peers through the shared database.

Decision outcome​

Two replicas with KC_CACHE=ispn and KC_CACHE_STACK=jdbc-ping, no headless Service. jdbc-ping is the documented default for 26.1 and what the running pod already used, so old and new pods formed one cluster during the roll. The manifest adds a PodDisruptionBudget (minAvailable: 1), a rolling update with maxUnavailable: 0, a topology spread constraint across nodes, probes on management port 9000, KC_METRICS_ENABLED and a capped JVM heap. The kubernetes stack with a headless Service is the documented plan B.

Consequences​

  • Good: the rehearsal showed zero non-2xx responses through graceful termination and rolling restarts, and tokens issued by one pod were accepted by the other.
  • Bad: Keycloak still depends on one PostgreSQL pod, so it is one database outage from unavailable.
  • Bad: JGroups traffic between the pods is not encrypted.
  • Bad: the rehearsal could not reproduce kube-proxy endpoint propagation, so a live failover test was still required.

Confirmation​

On 2026-11-07 the Deployment showed 2 of 2 replicas ready and keycloak-pdb existed. The local rehearsal harness is infra/keycloak/rehearsal/.

Sources​

.agents/tasks/keycloak-fix/plan.md, docs/runbooks/keycloak-fix-final-report.md, infra/k8s/keycloak.yaml, live read on 2026-11-07.