feat(replication): support application health and controlled two-node recovery

Allow explicit 200 health responses for application origins and share only
in-flight readiness probes without caching their results. Document the accepted
native primary/replica pattern over the existing Xray transport, with fenced
manual promotion and independent node repositories.

Validation: 18 Worker tests, 17 readiness tests, and git diff --check passed.
Production SSO migration and traffic activation remain separate rollout steps.

Disclosure: I'm an AI assistant helping the author; I'm not the author.

docs/rfcs/composition-replication.md

@@ -1,6 +1,6 @@
 # RFC Reusable composition replication patterns
 
-Status: Accepted architecture. The initial SSO rollout remains pending compatibility, routing, and failure verification; this document does not claim achieved high availability.
+Status: Accepted patterns; SSO revised to two locations with controlled promotion. The rollout remains pending migration, routing, and failure verification; this document does not claim achieved high availability.
 
 ## Purpose and decision
 
@@ -85,6 +85,10 @@ Shared helpers may cover common tasks. Native backend tools retain ownership of
 
 Reuse an existing private transport when tests demonstrate that it supports the application's actual protocols. Validate advertised addresses, bidirectional connectivity, required TCP and UDP ports, reconnect behavior, and relevant MTU constraints. Do not silently replace the fleet's routing with a new overlay.
 
+Validate the peer identities actually learned after reconnect, not just the configured addresses or successful initial connections. Some protocols derive membership from the incoming TCP source address: Galera combines that address with the advertised port. A TCP proxy that replaces multiple peer sources with a gateway address does not preserve this contract. Such groups need a transport that preserves distinct peer source addresses. [Galera handshake implementation](https://github.com/codership/galera/blob/release_26.4.27/gcomm/src/gmcast_proto.cpp#L222)
+
+The current fleet rollout uses the existing Xray TPROXY tunnels only. WireGuard is excluded. Restrict peer addresses and ports, reject fallback through unrelated routes when a peer is unreachable, and verify reconnect and container-network replacement behavior. A protocol incompatible with this transport requires a different application replication pattern, not an unapproved network overlay.
+
 Preserve distinct node-local bridge networks and addresses. Docker service names and host-local container IP assumptions must not be mistaken for cross-host service discovery. Peer endpoints must be explicitly reachable from the consumers that use them.
 
 A logical application endpoint routes only to ready instances. Dependency failure and loss of write authority must affect readiness. Both browser-facing endpoints and internal application calls must survive the same location-loss scenario. A single reverse proxy on one participating site must not be the only traffic entry point.
@@ -117,21 +121,21 @@ Backups remain independent of replication and cover accidental deletion, corrupt
 
 ## SSO first application
 
-Daedalus and Metis are separate repositories and application locations. Icarus is the proposed independent location for backend voting services. Other compositions on those nodes do not become replicas by association.
+Daedalus and Metis are separate repositories and application locations. The selected SSO pattern is one active deployment and one standby, with controlled manual promotion. Icarus is not a member of the target SSO topology. Other compositions on these nodes do not become replicas by association.
 
-The candidate SSO arrangement has two Authelia/nginx deployments and replicated persistent and session state. MariaDB/Galera can use two full data members plus an Icarus arbitrator. Valkey uses a native replication topology with three Sentinel voters. These may be dedicated compositions or reusable backend groups; SSO does not require a global SQL/cache platform for every APEX application.
+The target uses native MariaDB primary/replica replication and a primary/replica session store over the existing Xray tunnels. It excludes Galera, its arbitrator, Sentinel elections, and custom automatic promotion controllers. The standby must not accept application writes until an operator has fenced the previous writer and explicitly promoted the replacement. An unreachable SSH or tunnel endpoint alone does not prove fencing. Native replication status and the last applied position must be checked and any possible data loss recorded before promotion.
 
-The Galera arbitrator provides voting without another SQL data copy. The remaining data member and arbitrator can preserve quorum after one data location fails. WAN latency and stable connectivity must be checked before choosing this topology. [MariaDB deployment variants](https://mariadb.com/docs/galera-cluster/galera-architecture/galera-cluster-deployment-variants/)
+The existing public users source is a manually maintained YAML file. The selected replacement is LLDAP backed by MariaDB, with its enrolled directory preserved during the change from the staged Galera deployment to ordinary replication. Existing account identifiers, profile attributes and application groups must survive migration. The operator has already completed the one-time password enrollment; changing replication must not require another enrollment or overwrite the directory. Keep the old public SSO authoritative until migration checks pass. [Authelia LLDAP integration](https://www.authelia.com/integration/ldap/lldap/)
 
-The existing users source is a manually maintained YAML file. The selected replacement is LLDAP backed by the replicated MariaDB group, with a local LLDAP instance at each SSO location. Authelia supports this directory implementation. Existing account identifiers, profile attributes and application groups must survive migration. LLDAP cannot import the existing YAML password hash; the operator has accepted one password enrollment through a private setup page. Keep the old SSO authoritative until enrollment and migration checks pass. [Authelia LLDAP integration](https://www.authelia.com/integration/ldap/lldap/)
+The two identity locations use `198.18.16.150` on Daedalus and `198.18.18.150` on Metis. The existing SSO frontend addresses remain separate. These are explicit allocations for this deployment, not reserved addresses or mandatory topology for other APEX applications. The staged Icarus voting services must be retired only after verifying that the preserved data and new replication no longer depend on them; unrelated Icarus applications remain outside this change.
 
-The initial placement uses a separate `identity` composition at `198.18.16.150` on Daedalus, `198.18.18.150` on Metis and `198.18.17.150` on Icarus. Daedalus and Metis own SQL data and directory instances; Icarus owns the Galera arbitrator. Each location also owns a Valkey data member and Sentinel voter. The existing SSO frontend addresses remain separate. These are explicit allocations for this deployment, not reserved addresses or mandatory topology for other APEX applications.
+Stage the new backend beside the existing public SSO database. Preserve a verified backup and use a consistent logical export for migration. Ordinary container restart must never initialize a replacement database, replay a seed snapshot, or promote a replica. Test direct Daedalus–Metis Xray connectivity without Icarus as an intermediary. Returning former primaries remain fenced until explicitly resynchronized as replicas.
 
-Stage the new backend beside the existing SSO database. Use an explicit one-time SQL bootstrap, normal native joins thereafter, and a consistent logical export for migration. Ordinary container restart must never repeat cluster bootstrap or replay an old session snapshot. A direct private Daedalus–Metis transport is required so that Icarus loss does not also remove connectivity between the surviving data members. Password enrollment, verified replication transport and local dependency readiness are deployment gates, not consequences of starting containers.
+Asynchronous database and session replication can lose acknowledged recent changes. This pattern does not promise seamless preservation of every account change, session, or revocation. The promotion procedure must document the replication position, possible loss and whether users need to log in again. Session invalidation after promotion must be tested so that stale replicated sessions do not silently restore revoked access.
 
-Valkey replication is asynchronous and can lose acknowledged recent changes. The proposed SSO pattern therefore cannot claim seamless preservation of every session or revocation. Its production profile must state and test the resulting security and user-visible behavior. A requirement for lossless session state requires a different validated arrangement, not a stronger label on Sentinel.
+The earlier automatic recovery target of 60 seconds is superseded by controlled promotion. Loss of the active stateful deployment can interrupt authentication until promotion completes; loss of the standby must not stop the active deployment. Record the measured promotion and recovery time rather than claiming an automatic recovery bound.
 
-The starting recovery target is automatic restoration of authentication within 60 seconds after loss of any one participating location. This is a proposed acceptance target, not an achieved guarantee. Session behavior is assessed separately. All SSO dependencies, including user data, required notifications, internal authentication routes, and public routing, must be included before declaring that target met.
+Cloudflare traffic failover is requested for `sso.ermnvldmr.com`, `jef.ermnvldmr.com`, `jpl.ermnvldmr.com`, `sea.ermnvldmr.com`, and `imh.ermnvldmr.com`, using the no-additional-cost Worker approach. Traffic selection must not promote databases or treat an unpromoted standby as ready. Each hostname needs independently verified origin routing and application readiness. Two ingress paths to a single application backend provide ingress redundancy only; they do not replicate that backend or its data. Preserve existing streaming, WebSocket, authentication, and path-specific routes.
 
 ## Adding the next application