feat(replication): support application health and controlled two-node recovery

Allow explicit 200 health responses for application origins and share only
in-flight readiness probes without caching their results. Document the accepted
native primary/replica pattern over the existing Xray transport, with fenced
manual promotion and independent node repositories.

Validation: 18 Worker tests, 17 readiness tests, and git diff --check passed.
Production SSO migration and traffic activation remain separate rollout steps.

Disclosure: I'm an AI assistant helping the author; I'm not the author.

assets/composition-failover/README.md

@@ -1,6 +1,6 @@
 # Readiness-gated origin routing on Workers Free
 
-This local ES-module Worker asset selects between two origins for one public HTTPS hostname. It implements the ingress part of the [composition replication RFC](../../docs/rfcs/composition-replication.md). It is not deployed and does not establish production high availability. Activate it only after backend HA and authenticated application readiness are verified.
+This local ES-module Worker asset selects between two origins for one public HTTPS hostname. It implements the ingress part of the [composition replication RFC](../../docs/rfcs/composition-replication.md). It is not deployed and does not establish production high availability. Deploy one instance per public hostname, with independently verified origin routes and readiness. Activate only after the declared backend failure behavior and authenticated application readiness are verified. The Worker does not promote databases or activate a standby.
 
 The module has no runtime dependencies, paid bindings, KV, R2, storage or scheduled jobs. This directory does not enable billing, enroll an account, upload a Worker or change DNS/routes.
 
@@ -16,6 +16,7 @@ Set these string bindings through the operator's chosen Cloudflare deployment pr
 | `APEX_ORIGIN_SECONDARY` | Dedicated DNS-only secondary origin hostname in that zone |
 | `APEX_READY_PATH` | Dedicated absolute readiness path, for example `/.well-known/apex-ready` |
 | `APEX_READY_TOKEN` | High-entropy secret shared only by this Worker and both readiness adapters |
+| `APEX_READY_STATUS` | Optional exact status string: `204` (default) or `200` for a dedicated application health route |
 
 Hostnames accept ordinary ASCII DNS labels, without a scheme, port, trailing dot, wildcard or IP literal. Case is normalized. Public and origin hostnames must be distinct and within the declared zone. Readiness paths accept ASCII letters, digits, `.`, `_`, `~`, `-` and `/`; queries, fragments, escapes, repeated slashes and dot traversal segments are rejected. The secret must be nonempty visible ASCII without whitespace. Invalid/missing bindings produce a generic no-store 503 before any fetch. Syntax validation cannot establish actual DNS ownership or prevent an operator from naming a separately delegated zone; deployment must verify both origins are in the actual same Cloudflare zone.
 
@@ -42,9 +43,9 @@ The required non-secret upload metadata fields are:
 
 Use this as the multipart API `metadata` portion, with the ES-module file part named `worker.mjs`. Bindings and the readiness secret are configured separately through the operator-owned enrollment process; they are deliberately absent from this example, as are paid bindings. This metadata is not a complete enrollment or route activation command. Verify the resulting effective settings before activation, including after later uploads. [Multipart upload metadata](https://developers.cloudflare.com/workers/configuration/multipart-upload-metadata/)
 
-Every accepted HTTPS request first probes the primary with a GET at `https://APEX_PUBLIC_HOST/APEX_READY_PATH`. The request carries only `X-APEX-Ready-Token`, rather than client cookies or authorization. `cf.resolveOverride` selects the DNS-only origin while the URL remains the public hostname. Only status 204 is ready; redirects are manual and never accepted. Each probe has a two-second deadline. Unread response bodies are canceled, including responses arriving after timeout. Failed primary readiness leads to a secondary probe; neither ready produces a no-store 503. There is no readiness cache or background health state.
+Every accepted HTTPS request first probes the primary with a GET at `https://APEX_PUBLIC_HOST/APEX_READY_PATH`. The request carries only `X-APEX-Ready-Token`, rather than client cookies or authorization. `cf.resolveOverride` selects the DNS-only origin while the URL remains the public hostname. Only the configured exact status is ready: 204 by default, or 200 when explicitly selected with `APEX_READY_STATUS`. Redirects are manual and never accepted. Do not use 200 with a generic login page or native liveness endpoint; a protected application health route must verify both active SSO dependencies and its own backend. Each probe has a two-second deadline. Unread response bodies are canceled, including responses arriving after timeout. Failed primary readiness leads to a secondary probe; neither ready produces a no-store 503. There is no readiness cache or background health state.
 
-The dedicated origin adapter must authenticate the token, reject its absence/incorrect value and prove that this application instance can safely serve traffic: required dependencies, compatible configuration, current write authority/quorum, session state and any application-specific prerequisites. Authelia `/api/health`, an nginx 204, a TCP check or a running container alone is insufficient. Protect adapter aliases and normalize paths consistently at the origin. The Worker rejects public requests to the configured readiness path (also with a query or percent-encoded spelling), and removes every client-supplied readiness header before application dispatch. This asset does not implement or enroll the backend adapter.
+The dedicated origin adapter must authenticate the token, reject its absence/incorrect value and prove that this application instance can safely serve traffic: required dependencies, compatible configuration, current write authority for the declared topology, session state and any application-specific prerequisites. Authelia `/api/health`, an nginx 204, a TCP check or a running container alone is insufficient. Protect adapter aliases and normalize paths consistently at the origin. The Worker rejects public requests to the configured readiness path (also with a query or percent-encoded spelling), and removes every client-supplied readiness header before application dispatch. An unpromoted standby must remain unready. This asset does not implement or enroll the backend adapter. Preserve existing path-specific routes when attaching the Worker; two ingress paths do not replicate a shared application backend.
 
 Cloudflare documents `resolveOverride` as DNS substitution while retaining the URL's Host, subject to both names belonging to the same zone. The asset retains the HTTPS public URL for routing; actual origin TLS SNI/certificate selection and routing must be demonstrated natively before activation. No local Node test proves Cloudflare DNS/TLS behavior. [Request API](https://developers.cloudflare.com/workers/runtime-apis/request/)
 
@@ -62,7 +63,7 @@ Other hostnames, non-default ports and non-HTTPS requests receive no-store 421.
 
 ## Free-tier deployment gates
 
-Workers Free currently provides **100,000 incoming requests per day per account**, reset at midnight UTC, and **10 ms CPU per HTTP request**. Quota is shared with the account's other Workers. This asset performs two fetch subrequests normally (one probe plus application dispatch), or three when primary readiness fails. Waiting for network I/O does not count toward CPU time, but actual Worker CPU consumption must be measured; Node wall time is not that measurement. [Cloudflare Workers limits](https://developers.cloudflare.com/workers/platform/limits/)
+Workers Free currently provides **100,000 incoming requests per day per account**, reset at 03:00 MSK (midnight UTC), and **10 ms CPU per HTTP request**. Quota is shared with the account's other Workers. This asset performs two fetch subrequests normally (one probe plus application dispatch), or three when primary readiness fails. Waiting for network I/O does not count toward CPU time, but actual Worker CPU consumption must be measured; Node wall time is not that measurement. [Cloudflare Workers limits](https://developers.cloudflare.com/workers/platform/limits/)
 
 Keep the account on Workers Free. Do not enable a paid plan, paid load balancer or paid bindings for this asset. An operator must inspect existing account usage and peak traffic to confirm the free quota is sufficient. This is a no-added-cost deployment candidate, not unlimited availability.
 
@@ -70,11 +71,11 @@ Set the Worker Route to **fail closed**. On Free quota exhaustion, Cloudflare do
 
 Operator-owned enrollment gates:
 
-1. Validate backend HA, adapter authentication/readiness, dependency loss and write-authority withdrawal independently of this Worker.
+1. Validate the declared backend failure behavior, adapter authentication/readiness, dependency loss and write-authority withdrawal independently of this Worker.
 2. Confirm account Free plan/quota, same-zone DNS-only origin records, public proxied Route, strict TLS, native SNI/Host at each origin, required private-origin/cache compatibility flags and date, and no Worker/route recursion or pre-Worker caching.
 3. Test native primary/secondary readiness, both-unready failure, redirects, cookies, cache bypass, POST and GET callback non-replay, streaming and any required WebSockets. Measure Worker CPU and account request headroom.
 4. Verify fail-closed quota handling and exercise location loss, network partitions, stale/rejoining origins and recovery. Record observed recovery time and session behavior before making an HA claim.
-5. Activate only after those gates pass with the intended backend HA. Maintain a documented operator rollback that does not silently route to an unsafe origin.
+5. Activate only after those gates pass with the intended backend failure behavior. Maintain a documented operator rollback that does not silently route to an unsafe origin.
 
 ## Local verification
 

assets/composition-failover/worker.mjs

@@ -23,7 +23,9 @@ function configuration(env) {
       || path.includes('//') || path.split('/').some((part) => part === '.' || part === '..')) return null;
   const token = env.APEX_READY_TOKEN;
   if (typeof token !== 'string' || !/^[\x21-\x7e]+$/.test(token)) return null;
-  return { publicHost, primary, secondary, path, token };
+  const status = env.APEX_READY_STATUS === undefined ? '204' : env.APEX_READY_STATUS;
+  if (!['200', '204'].includes(status)) return null;
+  return { publicHost, primary, secondary, path, token, status: Number(status) };
 }
 
 function unavailable(status = 503, message = 'Service unavailable') {
@@ -60,7 +62,7 @@ async function ready(config, origin) {
     });
     const inspection = fetch(probe, { ...options(origin), signal: controller.signal })
       .then((response) => {
-        const accepted = !controller.signal.aborted && response.status === 204;
+        const accepted = !controller.signal.aborted && response.status === config.status;
         discard(response);
         return accepted;
       }, () => false);

assets/composition-failover/worker.test.mjs

@@ -114,6 +114,23 @@ test('readiness redirect is never accepted or followed', async (t) => {
   assert.equal(calls[0].options.redirect, 'manual');
 });
 
+test('application health can explicitly require 200 while 204 remains the default', async (t) => {
+  const bindings = { ...env, APEX_READY_STATUS: '200' };
+  const calls = intercept(t, (_, __, count) => count === 1
+    ? new Response(null, { status: 204 })
+    : count === 2 ? new Response('Healthy', { status: 200 }) : new Response('secondary'));
+  const response = await worker.fetch(new Request(url), bindings);
+  assert.equal(await response.text(), 'secondary');
+  assert.equal(calls.length, 3);
+  assert.equal(calls[2].options.cf.resolveOverride, env.APEX_ORIGIN_SECONDARY);
+});
+
+test('an unconfigured 200 health response cannot release application traffic', async (t) => {
+  const calls = intercept(t, () => new Response('login page', { status: 200 }));
+  assert.equal((await worker.fetch(new Request(url), env)).status, 503);
+  assert.equal(calls.length, 2);
+});
+
 for (const method of ['GET', 'POST']) {
   test(`failed ${method} dispatch is never replayed on another origin`, async (t) => {
     const calls = intercept(t, (_, __, count) => {
@@ -174,6 +191,9 @@ test('missing and invalid bindings fail closed without diagnostics or probes', a
     ['APEX_READY_PATH', '//evil.example/'], ['APEX_READY_PATH', '/ready?query=yes'],
     ['APEX_READY_PATH', '/ready#fragment'], ['APEX_READY_PATH', '/a/../ready'],
     ['APEX_READY_PATH', '/%72eady'], ['APEX_READY_TOKEN', 'secret\r\ninjected: yes'],
+    ['APEX_READY_STATUS', '302'], ['APEX_READY_STATUS', '401'],
+    ['APEX_READY_STATUS', '503'], ['APEX_READY_STATUS', '2xx'],
+    ['APEX_READY_STATUS', ''], ['APEX_READY_STATUS', 200], ['APEX_READY_STATUS', null],
   ]) invalid.push({ ...env, [name]: value });
   invalid.push(null);
   for (const bindings of invalid) {

assets/composition-readiness/README.md

@@ -1,6 +1,6 @@
 # Authenticated composition readiness adapter
 
-This dependency-free Python adapter exposes one local HTTP endpoint for the readiness-gated Worker and origin proxy. It runs an operator-supplied executable for every authenticated probe; exit0 means ready, every other outcome means unavailable. The executable must establish the application's actual readiness contract. A successful TCP connection or container health flag is not enough for replicated SSO.
+This dependency-free Python adapter exposes one local HTTP endpoint for the readiness-gated Worker and origin proxy. By default it runs an operator-supplied executable for every authenticated probe; exit0 means ready, every other outcome means unavailable. The executable must establish the application's actual readiness contract. A successful TCP connection or container health flag is not enough for replicated SSO.
 
 Example private configuration:
 
@@ -12,7 +12,8 @@ Example private configuration:
   "token_file": "/private/readiness-token",
   "command": ["/usr/local/bin/python3", "/assets/probe.py", "--config", "/private/probe.json"],
   "timeout": 1.5,
-  "slots": 4
+  "slots": 4,
+  "single_flight": false
 }
 ```
 
@@ -22,6 +23,8 @@ Only GET at the exact canonical path, with one matching `X-APEX-Ready-Token` hea
 
 Probe timeout is bounded to1.5seconds so an origin can answer within the Worker's two-second deadline. A timeout kills the probe's process group. Probe slots and request threads are bounded; slow socket operations time out after two seconds. Operators must account for proxy overhead and actual probe execution time. Probe programs must not detach into a new session or launch background jobs.
 
-The SSO probe must verify the local Authelia instance/config generation, local authenticated SQL and directory access, completed migration, SQL Primary/Synced/write authority, and fresh session-controller authority plus actual verified TLS application access to its selected primary. It must reject uncertain/stale state and avoid inventing a competing election. A short-lived controller status file alone is not a consensus lease. The generic adapter supplies none of those application checks; do not deploy a placeholder-success probe or claim these tests establish HA.
+Optional `single_flight: true` lets overlapping authenticated requests share one running fixed-command probe. It accepts only a JSON boolean and defaults to false. Waiters use that probe's remaining monotonic deadline; they never queue another probe for the same request. The existing connection limit (`slots * 2 + 4`) also bounds waiting requests. Timeout or shutdown returns unavailable and terminates the probe group. Completed results are not cached: each subsequent request starts a fresh probe, with no success TTL or failure cache. Coalescing reduces concurrent work but does not make a slow individual probe meet its deadline.
+
+For the controlled active/standby SSO pattern, the probe must reject a node not explicitly configured as active, verify completed migration and unchanged Authelia configuration, and check local Authelia health, authenticated SQL access/schema with read_only disabled, directory access, and an authenticated session write/read/delete over verified TLS. All dependencies belong to the selected local identity instance. These checks do not establish exclusive primary ownership: the operator must fence the previous writer before promotion, and the standby application and ingress stay disabled. No quorum, Sentinel, or session-controller authority is required by this pattern. The generic adapter supplies none of those application checks; do not deploy a placeholder-success probe or claim these tests establish HA.
 
 Run `python3 test_adapter.py` for loopback HTTP and subprocess behavior checks. No third-party packages are required. Production native image/probe/proxy integration and location-loss tests remain separate acceptance gates.

assets/composition-readiness/adapter.py

@@ -9,6 +9,7 @@ import re
 import signal
 import subprocess
 import threading
+import time
 
 
 class ReadinessServer(http.server.ThreadingHTTPServer):
@@ -16,7 +17,7 @@ class ReadinessServer(http.server.ThreadingHTTPServer):
     block_on_close = False
     allow_reuse_address = True
 
-    def __init__(self, address, *, token, path, command, timeout=1.5, slots=4):
+    def __init__(self, address, *, token, path, command, timeout=1.5, slots=4, single_flight=False):
         if not isinstance(token, str) or not re.fullmatch(r'[!-~]{16,512}', token):
             raise ValueError('A private visible-ASCII token is required')
         if not isinstance(path, str) or not re.fullmatch(r'/[A-Za-z0-9._~/-]+', path) or any(part in ('', '.', '..') for part in path[1:].split('/')):
@@ -25,10 +26,14 @@ class ReadinessServer(http.server.ThreadingHTTPServer):
             raise ValueError('A fixed absolute executable and argv are required')
         if isinstance(timeout, bool) or not 0 < timeout <= 1.5 or isinstance(slots, bool) or not isinstance(slots, int) or not 1 <= slots <= 16:
             raise ValueError('Invalid probe bounds')
+        if not isinstance(single_flight, bool):
+            raise ValueError('single_flight must be a boolean')
         self.token = token.encode('ascii')
         self.ready_path = path
         self.command = command
         self.timeout = timeout
+        self.single_flight = single_flight
+        self.flight = None
         self.probe_slots = threading.BoundedSemaphore(slots)
         self.connection_slots = threading.BoundedSemaphore(slots * 2 + 4)
         self.active = set()
@@ -74,6 +79,31 @@ class ReadinessServer(http.server.ThreadingHTTPServer):
         process.wait()
 
     def probe(self):
+        if not self.single_flight:
+            return self.run_probe()
+        with self.active_lock:
+            if self.closing:
+                return False
+            flight = self.flight
+            leader = flight is None
+            if leader:
+                flight = {'deadline': time.monotonic() + self.timeout, 'done': threading.Event(), 'ready': False}
+                self.flight = flight
+        if not leader:
+            flight['done'].wait(max(0, flight['deadline'] - time.monotonic()))
+            with self.active_lock:
+                return flight['ready'] and not self.closing and time.monotonic() < flight['deadline']
+        ready = False
+        try:
+            ready = self.run_probe(flight['deadline'])
+        finally:
+            with self.active_lock:
+                flight['ready'] = ready and not self.closing and time.monotonic() < flight['deadline']
+                self.flight = None
+                flight['done'].set()
+        return flight['ready']
+
+    def run_probe(self, deadline=None):
         if not self.probe_slots.acquire(blocking=False):
             return False
         process = None
@@ -84,7 +114,8 @@ class ReadinessServer(http.server.ThreadingHTTPServer):
                 process = subprocess.Popen(self.command, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, start_new_session=True, close_fds=True)
                 self.active.add(process)
             try:
-                return process.wait(timeout=self.timeout) == 0
+                timeout = self.timeout if deadline is None else max(0, deadline - time.monotonic())
+                return process.wait(timeout=timeout) == 0
             except subprocess.TimeoutExpired:
                 self.stop_probe(process)
                 return False
@@ -99,6 +130,9 @@ class ReadinessServer(http.server.ThreadingHTTPServer):
     def server_close(self):
         with self.active_lock:
             self.closing = True
+            if self.flight is not None:
+                self.flight['ready'] = False
+                self.flight['done'].set()
             active = list(self.active)
         for process in active:
             self.stop_probe(process)

assets/composition-readiness/test_adapter.py

@@ -6,9 +6,9 @@ class AdapterTests(unittest.TestCase):
  def tearDown(self):
   for s,t in self.servers:s.shutdown();s.server_close();t.join()
   self.tmp.cleanup()
- def server(self,code=None,timeout=.3,slots=2):
+ def server(self,code=None,timeout=.3,slots=2,**options):
   code=code or f"from pathlib import Path;Path({str(self.marker)!r}).write_text('yes')"
-  server=ReadinessServer(('127.0.0.1',0),token='private-test-token',path='/.well-known/apex-ready',command=[sys.executable,'-c',code],timeout=timeout,slots=slots)
+  server=ReadinessServer(('127.0.0.1',0),token='private-test-token',path='/.well-known/apex-ready',command=[sys.executable,'-c',code],timeout=timeout,slots=slots,**options)
   t=threading.Thread(target=server.serve_forever,kwargs={'poll_interval':.01},daemon=True);t.start();self.servers.append((server,t));return server
  def request(self,s,path='/.well-known/apex-ready',token='private-test-token',method='GET',extra_token=False):
   c=http.client.HTTPConnection(*s.server_address,timeout=3);c.putrequest(method,path)
@@ -50,4 +50,70 @@ class AdapterTests(unittest.TestCase):
   self.assertFalse(self.marker.exists())
  def test_no_cached_readiness(self):
   s=self.server();self.assertEqual(self.request(s)[0],204);s.command=[sys.executable,'-c','raise SystemExit(1)'];self.assertEqual(self.request(s)[0],503)
+ def overlapping_requests(self,s,count=4):
+  barrier=threading.Barrier(count)
+  def one():
+   barrier.wait(timeout=2);start=time.monotonic();result=self.request(s);return result,time.monotonic()-start
+  with concurrent.futures.ThreadPoolExecutor(max_workers=count) as pool:return list(pool.map(lambda _:one(),range(count)))
+ def test_single_flight_shares_success_and_sequential_request_reprobes(self):
+  code=f"import pathlib,time;p=pathlib.Path({str(self.marker)!r});f=p.open('a');f.write('probe\\n');f.close();time.sleep(.2)"
+  s=self.server(code,timeout=1,slots=1,single_flight=True)
+  results=self.overlapping_requests(s)
+  self.assertEqual([r for r,_ in results],[(204,'no-store',b'')]*4)
+  self.assertEqual(self.marker.read_text().splitlines(),['probe'])
+  self.assertEqual(self.request(s)[0],204)
+  self.assertEqual(self.marker.read_text().splitlines(),['probe','probe'])
+ def test_single_flight_shares_failure_without_caching_it(self):
+  code=f"import pathlib,time;p=pathlib.Path({str(self.marker)!r});f=p.open('a');f.write('probe\\n');f.close();time.sleep(.2);raise SystemExit(1)"
+  s=self.server(code,timeout=1,slots=4,single_flight=True)
+  self.assertEqual([r for r,_ in self.overlapping_requests(s)],[(503,'no-store',b'')]*4)
+  self.assertEqual(self.marker.read_text().splitlines(),['probe'])
+  s.command=[sys.executable,'-c','pass'];self.assertEqual(self.request(s)[0],204)
+ def test_single_flight_timeout_wakes_waiters_and_kills_descendant(self):
+  code=f"import subprocess,time,sys;subprocess.Popen([sys.executable,'-c',\"import time,pathlib;time.sleep(.8);pathlib.Path({str(self.marker)!r}).write_text('leaked')\"]);time.sleep(3)"
+  s=self.server(code,timeout=.25,slots=1,single_flight=True)
+  results=self.overlapping_requests(s)
+  self.assertEqual([r for r,_ in results],[(503,'no-store',b'')]*4)
+  self.assertTrue(all(.15<elapsed<.7 for _,elapsed in results),results)
+  time.sleep(.9);self.assertFalse(self.marker.exists())
+ def test_single_flight_shutdown_wakes_waiters_and_rejects_new_requests(self):
+  leaked=self.root/'descendant'
+  child=f"import time,pathlib;pathlib.Path({str(self.marker)!r}).write_text('started');time.sleep(.8);pathlib.Path({str(leaked)!r}).write_text('leaked')"
+  code=f"import subprocess,time,sys;subprocess.Popen([sys.executable,'-c',{child!r}]);time.sleep(3)"
+  s=self.server(code,timeout=1.5,slots=1,single_flight=True)
+  with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool:
+   futures=[pool.submit(self.request,s) for _ in range(4)]
+   for _ in range(100):
+    if self.marker.exists():break
+    time.sleep(.005)
+   self.assertTrue(self.marker.exists());start=time.monotonic();s.server_close()
+   self.assertEqual([f.result()[0] for f in futures],[503]*4);self.assertLess(time.monotonic()-start,.7)
+  self.assertFalse(s.probe());time.sleep(.9);self.assertFalse(leaked.exists())
+ def test_single_flight_late_waiter_uses_remaining_deadline(self):
+  code=f"import pathlib,time;p=pathlib.Path({str(self.marker)!r});f=p.open('a');f.write('probe\\n');f.close();time.sleep(3)"
+  s=self.server(code,timeout=.4,slots=1,single_flight=True)
+  with concurrent.futures.ThreadPoolExecutor() as pool:
+   first=pool.submit(self.request,s)
+   for _ in range(100):
+    if self.marker.exists():break
+    time.sleep(.005)
+   self.assertTrue(self.marker.exists());time.sleep(.2);start=time.monotonic()
+   self.assertEqual(self.request(s)[0],503);self.assertLess(time.monotonic()-start,.3)
+   self.assertEqual(first.result()[0],503)
+  self.assertEqual(self.marker.read_text().splitlines(),['probe'])
+ def test_single_flight_connection_limit_remains_bounded(self):
+  s=self.server(slots=1,single_flight=True);connections=[]
+  try:
+   for _ in range(6):
+    sock=socket.create_connection(s.server_address,timeout=2);sock.sendall(b'GET /.well-known/apex-ready HTTP/1.1\r\n');connections.append(sock)
+   self.assertEqual(self.request(s),(503,'no-store',b''));self.assertFalse(self.marker.exists())
+  finally:
+   for sock in connections:sock.close()
+ def test_single_flight_still_authenticates_before_probe(self):
+  s=self.server(single_flight=True)
+  for token,duplicate in [(None,False),('bad',False),('private-test-token',True)]:self.assertEqual(self.request(s,token=token,extra_token=duplicate)[0],403)
+  self.assertFalse(self.marker.exists())
+ def test_single_flight_requires_explicit_boolean(self):
+  for value in [0,1,None,'true']:
+   with self.subTest(value=value),self.assertRaises(ValueError):self.server(single_flight=value)
 if __name__=='__main__':unittest.main()

docs/rfcs/composition-replication.md

@@ -1,6 +1,6 @@
 # RFC Reusable composition replication patterns
 
-Status: Accepted architecture. The initial SSO rollout remains pending compatibility, routing, and failure verification; this document does not claim achieved high availability.
+Status: Accepted patterns; SSO revised to two locations with controlled promotion. The rollout remains pending migration, routing, and failure verification; this document does not claim achieved high availability.
 
 ## Purpose and decision
 
@@ -85,6 +85,10 @@ Shared helpers may cover common tasks. Native backend tools retain ownership of
 
 Reuse an existing private transport when tests demonstrate that it supports the application's actual protocols. Validate advertised addresses, bidirectional connectivity, required TCP and UDP ports, reconnect behavior, and relevant MTU constraints. Do not silently replace the fleet's routing with a new overlay.
 
+Validate the peer identities actually learned after reconnect, not just the configured addresses or successful initial connections. Some protocols derive membership from the incoming TCP source address: Galera combines that address with the advertised port. A TCP proxy that replaces multiple peer sources with a gateway address does not preserve this contract. Such groups need a transport that preserves distinct peer source addresses. [Galera handshake implementation](https://github.com/codership/galera/blob/release_26.4.27/gcomm/src/gmcast_proto.cpp#L222)
+
+The current fleet rollout uses the existing Xray TPROXY tunnels only. WireGuard is excluded. Restrict peer addresses and ports, reject fallback through unrelated routes when a peer is unreachable, and verify reconnect and container-network replacement behavior. A protocol incompatible with this transport requires a different application replication pattern, not an unapproved network overlay.
+
 Preserve distinct node-local bridge networks and addresses. Docker service names and host-local container IP assumptions must not be mistaken for cross-host service discovery. Peer endpoints must be explicitly reachable from the consumers that use them.
 
 A logical application endpoint routes only to ready instances. Dependency failure and loss of write authority must affect readiness. Both browser-facing endpoints and internal application calls must survive the same location-loss scenario. A single reverse proxy on one participating site must not be the only traffic entry point.
@@ -117,21 +121,21 @@ Backups remain independent of replication and cover accidental deletion, corrupt
 
 ## SSO first application
 
-Daedalus and Metis are separate repositories and application locations. Icarus is the proposed independent location for backend voting services. Other compositions on those nodes do not become replicas by association.
+Daedalus and Metis are separate repositories and application locations. The selected SSO pattern is one active deployment and one standby, with controlled manual promotion. Icarus is not a member of the target SSO topology. Other compositions on these nodes do not become replicas by association.
 
-The candidate SSO arrangement has two Authelia/nginx deployments and replicated persistent and session state. MariaDB/Galera can use two full data members plus an Icarus arbitrator. Valkey uses a native replication topology with three Sentinel voters. These may be dedicated compositions or reusable backend groups; SSO does not require a global SQL/cache platform for every APEX application.
+The target uses native MariaDB primary/replica replication and a primary/replica session store over the existing Xray tunnels. It excludes Galera, its arbitrator, Sentinel elections, and custom automatic promotion controllers. The standby must not accept application writes until an operator has fenced the previous writer and explicitly promoted the replacement. An unreachable SSH or tunnel endpoint alone does not prove fencing. Native replication status and the last applied position must be checked and any possible data loss recorded before promotion.
 
-The Galera arbitrator provides voting without another SQL data copy. The remaining data member and arbitrator can preserve quorum after one data location fails. WAN latency and stable connectivity must be checked before choosing this topology. [MariaDB deployment variants](https://mariadb.com/docs/galera-cluster/galera-architecture/galera-cluster-deployment-variants/)
+The existing public users source is a manually maintained YAML file. The selected replacement is LLDAP backed by MariaDB, with its enrolled directory preserved during the change from the staged Galera deployment to ordinary replication. Existing account identifiers, profile attributes and application groups must survive migration. The operator has already completed the one-time password enrollment; changing replication must not require another enrollment or overwrite the directory. Keep the old public SSO authoritative until migration checks pass. [Authelia LLDAP integration](https://www.authelia.com/integration/ldap/lldap/)
 
-The existing users source is a manually maintained YAML file. The selected replacement is LLDAP backed by the replicated MariaDB group, with a local LLDAP instance at each SSO location. Authelia supports this directory implementation. Existing account identifiers, profile attributes and application groups must survive migration. LLDAP cannot import the existing YAML password hash; the operator has accepted one password enrollment through a private setup page. Keep the old SSO authoritative until enrollment and migration checks pass. [Authelia LLDAP integration](https://www.authelia.com/integration/ldap/lldap/)
+The two identity locations use `198.18.16.150` on Daedalus and `198.18.18.150` on Metis. The existing SSO frontend addresses remain separate. These are explicit allocations for this deployment, not reserved addresses or mandatory topology for other APEX applications. The staged Icarus voting services must be retired only after verifying that the preserved data and new replication no longer depend on them; unrelated Icarus applications remain outside this change.
 
-The initial placement uses a separate `identity` composition at `198.18.16.150` on Daedalus, `198.18.18.150` on Metis and `198.18.17.150` on Icarus. Daedalus and Metis own SQL data and directory instances; Icarus owns the Galera arbitrator. Each location also owns a Valkey data member and Sentinel voter. The existing SSO frontend addresses remain separate. These are explicit allocations for this deployment, not reserved addresses or mandatory topology for other APEX applications.
+Stage the new backend beside the existing public SSO database. Preserve a verified backup and use a consistent logical export for migration. Ordinary container restart must never initialize a replacement database, replay a seed snapshot, or promote a replica. Test direct Daedalus–Metis Xray connectivity without Icarus as an intermediary. Returning former primaries remain fenced until explicitly resynchronized as replicas.
 
-Stage the new backend beside the existing SSO database. Use an explicit one-time SQL bootstrap, normal native joins thereafter, and a consistent logical export for migration. Ordinary container restart must never repeat cluster bootstrap or replay an old session snapshot. A direct private Daedalus–Metis transport is required so that Icarus loss does not also remove connectivity between the surviving data members. Password enrollment, verified replication transport and local dependency readiness are deployment gates, not consequences of starting containers.
+Asynchronous database and session replication can lose acknowledged recent changes. This pattern does not promise seamless preservation of every account change, session, or revocation. The promotion procedure must document the replication position, possible loss and whether users need to log in again. Session invalidation after promotion must be tested so that stale replicated sessions do not silently restore revoked access.
 
-Valkey replication is asynchronous and can lose acknowledged recent changes. The proposed SSO pattern therefore cannot claim seamless preservation of every session or revocation. Its production profile must state and test the resulting security and user-visible behavior. A requirement for lossless session state requires a different validated arrangement, not a stronger label on Sentinel.
+The earlier automatic recovery target of 60 seconds is superseded by controlled promotion. Loss of the active stateful deployment can interrupt authentication until promotion completes; loss of the standby must not stop the active deployment. Record the measured promotion and recovery time rather than claiming an automatic recovery bound.
 
-The starting recovery target is automatic restoration of authentication within 60 seconds after loss of any one participating location. This is a proposed acceptance target, not an achieved guarantee. Session behavior is assessed separately. All SSO dependencies, including user data, required notifications, internal authentication routes, and public routing, must be included before declaring that target met.
+Cloudflare traffic failover is requested for `sso.ermnvldmr.com`, `jef.ermnvldmr.com`, `jpl.ermnvldmr.com`, `sea.ermnvldmr.com`, and `imh.ermnvldmr.com`, using the no-additional-cost Worker approach. Traffic selection must not promote databases or treat an unpromoted standby as ready. Each hostname needs independently verified origin routing and application readiness. Two ingress paths to a single application backend provide ingress redundancy only; they do not replicate that backend or its data. Preserve existing streaming, WebSocket, authentication, and path-specific routes.
 
 ## Adding the next application