diff --git a/assets/composition-failover/README.md b/assets/composition-failover/README.md index c560336..cce3e28 100644 --- a/assets/composition-failover/README.md +++ b/assets/composition-failover/README.md @@ -1,6 +1,6 @@ # Readiness-gated origin routing on Workers Free -This local ES-module Worker asset selects between two origins for one public HTTPS hostname. It implements the ingress part of the [composition replication RFC](../../docs/rfcs/composition-replication.md). It is not deployed and does not establish production high availability. Activate it only after backend HA and authenticated application readiness are verified. +This local ES-module Worker asset selects between two origins for one public HTTPS hostname. It implements the ingress part of the [composition replication RFC](../../docs/rfcs/composition-replication.md). It is not deployed and does not establish production high availability. Deploy one instance per public hostname, with independently verified origin routes and readiness. Activate only after the declared backend failure behavior and authenticated application readiness are verified. The Worker does not promote databases or activate a standby. The module has no runtime dependencies, paid bindings, KV, R2, storage or scheduled jobs. This directory does not enable billing, enroll an account, upload a Worker or change DNS/routes. @@ -16,6 +16,7 @@ Set these string bindings through the operator's chosen Cloudflare deployment pr | `APEX_ORIGIN_SECONDARY` | Dedicated DNS-only secondary origin hostname in that zone | | `APEX_READY_PATH` | Dedicated absolute readiness path, for example `/.well-known/apex-ready` | | `APEX_READY_TOKEN` | High-entropy secret shared only by this Worker and both readiness adapters | +| `APEX_READY_STATUS` | Optional exact status string: `204` (default) or `200` for a dedicated application health route | Hostnames accept ordinary ASCII DNS labels, without a scheme, port, trailing dot, wildcard or IP literal. Case is normalized. Public and origin hostnames must be distinct and within the declared zone. Readiness paths accept ASCII letters, digits, `.`, `_`, `~`, `-` and `/`; queries, fragments, escapes, repeated slashes and dot traversal segments are rejected. The secret must be nonempty visible ASCII without whitespace. Invalid/missing bindings produce a generic no-store 503 before any fetch. Syntax validation cannot establish actual DNS ownership or prevent an operator from naming a separately delegated zone; deployment must verify both origins are in the actual same Cloudflare zone. @@ -42,9 +43,9 @@ The required non-secret upload metadata fields are: Use this as the multipart API `metadata` portion, with the ES-module file part named `worker.mjs`. Bindings and the readiness secret are configured separately through the operator-owned enrollment process; they are deliberately absent from this example, as are paid bindings. This metadata is not a complete enrollment or route activation command. Verify the resulting effective settings before activation, including after later uploads. [Multipart upload metadata](https://developers.cloudflare.com/workers/configuration/multipart-upload-metadata/) -Every accepted HTTPS request first probes the primary with a GET at `https://APEX_PUBLIC_HOST/APEX_READY_PATH`. The request carries only `X-APEX-Ready-Token`, rather than client cookies or authorization. `cf.resolveOverride` selects the DNS-only origin while the URL remains the public hostname. Only status 204 is ready; redirects are manual and never accepted. Each probe has a two-second deadline. Unread response bodies are canceled, including responses arriving after timeout. Failed primary readiness leads to a secondary probe; neither ready produces a no-store 503. There is no readiness cache or background health state. +Every accepted HTTPS request first probes the primary with a GET at `https://APEX_PUBLIC_HOST/APEX_READY_PATH`. The request carries only `X-APEX-Ready-Token`, rather than client cookies or authorization. `cf.resolveOverride` selects the DNS-only origin while the URL remains the public hostname. Only the configured exact status is ready: 204 by default, or 200 when explicitly selected with `APEX_READY_STATUS`. Redirects are manual and never accepted. Do not use 200 with a generic login page or native liveness endpoint; a protected application health route must verify both active SSO dependencies and its own backend. Each probe has a two-second deadline. Unread response bodies are canceled, including responses arriving after timeout. Failed primary readiness leads to a secondary probe; neither ready produces a no-store 503. There is no readiness cache or background health state. -The dedicated origin adapter must authenticate the token, reject its absence/incorrect value and prove that this application instance can safely serve traffic: required dependencies, compatible configuration, current write authority/quorum, session state and any application-specific prerequisites. Authelia `/api/health`, an nginx 204, a TCP check or a running container alone is insufficient. Protect adapter aliases and normalize paths consistently at the origin. The Worker rejects public requests to the configured readiness path (also with a query or percent-encoded spelling), and removes every client-supplied readiness header before application dispatch. This asset does not implement or enroll the backend adapter. +The dedicated origin adapter must authenticate the token, reject its absence/incorrect value and prove that this application instance can safely serve traffic: required dependencies, compatible configuration, current write authority for the declared topology, session state and any application-specific prerequisites. Authelia `/api/health`, an nginx 204, a TCP check or a running container alone is insufficient. Protect adapter aliases and normalize paths consistently at the origin. The Worker rejects public requests to the configured readiness path (also with a query or percent-encoded spelling), and removes every client-supplied readiness header before application dispatch. An unpromoted standby must remain unready. This asset does not implement or enroll the backend adapter. Preserve existing path-specific routes when attaching the Worker; two ingress paths do not replicate a shared application backend. Cloudflare documents `resolveOverride` as DNS substitution while retaining the URL's Host, subject to both names belonging to the same zone. The asset retains the HTTPS public URL for routing; actual origin TLS SNI/certificate selection and routing must be demonstrated natively before activation. No local Node test proves Cloudflare DNS/TLS behavior. [Request API](https://developers.cloudflare.com/workers/runtime-apis/request/) @@ -62,7 +63,7 @@ Other hostnames, non-default ports and non-HTTPS requests receive no-store 421. ## Free-tier deployment gates -Workers Free currently provides **100,000 incoming requests per day per account**, reset at midnight UTC, and **10 ms CPU per HTTP request**. Quota is shared with the account's other Workers. This asset performs two fetch subrequests normally (one probe plus application dispatch), or three when primary readiness fails. Waiting for network I/O does not count toward CPU time, but actual Worker CPU consumption must be measured; Node wall time is not that measurement. [Cloudflare Workers limits](https://developers.cloudflare.com/workers/platform/limits/) +Workers Free currently provides **100,000 incoming requests per day per account**, reset at 03:00 MSK (midnight UTC), and **10 ms CPU per HTTP request**. Quota is shared with the account's other Workers. This asset performs two fetch subrequests normally (one probe plus application dispatch), or three when primary readiness fails. Waiting for network I/O does not count toward CPU time, but actual Worker CPU consumption must be measured; Node wall time is not that measurement. [Cloudflare Workers limits](https://developers.cloudflare.com/workers/platform/limits/) Keep the account on Workers Free. Do not enable a paid plan, paid load balancer or paid bindings for this asset. An operator must inspect existing account usage and peak traffic to confirm the free quota is sufficient. This is a no-added-cost deployment candidate, not unlimited availability. @@ -70,11 +71,11 @@ Set the Worker Route to **fail closed**. On Free quota exhaustion, Cloudflare do Operator-owned enrollment gates: -1. Validate backend HA, adapter authentication/readiness, dependency loss and write-authority withdrawal independently of this Worker. +1. Validate the declared backend failure behavior, adapter authentication/readiness, dependency loss and write-authority withdrawal independently of this Worker. 2. Confirm account Free plan/quota, same-zone DNS-only origin records, public proxied Route, strict TLS, native SNI/Host at each origin, required private-origin/cache compatibility flags and date, and no Worker/route recursion or pre-Worker caching. 3. Test native primary/secondary readiness, both-unready failure, redirects, cookies, cache bypass, POST and GET callback non-replay, streaming and any required WebSockets. Measure Worker CPU and account request headroom. 4. Verify fail-closed quota handling and exercise location loss, network partitions, stale/rejoining origins and recovery. Record observed recovery time and session behavior before making an HA claim. -5. Activate only after those gates pass with the intended backend HA. Maintain a documented operator rollback that does not silently route to an unsafe origin. +5. Activate only after those gates pass with the intended backend failure behavior. Maintain a documented operator rollback that does not silently route to an unsafe origin. ## Local verification diff --git a/assets/composition-failover/worker.mjs b/assets/composition-failover/worker.mjs index fa8a6ab..fa7b925 100644 --- a/assets/composition-failover/worker.mjs +++ b/assets/composition-failover/worker.mjs @@ -23,7 +23,9 @@ function configuration(env) { || path.includes('//') || path.split('/').some((part) => part === '.' || part === '..')) return null; const token = env.APEX_READY_TOKEN; if (typeof token !== 'string' || !/^[\x21-\x7e]+$/.test(token)) return null; - return { publicHost, primary, secondary, path, token }; + const status = env.APEX_READY_STATUS === undefined ? '204' : env.APEX_READY_STATUS; + if (!['200', '204'].includes(status)) return null; + return { publicHost, primary, secondary, path, token, status: Number(status) }; } function unavailable(status = 503, message = 'Service unavailable') { @@ -60,7 +62,7 @@ async function ready(config, origin) { }); const inspection = fetch(probe, { ...options(origin), signal: controller.signal }) .then((response) => { - const accepted = !controller.signal.aborted && response.status === 204; + const accepted = !controller.signal.aborted && response.status === config.status; discard(response); return accepted; }, () => false); diff --git a/assets/composition-failover/worker.test.mjs b/assets/composition-failover/worker.test.mjs index 7665c6d..14f7226 100644 --- a/assets/composition-failover/worker.test.mjs +++ b/assets/composition-failover/worker.test.mjs @@ -114,6 +114,23 @@ test('readiness redirect is never accepted or followed', async (t) => { assert.equal(calls[0].options.redirect, 'manual'); }); +test('application health can explicitly require 200 while 204 remains the default', async (t) => { + const bindings = { ...env, APEX_READY_STATUS: '200' }; + const calls = intercept(t, (_, __, count) => count === 1 + ? new Response(null, { status: 204 }) + : count === 2 ? new Response('Healthy', { status: 200 }) : new Response('secondary')); + const response = await worker.fetch(new Request(url), bindings); + assert.equal(await response.text(), 'secondary'); + assert.equal(calls.length, 3); + assert.equal(calls[2].options.cf.resolveOverride, env.APEX_ORIGIN_SECONDARY); +}); + +test('an unconfigured 200 health response cannot release application traffic', async (t) => { + const calls = intercept(t, () => new Response('login page', { status: 200 })); + assert.equal((await worker.fetch(new Request(url), env)).status, 503); + assert.equal(calls.length, 2); +}); + for (const method of ['GET', 'POST']) { test(`failed ${method} dispatch is never replayed on another origin`, async (t) => { const calls = intercept(t, (_, __, count) => { @@ -174,6 +191,9 @@ test('missing and invalid bindings fail closed without diagnostics or probes', a ['APEX_READY_PATH', '//evil.example/'], ['APEX_READY_PATH', '/ready?query=yes'], ['APEX_READY_PATH', '/ready#fragment'], ['APEX_READY_PATH', '/a/../ready'], ['APEX_READY_PATH', '/%72eady'], ['APEX_READY_TOKEN', 'secret\r\ninjected: yes'], + ['APEX_READY_STATUS', '302'], ['APEX_READY_STATUS', '401'], + ['APEX_READY_STATUS', '503'], ['APEX_READY_STATUS', '2xx'], + ['APEX_READY_STATUS', ''], ['APEX_READY_STATUS', 200], ['APEX_READY_STATUS', null], ]) invalid.push({ ...env, [name]: value }); invalid.push(null); for (const bindings of invalid) { diff --git a/assets/composition-readiness/README.md b/assets/composition-readiness/README.md index ddbd3f8..7de64d5 100644 --- a/assets/composition-readiness/README.md +++ b/assets/composition-readiness/README.md @@ -1,6 +1,6 @@ # Authenticated composition readiness adapter -This dependency-free Python adapter exposes one local HTTP endpoint for the readiness-gated Worker and origin proxy. It runs an operator-supplied executable for every authenticated probe; exit0 means ready, every other outcome means unavailable. The executable must establish the application's actual readiness contract. A successful TCP connection or container health flag is not enough for replicated SSO. +This dependency-free Python adapter exposes one local HTTP endpoint for the readiness-gated Worker and origin proxy. By default it runs an operator-supplied executable for every authenticated probe; exit0 means ready, every other outcome means unavailable. The executable must establish the application's actual readiness contract. A successful TCP connection or container health flag is not enough for replicated SSO. Example private configuration: @@ -12,7 +12,8 @@ Example private configuration: "token_file": "/private/readiness-token", "command": ["/usr/local/bin/python3", "/assets/probe.py", "--config", "/private/probe.json"], "timeout": 1.5, - "slots": 4 + "slots": 4, + "single_flight": false } ``` @@ -22,6 +23,8 @@ Only GET at the exact canonical path, with one matching `X-APEX-Ready-Token` hea Probe timeout is bounded to1.5seconds so an origin can answer within the Worker's two-second deadline. A timeout kills the probe's process group. Probe slots and request threads are bounded; slow socket operations time out after two seconds. Operators must account for proxy overhead and actual probe execution time. Probe programs must not detach into a new session or launch background jobs. -The SSO probe must verify the local Authelia instance/config generation, local authenticated SQL and directory access, completed migration, SQL Primary/Synced/write authority, and fresh session-controller authority plus actual verified TLS application access to its selected primary. It must reject uncertain/stale state and avoid inventing a competing election. A short-lived controller status file alone is not a consensus lease. The generic adapter supplies none of those application checks; do not deploy a placeholder-success probe or claim these tests establish HA. +Optional `single_flight: true` lets overlapping authenticated requests share one running fixed-command probe. It accepts only a JSON boolean and defaults to false. Waiters use that probe's remaining monotonic deadline; they never queue another probe for the same request. The existing connection limit (`slots * 2 + 4`) also bounds waiting requests. Timeout or shutdown returns unavailable and terminates the probe group. Completed results are not cached: each subsequent request starts a fresh probe, with no success TTL or failure cache. Coalescing reduces concurrent work but does not make a slow individual probe meet its deadline. + +For the controlled active/standby SSO pattern, the probe must reject a node not explicitly configured as active, verify completed migration and unchanged Authelia configuration, and check local Authelia health, authenticated SQL access/schema with read_only disabled, directory access, and an authenticated session write/read/delete over verified TLS. All dependencies belong to the selected local identity instance. These checks do not establish exclusive primary ownership: the operator must fence the previous writer before promotion, and the standby application and ingress stay disabled. No quorum, Sentinel, or session-controller authority is required by this pattern. The generic adapter supplies none of those application checks; do not deploy a placeholder-success probe or claim these tests establish HA. Run `python3 test_adapter.py` for loopback HTTP and subprocess behavior checks. No third-party packages are required. Production native image/probe/proxy integration and location-loss tests remain separate acceptance gates. diff --git a/assets/composition-readiness/adapter.py b/assets/composition-readiness/adapter.py index 830b7c0..5c22034 100644 --- a/assets/composition-readiness/adapter.py +++ b/assets/composition-readiness/adapter.py @@ -9,6 +9,7 @@ import re import signal import subprocess import threading +import time class ReadinessServer(http.server.ThreadingHTTPServer): @@ -16,7 +17,7 @@ class ReadinessServer(http.server.ThreadingHTTPServer): block_on_close = False allow_reuse_address = True - def __init__(self, address, *, token, path, command, timeout=1.5, slots=4): + def __init__(self, address, *, token, path, command, timeout=1.5, slots=4, single_flight=False): if not isinstance(token, str) or not re.fullmatch(r'[!-~]{16,512}', token): raise ValueError('A private visible-ASCII token is required') if not isinstance(path, str) or not re.fullmatch(r'/[A-Za-z0-9._~/-]+', path) or any(part in ('', '.', '..') for part in path[1:].split('/')): @@ -25,10 +26,14 @@ class ReadinessServer(http.server.ThreadingHTTPServer): raise ValueError('A fixed absolute executable and argv are required') if isinstance(timeout, bool) or not 0 < timeout <= 1.5 or isinstance(slots, bool) or not isinstance(slots, int) or not 1 <= slots <= 16: raise ValueError('Invalid probe bounds') + if not isinstance(single_flight, bool): + raise ValueError('single_flight must be a boolean') self.token = token.encode('ascii') self.ready_path = path self.command = command self.timeout = timeout + self.single_flight = single_flight + self.flight = None self.probe_slots = threading.BoundedSemaphore(slots) self.connection_slots = threading.BoundedSemaphore(slots * 2 + 4) self.active = set() @@ -74,6 +79,31 @@ class ReadinessServer(http.server.ThreadingHTTPServer): process.wait() def probe(self): + if not self.single_flight: + return self.run_probe() + with self.active_lock: + if self.closing: + return False + flight = self.flight + leader = flight is None + if leader: + flight = {'deadline': time.monotonic() + self.timeout, 'done': threading.Event(), 'ready': False} + self.flight = flight + if not leader: + flight['done'].wait(max(0, flight['deadline'] - time.monotonic())) + with self.active_lock: + return flight['ready'] and not self.closing and time.monotonic() < flight['deadline'] + ready = False + try: + ready = self.run_probe(flight['deadline']) + finally: + with self.active_lock: + flight['ready'] = ready and not self.closing and time.monotonic() < flight['deadline'] + self.flight = None + flight['done'].set() + return flight['ready'] + + def run_probe(self, deadline=None): if not self.probe_slots.acquire(blocking=False): return False process = None @@ -84,7 +114,8 @@ class ReadinessServer(http.server.ThreadingHTTPServer): process = subprocess.Popen(self.command, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, start_new_session=True, close_fds=True) self.active.add(process) try: - return process.wait(timeout=self.timeout) == 0 + timeout = self.timeout if deadline is None else max(0, deadline - time.monotonic()) + return process.wait(timeout=timeout) == 0 except subprocess.TimeoutExpired: self.stop_probe(process) return False @@ -99,6 +130,9 @@ class ReadinessServer(http.server.ThreadingHTTPServer): def server_close(self): with self.active_lock: self.closing = True + if self.flight is not None: + self.flight['ready'] = False + self.flight['done'].set() active = list(self.active) for process in active: self.stop_probe(process) diff --git a/assets/composition-readiness/test_adapter.py b/assets/composition-readiness/test_adapter.py index adb86a5..b8ad498 100644 --- a/assets/composition-readiness/test_adapter.py +++ b/assets/composition-readiness/test_adapter.py @@ -6,9 +6,9 @@ class AdapterTests(unittest.TestCase): def tearDown(self): for s,t in self.servers:s.shutdown();s.server_close();t.join() self.tmp.cleanup() - def server(self,code=None,timeout=.3,slots=2): + def server(self,code=None,timeout=.3,slots=2,**options): code=code or f"from pathlib import Path;Path({str(self.marker)!r}).write_text('yes')" - server=ReadinessServer(('127.0.0.1',0),token='private-test-token',path='/.well-known/apex-ready',command=[sys.executable,'-c',code],timeout=timeout,slots=slots) + server=ReadinessServer(('127.0.0.1',0),token='private-test-token',path='/.well-known/apex-ready',command=[sys.executable,'-c',code],timeout=timeout,slots=slots,**options) t=threading.Thread(target=server.serve_forever,kwargs={'poll_interval':.01},daemon=True);t.start();self.servers.append((server,t));return server def request(self,s,path='/.well-known/apex-ready',token='private-test-token',method='GET',extra_token=False): c=http.client.HTTPConnection(*s.server_address,timeout=3);c.putrequest(method,path) @@ -50,4 +50,70 @@ class AdapterTests(unittest.TestCase): self.assertFalse(self.marker.exists()) def test_no_cached_readiness(self): s=self.server();self.assertEqual(self.request(s)[0],204);s.command=[sys.executable,'-c','raise SystemExit(1)'];self.assertEqual(self.request(s)[0],503) + def overlapping_requests(self,s,count=4): + barrier=threading.Barrier(count) + def one(): + barrier.wait(timeout=2);start=time.monotonic();result=self.request(s);return result,time.monotonic()-start + with concurrent.futures.ThreadPoolExecutor(max_workers=count) as pool:return list(pool.map(lambda _:one(),range(count))) + def test_single_flight_shares_success_and_sequential_request_reprobes(self): + code=f"import pathlib,time;p=pathlib.Path({str(self.marker)!r});f=p.open('a');f.write('probe\\n');f.close();time.sleep(.2)" + s=self.server(code,timeout=1,slots=1,single_flight=True) + results=self.overlapping_requests(s) + self.assertEqual([r for r,_ in results],[(204,'no-store',b'')]*4) + self.assertEqual(self.marker.read_text().splitlines(),['probe']) + self.assertEqual(self.request(s)[0],204) + self.assertEqual(self.marker.read_text().splitlines(),['probe','probe']) + def test_single_flight_shares_failure_without_caching_it(self): + code=f"import pathlib,time;p=pathlib.Path({str(self.marker)!r});f=p.open('a');f.write('probe\\n');f.close();time.sleep(.2);raise SystemExit(1)" + s=self.server(code,timeout=1,slots=4,single_flight=True) + self.assertEqual([r for r,_ in self.overlapping_requests(s)],[(503,'no-store',b'')]*4) + self.assertEqual(self.marker.read_text().splitlines(),['probe']) + s.command=[sys.executable,'-c','pass'];self.assertEqual(self.request(s)[0],204) + def test_single_flight_timeout_wakes_waiters_and_kills_descendant(self): + code=f"import subprocess,time,sys;subprocess.Popen([sys.executable,'-c',\"import time,pathlib;time.sleep(.8);pathlib.Path({str(self.marker)!r}).write_text('leaked')\"]);time.sleep(3)" + s=self.server(code,timeout=.25,slots=1,single_flight=True) + results=self.overlapping_requests(s) + self.assertEqual([r for r,_ in results],[(503,'no-store',b'')]*4) + self.assertTrue(all(.15