diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..f3a48f8 --- /dev/null +++ b/Makefile @@ -0,0 +1,21 @@ +.PHONY: test lint fmt check + +test: + python3 -m unittest discover -s tests + +lint: + @if command -v ruff >/dev/null 2>&1; then \ + ruff check .; \ + else \ + python3 -m py_compile engine/**/*.py actions/**/*.py tests/*.py; \ + echo "ruff not installed in PATH; verified python bytecode compilation."; \ + fi + +fmt: + @if command -v ruff >/dev/null 2>&1; then \ + ruff format .; \ + else \ + echo "ruff not installed in PATH; skipping fmt."; \ + fi + +check: lint test diff --git a/README.md b/README.md index 16228ca..bd565cf 100644 --- a/README.md +++ b/README.md @@ -2,140 +2,101 @@ apex.ermnvldmr.com logo

-# apex.ermnvldmr.com commons +# [apex.ermnvldmr.com](https://apex.ermnvldmr.com) -apex helps you run a bunch of self-hosted apps across a few machines — a VPS here, a home server -there — and keep them consistent, reproducible, and under version control. Instead of hand-tuning -every box and copy-pasting compose files around, you describe each machine once in its own git -repository, share a common set of conventions and building blocks between them, and manage the -whole thing with one small command: `apex`. +The boring, standard-library fleet operations framework powering self-hosted infrastructure across independent nodes. -It's for the person who self-hosts a handful of mostly-unrelated things — a media server, a -password vault, a mail server, some *arr apps — and wants the fleet to feel like one tidy, -git-tracked system instead of a pile of snowflakes. +## Quick Start -## Why - -Self-hosting across a few boxes tends to sprawl. Every host slowly becomes a one-off, -configuration drifts away from whatever you last wrote down, and "how did I set this up again?" -turns into a recurring question. apex keeps the shared parts in one versioned place and every host -as a plain, reviewable git repo — so your setup is something you can read, diff, and rebuild, not -something you hope you still remember. - -A few things that keep it pleasant to live with: - -- **No control plane.** Nothing central to run or babysit — every command acts on the machine you - run it on. The "fleet" is just a set of repositories that share the same building blocks. -- **No lock-in, no build step.** Just `git` and Python's standard library — no pip, Node, Go, or - code generation. The compose files are hand-written and stay readable. -- **Assumes nothing about your setup.** A machine says who it is in a small `node.env` file; apex - makes no assumptions about your domain, hosting provider, or topology. - -## How it works - -- **Each host is a git repository** that pins apex as a shared `commons/` submodule, then adds its - own `compositions/` (the services that host runs) and `proprietaries/` (anything special to that - host). -- **One command, grouped actions.** You drive everything with `apex /` — the path - *is* the name (`configure/ufw`, `compose up`, `sync/repository`, …). A host can override or - extend any shared action with its own, through a small overlay. -- **A shared compose core** gives every host the same foundation — an edge reverse proxy (Traefik) - and a Docker-socket proxy — plus opt-in pieces you switch on per host: AdGuard, an Xray edge, a - Grafana Alloy metrics/logs agent, and label-driven `restic` backups. -- **Your setup captures itself.** A scheduled "capture-up" records each host's live state back into - its repo on a `sync/` branch, so drift shows up as a diff you can review. - -Under the hood it's deliberately boring: a `python3`-stdlib engine that resolves the overlay, works -out the host's identity, builds a context of helpers, and hands it to your action. - -📖 **[Read the docs](https://docs.ermnvldmr.com/en/apex/)** for the full picture — concepts, guides, -and reference — also available in [Russian](https://docs.ermnvldmr.com/ru/apex/). - -## Quick start - -Every host is its own git repository that pins apex. Bringing up a host called -`selfhost.example.com` looks like this: +- **Live Documentation**: (also available in [Russian](https://docs.ermnvldmr.com/ru/apex/)) +- **Getting Started Guide**: +- **Report an Issue**: +- **License**: [BSD-2-Clause](#copyright--license) ```sh -# Get the host's repo (it pins apex as commons/) and put `apex` on your PATH +# Clone your node's repository (which pins apex as commons/) git clone --recurse-submodules git@github.com:you/selfhost.example.com.git cd selfhost.example.com && ./init.sh && source ~/.bashrc -# Tell apex who this host is +# Configure node identity cat > node.env <<'EOF' APEX_NODE_FQDN=selfhost.example.com APEX_NODE_HOST=selfhost.example.com APEX_SUBNET=198.18.16.0/24 EOF -apex configure # set the host up: firewall, crowdsec, cron, systemd, routing -apex compose up # bring its services online +# Configure host baseline and bring up compositions +apex configure # firewall, crowdsec, cron, systemd, routing +apex compose up # bring node services online ``` -A couple of steps sit in between on a real host — linking storage, provisioning users, filling in -secrets. The **[Getting started guide](https://docs.ermnvldmr.com/en/apex/getting-started/)** walks -through the whole thing. +## Essential Information -## Requirements +You should understand the following invariants before exploring the codebase: -- `git` and `python3` — standard library only, nothing to install. -- A Debian-based host with Docker and `docker compose` v2. +### Core Principles -## Initialize a provider-installed server +- **No Control Plane**: No central orchestrator, agent daemon, or cluster controller. Commands act locally on the node you run them on. +- **Zero Build Step & Stdlib-Only**: Pure Python 3 standard library and Git. No `pip` dependencies, Node, Go, or code-generation steps. +- **Node-Owned Repositories**: Each machine is an independent Git repository pinning `apex` as a `commons/` submodule. It declares its own identity, compositions, and local overrides. +- **Overlay Architecture**: Node repositories can extend or override any shared action by placing an executable in `proprietaries/actions/`. -The SSH initializer targets clean Debian 13 amd64 machines and uses the same -provisioning modules as Packer. Copy [the user-data template](ci/cloud-init/user-data.template.yaml), -fill in the target identity, operator public key, and independent node repository, then run: +### Repository Structure -```sh -./apex init root@server --user-data /private/path/location.user-data.yaml -``` - -The workstation needs Python and OpenSSH. Target host trust follows your SSH -configuration; private Git repositories also need a deploy key and trusted Git -host keys in the supported user-data fields. User-data contains credentials and -belongs outside tracked files. This is a strict APEX cloud-config subset, not a -general cloud-init interpreter. - -Initialization prepares packages, the administrator, storage tiers, and a pinned -repository checkout. It preserves provider networking and machine identity. A -persistent watchdog protects the switch to key-only `adam` access on port 2222 -and activation of UFW, its Docker integration, CrowdSec, and the firewall bouncer. -Fresh firewall defaults match the image baseline; existing custom policies require -compatible configuration. Unsupported conditional SSH policies stop enrollment -before baseline setup. Recovery preserves concurrent administrator edits and -reports `recovery_pending` when cleanup needs reconciliation, keeping the watchdog -armed while attempting to restore SSH independently. -After that switch, retries use `adam@server --port 2222`; the command reports the -safe resume invocation and root-owned stage records. Existing data and divergent -checkouts are preserved and reported as conflicts. Initialization does not deploy -applications or create database clusters. - -Supported node repositories declare `APEX_ENROLLMENT_INTERFACE=1` in `node.env` -alongside their compatible `init.sh`. Missing or unsupported declarations stop -enrollment before the checkout hook runs. Results record the provisioner and -baseline revisions separately, plus the node and commons commits. - -The running Debian kernel must have matching headers available. If the provider -installed an obsolete kernel, install a supported kernel and headers and reboot -before enrollment; the initializer does not reboot the server automatically. - -Isolated Debian 13 VM checks cover the shared package baseline, credential repair, -SSH and security activation, completed-run retries, watchdog and reboot recovery, -and preservation of administrator edits during interrupted recovery. Earlier checks -also cover safe storage handling and two independent Packer/cloud-init clones. -Production enrollment requires a published node repository with a compatible, -available commons pin. -The [composition replication RFC](docs/rfcs/composition-replication.md) defines how -independent node repositories can host replicas of selected compositions. - -## License - -BSD-2-Clause — see [`LICENSE`](LICENSE). +- **[`actions/`](./actions)**: Built-in engine actions (`compose.py`, `configure/`, `sync/`, `tiers/`, `backup/`, `utils/`). +- **[`compositions/`](./compositions)**: The shared core Compose stack (`traefik`, `socket-proxy`, `alloy`, `adguard`, `xray`, `restic`). +- **[`engine/`](./engine)**: Framework engine — action discovery, overlay resolution, context injection, and logging. +- **[`etc/`](./etc)**: Static service configurations, daemon templates, and helper scripts (Alloy configs, Happ generator). +- **[`ci/`](./ci)**: Host image build recipes (Packer for Debian 13) and cloud-init bootstrap templates. +- **[`assets/`](./assets)**: Repository branding assets (`logo.png`). --- -Portions of this project's source code may have been developed with the assistance of AI -code-generation tools. Contributions made with the help of such tools are welcome, provided the -code quality stays within an acceptable range and the contributor fully understands the submission -they are making. +## Who Are You? + +### Node Operator + +Running and maintaining services on an existing host. + +- **Requirements**: Debian-based host with Docker, Docker Compose v2, Git, and Python 3. +- **Workflow**: + - Update services: `apex compose pull && apex compose up` + - Run backups: `apex backup/run` + - Sync repository state: `apex sync/repository` +- **Replication**: For multi-node high availability patterns and contracts, consult the [Composition Replication Documentation](https://docs.ermnvldmr.com/en/apex/concepts/replication/). + +### Infrastructure Administrator + +Bootstrapping clean Debian 13 hosts on bare metal or cloud providers without cloud-init support. + +- **Initialization Tool**: A standalone workstation bootstrap tool targets clean Debian 13 amd64 installations using the same baseline as Packer. +- **Template**: Copy and fill out [`ci/cloud-init/user-data.template.yaml`](./ci/cloud-init/user-data.template.yaml) with target identity, operator keys, and node repository. +- **Run Enrollment**: + ```sh + python3 ci/scripts/init-node.py root@server --user-data /private/path/location.user-data.yaml + ``` +- **Security Handover**: Safely switches SSH to key-only authentication on port 2222 (`adam@server`) protected by an automated rollback watchdog, activates UFW firewall rules, Docker iptables integration, and CrowdSec. +- **Resuming**: Post-enrollment commands connect via `adam@server --port 2222`. + +### Developer & Contributor + +Adding shared actions or improving the engine. + +- **Conventions**: Maintain strict Python standard library compliance. No external runtime packages. +- **Testing**: Run test suite via `python3 -m unittest discover -s tests`. +- **Linting**: Keep code formatted and clean according to PEP 8 standards. + +--- + +## Communication & Support + +- **Issues**: +- **Email**: + +## Copyright & License + +Licensed under the [BSD-2-Clause License](./LICENSE). + +--- + +**AI Assistance**: Portions of this project's source code and configuration may have been developed with the assistance of AI code-generation tools. diff --git a/actions/backup/run.py b/actions/backup/run.py index 85b2119..621b090 100644 --- a/actions/backup/run.py +++ b/actions/backup/run.py @@ -63,7 +63,8 @@ def run(ctx, args): "backup produced no new snapshot (no restic.* labels discovered?)") raise SystemExit(1) log.info("Applying retention (forget --prune)...") - if restic("forget", "--prune", "--keep-daily", "7", "--keep-weekly", "4", + if restic("forget", "--prune", "--host", host, "--tag", tag, + "--keep-daily", "7", "--keep-weekly", "4", "--keep-monthly", "12", check=False).returncode != 0: ctx.notify.error(TITLE, url, "restic forget/prune failed"); raise SystemExit(1) log.success("Restic backup completed.") diff --git a/actions/compose.py b/actions/compose.py index de716f5..1895e4b 100644 --- a/actions/compose.py +++ b/actions/compose.py @@ -1,32 +1,85 @@ # actions/compose.py -"""Port of compose — orchestrate docker compose across the node. apex core first -(owns networks), then services alpha; reverse on down. Exports derived APEX_* vars and -layers --env-file (.env secrets + apex.env committed scalars).""" +"""Orchestrate docker compose across the node. + +Supports targeting all compositions in sequence (default: apex core first, services next; +reverse on down) or targeting a specific composition (e.g. `apex compose restart apex` +or `apex compose up -d sso`). Exports derived APEX_* vars and layers --env-file (.env secrets + apex.env). +""" import os -from engine.descriptor import Meta, Arg, Flag, Rest +from engine.descriptor import Arg, Flag, Meta, Opt, Rest + +METADATA = Meta( + summary="Orchestrate docker compose across the node or on a specific composition.", + args=[ + Opt("-p", "Target composition name or 'all'.", default=None), + Flag("--dry-run", "Print commands only."), + Arg("action", "up | down | restart | ps | logs | ..."), + Rest("extra", "Extra args (target composition name, service names, or docker compose flags)."), + ], +) + + +def _clean_xray_sockets(ctx): + """Unlink stale host-side Xray sockets in /dev/shm to prevent REALITY panic crash loops.""" + shm = os.path.join(ctx.paths.stier3, "apex", "xray", "shm") + if os.path.isdir(shm): + for f in os.listdir(shm): + if f.endswith(".sock"): + try: + os.unlink(os.path.join(shm, f)) + ctx.log.info(f"Cleaned stale Xray socket: {f}") + except OSError as e: + ctx.log.warn(f"Failed to unlink {f}: {e}") -METADATA = Meta(summary="Orchestrate docker compose across the node (apex core first, services next).", - args=[Arg("action", "up | down | restart | ..."), - Flag("--dry-run", "Print commands only."), - Rest("extra", "Extra args passed through to docker compose.")]) def run(ctx, args): log, comp, action = ctx.log, ctx.paths.compositions, args.action - # argparse.REMAINDER swallows flags placed after the action, so strip --dry-run - # out of the passthrough args like bash did ("apex compose up --dry-run" must - # preview, not hand docker compose its own --dry-run). extra = [a for a in (args.extra or []) if a != "--dry-run"] dry_run = args.dry_run or len(extra) != len(args.extra or []) env = dict(os.environ) env.update({k: v for k, v in ctx.vars().items() if k.startswith("APEX_")}) + available = [] + if os.path.isdir(comp): + available = [ + d for d in os.listdir(comp) + if os.path.isdir(os.path.join(comp, d)) + and not d.startswith((".", "@")) + and os.path.isfile(os.path.join(comp, d, "docker-compose.yml")) + ] + + target = getattr(args, "p", None) + if not target and extra: + first = extra[0] + if first == "all": + target = "all" + extra = extra[1:] + elif first in available: + target = first + extra = extra[1:] + elif not first.startswith("-"): + target = first + extra = extra[1:] + elif extra[-1] in available and len(extra) > 1: + target = extra[-1] + extra = extra[:-1] + + if not target: + target = "all" + + if target != "all" and target not in available: + log.error(f"Unknown composition '{target}'. Available: {', '.join(sorted(available)) if available else 'none'}") + raise SystemExit(1) + + if action in ("up", "restart", "start") and target in ("all", "apex") and not dry_run: + _clean_xray_sockets(ctx) + def sequence(): + if target != "all": + return [os.path.join(comp, target)] core = os.path.join(comp, "apex") - others = sorted( - os.path.join(comp, d) for d in os.listdir(comp) - if os.path.isdir(os.path.join(comp, d)) and not d.startswith((".", "@")) and d != "apex" - and os.path.isfile(os.path.join(comp, d, "docker-compose.yml"))) - seq = ([core] if os.path.isfile(os.path.join(core, "docker-compose.yml")) else []) + others + others = sorted(os.path.join(comp, d) for d in available if d != "apex") + seq = ([core] if "apex" in available else []) + others return list(reversed(seq)) if action == "down" else seq def run_one(dirpath): @@ -35,7 +88,12 @@ def run(ctx, args): for ef in (".env", "apex.env"): if os.path.isfile(os.path.join(dirpath, ef)): base += ["--env-file", ef] - cmd = base + (["up", "-d"] if action == "up" else [action]) + extra + if action == "up": + up_flags = [] if "-d" in extra else ["-d"] + cmd = base + ["up"] + up_flags + extra + else: + cmd = base + [action] + extra + if dry_run: log.info(f"[DRY-RUN] (cd {dirpath} && {' '.join(cmd)})") else: @@ -44,4 +102,4 @@ def run(ctx, args): for d in sequence(): run_one(d) - log.success(f"Orchestrated {action}.") + log.success(f"Orchestrated {action} ({target}).") diff --git a/actions/configure/base.py b/actions/configure/base.py index 67fb3c0..e3cebf7 100644 --- a/actions/configure/base.py +++ b/actions/configure/base.py @@ -1,13 +1,89 @@ -"""Apply the bounded shared base package, journal and UFW helper policy.""" -from pathlib import Path - +# actions/configure/base.py +"""Port of configure/base — base packages + ufw + ufw-docker helper.""" +import os +import shutil +import tempfile +import urllib.request from engine.descriptor import Meta METADATA = Meta(summary="Configure base system packages (curl, wget, git, rsyslog, ufw, ufw-docker).") +UFW_DOCKER_URL = "https://github.com/chaifeng/ufw-docker/raw/master/ufw-docker" def run(ctx, args): - ctx.log.info("Starting base system initialization...") - commons = Path(__file__).resolve().parents[2] - ctx.sys.sudo(["python3", "-m", "engine.provisioning.baseline", "--base-policy"], cwd=str(commons)) - ctx.log.success("Base system initialization completed.") + log, s = ctx.log, ctx.sys + log.info("Starting base system initialization...") + if not os.path.isfile("/etc/debian_version"): + log.error("Requires a Debian-based system (Debian/Ubuntu).") + raise SystemExit(1) + needed = [p for p in ["curl", "wget", "git", "rsyslog"] if not shutil.which(p)] + if needed: + log.info("Updating package lists...") + s.sudo(["apt-get", "update", "-y"]) + log.info(f"Installing missing essential packages: {', '.join(needed)}...") + s.sudo(["apt-get", "install", "-y", "-o", "DPkg::Lock::Timeout=60", *needed]) + else: + log.info("Essential packages already installed.") + s.ensure_running("rsyslog") + + # Configure APT to discard downloaded archives + apt_clean_path = "/etc/apt/apt.conf.d/99-apex" + apt_clean_content = 'APT::Keep-Downloaded-Packages "0";\n' + if os.path.exists("/etc/apt/apt.conf.d/01clean"): + s.sudo(["rm", "-f", "/etc/apt/apt.conf.d/01clean"]) + current_apt = None + if os.path.exists(apt_clean_path): + try: + with open(apt_clean_path, encoding="utf-8") as f: + current_apt = f.read() + except OSError: + current_apt = None + if current_apt != apt_clean_content: + log.info("Configuring APT to discard downloaded archives...") + s.sudo(["tee", apt_clean_path], input=apt_clean_content) + + # Limit systemd-journald max disk usage + journal_d = "/etc/systemd/journald.conf.d" + journal_conf = os.path.join(journal_d, "99-apex.conf") + journal_content = "[Journal]\nSystemMaxUse=200M\n" + if os.path.exists(os.path.join(journal_d, "maxuse.conf")): + s.sudo(["rm", "-f", os.path.join(journal_d, "maxuse.conf")]) + current_journal = None + if os.path.exists(journal_conf): + try: + with open(journal_conf, encoding="utf-8") as f: + current_journal = f.read() + except OSError: + current_journal = None + if current_journal != journal_content: + log.info("Configuring systemd-journald SystemMaxUse=200M...") + s.sudo(["mkdir", "-p", journal_d]) + s.sudo(["tee", journal_conf], input=journal_content) + s.sudo(["systemctl", "restart", "systemd-journald"]) + + if not shutil.which("ufw"): + log.info("Installing UFW...") + s.sudo(["apt-get", "install", "-y", "-o", "DPkg::Lock::Timeout=60", "ufw"]) + else: + log.info("UFW already installed.") + + target_bin = "/usr/local/bin/ufw-docker" + if not (os.path.isfile(target_bin) and os.access(target_bin, os.X_OK)): + log.info("Downloading ufw-docker helper to /usr/local/bin/ufw-docker...") + with tempfile.NamedTemporaryFile(delete=False) as tf: + tmp_name = tf.name + try: + urllib.request.urlretrieve(UFW_DOCKER_URL, tmp_name) + s.sudo(["cp", tmp_name, target_bin]) + s.sudo(["chown", "root:root", target_bin]) + s.sudo(["chmod", "0755", target_bin]) + log.success(f"ufw-docker installed to {target_bin}.") + finally: + if os.path.exists(tmp_name): + os.unlink(tmp_name) + else: + log.info(f"ufw-docker already exists at {target_bin}.") + + log.info("Ensuring ufw-docker routing rules are installed in UFW...") + s.sudo([target_bin, "install"]) + log.success("Base system initialization completed.") diff --git a/actions/env.py b/actions/env.py new file mode 100644 index 0000000..b662614 --- /dev/null +++ b/actions/env.py @@ -0,0 +1,54 @@ +# actions/env.py +"""Export or synchronize synthesized APEX_* environment variables. + +Outputs shell export statements (for `eval $(apex env)`) or writes a .env file +for direct compatibility with bare `docker compose` commands. +""" +import os +import shlex +from engine.descriptor import Flag, Meta, Opt + +METADATA = Meta( + summary="Export or sync synthesized APEX_* variables for shell or docker compose.", + args=[ + Flag("--export", "Format as shell export statements (default)."), + Flag("--dotenv", "Format as raw KEY=VAL lines without 'export'."), + Opt("--write", "Write variables to the specified file path.", default=None), + Flag("--sync", "Sync variables into compositions/apex/.env for bare docker compose."), + ], +) + + +def run(ctx, args): + log = ctx.log + vars_dict = {k: v for k, v in ctx.vars().items() if k.startswith("APEX_")} + + target_file = None + if args.sync: + target_file = os.path.join(ctx.paths.compositions, "apex", ".env") + elif args.write: + target_file = args.write + + if target_file: + content = [ + "# Generated by apex env. Do not edit directly.", + "# Contains synthesized node and path scalars for docker compose interpolation.", + ] + for k in sorted(vars_dict.keys()): + content.append(f"{k}={vars_dict[k]}") + content.append("") + body = "\n".join(content) + + os.makedirs(os.path.dirname(os.path.abspath(target_file)), exist_ok=True) + ctx.host.write_file(target_file, body, mode="0600") + log.success(f"Synced {len(vars_dict)} variables to {target_file}.") + return 0 + + if args.dotenv: + for k in sorted(vars_dict.keys()): + print(f"{k}={vars_dict[k]}") + else: + for k in sorted(vars_dict.keys()): + print(f"export {k}={shlex.quote(str(vars_dict[k]))}") + + return 0 diff --git a/actions/init.py b/actions/init.py deleted file mode 100644 index a032ac7..0000000 --- a/actions/init.py +++ /dev/null @@ -1,16 +0,0 @@ -"""Initialize a Debian target without resolving a workstation node identity.""" -from engine.descriptor import Arg, Meta, Opt -from engine.provisioning.transport import initialize - -METADATA = Meta(summary='Initialize Debian 13 through protected SSH enrollment.', args=[ - Arg('target', 'Bootstrap USER@HOST; root or an existing administrator with sudo -n'), - Opt('--user-data', 'Supported APEX cloud-config file'), - Opt('--port', 'Bootstrap SSH port; otherwise use SSH configuration'), - Opt('--identity-file', 'SSH private identity file; otherwise use SSH configuration/agent'), -]) - - -def run(ctx, args): - initialize(ctx.paths.commons, args.target, args.user_data, args.port, args.identity_file, - progress=ctx.log.info) - return 0 diff --git a/actions/sync/repository.py b/actions/sync/repository.py index e0fafae..28c0e32 100644 --- a/actions/sync/repository.py +++ b/actions/sync/repository.py @@ -39,7 +39,7 @@ def run(ctx, args): else: git("checkout", "-b", branch) git("add", "-A") - git("submodule", "update", "--remote", "--recursive") + git("submodule", "update", "--init", "--recursive") git("add", "commons") if git("diff", "--cached", "--quiet", check=False).returncode != 0: git("commit", "-m", f"[{node}] Auto-commit: {ts}") diff --git a/actions/tiers/chown.py b/actions/tiers/chown.py index 97c5279..f04c2b7 100644 --- a/actions/tiers/chown.py +++ b/actions/tiers/chown.py @@ -13,10 +13,11 @@ SHARED_GROUP = "noroot-shared" def _env_val(path, key): if not os.path.exists(path): return "" - for line in open(os.path.realpath(path)): - line = line.strip() - if line.startswith(key + "="): - return line.split("=", 1)[1].strip() + with open(os.path.realpath(path), encoding="utf-8") as f: + for line in f: + line = line.strip() + if line.startswith(key + "="): + return line.split("=", 1)[1].strip() return "" def run(ctx, args): diff --git a/actions/utils/extract-traefik-certs.py b/actions/utils/extract-traefik-certs.py index ea5108e..ea74cd5 100644 --- a/actions/utils/extract-traefik-certs.py +++ b/actions/utils/extract-traefik-certs.py @@ -25,8 +25,9 @@ def run(ctx, args): "--source", f"/data/{src_file}", "--dest", "/output", "--domain-subdir=true", "--crt-name=fullchain", "--crt-ext=.pem", "--key-name=privkey", "--key-ext=.pem"], check=True) - log.info("Setting secure permissions (600) on extracted keys...") - ctx.sys.sudo(["find", abs_dest, "-type", "f", "-name", "*.pem", "-exec", "chmod", "600", "{}", "+"]) + log.info("Setting secure permissions on extracted certs and keys...") + ctx.sys.sudo(["find", abs_dest, "-type", "f", "-name", "*key*.pem", "-exec", "chmod", "600", "{}", "+"]) + ctx.sys.sudo(["find", abs_dest, "-type", "f", "-name", "*chain*.pem", "-exec", "chmod", "644", "{}", "+"]) ctx.sys.sudo(["chown", "-R", f"{os.getuid()}:{os.getgid()}", abs_dest]) log.success("Certificates extracted.") ctx.notify.success(TITLE, url, f"Certificates extracted for {node}") diff --git a/actions/utils/generate-happ-subscriptions.py b/actions/utils/generate-happ-subscriptions.py deleted file mode 100644 index 33dfcd0..0000000 --- a/actions/utils/generate-happ-subscriptions.py +++ /dev/null @@ -1,29 +0,0 @@ -"""Build and run the shared Happ subscription generator.""" -import os -import shutil -from engine.descriptor import Meta, Arg - -METADATA = Meta(summary="Generate .psub files from a JSON config via the happ generator (docker).", - args=[Arg("input_json", "input JSON config"), Arg("output_dir", "output directory")]) -IMAGE = "happ-subscription-generator" - -def run(ctx, args): - log = ctx.log - src_dir = os.path.join(ctx.paths.commons, "assets", "happ-subscription-generator") - inp, out = os.path.realpath(args.input_json), os.path.realpath(args.output_dir) - # GNU realpath (bash) fails unless all but the last component exist — without this - # guard a mistyped path reaches `docker run -v`, which creates it root-owned. - for label, p in (("input_json", inp), ("output_dir", out)): - parent = os.path.dirname(p) - if not os.path.isdir(parent): - log.error(f"Cannot resolve {label}: no such directory: {parent}") - raise SystemExit(1) - if not shutil.which("docker"): - log.error("docker is not installed or not in PATH.") - raise SystemExit(1) - log.info(f"Building {IMAGE}...") - ctx.sys.run(["docker", "build", "-t", IMAGE, src_dir], check=True) - log.info(f"Running generator: {inp} -> {out}") - ctx.sys.run(["docker", "run", "--rm", "-v", f"{inp}:/app/input.json:ro", - "-v", f"{out}:/app/output:rw", IMAGE, "/app/input.json", "/app/output"], check=True) - log.success("Subscription generation completed.") diff --git a/actions/utils/lint-docker-compose.py b/actions/utils/lint-docker-compose.py index e36e010..30c6e41 100644 --- a/actions/utils/lint-docker-compose.py +++ b/actions/utils/lint-docker-compose.py @@ -57,7 +57,8 @@ def _validate_anchor_ips(ctx) -> int: if not os.path.isfile(f): continue try: - text = open(f).read() + with open(f, encoding="utf-8") as fh: + text = fh.read() except OSError as e: ctx.log.error(f"Cannot read {f}: {e}"); rc = 1 continue diff --git a/assets/composition-failover/README.md b/assets/composition-failover/README.md deleted file mode 100644 index fef3c5d..0000000 --- a/assets/composition-failover/README.md +++ /dev/null @@ -1,102 +0,0 @@ -# Readiness-gated origin routing on Workers Free - -This local ES-module Worker asset selects between two origins for one public HTTPS hostname. It implements the ingress part of the [composition replication RFC](../../docs/rfcs/composition-replication.md). It is not deployed and does not establish production high availability. Use the multi-host entrypoint below when hostnames share origins and a readiness secret. Verify origin routes and readiness independently for every hostname. Activate only after the declared backend failure behavior and authenticated application readiness are verified. The Worker does not promote databases or activate a standby. - -The module has no runtime dependencies, paid bindings, KV, R2, storage or scheduled jobs. This directory does not enable billing, enroll an account, upload a Worker or change DNS/routes. - -## Bindings - -Set these string bindings through the operator's chosen Cloudflare deployment process. Keep the readiness token in a Worker secret, never a committed variable, URL or log. - -| Binding | Value | -| --- | --- | -| `APEX_PUBLIC_HOST` | Existing public SSO hostname, for example `sso.example.com` | -| `APEX_ZONE` | The actual Cloudflare DNS zone, for example `example.com` | -| `APEX_ORIGIN_PRIMARY` | Dedicated DNS-only primary origin hostname in that zone | -| `APEX_ORIGIN_SECONDARY` | Dedicated DNS-only secondary origin hostname in that zone | -| `APEX_READY_PATH` | Dedicated absolute readiness path, for example `/.well-known/apex-ready` | -| `APEX_READY_TOKEN` | High-entropy secret shared only by this Worker and both readiness adapters | -| `APEX_READY_STATUS` | Optional exact status string: `204` (default) or `200` for a dedicated application health route | - -Hostnames accept ordinary ASCII DNS labels, without a scheme, port, trailing dot, wildcard or IP literal. Case is normalized. Public and origin hostnames must be distinct and within the declared zone. Readiness paths accept ASCII letters, digits, `.`, `_`, `~`, `-` and `/`; queries, fragments, escapes, repeated slashes and dot traversal segments are rejected. The secret must be nonempty visible ASCII without whitespace. Invalid/missing bindings produce a generic no-store 503 before any fetch. Syntax validation cannot establish actual DNS ownership or prevent an operator from naming a separately delegated zone; deployment must verify both origins are in the actual same Cloudflare zone. - -## Routing and readiness contract - -Create two dedicated DNS-only origin records pointing to the participating servers; do not point either at the public hostname or a Worker. Retain the existing proxied public SSO DNS record. Use a **Worker Route** such as `sso.example.com/*` on that existing hostname, not a Worker Custom Domain. Both origins must serve HTTPS with a valid certificate and virtual host for the public name. Configure strict origin TLS verification. Disable any Workers Cache in front of this Worker and any cache policy that would bypass its invocation. [Cloudflare Routes](https://developers.cloudflare.com/workers/configuration/routing/routes/) - -Require `global_fetch_private_origin` in the Worker compatibility flags and prohibit `global_fetch_strictly_public`. The strictly-public flag sends same-zone global fetches through Cloudflare's public routing layer and can re-enter this Worker; a readiness probe would then hit its public-path rejection instead of the origin. Private-origin behavior routes same-zone subrequests to the origin, bypassing mapped Workers and Cloudflare security settings, so the origin adapter must still enforce its own authentication. [Global fetch compatibility](https://developers.cloudflare.com/workers/configuration/compatibility-flags/#global-fetch-strictly-public) - -Set `compatibility_date` to **2024-11-11 or later** and do not set `cache_option_disabled`. `cache_option_enabled` became the default on that date; the example also enables it explicitly. Without this behavior, constructing requests with `cache: "no-store"` throws instead of dispatching. [Request cache compatibility](https://developers.cloudflare.com/workers/configuration/compatibility-flags/#enable-cache-no-store-http-standard-api) - -The required non-secret upload metadata fields are: - -```json -{ - "main_module": "worker.mjs", - "compatibility_date": "2026-10-10", - "compatibility_flags": [ - "global_fetch_private_origin", - "cache_option_enabled" - ] -} -``` - -Use this as the multipart API `metadata` portion, with the ES-module file part named `worker.mjs`. Bindings and the readiness secret are configured separately through the operator-owned enrollment process; they are deliberately absent from this example, as are paid bindings. This metadata is not a complete enrollment or route activation command. Verify the resulting effective settings before activation, including after later uploads. [Multipart upload metadata](https://developers.cloudflare.com/workers/configuration/multipart-upload-metadata/) - -Every accepted HTTPS request first probes the primary with a GET at `https://APEX_PUBLIC_HOST/APEX_READY_PATH`. The request carries only `X-APEX-Ready-Token`, rather than client cookies or authorization. `cf.resolveOverride` selects the DNS-only origin while the URL remains the public hostname. Only the configured exact status is ready: 204 by default, or 200 when explicitly selected with `APEX_READY_STATUS`. Redirects are manual and never accepted. Do not use 200 with a generic login page or native liveness endpoint; a protected application health route must verify both active SSO dependencies and its own backend. Each probe has a five-second deadline. Unread response bodies are canceled, including responses arriving after timeout. Failed primary readiness leads to a secondary probe; neither ready produces a no-store 503. There is no readiness cache or background health state. - -The dedicated origin adapter must authenticate the token, reject its absence/incorrect value and prove that this application instance can safely serve traffic: required dependencies, compatible configuration, current write authority for the declared topology, session state and any application-specific prerequisites. Authelia `/api/health`, an nginx 204, a TCP check or a running container alone is insufficient. Protect adapter aliases and normalize paths consistently at the origin. The Worker rejects public requests to the configured readiness path (also with a query or percent-encoded spelling), and removes every client-supplied readiness header before application dispatch. An unpromoted standby must remain unready. This asset does not implement or enroll the backend adapter. Preserve existing path-specific routes when attaching the Worker; two ingress paths do not replicate a shared application backend. - -Cloudflare documents `resolveOverride` as DNS substitution while retaining the URL's Host, subject to both names belonging to the same zone. The asset retains the HTTPS public URL for routing; actual origin TLS SNI/certificate selection and routing must be demonstrated natively before activation. No local Node test proves Cloudflare DNS/TLS behavior. [Request API](https://developers.cloudflare.com/workers/runtime-apis/request/) - -## Application request behavior - -After selection, the original method, URL/path/query, body, cookies and authorization are forwarded once. The body is streamed; no request-body buffering or replay copy is made. The client readiness header is removed. Application redirects remain manual, and response status, body, Location and separate Set-Cookie headers are preserved. - -**No application request is replayed after an origin failure, including GET callbacks.** A fetch exception returns a generic no-store 502; an origin HTTP error is returned with its original status/body. There is no second probe or alternative-origin dispatch after this point. Readiness can become stale between probe and dispatch; failover applies to subsequent requests. Application requests have no additional Worker timeout so streaming is not cut off; platform/client deadlines still apply. - -All probe and application fetches use `cache: "no-store"`, `cacheEverything: false` and negative `cacheTtlByStatus`. Ordinary responses override browser/CDN cache headers to no-store. There is no Cache API use. Verify native cache bypass with cookies, authorization, redirects and authentication error responses before enrollment. [Fetch caching](https://developers.cloudflare.com/workers/runtime-apis/fetch/) - -A platform status-101 response is returned unchanged to preserve Cloudflare's WebSocket upgrade object. The Worker does not terminate or inspect WebSocket messages, migrate existing connections, reconnect clients or claim protocol HA. Ordinary response bodies remain streams. Native WebSocket behavior is an optional deployment test if the application needs it; the local test only checks unchanged forwarding of the platform response. XHTTP and XHTTP-over-Cloudflare are excluded from this SSO HA scope. [Workers Response API](https://developers.cloudflare.com/workers/runtime-apis/response/) - -Other hostnames, non-default ports and non-HTTPS requests receive no-store 421. The public route must already have the intended HTTPS policy. Client errors and unavailable responses contain no secret diagnostics; the module emits no logs. - -## Free-tier deployment gates - -Workers Free currently provides **100,000 incoming requests per day per account**, reset at 03:00 MSK (midnight UTC), and **10 ms CPU per HTTP request**. Quota is shared with the account's other Workers. This asset performs two fetch subrequests normally (one probe plus application dispatch), or three when primary readiness fails. Waiting for network I/O does not count toward CPU time, but actual Worker CPU consumption must be measured; Node wall time is not that measurement. [Cloudflare Workers limits](https://developers.cloudflare.com/workers/platform/limits/) - -Keep the account on Workers Free. Do not enable a paid plan, paid load balancer or paid bindings for this asset. An operator must inspect existing account usage and peak traffic to confirm the free quota is sufficient. This is a no-added-cost deployment candidate, not unlimited availability. - -Set the Worker Route to **fail closed**. On Free quota exhaustion, Cloudflare documents a 1027 error page for this mode; fail open bypasses the Worker and is unacceptable here. The module cannot enforce the platform route setting or prevent quota exhaustion. Record and verify that setting before activation. - -Operator-owned enrollment gates: - -1. Validate the declared backend failure behavior, adapter authentication/readiness, dependency loss and write-authority withdrawal independently of this Worker. -2. Confirm account Free plan/quota, same-zone DNS-only origin records, public proxied Route, strict TLS, native SNI/Host at each origin, required private-origin/cache compatibility flags and date, and no Worker/route recursion or pre-Worker caching. -3. Test native primary/secondary readiness, both-unready failure, redirects, cookies, cache bypass, POST and GET callback non-replay, streaming and any required WebSockets. Measure Worker CPU and account request headroom. -4. Verify fail-closed quota handling and exercise location loss, network partitions, stale/rejoining origins and recovery. Record observed recovery time and session behavior before making an HA claim. -5. Activate only after those gates pass with the intended backend failure behavior. Maintain a documented operator rollback that does not silently route to an unsafe origin. - -## Local verification - -From the shared APEX repository root, with Node available: - -```sh -node --test assets/composition-failover/worker.test.mjs -``` - -Tested with Node v24.11.1 using its built-in test runner and Web APIs. Tests call the exported handler and replace only the external fetch boundary, with fake timers for probe deadlines. They do not call Cloudflare or use credentials. No npm install, new dependency or change to the default Python test workflow is required. Native Cloudflare deployment is deliberately outside this local test's proof. - -## Several hostnames in one Worker - -Use `multi-host.mjs` as the main module and upload `worker.mjs` alongside it when -several hostnames share the same origins and readiness secret. Attach one Route -per hostname to that single Worker. Set `APEX_PUBLIC_HOSTS` to a JSON object of -allowed hostnames and exact health status strings, for example -`{"sso.example.com":"204","app.example.com":"200"}`. It replaces the -`APEX_PUBLIC_HOST` and `APEX_READY_STATUS` bindings; all other bindings and -deployment gates above still apply. Unknown hostnames are rejected before any -origin request. Each allowed hostname uses the existing single-host handler, -including its authentication, caching, streaming and no-replay behavior. - -Verify both entrypoints with `node --test assets/composition-failover/*.test.mjs`. diff --git a/assets/composition-failover/multi-host.mjs b/assets/composition-failover/multi-host.mjs deleted file mode 100644 index 0575eb7..0000000 --- a/assets/composition-failover/multi-host.mjs +++ /dev/null @@ -1,23 +0,0 @@ -import singleHost from './worker.mjs'; - -function unavailable(status) { - return new Response(status === 421 ? 'Misdirected request' : 'Service unavailable', { - status, headers: { 'Cache-Control': 'private, no-store' }, - }); -} - -export default { - async fetch(request, env) { - let hosts; - try { hosts = JSON.parse(env?.APEX_PUBLIC_HOSTS); } catch { return unavailable(503); } - if (!hosts || Array.isArray(hosts) || typeof hosts !== 'object') return unavailable(503); - const entries = Object.entries(hosts); - if (!entries.length || entries.length > 64 || entries.some(([name, status]) => - name !== name.toLowerCase() || name.length > 253 || !name.includes('.') - || !name.split('.').every((label) => /^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$/.test(label)) - || !['200', '204'].includes(status))) return unavailable(503); - const host = new URL(request.url).hostname; - if (!Object.hasOwn(hosts, host)) return unavailable(421); - return singleHost.fetch(request, { ...env, APEX_PUBLIC_HOST: host, APEX_READY_STATUS: hosts[host] }); - }, -}; diff --git a/assets/composition-failover/multi-host.test.mjs b/assets/composition-failover/multi-host.test.mjs deleted file mode 100644 index 105fe5d..0000000 --- a/assets/composition-failover/multi-host.test.mjs +++ /dev/null @@ -1,53 +0,0 @@ -import assert from 'node:assert/strict'; -import test from 'node:test'; -import worker from './multi-host.mjs'; - -const settings = { - APEX_ZONE: 'example.com', - APEX_PUBLIC_HOSTS: JSON.stringify({ 'sso.example.com': '204', 'app.example.com': '200' }), - APEX_ORIGIN_PRIMARY: 'one.example.com', - APEX_ORIGIN_SECONDARY: 'two.example.com', - APEX_READY_PATH: '/.well-known/apex-ready', - APEX_READY_TOKEN: 'synthetic-private-token', -}; - -test('each allowed hostname uses its own health status and preserves the application request', async () => { - const original = globalThis.fetch; - try { - for (const [host, health] of [['sso.example.com', 204], ['app.example.com', 200]]) { - const calls = []; - globalThis.fetch = async (request) => { - calls.push(request); - return calls.length === 1 ? new Response(null, { status: health }) : new Response('accepted'); - }; - const result = await worker.fetch(new Request(`https://${host}/callback`, { method: 'POST', body: 'one-use-body' }), settings); - assert.equal(result.status, 200); - assert.equal(calls.length, 2); - assert.equal(new URL(calls[0].url).hostname, host); - assert.equal(calls[1].method, 'POST'); - assert.equal(await calls[1].text(), 'one-use-body'); - } - } finally { globalThis.fetch = original; } -}); - -test('unknown hosts do not probe or dispatch', async () => { - const original = globalThis.fetch; - globalThis.fetch = async () => { throw new Error('unexpected dispatch'); }; - try { - const response = await worker.fetch(new Request('https://other.example.com/'), settings); - assert.equal(response.status, 421); - } finally { globalThis.fetch = original; } -}); - -test('missing and malformed host policies fail closed before dispatch', async () => { - const original = globalThis.fetch; - let calls = 0; - globalThis.fetch = async () => { calls++; return new Response('unexpected'); }; - try { - for (const value of [undefined, '{', '[]', '{}', '{"app.example.com":"302"}', '{"APP.example.com":"200"}']) { - const response = await worker.fetch(new Request('https://app.example.com/'), { ...settings, APEX_PUBLIC_HOSTS: value }); - assert.equal(response.status, 503); - } - assert.equal(calls, 0); - } finally { globalThis.fetch = original; } -}); diff --git a/assets/composition-failover/worker.mjs b/assets/composition-failover/worker.mjs deleted file mode 100644 index 24f3b2f..0000000 --- a/assets/composition-failover/worker.mjs +++ /dev/null @@ -1,114 +0,0 @@ -const READY_HEADER = 'X-APEX-Ready-Token'; -const PROBE_TIMEOUT_MS = 5000; - -function hostname(value) { - if (typeof value !== 'string' || value.length > 253 || !value.includes('.')) return null; - const name = value.toLowerCase(); - if (!name.split('.').every((label) => /^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$/.test(label))) return null; - if (/^[0-9.]+$/.test(name)) return null; - return name; -} - -function configuration(env) { - if (!env) return null; - const zone = hostname(env.APEX_ZONE); - const publicHost = hostname(env.APEX_PUBLIC_HOST); - const primary = hostname(env.APEX_ORIGIN_PRIMARY); - const secondary = hostname(env.APEX_ORIGIN_SECONDARY); - if (!zone || ![publicHost, primary, secondary].every((name) => - name && (name === zone || name.endsWith('.' + zone)))) return null; - if (new Set([publicHost, primary, secondary]).size !== 3) return null; - const path = env.APEX_READY_PATH; - if (typeof path !== 'string' || !/^\/[a-zA-Z0-9._~/-]*$/.test(path) - || path.includes('//') || path.split('/').some((part) => part === '.' || part === '..')) return null; - const token = env.APEX_READY_TOKEN; - if (typeof token !== 'string' || !/^[\x21-\x7e]+$/.test(token)) return null; - const status = env.APEX_READY_STATUS === undefined ? '204' : env.APEX_READY_STATUS; - if (!['200', '204'].includes(status)) return null; - return { publicHost, primary, secondary, path, token, status: Number(status) }; -} - -function unavailable(status = 503, message = 'Service unavailable') { - return new Response(message, { status, headers: { 'Cache-Control': 'private, no-store' } }); -} - -function options(origin) { - return { - redirect: 'manual', - cache: 'no-store', - cf: { resolveOverride: origin, cacheEverything: false, cacheTtlByStatus: { '100-599': -1 } }, - }; -} - -function discard(response) { - // Do not let a slow body cancellation extend the probe deadline. - if (response.body) response.body.cancel().catch(() => {}); -} - -async function ready(config, origin) { - const controller = new AbortController(); - let timer; - const deadline = new Promise((resolve) => { - timer = setTimeout(() => { - controller.abort(); - resolve(false); - }, PROBE_TIMEOUT_MS); - }); - try { - const probe = new Request(`https://${config.publicHost}${config.path}`, { - headers: { [READY_HEADER]: config.token }, - redirect: 'manual', - cache: 'no-store', - }); - const inspection = fetch(probe, { ...options(origin), signal: controller.signal }) - .then((response) => { - const accepted = !controller.signal.aborted && response.status === config.status; - discard(response); - return accepted; - }, () => false); - return await Promise.race([inspection, deadline]); - } catch { - return false; - } finally { - clearTimeout(timer); - } -} - -export default { - async fetch(request, env) { - const config = configuration(env); - if (!config) return unavailable(); - const url = new URL(request.url); - if (url.hostname !== config.publicHost || url.port || url.protocol !== 'https:') { - return unavailable(421, 'Misdirected request'); - } - let path; - try { - path = decodeURIComponent(url.pathname); - } catch { - return unavailable(400, 'Invalid path'); - } - if (path === config.path) return unavailable(404, 'Not found'); - - let origin; - if (await ready(config, config.primary)) origin = config.primary; - else if (await ready(config, config.secondary)) origin = config.secondary; - else return unavailable(); - - try { - const headers = new Headers(request.headers); - headers.delete(READY_HEADER); - const forwarded = new Request(request, { headers, redirect: 'manual', cache: 'no-store' }); - // This is the only application dispatch. Its failure never triggers replay. - const response = await fetch(forwarded, options(origin)); - if (response.status === 101) return response; - const result = new Response(response.body, response); - result.headers.set('Cache-Control', 'private, no-store'); - result.headers.set('CDN-Cache-Control', 'no-store'); - result.headers.set('Cloudflare-CDN-Cache-Control', 'no-store'); - return result; - } catch { - return unavailable(502, 'Origin request failed'); - } - }, -}; diff --git a/assets/composition-failover/worker.test.mjs b/assets/composition-failover/worker.test.mjs deleted file mode 100644 index a0cd5dd..0000000 --- a/assets/composition-failover/worker.test.mjs +++ /dev/null @@ -1,265 +0,0 @@ -import assert from 'node:assert/strict'; -import { test } from 'node:test'; -import worker from './worker.mjs'; - -const env = { - APEX_PUBLIC_HOST: 'sso.example.com', - APEX_ZONE: 'example.com', - APEX_ORIGIN_PRIMARY: 'primary.example.com', - APEX_ORIGIN_SECONDARY: 'secondary.example.com', - APEX_READY_PATH: '/.well-known/apex-ready', - APEX_READY_TOKEN: 'test-only-readiness-secret', -}; -const url = 'https://sso.example.com/api/callback?state=a%2Fb'; -const ready = () => new Response(null, { status: 204 }); - -function intercept(t, handler) { - const calls = []; - t.mock.method(globalThis, 'fetch', async (request, options) => { - calls.push({ request, options }); - return handler(request, options, calls.length); - }); - return calls; -} - -function checkProbe(call, origin) { - assert.equal(call.request.url, 'https://sso.example.com/.well-known/apex-ready'); - assert.equal(call.request.method, 'GET'); - assert.equal(call.request.headers.get('X-APEX-Ready-Token'), env.APEX_READY_TOKEN); - assert.equal(call.request.headers.get('Cookie'), null); - assert.equal(call.request.headers.get('Authorization'), null); - assert.equal(call.options.cf.resolveOverride, origin); - assert.equal(call.options.redirect, 'manual'); - assert.equal(call.options.cache, 'no-store'); - assert.deepEqual(call.options.cf.cacheTtlByStatus, { '100-599': -1 }); -} - -test('ready primary receives the original POST exactly once with credentials and body', async (t) => { - const calls = intercept(t, async (request, options, count) => { - if (count === 1) return ready(); - assert.equal(await request.text(), 'code=one-use&state=a%2Fb'); - return new Response('accepted', { status: 201 }); - }); - const request = new Request(url, { - method: 'POST', body: 'code=one-use&state=a%2Fb', - headers: { Cookie: 'session=original', Authorization: 'Bearer original', - 'X-APEX-Ready-Token': 'client-supplied', 'Content-Type': 'application/x-www-form-urlencoded' }, - }); - const response = await worker.fetch(request, env); - assert.equal(response.status, 201); - assert.equal(await response.text(), 'accepted'); - assert.equal(calls.length, 2); - checkProbe(calls[0], env.APEX_ORIGIN_PRIMARY); - const forwarded = calls[1]; - assert.equal(forwarded.request.url, url); - assert.equal(forwarded.request.method, 'POST'); - assert.equal(forwarded.request.headers.get('Cookie'), 'session=original'); - assert.equal(forwarded.request.headers.get('Authorization'), 'Bearer original'); - assert.equal(forwarded.request.headers.get('X-APEX-Ready-Token'), null); - assert.equal(forwarded.options.cf.resolveOverride, env.APEX_ORIGIN_PRIMARY); - assert.equal(forwarded.options.redirect, 'manual'); - assert.equal(forwarded.options.cache, 'no-store'); - assert.match(response.headers.get('Cache-Control'), /no-store/); -}); - -test('unready primary cancels unread body and selects ready secondary', async (t) => { - let canceled = false; - const calls = intercept(t, (_, __, count) => count === 1 - ? new Response(new ReadableStream({ cancel() { canceled = true; } }), { status: 503 }) - : count === 2 ? ready() : new Response('secondary')); - const response = await worker.fetch(new Request(url), env); - assert.equal(await response.text(), 'secondary'); - assert.equal(canceled, true); - assert.equal(calls.length, 3); - checkProbe(calls[1], env.APEX_ORIGIN_SECONDARY); - assert.equal(calls[2].options.cf.resolveOverride, env.APEX_ORIGIN_SECONDARY); -}); - -test('probe deadline aborts at five seconds even if fetch ignores cancellation', async (t) => { - t.mock.timers.enable({ apis: ['setTimeout'] }); - let finishPrimary; - let canceled = false; - const calls = intercept(t, (_, __, count) => count === 1 - ? new Promise((resolve) => { finishPrimary = resolve; }) - : count === 2 ? ready() : new Response('secondary')); - const pending = worker.fetch(new Request(url), env); - await Promise.resolve(); - t.mock.timers.tick(4999); - assert.equal(calls.length, 1); - assert.equal(calls[0].options.signal.aborted, false); - t.mock.timers.tick(1); - const response = await pending; - assert.equal(await response.text(), 'secondary'); - assert.equal(calls[0].options.signal.aborted, true); - finishPrimary(new Response(new ReadableStream({ cancel() { canceled = true; } }))); - await new Promise((resolve) => setImmediate(resolve)); - assert.equal(canceled, true); - assert.equal(calls.length, 3); -}); - -test('two unavailable origins return no-store 503 without client dispatch', async (t) => { - const calls = intercept(t, () => new Response('down', { status: 503 })); - const response = await worker.fetch(new Request(url), env); - assert.equal(response.status, 503); - assert.match(response.headers.get('Cache-Control'), /no-store/); - assert.equal(calls.length, 2); - assert.ok(calls.every(({ request }) => new URL(request.url).pathname === env.APEX_READY_PATH)); -}); - -test('readiness redirect is never accepted or followed', async (t) => { - const calls = intercept(t, (_, __, count) => count === 1 - ? new Response(null, { status: 302, headers: { Location: 'https://evil.example/' } }) - : count === 2 ? ready() : new Response('secondary')); - assert.equal(await (await worker.fetch(new Request(url), env)).text(), 'secondary'); - assert.equal(calls.length, 3); - assert.equal(calls[0].options.redirect, 'manual'); -}); - -test('application health can explicitly require 200 while 204 remains the default', async (t) => { - const bindings = { ...env, APEX_READY_STATUS: '200' }; - const calls = intercept(t, (_, __, count) => count === 1 - ? new Response(null, { status: 204 }) - : count === 2 ? new Response('Healthy', { status: 200 }) : new Response('secondary')); - const response = await worker.fetch(new Request(url), bindings); - assert.equal(await response.text(), 'secondary'); - assert.equal(calls.length, 3); - assert.equal(calls[2].options.cf.resolveOverride, env.APEX_ORIGIN_SECONDARY); -}); - -test('an unconfigured 200 health response cannot release application traffic', async (t) => { - const calls = intercept(t, () => new Response('login page', { status: 200 })); - assert.equal((await worker.fetch(new Request(url), env)).status, 503); - assert.equal(calls.length, 2); -}); - -for (const method of ['GET', 'POST']) { - test(`failed ${method} dispatch is never replayed on another origin`, async (t) => { - const calls = intercept(t, (_, __, count) => { - if (count === 1) return ready(); - throw new DOMException('origin timed out', 'TimeoutError'); - }); - const response = await worker.fetch(new Request(url, { method, - ...(method === 'POST' ? { body: 'one-use' } : {}) }), env); - assert.equal(response.status, 502); - assert.match(response.headers.get('Cache-Control'), /no-store/); - assert.equal(calls.length, 2); - }); -} - -test('origin errors preserve status and body without probing secondary', async (t) => { - const calls = intercept(t, (_, __, count) => count === 1 ? ready() : new Response('origin error', { status: 500 })); - const response = await worker.fetch(new Request(url), env); - assert.equal(response.status, 500); - assert.equal(await response.text(), 'origin error'); - assert.equal(calls.length, 2); -}); - -test('auth redirect retains distinct Set-Cookie headers and Location but cannot cache', async (t) => { - const cookies = ['session=first; Secure; HttpOnly', 'state=second; Expires=Wed, 21 Oct 2030 07:28:00 GMT']; - const headers = new Headers({ Location: '/continue?state=a%2Fb', 'Cache-Control': 'public, max-age=3600' }); - for (const cookie of cookies) headers.append('Set-Cookie', cookie); - const calls = intercept(t, (_, __, count) => count === 1 ? ready() : new Response(null, { status: 302, headers })); - const response = await worker.fetch(new Request(url), env); - assert.equal(response.status, 302); - assert.equal(response.headers.get('Location'), '/continue?state=a%2Fb'); - assert.deepEqual(response.headers.getSetCookie(), cookies); - assert.match(response.headers.get('Cache-Control'), /no-store/); - assert.equal(response.headers.get('Cloudflare-CDN-Cache-Control'), 'no-store'); - assert.equal(calls.length, 2); -}); - -test('hostile hosts and public readiness requests never reach origins', async (t) => { - const calls = intercept(t, () => { throw new Error('must not dispatch'); }); - for (const target of ['https://evil.example/', 'https://sso.example.com.evil.example/', 'https://sso.example.com:444/']) { - assert.equal((await worker.fetch(new Request(target), env)).status, 421); - } - for (const path of [env.APEX_READY_PATH, env.APEX_READY_PATH + '?probe=1', '/.well-known/%61pex-ready']) { - const response = await worker.fetch(new Request('https://sso.example.com' + path), env); - assert.equal(response.status, 404); - assert.match(response.headers.get('Cache-Control'), /no-store/); - } - assert.equal(calls.length, 0); -}); - -test('missing and invalid bindings fail closed without diagnostics or probes', async (t) => { - const calls = intercept(t, () => { throw new Error('must not dispatch'); }); - const invalid = Object.keys(env).map((name) => ({ ...env, [name]: undefined })); - for (const [name, value] of [ - ['APEX_ZONE', 'com'], ['APEX_PUBLIC_HOST', 'https://sso.example.com'], - ['APEX_ORIGIN_PRIMARY', 'primary.other.example'], ['APEX_ORIGIN_PRIMARY', env.APEX_PUBLIC_HOST], - ['APEX_ORIGIN_PRIMARY', env.APEX_ORIGIN_SECONDARY], ['APEX_ORIGIN_SECONDARY', '127.0.0.1'], - ['APEX_ORIGIN_SECONDARY', '-invalid.example.com'], ['APEX_ORIGIN_PRIMARY', 'a..example.com'], - ['APEX_READY_PATH', '//evil.example/'], ['APEX_READY_PATH', '/ready?query=yes'], - ['APEX_READY_PATH', '/ready#fragment'], ['APEX_READY_PATH', '/a/../ready'], - ['APEX_READY_PATH', '/%72eady'], ['APEX_READY_TOKEN', 'secret\r\ninjected: yes'], - ['APEX_READY_STATUS', '302'], ['APEX_READY_STATUS', '401'], - ['APEX_READY_STATUS', '503'], ['APEX_READY_STATUS', '2xx'], - ['APEX_READY_STATUS', ''], ['APEX_READY_STATUS', 200], ['APEX_READY_STATUS', null], - ]) invalid.push({ ...env, [name]: value }); - invalid.push(null); - for (const bindings of invalid) { - const response = await worker.fetch(new Request(url), bindings); - assert.equal(response.status, 503); - assert.match(response.headers.get('Cache-Control'), /no-store/); - assert.equal(await response.text(), 'Service unavailable'); - } - assert.equal(calls.length, 0); -}); - -test('response remains streaming without buffering the body', async (t) => { - let controller; - const body = new ReadableStream({ start(value) { controller = value; } }); - intercept(t, (_, __, count) => count === 1 ? ready() : new Response(body)); - const response = await worker.fetch(new Request(url), env); - const reader = response.body.getReader(); - controller.enqueue(new TextEncoder().encode('first')); - assert.equal(new TextDecoder().decode((await reader.read()).value), 'first'); - controller.close(); - assert.equal((await reader.read()).done, true); -}); - -test('failed readiness fetch selects secondary without exposing exception details', async (t) => { - const calls = intercept(t, (_, __, count) => { - if (count === 1) throw new Error(env.APEX_READY_TOKEN); - return count === 2 ? ready() : new Response('secondary'); - }); - const response = await worker.fetch(new Request(url), env); - assert.equal(await response.text(), 'secondary'); - assert.equal(calls.length, 3); -}); - -test('secondary probe has its own five-second deadline and no client dispatch follows', async (t) => { - t.mock.timers.enable({ apis: ['setTimeout'] }); - const calls = intercept(t, () => new Promise(() => {})); - const pending = worker.fetch(new Request(url), env); - t.mock.timers.tick(5000); - await new Promise((resolve) => setImmediate(resolve)); - assert.equal(calls.length, 2); - assert.equal(calls[0].options.signal.aborted, true); - assert.equal(calls[1].options.signal.aborted, false); - t.mock.timers.tick(4999); - assert.equal(calls[1].options.signal.aborted, false); - t.mock.timers.tick(1); - const response = await pending; - assert.equal(response.status, 503); - assert.equal(calls[1].options.signal.aborted, true); - assert.equal(calls.length, 2); -}); - -test('unread probe cancellation cannot stall selection', async (t) => { - const calls = intercept(t, (_, __, count) => count === 1 - ? new Response(new ReadableStream({ cancel() { return new Promise(() => {}); } }), { status: 500 }) - : count === 2 ? ready() : new Response('secondary')); - assert.equal(await (await worker.fetch(new Request(url), env)).text(), 'secondary'); - assert.equal(calls.length, 3); -}); - -test('platform WebSocket upgrade response is returned unchanged', async (t) => { - // Node cannot construct status 101; the fetch boundary represents Workers' response. - const upgrade = { status: 101, webSocket: {}, headers: new Headers({ Upgrade: 'websocket' }) }; - const calls = intercept(t, (_, __, count) => count === 1 ? ready() : upgrade); - const response = await worker.fetch(new Request(url, { headers: { Upgrade: 'websocket' } }), env); - assert.equal(response, upgrade); - assert.equal(calls.length, 2); - assert.equal(calls[1].request.headers.get('Upgrade'), 'websocket'); -}); diff --git a/assets/composition-readiness/README.md b/assets/composition-readiness/README.md deleted file mode 100644 index 544bc13..0000000 --- a/assets/composition-readiness/README.md +++ /dev/null @@ -1,34 +0,0 @@ -# Authenticated composition readiness adapter - -This dependency-free Python adapter exposes one local HTTP endpoint for the readiness-gated Worker and origin proxy. By default it runs an operator-supplied executable for every authenticated probe; exit0 means ready, every other outcome means unavailable. The executable must establish the application's actual readiness contract. A successful TCP connection or container health flag is not enough for replicated SSO. - -Example private configuration: - -```json -{ - "host": "127.0.0.1", - "port": 9092, - "path": "/.well-known/apex-ready", - "token_file": "/private/readiness-token", - "command": ["/usr/local/bin/python3", "/assets/probe.py", "--config", "/private/probe.json"], - "timeout": 1.5, - "slots": 4, - "single_flight": false -} -``` - -Run `python3 adapter.py --config /private/adapter.json` inside the application frontend namespace. Keep the token and probe credentials private. A64-character lowercase hexadecimal random token is suitable. This adapter does not need issuer private keys, a Docker socket, root, writable application data or network-administration capabilities. Bind only to loopback; route authorized origin readiness through a separately validated local proxy. - -Only GET at the exact canonical path, with one matching `X-APEX-Ready-Token` header, can request readiness. In the default mode it executes the fixed command. Missing/wrong/duplicate credentials return403; other paths404; supported non-GET methods405. Ready returns204; probe failure, timeout or concurrency exhaustion returns503. These responses have empty bodies and `Cache-Control: no-store`. The adapter does not log requests, tokens or command output, or accept a client-supplied command. Each connection closes after its response. - -Probe timeout defaults to1.5seconds and can be configured up to3seconds. With a3-second probe, allow4seconds at the readiness proxy and5seconds at the Worker. A timeout kills the probe's process group. Probe slots and request threads are bounded; slow socket operations time out after two seconds. Operators must account for proxy overhead and actual probe execution time. Probe programs must not detach into a new session or launch background jobs. - -Optional `single_flight: true` lets overlapping authenticated requests share one running fixed-command probe. It accepts only a JSON boolean and defaults to false. Waiters use that probe's remaining monotonic deadline; they never queue another probe for the same request. The existing connection limit (`slots * 2 + 4`) also bounds waiting requests. Timeout or shutdown returns unavailable and terminates the probe group. Completed results are not cached: each subsequent request starts a fresh probe, with no success TTL or failure cache. Coalescing reduces concurrent work but does not make a slow individual probe meet its deadline. - -For checks containing expensive authentication, optional `refresh_interval: 2` and `max_age: 5` run one background probe every two seconds and serve its latest result immediately. This avoids repeating SQL, directory and session checks for each page or asset, including the Worker probe followed by the origin proxy's own check. Probes never overlap; a slow probe delays the next start. Startup, a failed or timed-out probe, shutdown, and an expired result all return503. Freshness uses monotonic time measured from the probe's **start**, not completion, so a stopped refresher cannot retain success indefinitely. With these settings readiness can lag a dependency or role change by at most five seconds. This does not confer write authority or replace fencing during manual promotion. - -Background mode is disabled by default (`refresh_interval: 0`). Its interval must be positive and at most five seconds; `max_age` must cover the interval plus probe timeout and cannot exceed ten seconds. A three-second timeout, two-second interval and five-second maximum age are suitable for the current SSO checks. Token validation, bounded connections, process-group termination and no-store HTTP responses still apply. Request traffic does not launch extra probes in this mode; `single_flight` is used only in the default request-driven mode. - -For the controlled active/standby SSO pattern, the probe must reject a node not explicitly configured as active, verify completed migration and unchanged Authelia configuration, and check local Authelia health, authenticated SQL access/schema with read_only disabled, directory access, and an authenticated session write/read/delete over verified TLS. All dependencies belong to the selected local identity instance. These checks do not establish exclusive primary ownership: the operator must fence the previous writer before promotion, and the standby application and ingress stay disabled. No quorum, Sentinel, or session-controller authority is required by this pattern. The generic adapter supplies none of those application checks; do not deploy a placeholder-success probe or claim these tests establish HA. - -Run `python3 test_adapter.py` for loopback HTTP and subprocess behavior checks. No third-party packages are required. Production native image/probe/proxy integration and location-loss tests remain separate acceptance gates. diff --git a/assets/composition-readiness/adapter.py b/assets/composition-readiness/adapter.py deleted file mode 100644 index 5975492..0000000 --- a/assets/composition-readiness/adapter.py +++ /dev/null @@ -1,233 +0,0 @@ -"""Authenticated readiness endpoint for a fixed, bounded application probe.""" -import argparse -import hmac -import http.server -import json -import os -import pathlib -import re -import signal -import subprocess -import threading -import time - - -class ReadinessServer(http.server.ThreadingHTTPServer): - daemon_threads = True - block_on_close = False - allow_reuse_address = True - request_queue_size = 64 - - def __init__(self, address, *, token, path, command, timeout=1.5, slots=4, single_flight=False, refresh_interval=0, max_age=5): - if not isinstance(token, str) or not re.fullmatch(r'[!-~]{16,512}', token): - raise ValueError('A private visible-ASCII token is required') - if not isinstance(path, str) or not re.fullmatch(r'/[A-Za-z0-9._~/-]+', path) or any(part in ('', '.', '..') for part in path[1:].split('/')): - raise ValueError('A canonical absolute readiness path is required') - if not isinstance(command, list) or not command or not all(isinstance(arg, str) and '\0' not in arg for arg in command) or not os.path.isabs(command[0]): - raise ValueError('A fixed absolute executable and argv are required') - if isinstance(timeout, bool) or not 0 < timeout <= 3 or isinstance(slots, bool) or not isinstance(slots, int) or not 1 <= slots <= 16: - raise ValueError('Invalid probe bounds') - if not isinstance(single_flight, bool): - raise ValueError('single_flight must be a boolean') - if isinstance(refresh_interval, bool) or not isinstance(refresh_interval, (int, float)) or not 0 <= refresh_interval <= 5: - raise ValueError('Invalid refresh interval') - if isinstance(max_age, bool) or not isinstance(max_age, (int, float)) or not 0 < max_age <= 10 or (refresh_interval and max_age < refresh_interval + timeout): - raise ValueError('Invalid readiness freshness bound') - self.token = token.encode('ascii') - self.ready_path = path - self.command = command - self.timeout = timeout - self.single_flight = single_flight - self.flight = None - self.probe_slots = threading.BoundedSemaphore(slots) - self.connection_slots = threading.BoundedSemaphore(slots * 2 + 4) - self.active = set() - self.active_lock = threading.Lock() - self.closing = False - self.refresh_interval = refresh_interval - self.max_age = max_age - self.snapshot = None - self.refresh_stop = threading.Event() - self.refresh_thread = None - super().__init__(address, Handler) - if refresh_interval: - self.refresh_thread = threading.Thread(target=self.refresh, daemon=True) - self.refresh_thread.start() - - def get_request(self): - connection, address = super().get_request() - connection.settimeout(2) - return connection, address - - def process_request(self, request, address): - if not self.connection_slots.acquire(blocking=False): - try: - request.sendall(b'HTTP/1.1 503 Service Unavailable\r\nCache-Control: no-store\r\nContent-Length: 0\r\nConnection: close\r\n\r\n') - except OSError: - pass - finally: - self.shutdown_request(request) - return - try: - super().process_request(request, address) - except BaseException: - self.connection_slots.release() - raise - - def process_request_thread(self, request, address): - try: - super().process_request_thread(request, address) - finally: - self.connection_slots.release() - - def handle_error(self, request, address): - pass - - @staticmethod - def stop_probe(process): - try: - os.killpg(process.pid, signal.SIGKILL) - except ProcessLookupError: - pass - process.wait() - - def probe(self): - if self.refresh_interval: - with self.active_lock: - return bool(not self.closing and self.snapshot and self.snapshot['ready'] - and 0 <= time.monotonic() - self.snapshot['started'] < self.max_age) - if not self.single_flight: - return self.run_probe() - with self.active_lock: - if self.closing: - return False - flight = self.flight - leader = flight is None - if leader: - flight = {'deadline': time.monotonic() + self.timeout, 'done': threading.Event(), 'ready': False} - self.flight = flight - if not leader: - flight['done'].wait(max(0, flight['deadline'] - time.monotonic())) - with self.active_lock: - return flight['ready'] and not self.closing and time.monotonic() < flight['deadline'] - ready = False - try: - ready = self.run_probe(flight['deadline']) - finally: - with self.active_lock: - flight['ready'] = ready and not self.closing and time.monotonic() < flight['deadline'] - self.flight = None - flight['done'].set() - return flight['ready'] - - def refresh(self): - while not self.refresh_stop.is_set(): - started = time.monotonic() - try: - ready = self.run_probe(started + self.timeout) - except Exception: - ready = False - with self.active_lock: - self.snapshot = {'started': started, 'ready': ready and not self.closing and time.monotonic() < started + self.timeout} - if self.refresh_stop.wait(max(0, started + self.refresh_interval - time.monotonic())): - break - - def run_probe(self, deadline=None): - if not self.probe_slots.acquire(blocking=False): - return False - process = None - try: - with self.active_lock: - if self.closing: - return False - process = subprocess.Popen(self.command, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, start_new_session=True, close_fds=True) - self.active.add(process) - try: - timeout = self.timeout if deadline is None else max(0, deadline - time.monotonic()) - return process.wait(timeout=timeout) == 0 - except subprocess.TimeoutExpired: - self.stop_probe(process) - return False - except (OSError, ValueError): - return False - finally: - if process is not None: - with self.active_lock: - self.active.discard(process) - self.probe_slots.release() - - def server_close(self): - self.refresh_stop.set() - with self.active_lock: - self.closing = True - if self.flight is not None: - self.flight['ready'] = False - self.flight['done'].set() - active = list(self.active) - for process in active: - self.stop_probe(process) - if self.refresh_thread is not None: - self.refresh_thread.join(self.timeout + 1) - super().server_close() - - -class Handler(http.server.BaseHTTPRequestHandler): - server_version = 'APEX' - sys_version = '' - - def log_message(self, *args): - pass - - def send_error(self, code, message=None, explain=None): - self.respond(code) - - def respond(self, status): - if self.request_version == 'HTTP/0.9': - self.request_version = 'HTTP/1.0' - self.send_response(status) - self.send_header('Cache-Control', 'no-store') - self.send_header('Content-Length', '0') - self.send_header('Connection', 'close') - self.end_headers() - self.close_connection = True - - def dispatch(self): - target = self.requestline.split()[1] - if target != self.server.ready_path or self.path != self.server.ready_path: - return self.respond(404) - if self.command != 'GET': - return self.respond(405) - headers = self.headers.get_all('X-APEX-Ready-Token', []) - supplied = headers[0].encode('latin-1') if len(headers) == 1 else b'' - if len(headers) != 1 or not hmac.compare_digest(supplied, self.server.token): - return self.respond(403) - return self.respond(204 if self.server.probe() else 503) - - do_GET = dispatch - do_HEAD = dispatch - do_POST = dispatch - do_PUT = dispatch - do_DELETE = dispatch - do_OPTIONS = dispatch - do_PATCH = dispatch - - -def main(): - parser = argparse.ArgumentParser() - parser.add_argument('--config', required=True) - args = parser.parse_args() - config = json.loads(pathlib.Path(args.config).read_text()) - token = pathlib.Path(config.pop('token_file')).read_text().strip() - address = (config.pop('host', '127.0.0.1'), config.pop('port', 9092)) - server = ReadinessServer(address, token=token, **config) - def terminate(signum, frame): - raise SystemExit(0) - signal.signal(signal.SIGTERM, terminate) - try: - server.serve_forever() - finally: - server.server_close() - - -if __name__ == '__main__': - main() diff --git a/assets/composition-readiness/test_adapter.py b/assets/composition-readiness/test_adapter.py deleted file mode 100644 index 9c36c4d..0000000 --- a/assets/composition-readiness/test_adapter.py +++ /dev/null @@ -1,182 +0,0 @@ -import concurrent.futures,http.client,pathlib,socket,sys,tempfile,threading,time,unittest -from adapter import ReadinessServer -class AdapterTests(unittest.TestCase): - def setUp(self): - self.tmp=tempfile.TemporaryDirectory();self.root=pathlib.Path(self.tmp.name);self.marker=self.root/'ran';self.servers=[] - def tearDown(self): - for s,t in self.servers:s.shutdown();s.server_close();t.join() - self.tmp.cleanup() - def server(self,code=None,timeout=.3,slots=2,**options): - code=code or f"from pathlib import Path;Path({str(self.marker)!r}).write_text('yes')" - server=ReadinessServer(('127.0.0.1',0),token='private-test-token',path='/.well-known/apex-ready',command=[sys.executable,'-c',code],timeout=timeout,slots=slots,**options) - t=threading.Thread(target=server.serve_forever,kwargs={'poll_interval':.01},daemon=True);t.start();self.servers.append((server,t));return server - def request(self,s,path='/.well-known/apex-ready',token='private-test-token',method='GET',extra_token=False): - c=http.client.HTTPConnection(*s.server_address,timeout=3);c.putrequest(method,path) - if token is not None:c.putheader('X-APEX-Ready-Token',token) - if extra_token:c.putheader('X-APEX-Ready-Token',token) - c.endheaders();r=c.getresponse();out=(r.status,r.getheader('Cache-Control'),r.read());c.close();return out - def test_authenticated_success_runs_fixed_probe(self): - self.assertEqual(self.request(self.server()),(204,'no-store',b''));self.assertTrue(self.marker.exists()) - def test_configured_three_seconds_allows_a_probe_longer_than_default(self): - code=f"import time;from pathlib import Path;time.sleep(1.6);Path({str(self.marker)!r}).write_text('completed')" - s=self.server(code,timeout=3,single_flight=True) - self.assertEqual(self.request(s),(204,'no-store',b''));self.assertEqual(self.marker.read_text(),'completed') - def test_timeout_over_three_seconds_is_rejected_before_probe(self): - with self.assertRaises(ValueError):self.server(timeout=3.01) - self.assertFalse(self.marker.exists()) - def test_absent_wrong_and_duplicate_tokens_never_probe(self): - s=self.server() - for token,duplicate in [(None,False),('bad',False),('private-test-token',True)]:self.assertEqual(self.request(s,token=token,extra_token=duplicate)[0],403) - self.assertFalse(self.marker.exists()) - def test_noncanonical_paths_and_methods_never_probe(self): - s=self.server() - for path in ['/other','/.well-known/apex-ready?x=1','/.well-known/%61pex-ready','//.well-known/apex-ready']: - self.assertEqual(self.request(s,path=path)[0],404) - self.assertEqual(self.request(s,method='POST')[0],405);self.assertFalse(self.marker.exists()) - def test_nonzero_exit_and_missing_executable_fail_closed(self): - s=self.server('raise SystemExit(1)');self.assertEqual(self.request(s),(503,'no-store',b''));s.command=['/nonexistent/apex-probe'];self.assertEqual(self.request(s)[0],503) - def test_timeout_terminates_probe_and_descendant(self): - code=f"import subprocess,time,sys;subprocess.Popen([sys.executable,'-c',\"import time,pathlib;time.sleep(.8);pathlib.Path({str(self.marker)!r}).write_text('leaked')\"]);time.sleep(3)" - s=self.server(code,timeout=.1);start=time.monotonic();self.assertEqual(self.request(s)[0],503);self.assertLess(time.monotonic()-start,.7);time.sleep(.9);self.assertFalse(self.marker.exists()) - def test_concurrency_limit_rejects_before_spawn(self): - code=f"from pathlib import Path;import time;Path({str(self.marker)!r}).write_text('yes');time.sleep(.4)" - s=self.server(code,timeout=1,slots=1) - with concurrent.futures.ThreadPoolExecutor() as pool: - first=pool.submit(self.request,s) - for _ in range(100): - if self.marker.exists():break - time.sleep(.005) - self.assertTrue(self.marker.exists());self.assertEqual(self.request(s)[0],503);self.assertEqual(first.result()[0],204) - def test_unsupported_method_is_empty_and_not_cached(self): - self.assertEqual(self.request(self.server(),method='TRACE'),(501,'no-store',b''));self.assertFalse(self.marker.exists()) - def test_parser_error_is_empty_and_not_cached(self): - s=self.server() - with socket.create_connection(s.server_address,timeout=2) as sock: - sock.sendall(b'GET /.well-known/apex-ready HTTP/banana\r\n\r\n') - r=http.client.HTTPResponse(sock);r.begin();self.assertEqual((r.status,r.getheader('Cache-Control'),r.read()),(400,'no-store',b'')) - self.assertFalse(self.marker.exists()) - def test_no_cached_readiness(self): - s=self.server();self.assertEqual(self.request(s)[0],204);s.command=[sys.executable,'-c','raise SystemExit(1)'];self.assertEqual(self.request(s)[0],503) - def overlapping_requests(self,s,count=4): - barrier=threading.Barrier(count) - def one(): - barrier.wait(timeout=2);start=time.monotonic();result=self.request(s);return result,time.monotonic()-start - with concurrent.futures.ThreadPoolExecutor(max_workers=count) as pool:return list(pool.map(lambda _:one(),range(count))) - def test_browser_asset_burst_connects_and_shares_one_slow_probe(self): - code=f"import pathlib,time;p=pathlib.Path({str(self.marker)!r});f=p.open('a');f.write('probe\\n');f.close();time.sleep(.8)" - s=self.server(code,timeout=3,slots=16,single_flight=True) - process_request=s.process_request - def delayed_accept(request,address): - time.sleep(.01) - process_request(request,address) - s.process_request=delayed_accept - barrier=threading.Barrier(32) - def one(_): - barrier.wait(timeout=2) - c=http.client.HTTPConnection(*s.server_address,timeout=.5) - try: - c.connect();c.sock.settimeout(3) - c.request('GET','/.well-known/apex-ready',headers={'X-APEX-Ready-Token':'private-test-token'}) - r=c.getresponse();return r.status,r.read() - finally:c.close() - with concurrent.futures.ThreadPoolExecutor(max_workers=32) as pool:results=list(pool.map(one,range(32))) - self.assertEqual(results,[(204,b'')]*32) - self.assertEqual(self.marker.read_text().splitlines(),['probe']) - def test_single_flight_shares_success_and_sequential_request_reprobes(self): - code=f"import pathlib,time;p=pathlib.Path({str(self.marker)!r});f=p.open('a');f.write('probe\\n');f.close();time.sleep(.2)" - s=self.server(code,timeout=1,slots=1,single_flight=True) - results=self.overlapping_requests(s) - self.assertEqual([r for r,_ in results],[(204,'no-store',b'')]*4) - self.assertEqual(self.marker.read_text().splitlines(),['probe']) - self.assertEqual(self.request(s)[0],204) - self.assertEqual(self.marker.read_text().splitlines(),['probe','probe']) - def test_single_flight_shares_failure_without_caching_it(self): - code=f"import pathlib,time;p=pathlib.Path({str(self.marker)!r});f=p.open('a');f.write('probe\\n');f.close();time.sleep(.2);raise SystemExit(1)" - s=self.server(code,timeout=1,slots=4,single_flight=True) - self.assertEqual([r for r,_ in self.overlapping_requests(s)],[(503,'no-store',b'')]*4) - self.assertEqual(self.marker.read_text().splitlines(),['probe']) - s.command=[sys.executable,'-c','pass'];self.assertEqual(self.request(s)[0],204) - def test_single_flight_timeout_wakes_waiters_and_kills_descendant(self): - code=f"import subprocess,time,sys;subprocess.Popen([sys.executable,'-c',\"import time,pathlib;time.sleep(.8);pathlib.Path({str(self.marker)!r}).write_text('leaked')\"]);time.sleep(3)" - s=self.server(code,timeout=.25,slots=1,single_flight=True) - results=self.overlapping_requests(s) - self.assertEqual([r for r,_ in results],[(503,'no-store',b'')]*4) - self.assertTrue(all(.15 (routing profile, unchanged behaviour) - line N : a share link for every config the user has - -Design ------- -The config is written 1:1 against the HAPP / Xray share-link format. A template -is just the literal pieces of a URI: - - scheme -> vless:// , hy2:// , trojan:// , ss:// , socks:// , vmess:// - host -> the @host part - port -> the :port part - id -> the userinfo before @ (uuid / password / auth) - params -> the ?a=b&c=d query, keys are the REAL link keys - (type, security, sni, fp, alpn, flow, encryption, - pbk, sid, spx, mode, path, host, serviceName, - headerType, mux, obfs, ... whatever the link supports) - -Nothing is translated or guessed. Whatever you put in `params` is what goes in -the link, url-encoded. This keeps the generator dumb and the config honest. - -Per-user `configs` entries deep-merge onto the named template, so a user only -supplies the bits that differ (their `id`, their `params.sid`, etc.). - -The `#remark` is built from template metadata (country / nodes) + the user name. -""" - -import json -import base64 -import sys -import os -import copy -from urllib.parse import urlencode, quote - -# ISO 3166-1 alpha-2 -> (flag, display name). Extend as needed. -COUNTRY_MAP = { - "NL": ("\U0001F1F3\U0001F1F1", "Netherlands"), - "LV": ("\U0001F1F1\U0001F1FB", "Latvia"), - "RU": ("\U0001F1F7\U0001F1FA", "Russia"), - "KZ": ("\U0001F1F0\U0001F1FF", "Kazakhstan"), - "BY": ("\U0001F1E7\U0001F1FE", "Belarus"), -} - - -def deep_merge(base, override): - """Recursively merge `override` into a copy of `base`. Lists/scalars replace.""" - out = copy.deepcopy(base) - for key, value in override.items(): - if isinstance(out.get(key), dict) and isinstance(value, dict): - out[key] = deep_merge(out[key], value) - else: - out[key] = copy.deepcopy(value) - return out - - -def routing_link(routing): - """happ://routing/onadd/""" - payload = json.dumps(routing, separators=(",", ":")).encode() - return "happ://routing/onadd/" + base64.b64encode(payload).decode() - - -def build_remark(node, user_name): - code = node.get("country", "") - flag, name = COUNTRY_MAP.get(code, ("\U0001F310", code or "??")) - transport = ("hysteria" if node.get("scheme") in ("hysteria2", "hy2") - else node.get("params", {}).get("type", "tcp")) - chain = " > ".join(node.get("nodes", ["unknown"])) - return f"{flag} {name} [{transport}] | {chain} | {user_name}" - - -def build_link(node, user_name): - """Assemble one share link from a fully-merged node dict.""" - scheme = node["scheme"] - host = node["host"] - port = node["port"] - userinfo = node.get("id", "") - - # Drop empty params so the query stays clean; preserve insertion order. - params = {k: v for k, v in node.get("params", {}).items() - if v is not None and v != ""} - - # Lists (e.g. alpn) -> comma-joined; everything else stringified. - flat = {} - for k, v in params.items(): - flat[k] = ",".join(map(str, v)) if isinstance(v, list) else str(v) - - query = urlencode(flat, safe="/:") - remark = quote(build_remark(node, user_name)) - - base = f"{scheme}://{userinfo}@{host}:{port}" - return f"{base}?{query}#{remark}" if query else f"{base}#{remark}" - - -def main(): - if len(sys.argv) < 3: - print("Usage: generator.py ") - sys.exit(1) - - input_path, output_dir = sys.argv[1], sys.argv[2] - - with open(input_path, "r", encoding="utf-8") as f: - data = json.load(f) - - routing = routing_link(data.get("routing", {})) - templates = data.get("templates", {}) - users = data.get("users", []) - - os.makedirs(output_dir, exist_ok=True) - - for user in users: - name = user["name"] - out_path = os.path.join(output_dir, user.get("psub", name)) - - lines = [routing] - for entry in user.get("configs", []): - tmpl = entry.get("template") - node = deep_merge(templates.get(tmpl, {}), entry) if tmpl else entry - lines.append(build_link(node, name)) - - with open(out_path, "w", encoding="utf-8") as f: - f.write("\n".join(lines) + "\n") - print(f"Generated: {out_path}") - - -if __name__ == "__main__": - main() diff --git a/ci/cloud-init/user-data.template.yaml b/ci/cloud-init/user-data.template.yaml index ca2c3e4..faf1baf 100644 --- a/ci/cloud-init/user-data.template.yaml +++ b/ci/cloud-init/user-data.template.yaml @@ -1,11 +1,11 @@ #cloud-config -# Use this same file for cloud-init or: ./apex init root@HOST --user-data FILE +# Use this same file for cloud-init or: python3 ci/scripts/init-node.py root@HOST --user-data FILE # Replace the hostname, repository and public key. Keep filled files outside Git. # SSH repositories additionally require approved deferred id_ed25519 (0600) # and known_hosts (0644) writes under /home/adam/.ssh, owned by adam:adam. # Supply trusted Git host keys; do not infer trust from ssh-keyscan alone. preserve_hostname: false -hostname: metis.a1.apex.ermnvldmr.com +hostname: node1.example.com users: - name: adam sudo: ALL=(ALL) NOPASSWD:ALL @@ -18,8 +18,8 @@ write_files: owner: root:root permissions: '0600' content: | - APEX_REPO_NAME="metis.apex.ermnvldmr.com" - APEX_REPO_URL="https://github.com/deytenit/metis.apex.ermnvldmr.com.git" + APEX_REPO_NAME="node1.example.com" + APEX_REPO_URL="https://github.com/you/node1.example.com.git" APEX_TIER1_MODE="directory" APEX_TIER1_DEVICE="" APEX_TIER2_MODE="directory" diff --git a/ci/scripts/init-node.py b/ci/scripts/init-node.py new file mode 100755 index 0000000..0eb6b53 --- /dev/null +++ b/ci/scripts/init-node.py @@ -0,0 +1,64 @@ +#!/usr/bin/env python3 +"""Initialize a plain Debian target server via SSH enrollment. + +Standalone tool for hosts on providers that do not support cloud-init. +Kept separate from the core local-only apex engine. +""" +import argparse +import os +import sys + +_here = os.path.dirname(os.path.abspath(__file__)) +_repo_root = os.path.dirname(os.path.dirname(_here)) +_prov_dir = os.path.join(_here, "init-node") +sys.path.insert(0, _prov_dir) + +from provisioning.transport import initialize + + +def main(argv=None): + parser = argparse.ArgumentParser( + description="Initialize a plain Debian server through protected SSH enrollment.", + formatter_class=argparse.ArgumentDefaultsHelpFormatter, + ) + parser.add_argument( + "target", + help="Bootstrap USER@HOST (root or an existing administrator with passwordless sudo)", + ) + parser.add_argument( + "--user-data", + dest="user_data", + help="Path to APEX cloud-config file (e.g., ci/cloud-init/user-data.template.yaml)", + ) + parser.add_argument( + "--port", + dest="port", + help="Bootstrap SSH port; otherwise uses standard SSH configuration", + ) + parser.add_argument( + "--identity-file", + dest="identity_file", + help="SSH private identity file; otherwise uses SSH configuration/agent", + ) + + args = parser.parse_args(argv) + + print(f"[init-node] Starting enrollment for {args.target}...") + try: + initialize( + _repo_root, + args.target, + args.user_data, + args.port, + args.identity_file, + progress=lambda msg: print(f"[init-node] {msg}"), + ) + print(f"[init-node] Successfully initialized {args.target}.") + return 0 + except Exception as e: + sys.stderr.write(f"[init-node] Error: {e}\n") + return 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/engine/provisioning/__init__.py b/ci/scripts/init-node/provisioning/__init__.py similarity index 100% rename from engine/provisioning/__init__.py rename to ci/scripts/init-node/provisioning/__init__.py diff --git a/engine/provisioning/access.py b/ci/scripts/init-node/provisioning/access.py similarity index 100% rename from engine/provisioning/access.py rename to ci/scripts/init-node/provisioning/access.py diff --git a/engine/provisioning/baseline.py b/ci/scripts/init-node/provisioning/baseline.py similarity index 100% rename from engine/provisioning/baseline.py rename to ci/scripts/init-node/provisioning/baseline.py diff --git a/engine/provisioning/contract.py b/ci/scripts/init-node/provisioning/contract.py similarity index 100% rename from engine/provisioning/contract.py rename to ci/scripts/init-node/provisioning/contract.py diff --git a/engine/provisioning/image.py b/ci/scripts/init-node/provisioning/image.py similarity index 100% rename from engine/provisioning/image.py rename to ci/scripts/init-node/provisioning/image.py diff --git a/engine/provisioning/repository.py b/ci/scripts/init-node/provisioning/repository.py similarity index 100% rename from engine/provisioning/repository.py rename to ci/scripts/init-node/provisioning/repository.py diff --git a/engine/provisioning/security.py b/ci/scripts/init-node/provisioning/security.py similarity index 100% rename from engine/provisioning/security.py rename to ci/scripts/init-node/provisioning/security.py diff --git a/engine/provisioning/seed.sh b/ci/scripts/init-node/provisioning/seed.sh similarity index 100% rename from engine/provisioning/seed.sh rename to ci/scripts/init-node/provisioning/seed.sh diff --git a/engine/provisioning/state.py b/ci/scripts/init-node/provisioning/state.py similarity index 100% rename from engine/provisioning/state.py rename to ci/scripts/init-node/provisioning/state.py diff --git a/engine/provisioning/storage.py b/ci/scripts/init-node/provisioning/storage.py similarity index 100% rename from engine/provisioning/storage.py rename to ci/scripts/init-node/provisioning/storage.py diff --git a/engine/provisioning/target.py b/ci/scripts/init-node/provisioning/target.py similarity index 100% rename from engine/provisioning/target.py rename to ci/scripts/init-node/provisioning/target.py diff --git a/engine/provisioning/transport.py b/ci/scripts/init-node/provisioning/transport.py similarity index 100% rename from engine/provisioning/transport.py rename to ci/scripts/init-node/provisioning/transport.py diff --git a/engine/provisioning/userdata.py b/ci/scripts/init-node/provisioning/userdata.py similarity index 100% rename from engine/provisioning/userdata.py rename to ci/scripts/init-node/provisioning/userdata.py diff --git a/tests/provision_fakes.py b/ci/scripts/init-node/tests/provision_fakes.py similarity index 100% rename from tests/provision_fakes.py rename to ci/scripts/init-node/tests/provision_fakes.py diff --git a/tests/test_provision_access.py b/ci/scripts/init-node/tests/test_provision_access.py similarity index 100% rename from tests/test_provision_access.py rename to ci/scripts/init-node/tests/test_provision_access.py diff --git a/tests/test_provision_baseline.py b/ci/scripts/init-node/tests/test_provision_baseline.py similarity index 100% rename from tests/test_provision_baseline.py rename to ci/scripts/init-node/tests/test_provision_baseline.py diff --git a/tests/test_provision_contract.py b/ci/scripts/init-node/tests/test_provision_contract.py similarity index 100% rename from tests/test_provision_contract.py rename to ci/scripts/init-node/tests/test_provision_contract.py diff --git a/tests/test_provision_image.py b/ci/scripts/init-node/tests/test_provision_image.py similarity index 100% rename from tests/test_provision_image.py rename to ci/scripts/init-node/tests/test_provision_image.py diff --git a/tests/test_provision_repository.py b/ci/scripts/init-node/tests/test_provision_repository.py similarity index 100% rename from tests/test_provision_repository.py rename to ci/scripts/init-node/tests/test_provision_repository.py diff --git a/tests/test_provision_security.py b/ci/scripts/init-node/tests/test_provision_security.py similarity index 100% rename from tests/test_provision_security.py rename to ci/scripts/init-node/tests/test_provision_security.py diff --git a/tests/test_provision_security_access.py b/ci/scripts/init-node/tests/test_provision_security_access.py similarity index 100% rename from tests/test_provision_security_access.py rename to ci/scripts/init-node/tests/test_provision_security_access.py diff --git a/tests/test_provision_state.py b/ci/scripts/init-node/tests/test_provision_state.py similarity index 100% rename from tests/test_provision_state.py rename to ci/scripts/init-node/tests/test_provision_state.py diff --git a/tests/test_provision_storage.py b/ci/scripts/init-node/tests/test_provision_storage.py similarity index 100% rename from tests/test_provision_storage.py rename to ci/scripts/init-node/tests/test_provision_storage.py diff --git a/tests/test_provision_target.py b/ci/scripts/init-node/tests/test_provision_target.py similarity index 100% rename from tests/test_provision_target.py rename to ci/scripts/init-node/tests/test_provision_target.py diff --git a/tests/test_provision_transport.py b/ci/scripts/init-node/tests/test_provision_transport.py similarity index 100% rename from tests/test_provision_transport.py rename to ci/scripts/init-node/tests/test_provision_transport.py diff --git a/tests/test_provision_userdata.py b/ci/scripts/init-node/tests/test_provision_userdata.py similarity index 100% rename from tests/test_provision_userdata.py rename to ci/scripts/init-node/tests/test_provision_userdata.py diff --git a/compositions/apex/docker-compose.yml b/compositions/apex/docker-compose.yml index b0e06d0..ee3bad9 100644 --- a/compositions/apex/docker-compose.yml +++ b/compositions/apex/docker-compose.yml @@ -7,8 +7,6 @@ x-environment: &shared-environment TZ: "${TZ:-Etc/UTC}" # override per node via TZ in the env; default UTC services: - - # --- optional: adguard (shares traefik netns) --- adguard: image: adguard/adguardhome:v0.107.79 container_name: apex-adguard @@ -42,7 +40,7 @@ services: image: grafana/alloy:v1.20.0 container_name: apex-alloy volumes: - - ${APEX_COMMONS}/assets/alloy/config.alloy:/etc/alloy/config.alloy:ro + - ${APEX_COMMONS}/etc/alloy/config.alloy:/etc/alloy/config.alloy:ro - ${APEX_TIER2}/alloy/data:/var/lib/alloy - /var/run/docker.sock:/var/run/docker.sock:ro - /run/containerd/containerd.sock:/run/containerd/containerd.sock:ro # cadvisor per-container stats on nodes using the containerd snapshotter (overlayfs) @@ -105,10 +103,10 @@ services: environment: <<: *shared-environment RESTIC_HOSTFS: "/" # mount sources are already absolute (/srv/…) - RESTIC_REPOSITORY: "${APEX_RESTIC_REPOSITORY?error}" - RESTIC_PASSWORD: "${APEX_RESTIC_PASSWORD?error}" - AWS_ACCESS_KEY_ID: "${APEX_RESTIC_AWS_ACCESS_KEY_ID?error}" - AWS_SECRET_ACCESS_KEY: "${APEX_RESTIC_AWS_SECRET_ACCESS_KEY?error}" + RESTIC_REPOSITORY: "${APEX_RESTIC_REPOSITORY:-}" + RESTIC_PASSWORD: "${APEX_RESTIC_PASSWORD:-}" + AWS_ACCESS_KEY_ID: "${APEX_RESTIC_AWS_ACCESS_KEY_ID:-}" + AWS_SECRET_ACCESS_KEY: "${APEX_RESTIC_AWS_SECRET_ACCESS_KEY:-}" # S3 backend region + endpoint. Defaults suit AWS S3 (us-east-1, no custom # endpoint); set APEX_RESTIC_AWS_REGION / APEX_RESTIC_AWS_ENDPOINT (node apex.env) # for any other S3-compatible store (Yandex, Backblaze B2, MinIO, …). @@ -149,7 +147,7 @@ services: - ${APEX_TIER1}/traefik/dynamic:/etc/traefik/dynamic:ro environment: <<: *shared-environment - CF_DNS_API_TOKEN: "${APEX_TRAEFIK_CF_DNS_API_TOKEN?error}" + CF_DNS_API_TOKEN: "${APEX_TRAEFIK_CF_DNS_API_TOKEN:-}" UMASK: "0002" ports: - '0.0.0.0:80:80' @@ -182,7 +180,7 @@ services: - "--entrypoints.websecure.http3" - "--certificatesresolvers.cloudflare.acme.dnschallenge=true" - "--certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare" - - "--certificatesresolvers.cloudflare.acme.email=${APEX_TRAEFIK_ACME_EMAIL?error}" + - "--certificatesresolvers.cloudflare.acme.email=${APEX_TRAEFIK_ACME_EMAIL:-admin@localhost}" - "--certificatesresolvers.cloudflare.acme.storage=/letsencrypt/acme.json" restart: unless-stopped labels: diff --git a/docs/rfcs/composition-replication.md b/docs/rfcs/composition-replication.md deleted file mode 100644 index d30c50f..0000000 --- a/docs/rfcs/composition-replication.md +++ /dev/null @@ -1,170 +0,0 @@ -# RFC Reusable composition replication patterns - -Status: Accepted patterns; SSO revised to two locations with controlled promotion. The rollout remains pending migration, routing, and failure verification; this document does not claim achieved high availability. - -## Purpose and decision - -APEX should make it repeatable to deploy another instance of an individual composition on another node and verify its declared failure behavior. Nodes and repositories always remain distinct. Applications are mostly unrelated, and duplicating configuration or running dedicated supporting services is acceptable when it preserves understandable ownership. - -The unit of replication is a logical application deployment, represented by separately owned Compose projects on participating nodes. Replication does not mean cloning a node, synchronizing repositories, sharing every composition, or applying one storage mechanism to every application. - -Adopt a small catalog of replication patterns, a common deployment contract, and a common verification workflow. Use application and database native mechanisms for data replication and leader election. Shared backend clusters are optional. SSO is the first worked example, not the architecture imposed on later applications. - -## Existing foundations - -APEX is a Python standard-library engine with node-local actions and handwritten Compose files. Each node repository pins commons and owns its compositions, configuration, and proprietary actions. Keep these characteristics. - -`node.env` declares node identity and a node-local `APEX_SUBNET`. The core composition creates a Docker bridge on that subnet. The checked-in fleet convention assigns separate /24 networks to nodes within a larger cluster address block. Existing firewall and routing configuration includes Xray transparent-proxy rules; this is not evidence that arbitrary replication protocols already work across every node. - -`apex compose` currently orchestrates local projects, with the core first. The action overlay provides a natural extension point for application-specific operations. Tier storage separates runtime state and secrets from tracked Compose definitions. - -## Ownership and terminology - -A **node** is an independently managed server with its own repository, identity, addresses, storage, credentials, and lifecycle. - -A **composition instance** is a local Compose project. Its repository owns its complete definition, even when another repository owns a similar project. - -A **replication group** identifies composition instances providing the same logical application. One node can participate in several unrelated groups, each with different members, roles, backend choices, and availability guarantees. A Compose project name alone does not establish group membership. - -A **replication domain** identifies the permitted network scope for a group. The user referred to this scope as Na. This RFC treats it as a set of mutually reachable participating nodes rather than requiring a shared layer-2 network. The label is independent of the engine's existing node-local `APEX_SUBNET`; no address allocations or cross-cluster membership are implied. - -A **failure domain** identifies what can disappear together, such as a physical site. Distinct node names or subnet labels are not sufficient evidence of independent failures. - -A **backend group** provides a data service to one or more application groups. It has its own members and failure contract. It can be dedicated to one application or deliberately shared by compatible applications. - -## Supported patterns - -| Pattern | Suitable application | Replication and failover responsibility | -| --- | --- | --- | -| Stateless copies | Instances can serve concurrently without unique mutable local state | Deploy equivalent configuration; route to ready instances | -| Copies using replicated backends | Application supports concurrent instances and external data services | Application instances share logical state through compatible replicated backends; backends own consistency and failover | -| Application-native cluster | Application provides supported membership, replication, and recovery | Follow its native topology and lifecycle; expose readiness and recovery through APEX conventions | -| Single active instance with standby | Application requires exclusive ownership of mutable state | Use a supported consistent state-transfer method and proven fencing before automatic promotion | -| Recovery-only deployment | Application has no validated safe live-replication arrangement | Reproducible restore and explicit activation; report that automatic availability is unsupported | - -Every service can be assessed through this catalog. Not every service can safely provide concurrent active copies. Supporting an arbitrary service means allowing an appropriate adapter or documented recovery pattern, not promising automatic active-active operation for unmodified software. - -A composition can combine patterns: stateless HTTP instances, a replicated SQL backend, and a singleton scheduled worker. Starting duplicate workers or schedulers is a separate correctness decision from starting duplicate web containers. - -Do not synchronize live database directories as a generic replication mechanism. Applications using SQLite require an application-supported solution or exclusive-writer failover; placing their database on a network filesystem is not a universal fix. [SQLite network storage guidance](https://www.sqlite.org/useovernet.html) - -## Required deployment contract - -Each participating repository must record the following non-secret information alongside the composition using the project's established configuration and documentation conventions. This RFC does not introduce a new manifest parser or require generated Compose files. - -| Contract element | Required content | -| --- | --- | -| Identity | Logical group name, local instance identity, replication domain, failure domain, topology revision | -| Members | Explicit peer identities, stable reachable endpoints, and desired membership roles | -| Pattern | Selected replication pattern and supported application/backend versions | -| State inventory | Every persistent volume, uploaded file store, session store, user directory, queue, and mutable configuration source | -| State ownership | Replication method, write authority, consistency limits, and recovery procedure for each state class | -| Dependencies | Named backend groups, access requirements, and the failure assumptions of each dependency | -| Configuration | Required common settings, intentionally local settings, secret references, and update procedure | -| Traffic | Public and internal client endpoints, readiness conditions, routing and withdrawal behavior | -| Lifecycle | Initial creation, joining, draining, upgrade order, departure, rejoin, and full-outage recovery | -| Guarantees | Maximum recovery time target, acknowledged-data-loss policy, session behavior, and supported failure scenarios | - -Desired roles must not overwrite runtime decisions. For example, a replica promoted by Sentinel must not be forcibly returned to an obsolete role on every ordinary restart. - -Repositories can adopt application changes independently only within the documented version-compatibility window. Changes to group membership, shared schemas, or wire protocols require coordinated sequencing. Planned membership changes must validate agreement among the participating reachable nodes against the native system's authoritative membership. Removing a failed member follows the native recovery procedure and must not require that failed member to acknowledge the change. - -Keep private keys, credentials, password hashes, and application encryption secrets outside tracked configuration. State explicitly which application secrets must match across replicas; node identity and node-specific credentials remain distinct. - -## APEX responsibilities - -APEX should provide reusable pattern documentation, preflight checks, composition-scoped lifecycle entry points, and a consistent way to inspect readiness and replication health. A future implementation should use the existing action and overlay model. Exact new command names and machine-readable schemas belong to the implementation design after this RFC is accepted. - -Actions operate on explicitly selected local compositions. Optional SSH invocation transports the same operations to named nodes; it does not create a fleet-wide desired-state controller. APEX must not make automatic quorum decisions or promote database primaries based on its own ping checks. - -Checks must distinguish running containers from a service that can safely serve traffic. They must report incompatible versions, unreachable required peers, mismatched membership, missing configuration, unsafe state-sharing arrangements, and absent capacity for the declared failure target. - -Shared helpers may cover common tasks. Native backend tools retain ownership of elections, replication, fencing, and recovery. Application-specific behavior stays in a bounded adapter or runbook rather than an expanding set of SSO conditionals in the engine. - -## Networking and routing - -Reuse an existing private transport when tests demonstrate that it supports the application's actual protocols. Validate advertised addresses, bidirectional connectivity, required TCP and UDP ports, reconnect behavior, and relevant MTU constraints. Do not silently replace the fleet's routing with a new overlay. - -Validate the peer identities actually learned after reconnect, not just the configured addresses or successful initial connections. Some protocols derive membership from the incoming TCP source address: Galera combines that address with the advertised port. A TCP proxy that replaces multiple peer sources with a gateway address does not preserve this contract. Such groups need a transport that preserves distinct peer source addresses. [Galera handshake implementation](https://github.com/codership/galera/blob/release_26.4.27/gcomm/src/gmcast_proto.cpp#L222) - -The current fleet rollout uses the existing Xray TPROXY tunnels only. WireGuard is excluded. Restrict peer addresses and ports, reject fallback through unrelated routes when a peer is unreachable, and verify reconnect and container-network replacement behavior. A protocol incompatible with this transport requires a different application replication pattern, not an unapproved network overlay. - -Preserve distinct node-local bridge networks and addresses. Docker service names and host-local container IP assumptions must not be mistaken for cross-host service discovery. Peer endpoints must be explicitly reachable from the consumers that use them. - -A logical application endpoint routes only to ready instances. Dependency failure and loss of write authority must affect readiness. Both browser-facing endpoints and internal application calls must survive the same location-loss scenario. A single reverse proxy on one participating site must not be the only traffic entry point. - -Plain round-robin DNS distributes addresses but does not supply application health checks. DNS-based failover also needs its caching behavior included in the measured recovery time. [DNS load balancing](https://www.cloudflare.com/learning/performance/what-is-dns-load-balancing/) - -Automatic failover must keep working when the operator laptop and APEX CLI are absent. Its runtime mechanisms belong to the deployed services and traffic infrastructure. - -## Shared and dedicated backends - -Allow both arrangements. Prefer dedicated resources when versions, retention, durability, eviction policy, performance, or trust boundaries differ. Reuse an existing compatible cluster when doing so materially simplifies operations and its shared failure impact is accepted. - -Several applications can use separate databases and credentials within one MariaDB cluster; each does not automatically require a new Galera installation. Applications requiring PostgreSQL need an appropriate PostgreSQL solution instead. Shared database credentials are not an acceptable shortcut. - -Caches, sessions, and durable queues must not be combined solely because all speak the Redis protocol. Separate Valkey replication groups remain valid, and Sentinel can monitor multiple named groups. A Sentinel voting set is not a universal witness for unrelated database technologies. [Valkey Sentinel](https://valkey.io/topics/sentinel/) - -Application placement and backend placement must satisfy the whole dependency graph. Two application copies are not location-independent when both require an unreplicated database on one of those locations. - -## Lifecycle and failure behavior - -Initial creation and joining are different operations. Joining must verify the group identity and synchronize from the authorized source. Existing non-empty state is not overwritten implicitly. Formatting storage, discarding divergent state, and forced cluster recovery are explicit operations. - -Ordinary deployment must not bootstrap a new database cluster when peers are unreachable. A returning member catches up and passes readiness checks before it receives traffic. Repeated operations should converge or stop with an actionable diagnosis rather than silently forming a second cluster. - -Upgrade one failure domain at a time where the native system supports it. Verify redundancy and capacity before removing a member. Schema migration ownership and compatible mixed-version operation must be documented per application; reverting a container image does not automatically reverse a database migration. - -Automatic promotion of an exclusive writer requires a native election mechanism or fencing that prevents the previous writer from continuing. If this prerequisite is absent, classify promotion as manual rather than claiming safe automatic failover. - -Backups remain independent of replication and cover accidental deletion, corruption, and total group loss. Restoring a backup must not introduce an additional active writer into an existing group without its join procedure. - -## SSO first application - -Daedalus and Metis are separate repositories and application locations. The selected SSO pattern is one active deployment and one standby, with controlled manual promotion. Icarus is not a member of the target SSO topology. Other compositions on these nodes do not become replicas by association. - -The target uses native MariaDB primary/replica replication and a primary/replica session store over the existing Xray tunnels. It excludes Galera, its arbitrator, Sentinel elections, and custom automatic promotion controllers. The standby must not accept application writes until an operator has fenced the previous writer and explicitly promoted the replacement. An unreachable SSH or tunnel endpoint alone does not prove fencing. Native replication status and the last applied position must be checked and any possible data loss recorded before promotion. - -The existing public users source is a manually maintained YAML file. The selected replacement is LLDAP backed by MariaDB, with its enrolled directory preserved during the change from the staged Galera deployment to ordinary replication. Existing account identifiers, profile attributes and application groups must survive migration. The operator has already completed the one-time password enrollment; changing replication must not require another enrollment or overwrite the directory. Keep the old public SSO authoritative until migration checks pass. [Authelia LLDAP integration](https://www.authelia.com/integration/ldap/lldap/) - -The two identity locations use `198.18.16.150` on Daedalus and `198.18.18.150` on Metis. The existing SSO frontend addresses remain separate. These are explicit allocations for this deployment, not reserved addresses or mandatory topology for other APEX applications. The staged Icarus voting services must be retired only after verifying that the preserved data and new replication no longer depend on them; unrelated Icarus applications remain outside this change. - -Stage the new backend beside the existing public SSO database. Preserve a verified backup and use a consistent logical export for migration. Ordinary container restart must never initialize a replacement database, replay a seed snapshot, or promote a replica. Test direct Daedalus–Metis Xray connectivity without Icarus as an intermediary. Returning former primaries remain fenced until explicitly resynchronized as replicas. - -Asynchronous database and session replication can lose acknowledged recent changes. This pattern does not promise seamless preservation of every account change, session, or revocation. The promotion procedure must document the replication position, possible loss and whether users need to log in again. Session invalidation after promotion must be tested so that stale replicated sessions do not silently restore revoked access. - -The earlier automatic recovery target of 60 seconds is superseded by controlled promotion. Loss of the active stateful deployment can interrupt authentication until promotion completes; loss of the standby must not stop the active deployment. Record the measured promotion and recovery time rather than claiming an automatic recovery bound. - -Cloudflare traffic failover is requested for `sso.ermnvldmr.com`, `jef.ermnvldmr.com`, `jpl.ermnvldmr.com`, `sea.ermnvldmr.com`, and `imh.ermnvldmr.com`, using the no-additional-cost Worker approach. Traffic selection must not promote databases or treat an unpromoted standby as ready. Each hostname needs independently verified origin routing and application readiness. Two ingress paths to a single application backend provide ingress redundancy only; they do not replicate that backend or its data. Preserve existing streaming, WebSocket, authentication, and path-specific routes. - -## Adding the next application - -1. Select the compositions and target nodes; keep node identities and unrelated projects untouched. -2. Inventory mutable state, background jobs, dependencies, and application-supported concurrency. -3. Select a pattern. Reuse compatible infrastructure or provision dedicated supporting groups deliberately. -4. Add the local composition and deployment contract to each participating repository. Provide configuration, secrets, and reachable endpoints. -5. Run preflight checks, seed or join through the native mechanism, and confirm synchronization. -6. Add the ready instance to traffic routing. -7. Execute the failure suite and record the measured guarantee before describing the application as highly available. - -A stateless dashboard should follow these steps without database machinery. A SQL-backed application may reuse a cluster. A SQLite-backed application may need a standby arrangement or remain recovery-only. This range is intentional for a fleet of unrelated applications. - -## Acceptance and rollout - -Validate the framework against at least a stateless composition and the SSO stateful example before generalizing helpers further. Do not migrate unrelated existing applications as part of the first implementation. - -For each claimed automatic-availability profile, test loss of every location individually, process failure, network partition, stale/rejoining members, and a rolling update. Observe end-user operations and dependency health, not merely container exit codes. Test without the operator laptop participating. - -Record recovery duration, acknowledged data lost, session behavior, whether conflicting writers existed, and whether manual intervention was needed. Full-group loss and backup restoration have a separate recovery test and do not inherit the single-location availability promise. - -A failed prerequisite must produce an honest unsupported or degraded result. The word replicated alone must not imply the availability target has been verified. - -## Relationship to remote initialization - -Remote initialization remains a separate companion design: a local `apex init` would prepare a Debian server using the same versioned baseline and supported enrollment inputs as the golden-image/cloud-init path. It ends before application state migration or replication-group creation. Replication recipes must not format disks or initialize clusters as a side effect of preparing a host. - -## Alternatives considered - -Whole-node/repository mirroring violates the ownership model and couples unrelated applications. A mandatory shared database platform couples applications that may need different engines or policies. A universal volume-sync flag cannot supply arbitrary application consistency. A fleet scheduler is a possible future infrastructure choice, but introducing one is unnecessary for this RFC and would not itself solve application data replication. - -The proposed pattern catalog preserves independent repositories, permits useful redundancy, and makes later composition replicas follow a repeatable, testable process. diff --git a/engine/cli.py b/engine/cli.py index 71b9643..403e373 100644 --- a/engine/cli.py +++ b/engine/cli.py @@ -12,17 +12,19 @@ from engine.descriptor import build_parser, Meta def _resolve_layout(): - commons_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) + commons_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) parent = os.path.dirname(commons_dir) - is_node = (os.path.basename(commons_dir) == "commons" - and os.path.isfile(os.path.join(parent, "node.env"))) - if is_node: + cwd = os.getcwd() + + if os.path.isfile(os.path.join(parent, "node.env")): repo_root = parent prop_actions = os.path.join(repo_root, "proprietaries", "actions") - else: - repo_root = commons_dir # standalone commons repo is its own root - prop_actions = None - return commons_dir, repo_root, prop_actions, is_node + return commons_dir, repo_root, prop_actions, True + if os.path.isfile(os.path.join(cwd, "node.env")): + repo_root = cwd + prop_actions = os.path.join(repo_root, "proprietaries", "actions") + return commons_dir, repo_root, prop_actions, True + return commons_dir, commons_dir, None, False def _print_list(commons_actions, prop_actions): @@ -47,7 +49,7 @@ def main(argv) -> int: sys.stderr.write(f"[global] [core] [ERROR] Unknown action: {name}\n") return 1 - node, warns = (resolve(socket.getfqdn(), repo_root) if is_node and name != 'init' else (None, [])) + node, warns = (resolve(socket.getfqdn(), repo_root) if is_node else (None, [])) ctx = Context(node, Paths(repo_root, commons_dir), name, shadowed=r.shadowed) for w in warns: ctx.log.warn(w) diff --git a/engine/lib/crowdsec.py b/engine/lib/crowdsec.py index ab0f4b7..46eec39 100644 --- a/engine/lib/crowdsec.py +++ b/engine/lib/crowdsec.py @@ -13,7 +13,7 @@ class Crowdsec: if shutil.which("cscli"): return self.log.info("Installing CrowdSec via official script...") - self.sys.run(["bash", "-c", "set -o pipefail; curl -s https://install.crowdsec.net | sudo sh"], + self.sys.run(["bash", "-c", "set -o pipefail; curl -sSfL https://install.crowdsec.net | sudo sh"], check=True) self.sys.sudo(["apt-get", "install", "-y", "crowdsec"]) diff --git a/engine/lib/host.py b/engine/lib/host.py index c5341bb..6286098 100644 --- a/engine/lib/host.py +++ b/engine/lib/host.py @@ -5,6 +5,7 @@ Ports root_template_crontab (install + rollback) and the ufw block-injection log (root_template.sh _root_template_inject_rules_to_ufw_file), generalized + self-healing. """ from __future__ import annotations + import os import shutil import subprocess @@ -51,7 +52,7 @@ def apply_block(text: str, content: str, start: str, end: str, if not inserted and anchor in ln: result.extend(block) inserted = True - return "\n".join(result) + "\n", True + return "\n".join(result) + "\n", inserted class Host: @@ -64,7 +65,7 @@ class Host: if sudo: cp = self.sys.run(["sudo", "cat", path], check=True, capture=True) return cp.stdout or "" - with open(path) as f: + with open(path, encoding="utf-8") as f: return f.read() def backup(self, path: str, sudo: bool = False) -> Optional[str]: @@ -80,27 +81,29 @@ class Host: self.log.info(f"Backed up {path} -> {bkp}") return bkp - def write_file(self, path: str, content: str, backup: bool = True, - dry_run: bool = False, sudo: bool = False) -> None: + def write_file(self, path: str, content: str, mode: Optional[str] = None, + backup: bool = True, dry_run: bool = False, sudo: bool = False) -> None: if dry_run: self.log.info(f"[DRY-RUN] would write {len(content)} bytes to {path}") return if backup: self.backup(path, sudo=sudo) - tmp = tempfile.NamedTemporaryFile("w", delete=False) - tmp.write(content) - tmp.close() - # 0644 like the bash renders (umask-022 redirect); mkstemp's 0600 would make - # freshly created units/configs owner-only. Overwrites via sudo cp keep the - # destination's existing mode either way. - os.chmod(tmp.name, 0o644) - if sudo: - self.sys.sudo(["mkdir", "-p", os.path.dirname(path)]) - self.sys.sudo(["cp", tmp.name, path]) - os.unlink(tmp.name) - else: - os.makedirs(os.path.dirname(path), exist_ok=True) - shutil.move(tmp.name, path) + fd, tmp_name = tempfile.mkstemp(prefix="apex-") + try: + with os.fdopen(fd, "w", encoding="utf-8") as tmp: + tmp.write(content) + target_mode = mode or "0644" + if sudo: + self.sys.sudo(["mkdir", "-p", os.path.dirname(path)]) + self.sys.sudo(["cp", tmp_name, path]) + self.sys.sudo(["chmod", target_mode, path]) + else: + os.makedirs(os.path.dirname(path), exist_ok=True) + shutil.copyfile(tmp_name, path) + os.chmod(path, int(target_mode, 8)) + finally: + if os.path.exists(tmp_name): + os.unlink(tmp_name) self.log.success(f"Wrote {path}") def inject_block(self, target: str, content: str, start: str, end: str, diff --git a/engine/lib/notify.py b/engine/lib/notify.py index 535c5bf..aa2e81c 100644 --- a/engine/lib/notify.py +++ b/engine/lib/notify.py @@ -5,12 +5,14 @@ The alert `instance` is the node's public host (APEX_NODE_HOST), passed in from context. POSTs application/x-www-form-urlencoded via urllib (stdlib). """ from __future__ import annotations + import html import urllib.parse import urllib.request from datetime import datetime NOTIFY_TIMEOUT = 20 +MAX_MESSAGE_LEN = 3500 def resolve_bot_url(value: str) -> str: @@ -20,7 +22,7 @@ def resolve_bot_url(value: str) -> str: path = value[1:] if not path: raise ValueError("empty Telegram Bot URL file reference") - with open(path) as f: + with open(path, encoding="utf-8") as f: return f.read().strip() @@ -33,7 +35,7 @@ def build_text(title: str, message: str, node: str, level: str, ts: str) -> str: elif level == "WARN": severity = "warning" instance = node - msg = message if len(message) <= 200 else message[:200] + "..." + msg = message if len(message) <= MAX_MESSAGE_LEN else message[:MAX_MESSAGE_LEN] + "..." msg = html.escape(msg) safe_title = html.escape(title) return (f"{icon} {safe_title}\n" @@ -60,6 +62,12 @@ class Notify: if not bot_url: self.log.warn("Telegram Bot URL not provided. Skipping notification.") return False + + parsed = urllib.parse.urlparse(bot_url) + if parsed.scheme != "https": + self.log.error(f"Telegram Bot URL must use HTTPS, got scheme: {parsed.scheme!r}") + return False + ts = datetime.now().astimezone().strftime("%Y-%m-%d %H:%M:%S %Z") text = build_text(title, message, node, level, ts) data = urllib.parse.urlencode({"parse_mode": "HTML", "text": text}).encode() diff --git a/engine/lib/ufw.py b/engine/lib/ufw.py index c925760..ee56c33 100644 --- a/engine/lib/ufw.py +++ b/engine/lib/ufw.py @@ -6,13 +6,16 @@ and the new APEX blocks first (so re-running after cutover migrates the marker i place — SP2 §6). Docker rules go through ufw-docker; host rules through plain ufw. """ from __future__ import annotations + import os +import shlex import shutil APEX_START = "# START APEX.ERMNVLDMR.COM RULES" APEX_END = "# END APEX.ERMNVLDMR.COM RULES" OLD_PAIRS = [("# START ROOT.ERMNVLDMR.COM RULES", "# END ROOT.ERMNVLDMR.COM RULES")] + def resolve_ufw_docker_bin() -> str: """Find ufw-docker in PATH, /usr/local/bin, or fallback to ~/.local/bin/ufw-docker.""" found = shutil.which("ufw-docker") @@ -22,6 +25,7 @@ def resolve_ufw_docker_bin() -> str: return "/usr/local/bin/ufw-docker" return os.path.expanduser("~/.local/bin/ufw-docker") + UFW_DOCKER_BIN = resolve_ufw_docker_bin() @@ -40,10 +44,18 @@ class Ufw: if not self.sys.ok(["sudo", "test", "-f", target]): self.log.error(f"Target file does not exist: {target}") raise SystemExit(1) - content = "\n".join(clean_rules(open(source_file).read())) - self.host.inject_block(target, content, APEX_START, APEX_END, - anchor="# End required lines", old_pairs=OLD_PAIRS, - dry_run=dry_run, sudo=True) + with open(source_file, encoding="utf-8") as f: + content = "\n".join(clean_rules(f.read())) + self.host.inject_block( + target, + content, + APEX_START, + APEX_END, + anchor="# End required lines", + old_pairs=OLD_PAIRS, + dry_run=dry_run, + sudo=True, + ) def apply(self, rules_dir: str | None = None, config_dir: str | None = None, dry_run: bool = False) -> None: if rules_dir and not os.path.isdir(rules_dir): @@ -65,27 +77,30 @@ class Ufw: if config_dir: host_rules = os.path.join(config_dir, "host.rules") if os.path.isfile(host_rules): - for rule in clean_rules(open(host_rules).read()): + with open(host_rules, encoding="utf-8") as f: + rules = clean_rules(f.read()) + for rule in rules: self.log.info(f"Applying host ufw rule: {rule}") if dry_run: self.log.info(f"[DRY-RUN] sudo ufw {rule}") else: - self.sys.run(["bash", "-c", f"sudo ufw {rule}"], check=True) + self.sys.sudo(["ufw", *shlex.split(rule)], check=True) self._inject_file("/etc/ufw/before.rules", os.path.join(config_dir, "before.rules"), dry_run) self._inject_file("/etc/ufw/after.rules", os.path.join(config_dir, "after.rules"), dry_run) if rules_dir: - rule_files = [fn for fn in sorted(os.listdir(rules_dir)) - if fn.endswith(".rules") and not fn.startswith(".")] + rule_files = [fn for fn in sorted(os.listdir(rules_dir)) if fn.endswith(".rules") and not fn.startswith(".")] if not rule_files: self.log.warn(f"No .rules files found in {rules_dir}, nothing to apply.") for fn in rule_files: - for rule in clean_rules(open(os.path.join(rules_dir, fn)).read()): + with open(os.path.join(rules_dir, fn), encoding="utf-8") as f: + rules = clean_rules(f.read()) + for rule in rules: self.log.info(f"Applying ufw-docker rule ({fn}): {rule}") if dry_run: self.log.info(f"[DRY-RUN] sudo {UFW_DOCKER_BIN} {rule}") else: - self.sys.run(["sudo", UFW_DOCKER_BIN, *rule.split()], check=True) + self.sys.sudo([UFW_DOCKER_BIN, *shlex.split(rule)], check=True) self.log.success("All ufw rules applied.") self.log.warn("Run `sudo systemctl restart ufw` to apply. Ensure ssh is allowed via plain ufw!") diff --git a/assets/alloy/config.alloy b/etc/alloy/config.alloy similarity index 100% rename from assets/alloy/config.alloy rename to etc/alloy/config.alloy diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..221b5ea --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,21 @@ +[project] +name = "apex-commons" +version = "2026.10.0" +description = "Core runtime and orchestration framework for apex.ermnvldmr.com node fleet" +readme = "README.md" +license = { text = "BSD-2-Clause" } +requires-python = ">=3.10" + +[tool.ruff] +line-length = 100 +target-version = "py310" + +[tool.ruff.lint] +select = ["E", "F", "W", "I", "B", "UP", "S"] +ignore = ["S603", "S607"] # Subprocess invocations are mediated via ctx.sys + +[tool.mypy] +python_version = "3.10" +warn_return_any = true +warn_unused_configs = true +disallow_untyped_defs = false diff --git a/tests/test_compose.py b/tests/test_compose.py new file mode 100644 index 0000000..9d997f9 --- /dev/null +++ b/tests/test_compose.py @@ -0,0 +1,119 @@ +# tests/test_compose.py +import os +import tempfile +import unittest +from unittest.mock import MagicMock + +from actions import compose + + +class TestCompose(unittest.TestCase): + def setUp(self): + self.tmpdir = tempfile.TemporaryDirectory() + self.addCleanup(self.tmpdir.cleanup) + self.comp_dir = os.path.join(self.tmpdir.name, "compositions") + os.makedirs(self.comp_dir) + + # Create compositions: apex, sso, arr + for name in ("apex", "sso", "arr"): + d = os.path.join(self.comp_dir, name) + os.makedirs(d) + with open(os.path.join(d, "docker-compose.yml"), "w") as f: + f.write(f"name: {name}\n") + + self.stier3 = os.path.join(self.tmpdir.name, "tier3_shared") + os.makedirs(os.path.join(self.stier3, "apex", "xray", "shm")) + + self.ctx = MagicMock() + self.ctx.paths.compositions = self.comp_dir + self.ctx.paths.stier3 = self.stier3 + self.ctx.vars.return_value = { + "APEX_SUBNET": "198.18.16.0/24", + "APEX_NODE_HOST": "test-host", + } + self.ctx.sys.run = MagicMock() + + def test_sequence_all_by_default(self): + args = MagicMock(action="restart", p=None, dry_run=False, extra=[]) + compose.run(self.ctx, args) + + # Should execute on apex, arr, sso (apex core first, others sorted) + calls = self.ctx.sys.run.call_args_list + self.assertEqual(len(calls), 3) + visited = [os.path.basename(c.kwargs["cwd"]) for c in calls] + self.assertEqual(visited, ["apex", "arr", "sso"]) + + def test_down_reverse_sequence(self): + args = MagicMock(action="down", p=None, dry_run=False, extra=[]) + compose.run(self.ctx, args) + + calls = self.ctx.sys.run.call_args_list + self.assertEqual(len(calls), 3) + visited = [os.path.basename(c.kwargs["cwd"]) for c in calls] + # Reverse on down: sso, arr, apex + self.assertEqual(visited, ["sso", "arr", "apex"]) + + def test_target_single_composition_positional(self): + args = MagicMock(action="restart", p=None, dry_run=False, extra=["apex"]) + compose.run(self.ctx, args) + + calls = self.ctx.sys.run.call_args_list + self.assertEqual(len(calls), 1) + self.assertEqual(os.path.basename(calls[0].kwargs["cwd"]), "apex") + self.assertEqual(calls[0].args[0], ["docker", "compose", "restart"]) + + def test_target_single_composition_flag(self): + args = MagicMock(action="restart", p="sso", dry_run=False, extra=[]) + compose.run(self.ctx, args) + + calls = self.ctx.sys.run.call_args_list + self.assertEqual(len(calls), 1) + self.assertEqual(os.path.basename(calls[0].kwargs["cwd"]), "sso") + + def test_target_with_service_passthrough(self): + args = MagicMock(action="restart", p=None, dry_run=False, extra=["apex", "traefik"]) + compose.run(self.ctx, args) + + calls = self.ctx.sys.run.call_args_list + self.assertEqual(len(calls), 1) + self.assertEqual(os.path.basename(calls[0].kwargs["cwd"]), "apex") + self.assertEqual(calls[0].args[0], ["docker", "compose", "restart", "traefik"]) + + def test_target_up_detached_order(self): + args = MagicMock(action="up", p=None, dry_run=False, extra=["-d", "sso"]) + compose.run(self.ctx, args) + + calls = self.ctx.sys.run.call_args_list + self.assertEqual(len(calls), 1) + self.assertEqual(os.path.basename(calls[0].kwargs["cwd"]), "sso") + self.assertEqual(calls[0].args[0], ["docker", "compose", "up", "-d"]) + + def test_dry_run_skips_execution(self): + args = MagicMock(action="restart", p=None, dry_run=True, extra=["apex"]) + compose.run(self.ctx, args) + + self.ctx.sys.run.assert_not_called() + self.ctx.log.info.assert_any_call( + f"[DRY-RUN] (cd {os.path.join(self.comp_dir, 'apex')} && docker compose restart)" + ) + + def test_unknown_target_raises_system_exit(self): + args = MagicMock(action="restart", p=None, dry_run=False, extra=["nonexistent"]) + with self.assertRaises(SystemExit) as cm: + compose.run(self.ctx, args) + self.assertEqual(cm.exception.code, 1) + + def test_xray_socket_cleanup_on_apex_restart(self): + shm = os.path.join(self.stier3, "apex", "xray", "shm") + sock_file = os.path.join(shm, "stale.sock") + with open(sock_file, "w") as f: + f.write("") + + args = MagicMock(action="restart", p=None, dry_run=False, extra=["apex"]) + compose.run(self.ctx, args) + + self.assertFalse(os.path.exists(sock_file)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_env.py b/tests/test_env.py new file mode 100644 index 0000000..d3c9ccc --- /dev/null +++ b/tests/test_env.py @@ -0,0 +1,68 @@ +# tests/test_env.py +import io +import os +import sys +import tempfile +import unittest +from unittest.mock import MagicMock + +from actions import env + + +class TestEnv(unittest.TestCase): + def setUp(self): + self.tmpdir = tempfile.TemporaryDirectory() + self.addCleanup(self.tmpdir.cleanup) + self.comp_dir = os.path.join(self.tmpdir.name, "compositions") + os.makedirs(os.path.join(self.comp_dir, "apex")) + + self.ctx = MagicMock() + self.ctx.paths.compositions = self.comp_dir + self.ctx.vars.return_value = { + "APEX_SUBNET": "198.18.16.0/24", + "APEX_NODE_HOST": "test-host", + "OTHER_VAR": "ignore_me", + } + + def test_export_format_stdout(self): + args = MagicMock(sync=False, write=None, dotenv=False) + captured = io.StringIO() + old_stdout = sys.stdout + try: + sys.stdout = captured + env.run(self.ctx, args) + finally: + sys.stdout = old_stdout + + out = captured.getvalue() + self.assertIn("export APEX_NODE_HOST=test-host", out) + self.assertIn("export APEX_SUBNET=198.18.16.0/24", out) + self.assertNotIn("OTHER_VAR", out) + + def test_dotenv_format_stdout(self): + args = MagicMock(sync=False, write=None, dotenv=True) + captured = io.StringIO() + old_stdout = sys.stdout + try: + sys.stdout = captured + env.run(self.ctx, args) + finally: + sys.stdout = old_stdout + + out = captured.getvalue() + self.assertIn("APEX_NODE_HOST=test-host", out) + self.assertNotIn("export", out) + + def test_sync_to_file(self): + target = os.path.join(self.comp_dir, "apex", ".env") + args = MagicMock(sync=True, write=None, dotenv=False) + env.run(self.ctx, args) + + self.ctx.host.write_file.assert_called_once() + call_args = self.ctx.host.write_file.call_args + self.assertEqual(call_args[0][0], target) + self.assertIn("APEX_SUBNET=198.18.16.0/24", call_args[0][1]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_happ_subscriptions.py b/tests/test_happ_subscriptions.py deleted file mode 100644 index 9b5d79d..0000000 --- a/tests/test_happ_subscriptions.py +++ /dev/null @@ -1,62 +0,0 @@ -import json -import pathlib -import subprocess -import sys -import tempfile -import unittest -from urllib.parse import parse_qs, unquote, urlsplit - - -ROOT = pathlib.Path(__file__).resolve().parents[1] -GENERATOR = ROOT / 'assets/happ-subscription-generator/generator.py' - - -class TestHappSubscriptions(unittest.TestCase): - def generate(self, scheme, params, country='NL'): - data = { - 'routing': {'name': 'test-routing'}, - 'templates': {'edge': { - 'country': country, 'nodes': ['metis'], 'scheme': scheme, - 'host': 'edge.example.com', 'port': 443, 'params': params, - }}, - 'users': [{'name': 'test-user', 'psub': 'test.psub', 'configs': [ - {'template': 'edge', 'id': 'test-credential', 'params': {'sni': 'tls.example.com'}}, - ]}], - } - with tempfile.TemporaryDirectory() as tmp: - source = pathlib.Path(tmp) / 'input.json' - source.write_text(json.dumps(data)) - result = subprocess.run( - [sys.executable, str(GENERATOR), str(source), tmp], - capture_output=True, text=True, - ) - self.assertEqual(result.returncode, 0, result.stderr) - lines = (pathlib.Path(tmp) / 'test.psub').read_text().splitlines() - self.assertEqual(len(lines), 2) - return urlsplit(lines[1]) - - def test_hysteria_names_follow_protocol_without_changing_link_parameters(self): - for scheme in ('hysteria2', 'hy2'): - for params in ({}, {'type': 'tcp'}): - with self.subTest(scheme=scheme, params=params): - link = self.generate(scheme, params) - self.assertEqual(unquote(link.fragment), - '🇳🇱 Netherlands [hysteria] | metis | test-user') - self.assertEqual(link.scheme, scheme) - self.assertEqual(link.netloc, 'test-credential@edge.example.com:443') - self.assertEqual(parse_qs(link.query), - {**{k: [v] for k, v in params.items()}, 'sni': ['tls.example.com']}) - - def test_vless_keeps_transport_label_and_user_overrides(self): - link = self.generate('vless', {'type': 'xhttp', 'sni': 'old.example.com'}) - self.assertEqual(unquote(link.fragment), - '🇳🇱 Netherlands [xhttp] | metis | test-user') - self.assertEqual(parse_qs(link.query), {'type': ['xhttp'], 'sni': ['tls.example.com']}) - - def test_existing_country_and_tcp_label_are_preserved(self): - link = self.generate('vless', {'type': 'tcp'}, country='LV') - self.assertEqual(unquote(link.fragment), '🇱🇻 Latvia [tcp] | metis | test-user') - - -if __name__ == '__main__': - unittest.main() diff --git a/tests/test_notify.py b/tests/test_notify.py index 93504ab..9d9c6be 100644 --- a/tests/test_notify.py +++ b/tests/test_notify.py @@ -11,7 +11,7 @@ class TestNotify(unittest.TestCase): self.assertIn("#instance_node1_example_com", t) # dots -> underscores in the hashtag def test_html_escaped_and_truncated(self): - t = build_text("A & B", "