Recover interrupted Docker policy changes and validate fleet bootstrap pins

- Purge misplaced RFCs, scratch tokens, and node-proprietary assets
- Decouple remote provisioning to ci/scripts/init-node.py
- Add targeted composition orchestration in actions/compose.py
- Add host-side Xray stale socket cleanup in /dev/shm
- Add actions/env.py for shell exports and .env synchronization
- Harden UFW, backup retention, host tempfiles, and notify dispatcher
- Add pyproject.toml and Makefile with 78 passing unit tests

.agents/skills/update-compositions/SKILL.md

@@ -0,0 +1,234 @@
+---
+name: update-compositions
+description: Use when updating, upgrading, or bumping Docker container images, tags, composition services, and OS packages across one or more APEX fleet nodes.
+---
+
+# Updating APEX Compositions & System Packages
+
+This skill guides the safe, structured update of container images, compose definitions, and base operating system packages across nodes in an APEX fleet.
+
+## Core Rules
+
+1. **Verify access first**: Never execute commands against a node without confirming SSH connectivity and credentials.
+2. **Hold critical services**: Always ask the user which services or compositions must NOT be touched before discovering or updating images.
+3. **Classify update scope**: Separate updates into **Minor/Patch** (non-breaking), **Major** (breaking / schema migration risks), and **Unstable SemVer / CalVer** (requiring changelog scrutiny).
+4. **Gate major and breaking updates**: Never apply a Major version bump, SemVer `0.x` minor change, or CalVer jump without migration research and explicit user approval.
+5. **Preserve APEX engine context**: Always orchestrate compose actions through `./commons/apex composition/run` (or with derived `APEX_*` variables and layered env files) rather than bare `docker compose`.
+6. **Safeguard in-tree patches**: Do not bump images for containers with host bind mounts that patch application source code without explicit review.
+7. **Verify fleet observability post-rollout**: An update is not complete until containers reach `Up` and `healthy`, logs are clean, Prometheus scrape targets report `UP`, blackbox probes pass, and remote write streams are established.
+
+---
+
+## Workflow Steps
+
+### Step 1: Confirm Node Access
+
+1. Identify the target nodes from the user request.
+2. If access credentials, custom SSH ports, or key paths are not declared in the environment's SSH configuration, confirm:
+   - SSH hostname / IP
+   - SSH port (APEX default: `2222`)
+   - SSH user
+   - Identity file / key
+3. Test connectivity to each node before proceeding:
+   ```bash
+   ssh -p <port> <user>@<node_fqdn> "hostname && uptime"
+   ```
+4. **Completion check**: All target nodes return exit code 0 on the connectivity probe.
+
+---
+
+### Step 2: Inquire About Critical Services & Holds
+
+1. Prompt the user explicitly before inspecting or modifying any services:
+   > "Which critical services or compositions on these nodes should **NOT** be updated during this run (e.g. databases, SSO/Identity providers, Mail, Vaults, core routing)?"
+2. Record the freeze list (held services/compositions).
+3. Ensure all subsequent steps skip services on the freeze list.
+4. **Completion check**: User response received, and the list of excluded services is confirmed.
+
+---
+
+### Step 3: Inventory Running Services, Images, and Custom Patches
+
+For each target node:
+
+1. List active containers, current images, and status:
+   ```bash
+   ssh <node> "docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}'"
+   ```
+2. Locate the compose files on the node:
+   - Shared core: `/srv/<node_fqdn>/compositions/apex/docker-compose.yml`
+   - Node applications: `/srv/<node_fqdn>/compositions/*/docker-compose.yml`
+   - Overrides: `docker-compose.override.yml` in each composition directory
+3. Map every running container to its compose declaration:
+   - Image repository
+   - Pinned tag / SHA digest
+   - Whether the image is locally built (`build:` context present)
+4. **Inspect for in-tree patches & mounts**:
+   - Check container volume declarations for bind mounts into application code paths (e.g. `./patches/...:/app/...`, monkeypatched modules, or custom entrypoint scripts).
+   - If a service mounts patches into internal software paths, flag it: upstream image updates may alter internal APIs and break the patch. Add to hold or require explicit user confirmation.
+5. Exclude any services identified on the freeze list from Step 2.
+6. **Completion check**: A complete table of candidate compositions, running tags, compose file paths, and in-tree patch status is constructed.
+
+---
+
+### Step 4: Seek Latest Tags and Classify Update Types
+
+For each candidate service:
+
+1. Query the container registry (Docker Hub, GitHub Container Registry `ghcr.io`, Quay, etc.) for available release tags. Refer to [references/registry-queries.md](references/registry-queries.md) for registry API patterns.
+2. Determine the latest stable versioned tag:
+   - Prefer semantic versioned tags matching the existing packaging flavor (e.g. `-alpine`, `-bookworm`).
+   - Ignore pre-release, alpha, beta, or release-candidate (`-rc`) tags unless the running container already pins one.
+   - Do not replace a pinned version tag with bare `:latest`.
+3. Compare the current tag with the latest tag:
+
+| Tag Transition | Category | Classification | Action Path |
+| :--- | :--- | :--- | :--- |
+| `1.37.1` -> `1.37.2` | SemVer `1.x+` | **Patch** | Safe -> Proceed to Step 6 |
+| `1.37.1` -> `1.38.0` | SemVer `1.x+` | **Minor** | Safe -> Proceed to Step 6 |
+| `0.4.1` -> `0.4.2` | SemVer `0.x` | **0.x Patch** | Safe -> Proceed to Step 6 |
+| `0.4.1` -> `0.7.0` | SemVer `0.x` | **0.x Minor** | **Route to Step 5 (Strict Gate)** |
+| `YY.M.D` -> `YY.M.D` | CalVer / Date | **CalVer Jump** | Multi-month jump -> **Route to Step 5** |
+| Floating tag (e.g. `:4`, `:16`) | Floating | **In-place Digest Bump** | Safe -> Proceed to Step 6 |
+| `1.x` -> `2.x` or `16` -> `17` | SemVer `1.x+` | **Major** | **Route to Step 5 (Strict Gate)** |
+| Locally built image (`build:`) | Local Build | **Local Build** | Git update + build (Step 6) |
+
+> [!IMPORTANT]
+> **SemVer 0.x Rule**: Per SemVer specification Clause 4, version `0.x` software has no public API stability guarantees. Minor version bumps (e.g. `0.4` -> `0.7`) often contain breaking changes, renamed environment variables, or schema changes. Treat them with the same scrutiny as Major updates.
+>
+> **CalVer Rule**: Projects using date-based versioning (e.g. Xray, Ubuntu base images) do not follow SemVer breaking change rules. Releases spanning several months must be vetted against upstream changelogs for breaking configuration or default behavior shifts.
+
+4. **Completion check**: Every eligible service has a determined target tag and update classification.
+
+---
+
+### Step 5: Major & Breaking Update Protocol (Strict Gating)
+
+For each service classified as **Major**, **0.x Minor**, or **CalVer Jump**:
+
+1. **Research Upstream Changes**:
+   - Check upstream release notes, migration guides, and changelogs.
+   - Identify:
+     - Breaking configuration changes (renamed environment variables, CLI flags, config format alterations).
+     - Altered default security behaviors (e.g. new access control rules, loopback/private IP blocking, required authentication headers).
+     - Storage schema migrations (e.g., PostgreSQL major upgrade requirements).
+2. **Formulate Migration & Rollback Plan**:
+   - Outline pre-upgrade steps (e.g. explicit logical database dump `pg_dump` / `mysqldump` beyond normal snapshots).
+   - Outline required compose file changes (volumes, environment variables).
+   - Outline fallback/rollback steps if the update fails.
+3. **Request Explicit User Approval**:
+   - Present findings to the user:
+     - Target service and node.
+     - Current version vs New version.
+     - Summary of breaking changes and migration strategy.
+   - Ask: *"Do you approve proceeding with the update for `<service>` on `<node>` under this strategy?"*
+4. **Gate**:
+   - If approved: Proceed to Step 6 for this service.
+   - If rejected or deferred: Add service to the freeze list and leave it untouched.
+5. **Completion check**: Every breaking or major update has documented strategy + user approval or is explicitly skipped.
+
+---
+
+### Step 6: Apply Container Updates via APEX Engine
+
+Follow node and service rollout order to minimize blast radius:
+1. **Node ordering**:
+   - Application & worker nodes first.
+   - Communications, data, and storage nodes next.
+   - Core edge / gateway / SSO nodes last, ensuring routing and auth remain stable during earlier rollouts.
+2. **Service dependency sequencing**:
+   - Infrastructure providers first (DNS resolvers, database servers, mail exchangers).
+   - Verify provider health before updating or restarting dependent consumer services (e.g. SSO forwardAuth, telemetry agents).
+
+For each node:
+
+1. **Pre-upgrade backup**:
+   - Trigger a backup of stateful volumes:
+     ```bash
+     ssh <node> "apex tier/backup"
+     ```
+   - For database services being updated, execute an in-container logical export to host storage.
+2. **Update image tags in compose definitions**:
+   - Modify the target `docker-compose.yml`, `docker-compose.override.yml`, or node configuration repository.
+3. **Pull and recreate services via APEX CLI**:
+   - Always run through the node's APEX CLI to preserve derived `APEX_*` variables and layered env files:
+     ```bash
+     ssh <node> "cd /srv/<node_fqdn> && ./commons/apex composition/run pull"
+     ssh <node> "cd /srv/<node_fqdn> && ./commons/apex composition/run up -d"
+     ```
+   - *Note on `--force-recreate`*: When updating containers using floating tags, recovering from host reboots, or handling services with `tmpfs` state (e.g. mail spools), pass `--force-recreate` to ensure the container entrypoint runs a clean initialization:
+     ```bash
+     ssh <node> "cd /srv/<node_fqdn> && ./commons/apex composition/run up -d --force-recreate"
+     ```
+4. **Completion check**: Compose commands finish with exit code 0.
+
+---
+
+### Step 7: Apply System Packages and OS Updates
+
+When the update scope includes host operating system packages:
+
+1. **Pre-upgrade check**:
+   - Check available disk space: `df -h /`
+   - Check for broken package states: `sudo dpkg --audit`
+2. **Handle unattended package configuration (debconf)**:
+   - Prevent interactive prompts (e.g. GRUB drive selection on VPS instances) from blocking non-interactive execution:
+     ```bash
+     # Example for grub-pc on virtualized hosts using /dev/sda:
+     echo 'grub-pc grub-pc/install_devices multiselect /dev/sda' | sudo debconf-set-selections
+     ```
+3. **Execute system package upgrade**:
+   ```bash
+   ssh <node> "sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade -y"
+   ```
+4. **Clean package cache**:
+   ```bash
+   ssh <node> "sudo apt-get autoremove -y && sudo apt-get clean"
+   ```
+5. **Reboot handling & coordination**:
+   - Check if a system reboot is required:
+     ```bash
+     ssh <node> "test -f /var/run/reboot-required && echo 'REBOOT_REQUIRED' || echo 'OK'"
+     ```
+   - If reboot is required:
+     1. Notify the user before initiating the reboot.
+     2. Reboot node: `ssh <node> "sudo reboot"`
+     3. Wait for SSH availability: poll connectivity with timeout.
+     4. After boot, verify systemd state (`systemctl is-system-running`) and check that Docker containers auto-started.
+     5. If services with `tmpfs` mounts fail startup checks (e.g. Postfix spool directories in mail containers), run `./commons/apex composition/run up -d --force-recreate`.
+6. **Completion check**: Host packages are up to date and node is online.
+
+---
+
+### Step 8: Verify Service Health & Fleet Observability
+
+Immediately after recreation and host reboots on each node:
+
+1. **Inspect container state**:
+   ```bash
+   ssh <node> "docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'"
+   ```
+   - Verify status is `Up`.
+   - Verify healthchecks transition to `(healthy)` and do not hang in `(unhealthy)` or `(starting)`.
+2. **Inspect recent logs**:
+   ```bash
+   ssh <node> "docker logs --tail 50 <container_name>"
+   ```
+   - Check for crash loops, panic traces, database connection errors, or authentication failures.
+3. **Verify HTTP/network response (edge/web services)**:
+   ```bash
+   curl -IfsS -k https://<service_fqdn> || true
+   ```
+4. **Observability & Telemetry Verification**:
+   - If the fleet runs Prometheus:
+     - Check scrape targets: `curl -s http://<prom_internal_ip>:9090/prometheus/api/v1/targets | jq -r '.data.activeTargets[] | "\(.labels.job) | \(.health)"'` — verify all targets report `up`.
+     - Check blackbox probes: `curl -s 'http://<prom_internal_ip>:9090/prometheus/api/v1/query?query=probe_success==0'` — verify 0 failing probes.
+     - Check active alerts: `curl -s http://<prom_internal_ip>:9090/prometheus/api/v1/alerts | jq '.data.alerts[] | select(.state=="firing")'` — verify only expected watchdogs fire.
+     - Check remote write: confirm telemetry agent (e.g. Alloy) is actively shipping host/container metrics without push retry errors.
+5. **Rollback protocol**:
+   - If a container enters a crash loop (`Restarting`, `Exit 1`) or breaks ingress:
+     1. Capture failure logs.
+     2. Revert the image tag in the compose file to the prior tag.
+     3. Re-run `./commons/apex composition/run up -d --force-recreate`.
+     4. Notify user immediately with logs and root cause.
+6. **Completion check**: All containers are confirmed `Up` and `healthy`, external ingress functions normally, and telemetry metrics/targets are green.

.agents/skills/update-compositions/references/registry-queries.md

@@ -0,0 +1,99 @@
+# Container Registry Tag Queries
+
+This reference details how to discover available image tags across common container registries without requiring external tooling.
+
+## 1. Docker Hub (`hub.docker.com`)
+
+Docker Hub provides an unauthenticated v2 API for public repositories.
+
+### Official Images (Library)
+For top-level images like `postgres`, `redis`, `traefik`, `mariadb`:
+
+```bash
+IMAGE="postgres"
+curl -s "https://registry.hub.docker.com/v2/repositories/library/${IMAGE}/tags?page_size=100" \
+  | python3 -c '
+import sys, json
+data = json.load(sys.stdin)
+tags = [t["name"] for t in data.get("results", [])]
+print("\n".join(tags[:30]))
+'
+```
+
+### Community / User Images
+For namespace images like `adguard/adguardhome`, `prom/prometheus`, `grafana/alloy`:
+
+```bash
+ORG="adguard"
+IMAGE="adguardhome"
+curl -s "https://registry.hub.docker.com/v2/repositories/${ORG}/${IMAGE}/tags?page_size=100" \
+  | python3 -c '
+import sys, json
+data = json.load(sys.stdin)
+tags = [t["name"] for t in data.get("results", [])]
+print("\n".join(tags[:30]))
+'
+```
+
+---
+
+## 2. GitHub Container Registry (`ghcr.io`)
+
+GHCR implements OCI distribution spec. Public image tags can be queried with an anonymous bearer token.
+
+```bash
+IMAGE="xtls/xray-core"
+TOKEN=$(curl -s "https://ghcr.io/token?scope=repository:${IMAGE}:pull" | python3 -c 'import sys, json; print(json.load(sys.stdin)["token"])')
+curl -s -H "Authorization: Bearer ${TOKEN}" "https://ghcr.io/v2/${IMAGE}/tags/list" \
+  | python3 -c '
+import sys, json
+data = json.load(sys.stdin)
+tags = data.get("tags", [])
+print("\n".join(sorted(tags, reverse=True)[:30]))
+'
+```
+
+---
+
+## 3. Quay.io (`quay.io`)
+
+Quay provides a REST API:
+
+```bash
+REPO="repository_name"
+curl -s "https://quay.io/api/v1/repository/${REPO}/tag/?limit=50&onlyActiveTags=true" \
+  | python3 -c '
+import sys, json
+data = json.load(sys.stdin)
+tags = [t["name"] for t in data.get("tags", [])]
+print("\n".join(tags[:30]))
+'
+```
+
+---
+
+## 4. SemVer Matching & Sorting
+
+When filtering tags to find the latest stable version matching the current container's flavor:
+
+```python
+import re
+
+def parse_semver(tag: str):
+    # Match patterns like v1.2.3, 1.2.3-alpine, 16.4
+    m = re.match(r"^v?(\d+)(?:\.(\d+))?(?:\.(\d+))?(?:-(.+))?$", tag)
+    if not m:
+        return None
+    major = int(m.group(1))
+    minor = int(m.group(2) or 0)
+    patch = int(m.group(3) or 0)
+    flavor = m.group(4) or ""
+    return (major, minor, patch, flavor)
+```
+
+Rules for candidate selection:
+1. Preserve flavor: If currently running `16-alpine`, match only tags with suffix `-alpine`.
+2. Ignore pre-releases: Filter out tags containing `alpha`, `beta`, `rc`, `dev`, `test`, or git SHAs.
+3. Classify:
+   - Same `major`, higher `minor` or `patch` -> **Minor / Patch**.
+   - Higher `major` -> **Major** (requires Step 5 approval).

.gitignore

@@ -9,9 +9,12 @@
 __pycache__/
 *.pyc
 .pytest_cache/
+.venv/
 
 # Runtime / secrets (never in commons)
 *.log
 *.env
 .env
 *.bak
+._*
+.DS_Store

LICENSE

@@ -1,21 +1,24 @@
-MIT License
+BSD 2-Clause License
 
-Copyright (c) 2026-present Vladimir Eremin
+Copyright (c) 2026, Vladimir Eremin
 
-Permission is hereby granted, free of charge, to any person obtaining a copy
+Redistribution and use in source and binary forms, with or without
-of this software and associated documentation files (the "Software"), to deal
+modification, are permitted provided that the following conditions are met:
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
 
-The above copyright notice and this permission notice shall be included in all
+1. Redistributions of source code must retain the above copyright notice, this
-copies or substantial portions of the Software.
+   list of conditions and the following disclaimer.
 
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+2. Redistributions in binary form must reproduce the above copyright notice,
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+   this list of conditions and the following disclaimer in the documentation
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+   and/or other materials provided with the distribution.
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-SOFTWARE.
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Makefile

@@ -0,0 +1,22 @@
+.PHONY: test lint fmt check
+
+test:
+	bash tests/run.sh
+
+lint:
+	@if command -v ruff >/dev/null 2>&1; then \
+		ruff check . scripts/bootstrap-node; \
+	else \
+		python3 -m compileall -q actions tests scripts && \
+		python3 -m py_compile scripts/bootstrap-node && \
+		echo "ruff not installed in PATH; verified python bytecode compilation."; \
+	fi
+
+fmt:
+	@if command -v ruff >/dev/null 2>&1; then \
+		ruff format .; \
+	else \
+		echo "ruff not installed in PATH; skipping fmt."; \
+	fi
+
+check: lint test

PARITY.md

@@ -1,76 +0,0 @@
-# Port parity map (bash -> python)
-
-Behavioral parity is validated on a scratch host in Phase 3. This maps each ported
-action to its bash source and the behaviors that must match.
-
-| apex action | bash source (root commons) | must-match behaviors |
-|---|---|---|
-| configure/base | actions/configure/base | apt update+install curl/wget/git/rsyslog; rsyslog running; ufw + ufw-docker at ~/.local/bin |
-| configure/ufw | actions/configure/ufw + root_template_ufw | host.rules via plain ufw; before/after.rules block inject (APEX markers, self-heal ROOT); docker rules via ufw-docker |
-| configure/crowdsec | actions/configure/crowdsec | install if cscli absent; render configs->/etc/crowdsec; collections linux+traefik; firewall bouncer if bouncer file present |
-| configure/cron | actions/configure/cron + root_template_crontab | render crontab; backup+install; rollback on empty/failure |
-| configure/systemd | actions/configure/systemd | render each unit -> /etc/systemd/system; daemon-reload; enable+start .service |
-| configure/routing | actions/configure/routing | ip_forward=1; fwmark 0x1->table100 rule; local default route table100 (idempotent) |
-| configure | (new umbrella) | run base,ufw,crowdsec,cron,systemd,routing in order |
-| sync/repository | actions/sync/repository | sync/<node> branch; add -A + commons submodule; commit; rebase origin/main (was trunk in the monorepo); force-with-lease; telegram |
-| sync/packages | actions/sync/packages | apt --just-print upgrade Inst list; docker+skopeo digest compare; telegram info |
-| backup/run | actions/sync/tiers (restic-backup) | init-if-needed; label-driven backup via resticontainer (discovers `restic.*`-labelled services, runs pre-hooks/stops, snapshots the resolved host paths) --host <node>.apex.ermnvldmr.com --tag biweekly --compression max; guard on a fresh snapshot; forget --prune 7/4/12; telegram |
-| tiers/link | actions/tiers/link | node + per-project @tierN symlinks; shared/ links; .env symlink |
-| tiers/useradd | actions/tiers/useradd | noroot-<proj> system users; noroot-shared group; APEX_UID/GID block in .env |
-| tiers/chown | actions/tiers/chown | {tier}/{proj}=uid:gid; {tier}/shared/{proj}=uid:sgid+sticky; base=root:root |
-| utils/extract-traefik-certs | actions/utils/extract-traefik-certs | traefik-certs-dumper v3; 600 perms; chown caller |
-| utils/generate-happ-subscriptions | actions/utils/generate-happ-subscriptions | MOVED to daedalus proprietaries (node-specific tooling; commons keeps only shared stuff) |
-| utils/lint-docker-compose | actions/utils/lint-docker-compose | dclint --fix via docker; --hook staged-only + restage; + anchor-IP check |
-| compose | actions/compose | apex core first then services alpha; reverse on down; --dry-run; extra passthrough |
-
-## Deliberate deltas (rename + fixes)
-- root->apex everywhere (CLI, container names apex-*, telegram instance com.ermnvldmr.apex.*, ufw markers APEX.*, networks direct/enclave/socket).
-- No <node> arg; identity from FQDN + node.env.
-- backup superseded by the resticontainer migration (see below): the old whole-tier-root
-  restic snapshot is replaced by per-service, `restic.*`-label-driven backups.
-- ufw markers self-heal (delete ROOT + APEX blocks before inject).
-- envsubst -> string.Template.safe_substitute; wget/curl download -> urllib. Unset template
-  vars stay LITERAL (envsubst emptied them) — typos surface instead of silently blanking.
-- crowdsec install auto-confirms (`apt-get install -y`; bash ran promptful `apt install`,
-  which blocked on a tty and aborted non-interactive runs).
-- pre-commit staged-compose pathspec widened to `*docker-compose.yml` — the old
-  `*.docker-compose.yml`/root-anchored patterns never matched nested compose files.
-- anchor-IP validation in utils/lint-docker-compose is a planned addition (SP3 §6.2).
-- new opt-in `--dry-run` flags on configure/* have no bash counterpart; the default
-  (flag absent) path is the ported behavior.
-- ufw/crowdsec deploys keep a timestamped `.bak` next to overwritten system files
-  (bash: none / `/tmp`) as a rollback net; tiers/useradd `.env` rewrites deliberately do
-  NOT (no secret-bearing copies).
-- non-conventional tool exit codes normalize to 1 (bash sometimes propagated e.g.
-  apt-get's 100); 64/65/66 conventions preserved.
-
-## Core-compose consolidation deltas (beyond the rename)
-- icarus: traefik no longer publishes 53/tcp+udp (AdGuard DNS) from the shared file —
-  restored via an icarus-only override layered in the node repo's include (Phase 2).
-- icarus: ACME propagation-tuning flags and DEBUG log level drop to the shared defaults;
-  traefik now runs noroot (APEX_UID) with file provider + HTTP/3 like daedalus.
-- xray is profile-gated (`xray` in COMPOSE_PROFILES): daedalus + icarus opt in,
-  morpheus does not (it never ran xray — owner decision 2026-07-05).
-- loki (profile `loki`, no node opts in yet) is exposed via traefik at
-  `https://<node-fqdn>/loki` behind basicauth (`APEX_LOKI_BASIC_AUTH` in the node .env).
-- morpheus: dashboard auth changes basicauth (TRAEFIK_INTERNAL_API_AUTH) -> enclave IP
-  allowlist; the adguard router labels exist without a backend (502, was 404, on /adguard).
-- restic: per-node selective tier2 mounts replaced by whole-tier-root RO mounts
-  (superset; snapshot paths preserved); RESTIC_COMPRESSION per-node via
-  `APEX_RESTIC_COMPRESSION` (default auto).
-
-## resticontainer migration (SP4 realized)
-- The `restic` core service is replaced by `resticontainer`
-  (`ghcr.io/deytenit/resticontainer`), a restic wrapper that reads per-service
-  `restic.*` compose labels: `restic.enable`, `restic.backup.paths` (comma-separated
-  container mount destinations — matched exactly, whole mount dir backed up),
-  `restic.backup.stop`, and `restic.hooks.pre-backup`/`post-backup` (run via docker exec).
-- It mounts the docker socket (list/inspect/exec/stop/start) and `/srv:/srv:ro`
-  (`RESTIC_HOSTFS=/`); apex data + checkouts live under /srv and the `@tierN` mounts
-  are absolute symlinks into /srv, so mounting /srv at the same path lets restic
-  follow them and record the real host paths, and snapshots the union in one run.
-- The whole-tier-root RO mounts and the explicit `/data/@tier1,2` backup args are gone;
-  a service is backed up only if it carries `restic.enable=true`. DB consistency is
-  per-service: a pre-backup hook dumps into a dedicated backed-up mount (e.g. postgres
-  `pg_dump … > /dumps/dump.sql`), never the live data dir. `backup/run` now guards on a
-  fresh snapshot landing for the host (a no-label repo would otherwise be a silent no-op).

README.md

@@ -1,31 +1,172 @@
-# apex.ermnvldmr.com
+<p align="center">
+  <img alt="apex.ermnvldmr.com logo" src="./etc/assets/logo.png" height="128" />
+</p>
 
-The **commons** of the apex fleet: a stock-python engine (`apex <group>/<action>`) and
+# [apex.ermnvldmr.com](https://apex.ermnvldmr.com)
-the shared core docker-compose, consumed by every node as the `commons/` git submodule.
 
-Full documentation: [docs.ermnvldmr.com/en/apex](https://docs.ermnvldmr.com/en/apex/).
+Local infrastructure operations for independently owned nodes, with a Python standard-library core.
 
-## Layout
+## Quick Start
-- `apex` — launcher (python3, stdlib only); node `init.sh` symlinks it onto `PATH`.
-- `engine/` — the runner: CLI, identity, overlay, context, and the `ctx.*` helper library.
-- `actions/` — grouped actions; the path IS the name (`configure/ufw`, `tiers/link`, …).
-- `compositions/apex/docker-compose.yml` — the shared, `${env}`-parameterized core compose.
-- `vendor/` — pinned pure-python third-party (empty; see `vendor/VENDORED.md`).
-- `tests/` — stdlib `unittest` suite (`tests/run.sh`).
 
-Node-specific tooling lives in each node repo's `proprietaries/` — commons holds only
+- **Live Documentation**: <https://docs.ermnvldmr.com/en/apex/> (also available in [Russian](https://docs.ermnvldmr.com/ru/apex/))
-what every node shares.
+- **Getting Started Guide**: <https://docs.ermnvldmr.com/en/apex/getting-started/>
+- **Report an Issue**: <https://git.ermnvldmr.com/deytenit/apex.ermnvldmr.com/issues>
+- **License**: [BSD-2-Clause](#copyright--license)
 
-## Usage (on a node)
+From an APEX checkout on your workstation, connect to the installed Debian or Proxmox VE host as root or as a user with passwordless sudo. The workstation needs Python 3.10 or newer and OpenSSH. The script installs Python 3 through APT if the host lacks it. Verify its SSH host key through a trusted channel and record it in your workstation’s SSH known hosts before connecting.
-`apex` acts on the **local checkout** — no `<node>` argument. Identity comes from the
-host FQDN (`<node>.a<x>.apex.ermnvldmr.com`) and `node.env`.
-- `apex` / `apex --help` — list actions
-- `apex configure/ufw` — render + apply this node's ufw config
-- `apex configure` — run every `configure/*` in order
-- `apex compose up` — bring up the node's compositions
 
-## Constraints
+```sh
-git + python3 stdlib only. No pip/Node/Go/CUE. Composes are hand-authored (no codegen).
+scripts/bootstrap-node \
+  --target root@node.example.com \
+  --repo ssh://git@git.example.com:22222/infrastructure/selfhost.example.com.git \
+  --operator-key ~/.ssh/operator.pub \
+  --deploy-key ~/.ssh/selfhost-deploy \
+  --known-hosts ~/.ssh/git-known-hosts
+```
 
-## License
+The operator public key grants SSH access to `adam`, who receives passwordless sudo. Use a separate, passphrase-free SSH deploy key for this node. Register its public half as a deploy key on the Git server before running the script. The script installs the private half on the node. The key must have read access to the node repository and any private SSH submodules it needs on that Git server. Grant write access to the node repository only if you use `apex repository/sync` to push changes. Supply verified Git server host keys in OpenSSH known_hosts format, including the port for nonstandard listeners (for example, `[git.example.com]:22222`).
-MIT (see `LICENSE`).
+
+The script clones the repository under `/srv` as `adam`. Use `--node-name NAME` to choose the checkout directory under `/srv`. Before changing the SSH policy, log in with the matching operator private key on the original SSH port. Use the same host or SSH alias as the bootstrap target.
+
+```sh
+ssh -o StrictHostKeyChecking=yes -o IdentitiesOnly=yes -i ~/.ssh/operator adam@node.example.com
+```
+
+In that session, enter the checkout. If the node repository includes `init.sh`, review and run it to install the shell shortcuts. Otherwise call `./commons/apex` directly.
+
+```sh
+cd /srv/selfhost.example.com
+if [ -f ./init.sh ]; then ./init.sh && source ~/.bashrc; fi
+./commons/apex --help
+```
+
+On existing checkouts that use the APEX Git hooks, run `git config --local core.hooksPath commons/scripts/githooks` from the node repository after updating the submodule.
+
+Configure the node in its checkout. For example, on a stock Debian Docker host:
+
+```sh
+# Stock Debian Docker host, after configuring the node repository
+apex config/packages
+apex config/base
+apex config/docker
+apex config/ufw-docker
+apex config/ufw       # apply the node firewall rules
+apex composition/run up       # bring node services online
+```
+
+## Essential Information
+
+You should understand the following invariants before exploring the codebase:
+
+### Core Principles
+
+- **No Control Plane**: No central orchestrator, agent daemon, or cluster controller. Commands act locally on the node you run them on.
+- **Core Runtime**: Python 3.10 or newer and Git, with no `pip` dependencies or build step. Standalone host provisioning and workstation checks have separate requirements below.
+- **Node-Owned Repositories**: Each machine is an independent Git repository pinning `apex` as a `commons/` submodule. It declares its own identity, compositions, and local overrides.
+- **Overlay Architecture**: Node repositories can extend or override any shared action by placing an executable in `proprietaries/actions/`.
+
+### Repository Structure
+
+- **[`actions/`](./actions)**: Built-in engine actions (`composition/`, `config/`, `repository/`, `tier/`, `util/`).
+- **[`compositions/`](./compositions)**: The shared core Compose stack (`traefik`, `socket-proxy`, `alloy`, `adguard`, `xray`, `restic`).
+- **[`scripts/`](./scripts)**: Workstation bootstrap, the Python engine, host provisioning helpers, and Git hooks.
+- **[`etc/`](./etc)**: Shared configuration and static assets, including Alloy templates and the project logo.
+
+---
+
+## Who Are You?
+
+### Node Operator
+
+Running and maintaining services on an existing host.
+
+- **Requirements**: Debian-based host with Docker, Docker Compose v2, Git, and Python 3.
+- **Workflow**:
+  - Update services: `apex composition/run pull && apex composition/run up`
+  - Run backups: `apex tier/backup`
+  - Sync repository state: `apex repository/sync`
+
+Target a composition explicitly when operating on one application:
+
+```sh
+apex composition/run -p apex restart traefik
+apex composition/run -p app up -d
+apex composition/run --dry-run -p apex logs --tail 20 xray
+```
+
+The existing `apex compose` spelling is an alias for `apex composition/run`.
+Wrapper options precede the action; remaining arguments go to Docker Compose.
+The positional forms `apex composition/run restart apex traefik` and `apex composition/run up -d apex`
+remain supported. Use `-p` when service names or option values could be mistaken
+for composition names. Without a target, APEX visits core first, then other
+compositions alphabetically; `down` reverses that order.
+
+Replace `app` with a composition defined in your repository. Use targeted `up -d`
+after changing a container definition; `restart` retains the existing definition.
+
+For direct Docker Compose use in the core composition:
+
+```sh
+apex config/env --sync
+cd compositions/apex
+docker compose config --quiet
+docker compose restart traefik
+```
+
+Sync preserves existing `.env` credentials and UID/GID entries, including tier
+symlinks. It updates a managed block containing `apex.env` settings (including
+profiles) followed by synthesized node/path values, and writes with mode `0600`.
+Resync after changing `apex.env`, node identity, or tier paths. Shell environment
+values still take precedence in Docker Compose. Do not edit the generated block.
+
+`eval "$(apex config/env)"` exports synthesized APEX values to the current shell;
+`apex config/env --dotenv` prints dotenv syntax, and `apex config/env --write generated.env`
+writes those values to a separate file. These three modes do not merge the
+composition's secrets or `apex.env`; use `--sync` for the complete core bridge.
+Choose one output mode per invocation.
+
+### Preparing an existing host
+
+Run each setup action explicitly from the node checkout. Use `apex --help` to list the shared and node-owned actions. New actions follow `<singular-entity>/<action>`. Use `apex <entity>/<action> --help` for the arguments.
+
+On an installed Proxmox VE host, `config/packages` installs the administrator tools and `config/base` applies APT and journald policy. The PVE host keeps its own storage, networking, firewall, and management listeners. Run workload configuration inside the guest that owns the service.
+
+Before moving SSH to port 2222, allow inbound TCP/2222 from the operator’s address through the provider firewall and the node’s firewall (UFW on stock Debian or the PVE firewall). Keep the original administrator session open throughout the transition.
+
+1. Run `./commons/apex config/hardening begin` from the node checkout in the original SSH session.
+2. On your workstation, run `ssh -o ControlPath=none -o BatchMode=yes -p 2222 adam@node.example.com` using the operator key. In the new session, run `./commons/apex config/hardening finalize` from the checkout.
+3. Run the same SSH command again from your workstation. From the checkout in this second fresh session, run `./commons/apex config/hardening commit` to cancel automatic rollback.
+
+On stock Debian the final policy allows only `adam` on port 2222 using a public key. PVE retains ports 22 and 2222, requires public keys globally, and keeps root key access with `PermitRootLogin prohibit-password` and no `AllowUsers` restriction.
+
+If a login fails, run `./commons/apex config/hardening rollback` from the checkout in the original session or let the timer restore the original SSH configuration five minutes after `begin`. Use `./commons/apex config/hardening status` to inspect the transition. [Action reference](https://docs.ermnvldmr.com/en/apex/reference/actions/#confighardening)
+
+### Developer & Contributor
+
+Adding shared actions or improving the engine.
+
+- **Conventions**: Maintain strict Python standard library compliance. No external runtime packages.
+- **Public documentation**: Describe supported interfaces using synthetic examples. Keep deployment inventories, internal plans, operational runbooks, and host-specific configuration in their owning private repositories.
+- **Setup**:
+  ```sh
+  python3 -m venv .venv
+  . .venv/bin/activate
+  python3 -m pip install ruff==0.17.2
+  ```
+- **Verification**: `make check` runs lint and the core and host provisioning suites. `make test` (or `bash tests/run.sh`) runs both suites. Use `PYTHONPATH=scripts` when running individual test modules. Environment integration tests use `docker compose` or standalone `docker-compose` with `config`; no daemon is needed.
+- **Linting**: Ruff checks correctness and security rules. Existing import-order, line-length, compact-statement, and annotation-style debt is explicitly deferred in `pyproject.toml` to avoid unrelated formatting changes. Without Ruff, `make lint` recursively checks Python compilation and fails on syntax errors; that fallback is not equivalent to Ruff. `make fmt` formats files and is a separate, deliberate operation.
+
+---
+
+## Communication & Support
+
+- **Issues**: <https://git.ermnvldmr.com/deytenit/apex.ermnvldmr.com/issues>
+- **Email**: <personal@ermnvldmr.com>
+
+## Copyright & License
+
+Licensed under the [BSD-2-Clause License](./LICENSE).
+
+---
+
+**AI Assistance**: Portions of this project's source code and configuration may have been developed with the assistance of AI code-generation tools.

actions/compose.py

@@ -1,47 +0,0 @@
-# actions/compose.py
-"""Port of compose — orchestrate docker compose across the node. apex core first
-(owns networks), then services alpha; reverse on down. Exports derived APEX_* vars and
-layers --env-file (.env secrets + apex.env committed scalars)."""
-import os
-from engine.descriptor import Meta, Arg, Flag, Rest
-
-METADATA = Meta(summary="Orchestrate docker compose across the node (apex core first, services next).",
-                args=[Arg("action", "up | down | restart | ..."),
-                      Flag("--dry-run", "Print commands only."),
-                      Rest("extra", "Extra args passed through to docker compose.")])
-
-def run(ctx, args):
-    log, comp, action = ctx.log, ctx.paths.compositions, args.action
-    # argparse.REMAINDER swallows flags placed after the action, so strip --dry-run
-    # out of the passthrough args like bash did ("apex compose up --dry-run" must
-    # preview, not hand docker compose its own --dry-run).
-    extra = [a for a in (args.extra or []) if a != "--dry-run"]
-    dry_run = args.dry_run or len(extra) != len(args.extra or [])
-    env = dict(os.environ)
-    env.update({k: v for k, v in ctx.vars().items() if k.startswith("APEX_")})
-
-    def sequence():
-        core = os.path.join(comp, "apex")
-        others = sorted(
-            os.path.join(comp, d) for d in os.listdir(comp)
-            if os.path.isdir(os.path.join(comp, d)) and not d.startswith((".", "@")) and d != "apex"
-            and os.path.isfile(os.path.join(comp, d, "docker-compose.yml")))
-        seq = ([core] if os.path.isfile(os.path.join(core, "docker-compose.yml")) else []) + others
-        return list(reversed(seq)) if action == "down" else seq
-
-    def run_one(dirpath):
-        name = os.path.basename(dirpath)
-        base = ["docker", "compose"]
-        for ef in (".env", "apex.env"):
-            if os.path.isfile(os.path.join(dirpath, ef)):
-                base += ["--env-file", ef]
-        cmd = base + (["up", "-d"] if action == "up" else [action]) + extra
-        if dry_run:
-            log.info(f"[DRY-RUN] (cd {dirpath} && {' '.join(cmd)})")
-        else:
-            log.info(f"{action} {name}...")
-            ctx.sys.run(cmd, cwd=dirpath, env=env, check=True)
-
-    for d in sequence():
-        run_one(d)
-    log.success(f"Orchestrated {action}.")

actions/composition/run.py

@@ -0,0 +1,98 @@
+# actions/composition/run.py
+"""Orchestrate docker compose across the node.
+
+Supports targeting all compositions in sequence (default: apex core first, services next;
+reverse on down) or targeting one with `apex composition/run -p apex restart traefik`.
+Wrapper options precede the action. Exports derived APEX_* vars and layers env files.
+"""
+import os
+import shlex
+
+from engine.descriptor import Flag, Meta, Opt, Rest
+
+METADATA = Meta(
+    summary="Orchestrate docker compose across the node or on a specific composition.",
+    args=[
+        Opt("-p", "Target composition name or 'all'.", default=None),
+        Flag("--dry-run", "Print commands only."),
+        Rest("command", "ACTION [COMPOSE_ARGS...]; wrapper options must precede ACTION."),
+    ],
+)
+
+
+def run(ctx, args):
+    log, comp = ctx.log, ctx.paths.compositions
+    command = list(args.command)
+    if command and command[0] == "--":
+        command.pop(0)
+    if not command:
+        log.error("ACTION is required (up, down, restart, logs, ...).")
+        raise SystemExit(64)
+    action, extra = command[0], command[1:]
+    dry_run = args.dry_run
+    before_separator = extra[:extra.index("--")] if "--" in extra else extra
+    if any(arg in ("-p", "--dry-run") or arg.startswith("-p=")
+           or arg.startswith("--dry-run=") for arg in before_separator):
+        log.error("Wrapper options -p and --dry-run must precede ACTION; use -- for passthrough.")
+        raise SystemExit(64)
+    env = dict(os.environ)
+    env.update({k: v for k, v in ctx.vars().items() if k.startswith("APEX_")})
+
+    available = []
+    if os.path.isdir(comp):
+        available = [
+            d for d in os.listdir(comp)
+            if os.path.isdir(os.path.join(comp, d))
+            and not d.startswith((".", "@"))
+            and os.path.isfile(os.path.join(comp, d, "docker-compose.yml"))
+        ]
+
+    target = getattr(args, "p", None)
+    if target is None and extra:
+        if extra[0] in ["all", *available]:
+            target, extra = extra[0], extra[1:]
+        elif action == "up" and len(extra) >= 2 and extra[0] == "-d":
+            # Preserve the documented `up -d apex` spelling without scanning flag values.
+            if extra[1] in ["all", *available]:
+                if len(extra) != 2:
+                    log.error("Use -p COMPOSITION before up when adding service arguments.")
+                    raise SystemExit(64)
+                target, extra = extra[1], ["-d"]
+
+    if target is None:
+        target = "all"
+
+    if target != "all" and target not in available:
+        names = ", ".join(sorted(available)) if available else "none"
+        log.error(f"Unknown composition '{target}'. Available: {names}")
+        raise SystemExit(1)
+
+    def sequence():
+        if target != "all":
+            return [os.path.join(comp, target)]
+        core = os.path.join(comp, "apex")
+        others = sorted(os.path.join(comp, d) for d in available if d != "apex")
+        seq = ([core] if "apex" in available else []) + others
+        return list(reversed(seq)) if action == "down" else seq
+
+    def run_one(dirpath):
+        name = os.path.basename(dirpath)
+        base = ["docker", "compose"]
+        for ef in (".env", "apex.env"):
+            if os.path.isfile(os.path.join(dirpath, ef)):
+                base += ["--env-file", ef]
+        if action == "up":
+            up_flags = [] if any(flag in extra for flag in ("-d", "--detach")) else ["-d"]
+            cmd = base + ["up"] + up_flags + extra
+        else:
+            cmd = base + [action] + extra
+
+        if dry_run:
+            log.info(f"[DRY-RUN] (cd {shlex.quote(dirpath)} && {shlex.join(cmd)})")
+        else:
+            log.info(f"{action} {name}...")
+            ctx.sys.run(cmd, cwd=dirpath, env=env, check=True)
+
+    for d in sequence():
+        run_one(d)
+    log.success(f"Orchestrated {action} ({target}).")

actions/config/alloy.py

@@ -0,0 +1,77 @@
+"""Deploy the opt-in native Alloy service using shared host/guest configs."""
+import json
+from pathlib import Path
+import re
+import tempfile
+
+from engine.descriptor import Flag, Meta
+
+METADATA = Meta(summary="Configure native Alloy from configs/alloy/config.json.",
+                args=[Flag("--dry-run", "Validate the deployment plan without changes.")])
+
+
+def deployment(ctx):
+    settings = Path(ctx.paths.configs) / "alloy/config.json"
+    if not settings.exists():
+        return None
+    values = json.loads(settings.read_text())
+    role = values.get("role")
+    if role not in ("host", "guest"):
+        raise ValueError("Alloy role must be host or guest")
+    env_file = values.get("environment_file", "")
+    if not re.fullmatch(r"/[A-Za-z0-9_./-]+", env_file):
+        raise ValueError("Alloy environment_file must be an absolute path without whitespace")
+    source = Path(ctx.paths.commons) / "etc/alloy" / (
+        "native-host.alloy" if role == "host" else "native.alloy")
+    config = source.read_text()
+    extra = Path(ctx.paths.configs) / "alloy/extra.alloy"
+    if extra.is_file():
+        config += "\n" + extra.read_text()
+    unit = ("[Service]\nEnvironmentFile=" + env_file + "\n"
+            'ExecStart=\nExecStart=/usr/bin/alloy run --storage.path=/var/lib/alloy --server.http.listen-addr=127.0.0.1:12346 /etc/alloy/config.alloy\n'
+            + ("User=root\nGroup=root\n" if role == "guest" else "") +
+            "SupplementaryGroups=systemd-journal" + (" docker" if role == "guest" else "") + "\n")
+    return config, unit, env_file
+
+
+def run(ctx, args):
+    plan = deployment(ctx)
+    if plan is None:
+        ctx.log.info("Native Alloy is not configured for this node; skipping.")
+        return
+    config, unit, env_file = plan
+    if args.dry_run:
+        ctx.log.info("Native Alloy deployment plan validated; no changes made.")
+        return
+    ctx.sys.sudo(["test", "-f", env_file])
+    # Validate with the actual service environment without printing credentials.
+    validator = '''import os, stat, subprocess, sys
+from pathlib import Path
+# Environment files deliberately use plain KEY=value, with no shell evaluation.
+secret = Path(sys.argv[1])
+info = secret.stat()
+if info.st_uid != 0 or stat.S_IMODE(info.st_mode) & 0o077:
+    raise RuntimeError('Alloy environment file must be root-owned with mode 0600 or stricter')
+for line in secret.read_text().splitlines():
+    if not line.strip() or line.lstrip().startswith('#'): continue
+    key, value = line.split('=', 1)
+    os.environ[key] = value
+subprocess.run(['alloy', 'validate', sys.argv[2]], check=True)
+'''
+    with tempfile.TemporaryDirectory(prefix="apex-alloy-") as directory:
+        config_path = Path(directory) / "config.alloy"
+        unit_path = Path(directory) / "apex.conf"
+        config_path.write_text(config)
+        unit_path.write_text(unit)
+        ctx.sys.sudo(["python3", "-c", validator, env_file, str(config_path)])
+        ctx.sys.sudo(["install", "-d", "-m", "0755", "/etc/alloy",
+                      "/etc/systemd/system/alloy.service.d"])
+        ctx.sys.sudo(["install", "-m", "0644", str(config_path), "/etc/alloy/config.alloy"])
+        ctx.sys.sudo(["install", "-m", "0644", str(unit_path),
+                      "/etc/systemd/system/alloy.service.d/apex.conf"])
+    ctx.sys.sudo(["systemctl", "unmask", "alloy.service"])
+    ctx.sys.daemon_reload()
+    ctx.sys.sudo(["systemctl", "enable", "alloy.service"])
+    ctx.sys.sudo(["systemctl", "restart", "alloy.service"])
+    ctx.sys.sudo(["systemctl", "is-active", "--quiet", "alloy.service"])
+    ctx.log.success("Configured native Alloy.")

actions/config/base.py

@@ -0,0 +1,40 @@
+"""Apply the common APT archive and journald retention policy."""
+from pathlib import Path
+from engine.descriptor import Meta
+
+METADATA = Meta(summary="Configure APT archive cleanup and journald retention.")
+
+
+def _configure(ctx, path, legacy, content, restart=False):
+    for target in (path, legacy):
+        if target.is_symlink():
+            raise RuntimeError(f"Refusing redirected configuration: {target}")
+    previous = path.read_text() if path.exists() else None
+    backup = ctx.host.backup(str(legacy), sudo=True) if legacy.exists() else None
+    if previous == content and not backup:
+        return
+    if previous != content:
+        ctx.host.write_file(str(path), content, sudo=True)
+    if backup:
+        ctx.sys.sudo(["rm", "--", str(legacy)])
+    if restart:
+        try:
+            ctx.sys.sudo(["systemctl", "restart", "systemd-journald"])
+        except Exception:
+            if backup:
+                ctx.sys.sudo(["cp", "--preserve=all", "--", backup, str(legacy)])
+            if previous is None:
+                ctx.sys.sudo(["rm", "--", str(path)])
+            elif previous != content:
+                ctx.host.write_file(str(path), previous, backup=False, sudo=True)
+            raise
+
+
+def run(ctx, args):
+    if not Path("/etc/debian_version").is_file():
+        raise RuntimeError("Debian required.")
+    apt = Path("/etc/apt/apt.conf.d")
+    journal = Path("/etc/systemd/journald.conf.d")
+    _configure(ctx, apt / "99-apex", apt / "01clean", 'APT::Keep-Downloaded-Packages "0";\n')
+    _configure(ctx, journal / "99-apex.conf", journal / "maxuse.conf",
+               "[Journal]\nSystemMaxUse=200M\n", restart=True)

actions/config/cron.py

@@ -1,5 +1,5 @@
-# actions/configure/cron.py
+# actions/config/cron.py
-"""Port of configure/cron — install crontab from configs/cron/crontab."""
+"""Port of config/cron — install crontab from configs/cron/crontab."""
 import os
 from engine.descriptor import Meta, Flag
 

actions/config/crowdsec.py

@@ -1,5 +1,5 @@
-# actions/configure/crowdsec.py
+# actions/config/crowdsec.py
-"""Port of configure/crowdsec — install, deploy /etc/crowdsec, collections, bouncer."""
+"""Port of config/crowdsec — install, deploy /etc/crowdsec, collections, bouncer."""
 import os
 from engine.descriptor import Meta, Flag
 

actions/config/docker.py

@@ -0,0 +1,12 @@
+"""Install a local Docker Engine, Compose and Buildx on Debian 13."""
+from pathlib import Path
+
+from engine.descriptor import Meta
+
+METADATA = Meta(summary="Verify a working Docker toolchain or install it on stock Debian 13.")
+
+
+def run(ctx, args):
+    provisioning = Path(ctx.paths.commons) / "scripts"
+    ctx.sys.sudo(["python3", "-E", "-s", "-m", "provisioning.docker"],
+                 cwd=str(provisioning))

actions/config/env.py

@@ -0,0 +1,82 @@
+"""Export synthesized APEX values or bridge a composition's environment for Compose."""
+import os
+import shlex
+
+from engine.descriptor import Flag, Meta, Opt
+
+METADATA = Meta(
+    summary="Export or sync synthesized APEX_* variables for shell or docker compose.",
+    args=[
+        Flag("--export", "Format as shell export statements (default)."),
+        Flag("--dotenv", "Print quoted generated APEX_* dotenv values only."),
+        Opt("--write", "Write a standalone generated dotenv file (0600).", default=None),
+        Flag("--sync", "Preserve user .env text and refresh the core Compose managed block."),
+    ],
+)
+
+_START = "# BEGIN APEX MANAGED ENV"
+_END = "# END APEX MANAGED ENV"
+
+
+def _dotenv_value(value):
+    """Quote a literal using Compose dotenv escapes, including interpolation dollars."""
+    value = str(value)
+    for old, new in (("\\", "\\\\"), ('"', '\\"'), ("$", "$$"),
+                     ("\r", "\\r"), ("\n", "\\n"), ("\t", "\\t")):
+        value = value.replace(old, new)
+    return f'"{value}"'
+
+
+def _read_optional(path):
+    try:
+        with open(path, encoding="utf-8", newline="") as source:
+            return source.read()
+    except FileNotFoundError:
+        return ""
+
+
+def _sync_body(existing, public, generated):
+    if any(marker in public or marker in generated for marker in (_START, _END)):
+        raise ValueError("Managed environment marker collides with source content.")
+    lines = existing.splitlines(keepends=True)
+    markers = [(index, line.rstrip("\r\n")) for index, line in enumerate(lines)
+               if _START in line or _END in line]
+    if markers:
+        if (len(markers) != 2 or markers[0][1] != _START or markers[1][1] != _END):
+            raise ValueError("Malformed or duplicate managed environment markers; .env unchanged.")
+        start, end = markers[0][0], markers[1][0]
+        existing = "".join(lines[:start] + lines[end + 1:])
+    # Keep user bytes in order, but place managed values last to preserve precedence.
+    separator = "" if not existing or existing.endswith("\n") else "\n"
+    public_separator = "" if not public or public.endswith("\n") else "\n"
+    return (existing + separator + _START + "\n" + public + public_separator
+            + generated + _END + "\n")
+
+
+def run(ctx, args):
+    modes = (args.export, args.dotenv, args.write is not None, args.sync)
+    if sum(bool(mode) for mode in modes) > 1:
+        ctx.log.error("Choose only one of --export, --dotenv, --write, or --sync.")
+        raise SystemExit(64)
+    variables = {k: v for k, v in ctx.vars().items() if k.startswith("APEX_")}
+    generated = "".join(f"{key}={_dotenv_value(value)}\n"
+                        for key, value in sorted(variables.items()))
+    target = args.write
+    if args.sync:
+        core = os.path.join(ctx.paths.compositions, "apex")
+        target = os.path.join(core, ".env")
+        try:
+            generated = _sync_body(_read_optional(target),
+                                   _read_optional(os.path.join(core, "apex.env")), generated)
+        except ValueError as error:
+            ctx.log.error(str(error))
+            raise SystemExit(65) from error
+    if target is not None:
+        ctx.host.write_file(target, generated, mode="0600")
+        ctx.log.success(f"Wrote {len(variables)} synthesized variables to {target}.")
+    elif args.dotenv:
+        print(generated, end="")
+    else:
+        for key, value in sorted(variables.items()):
+            print(f"export {key}={shlex.quote(str(value))}")
+    return 0

actions/config/hardening.py

@@ -0,0 +1,189 @@
+"""Hand SSH access over to fresh, verified adam sessions on port 2222."""
+import ipaddress
+import json
+import os
+import platform
+import pwd
+import re
+import time
+import uuid
+from pathlib import Path
+from engine.descriptor import Arg, Meta, Opt
+
+METADATA = Meta(summary="Manage the protected SSH handover to port 2222.", args=[
+    Arg("operation", "begin (default), finalize, commit, rollback, or status", required=False),
+    Opt("--original-port", "Current SSH port when beginning from a console"),
+    Opt("--client-address", "SSH client IP when beginning from a console"),
+])
+STATE = "/var/lib/apex-init/access/manual-state.json"
+BUNDLES = Path("/usr/local/lib/apex-provisioner")
+
+
+def _ancestors():
+    ancestors = {}
+    pid = os.getppid()
+    ticks = os.sysconf("SC_CLK_TCK")
+    # proc starttime includes suspend time, while journald monotonic timestamps do not.
+    suspend_us = (time.clock_gettime_ns(time.CLOCK_BOOTTIME) - time.monotonic_ns()) // 1000
+    while pid > 1:
+        # comm can contain spaces and parentheses; the remaining fields start at state.
+        fields = Path(f"/proc/{pid}/stat").read_text().rsplit(")", 1)[1].split()
+        ancestors[pid] = int(fields[19]) * 1_000_000 // ticks - max(0, suspend_us)
+        pid = int(fields[1])
+    return ancestors
+
+
+def _socket(ctx, ancestors):
+    sockets = ctx.sys.sudo(["ss", "-Hntp", "state", "established"], capture=True).stdout
+    connections = set()
+    for line in sockets.splitlines():
+        fields = line.split(None, 4)
+        if len(fields) == 5 and any(int(pid) in ancestors for pid in
+                                    re.findall(r'"sshd(?:-session)?",pid=(\d+),', fields[4])):
+            connections.add((fields[2], fields[3]))
+    if not connections:
+        return None
+    if len(connections) != 1:
+        raise RuntimeError("Cannot identify a unique established SSH socket for this process.")
+    local, peer = connections.pop()
+    peer, peer_port = peer.rsplit(":", 1)
+    local, local_port = local.rsplit(":", 1)
+    def address(value):
+        ip = ipaddress.ip_address(value.strip("[]"))
+        return str(getattr(ip, "ipv4_mapped", None) or ip)
+    peer, local = address(peer), address(local)
+    return local, int(local_port), peer, int(peer_port)
+
+
+def _connection(ctx, *, after=None):
+    if pwd.getpwuid(os.getuid()).pw_name != "adam":
+        raise RuntimeError("Run this step inside a fresh SSH public-key session as adam.")
+    ancestors = _ancestors()
+    endpoint = _socket(ctx, ancestors)
+    if not endpoint:
+        raise RuntimeError("Cannot identify the SSH connection for this process.")
+    local, local_port, peer, peer_port = endpoint
+    if after is not None and local_port != 2222:
+        raise RuntimeError("Open a fresh adam SSH connection on port 2222.")
+    record = rf"^Accepted publickey for adam from {re.escape(peer)} port {int(peer_port)} ssh2:"
+    result = ctx.sys.sudo(["journalctl", "-b", "_UID=0", "_EXE=/usr/sbin/sshd",
+                           "_EXE=/usr/lib/openssh/sshd-session", "_EXE=/usr/lib/openssh/sshd-auth",
+                           "--no-pager", "-o", "json", "--grep", record], capture=True, check=False)
+    if not result.returncode:
+        for line in result.stdout.splitlines():
+            entry = json.loads(line)
+            pid = int(entry["_PID"])
+            if (pid in ancestors and int(entry["__MONOTONIC_TIMESTAMP"]) >= ancestors[pid]
+                    and (after is None or ancestors[pid] > after)
+                    and re.match(record, entry["MESSAGE"])):
+                ctx.sys.run(["sudo", "-n", "true"])
+                return f"user=adam,host={peer},addr={peer},laddr={local},lport={int(local_port)}"
+    raise RuntimeError("Cannot verify a fresh public-key login for this SSH process in the root sshd journal.")
+
+
+def _operator_access(ctx):
+    account = pwd.getpwnam("adam")
+    if account.pw_uid < 1000 or account.pw_dir != "/home/adam":
+        raise RuntimeError("Prepare the adam account from your workstation with scripts/bootstrap-node first.")
+    if pwd.getpwuid(os.getuid()).pw_name == "adam":
+        return _connection(ctx)
+    loginuid = int(Path("/proc/self/loginuid").read_text())
+    if loginuid == account.pw_uid or (loginuid == 4294967295 and os.getuid() != 0) or loginuid < 0:
+        raise RuntimeError("Use an independent administrator login or an SSH public-key session as adam.")
+    if os.getuid() not in (0, loginuid):
+        raise RuntimeError("Run from the independent administrator's own login.")
+    ctx.sys.sudo(["true"])
+    home = Path(account.pw_dir)
+    authorized = home / ".ssh" / "authorized_keys"
+    # Inspect as adam so private .ssh directories remain usable from another administrator.
+    command = ["runuser", "-u", "adam", "--"]
+    for path in (home, authorized.parent, authorized):
+        metadata = ctx.sys.sudo([*command, "env", "LC_ALL=C", "stat", "-c", "%u %a %F", "--", str(path)],
+                                capture=True).stdout.strip().split(" ", 2)
+        owner, mode, kind = metadata
+        expected = "regular file" if path == authorized else "directory"
+        if int(owner) not in (0, account.pw_uid) or int(mode, 8) & 0o022 or kind != expected:
+            raise RuntimeError(f"Unsafe operator SSH path: {path}")
+    keys = ctx.sys.sudo([*command, "cat", "--", str(authorized)], capture=True).stdout
+    for key in keys.splitlines():
+        if key.startswith(("ssh-", "ecdsa-", "sk-")) and ctx.sys.run(
+            ["ssh-keygen", "-lf", "-"], input=key + "\n", capture=True, check=False,
+        ).returncode == 0:
+            break
+    else:
+        raise RuntimeError("Install a valid unrestricted operator public key before applying SSH hardening.")
+    ctx.sys.sudo([*command, "sudo", "-n", "true"])
+    connection = "user=adam,host=localhost,addr=127.0.0.1"
+    paths = _effective(ctx.sys, connection).get("authorizedkeysfile", "").split()
+    if not {".ssh/authorized_keys", "%h/.ssh/authorized_keys", str(authorized)}.intersection(paths):
+        raise RuntimeError("sshd does not use the operator's verified authorized_keys file.")
+    return connection
+
+
+def _effective(system, connection):
+    result = system.sudo(["/usr/sbin/sshd", "-T", "-C", connection], capture=True)
+    return dict(line.split(" ", 1) for line in result.stdout.splitlines() if " " in line)
+
+
+def _state(system):
+    if system.sudo(["test", "-f", STATE], check=False).returncode:
+        return {}
+    return json.loads(system.sudo(["cat", STATE], capture=True).stdout)
+
+
+def _original_endpoint(ctx, args):
+    endpoint = _socket(ctx, _ancestors())
+    port = getattr(args, "original_port", None)
+    address = getattr(args, "client_address", None)
+    if endpoint:
+        if port or address:
+            raise ValueError("SSH sessions detect the peer address and port automatically.")
+        return endpoint[1], endpoint[2]
+    if not port or not address or not 1 <= int(port) <= 65535:
+        raise ValueError("From a console, supply --original-port and --client-address for the SSH client.")
+    return int(port), str(ipaddress.ip_address(address))
+
+
+def run(ctx, args):
+    operation = getattr(args, "operation", None) or "begin"
+    if operation not in ("begin", "finalize", "commit", "rollback", "status"):
+        raise ValueError("Choose begin, finalize, commit, rollback, or status.")
+    if operation == "begin":
+        _operator_access(ctx)
+    state = _state(ctx.sys)
+    phase = state.get("status", "absent")
+    if operation == "status" and phase == "absent":
+        print("absent")
+        return
+    if operation == "begin" and phase in ("transition", "final", "recovery_pending"):
+        raise RuntimeError(f"Handover is {phase}; inspect status, continue from a fresh SSH login, or roll back.")
+    if operation in ("finalize", "commit"):
+        required = "transition" if operation == "finalize" else "final"
+        if phase != required:
+            raise RuntimeError(f"{operation} requires phase {required}; current phase is {phase}.")
+        _connection(ctx, after=state["authenticated_after_us"])
+    profile = state.get("profile") if operation != "begin" else None
+    if not profile:
+        if Path("/usr/bin/pveversion").is_file():
+            profile = "proxmox"
+        elif platform.freedesktop_os_release().get("ID") == "debian":
+            profile = "debian"
+        else:
+            raise RuntimeError("SSH hardening supports Debian and Proxmox VE.")
+    options = ["--profile", profile]
+    if operation == "begin":
+        port, address = _original_endpoint(ctx, args)
+        options.extend(["--original-port", str(port), "--client-address", address, "--token", uuid.uuid4().hex])
+        installer = Path(__file__).resolve().parents[2] / "scripts/provisioning/manual_bundle.py"
+        bundle = ctx.sys.sudo(["/usr/bin/python3", "-I", "-B", str(installer)], capture=True).stdout.strip()
+    else:
+        if not state:
+            raise RuntimeError("No SSH handover has started.")
+        bundle = state["bundle"]
+        options.extend(["--token", state["token"]])
+    if not re.fullmatch(re.escape(str(BUNDLES)) + "/[0-9a-f]{64}", bundle):
+        raise RuntimeError("Invalid installed rollback bundle location.")
+    command = ["/usr/bin/python3", "-I", "-c",
+               "import sys; sys.path.insert(0, sys.argv.pop(1)); from provisioning.access import main; sys.exit(main())",
+               bundle, "tick" if operation == "status" else operation, *options]
+    ctx.sys.sudo(command)

actions/config/packages.py

@@ -0,0 +1,50 @@
+"""Install common Debian administrator tools without replacing host packages."""
+import re
+from pathlib import Path
+from engine.descriptor import Meta
+
+METADATA = Meta(summary="Install Debian administrator packages and enable rsyslog.")
+BASE_PACKAGES = ("curl", "wget", "git", "rsyslog", "ufw")
+
+
+ADMIN_PACKAGES = (
+    "tcpdump", "socat", "rsync", "bind9-dnsutils", "iproute2", "net-tools",
+    "traceroute", "ca-certificates", "jq", "sed", "gawk", "tar", "gzip",
+    "bzip2", "xz-utils", "unzip", "htop", "iotop", "strace", "lsof",
+    "sysstat", "parted", "fdisk", "pciutils", "usbutils", "smartmontools",
+    "tmux", "vim", "less", "tree", "bash-completion", "python3",
+    "python3-minimal", "python3-yaml", "sudo", "cron", "iptables", "ipset",
+    "debian-archive-keyring", "kmod",
+)
+
+def _install_missing(s, packages):
+    missing = []
+    for package in packages:
+        result = s.run(["dpkg-query", "-W", "-f=${Status}", package],
+                       check=False, capture=True)
+        if result.returncode or result.stdout.split()[1:] != ["ok", "installed"]:
+            missing.append(package)
+    if not missing:
+        return
+    apt = ["env", "LC_ALL=C", "DEBIAN_FRONTEND=noninteractive", "apt-get",
+           "-o", "DPkg::Lock::Timeout=60", "-o", "Acquire::Retries=2",
+           "-o", "Acquire::http::Timeout=30", "-o", "Acquire::https::Timeout=30"]
+    s.sudo([*apt, "update", "--error-on=any"])
+    command = [*apt, "install", "-y", "--no-install-recommends", "--no-upgrade",
+               "--no-remove", *missing]
+    plan = s.sudo([*command, "--simulate"], capture=True)
+    if any(line.startswith("Remv ") or re.match(r"^Inst \S+ \[", line)
+           for line in plan.stdout.splitlines()):
+        raise RuntimeError("Missing packages require upgrades/removals; reconcile packages first.")
+    s.sudo(command)
+
+
+def run(ctx, args):
+    if not Path("/etc/debian_version").is_file():
+        raise RuntimeError("Debian required.")
+    packages = (*BASE_PACKAGES, *ADMIN_PACKAGES)
+    if Path("/usr/bin/pveversion").exists():
+        packages = tuple(package for package in packages if package != "ufw")
+    _install_missing(ctx.sys, packages)
+    if not ctx.sys.start_and_enable("rsyslog"):
+        raise RuntimeError("rsyslog failed to start.")

actions/config/routing.py

@@ -1,5 +1,5 @@
-# actions/configure/routing.py
+# actions/config/routing.py
-"""Port of configure/routing — idempotent node routing (ip_forward, fwmark rule, table 100)."""
+"""Port of config/routing — idempotent node routing (ip_forward, fwmark rule, table 100)."""
 from engine.descriptor import Meta
 
 METADATA = Meta(summary="Idempotent node-level routing rules and routes.")

actions/config/systemd.py

@@ -1,5 +1,5 @@
-# actions/configure/systemd.py
+# actions/config/systemd.py
-"""Port of configure/systemd — deploy + enable units from configs/systemd."""
+"""Port of config/systemd — deploy + enable units from configs/systemd."""
 import os
 from engine.descriptor import Meta, Flag
 

actions/config/timezone.py

@@ -0,0 +1,33 @@
+"""Apply the optional node timezone before installing scheduled jobs."""
+from pathlib import Path
+from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
+
+from engine.descriptor import Meta, Flag
+
+METADATA = Meta(summary="Apply configs/timezone and refresh cron when the timezone changes.",
+                args=[Flag("--dry-run", "Validate and preview without changing the host.")])
+
+
+def run(ctx, args):
+    source = Path(ctx.paths.configs) / "timezone"
+    if not source.is_file():
+        ctx.log.info("No configs/timezone declaration; preserving host timezone.")
+        return
+    zone = source.read_text().strip()
+    try:
+        ZoneInfo(zone)
+    except (ValueError, ZoneInfoNotFoundError):
+        ctx.log.error("configs/timezone must name an installed IANA timezone.")
+        raise SystemExit(1) from None
+    current = ctx.sys.run(["timedatectl", "show", "--property=Timezone", "--value"], capture=True).stdout.strip()
+    if current == zone:
+        ctx.log.info(f"Timezone already set to {zone}.")
+        return
+    if args.dry_run:
+        ctx.log.info(f"[DRY-RUN] would change timezone from {current} to {zone} and refresh active cron.")
+        return
+    ctx.sys.sudo(["timedatectl", "set-timezone", zone])
+    if ctx.sys.service_is_active("cron") and not ctx.sys.restart("cron"):
+        ctx.log.error("Timezone changed, but cron did not become active after restart.")
+        raise SystemExit(1)
+    ctx.log.success(f"Timezone set to {zone}.")

actions/config/ufw-docker.py

@@ -0,0 +1,65 @@
+"""Install the pinned ufw-docker integration on Docker hosts."""
+import hashlib
+import os
+from pathlib import Path
+import stat
+import urllib.request
+from engine.descriptor import Meta
+
+METADATA = Meta(summary="Install the pinned Docker integration for UFW.")
+UFW_DOCKER_REVISION = "020a8699f95592561f254d8d4ad1bb40d401dfc7"
+UFW_DOCKER_SHA256 = "643e56b080567c567b4aa28650196849b2a2da5dd0473fd3e5216b0886035ab0"
+UFW_DOCKER_URL = (
+    "https://raw.githubusercontent.com/chaifeng/ufw-docker/"
+    + UFW_DOCKER_REVISION + "/ufw-docker"
+)
+UFW_DOCKER_PATH = Path("/usr/local/bin/ufw-docker")
+def _check_helper_path(path):
+    for parent in reversed(path.parents):
+        info = parent.lstat()
+        if (not stat.S_ISDIR(info.st_mode) or info.st_uid != 0 or
+                info.st_mode & 0o022):
+            raise RuntimeError(f"Unsafe ufw-docker ancestor: {parent}")
+    try:
+        info = path.lstat()
+    except FileNotFoundError:
+        return False
+    if (not stat.S_ISREG(info.st_mode) or info.st_uid != 0 or info.st_gid != 0 or
+            info.st_mode & 0o022 or info.st_nlink != 1):
+        raise RuntimeError(f"Unsafe ufw-docker target: {path}")
+    return True
+
+
+def _install_helper(ctx):
+    path = UFW_DOCKER_PATH
+    exists = _check_helper_path(path)
+    payload = path.read_bytes() if exists else None
+    if payload is None or hashlib.sha256(payload).hexdigest() != UFW_DOCKER_SHA256:
+        ctx.log.info(f"Downloading pinned ufw-docker helper to {path}...")
+        # The immutable HTTPS URL is authenticated by the digest below.
+        with urllib.request.urlopen(UFW_DOCKER_URL, timeout=30) as response:  # noqa: S310
+            payload = response.read(1024 * 1024 + 1)
+        if len(payload) > 1024 * 1024 or hashlib.sha256(payload).hexdigest() != UFW_DOCKER_SHA256:
+            raise RuntimeError("ufw-docker checksum mismatch; refusing installation.")
+        _check_helper_path(path)
+        ctx.host.write_file(str(path), payload.decode("utf-8"), mode="0755",
+                            backup=False, sudo=True)
+    elif stat.S_IMODE(path.stat().st_mode) != 0o755:
+        ctx.host.write_file(str(path), payload.decode("utf-8"), mode="0755",
+                            backup=False, sudo=True)
+    if (not _check_helper_path(path) or
+            hashlib.sha256(path.read_bytes()).hexdigest() != UFW_DOCKER_SHA256 or
+            stat.S_IMODE(path.stat().st_mode) != 0o755):
+        raise RuntimeError("ufw-docker verification failed; refusing execution.")
+    ctx.log.info("Ensuring ufw-docker routing rules are installed in UFW...")
+    ctx.sys.sudo([str(path), "install"])
+
+
+def run(ctx, args):
+    _check_helper_path(UFW_DOCKER_PATH)
+    if os.path.exists("/usr/bin/pveversion"):
+        raise RuntimeError("Use the Proxmox firewall on a PVE host.")
+    for binary in ("ufw", "docker"):
+        if not ctx.sys.ok(["which", binary]):
+            raise RuntimeError(f"Install {binary} before configuring ufw-docker.")
+    _install_helper(ctx)

actions/config/ufw.py

@@ -1,5 +1,5 @@
-# actions/configure/ufw.py
+# actions/config/ufw.py
-"""Port of configure/ufw — apply docker + host ufw rules from configs/ufw."""
+"""Port of config/ufw — apply docker + host ufw rules from configs/ufw."""
 import os
 from engine.descriptor import Meta, Flag
 
@@ -13,12 +13,12 @@ def run(ctx, args):
     docker_dir = os.path.join(cfg, "docker")
     host_dir = os.path.join(cfg, "host")
     ctx.log.info("Configuring UFW rules...")
-    # Mirrors the bash branch structure: host dir is passed unconditionally (a missing
+    has_docker = os.path.isdir(docker_dir)
-    # one exits 66 inside apply); both dirs missing = warn + success, like bash.
+    has_host = os.path.isdir(host_dir)
-    if os.path.isdir(docker_dir):
+    if not has_docker:
-        ctx.ufw.apply(docker_dir, host_dir, dry_run=args.dry_run)
-    else:
         ctx.log.warn(f"Docker rules dir not found at {docker_dir}. Skipping ufw-docker rules.")
-        if os.path.isdir(host_dir):
+    if not has_host:
-            ctx.ufw.apply(docker_dir, host_dir, dry_run=args.dry_run)
+        ctx.log.warn(f"Host rules dir not found at {host_dir}. Skipping host rules.")
+    if has_docker or has_host:
+        ctx.ufw.apply(docker_dir if has_docker else None, host_dir if has_host else None, dry_run=args.dry_run)
     ctx.log.success("Configured ufw.")

actions/configure.py

@@ -1,34 +0,0 @@
-# actions/configure.py
-"""Umbrella: run every configure/* action in a stable order (SP2 §3.1)."""
-import os
-from engine.descriptor import Meta, Flag, build_parser
-from engine import overlay
-
-METADATA = Meta(summary="Run every configure/* action in stable order.",
-                args=[Flag("--dry-run", "Pass --dry-run to sub-actions that support it.")])
-ORDER = ["configure/base", "configure/ufw", "configure/crowdsec",
-         "configure/cron", "configure/systemd", "configure/routing"]
-
-def run(ctx, args):
-    commons_actions = os.path.join(ctx.paths.commons, "actions")
-    prop = os.path.join(ctx.paths.proprietaries, "actions")
-    prop = prop if os.path.isdir(prop) else None
-    names = [i.name for i in overlay.discover(commons_actions, prop) if i.name.startswith("configure/")]
-    ordered = [n for n in ORDER if n in names] + sorted(n for n in names if n not in ORDER)
-    rc = 0
-    for name in ordered:
-        r = overlay.resolve(name, commons_actions, prop)
-        mod = overlay.load_module(r.path, "apex_umbrella_" + name.replace("/", "_"))
-        if getattr(mod, "DISABLED", None):
-            ctx.log.warn(f"Skipping disabled {name}"); continue
-        meta = getattr(mod, "METADATA")
-        has_dry = any(getattr(a, "name", None) == "--dry-run" for a in meta.args)
-        argv = ["--dry-run"] if (args.dry_run and has_dry) else []
-        sub_args = build_parser(meta, f"apex {name}").parse_args(argv)
-        ctx.log.info(f"=== {name} ===")
-        try:
-            mod.run(ctx, sub_args)
-        except SystemExit as e:
-            if e.code:
-                ctx.log.error(f"{name} exited {e.code}"); rc = rc or e.code
-    return rc

actions/configure/base.py

@@ -1,31 +0,0 @@
-# actions/configure/base.py
-"""Port of configure/base — base packages + ufw + ufw-docker helper."""
-import os, shutil, stat, urllib.request
-from engine.descriptor import Meta
-
-METADATA = Meta(summary="Configure base system packages (curl, wget, git, rsyslog, ufw, ufw-docker).")
-UFW_DOCKER_URL = "https://github.com/chaifeng/ufw-docker/raw/master/ufw-docker"
-
-def run(ctx, args):
-    log, s = ctx.log, ctx.sys
-    log.info("Starting base system initialization...")
-    if not os.path.isfile("/etc/debian_version"):
-        log.error("Requires a Debian-based system (Debian/Ubuntu)."); raise SystemExit(1)
-    log.info("Updating package lists..."); s.sudo(["apt-get", "update", "-y"])
-    log.info("Installing essential packages...")
-    s.sudo(["apt-get", "install", "-y", "curl", "wget", "git", "rsyslog"])
-    s.ensure_running("rsyslog")
-    if not shutil.which("ufw"):
-        log.info("Installing UFW..."); s.sudo(["apt-get", "install", "-y", "ufw"])
-    else:
-        log.info("UFW already installed.")
-    binp = os.path.expanduser("~/.local/bin/ufw-docker")
-    if not (os.path.isfile(binp) and os.access(binp, os.X_OK)):
-        log.info("Downloading ufw-docker helper...")
-        os.makedirs(os.path.dirname(binp), exist_ok=True)
-        urllib.request.urlretrieve(UFW_DOCKER_URL, binp)       # stdlib; replaces wget/curl
-        os.chmod(binp, os.stat(binp).st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
-        log.success(f"ufw-docker downloaded to {binp}.")
-    else:
-        log.info(f"ufw-docker already exists at {binp}.")
-    log.success("Base system initialization completed.")

actions/repository/chown.py

@@ -0,0 +1,62 @@
+"""Give adam ownership of the active node's tracked checkout and Git metadata."""
+import inspect
+import json
+from pathlib import Path
+import re
+from engine.descriptor import Meta
+
+METADATA = Meta(summary="Set ownership of tracked files and Git metadata under /srv to adam.")
+
+
+def _repair(root, tracked, mounts, uid, gid):
+    import os
+    import stat
+
+    directories = {""}
+    for item in tracked:
+        directory = os.path.dirname(item)
+        while directory and directory not in directories:
+            directories.add(directory)
+            directory = os.path.dirname(directory)
+    metadata = os.path.join(root, ".git")
+
+    def fail(error):
+        raise error
+
+    for parent, dirs, files, fd in os.fwalk(root, onerror=fail, follow_symlinks=False):
+        os.fchown(fd, uid, gid)
+        for name in dirs.copy() + files:
+            path = os.path.join(parent, name)
+            relative = os.path.relpath(path, root)
+            if path in mounts or not (relative in tracked or relative in directories or
+                                      path == metadata or path.startswith(metadata + "/")):
+                if name in dirs:
+                    dirs.remove(name)
+                continue
+            info = os.stat(name, dir_fd=fd, follow_symlinks=False)
+            if stat.S_ISDIR(info.st_mode):
+                continue
+            if (stat.S_ISREG(info.st_mode) or stat.S_ISLNK(info.st_mode)) and info.st_nlink == 1:
+                os.chown(name, uid, gid, dir_fd=fd, follow_symlinks=False)
+
+
+def run(ctx, args):
+    root = Path(ctx.paths.repo_root)
+    if (root.parent != Path("/srv") or root != root.resolve(strict=True) or
+            not (root / "node.env").is_file() or (root / ".git").is_symlink() or
+            not (root / ".git").is_dir()):
+        raise RuntimeError("Expected the current node checkout directly under /srv.")
+    git = ["git", "-c", f"safe.directory={root}", "-c", f"safe.directory={root}/commons",
+               "-C", str(root)]
+    entries = ctx.sys.run([*git, "ls-files", "--cached", "--recurse-submodules", "-z"],
+                          capture=True).stdout
+    gitfiles = ctx.sys.run([*git, "submodule", "foreach", "--quiet", "--recursive",
+                            'printf "%s\\0" "$displaypath/.git"'], capture=True).stdout
+    mounts = [re.sub(r"\\([0-7]{3})", lambda match: chr(int(match[1], 8)), line.split()[4])
+              for line in Path("/proc/self/mountinfo").read_text().splitlines()]
+    script = (inspect.getsource(_repair) + "\nimport json, pwd, sys\n"
+              "root, entries, mounts = json.load(sys.stdin)\n"
+              "adam = pwd.getpwnam('adam')\n"
+              "_repair(root, set(entries), set(mounts), adam.pw_uid, adam.pw_gid)")
+    ctx.sys.sudo(["python3", "-I", "-c", script],
+                 input=json.dumps([str(root), (entries + gitfiles).split("\0"), mounts]))

actions/repository/sync.py

@@ -0,0 +1,59 @@
+"""Commit node configuration and synchronize main directly with its remote."""
+import os
+from datetime import datetime
+from engine.descriptor import Meta, Arg
+from engine.identity import read_env
+
+METADATA = Meta(summary="Commit node changes, rebase onto origin/main, and push main.",
+                args=[Arg("telegram_bot_url", "Optional notification URL", required=False)])
+TITLE = "Repository Sync"
+
+
+def run(ctx, args):
+    log, node = ctx.log, ctx.node.name
+    url = args.telegram_bot_url
+    repo, commons = ctx.paths.repo_root, ctx.paths.commons
+    ne = read_env(os.path.join(repo, "node.env"))
+    author = ne.get("APEX_GIT_AUTHOR_NAME", "apex [bot]")
+    email = ne.get("APEX_GIT_AUTHOR_EMAIL", "apex@localhost")
+    env = dict(os.environ, GIT_AUTHOR_NAME=author, GIT_AUTHOR_EMAIL=email,
+               GIT_COMMITTER_NAME=author, GIT_COMMITTER_EMAIL=email)
+
+    def git(*a, check=True, capture=False):
+        return ctx.sys.run(["git", "-C", repo, *a], check=check, capture=capture, env=env)
+
+    try:
+        branch = git("symbolic-ref", "--quiet", "--short", "HEAD", capture=True).stdout.strip()
+        if branch != "main":
+            raise RuntimeError("Repository sync requires main; switch branches explicitly first.")
+        cp = ctx.sys.run(["git", "-C", commons, "status", "--porcelain"], capture=True)
+        if (cp.stdout or "").strip():
+            raise RuntimeError(f"Commons submodule at {commons} has uncommitted changes.")
+        if not os.path.isdir(ctx.paths.compositions):
+            raise RuntimeError(f"Directory '{ctx.paths.compositions}' does not exist.")
+        expected = git("rev-parse", "HEAD:commons", capture=True).stdout.strip()
+        actual = ctx.sys.run(["git", "-C", commons, "rev-parse", "HEAD"], capture=True).stdout.strip()
+        if actual != expected:
+            raise RuntimeError("Commons checkout differs from the committed pin; update the submodule before retrying.")
+        git("add", "-A")
+        if git("diff", "--cached", "--quiet", check=False).returncode != 0:
+            ts = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+            git("commit", "-m", f"[{node}] Auto-commit: {ts}")
+        else:
+            log.info("No changes to commit.")
+        git("fetch", "origin", "main")
+        if git("rebase", "origin/main", check=False).returncode != 0:
+            git("rebase", "--abort", check=False)
+            raise RuntimeError("Rebase conflict; local commits retained for manual resolution.")
+        git("submodule", "update", "--init", "--recursive")
+        git("push", "origin", "HEAD:refs/heads/main")
+        log.success("Synchronized main.")
+        if url and not ctx.notify.success(TITLE, url, "Sync to main completed."):
+            raise SystemExit(1)
+    except SystemExit:
+        raise
+    except Exception as e:
+        log.error(f"{type(e).__name__}: {e}")
+        if url:
+            ctx.notify.error(TITLE, url, f"Sync failed: {e}")
+        raise SystemExit(1) from e

actions/sync/packages.py

@@ -1,71 +0,0 @@
-# actions/sync/packages.py
-"""Port of sync/packages — apt upgradable + docker image (skopeo) update check; Telegram."""
-import os, re, shutil
-from engine.descriptor import Meta, Arg
-
-METADATA = Meta(summary="Check pending apt updates + docker image updates; Telegram notify.",
-                args=[Arg("telegram_bot_url", "Telegram Bot URL for notifications")])
-TITLE = "Node Security Updates"
-
-def run(ctx, args):
-    log, node, url, s = ctx.log, ctx.node.name, args.telegram_bot_url, ctx.sys
-    try:
-        _check(ctx, args, log, node, url, s)
-    except SystemExit:
-        raise
-    except Exception as e:   # bash ERR trap parity: unhandled failures page via Telegram
-        log.error(f"{type(e).__name__}: {e}")
-        ctx.notify.error(TITLE, url, f"Critical error: {e}", node)
-        raise SystemExit(1)
-
-def _check(ctx, args, log, node, url, s):
-    if not os.path.isfile("/etc/debian_version"):
-        log.error("Requires a Debian-based system.")
-        ctx.notify.error(TITLE, url, "Requires a Debian-based system.", node)
-        raise SystemExit(1)
-
-    log.info("Updating package lists...")
-    if s.ok(["sudo", "apt-get", "update"]):
-        cp = s.run(["bash", "-c",
-                    "sudo apt-get --just-print upgrade 2>/dev/null | grep '^Inst' | cut -d' ' -f2 | sort"],
-                   check=False, capture=True)
-        upd = [l for l in (cp.stdout or "").splitlines() if l.strip()]
-        packages = (f"*Packages ({len(upd)}):* " + ", ".join(f"`{u}`" for u in upd)) if upd else "*Packages:* _none_"
-    else:
-        packages = "*Packages:* _Failed to check_"
-
-    if not shutil.which("docker"):
-        images = "*Images:* _Docker not available_"
-    elif not shutil.which("skopeo"):
-        images = "*Images:* _skopeo not available_"
-    else:
-        cp = s.run(["bash", "-c", "docker ps --format '{{.Image}}' | sort -u"], check=False, capture=True)
-        outdated = []
-        for image in [l for l in (cp.stdout or "").splitlines() if l.strip()]:
-            name, sep, tag = image.partition(":")
-            tag = tag or "latest"
-            ld = s.run(["bash", "-c",
-                        f"docker image inspect '{image}' --format '{{{{index .RepoDigests 0}}}}' 2>/dev/null | cut -d'@' -f2"],
-                       check=False, capture=True).stdout.strip()
-            if not ld:
-                continue
-            if "/" not in name:
-                reg = f"docker.io/library/{name}"
-            elif not re.match(r".*\..*/", name):
-                reg = f"docker.io/{name}"
-            else:
-                reg = name
-            rd = s.run(["bash", "-c",
-                        f"skopeo inspect 'docker://{reg}:{tag}' 2>/dev/null | "
-                        f"grep -o '\"Digest\"[[:space:]]*:[[:space:]]*\"[^\"]*\"' | cut -d'\"' -f4"],
-                       check=False, capture=True).stdout.strip()
-            if rd and ld != rd:
-                disp = name[len("docker.io/"):] if name.startswith("docker.io/") else name
-                disp = disp[len("library/"):] if disp.startswith("library/") else disp
-                outdated.append(disp)
-        images = (f"*Images ({len(outdated)}):* " + ", ".join(f"`{i}`" for i in outdated)) if outdated else "*Images:* _none_"
-
-    # A failed send exits 1 (bash: telegram_info's return 1 tripped set -e).
-    if not ctx.notify.info(TITLE, url, f"*APT:*\n{packages}\n\n*Skopeo:*\n{images}", node):
-        raise SystemExit(1)
-    log.success("Security update check completed.")

actions/sync/repository.py

@@ -1,60 +0,0 @@
-# actions/sync/repository.py
-"""Port of sync/repository — capture-up commit + rebase onto origin/main + force-push."""
-import os
-from datetime import datetime
-from engine.descriptor import Meta, Arg
-
-METADATA = Meta(summary="Capture-up: commit node changes, rebase onto origin/main, force-push sync/<node>.",
-                args=[Arg("telegram_bot_url", "Telegram Bot URL for notifications")])
-TITLE = "Annual Remote Sync"
-
-def run(ctx, args):
-    log, node, url = ctx.log, ctx.node.name, args.telegram_bot_url
-    repo, commons = ctx.paths.repo_root, ctx.paths.commons
-    env = dict(os.environ,
-               GIT_AUTHOR_NAME="Adam Jensen [bot]", GIT_AUTHOR_EMAIL="adam@ermnvldmr.com",
-               GIT_COMMITTER_NAME="Adam Jensen [bot]", GIT_COMMITTER_EMAIL="adam@ermnvldmr.com")
-
-    def git(*a, check=True):
-        return ctx.sys.run(["git", "-C", repo, *a], check=check, env=env)
-
-    try:
-        cp = ctx.sys.run(["git", "-C", commons, "status", "--porcelain"], capture=True)
-        if (cp.stdout or "").strip():
-            log.error(f"Commons submodule at {commons} has uncommitted changes. Aborting.")
-            raise SystemExit(1)
-        if not os.path.isdir(ctx.paths.compositions):
-            # Without this, `git add -A` would stage a deleted node tree and an
-            # unattended cron run would commit + force-push the deletion.
-            log.error(f"Directory '{ctx.paths.compositions}' does not exist. Cannot sync.")
-            raise SystemExit(1)
-        branch = f"sync/{node}"
-        ts = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
-        if ctx.sys.ok(["git", "-C", repo, "rev-parse", "--verify", branch]):
-            git("checkout", branch)
-        else:
-            git("checkout", "-b", branch)
-        git("add", "-A")
-        git("submodule", "update", "--remote", "--recursive")
-        git("add", "commons")
-        if git("diff", "--cached", "--quiet", check=False).returncode != 0:
-            git("commit", "-m", f"[{node}] Auto-commit: {ts}")
-        else:
-            log.info("No changes to commit.")
-        git("fetch", "origin", "main")
-        if git("rebase", "origin/main", check=False).returncode != 0:
-            log.error("Rebase conflict — aborting rebase; manual intervention required.")
-            git("rebase", "--abort", check=False)
-            ctx.notify.error(TITLE, url, "Rebase conflict. Manual intervention required.", node)
-            raise SystemExit(1)
-        git("submodule", "update", "--init", "--recursive")
-        git("push", "--force-with-lease", "origin", branch)
-        log.success(f"Pushed {branch}.")
-        if not ctx.notify.success(TITLE, url, f"Sync to '{branch}' completed.", node):
-            raise SystemExit(1)
-    except SystemExit:
-        raise
-    except Exception as e:
-        log.error(f"{type(e).__name__}: {e}")
-        ctx.notify.error(TITLE, url, f"Sync failed: {e}", node)
-        raise SystemExit(1)

actions/tier/backup.py

@@ -1,5 +1,5 @@
-# actions/backup/run.py
+# actions/tier/backup.py
-"""Label-driven restic backup via resticontainer (github.com/deytenit/resticontainer).
+"""Label-driven restic backup via resticontainer (git.ermnvldmr.com/deytenit/resticontainer).
 Runs the resticontainer core service one-shot: init-if-needed -> backup (discovers
 `restic.*`-labelled services, runs their pre-hooks/stops, snapshots the union of the
 resolved host paths in a single restic run) -> forget --prune -> Telegram. Every
@@ -7,16 +7,48 @@ non-`backup` subcommand passes straight through to restic, so init/snapshots/for
 behave exactly as before."""
 import json
 import os
+from pathlib import Path
+import time
 from engine.descriptor import Meta, Arg
 
 METADATA = Meta(summary="Run the resticontainer label-driven backup (init-if-needed, backup, forget --prune).",
                 args=[Arg("telegram_bot_url", "Telegram Bot URL for notifications")])
 TITLE = "Restic Host Backup"
 
+def write_backup_status(path, state, timestamp):
+    path = Path(path)
+    previous = json.loads(path.read_text()) if path.exists() else {}
+    status = {"state": state, "updated_at": timestamp,
+              "last_success": timestamp if state == "success" else previous.get("last_success", 0)}
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_name(path.name + f".{os.getpid()}.tmp")
+    temporary.write_text(json.dumps(status) + "\n")
+    temporary.chmod(0o644)
+    temporary.replace(path)
+
+def restore_containers(ctx, container_ids):
+    """Recover previously running backup targets, retrying namespace dependents."""
+    pending = list(container_ids)
+    if not pending:
+        return
+    for _ in range(len(pending) + 1):
+        cp = ctx.sys.run(["docker", "inspect", *pending], capture=True)
+        stopped = [c["Id"] for c in json.loads(cp.stdout) if not c["State"]["Running"]]
+        if not stopped:
+            return
+        progress = False
+        for container_id in stopped:
+            cp = ctx.sys.run(["docker", "start", container_id], check=False, capture=True)
+            progress = progress or cp.returncode == 0
+        if not progress:
+            raise RuntimeError("Backup left containers stopped: " + ", ".join(stopped))
+        pending = stopped
+    raise RuntimeError("Backup containers did not stay running: " + ", ".join(pending))
+
 def run(ctx, args):
     log, node, url = ctx.log, ctx.node.name, args.telegram_bot_url
     core = ctx.paths.core
-    host = f"{node}.apex.ermnvldmr.com"         # restic --host: the per-node public name (matches the bucket)
+    host = ctx.node.host                        # restic --host: the node's public host name (APEX_NODE_HOST)
     tag = "biweekly"                            # restic --tag: snapshot cadence (this cron fires on the 5th & 20th)
     if not os.path.isfile(os.path.join(core, "docker-compose.yml")):
         log.error(f"Core compose not found in {core}"); raise SystemExit(1)
@@ -42,17 +74,27 @@ def run(ctx, args):
         except (ValueError, TypeError):
             return None
 
+    status_path = Path("/var/lib/apex/backup-status.json")
+    write_backup_status(status_path, "running", time.time())
     try:
         log.info("Checking restic repository availability...")
         if restic("snapshots", check=False).returncode == 0:
             log.info("Restic repository reachable.")
         elif restic("init", check=False).returncode != 0:
-            ctx.notify.error(TITLE, url, "restic init failed", node); raise SystemExit(1)
+            ctx.notify.error(TITLE, url, "restic init failed"); raise SystemExit(1)
         before = host_snapshot_ids()
         log.info("Backing up `restic.*`-labelled services (resticontainer discovers paths)...")
-        if restic("backup", "--host", host, "--tag", tag, "--compression", "max",
+        targets = ctx.sys.run(["docker", "ps", "-q", "--no-trunc",
-                  check=False).returncode != 0:
+                               "--filter", "label=restic.enable=true",
-            ctx.notify.error(TITLE, url, "restic backup failed", node); raise SystemExit(1)
+                               "--filter", "label=restic.backup.stop=true"], capture=True).stdout.split()
+        try:
+            result = restic("backup", "--host", host, "--tag", tag, "--compression", "max",
+                            check=False)
+        finally:
+            # Resticontainer may start a network-namespace dependent before its owner.
+            restore_containers(ctx, targets)
+        if result.returncode != 0:
+            ctx.notify.error(TITLE, url, "restic backup failed"); raise SystemExit(1)
         # Guard: the label-driven backup is a silent no-op if no service carries the
         # `restic.enable` label (resticontainer finds zero paths and skips restic).
         # Prove a fresh snapshot for this host actually landed before we prune/notify.
@@ -60,18 +102,22 @@ def run(ctx, args):
         if before is not None and after is not None and not (after - before):
             log.error("No new snapshot produced — refusing to report success.")
             ctx.notify.error(TITLE, url,
-                             "backup produced no new snapshot (no restic.* labels discovered?)", node)
+                             "backup produced no new snapshot (no restic.* labels discovered?)")
             raise SystemExit(1)
         log.info("Applying retention (forget --prune)...")
-        if restic("forget", "--prune", "--keep-daily", "7", "--keep-weekly", "4",
+        if restic("forget", "--prune", "--host", host, "--tag", tag,
+                  "--keep-daily", "7", "--keep-weekly", "4",
                   "--keep-monthly", "12", check=False).returncode != 0:
-            ctx.notify.error(TITLE, url, "restic forget/prune failed", node); raise SystemExit(1)
+            ctx.notify.error(TITLE, url, "restic forget/prune failed"); raise SystemExit(1)
         log.success("Restic backup completed.")
-        if not ctx.notify.success(TITLE, url, f"Restic backup + retention succeeded for {node}", node):
+        if not ctx.notify.success(TITLE, url, f"Restic backup + retention succeeded for {node}"):
             raise SystemExit(1)   # bash exited 1 when the success telegram failed to send
+        write_backup_status(status_path, "success", time.time())
     except SystemExit:
+        write_backup_status(status_path, "failed", time.time())
         raise
     except Exception as e:
+        write_backup_status(status_path, "failed", time.time())
         log.error(f"{type(e).__name__}: {e}")
-        ctx.notify.error(TITLE, url, f"backup failed: {e}", node)
+        ctx.notify.error(TITLE, url, f"backup failed: {e}")
-        raise SystemExit(1)
+        raise SystemExit(1) from e

actions/tier/chown.py

@@ -1,5 +1,5 @@
-# actions/tiers/chown.py
+# actions/tier/chown.py
-"""Port of tiers/chown — ownership: {tier}/{proj} -> uid:gid; {tier}/shared/{proj} ->
+"""Port of tier/chown — ownership: {tier}/{proj} -> uid:gid; {tier}/shared/{proj} ->
 uid:shared_gid (+sticky); base/node-infra -> root:root. Reads APEX_UID/GID from each .env."""
 import os
 from engine.descriptor import Meta, Arg
@@ -13,10 +13,11 @@ SHARED_GROUP = "noroot-shared"
 def _env_val(path, key):
     if not os.path.exists(path):
         return ""
-    for line in open(os.path.realpath(path)):
+    with open(os.path.realpath(path), encoding="utf-8") as f:
-        line = line.strip()
+        for line in f:
-        if line.startswith(key + "="):
+            line = line.strip()
-            return line.split("=", 1)[1].strip()
+            if line.startswith(key + "="):
+                return line.split("=", 1)[1].strip()
     return ""
 
 def run(ctx, args):
@@ -30,7 +31,7 @@ def run(ctx, args):
 
     cp = s.run(["getent", "group", SHARED_GROUP], check=False, capture=True)
     if cp.returncode != 0:
-        log.error(f"Shared group '{SHARED_GROUP}' not found. Run tiers/useradd first."); raise SystemExit(1)
+        log.error(f"Shared group '{SHARED_GROUP}' not found. Run tier/useradd first."); raise SystemExit(1)
 
     log.info("Ensuring base structures owned by root...")
     for tp in tiers:
@@ -61,7 +62,7 @@ def run(ctx, args):
         gid = _env_val(env_path, "APEX_GID")
         sgid = _env_val(env_path, "APEX_SHARED_GID")
         if not (uid and gid and sgid):
-            log.warn(f"Incomplete APEX_UID/GID/SHARED_GID for {proj}. Run tiers/useradd first. Skipping.")
+            log.warn(f"Incomplete APEX_UID/GID/SHARED_GID for {proj}. Run tier/useradd first. Skipping.")
             continue
         log.info(f"Applying permissions for {proj} (UID={uid} GID={gid} SHARED_GID={sgid})")
         for tp in tiers:

actions/tier/link.py

@@ -1,5 +1,5 @@
-# actions/tiers/link.py
+# actions/tier/link.py
-"""Port of tiers/link — @tier symlink structure for the node + each composition project."""
+"""Port of tier/link — @tier symlink structure for the node + each composition project."""
 import os
 from engine.descriptor import Meta, Arg
 

actions/tier/useradd.py

@@ -1,5 +1,5 @@
-# actions/tiers/useradd.py
+# actions/tier/useradd.py
-"""Port of tiers/useradd — one noroot system user per composition project; inject the
+"""Port of tier/useradd — one noroot system user per composition project; inject the
 APEX_UID/APEX_GID/APEX_SHARED_GID block into each project .env."""
 import os, re
 from engine.descriptor import Meta
@@ -36,7 +36,7 @@ def run(ctx, args):
             continue
         # opt-out: a project marked .apex-root-owned runs as root and keeps its tier
         # data root-owned (e.g. seafile-mc, whose entrypoint refuses non-root-owned
-        # data). Skip the noroot user + .env UID injection; tiers/chown forces root.
+        # data). Skip the noroot user + .env UID injection; tier/chown forces root.
         if os.path.isfile(os.path.join(full, ".apex-root-owned")):
             log.info(f"{proj}: .apex-root-owned — runs as root; skipping noroot user")
             continue

actions/util/check-updates.py

@@ -0,0 +1,249 @@
+# actions/util/check-updates.py
+"""Check pending APT and running-image updates without pulling images."""
+import json
+import os
+import re
+import shutil
+from engine.descriptor import Meta, Arg
+
+METADATA = Meta(summary="Check pending apt updates + docker image updates; Telegram notify.",
+                args=[Arg("telegram_bot_url", "Telegram Bot URL for notifications")])
+TITLE = "Node Security Updates"
+
+MANIFEST_TYPES = {"application/vnd.oci.image.manifest.v1+json",
+                  "application/vnd.docker.distribution.manifest.v2+json"}
+INDEX_TYPES = {"application/vnd.oci.image.index.v1+json",
+               "application/vnd.docker.distribution.manifest.list.v2+json"}
+
+
+def _output(s, command):
+    result = s.run(command, check=False, capture=True)
+    if result.returncode:
+        # Registry errors can contain authentication details; don't relay stderr.
+        raise ValueError("inspection command failed")
+    return result.stdout or ""
+
+
+def _digest(value):
+    if not isinstance(value, str) or not re.fullmatch(r"sha256:[0-9a-f]{64}", value):
+        raise ValueError("missing or unsupported image digest")
+    return value
+
+
+def _platform(os_name, architecture, variant=""):
+    if not isinstance(os_name, str) or not os_name or not isinstance(architecture, str) or not architecture:
+        raise ValueError("missing image platform")
+    variant = variant or ""
+    if architecture == "arm64" and variant == "v8":
+        variant = ""
+    return os_name, architecture, variant
+
+
+def _repository(reference):
+    name = reference.split("@", 1)[0]
+    # Only a colon in the final path component is a tag separator.
+    return name.rsplit(":", 1)[0] if ":" in name.rsplit("/", 1)[-1] else name
+
+
+def _remote_digests(s, reference, local, cache, running_manifest=None):
+    def inspect(ref, raw=False, field="Manifest"):
+        command = ["docker", "buildx", "imagetools", "inspect"]
+        command += ["--raw"] if raw else ["--format", "{{json ." + field + "}}"]
+        key = tuple(command + [ref])
+        if key not in cache:
+            value = json.loads(_output(s, ["timeout", "--signal=TERM", "--kill-after=5s",
+                                          "30s", *key]))
+            if not isinstance(value, dict):
+                raise ValueError("invalid registry JSON object")
+            cache[key] = value
+        return cache[key]
+
+    platform = _platform(local.get("Os"), local.get("Architecture"), local.get("Variant"))
+    descriptor = inspect(reference)
+    media_type = descriptor.get("mediaType")
+    selected = _digest(descriptor.get("digest"))
+    if media_type in INDEX_TYPES:
+        manifests = descriptor.get("manifests")
+        if not isinstance(manifests, list):
+            raise ValueError("invalid image index")
+        matches = []
+        for item in manifests:
+            p = item.get("platform", {})
+            if _platform(p.get("os"), p.get("architecture"), p.get("variant")) == platform:
+                if item.get("mediaType") not in MANIFEST_TYPES:
+                    raise ValueError("unsupported nested image index")
+                matches.append(_digest(item.get("digest")))
+        if len(matches) != 1:
+            raise ValueError("missing or ambiguous matching image platform")
+        selected = matches[0]
+    elif media_type not in MANIFEST_TYPES:
+        raise ValueError("unsupported registry manifest")
+
+    immutable = _repository(reference) + "@" + selected
+    manifest = inspect(immutable, raw=True)
+    if manifest.get("schemaVersion") != 2 or manifest.get("mediaType") not in MANIFEST_TYPES:
+        raise ValueError("invalid platform manifest")
+    remote_id = _digest(manifest.get("config", {}).get("digest"))
+    if media_type in MANIFEST_TYPES and selected != running_manifest and remote_id != local["Id"]:
+        config = inspect(immutable, field="Image")
+        if _platform(config.get("os"), config.get("architecture"), config.get("variant")) != platform:
+            raise ValueError("registry image platform changed")
+    return selected, remote_id
+
+
+def _compose_builds(ctx):
+    builds = {}
+    env = dict(os.environ)
+    env.update({k: v for k, v in ctx.vars().items() if k.startswith("APEX_")})
+    for name in sorted(os.listdir(ctx.paths.compositions)):
+        directory = os.path.join(ctx.paths.compositions, name)
+        if name.startswith((".", "@")) or not os.path.isfile(os.path.join(directory, "docker-compose.yml")):
+            continue
+        command = ["docker", "compose"]
+        for filename in (".env", "apex.env"):
+            if os.path.isfile(os.path.join(directory, filename)):
+                command += ["--env-file", filename]
+        # Resolved Compose includes secrets: capture privately and never log it.
+        result = ctx.sys.run(["timeout", "--signal=TERM", "--kill-after=5s", "30s",
+                              *command, "config", "--format", "json"],
+                             cwd=directory, env=env, check=False, capture=True)
+        if result.returncode:
+            raise ValueError("Compose inspection failed")
+        config = json.loads(result.stdout)
+        project, services = config["name"], config["services"]
+        if not isinstance(project, str) or not project or not isinstance(services, dict):
+            raise ValueError("invalid Compose model")
+        for service, settings in services.items():
+            if settings.get("build") is not None:
+                reference = settings.get("image")
+                if reference is not None and not isinstance(reference, str):
+                    raise ValueError("invalid Compose image")
+                builds[(project, service)] = reference
+    return builds
+
+
+def _images(s, log, ctx=None):
+    if not shutil.which("docker"):
+        return "Images: unchecked (Docker unavailable)"
+    if not shutil.which("timeout"):
+        return "Images: unchecked (coreutils timeout unavailable)"
+    try:
+        ids = _output(s, ["docker", "ps", "--quiet", "--no-trunc"]).split()
+        if not ids:
+            return "Images: no running containers"
+        output = _output(s, ["docker", "container", "inspect", "--format",
+                            '{"Image":{{json .Image}},"Reference":{{json .Config.Image}},'
+                            '"ImageManifestDescriptor":{{json (index . "ImageManifestDescriptor")}},'
+                            '"Labels":{{json .Config.Labels}}}', *ids])
+        containers = [json.loads(line) for line in output.splitlines()]
+        if len(containers) != len(ids):
+            raise ValueError("incomplete running-container inspection")
+        if any(not isinstance(c["Reference"], str) for c in containers):
+            raise ValueError("invalid running-image reference")
+    except (ValueError, KeyError, TypeError, OSError) as error:
+        log.warn(f"Images unchecked: {type(error).__name__}")
+        return "Images: unchecked (container inspection failed)"
+
+    try:
+        builds = _compose_builds(ctx) if ctx is not None else {}
+    except (ValueError, KeyError, TypeError, AttributeError, OSError):
+        log.warn("Images unchecked: Compose build provenance could not be resolved")
+        return "Images: unchecked (Compose inspection failed)"
+
+    buildx = s.run(["docker", "buildx", "version"], check=False, capture=True).returncode == 0
+    counts = dict.fromkeys(("updates", "current", "pinned", "local/untracked", "unchecked"), 0)
+    cache, images, seen = {}, {}, set()
+    for container in containers:
+        reference = container["Reference"]
+        reason = ""
+        try:
+            image_id = _digest(container["Image"])
+            descriptor = container.get("ImageManifestDescriptor")
+            labels = container.get("Labels") or {}
+            build_key = (labels.get("com.docker.compose.project"), labels.get("com.docker.compose.service"))
+            declared_build = build_key in builds and builds[build_key] in (None, reference)
+            key = (reference, image_id, json.dumps(descriptor, sort_keys=True), declared_build)
+            if key in seen:
+                continue
+            seen.add(key)
+            if not isinstance(reference, str) or not reference or reference.startswith("-"):
+                raise ValueError("invalid image reference")
+            if "@" in reference or re.fullmatch(r"(?:sha256:)?[0-9a-f]{64}", reference):
+                status = "pinned"
+            elif declared_build:
+                status = "local/untracked"
+            else:
+                if image_id not in images:
+                    data = json.loads(_output(s, ["docker", "image", "inspect", image_id]))
+                    if not isinstance(data, list) or len(data) != 1 or data[0].get("Id") != image_id:
+                        raise ValueError("invalid running-image inspection")
+                    images[image_id] = data[0]
+                local = images[image_id]
+                repo_digests = local["RepoDigests"]
+                if repo_digests is None or repo_digests == []:
+                    status = "local/untracked"
+                elif not isinstance(repo_digests, list):
+                    raise ValueError("invalid repository digests")
+                elif not buildx:
+                    raise ValueError("Buildx unavailable")
+                else:
+                    running_manifest = None
+                    if descriptor is not None:
+                        if descriptor.get("mediaType") not in MANIFEST_TYPES:
+                            raise ValueError("invalid running platform manifest")
+                        running_manifest = _digest(descriptor.get("digest"))
+                        platform = descriptor.get("platform", {})
+                        os_name, architecture, variant = _platform(
+                            platform.get("os"), platform.get("architecture"), platform.get("variant"))
+                        local = dict(local, Os=os_name, Architecture=architecture, Variant=variant)
+                    elif local.get("Descriptor") is not None:
+                        raise ValueError("containerd running platform manifest unavailable")
+                    remote_manifest, remote_config = _remote_digests(
+                        s, reference, local, cache, running_manifest)
+                    # Classic Docker IDs are configs; containerd IDs may be indices.
+                    current = (remote_manifest == running_manifest if running_manifest
+                               else remote_config == image_id)
+                    status = "current" if current else "updates"
+        except (ValueError, KeyError, TypeError, AttributeError, OSError) as error:
+            status, reason = "unchecked", str(error) if isinstance(error, ValueError) else type(error).__name__
+        counts[status] += 1
+        log.info(f"Image {reference}: {status}" + (f" ({reason})" if reason else ""))
+    return "Images: " + ", ".join(f"{count} {status}" for status, count in counts.items())
+
+
+def run(ctx, args):
+    log, node, url, s = ctx.log, ctx.node.name, args.telegram_bot_url, ctx.sys
+    try:
+        _check(ctx, args, log, node, url, s)
+    except SystemExit:
+        raise
+    except Exception as e:   # bash ERR trap parity: unhandled failures page via Telegram
+        log.error(f"{type(e).__name__}: {e}")
+        ctx.notify.error(TITLE, url, f"Critical error: {e}")
+        raise SystemExit(1) from e
+
+def _check(ctx, args, log, node, url, s):
+    if not os.path.isfile("/etc/debian_version"):
+        log.error("Requires a Debian-based system.")
+        ctx.notify.error(TITLE, url, "Requires a Debian-based system.")
+        raise SystemExit(1)
+
+    log.info("Updating package lists...")
+    if s.ok(["sudo", "apt-get", "update"]):
+        cp = s.run(["bash", "-c",
+                    "sudo apt-get --just-print upgrade 2>/dev/null | grep '^Inst' | cut -d' ' -f2 | sort"],
+                   check=False, capture=True)
+        upd = [l for l in (cp.stdout or "").splitlines() if l.strip()]
+        packages = (f"*Packages ({len(upd)}):* " + ", ".join(f"`{u}`" for u in upd)) if upd else "*Packages:* _none_"
+        apt_summary = f"APT: {len(upd)} pending"
+    else:
+        packages = "*Packages:* _Failed to check_"
+        apt_summary = "APT: failed to check"
+
+    images = _images(s, log, ctx)
+    log.info(packages)
+
+    # A failed send exits 1 (bash: telegram_info's return 1 tripped set -e).
+    if not ctx.notify.info(TITLE, url, f"{apt_summary}\n{images}\n\n{packages}"):
+        raise SystemExit(1)
+    log.success("Security update check completed.")

actions/util/extract-traefik-certs.py

@@ -1,5 +1,5 @@
-# actions/utils/extract-traefik-certs.py
+# actions/util/extract-traefik-certs.py
-"""Port of utils/extract-traefik-certs — dump traefik v3 acme.json to pem via docker."""
+"""Port of util/extract-traefik-certs — dump traefik v3 acme.json to pem via docker."""
 import os
 from engine.descriptor import Meta, Arg
 
@@ -21,15 +21,16 @@ def run(ctx, args):
         abs_dest = os.path.realpath(args.dest_dir)
         log.info(f"Extracting certs: {abs_src} -> {abs_dest}")
         ctx.sys.run(["docker", "run", "--rm", "-v", f"{src_dir}:/data", "-v", f"{abs_dest}:/output",
-                     "ldez/traefik-certs-dumper:v2.9.3", "file", "--version", "v3",
+                     "ldez/traefik-certs-dumper:v2.11.4", "file", "--version", "v3",
                      "--source", f"/data/{src_file}", "--dest", "/output", "--domain-subdir=true",
                      "--crt-name=fullchain", "--crt-ext=.pem", "--key-name=privkey", "--key-ext=.pem"],
                     check=True)
-        log.info("Setting secure permissions (600) on extracted keys...")
+        log.info("Setting secure permissions on extracted certs and keys...")
-        ctx.sys.sudo(["find", abs_dest, "-type", "f", "-name", "*.pem", "-exec", "chmod", "600", "{}", "+"])
+        ctx.sys.sudo(["find", abs_dest, "-type", "f", "-name", "*key*.pem", "-exec", "chmod", "600", "{}", "+"])
+        ctx.sys.sudo(["find", abs_dest, "-type", "f", "-name", "*chain*.pem", "-exec", "chmod", "644", "{}", "+"])
         ctx.sys.sudo(["chown", "-R", f"{os.getuid()}:{os.getgid()}", abs_dest])
         log.success("Certificates extracted.")
-        ctx.notify.success(TITLE, url, f"Certificates extracted for {node}", node)
+        ctx.notify.success(TITLE, url, f"Certificates extracted for {node}")
     except Exception as e:
-        log.error(str(e)); ctx.notify.error(TITLE, url, f"cert extraction failed: {e}", node)
+        log.error(str(e)); ctx.notify.error(TITLE, url, f"cert extraction failed: {e}")
-        raise SystemExit(1)
+        raise SystemExit(1) from e

actions/util/lint-docker-compose.py

@@ -1,7 +1,7 @@
-# actions/utils/lint-docker-compose.py
+# actions/util/lint-docker-compose.py
-"""Port of utils/lint-docker-compose (dclint via docker) + stdlib anchor-IP validation:
+"""Run dclint through Docker and validate anchor IPs:
 every ipv4_address in compositions/*/docker-compose.yml must be unique and inside the
-node's /24 (SP3 §6.2). No new deps — ipaddress + re are stdlib."""
+node's /24."""
 import ipaddress, os, re, shutil
 from engine.descriptor import Meta, Flag
 
@@ -12,8 +12,6 @@ def run(ctx, args):
     log, s = ctx.log, ctx.sys
     repo = ctx.paths.repo_root
     rc_ip = _validate_anchor_ips(ctx)
-    if not shutil.which("docker"):
-        log.error("docker is not installed or not in PATH."); return rc_ip or 1
 
     files_string, targets = "", ["."]
     if args.hook:
@@ -25,12 +23,19 @@ def run(ctx, args):
             log.info("No staged docker-compose files to lint."); return rc_ip
         targets = [f"/workdir/{f}" for f in files_string.splitlines()]
 
+    if not shutil.which("docker"):
+        log.error("docker is not installed or not in PATH."); return rc_ip or 1
+
     log.info("Running dclint --fix (via docker)...")
+    env_args = []
+    for k in ("HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy", "NO_PROXY", "no_proxy"):
+        if k in os.environ:
+            env_args.extend(["-e", f"{k}={os.environ[k]}"])
     # "-e commons": never --fix inside the pinned submodule — a dirtied commons makes
-    # sync/repository abort the nightly capture-up.
+    # repository/sync abort the nightly capture-up.
-    r = s.run(["docker", "run", "--rm", "-v", f"{repo}:/workdir", "-w", "/workdir", "node:lts-alpine",
+    r = s.run(["docker", "run", "--rm", "--network", "host", *env_args, "-v", f"{repo}:/workdir", "-w", "/workdir", "node:lts-alpine",
                "npx", "--yes", "dclint", "-e", "@tier1", "-e", "@tier2", "-e", "@tier3", "-e", ".env",
-               "-e", "commons", "-r", "--fix", *targets], check=False)
+               "-e", "commons", "--disable-rule", "no-build-and-image", "-r", "--fix", *targets], check=False)
     if r.returncode != 0:
         log.error(f"dclint issues (exit {r.returncode})."); return r.returncode or rc_ip
     if args.hook and files_string:
@@ -53,7 +58,8 @@ def _validate_anchor_ips(ctx) -> int:
         if not os.path.isfile(f):
             continue
         try:
-            text = open(f).read()
+            with open(f, encoding="utf-8") as fh:
+                text = fh.read()
         except OSError as e:
             ctx.log.error(f"Cannot read {f}: {e}"); rc = 1
             continue

apex

@@ -1,10 +1,10 @@
 #!/usr/bin/env python3
-"""apex launcher — stdlib only. Puts commons/ (+ vendor/ + proprietaries/) on sys.path."""
+"""apex launcher — stdlib only. Loads the shared engine and optional checkout libraries."""
 import os
 import sys
 
 _here = os.path.dirname(os.path.realpath(__file__))     # commons dir (or standalone repo root)
-sys.path.insert(0, _here)                               # make `engine` importable
+sys.path.insert(0, os.path.join(_here, "scripts"))
 _vendor = os.path.join(_here, "vendor")
 if os.path.isdir(_vendor):
     sys.path.insert(0, _vendor)

assets/logo.svg

@@ -1,35 +0,0 @@
-<svg width="1410" height="293" viewBox="0 0 1410 293" fill="none" xmlns="http://www.w3.org/2000/svg">
-<mask id="path-1-outside-1_1_35" maskUnits="userSpaceOnUse" x="1" y="77.5" width="1080" height="129" fill="black">
-<rect fill="white" x="1" y="77.5" width="1080" height="129"/>
-<path d="M62.64 150.02C62.64 147.46 62.2667 145.06 61.52 142.82C60.8267 140.527 59.76 138.527 58.32 136.82C56.88 135.113 55.04 133.78 52.8 132.82C50.6133 131.807 48.0533 131.3 45.12 131.3C39.4133 131.3 34.9067 132.927 31.6 136.18C28.3467 139.433 26.2667 144.047 25.36 150.02H62.64ZM24.96 161.94C25.28 166.153 26.0267 169.807 27.2 172.9C28.3733 175.94 29.92 178.473 31.84 180.5C33.76 182.473 36.0267 183.967 38.64 184.98C41.3067 185.94 44.24 186.42 47.44 186.42C50.64 186.42 53.3867 186.047 55.68 185.3C58.0267 184.553 60.0533 183.727 61.76 182.82C63.52 181.913 65.04 181.087 66.32 180.34C67.6533 179.593 68.9333 179.22 70.16 179.22C71.8133 179.22 73.04 179.833 73.84 181.06L79.52 188.26C77.3333 190.82 74.88 192.98 72.16 194.74C69.44 196.447 66.5867 197.833 63.6 198.9C60.6667 199.913 57.6533 200.633 54.56 201.06C51.52 201.487 48.56 201.7 45.68 201.7C39.9733 201.7 34.6667 200.767 29.76 198.9C24.8533 196.98 20.5867 194.18 16.96 190.5C13.3333 186.767 10.48 182.18 8.4 176.74C6.32 171.247 5.28 164.9 5.28 157.7C5.28 152.1 6.18667 146.847 8 141.94C9.81333 136.98 12.4 132.687 15.76 129.06C19.1733 125.38 23.3067 122.473 28.16 120.34C33.0667 118.207 38.5867 117.14 44.72 117.14C49.8933 117.14 54.6667 117.967 59.04 119.62C63.4133 121.273 67.1733 123.7 70.32 126.9C73.4667 130.047 75.92 133.94 77.68 138.58C79.4933 143.167 80.4 148.42 80.4 154.34C80.4 157.327 80.08 159.353 79.44 160.42C78.8 161.433 77.5733 161.94 75.76 161.94H24.96ZM111.869 132.66C114.429 127.753 117.469 123.913 120.989 121.14C124.509 118.313 128.669 116.9 133.469 116.9C137.255 116.9 140.295 117.727 142.589 119.38L141.309 134.18C141.042 135.14 140.642 135.833 140.109 136.26C139.629 136.633 138.962 136.82 138.109 136.82C137.309 136.82 136.109 136.687 134.509 136.42C132.962 136.153 131.442 136.02 129.949 136.02C127.762 136.02 125.815 136.34 124.109 136.98C122.402 137.62 120.855 138.553 119.469 139.78C118.135 140.953 116.935 142.393 115.869 144.1C114.855 145.807 113.895 147.753 112.989 149.94V200.5H93.2288V118.42H104.829C106.855 118.42 108.269 118.793 109.069 119.54C109.869 120.233 110.402 121.513 110.669 123.38L111.869 132.66ZM150.916 200.5V118.42H162.996C165.556 118.42 167.236 119.62 168.036 122.02L169.316 128.1C170.756 126.5 172.25 125.033 173.796 123.7C175.396 122.367 177.076 121.22 178.836 120.26C180.65 119.3 182.57 118.553 184.596 118.02C186.676 117.433 188.943 117.14 191.396 117.14C196.57 117.14 200.81 118.553 204.116 121.38C207.476 124.153 209.983 127.86 211.636 132.5C212.916 129.78 214.516 127.46 216.436 125.54C218.356 123.567 220.463 121.967 222.756 120.74C225.05 119.513 227.476 118.607 230.036 118.02C232.65 117.433 235.263 117.14 237.876 117.14C242.41 117.14 246.436 117.833 249.956 119.22C253.476 120.607 256.436 122.633 258.836 125.3C261.236 127.967 263.05 131.22 264.276 135.06C265.556 138.9 266.196 143.3 266.196 148.26V200.5H246.436V148.26C246.436 143.033 245.29 139.113 242.996 136.5C240.703 133.833 237.343 132.5 232.916 132.5C230.89 132.5 228.996 132.847 227.236 133.54C225.53 134.233 224.01 135.247 222.676 136.58C221.396 137.86 220.383 139.487 219.636 141.46C218.89 143.433 218.516 145.7 218.516 148.26V200.5H198.676V148.26C198.676 142.767 197.556 138.767 195.316 136.26C193.13 133.753 189.903 132.5 185.636 132.5C182.756 132.5 180.063 133.22 177.556 134.66C175.103 136.047 172.81 137.967 170.676 140.42V200.5H150.916ZM301.459 128.5C303.112 126.847 304.845 125.327 306.659 123.94C308.525 122.5 310.472 121.3 312.499 120.34C314.579 119.327 316.792 118.553 319.139 118.02C321.485 117.433 324.045 117.14 326.819 117.14C331.299 117.14 335.272 117.913 338.739 119.46C342.205 120.953 345.085 123.087 347.379 125.86C349.725 128.58 351.485 131.86 352.659 135.7C353.885 139.487 354.499 143.673 354.499 148.26V200.5H334.739V148.26C334.739 143.247 333.592 139.38 331.299 136.66C329.005 133.887 325.512 132.5 320.819 132.5C317.405 132.5 314.205 133.273 311.219 134.82C308.232 136.367 305.405 138.473 302.739 141.14V200.5H282.979V118.42H295.059C297.619 118.42 299.299 119.62 300.099 122.02L301.459 128.5ZM443.416 118.42L410.856 200.5H392.936L360.376 118.42H376.696C378.19 118.42 379.416 118.767 380.376 119.46C381.39 120.153 382.056 121.033 382.376 122.1L398.216 165.94C399.07 168.5 399.816 171.007 400.456 173.46C401.096 175.913 401.656 178.367 402.136 180.82C402.67 178.367 403.256 175.913 403.896 173.46C404.536 171.007 405.31 168.5 406.216 165.94L422.376 122.1C422.75 121.033 423.416 120.153 424.376 119.46C425.336 118.767 426.483 118.42 427.816 118.42H443.416ZM471.948 81.62V200.5H452.188V81.62H471.948ZM541.155 139.38C538.915 136.66 536.462 134.74 533.795 133.62C531.182 132.5 528.355 131.94 525.315 131.94C522.328 131.94 519.635 132.5 517.235 133.62C514.835 134.74 512.782 136.447 511.075 138.74C509.368 140.98 508.062 143.86 507.155 147.38C506.248 150.847 505.795 154.953 505.795 159.7C505.795 164.5 506.168 168.58 506.915 171.94C507.715 175.247 508.835 177.967 510.275 180.1C511.715 182.18 513.475 183.7 515.555 184.66C517.635 185.567 519.955 186.02 522.515 186.02C526.622 186.02 530.115 185.167 Line truncated
-</mask>
-<path d="M62.64 150.02C62.64 147.46 62.2667 145.06 61.52 142.82C60.8267 140.527 59.76 138.527 58.32 136.82C56.88 135.113 55.04 133.78 52.8 132.82C50.6133 131.807 48.0533 131.3 45.12 131.3C39.4133 131.3 34.9067 132.927 31.6 136.18C28.3467 139.433 26.2667 144.047 25.36 150.02H62.64ZM24.96 161.94C25.28 166.153 26.0267 169.807 27.2 172.9C28.3733 175.94 29.92 178.473 31.84 180.5C33.76 182.473 36.0267 183.967 38.64 184.98C41.3067 185.94 44.24 186.42 47.44 186.42C50.64 186.42 53.3867 186.047 55.68 185.3C58.0267 184.553 60.0533 183.727 61.76 182.82C63.52 181.913 65.04 181.087 66.32 180.34C67.6533 179.593 68.9333 179.22 70.16 179.22C71.8133 179.22 73.04 179.833 73.84 181.06L79.52 188.26C77.3333 190.82 74.88 192.98 72.16 194.74C69.44 196.447 66.5867 197.833 63.6 198.9C60.6667 199.913 57.6533 200.633 54.56 201.06C51.52 201.487 48.56 201.7 45.68 201.7C39.9733 201.7 34.6667 200.767 29.76 198.9C24.8533 196.98 20.5867 194.18 16.96 190.5C13.3333 186.767 10.48 182.18 8.4 176.74C6.32 171.247 5.28 164.9 5.28 157.7C5.28 152.1 6.18667 146.847 8 141.94C9.81333 136.98 12.4 132.687 15.76 129.06C19.1733 125.38 23.3067 122.473 28.16 120.34C33.0667 118.207 38.5867 117.14 44.72 117.14C49.8933 117.14 54.6667 117.967 59.04 119.62C63.4133 121.273 67.1733 123.7 70.32 126.9C73.4667 130.047 75.92 133.94 77.68 138.58C79.4933 143.167 80.4 148.42 80.4 154.34C80.4 157.327 80.08 159.353 79.44 160.42C78.8 161.433 77.5733 161.94 75.76 161.94H24.96ZM111.869 132.66C114.429 127.753 117.469 123.913 120.989 121.14C124.509 118.313 128.669 116.9 133.469 116.9C137.255 116.9 140.295 117.727 142.589 119.38L141.309 134.18C141.042 135.14 140.642 135.833 140.109 136.26C139.629 136.633 138.962 136.82 138.109 136.82C137.309 136.82 136.109 136.687 134.509 136.42C132.962 136.153 131.442 136.02 129.949 136.02C127.762 136.02 125.815 136.34 124.109 136.98C122.402 137.62 120.855 138.553 119.469 139.78C118.135 140.953 116.935 142.393 115.869 144.1C114.855 145.807 113.895 147.753 112.989 149.94V200.5H93.2288V118.42H104.829C106.855 118.42 108.269 118.793 109.069 119.54C109.869 120.233 110.402 121.513 110.669 123.38L111.869 132.66ZM150.916 200.5V118.42H162.996C165.556 118.42 167.236 119.62 168.036 122.02L169.316 128.1C170.756 126.5 172.25 125.033 173.796 123.7C175.396 122.367 177.076 121.22 178.836 120.26C180.65 119.3 182.57 118.553 184.596 118.02C186.676 117.433 188.943 117.14 191.396 117.14C196.57 117.14 200.81 118.553 204.116 121.38C207.476 124.153 209.983 127.86 211.636 132.5C212.916 129.78 214.516 127.46 216.436 125.54C218.356 123.567 220.463 121.967 222.756 120.74C225.05 119.513 227.476 118.607 230.036 118.02C232.65 117.433 235.263 117.14 237.876 117.14C242.41 117.14 246.436 117.833 249.956 119.22C253.476 120.607 256.436 122.633 258.836 125.3C261.236 127.967 263.05 131.22 264.276 135.06C265.556 138.9 266.196 143.3 266.196 148.26V200.5H246.436V148.26C246.436 143.033 245.29 139.113 242.996 136.5C240.703 133.833 237.343 132.5 232.916 132.5C230.89 132.5 228.996 132.847 227.236 133.54C225.53 134.233 224.01 135.247 222.676 136.58C221.396 137.86 220.383 139.487 219.636 141.46C218.89 143.433 218.516 145.7 218.516 148.26V200.5H198.676V148.26C198.676 142.767 197.556 138.767 195.316 136.26C193.13 133.753 189.903 132.5 185.636 132.5C182.756 132.5 180.063 133.22 177.556 134.66C175.103 136.047 172.81 137.967 170.676 140.42V200.5H150.916ZM301.459 128.5C303.112 126.847 304.845 125.327 306.659 123.94C308.525 122.5 310.472 121.3 312.499 120.34C314.579 119.327 316.792 118.553 319.139 118.02C321.485 117.433 324.045 117.14 326.819 117.14C331.299 117.14 335.272 117.913 338.739 119.46C342.205 120.953 345.085 123.087 347.379 125.86C349.725 128.58 351.485 131.86 352.659 135.7C353.885 139.487 354.499 143.673 354.499 148.26V200.5H334.739V148.26C334.739 143.247 333.592 139.38 331.299 136.66C329.005 133.887 325.512 132.5 320.819 132.5C317.405 132.5 314.205 133.273 311.219 134.82C308.232 136.367 305.405 138.473 302.739 141.14V200.5H282.979V118.42H295.059C297.619 118.42 299.299 119.62 300.099 122.02L301.459 128.5ZM443.416 118.42L410.856 200.5H392.936L360.376 118.42H376.696C378.19 118.42 379.416 118.767 380.376 119.46C381.39 120.153 382.056 121.033 382.376 122.1L398.216 165.94C399.07 168.5 399.816 171.007 400.456 173.46C401.096 175.913 401.656 178.367 402.136 180.82C402.67 178.367 403.256 175.913 403.896 173.46C404.536 171.007 405.31 168.5 406.216 165.94L422.376 122.1C422.75 121.033 423.416 120.153 424.376 119.46C425.336 118.767 426.483 118.42 427.816 118.42H443.416ZM471.948 81.62V200.5H452.188V81.62H471.948ZM541.155 139.38C538.915 136.66 536.462 134.74 533.795 133.62C531.182 132.5 528.355 131.94 525.315 131.94C522.328 131.94 519.635 132.5 517.235 133.62C514.835 134.74 512.782 136.447 511.075 138.74C509.368 140.98 508.062 143.86 507.155 147.38C506.248 150.847 505.795 154.953 505.795 159.7C505.795 164.5 506.168 168.58 506.915 171.94C507.715 175.247 508.835 177.967 510.275 180.1C511.715 182.18 513.475 183.7 515.555 184.66C517.635 185.567 519.955 186.02 522.515 186.02C526.622 186.02 530.115 185.167 Line truncated
-<path d="M62.64 150.02V154.02H66.64V150.02H62.64ZM61.52 142.82L57.6912 143.978L57.7075 144.031L57.7253 144.085L61.52 142.82ZM52.8 132.82L51.1182 136.449L51.1709 136.474L51.2243 136.497L52.8 132.82ZM31.6 136.18L28.7947 133.329L28.7831 133.34L28.7716 133.352L31.6 136.18ZM25.36 150.02L21.4053 149.42L20.707 154.02H25.36V150.02ZM24.96 161.94V157.94H20.6447L20.9715 162.243L24.96 161.94ZM27.2 172.9L23.46 174.319L23.4641 174.329L23.4683 174.34L27.2 172.9ZM31.84 180.5L28.9362 183.251L28.9545 183.27L28.9731 183.289L31.84 180.5ZM38.64 184.98L37.1939 188.709L37.2393 188.727L37.2851 188.744L38.64 184.98ZM55.68 185.3L54.4672 181.488L54.4544 181.492L54.4417 181.497L55.68 185.3ZM61.76 182.82L59.9282 179.264L59.9057 179.276L59.8834 179.288L61.76 182.82ZM66.32 180.34L64.3656 176.85L64.3349 176.867L64.3045 176.885L66.32 180.34ZM73.84 181.06L70.4896 183.245L70.588 183.396L70.6996 183.537L73.84 181.06ZM79.52 188.26L82.5615 190.858L84.6944 188.361L82.6604 185.783L79.52 188.26ZM72.16 194.74L74.286 198.128L74.3096 198.113L74.333 198.098L72.16 194.74ZM63.6 198.9L64.9061 202.681L64.9258 202.674L64.9454 202.667L63.6 198.9ZM54.56 201.06L54.0135 197.098L54.004 197.099L54.56 201.06ZM29.76 198.9L28.3024 202.625L28.32 202.632L28.3377 202.639L29.76 198.9ZM16.96 190.5L14.0909 193.287L14.1009 193.297L14.111 193.308L16.96 190.5ZM8.4 176.74L4.65916 178.156L4.66379 178.169L8.4 176.74ZM8 141.94L11.752 143.327L11.7568 143.313L8 141.94ZM15.76 129.06L12.8273 126.34L12.8258 126.342L15.76 129.06ZM28.16 120.34L26.5651 116.672L26.5577 116.675L26.5504 116.678L28.16 120.34ZM59.04 119.62L60.4545 115.878L59.04 119.62ZM70.32 126.9L67.4679 129.705L67.4797 129.717L67.4916 129.728L70.32 126.9ZM77.68 138.58L73.94 139.999L73.9499 140.025L73.9602 140.051L77.68 138.58ZM79.44 160.42L82.822 162.556L82.8464 162.517L82.87 162.478L79.44 160.42ZM62.64 150.02H66.64C66.64 147.066 66.2083 144.236 65.3147 141.555L61.52 142.82L57.7253 144.085C58.325 145.884 58.64 147.854 58.64 150.02H62.64ZM61.52 142.82L65.3488 141.662C64.5095 138.886 63.1936 136.393 61.3772 134.241L58.32 136.82L55.2628 139.399C56.3264 140.66 57.1439 142.167 57.6912 143.978L61.52 142.82ZM58.32 136.82L61.3772 134.241C59.4997 132.015 57.1349 130.326 54.3757 129.143L52.8 132.82L51.2243 136.497C52.9451 137.234 54.2603 138.211 55.2628 139.399L58.32 136.82ZM52.8 132.82L54.4818 129.191C51.6664 127.886 48.5106 127.3 45.12 127.3V131.3V135.3C47.5961 135.3 49.5602 135.727 51.1182 136.449L52.8 132.82ZM45.12 131.3V127.3C38.6197 127.3 33.014 129.177 28.7947 133.329L31.6 136.18L34.4053 139.031C36.7993 136.676 40.207 135.3 45.12 135.3V131.3ZM31.6 136.18L28.7716 133.352C24.7503 137.373 22.4001 142.866 21.4053 149.42L25.36 150.02L29.3147 150.62C30.1332 145.228 31.9431 141.494 34.4284 139.008L31.6 136.18ZM25.36 150.02V154.02H62.64V150.02V146.02H25.36V150.02ZM24.96 161.94L20.9715 162.243C21.3129 166.738 22.1193 170.784 23.46 174.319L27.2 172.9L30.94 171.481C29.934 168.829 29.2471 165.569 28.9485 161.637L24.96 161.94ZM27.2 172.9L23.4683 174.34C24.7985 177.787 26.6026 180.788 28.9362 183.251L31.84 180.5L34.7438 177.749C33.2374 176.159 31.9482 174.093 30.9317 171.46L27.2 172.9ZM31.84 180.5L28.9731 183.289C31.3011 185.682 34.0549 187.492 37.1939 188.709L38.64 184.98L40.0861 181.251C37.9984 180.441 36.2189 179.265 34.7069 177.711L31.84 180.5ZM38.64 184.98L37.2851 188.744C40.4386 189.879 43.8371 190.42 47.44 190.42V186.42V182.42C44.6429 182.42 42.1747 182.001 39.9949 181.216L38.64 184.98ZM47.44 186.42V190.42C50.9318 190.42 54.1187 190.015 56.9183 189.103L55.68 185.3L54.4417 181.497C52.6546 182.078 50.3482 182.42 47.44 182.42V186.42ZM55.68 185.3L56.8928 189.112C59.4097 188.311 61.6691 187.398 63.6366 186.352L61.76 182.82L59.8834 179.288C58.4376 180.056 56.6436 180.796 54.4672 181.488L55.68 185.3ZM61.76 182.82L63.5918 186.376C65.3948 185.447 66.979 184.586 68.3355 183.795L66.32 180.34L64.3045 176.885C63.101 177.587 61.6452 178.38 59.9282 179.264L61.76 182.82ZM66.32 180.34L68.2744 183.83C69.1554 183.337 69.7664 183.22 70.16 183.22V179.22V175.22C68.1002 175.22 66.1512 175.85 64.3656 176.85L66.32 180.34ZM70.16 179.22V183.22C70.4918 183.22 70.576 183.28 70.5312 183.258C70.5118 183.248 70.4889 183.233 70.4713 183.217C70.4548 183.202 70.4631 183.204 70.4896 183.245L73.84 181.06L77.1904 178.875C75.5467 176.354 72.9601 175.22 70.16 175.22V179.22ZM73.84 181.06L70.6996 183.537L76.3796 190.737L79.52 188.26L82.6604 185.783L76.9804 178.583L73.84 181.06ZM79.52 188.26L76.4785 185.662C74.5331 187.94 72.3697 189.84 69.987 191.382L72.16 194.74L74.333 198.098C77.3903 196.12 80.1335 193.7 82.5615 190.858L79.52 188.26ZM72.16 194.74L70.034 191.352C67.5567 192.906 64.9647 194.165 62.2547 195.133L63.6 198.9L64.9454 202.667C68.2086 201.502 71.3233 199.987 74.286 198.128L72.16 194.74ZM63.6 198.9L62.2939 195.119C59.6088 196.047 56.8499 196.706 54.0135 197.098L54.56 201.06L55.1066 205.022C58.4568 204.56 61.7246 203.78 64.9061 202.681L63.6 198.9ZM54.56 201.06L54.004 197.099C51.1363 197.501 48.3623 197.7 45.68 197.7V201.7V205.7C48.7577 205.7 51.9037 205.472 55.116Line truncated
-<rect width="443" height="64" transform="translate(0 226.5)" fill="white"/>
-<path d="M25.92 257.58C26.944 255.361 28.2027 253.633 29.696 252.396C31.1893 251.137 33.0133 250.508 35.168 250.508C35.8507 250.508 36.5013 250.583 37.12 250.732C37.76 250.881 38.3253 251.116 38.816 251.436L38.4 255.692C38.272 256.225 37.952 256.492 37.44 256.492C37.1413 256.492 36.704 256.428 36.128 256.3C35.552 256.172 34.9013 256.108 34.176 256.108C33.152 256.108 32.2347 256.257 31.424 256.556C30.6347 256.855 29.92 257.303 29.28 257.9C28.6613 258.476 28.096 259.201 27.584 260.076C27.0933 260.929 26.6453 261.911 26.24 263.02V283.5H20.512V251.084H23.776C24.3947 251.084 24.8213 251.201 25.056 251.436C25.2907 251.671 25.4507 252.076 25.536 252.652L25.92 257.58ZM56.676 250.572C59.044 250.572 61.1773 250.967 63.076 251.756C64.9747 252.545 66.596 253.665 67.94 255.116C69.284 256.567 70.308 258.327 71.012 260.396C71.7373 262.444 72.1 264.737 72.1 267.276C72.1 269.836 71.7373 272.14 71.012 274.188C70.308 276.236 69.284 277.985 67.94 279.436C66.596 280.887 64.9747 282.007 63.076 282.796C61.1773 283.564 59.044 283.948 56.676 283.948C54.2867 283.948 52.132 283.564 50.212 282.796C48.3133 282.007 46.692 280.887 45.348 279.436C44.004 277.985 42.9693 276.236 42.244 274.188C41.54 272.14 41.188 269.836 41.188 267.276C41.188 264.737 41.54 262.444 42.244 260.396C42.9693 258.327 44.004 256.567 45.348 255.116C46.692 253.665 48.3133 252.545 50.212 251.756C52.132 250.967 54.2867 250.572 56.676 250.572ZM56.676 279.5C59.876 279.5 62.2653 278.433 63.844 276.3C65.4227 274.145 66.212 271.148 66.212 267.308C66.212 263.447 65.4227 260.439 63.844 258.284C62.2653 256.129 59.876 255.052 56.676 255.052C55.0547 255.052 53.636 255.329 52.42 255.884C51.2253 256.439 50.2227 257.239 49.412 258.284C48.6227 259.329 48.0253 260.62 47.62 262.156C47.236 263.671 47.044 265.388 47.044 267.308C47.044 271.148 47.8333 274.145 49.412 276.3C51.012 278.433 53.4333 279.5 56.676 279.5ZM93.4885 250.572C95.8565 250.572 97.9898 250.967 99.8885 251.756C101.787 252.545 103.409 253.665 104.753 255.116C106.097 256.567 107.121 258.327 107.825 260.396C108.55 262.444 108.913 264.737 108.913 267.276C108.913 269.836 108.55 272.14 107.825 274.188C107.121 276.236 106.097 277.985 104.753 279.436C103.409 280.887 101.787 282.007 99.8885 282.796C97.9898 283.564 95.8565 283.948 93.4885 283.948C91.0992 283.948 88.9445 283.564 87.0245 282.796C85.1258 282.007 83.5045 280.887 82.1605 279.436C80.8165 277.985 79.7818 276.236 79.0565 274.188C78.3525 272.14 78.0005 269.836 78.0005 267.276C78.0005 264.737 78.3525 262.444 79.0565 260.396C79.7818 258.327 80.8165 256.567 82.1605 255.116C83.5045 253.665 85.1258 252.545 87.0245 251.756C88.9445 250.967 91.0992 250.572 93.4885 250.572ZM93.4885 279.5C96.6885 279.5 99.0778 278.433 100.657 276.3C102.235 274.145 103.025 271.148 103.025 267.308C103.025 263.447 102.235 260.439 100.657 258.284C99.0778 256.129 96.6885 255.052 93.4885 255.052C91.8672 255.052 90.4485 255.329 89.2325 255.884C88.0378 256.439 87.0352 257.239 86.2245 258.284C85.4352 259.329 84.8378 260.62 84.4325 262.156C84.0485 263.671 83.8565 265.388 83.8565 267.308C83.8565 271.148 84.6458 274.145 86.2245 276.3C87.8245 278.433 90.2458 279.5 93.4885 279.5ZM126.027 284.012C123.467 284.012 121.493 283.297 120.107 281.868C118.741 280.439 118.059 278.38 118.059 275.692V255.852H114.155C113.813 255.852 113.525 255.756 113.291 255.564C113.056 255.351 112.939 255.031 112.939 254.604V252.332L118.251 251.66L119.562 241.644C119.627 241.324 119.765 241.068 119.979 240.876C120.213 240.663 120.512 240.556 120.875 240.556H123.755V251.724H133.131V255.852H123.755V275.308C123.755 276.673 124.085 277.687 124.747 278.348C125.408 279.009 126.261 279.34 127.307 279.34C127.904 279.34 128.416 279.265 128.843 279.116C129.291 278.945 129.675 278.764 129.995 278.572C130.315 278.38 130.581 278.209 130.795 278.06C131.029 277.889 131.232 277.804 131.403 277.804C131.701 277.804 131.968 277.985 132.203 278.348L133.867 281.068C132.885 281.985 131.701 282.711 130.315 283.244C128.928 283.756 127.499 284.012 126.027 284.012ZM178.947 256.844C178.776 257.079 178.606 257.26 178.435 257.388C178.264 257.516 178.03 257.58 177.731 257.58C177.411 257.58 177.059 257.452 176.675 257.196C176.291 256.919 175.811 256.62 175.235 256.3C174.68 255.98 173.987 255.692 173.155 255.436C172.344 255.159 171.342 255.02 170.147 255.02C168.547 255.02 167.139 255.308 165.923 255.884C164.707 256.439 163.683 257.249 162.851 258.316C162.04 259.383 161.422 260.673 160.995 262.188C160.59 263.703 160.387 265.399 160.387 267.276C160.387 269.239 160.611 270.988 161.059 272.524C161.507 274.039 162.136 275.319 162.947 276.364C163.779 277.388 164.771 278.177 165.923 278.732C167.096 279.265 168.408 279.532 169.859 279.532C171.246 279.532 172.387 279.372 173.283 279.052C174.179 278.711 174.915 278.337 175.491 277.932C176.088 277.527 176.579 277.164 176.963 276.844C177.368 276.503 177.763 276.332 178.147 276.332C178.616 276.332 178.979 276.513 179.235 276.876L180.835 278.956C180.131 279.831 179.331 280.577 178.435 281.196C177.539 281.815 176.568 282.337 175.523 Line truncated
-<path d="M1200 218.5H1260L1242 206.5H1182L1200 218.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1200 218.5V278.5L1182 266.5V206.5L1200 218.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1260 218.5H1200V278.5H1260V218.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1200 158.5H1260L1242 146.5H1182L1200 158.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1200 158.5V218.5L1182 206.5V146.5L1200 158.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1260 158.5H1200V218.5H1260V158.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1347 14.5H1407L1389 2.5H1329L1347 14.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1347 14.5V74.5L1329 62.5V2.5L1347 14.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1407 14.5H1347V74.5H1407V14.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1242 86.5H1302L1284 74.5H1224L1242 86.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1242 86.5V146.5L1224 134.5V74.5L1242 86.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1302 86.5H1242V146.5H1302V86.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1218 230.5H1278L1260 218.5H1200L1218 230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1218 230.5V290.5L1200 278.5V218.5L1218 230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1278 230.5H1218V290.5H1278V230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1158 230.5H1218L1200 218.5H1140L1158 230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1158 230.5V290.5L1140 278.5V218.5L1158 230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1218 230.5H1158V290.5H1218V230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1158 230.5H1218L1200 218.5H1140L1158 230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1158 230.5V290.5L1140 278.5V218.5L1158 230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1218 230.5H1158V290.5H1218V230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1098 230.5V290.5L1080 278.5V218.5L1098 230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1038 230.5H1098L1080 218.5H1020L1038 230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1038 230.5V290.5L1020 278.5V218.5L1038 230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1098 230.5H1038V290.5H1098V230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-</svg>

compositions/apex/docker-compose.yml

@@ -1,32 +1,75 @@
 # commons/compositions/apex/docker-compose.yml
-# The shared apex CORE composition. Consumed by each node via include: (SP3).
+# The shared apex CORE composition. Available to node compositions through include:.
 # FULLY ${env}-parameterized; NO './' host paths (relative paths in an included file
 # resolve against THIS dir — always use absolute ${APEX_*} vars). The engine exports
 # APEX_TIER*/APEX_TIER*_SHARED/APEX_TIER_ROOT*/APEX_SUBNET/APEX_NODE_FQDN at invoke.
 x-environment: &shared-environment
-  TZ: "Europe/Moscow"
+  TZ: "${TZ:-Etc/UTC}" # override per node via TZ in the env; default UTC
 
 services:
-  # --- edge (host network; profile-gated — a node opts in via COMPOSE_PROFILES) ---
+  adguard:
-  xray:
+    image: adguard/adguardhome:v0.107.79
-    image: ghcr.io/xtls/xray-core:26.3.27
+    container_name: apex-adguard
-    container_name: apex-xray
-    profiles: [xray]
     volumes:
-      - ${APEX_TIER1}/xray/data/config.jsonc:/etc/xray/config.jsonc:ro
+      - ${APEX_TIER1}/adguard/data:/opt/adguardhome/work
-      - ${APEX_TIER1_SHARED}/apex/letsencrypt:/etc/letsencrypt/live:ro
+      - ${APEX_TIER1}/adguard/conf:/opt/adguardhome/conf
-      - ${APEX_TIER3_SHARED}/apex/xray/log:/var/log/xray/
-      - ${APEX_TIER3_SHARED}/apex/xray/shm:/dev/shm/xray/
     environment:
       <<: *shared-environment
-    network_mode: "host"
+    network_mode: "service:traefik"
-    command: ["-c", "/etc/xray/config.jsonc"]
     restart: unless-stopped
-    user: "0:0"
     labels:
-      # backup (resticontainer, core/all-nodes): xray config only (logs + shm are ephemeral)
+      # backup (resticontainer): adguard config only (work data regenerable)
       - "restic.enable=true"
-      - "restic.backup.paths=/etc/xray/config.jsonc"
+      - "restic.backup.paths=/opt/adguardhome/conf"
+    cap_add:
+      - NET_BIND_SERVICE
+      - SYS_NICE
+      - CAP_CHOWN
+    profiles: [ adguard ]
+    security_opt:
+      - no-new-privileges:true
+
+  # --- optional: observability agent (profile-gated) ---
+  # Collect host/container metrics and logs for configured remote endpoints.
+  # Alloy requires outbound access to those endpoints through the direct network.
+  alloy:
+    image: grafana/alloy:v1.20.0
+    container_name: apex-alloy
+    volumes:
+      - ${APEX_COMMONS}/etc/alloy/config.alloy:/etc/alloy/config.alloy:ro
+      - ${APEX_TIER2}/alloy/data:/var/lib/alloy
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /run/containerd/containerd.sock:/run/containerd/containerd.sock:ro # cadvisor per-container stats on nodes using the containerd snapshotter (overlayfs)
+      - /proc:/host/proc:ro
+      - /sys:/host/sys:ro
+      - /:/rootfs:ro
+      - /var/log/journal:/var/log/journal:ro
+      - /run/log/journal:/run/log/journal:ro
+      - /etc/machine-id:/etc/machine-id:ro
+    environment:
+      <<: *shared-environment
+      APEX_NODE_HOST: "${APEX_NODE_HOST?error}" # node + instance metric/log labels
+      # push targets + machine auth (per-node scalars in apex.env / secret .env)
+      APEX_LOKI_PUSH_URL: "${APEX_LOKI_PUSH_URL?error}"
+      APEX_PROM_PUSH_URL: "${APEX_PROM_PUSH_URL?error}"
+      APEX_OBS_BASIC_AUTH_USER: "${APEX_OBS_BASIC_AUTH_USER?error}"
+      APEX_OBS_BASIC_AUTH_PASS: "${APEX_OBS_BASIC_AUTH_PASS?error}"
+      GOMEMLIMIT: "${ALLOY_GOMEMLIMIT:-160MiB}"
+      GOGC: "${ALLOY_GOGC:-75}"
+    networks:
+      direct: {}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    command:
+      - run
+      - --server.http.listen-addr=0.0.0.0:12345
+      - --storage.path=/var/lib/alloy
+      - /etc/alloy/config.alloy
+    restart: unless-stopped
+    user: "0:0" # root: required for host metrics, journal, and docker.sock
+    cgroup: host # host cgroup namespace: the cadvisor exporter needs it to read per-container cgroups (else machine-root only)
+    hostname: "${APEX_NODE_HOST}" # UTS hostname => node_exporter's node_uname_info `nodename` (else the container id)
+    profiles: [ alloy ]
     security_opt:
       - no-new-privileges:true
 
@@ -39,42 +82,42 @@ services:
   # container mounts to host paths (via the /srv mount), snapshots the union
   # in a single restic run, then restarts + runs post-hooks. Every other subcommand
   # (init/snapshots/forget/…) passes straight through to restic. Needs the docker
-  # socket (list/inspect/exec/stop/start) and /srv read-only (the fleet's data root)
+  # socket (list/inspect/exec/stop/start) and read-only access to /srv.
-  # — a privileged manual one-shot, same trust surface as the old restic core.
   resticontainer:
-    image: ghcr.io/deytenit/resticontainer:0.19.1
+    image: ghcr.io/deytenit/resticontainer:0.19.1@sha256:85457ee665b1c6c8438ac3494c286d267169ee023857da3a605a6fe16dab3d82
     container_name: apex-resticontainer
-    profiles: [manual]
     volumes:
       - ${APEX_TIER3}/restic/cache:/var/restic-cache
-      - /var/run/docker.sock:/var/run/docker.sock          # query + exec/stop/start
+      - /var/run/docker.sock:/var/run/docker.sock # query + exec/stop/start
       # apex tier data + checkouts live under /srv, and the @tierN bind mounts are
       # ABSOLUTE symlinks into /srv (compositions/<p>/@tierN -> /srv/com-…-tierN/<p>).
       # Mount /srv at the same path with RESTIC_HOSTFS=/ so restic follows those
       # symlinks and records the real host paths — a bare /:/hostfs cannot, because
       # the absolute symlink target /srv/… is dangling inside the container.
-      - /srv:/srv:ro                                        # read the data (resolves @tierN symlinks)
+      - /srv:/srv:ro # read the data (resolves @tierN symlinks)
     environment:
       <<: *shared-environment
-      RESTIC_HOSTFS: "/"                                    # mount sources are already absolute (/srv/…)
+      RESTIC_HOSTFS: "/" # mount sources are already absolute (/srv/…)
-      RESTIC_REPOSITORY: "${APEX_RESTIC_REPOSITORY?error}"
+      RESTIC_REPOSITORY: "${APEX_RESTIC_REPOSITORY:-}"
-      RESTIC_PASSWORD: "${APEX_RESTIC_PASSWORD?error}"
+      RESTIC_PASSWORD: "${APEX_RESTIC_PASSWORD:-}"
-      AWS_ACCESS_KEY_ID: "${APEX_RESTIC_AWS_ACCESS_KEY_ID?error}"
+      AWS_ACCESS_KEY_ID: "${APEX_RESTIC_AWS_ACCESS_KEY_ID:-}"
-      AWS_SECRET_ACCESS_KEY: "${APEX_RESTIC_AWS_SECRET_ACCESS_KEY?error}"
+      AWS_SECRET_ACCESS_KEY: "${APEX_RESTIC_AWS_SECRET_ACCESS_KEY:-}"
-      AWS_REGION: "ru-central1"
+      # S3 backend region + endpoint. Defaults suit AWS S3 (us-east-1, no custom
-      AWS_ENDPOINT_URL: "https://storage.yandexcloud.net/"
+      # endpoint); set APEX_RESTIC_AWS_REGION / APEX_RESTIC_AWS_ENDPOINT (node apex.env)
-      # max compression is fleet policy (backup/run also forces --compression max on
+      # for any other S3-compatible store (Yandex, Backblaze B2, MinIO, …).
-      # the CLI; needs repo format v2, which a fresh restic init creates). Per-node
+      AWS_REGION: "${APEX_RESTIC_AWS_REGION:-us-east-1}"
-      # override via APEX_RESTIC_COMPRESSION only if ever needed.
+      AWS_ENDPOINT_URL: "${APEX_RESTIC_AWS_ENDPOINT:-}"
+      # Compression requires repository format v2. The default is max;
+      # override it with APEX_RESTIC_COMPRESSION for direct container use.
       RESTIC_COMPRESSION: "${APEX_RESTIC_COMPRESSION:-max}"
       RESTIC_CACHE_DIR: /var/restic-cache
     network_mode: "host"
+    profiles: [ manual ]
 
   # --- docker socket proxy ---
   socket-proxy:
     image: tecnativa/docker-socket-proxy:0
     container_name: apex-socket-proxy
-    user: "0:0"
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock:ro
     environment:
@@ -84,14 +127,12 @@ services:
     networks:
       socket: {}
     restart: unless-stopped
+    user: "0:0"
 
   # --- edge reverse proxy (the core anchor on direct) ---
   traefik:
-    image: traefik:3.6
+    image: traefik:3.7
     container_name: apex-traefik
-    user: "${APEX_UID?error}:${APEX_GID?error}"
-    group_add:
-      - "${APEX_SHARED_GID?error}"
     depends_on:
       - socket-proxy
     volumes:
@@ -100,7 +141,7 @@ services:
       - ${APEX_TIER1}/traefik/dynamic:/etc/traefik/dynamic:ro
     environment:
       <<: *shared-environment
-      CF_DNS_API_TOKEN: "${APEX_TRAEFIK_CF_DNS_API_TOKEN?error}"
+      CF_DNS_API_TOKEN: "${APEX_TRAEFIK_CF_DNS_API_TOKEN:-}"
       UMASK: "0002"
     ports:
       - '0.0.0.0:80:80'
@@ -133,13 +174,13 @@ services:
       - "--entrypoints.websecure.http3"
       - "--certificatesresolvers.cloudflare.acme.dnschallenge=true"
       - "--certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare"
-      - "--certificatesresolvers.cloudflare.acme.email=${APEX_TRAEFIK_ACME_EMAIL?error}"
+      - "--certificatesresolvers.cloudflare.acme.email=${APEX_TRAEFIK_ACME_EMAIL:-admin@localhost}"
       - "--certificatesresolvers.cloudflare.acme.storage=/letsencrypt/acme.json"
     restart: unless-stopped
     labels:
       - "traefik.enable=true"
       - "traefik.http.routers.apex-traefik.rule=Host(`${APEX_NODE_FQDN}`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
-      - "traefik.http.middlewares.xray-enclave.ipallowlist.sourcerange=198.18.0.0/16"
+      - "traefik.http.middlewares.xray-enclave.ipallowlist.sourcerange=${APEX_ENCLAVE_SUBNET:-198.18.0.0/16}"
       - "traefik.http.routers.apex-traefik.middlewares=xray-enclave@docker"
       - "traefik.http.routers.apex-traefik.service=api@internal"
       # adguard route (backend only present when the adguard profile is active)
@@ -148,76 +189,34 @@ services:
       - "traefik.http.middlewares.apex-adguard-strip-prefix.stripprefix.prefixes=/adguard"
       - "traefik.http.routers.apex-adguard.service=apex-adguard"
       - "traefik.http.services.apex-adguard.loadbalancer.server.port=3000"
-      # backup (resticontainer, core/all-nodes): dynamic routing rules only
+      # backup (resticontainer): dynamic routing rules only
       # (letsencrypt certs are regenerable via ACME/Cloudflare DNS-01)
       - "restic.enable=true"
       - "restic.backup.paths=/etc/traefik/dynamic"
+    user: "${APEX_UID?error}:${APEX_GID?error}"
+    group_add:
+      - "${APEX_SHARED_GID?error}"
     security_opt:
       - no-new-privileges:true
-
+  xray:
-  # --- optional: adguard (shares traefik netns) ---
+    image: ghcr.io/xtls/xray-core:26.9.9
-  adguard:
+    container_name: apex-xray
-    image: adguard/adguardhome:v0.107.72
-    container_name: apex-adguard
-    profiles: [adguard]
     volumes:
-      - ${APEX_TIER1}/adguard/data:/opt/adguardhome/work
+      - ${APEX_TIER1}/xray/data/config.jsonc:/etc/xray/config.jsonc:ro
-      - ${APEX_TIER1}/adguard/conf:/opt/adguardhome/conf
+      - ${APEX_TIER1_SHARED}/apex/letsencrypt:/etc/letsencrypt/live:ro
+      - ${APEX_TIER3_SHARED}/apex/xray/log:/var/log/xray/
+      - ${APEX_TIER3_SHARED}/apex/xray/shm:/dev/shm/xray/
     environment:
       <<: *shared-environment
+    network_mode: "host"
+    command: [ "-c", "/etc/xray/config.jsonc" ]
+    restart: unless-stopped
     labels:
-      # backup (resticontainer, core/all-nodes): adguard config only (work data regenerable)
+      # backup (resticontainer): xray config only (logs + shm are ephemeral)
       - "restic.enable=true"
-      - "restic.backup.paths=/opt/adguardhome/conf"
+      - "restic.backup.paths=/etc/xray/config.jsonc"
-    network_mode: "service:traefik"
+    user: "0:0"
-    restart: unless-stopped
+    profiles: [ xray ]
-    cap_add:
-      - NET_BIND_SERVICE
-      - SYS_NICE
-      - CAP_CHOWN
-    security_opt:
-      - no-new-privileges:true
-
-  # --- optional: observability agent (profile-gated) ---
-  # Grafana Alloy: one agent per node. Collects host metrics (node_exporter-
-  # equivalent) + container metrics (cadvisor) and PUSHES them via remote_write;
-  # collects systemd-journal + docker logs and PUSHES them to Loki. All targets
-  # are OUTBOUND to the central gfn stack (Prometheus + Loki live on icarus), so
-  # Alloy needs egress (direct, not enclave). Replaces the old store-only `loki`
-  # service (which had no feeder). Config is a committed, env-parameterized asset.
-  alloy:
-    image: grafana/alloy:v1.17.1
-    container_name: apex-alloy
-    profiles: [alloy]
-    user: "0:0"  # root: required for host metrics, journal, and docker.sock
-    volumes:
-      - ${APEX_COMMONS}/assets/alloy/config.alloy:/etc/alloy/config.alloy:ro
-      - ${APEX_TIER2}/alloy/data:/var/lib/alloy
-      - /var/run/docker.sock:/var/run/docker.sock:ro
-      - /proc:/host/proc:ro
-      - /sys:/host/sys:ro
-      - /:/rootfs:ro
-      - /var/log/journal:/var/log/journal:ro
-      - /run/log/journal:/run/log/journal:ro
-      - /etc/machine-id:/etc/machine-id:ro
-    environment:
-      <<: *shared-environment
-      APEX_NODE: "${APEX_NODE?error}"
-      # push targets + machine auth (per-node scalars in apex.env / secret .env)
-      APEX_LOKI_PUSH_URL: "${APEX_LOKI_PUSH_URL?error}"
-      APEX_PROM_PUSH_URL: "${APEX_PROM_PUSH_URL?error}"
-      APEX_OBS_BASIC_AUTH_USER: "${APEX_OBS_BASIC_AUTH_USER?error}"
-      APEX_OBS_BASIC_AUTH_PASS: "${APEX_OBS_BASIC_AUTH_PASS?error}"
-    command:
-      - run
-      - --server.http.listen-addr=0.0.0.0:12345
-      - --storage.path=/var/lib/alloy
-      - /etc/alloy/config.alloy
-    networks:
-      direct: {}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    restart: unless-stopped
     security_opt:
       - no-new-privileges:true
 

engine/identity.py

@@ -1,76 +0,0 @@
-# engine/identity.py
-"""Node identity from host FQDN + node.env (replaces bash root_require_node).
-
-Decision B: the FQDN <node>.a<x>.apex.ermnvldmr.com keeps the cluster; we parse
-node+cluster from it. node.env owns APEX_SUBNET and mirrors APEX_CLUSTER; hostname
-wins on drift. Falls back to the repo directory name when the FQDN is not yet
-apex-shaped (mid-cutover).
-"""
-from __future__ import annotations
-
-import os
-import re
-import sys
-from dataclasses import dataclass
-from typing import Dict, List, Tuple
-
-_FQDN = re.compile(r"^(?P<node>[^.]+)\.(?P<cluster>a\d+)\.apex\.ermnvldmr\.com$")
-_DIRNAME = re.compile(r"^(?P<node>[^.]+)\.apex\.ermnvldmr\.com")
-
-
-@dataclass
-class Node:
-    name: str
-    cluster: str
-    subnet: str
-    fqdn: str
-
-
-def read_env(path: str) -> Dict[str, str]:
-    """Parse a KEY=value file (shell-sourceable). Ignores comments/blank lines."""
-    out: Dict[str, str] = {}
-    if not os.path.isfile(path):
-        return out
-    with open(path) as f:
-        for line in f:
-            line = line.strip()
-            if not line or line.startswith("#") or "=" not in line:
-                continue
-            key, _, val = line.partition("=")
-            out[key.strip()] = val.strip().strip('"').strip("'")
-    return out
-
-
-def resolve(fqdn: str, repo_root: str) -> Tuple[Node, List[str]]:
-    """Return (Node, warnings). Exits 66 (EX_NOINPUT) if APEX_SUBNET is missing."""
-    warns: List[str] = []
-    env = read_env(os.path.join(repo_root, "node.env"))
-
-    m = _FQDN.match(fqdn or "")
-    if m:
-        name, cluster = m.group("node"), m.group("cluster")
-        env_cluster = env.get("APEX_CLUSTER")
-        if env_cluster and env_cluster != cluster:
-            warns.append(
-                f"cluster drift: hostname says {cluster!r}, node.env says "
-                f"{env_cluster!r}; hostname wins."
-            )
-    else:
-        # Fallback: derive node from repo dir name; cluster from node.env.
-        warns.append(
-            f"FQDN {fqdn!r} is not apex-shaped; falling back to repo dir + node.env."
-        )
-        base = os.path.basename(os.path.realpath(repo_root))
-        dm = _DIRNAME.match(base)
-        name = dm.group("node") if dm else base
-        cluster = env.get("APEX_CLUSTER", "")
-        if not cluster:
-            sys.stderr.write("identity: no cluster in FQDN or node.env\n")
-            raise SystemExit(66)
-
-    subnet = env.get("APEX_SUBNET", "")
-    if not subnet:
-        sys.stderr.write("identity: APEX_SUBNET missing from node.env\n")
-        raise SystemExit(66)
-
-    return Node(name=name, cluster=cluster, subnet=subnet, fqdn=fqdn), warns

engine/lib/ufw.py

@@ -1,77 +0,0 @@
-# engine/lib/ufw.py
-"""ctx.ufw — ufw + ufw-docker rule application with self-healing marker migration.
-
-Injects the APEX marker block into before/after.rules, deleting BOTH the old ROOT
-and the new APEX blocks first (so re-running after cutover migrates the marker in
-place — SP2 §6). Docker rules go through ufw-docker; host rules through plain ufw.
-"""
-from __future__ import annotations
-import os
-
-APEX_START = "# START APEX.ERMNVLDMR.COM RULES"
-APEX_END = "# END APEX.ERMNVLDMR.COM RULES"
-OLD_PAIRS = [("# START ROOT.ERMNVLDMR.COM RULES", "# END ROOT.ERMNVLDMR.COM RULES")]
-UFW_DOCKER_BIN = os.path.expanduser("~/.local/bin/ufw-docker")
-
-
-def clean_rules(text: str):
-    """Strip whitespace, drop blank + comment lines (matches the bash awk filter)."""
-    return [s for s in (ln.strip() for ln in text.splitlines()) if s and not s.startswith("#")]
-
-
-class Ufw:
-    def __init__(self, log, sys_, host):
-        self.log, self.sys, self.host = log, sys_, host
-
-    def _inject_file(self, target: str, source_file: str, dry_run: bool):
-        if not os.path.isfile(source_file):
-            return
-        if not self.sys.ok(["sudo", "test", "-f", target]):
-            self.log.error(f"Target file does not exist: {target}")
-            raise SystemExit(1)
-        content = "\n".join(clean_rules(open(source_file).read()))
-        self.host.inject_block(target, content, APEX_START, APEX_END,
-                               anchor="# End required lines", old_pairs=OLD_PAIRS,
-                               dry_run=dry_run, sudo=True)
-
-    def apply(self, rules_dir: str, config_dir=None, dry_run: bool = False) -> None:
-        # EX_NOINPUT guards first, tools second — same order/codes as root_template_ufw.
-        if not os.path.isdir(rules_dir):
-            self.log.error(f"Rules directory does not exist: {rules_dir}")
-            raise SystemExit(66)
-        if config_dir and not os.path.isdir(config_dir):
-            self.log.error(f"Config directory does not exist: {config_dir}")
-            raise SystemExit(66)
-        if not self.sys.ok(["sudo", "ufw", "--version"]):
-            self.log.error("ufw not installed/working via sudo. Run configure/base first.")
-            raise SystemExit(1)
-        if not (os.path.isfile(UFW_DOCKER_BIN) and os.access(UFW_DOCKER_BIN, os.X_OK)):
-            self.log.error(f"ufw-docker not found at {UFW_DOCKER_BIN}. Run configure/base first.")
-            raise SystemExit(1)
-
-        if config_dir:
-            host_rules = os.path.join(config_dir, "host.rules")
-            if os.path.isfile(host_rules):
-                for rule in clean_rules(open(host_rules).read()):
-                    self.log.info(f"Applying host ufw rule: {rule}")
-                    if dry_run:
-                        self.log.info(f"[DRY-RUN] sudo ufw {rule}")
-                    else:
-                        self.sys.run(["bash", "-c", f"sudo ufw {rule}"], check=True)
-            self._inject_file("/etc/ufw/before.rules", os.path.join(config_dir, "before.rules"), dry_run)
-            self._inject_file("/etc/ufw/after.rules", os.path.join(config_dir, "after.rules"), dry_run)
-
-        rule_files = [fn for fn in sorted(os.listdir(rules_dir))
-                      if fn.endswith(".rules") and not fn.startswith(".")]
-        if not rule_files:
-            self.log.warn(f"No .rules files found in {rules_dir}, nothing to apply.")
-        for fn in rule_files:
-            for rule in clean_rules(open(os.path.join(rules_dir, fn)).read()):
-                self.log.info(f"Applying ufw-docker rule ({fn}): {rule}")
-                if dry_run:
-                    self.log.info(f"[DRY-RUN] sudo {UFW_DOCKER_BIN} {rule}")
-                else:
-                    self.sys.run(["sudo", UFW_DOCKER_BIN, *rule.split()], check=True)
-
-        self.log.success("All ufw rules applied.")
-        self.log.warn("Run `sudo systemctl restart ufw` to apply. Ensure ssh is allowed via plain ufw!")

etc/alloy/config.alloy

@@ -1,14 +1,6 @@
-// ============================================================================
+// Optional shared Grafana Alloy configuration.
-// APEX fleet observability agent — per-node Grafana Alloy config (shared).
+// Metrics and logs use the endpoints and credentials supplied through
-// Committed, env-parameterized: every node runs this SAME file; per-node values
+// APEX_PROM_PUSH_URL, APEX_LOKI_PUSH_URL, and APEX_OBS_BASIC_AUTH_USER/PASS.
-// come from sys.env(...) (APEX_NODE, push URLs, obs basic-auth creds).
-// Verified against grafana.com/docs/alloy/latest (Alloy v1.17.1).
-// ============================================================================
-// Model: PUSH. Metrics -> remote_write to the central Prometheus (on icarus);
-// logs -> loki.write to the central Loki (on icarus). daedalus/morpheus reach
-// them through the public gfn.ermnvldmr.com front (authelia-bypassed + basicauth
-// on /loki and /prometheus/api/v1/write); icarus resolves gfn locally.
-// ============================================================================
 
 // ----------------------------------------------------------------------------
 // METRICS
@@ -24,8 +16,21 @@ prometheus.exporter.unix "node" {
 
 // Container metrics (cadvisor-equivalent) via the docker socket.
 prometheus.exporter.cadvisor "containers" {
-  docker_host            = "unix:///var/run/docker.sock"
+  docker_host                  = "unix:///var/run/docker.sock"
-  store_container_labels = true
+  store_container_labels       = false
+  allowlisted_container_labels = [
+    "com.docker.compose.project",
+    "com.docker.compose.service",
+  ]
+  docker_only                  = true
+  enabled_metrics              = [
+    "cpu",
+    "cpuLoad",
+    "diskIO",
+    "memory",
+    "network",
+    "oom_event",
+  ]
 }
 
 // Stamp a static `node` label onto each exporter's targets.
@@ -34,7 +39,14 @@ discovery.relabel "node_metrics" {
   rule {
     action       = "replace"
     target_label = "node"
-    replacement  = sys.env("APEX_NODE")
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+  // Stamp a stable `instance` (the node's public host) instead of the default,
+  // which is the Alloy container id/address.
+  rule {
+    action       = "replace"
+    target_label = "instance"
+    replacement  = sys.env("APEX_NODE_HOST")
   }
 }
 discovery.relabel "container_metrics" {
@@ -42,22 +54,40 @@ discovery.relabel "container_metrics" {
   rule {
     action       = "replace"
     target_label = "node"
-    replacement  = sys.env("APEX_NODE")
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+  rule {
+    action       = "replace"
+    target_label = "instance"
+    replacement  = sys.env("APEX_NODE_HOST")
   }
 }
 
 prometheus.scrape "node" {
   targets         = discovery.relabel.node_metrics.output
   forward_to      = [prometheus.remote_write.default.receiver]
-  scrape_interval = "15s"
+  scrape_interval = "30s"
 }
 prometheus.scrape "containers" {
   targets         = discovery.relabel.container_metrics.output
   forward_to      = [prometheus.remote_write.default.receiver]
-  scrape_interval = "15s"
+  scrape_interval = "30s"
 }
 
-// Push metrics to the central Prometheus (needs --web.enable-remote-write-receiver).
+// The CrowdSec scrape requires a reachable host metrics endpoint on port 6060.
+// If no endpoint is configured, this target reports down.
+prometheus.scrape "crowdsec" {
+  targets = [{
+    __address__ = "host.docker.internal:6060",
+    instance    = sys.env("APEX_NODE_HOST"),
+    node        = sys.env("APEX_NODE_HOST"),
+    job         = "crowdsec",
+  }]
+  forward_to      = [prometheus.remote_write.default.receiver]
+  scrape_interval = "30s"
+}
+
+// The configured Prometheus-compatible endpoint must accept remote writes.
 prometheus.remote_write "default" {
   endpoint {
     url = sys.env("APEX_PROM_PUSH_URL")
@@ -65,6 +95,20 @@ prometheus.remote_write "default" {
       username = sys.env("APEX_OBS_BASIC_AUTH_USER")
       password = sys.env("APEX_OBS_BASIC_AUTH_PASS")
     }
+    queue_config {
+      capacity             = 5000
+      max_shards           = 10
+      min_shards           = 1
+      max_samples_per_send = 1000
+      batch_send_deadline  = "5s"
+      min_backoff          = "100ms"
+      max_backoff          = "10s"
+    }
+  }
+  wal {
+    truncate_frequency = "30m"
+    min_keepalive_time = "5m"
+    max_keepalive_time = "30m"
   }
 }
 
@@ -77,7 +121,7 @@ loki.source.journal "systemd" {
   path    = "/var/log/journal"
   max_age = "12h"
   labels  = {
-    node = sys.env("APEX_NODE"),
+    node = sys.env("APEX_NODE_HOST"),
     job  = "systemd-journal",
   }
   forward_to = [loki.write.default.receiver]
@@ -98,7 +142,7 @@ discovery.relabel "container_logs" {
   rule {
     action       = "replace"
     target_label = "node"
-    replacement  = sys.env("APEX_NODE")
+    replacement  = sys.env("APEX_NODE_HOST")
   }
 }
 loki.source.docker "containers" {

etc/alloy/native-host.alloy

@@ -0,0 +1,102 @@
+// Shared native Grafana Alloy configuration for the physical host.
+// Metrics and logs use the endpoints and credentials supplied through
+// APEX_PROM_PUSH_URL, APEX_LOKI_PUSH_URL, and APEX_OBS_BASIC_AUTH_USER/PASS.
+
+// ----------------------------------------------------------------------------
+// METRICS
+// ----------------------------------------------------------------------------
+
+// Native filesystem views; physical capacity is collected on the Proxmox host.
+prometheus.exporter.unix "node" {
+  enable_collectors = ["textfile"]
+  textfile {
+    directory = "/var/lib/alloy/textfile"
+  }
+  procfs_path = "/proc"
+  sysfs_path  = "/sys"
+  rootfs_path = "/"
+}
+
+// Container metrics (cadvisor-equivalent) via the docker socket.
+
+
+// Stamp a static `node` label onto each exporter's targets.
+discovery.relabel "node_metrics" {
+  targets = prometheus.exporter.unix.node.targets
+  rule {
+    action       = "replace"
+    target_label = "node"
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+  // Stamp a stable `instance` (the node's public host) instead of the default,
+  // which is the Alloy container id/address.
+  rule {
+    action       = "replace"
+    target_label = "instance"
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+}
+
+
+prometheus.scrape "node" {
+  targets         = discovery.relabel.node_metrics.output
+  forward_to      = [prometheus.remote_write.default.receiver]
+  scrape_interval = "30s"
+}
+
+
+// The configured Prometheus-compatible endpoint must accept remote writes.
+prometheus.remote_write "default" {
+  endpoint {
+    url = sys.env("APEX_PROM_PUSH_URL")
+    basic_auth {
+      username = sys.env("APEX_OBS_BASIC_AUTH_USER")
+      password = sys.env("APEX_OBS_BASIC_AUTH_PASS")
+    }
+    queue_config {
+      capacity             = 5000
+      max_shards           = 10
+      min_shards           = 1
+      max_samples_per_send = 1000
+      batch_send_deadline  = "5s"
+      min_backoff          = "100ms"
+      max_backoff          = "10s"
+    }
+  }
+  wal {
+    truncate_frequency = "30m"
+    min_keepalive_time = "5m"
+    max_keepalive_time = "30m"
+  }
+}
+
+// ----------------------------------------------------------------------------
+// LOGS
+// ----------------------------------------------------------------------------
+
+// systemd journal (host services, ufw, sshd, crowdsec, etc.).
+loki.source.journal "systemd" {
+  path    = "/var/log/journal"
+  max_age = "12h"
+  labels  = {
+    node = sys.env("APEX_NODE_HOST"),
+    job  = "systemd-journal",
+  }
+  forward_to = [loki.write.default.receiver]
+}
+
+// All docker containers' stdout/stderr.
+
+
+
+
+// Push logs to the central Loki.
+loki.write "default" {
+  endpoint {
+    url = sys.env("APEX_LOKI_PUSH_URL")
+    basic_auth {
+      username = sys.env("APEX_OBS_BASIC_AUTH_USER")
+      password = sys.env("APEX_OBS_BASIC_AUTH_PASS")
+    }
+  }
+}

etc/alloy/native.alloy

@@ -0,0 +1,128 @@
+// Shared native Grafana Alloy configuration for Docker guests.
+// Metrics and logs use the endpoints and credentials supplied through
+// APEX_PROM_PUSH_URL, APEX_LOKI_PUSH_URL, and APEX_OBS_BASIC_AUTH_USER/PASS.
+
+// ----------------------------------------------------------------------------
+// METRICS
+// ----------------------------------------------------------------------------
+
+// Native filesystem views; physical capacity is collected on the Proxmox host.
+
+
+// Container metrics (cadvisor-equivalent) via the docker socket.
+prometheus.exporter.cadvisor "containers" {
+  docker_host                  = "unix:///var/run/docker.sock"
+  store_container_labels       = false
+  allowlisted_container_labels = [
+    "com.docker.compose.project",
+    "com.docker.compose.service",
+  ]
+  docker_only                  = true
+  enabled_metrics              = [
+    "cpu",
+    "cpuLoad",
+    "diskIO",
+    "memory",
+    "network",
+    "oom_event",
+  ]
+}
+
+// Stamp a static `node` label onto each exporter's targets.
+
+discovery.relabel "container_metrics" {
+  targets = prometheus.exporter.cadvisor.containers.targets
+  rule {
+    action       = "replace"
+    target_label = "node"
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+  rule {
+    action       = "replace"
+    target_label = "instance"
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+}
+
+
+prometheus.scrape "containers" {
+  targets         = discovery.relabel.container_metrics.output
+  forward_to      = [prometheus.remote_write.default.receiver]
+  scrape_interval = "30s"
+}
+
+// The configured Prometheus-compatible endpoint must accept remote writes.
+prometheus.remote_write "default" {
+  endpoint {
+    url = sys.env("APEX_PROM_PUSH_URL")
+    basic_auth {
+      username = sys.env("APEX_OBS_BASIC_AUTH_USER")
+      password = sys.env("APEX_OBS_BASIC_AUTH_PASS")
+    }
+    queue_config {
+      capacity             = 5000
+      max_shards           = 10
+      min_shards           = 1
+      max_samples_per_send = 1000
+      batch_send_deadline  = "5s"
+      min_backoff          = "100ms"
+      max_backoff          = "10s"
+    }
+  }
+  wal {
+    truncate_frequency = "30m"
+    min_keepalive_time = "5m"
+    max_keepalive_time = "30m"
+  }
+}
+
+// ----------------------------------------------------------------------------
+// LOGS
+// ----------------------------------------------------------------------------
+
+// systemd journal (host services, ufw, sshd, crowdsec, etc.).
+loki.source.journal "systemd" {
+  path    = "/var/log/journal"
+  max_age = "12h"
+  labels  = {
+    node = sys.env("APEX_NODE_HOST"),
+    job  = "systemd-journal",
+  }
+  forward_to = [loki.write.default.receiver]
+}
+
+// All docker containers' stdout/stderr.
+discovery.docker "containers" {
+  host = "unix:///var/run/docker.sock"
+}
+discovery.relabel "container_logs" {
+  targets = discovery.docker.containers.targets
+  rule {
+    action        = "replace"
+    source_labels = ["__meta_docker_container_name"]
+    regex         = "/?(.*)"
+    target_label  = "container"
+  }
+  rule {
+    action       = "replace"
+    target_label = "node"
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+}
+loki.source.docker "containers" {
+  host          = "unix:///var/run/docker.sock"
+  targets       = discovery.relabel.container_logs.output
+  relabel_rules = discovery.relabel.container_logs.rules
+  forward_to    = [loki.write.default.receiver]
+}
+
+// Push logs to the central Loki.
+loki.write "default" {
+  endpoint {
+    url = sys.env("APEX_LOKI_PUSH_URL")
+    basic_auth {
+      username = sys.env("APEX_OBS_BASIC_AUTH_USER")
+      password = sys.env("APEX_OBS_BASIC_AUTH_PASS")
+    }
+  }
+}

pyproject.toml

@@ -0,0 +1,26 @@
+[project]
+name = "apex-commons"
+version = "2026.10.0"
+description = "Core runtime and orchestration framework for apex.ermnvldmr.com node fleet"
+readme = "README.md"
+license = { text = "BSD-2-Clause" }
+requires-python = ">=3.10"
+
+[project.optional-dependencies]
+dev = ["ruff==0.17.2"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "B", "UP", "S"]
+# Existing formatting and typing modernization are outside correctness checks.
+# Subprocess arguments are constructed explicitly; no implicit shell execution.
+ignore = ["E401", "E501", "E702", "E741", "I001", "UP006", "UP035", "UP045", "S603", "S607"]
+
+[tool.mypy]
+python_version = "3.10"
+warn_return_any = true
+warn_unused_configs = true
+disallow_untyped_defs = false

scripts/bootstrap-node

@@ -0,0 +1,95 @@
+#!/usr/bin/env python3
+"""Bootstrap a fleet node through a trusted OpenSSH connection."""
+
+import argparse
+import json
+import re
+import shlex
+import subprocess
+import sys
+from pathlib import Path
+from urllib.parse import urlsplit
+
+
+def repository(url):
+    if any(ch.isspace() or ord(ch) < 32 for ch in url):
+        raise ValueError("invalid repository URL")
+    if url.startswith("ssh://"):
+        parts = urlsplit(url)
+        if parts.password is not None or parts.query or parts.fragment:
+            raise ValueError("expected an SSH repository URL without password, query or fragment")
+        host, port, path = parts.hostname, parts.port, parts.path
+    else:
+        match = re.fullmatch(r"(?:[\w.-]+@)?(\[[\da-fA-F:]+\]|[\w.-]+):(.+)", url)
+        if not match:
+            raise ValueError("expected git@host:path or ssh://user@host[:port]/path")
+        host, path = match.groups()
+        host, port = host.strip("[]"), None
+    if not host or not re.fullmatch(r"[a-zA-Z0-9:][a-zA-Z0-9.:-]*", host) or not path:
+        raise ValueError("invalid Git host or repository path")
+    return f"[{host}]:{port}" if port not in (None, 22) else host, Path(path).name.removesuffix(".git")
+
+
+def keygen(args, data=None):
+    return subprocess.run(["ssh-keygen", *args], input=data or b"", capture_output=True)
+
+
+def bootstrap(args):
+    endpoint, basename = repository(args.repo)
+    name = args.node_name if args.node_name is not None else basename
+    if not re.fullmatch(r"[a-zA-Z0-9][a-zA-Z0-9._-]*", name):
+        raise ValueError("invalid node name")
+    operator = args.operator_key.read_text().strip()
+    fields = operator.split()
+    if (len(operator.splitlines()) != 1 or len(fields) < 2
+            or keygen(["-lf", "-"], (" ".join(fields[:2]) + "\n").encode()).returncode):
+        raise ValueError("operator key must contain one plain SSH public key")
+    derived = keygen(["-y", "-P", "", "-f", str(args.deploy_key.resolve())])
+    if derived.returncode or derived.stdout.split()[0].startswith(b"sk-"):
+        raise ValueError("deploy key must be a standalone private key with no passphrase and owner-only permissions")
+    if derived.stdout.split()[:2] == [v.encode() for v in fields[:2]]:
+        raise ValueError("operator and deploy keys must be different")
+    matches = keygen(["-F", endpoint, "-f", str(args.known_hosts.resolve())])
+    pins = [line for line in matches.stdout.decode().splitlines() if line and not line.startswith("#")]
+    if matches.returncode or not pins or not any(
+        not pin.startswith("@revoked ") and keygen(["-lf", "-"], (pin + "\n").encode()).returncode == 0
+        for pin in pins
+    ):
+        raise ValueError(f"no usable known_hosts pin for {endpoint}")
+    payload = json.dumps(dict(operator_key=operator, deploy_key=args.deploy_key.read_text(),
+                              known_hosts="\n".join(pins) + "\n", repo_url=args.repo, node_name=name))
+    helper = Path(__file__).resolve().with_name("bootstrap-node-remote.py").read_text()
+    # Package management must never consume the credential payload on standard input.
+    remote = """set -e
+if [ "$(id -u)" = 0 ]; then set --; else set -- sudo -n --; fi
+if ! command -v python3 >/dev/null; then
+    test -f /etc/debian_version
+    "$@" apt-get -o DPkg::Lock::Timeout=60 update </dev/null
+    "$@" env DEBIAN_FRONTEND=noninteractive apt-get -o DPkg::Lock::Timeout=60 install -y python3 </dev/null
+fi
+exec "$@" python3 -I -c """ + shlex.quote(helper)
+    result = subprocess.run(["ssh", "-T", "-a", "-S", "none", "-o", "StrictHostKeyChecking=yes",
+                           "-o", "ClearAllForwardings=yes", "--", args.target, remote],
+                          input=payload.encode())
+    if result.returncode == 0:
+        print(f"Checkout ready for adam at /srv/{name}.")
+    return result.returncode
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--target", required=True, help="trusted SSH target with root or noninteractive sudo")
+    parser.add_argument("--repo", required=True, help="SSH URL of the node repository")
+    parser.add_argument("--operator-key", required=True, type=Path, help="operator public key")
+    parser.add_argument("--deploy-key", required=True, type=Path, help="repository deploy private key")
+    parser.add_argument("--known-hosts", required=True, type=Path, help="trusted Git server host keys")
+    parser.add_argument("--node-name", help="defaults to repository basename without .git")
+    args = parser.parse_args()
+    try:
+        return bootstrap(args)
+    except (OSError, ValueError) as exc:
+        parser.exit(1, f"bootstrap-node: {exc}\n")
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/bootstrap-node-remote.py

@@ -0,0 +1,275 @@
+"""Root-side Debian/PVE bootstrap, staged and invoked by bootstrap-node."""
+import fcntl
+import json
+import os
+from pathlib import Path
+import pwd
+import re
+import stat
+import subprocess
+import sys
+import tempfile
+from urllib.parse import urlsplit
+
+
+HOME = Path("/home/adam")
+
+
+def checked(command, *, data=None, capture=False, **kwargs):
+    # A Git hook or remote can emit credentials. Report the operation, never its output.
+    result = subprocess.run(command, input=data, text=True, stderr=subprocess.DEVNULL,
+                            stdout=subprocess.PIPE if capture else subprocess.DEVNULL, **kwargs)
+    if result.returncode:
+        raise RuntimeError(f"{Path(command[0]).name} failed (exit {result.returncode})")
+    return result.stdout
+
+
+def repository_endpoint(url):
+    if not isinstance(url, str) or any(c.isspace() or ord(c) < 32 for c in url):
+        raise ValueError("Invalid SSH repository URL")
+    if url.startswith("ssh://"):
+        parsed = urlsplit(url)
+        if (parsed.password is not None or not parsed.path.strip("/") or parsed.fragment
+                or parsed.query or parsed.port == 0):
+            raise ValueError("Invalid SSH repository URL")
+        host, port = parsed.hostname, parsed.port
+    else:
+        match = re.fullmatch(r"(?:[\w.-]+@)?(\[[a-fA-F0-9:]+\]|[\w.-]+):(.+)", url, re.ASCII)
+        host, port = (match[1].strip("[]"), None) if match else (None, None)
+    if not host or not re.fullmatch(r"[a-zA-Z0-9:][a-zA-Z0-9.:-]*", host):
+        raise ValueError("An SSH repository host is required")
+    return host, f"[{host}]:{port}" if port not in (None, 22) else host
+
+
+def existing(path):
+    try:
+        info = path.lstat()
+    except FileNotFoundError:
+        return b""
+    if not stat.S_ISREG(info.st_mode) or info.st_nlink != 1 or info.st_uid != os.geteuid():
+        raise ValueError(f"Unsafe file: {path.name}")
+    return path.read_bytes()
+
+
+def install(path, content, mode):
+    existing(path)
+    fd, tmp = tempfile.mkstemp(dir=path.parent)
+    try:
+        with os.fdopen(fd, "wb") as stream:
+            stream.write(content)
+            os.fchmod(stream.fileno(), mode)
+        os.replace(tmp, path)
+    finally:
+        Path(tmp).unlink(missing_ok=True)
+
+
+def validate_credentials(spec, endpoint):
+    key = spec["operator_key"].strip()
+    if len(key.splitlines()) != 1 or not key.startswith(("ssh-", "ecdsa-", "sk-")):
+        raise ValueError("Expected one operator public key")
+    checked(["ssh-keygen", "-lf", "-"], data=key + "\n")
+    with tempfile.TemporaryDirectory() as temp:
+        private = Path(temp) / "deploy"
+        install(private, (spec["deploy_key"].rstrip() + "\n").encode(), 0o600)
+        public = checked(["ssh-keygen", "-y", "-P", "", "-f", str(private)], capture=True)
+        if public.startswith("sk-") or public.split()[:2] == key.split()[:2]:
+            raise ValueError("Use a separate unattended deploy key")
+        pins = Path(temp) / "known_hosts"
+        pins.write_text(spec["known_hosts"])
+        matches = checked(["ssh-keygen", "-F", endpoint, "-f", str(pins)], capture=True)
+        trusted = [line for line in matches.splitlines() if not line.startswith(("#", "@revoked "))]
+        if not trusted:
+            raise ValueError("No trusted host key for the repository")
+        for pin in trusted:
+            checked(["ssh-keygen", "-lf", "-"], data=pin + "\n")
+
+
+def configure_ssh(home, spec, host):
+    directory = home / ".ssh"
+    directory.mkdir(mode=0o700, exist_ok=True)
+    info = directory.lstat()
+    if not stat.S_ISDIR(info.st_mode) or info.st_uid != os.geteuid():
+        raise ValueError("Unsafe SSH directory")
+    directory.chmod(0o700)
+    key = spec["operator_key"].strip()
+    authorized = directory / "authorized_keys"
+    content = existing(authorized)
+    identity = key.split()[:2]
+    present = False
+    for line in content.decode(errors="replace").splitlines():
+        fields = line.split()
+        if fields[:2] == identity:
+            present = True
+        elif fields and not line.lstrip().startswith("#") and identity[1] in fields:
+            raise ValueError("Existing operator key has SSH authorization restrictions; inspect it")
+    if not present:
+        content = content + (b"\n" if content and not content.endswith(b"\n") else b"")
+        content += key.encode() + b"\n"
+    # Git also evaluates includeIf for the separate gitdirs of recursive submodules.
+    config = home / ".gitconfig"
+    old = existing(config)
+    gitdir = f"/srv/{spec['node_name']}/.git"
+    prefix = directory / ("apex-" + spec["node_name"])
+    include = (f'[includeIf "gitdir:{gitdir}"]\n\tpath = {prefix}.gitconf\n'
+               f'[includeIf "gitdir:{gitdir}/"]\n\tpath = {prefix}.gitconf\n').encode()
+    for suffix in ("key", "hosts", "sshconf", "gitconf"):
+        existing(Path(f"{prefix}.{suffix}"))
+    write_repository_credentials(prefix, spec, host)
+    install(Path(f"{prefix}.gitconf"),
+            f"[core]\n\tsshCommand = ssh -F {prefix}.sshconf\n".encode(), 0o600)
+    if include not in old:
+        install(config, old + b"\n" + include, 0o600)
+    install(authorized, content, 0o600)
+
+
+def write_repository_credentials(prefix, spec, host):
+    policy = (f"Host {host}\n    IdentitiesOnly yes\n"
+              f"    IdentityFile {prefix}.key\n    UserKnownHostsFile {prefix}.hosts\n"
+              "    GlobalKnownHostsFile /dev/null\nHost *\n    StrictHostKeyChecking yes\n"
+              "    BatchMode yes\n")
+    for suffix, content in (("key", spec["deploy_key"]), ("hosts", spec["known_hosts"]),
+                            ("sshconf", policy)):
+        install(Path(f"{prefix}.{suffix}"), (content.rstrip() + "\n").encode(), 0o600)
+
+
+def verify_and_clone(spec, host, account):
+    srv = Path("/srv")
+    if not srv.exists():
+        srv.mkdir()
+        srv.chmod(0o755)
+    if srv.resolve() != srv or srv.stat().st_uid != 0 or srv.stat().st_mode & 0o022:
+        raise RuntimeError("/srv must be a root-controlled directory")
+    with tempfile.TemporaryDirectory(prefix=".apex-credentials-", dir=srv) as temp:
+        stage = Path(temp)
+        prefix = stage / "git"
+        write_repository_credentials(prefix, spec, host)
+        for path in stage.iterdir():
+            os.chown(path, account.pw_uid, account.pw_gid)
+        stage.chmod(0o711)
+
+        def git(*args):
+            return checked(["runuser", "-u", "adam", "--", "env", "GIT_TERMINAL_PROMPT=0",
+                            f"GIT_SSH_COMMAND=ssh -F {prefix}.sshconf", "git", *args],
+                           capture=True, cwd="/")
+
+        try:
+            git("ls-remote", "--exit-code", "--", spec["repo_url"], "HEAD")
+        except RuntimeError:
+            raise RuntimeError("Git access check failed; check the repository URL, deploy-key "
+                               "grants and pinned host key") from None
+        checkout(srv, spec["node_name"], spec["repo_url"], account, git)
+
+
+def prepare_packages():
+    if not Path("/etc/debian_version").is_file():
+        raise RuntimeError("Debian or Proxmox VE is required")
+    missing = []
+    for package in ("git", "sudo", "openssh-client"):
+        result = subprocess.run(["dpkg-query", "-W", "-f=${Status}", package], text=True,
+                                capture_output=True)
+        if result.returncode or result.stdout.split()[1:] != ["ok", "installed"]:
+            missing.append(package)
+    if missing:
+        checked(["apt-get", "update"])
+        checked(["apt-get", "install", "-y", "--no-remove", "--no-install-recommends", *missing])
+
+
+def prepare_account():
+    try:
+        account = pwd.getpwnam("adam")
+    except KeyError:
+        if os.path.lexists(HOME):
+            raise RuntimeError("Existing /home/adam requires inspection") from None
+        checked(["useradd", "--create-home", "--user-group", "--shell", "/bin/bash", "adam"])
+        account = pwd.getpwnam("adam")
+    if (not 1000 <= account.pw_uid < 65534 or account.pw_gid == 0
+            or account.pw_dir != str(HOME) or HOME.resolve() != HOME
+            or HOME.stat().st_uid != account.pw_uid or HOME.stat().st_mode & 0o022
+            or not HOME.is_dir()):
+        raise RuntimeError("Existing adam account or home requires inspection")
+    groups = "sudo"
+    if not Path("/usr/bin/pveversion").exists():
+        checked(["groupadd", "--system", "--force", "docker"])
+        groups += ",docker"
+    checked(["usermod", "--shell", "/bin/bash", "--append", "--groups", groups, "adam"])
+    return account
+
+
+def checkout(srv, node, url, account, git):
+    destination = srv / node
+    if os.path.lexists(destination):
+        info = destination.lstat()
+        if not stat.S_ISDIR(info.st_mode) or info.st_uid != account.pw_uid:
+            raise RuntimeError("Existing checkout path requires inspection")
+
+        def query(*args):
+            return git("-C", str(destination), *args).strip()
+
+        if query("rev-parse", "--show-toplevel") != str(destination) or query(
+                "config", "--get", "remote.origin.url") != url:
+            raise RuntimeError("Existing checkout does not match the requested repository")
+        for line in query("submodule", "status", "--recursive").splitlines():
+            if line.startswith(("-", "+", "U")):
+                raise RuntimeError(f"Existing submodule does not match its recorded commit: {line}")
+        return
+    with tempfile.TemporaryDirectory(prefix=".apex-bootstrap-", dir=srv) as staging:
+        stage = Path(staging)
+        stage.chmod(0o711)
+        clone = stage / "checkout"
+        clone.mkdir()
+        os.chown(clone, account.pw_uid, account.pw_gid)
+        try:
+            git("clone", "--recurse-submodules", "--", url, str(clone))
+        except RuntimeError:
+            raise RuntimeError("Recursive clone failed; check access to pinned submodules "
+                               "and available disk space") from None
+        clone.rename(destination)
+
+
+def bootstrap(spec):
+    node = spec["node_name"]
+    if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._-]{0,252}", node):
+        raise ValueError("Invalid node checkout name")
+    host, endpoint = repository_endpoint(spec["repo_url"])
+    if not all(isinstance(spec[k], str) and spec[k].strip()
+               for k in ("known_hosts", "deploy_key", "operator_key")):
+        raise ValueError("Missing SSH credentials")
+    if os.geteuid() != 0:
+        raise RuntimeError("Root login is required")
+    os.umask(0o077)
+    os.environ.clear()
+    os.environ.update(PATH="/usr/sbin:/usr/bin:/sbin:/bin", LC_ALL="C",
+                      DEBIAN_FRONTEND="noninteractive", HOME="/root")
+    # Serialize account, credential and checkout changes from competing bootstraps.
+    with open("/run/apex-bootstrap.lock", "a") as lock:
+        fcntl.flock(lock, fcntl.LOCK_EX)
+        prepare_packages()
+        validate_credentials(spec, endpoint)
+        account = prepare_account()
+        verify_and_clone(spec, host, account)
+        child = os.fork()
+        if child == 0:
+            try:
+                os.initgroups("adam", account.pw_gid)
+                os.setgid(account.pw_gid)
+                os.setuid(account.pw_uid)
+                os.chdir(HOME)
+                configure_ssh(HOME, spec, host)
+            except Exception as error:
+                print(f"SSH setup failed: {error}", file=sys.stderr, flush=True)
+                os._exit(1)
+            os._exit(0)
+        if os.waitpid(child, 0)[1]:
+            raise RuntimeError("Unable to install SSH credentials")
+        policy = "adam ALL=(ALL:ALL) NOPASSWD: ALL\n"
+        checked(["visudo", "-cf", "-"], data=policy)
+        install(Path("/etc/sudoers.d/99-apex-adam"), policy.encode(), 0o440)
+        checked(["visudo", "-c"])
+        checked(["runuser", "-u", "adam", "--", "sudo", "-n", "true"])
+
+
+if __name__ == "__main__":
+    try:
+        bootstrap(json.load(sys.stdin))
+    except (ValueError, KeyError, RuntimeError, OSError) as error:
+        sys.exit(f"Bootstrap failed: {error}")

scripts/engine/cli.py

@@ -1,9 +1,10 @@
-# engine/cli.py
+# scripts/engine/cli.py
 """apex CLI: list / dispatch / help. Acts on the local checkout (no <node> arg)."""
 from __future__ import annotations
 import os
 import socket
 import sys
+from pathlib import Path
 
 from engine import overlay
 from engine.identity import resolve
@@ -12,21 +13,23 @@ from engine.descriptor import build_parser, Meta
 
 
 def _resolve_layout():
-    commons_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
+    commons_dir = str(Path(__file__).resolve().parents[2])
     parent = os.path.dirname(commons_dir)
-    is_node = (os.path.basename(commons_dir) == "commons"
+    cwd = os.getcwd()
-               and os.path.isfile(os.path.join(parent, "node.env")))
+
-    if is_node:
+    if os.path.isfile(os.path.join(parent, "node.env")):
         repo_root = parent
         prop_actions = os.path.join(repo_root, "proprietaries", "actions")
-    else:
+        return commons_dir, repo_root, prop_actions, True
-        repo_root = commons_dir          # standalone commons repo is its own root
+    if os.path.isfile(os.path.join(cwd, "node.env")):
-        prop_actions = None
+        repo_root = cwd
-    return commons_dir, repo_root, prop_actions, is_node
+        prop_actions = os.path.join(repo_root, "proprietaries", "actions")
+        return commons_dir, repo_root, prop_actions, True
+    return commons_dir, commons_dir, None, False
 
 
 def _print_list(commons_actions, prop_actions):
-    print("Usage: apex <group>/<action> [args...]\n\nAvailable actions:")
+    print("Usage: apex <entity>/<action> [args...]\n\nAvailable actions:")
     tag = {"commons": "", "local": "  (local)", "override": "  (local override)"}
     for i in overlay.discover(commons_actions, prop_actions):
         mark = f"  [DISABLED: {i.disabled}]" if i.disabled else ""
@@ -42,6 +45,8 @@ def main(argv) -> int:
         return 0
 
     name, rest = argv[0], argv[1:]
+    if name == "compose":
+        name = "composition/run"
     r = overlay.resolve(name, commons_actions, prop_actions)
     if r is None:
         sys.stderr.write(f"[global] [core] [ERROR] Unknown action: {name}\n")

scripts/engine/context.py

@@ -1,5 +1,5 @@
-# engine/context.py
+# scripts/engine/context.py
-"""Context (ctx) — the single seam between an action and the world (SP2 §2.1)."""
+"""Context and filesystem paths available to actions."""
 from __future__ import annotations
 import os
 
@@ -81,7 +81,7 @@ class Context:
         self.ufw = Ufw(self.log, self.sys, self.host)
         self.crowdsec = Crowdsec(self.log, self.sys, self.tpl, self.host)
         self.systemd = Systemd(self.log, self.sys, self.tpl, self.host)
-        self.notify = Notify(self.log, name)
+        self.notify = Notify(self.log, node.host if node else "global")
         self._shadowed = shadowed
         self._commons = None
 
@@ -98,10 +98,9 @@ class Context:
         v["APEX_REPO_ROOT"] = self.paths.repo_root
         if self.node:
             v.update({
-                "APEX_NODE": self.node.name,
+                "APEX_NODE_HOST": self.node.host,
-                "APEX_CLUSTER": self.node.cluster,
-                "APEX_SUBNET": self.node.subnet,
                 "APEX_NODE_FQDN": self.node.fqdn,
+                "APEX_SUBNET": self.node.subnet,
                 "APEX_TIER1": self.paths.tier1, "APEX_TIER2": self.paths.tier2,
                 "APEX_TIER3": self.paths.tier3,
                 "APEX_TIER1_SHARED": self.paths.stier1,

scripts/engine/descriptor.py

@@ -1,4 +1,4 @@
-# engine/descriptor.py
+# scripts/engine/descriptor.py
 """Declarative action descriptors — successor to the bash action_metadata() heredoc.
 
 Meta drives BOTH the help text and argparse parsing. Exit code 64 (EX_USAGE) on a

scripts/engine/identity.py

@@ -0,0 +1,64 @@
+# scripts/engine/identity.py
+"""Node identity — fully explicit, from node.env. The framework assumes no domain
+and no clustering: a node declares its real FQDN and its public host name, and the
+short `name` is just the first label. The OS hostname is only a fallback for the FQDN.
+
+node.env keys:
+  APEX_NODE_FQDN  real FQDN (traefik Host() routing).            default: OS hostname
+  APEX_NODE_HOST  public host name (backup --host, obs/notify    default: APEX_NODE_FQDN
+                  `instance`); often the FQDN minus a cluster
+                  segment, but that's the node's choice.
+  APEX_SUBNET     the node's /24 for the `direct` network.       required
+"""
+from __future__ import annotations
+
+import os
+import sys
+from dataclasses import dataclass
+from typing import Dict, List, Tuple
+
+
+@dataclass
+class Node:
+    name: str    # short label (first segment of the host), e.g. "node1" — logs, branch, messages
+    subnet: str  # the node's direct-network /24
+    fqdn: str    # real FQDN — traefik Host() routing
+    host: str    # public host name — backup --host, obs `instance`, notify instance
+
+
+def read_env(path: str) -> Dict[str, str]:
+    """Parse a KEY=value file (shell-sourceable). Ignores comments/blank lines."""
+    out: Dict[str, str] = {}
+    if not os.path.isfile(path):
+        return out
+    with open(path) as f:
+        for line in f:
+            line = line.strip()
+            if not line or line.startswith("#") or "=" not in line:
+                continue
+            key, _, val = line.partition("=")
+            out[key.strip()] = val.strip().strip('"').strip("'")
+    return out
+
+
+def resolve(hostname: str, repo_root: str) -> Tuple[Node, List[str]]:
+    """Return (Node, warnings). Exits 66 (EX_NOINPUT) if APEX_SUBNET is missing."""
+    warns: List[str] = []
+    env = read_env(os.path.join(repo_root, "node.env"))
+
+    fqdn = env.get("APEX_NODE_FQDN", "") or (hostname or "").strip()
+    if not fqdn:
+        sys.stderr.write("identity: APEX_NODE_FQDN missing from node.env and no hostname\n")
+        raise SystemExit(66)
+    if not env.get("APEX_NODE_FQDN"):
+        warns.append(f"APEX_NODE_FQDN not set in node.env; using hostname {fqdn!r}.")
+
+    host = env.get("APEX_NODE_HOST", "") or fqdn      # node's public name; defaults to the FQDN
+    name = host.split(".")[0]                          # short label for logs/branch/messages
+
+    subnet = env.get("APEX_SUBNET", "")
+    if not subnet:
+        sys.stderr.write("identity: APEX_SUBNET missing from node.env\n")
+        raise SystemExit(66)
+
+    return Node(name=name, subnet=subnet, fqdn=fqdn, host=host), warns

scripts/engine/lib/crowdsec.py

@@ -1,5 +1,5 @@
-# engine/lib/crowdsec.py
+# scripts/engine/lib/crowdsec.py
-"""ctx.crowdsec — install + config deploy + collections (port of configure/crowdsec)."""
+"""ctx.crowdsec — install + config deploy + collections (port of config/crowdsec)."""
 from __future__ import annotations
 import os
 import shutil
@@ -13,7 +13,7 @@ class Crowdsec:
         if shutil.which("cscli"):
             return
         self.log.info("Installing CrowdSec via official script...")
-        self.sys.run(["bash", "-c", "set -o pipefail; curl -s https://install.crowdsec.net | sudo sh"],
+        self.sys.run(["bash", "-c", "set -o pipefail; curl -sSfL https://install.crowdsec.net | sudo sh"],
                      check=True)
         self.sys.sudo(["apt-get", "install", "-y", "crowdsec"])
 

scripts/engine/lib/host.py

@@ -1,14 +1,15 @@
-# engine/lib/host.py
+# scripts/engine/lib/host.py
 """ctx.host — idempotent host mutation: backup, write_file, inject_block, install_crontab.
 
 Ports root_template_crontab (install + rollback) and the ufw block-injection logic
 (root_template.sh _root_template_inject_rules_to_ufw_file), generalized + self-healing.
 """
 from __future__ import annotations
+
+import inspect
+import json
 import os
-import shutil
 import subprocess
-import tempfile
 import time
 from typing import List, Optional, Sequence, Tuple
 
@@ -51,7 +52,61 @@ def apply_block(text: str, content: str, start: str, end: str,
         if not inserted and anchor in ln:
             result.extend(block)
             inserted = True
-    return "\n".join(result) + "\n", True
+    return "\n".join(result) + "\n", inserted
+
+
+def _atomic_write(path, content, mode):
+    """Self-contained so sudo executes the same operation without importing user code."""
+    import os
+    import stat
+    import tempfile
+
+    target = os.path.realpath(path)
+    parent = os.path.dirname(target)
+    try:
+        previous = os.stat(target)
+    except FileNotFoundError:
+        previous = None
+    if previous is not None and not stat.S_ISREG(previous.st_mode):
+        raise ValueError(f"Not a regular file: {path}")
+    target_mode = (int(mode, 8) if mode is not None else
+                   stat.S_IMODE(previous.st_mode) if previous is not None else 0o644)
+    os.makedirs(parent, exist_ok=True)
+    fd, temporary = tempfile.mkstemp(prefix=".apex-", dir=parent)
+    try:
+        with os.fdopen(fd, "w", encoding="utf-8", newline="") as stream:
+            stream.write(content)
+            stream.flush()
+            if previous is not None:
+                current = os.fstat(stream.fileno())
+                if (current.st_uid, current.st_gid) != (previous.st_uid, previous.st_gid):
+                    os.fchown(stream.fileno(), previous.st_uid, previous.st_gid)
+            os.fchmod(stream.fileno(), target_mode)
+            os.fsync(stream.fileno())
+        os.replace(temporary, target)
+    finally:
+        if os.path.exists(temporary):
+            os.unlink(temporary)
+
+
+def _atomic_backup(path, destination):
+    import os
+    import shutil
+    import tempfile
+
+    destination = os.path.abspath(destination)
+    fd, temporary = tempfile.mkstemp(prefix=".apex-", dir=os.path.dirname(destination))
+    os.close(fd)
+    try:
+        shutil.copyfile(path, temporary)
+        previous, current = os.stat(path), os.stat(temporary)
+        if (current.st_uid, current.st_gid) != (previous.st_uid, previous.st_gid):
+            os.chown(temporary, previous.st_uid, previous.st_gid)
+        shutil.copystat(path, temporary)
+        os.replace(temporary, destination)
+    finally:
+        if os.path.exists(temporary):
+            os.unlink(temporary)
 
 
 class Host:
@@ -59,55 +114,47 @@ class Host:
         self.log = log
         self.sys = sys_
 
+    def _file_operation(self, operation, args, sudo):
+        if sudo:
+            script = (inspect.getsource(operation) + "\nimport json, sys\n" +
+                      f"{operation.__name__}(*json.load(sys.stdin))\n")
+            self.sys.sudo(["python3", "-I", "-c", script], input=json.dumps(args))
+        else:
+            operation(*args)
+
     # --- read/write with sudo awareness ---
     def _read(self, path: str, sudo: bool = False) -> str:
         if sudo:
-            cp = self.sys.run(["sudo", "cat", path], check=True, capture=True)
+            cp = self.sys.sudo(["cat", path], check=True, capture=True)
             return cp.stdout or ""
-        with open(path) as f:
+        with open(path, encoding="utf-8") as f:
             return f.read()
 
     def backup(self, path: str, sudo: bool = False) -> Optional[str]:
-        exists = (self.sys.ok(["sudo", "test", "-f", path]) if sudo
+        exists = (self.sys.sudo(["test", "-f", path], check=False).returncode == 0 if sudo
                   else os.path.isfile(path))
         if not exists:
             return None
         bkp = f"{path}.{int(time.time())}.bak"
-        if sudo:
+        self._file_operation(_atomic_backup, [path, bkp], sudo)
-            self.sys.sudo(["cp", "-a", path, bkp])
-        else:
-            shutil.copy2(path, bkp)
         self.log.info(f"Backed up {path} -> {bkp}")
         return bkp
 
-    def write_file(self, path: str, content: str, backup: bool = True,
+    def write_file(self, path: str, content: str, mode: Optional[str] = None,
-                   dry_run: bool = False, sudo: bool = False) -> None:
+                   backup: bool = True, dry_run: bool = False, sudo: bool = False) -> None:
         if dry_run:
             self.log.info(f"[DRY-RUN] would write {len(content)} bytes to {path}")
             return
         if backup:
             self.backup(path, sudo=sudo)
-        tmp = tempfile.NamedTemporaryFile("w", delete=False)
+        self._file_operation(_atomic_write, [path, content, mode], sudo)
-        tmp.write(content)
-        tmp.close()
-        # 0644 like the bash renders (umask-022 redirect); mkstemp's 0600 would make
-        # freshly created units/configs owner-only. Overwrites via sudo cp keep the
-        # destination's existing mode either way.
-        os.chmod(tmp.name, 0o644)
-        if sudo:
-            self.sys.sudo(["mkdir", "-p", os.path.dirname(path)])
-            self.sys.sudo(["cp", tmp.name, path])
-            os.unlink(tmp.name)
-        else:
-            os.makedirs(os.path.dirname(path), exist_ok=True)
-            shutil.move(tmp.name, path)
         self.log.success(f"Wrote {path}")
 
     def inject_block(self, target: str, content: str, start: str, end: str,
                      anchor: Optional[str] = None,
                      old_pairs: Sequence[Tuple[str, str]] = (),
                      dry_run: bool = False, sudo: bool = True) -> bool:
-        exists = (self.sys.ok(["sudo", "test", "-f", target]) if sudo
+        exists = (self.sys.sudo(["test", "-f", target], check=False).returncode == 0 if sudo
                   else os.path.isfile(target))
         if not exists:
             self.log.error(f"Target file does not exist: {target}")

scripts/engine/lib/log.py

@@ -1,4 +1,4 @@
-# engine/lib/log.py
+# scripts/engine/lib/log.py
 """ctx.log — [node] [action] [level] message (port of root_log_*)."""
 from __future__ import annotations
 import sys

scripts/engine/lib/notify.py

@@ -1,16 +1,30 @@
-# engine/lib/notify.py
+# scripts/engine/lib/notify.py
-"""ctx.notify — Telegram notifications (port of root_core_telegram*).
+"""ctx.notify — Telegram notifications.
 
-instance renamed com.ermnvldmr.root.<node> -> com.ermnvldmr.apex.<node>. POSTs
+The alert `instance` is the node's public host (APEX_NODE_HOST), passed in from the
-application/x-www-form-urlencoded via urllib (stdlib) — functionally identical to
+context. POSTs application/x-www-form-urlencoded via urllib (stdlib).
-the bash curl multipart for Telegram sendMessage.
 """
 from __future__ import annotations
+
 import html
 import urllib.parse
 import urllib.request
 from datetime import datetime
 
+NOTIFY_TIMEOUT = 20
+MAX_MESSAGE_LEN = 3500
+
+
+def resolve_bot_url(value: str) -> str:
+    """Resolve a literal URL or an @file reference without putting secrets in argv."""
+    if not value or not value.startswith("@"):
+        return value
+    path = value[1:]
+    if not path:
+        raise ValueError("empty Telegram Bot URL file reference")
+    with open(path, encoding="utf-8") as f:
+        return f.read().strip()
+
 
 def build_text(title: str, message: str, node: str, level: str, ts: str) -> str:
     status, icon, severity = "firing", "🚨", "info"
@@ -20,8 +34,8 @@ def build_text(title: str, message: str, node: str, level: str, ts: str) -> str:
         severity = "critical"
     elif level == "WARN":
         severity = "warning"
-    instance = f"com.ermnvldmr.apex.{node}"
+    instance = node
-    msg = message if len(message) <= 200 else message[:200] + "..."
+    msg = message if len(message) <= MAX_MESSAGE_LEN else message[:MAX_MESSAGE_LEN] + "..."
     msg = html.escape(msg)
     safe_title = html.escape(title)
     return (f"{icon} <b>{safe_title}</b>\n"
@@ -40,15 +54,27 @@ class Notify:
 
     def telegram(self, title, bot_url, message, node=None, level="INFO") -> bool:
         node = node or self.node
+        try:
+            bot_url = resolve_bot_url(bot_url)
+        except (OSError, ValueError) as e:
+            self.log.error(f"Failed to read Telegram Bot URL: {e}")
+            return False
         if not bot_url:
             self.log.warn("Telegram Bot URL not provided. Skipping notification.")
             return False
+
+        parsed = urllib.parse.urlparse(bot_url)
+        if parsed.scheme != "https":
+            self.log.error(f"Telegram Bot URL must use HTTPS, got scheme: {parsed.scheme!r}")
+            return False
+
         ts = datetime.now().astimezone().strftime("%Y-%m-%d %H:%M:%S %Z")
         text = build_text(title, message, node, level, ts)
         data = urllib.parse.urlencode({"parse_mode": "HTML", "text": text}).encode()
         self.log.info(f"Sending Telegram notification ({level}): {title}")
         try:
-            with urllib.request.urlopen(urllib.request.Request(bot_url, data=data), timeout=120) as r:
+            with urllib.request.urlopen(urllib.request.Request(bot_url, data=data),  # noqa: S310 -- HTTPS validated above.
+                                        timeout=NOTIFY_TIMEOUT) as r:
                 r.read()
             return True
         except Exception as e:

scripts/engine/lib/sys.py

@@ -1,6 +1,7 @@
-# engine/lib/sys.py
+# scripts/engine/lib/sys.py
 """ctx.sys — command execution + systemd service management (port of root_sys.sh)."""
 from __future__ import annotations
+import os
 import shutil
 import subprocess
 import time
@@ -21,6 +22,8 @@ class Sys:
                               stderr=subprocess.PIPE if capture else None)
 
     def sudo(self, cmd: Sequence[str], **kw) -> subprocess.CompletedProcess:
+        if os.geteuid() == 0 and shutil.which("sudo") is None:
+            return self.run(cmd, **kw)
         return self.run(["sudo", *cmd], **kw)
 
     def ok(self, cmd: Sequence[str], **kw) -> bool:

scripts/engine/lib/systemd.py

@@ -1,4 +1,4 @@
-# engine/lib/systemd.py
+# scripts/engine/lib/systemd.py
 """ctx.systemd — render + install unit files, daemon-reload, enable+start."""
 from __future__ import annotations
 import os

scripts/engine/lib/tpl.py

@@ -1,9 +1,8 @@
-# engine/lib/tpl.py
+# scripts/engine/lib/tpl.py
-"""ctx.tpl — stdlib string.Template rendering (envsubst successor, SP2 decision #3).
+"""ctx.tpl — stdlib string.Template rendering.
 
-safe_substitute replaces known $VAR/${VAR} and leaves unknown ones literal (deliberate
+safe_substitute replaces known $VAR/${VAR} and leaves unknown ones literal.
-delta: envsubst emptied them — literal survivors surface typos instead of silently
+Shell $(...) command-substitution survives untouched.
-blanking). Shell $(...) command-substitution survives untouched.
 Loops/conditionals live in the python action, never in the template file.
 """
 from __future__ import annotations

scripts/engine/lib/ufw.py

@@ -0,0 +1,105 @@
+# scripts/engine/lib/ufw.py
+"""Apply ufw and ufw-docker rules.
+
+Replace existing supported marker blocks before inserting current rules.
+Docker rules go through ufw-docker; host rules through plain ufw.
+"""
+from __future__ import annotations
+
+import os
+import shlex
+import shutil
+
+APEX_START = "# START APEX.ERMNVLDMR.COM RULES"
+APEX_END = "# END APEX.ERMNVLDMR.COM RULES"
+OLD_PAIRS = [("# START ROOT.ERMNVLDMR.COM RULES", "# END ROOT.ERMNVLDMR.COM RULES")]
+
+
+def resolve_ufw_docker_bin() -> str:
+    """Find ufw-docker in PATH, /usr/local/bin, or fallback to ~/.local/bin/ufw-docker."""
+    found = shutil.which("ufw-docker")
+    if found:
+        return found
+    if os.path.isfile("/usr/local/bin/ufw-docker") and os.access("/usr/local/bin/ufw-docker", os.X_OK):
+        return "/usr/local/bin/ufw-docker"
+    return os.path.expanduser("~/.local/bin/ufw-docker")
+
+
+UFW_DOCKER_BIN = resolve_ufw_docker_bin()
+
+
+def clean_rules(text: str):
+    """Strip whitespace, drop blank + comment lines (matches the bash awk filter)."""
+    return [s for s in (ln.strip() for ln in text.splitlines()) if s and not s.startswith("#")]
+
+
+class Ufw:
+    def __init__(self, log, sys_, host):
+        self.log, self.sys, self.host = log, sys_, host
+
+    def _inject_file(self, target: str, source_file: str, dry_run: bool):
+        if not os.path.isfile(source_file):
+            return
+        if not self.sys.ok(["sudo", "test", "-f", target]):
+            self.log.error(f"Target file does not exist: {target}")
+            raise SystemExit(1)
+        with open(source_file, encoding="utf-8") as f:
+            content = "\n".join(clean_rules(f.read()))
+        self.host.inject_block(
+            target,
+            content,
+            APEX_START,
+            APEX_END,
+            anchor="# End required lines",
+            old_pairs=OLD_PAIRS,
+            dry_run=dry_run,
+            sudo=True,
+        )
+
+    def apply(self, rules_dir: str | None = None, config_dir: str | None = None, dry_run: bool = False) -> None:
+        if rules_dir and not os.path.isdir(rules_dir):
+            self.log.error(f"Rules directory does not exist: {rules_dir}")
+            raise SystemExit(66)
+        if config_dir and not os.path.isdir(config_dir):
+            self.log.error(f"Config directory does not exist: {config_dir}")
+            raise SystemExit(66)
+        if not rules_dir and not config_dir:
+            self.log.warn("Neither docker nor host ufw rules directory found; nothing to apply.")
+            return
+        if not self.sys.ok(["sudo", "ufw", "--version"]):
+            self.log.error("ufw not installed/working via sudo. Run config/packages first.")
+            raise SystemExit(1)
+        if not (os.path.isfile(UFW_DOCKER_BIN) and os.access(UFW_DOCKER_BIN, os.X_OK)):
+            self.log.error(f"ufw-docker not found at {UFW_DOCKER_BIN}. Run config/ufw-docker first.")
+            raise SystemExit(1)
+
+        if config_dir:
+            host_rules = os.path.join(config_dir, "host.rules")
+            if os.path.isfile(host_rules):
+                with open(host_rules, encoding="utf-8") as f:
+                    rules = clean_rules(f.read())
+                for rule in rules:
+                    self.log.info(f"Applying host ufw rule: {rule}")
+                    if dry_run:
+                        self.log.info(f"[DRY-RUN] sudo ufw {rule}")
+                    else:
+                        self.sys.sudo(["ufw", *shlex.split(rule)], check=True)
+            self._inject_file("/etc/ufw/before.rules", os.path.join(config_dir, "before.rules"), dry_run)
+            self._inject_file("/etc/ufw/after.rules", os.path.join(config_dir, "after.rules"), dry_run)
+
+        if rules_dir:
+            rule_files = [fn for fn in sorted(os.listdir(rules_dir)) if fn.endswith(".rules") and not fn.startswith(".")]
+            if not rule_files:
+                self.log.warn(f"No .rules files found in {rules_dir}, nothing to apply.")
+            for fn in rule_files:
+                with open(os.path.join(rules_dir, fn), encoding="utf-8") as f:
+                    rules = clean_rules(f.read())
+                for rule in rules:
+                    self.log.info(f"Applying ufw-docker rule ({fn}): {rule}")
+                    if dry_run:
+                        self.log.info(f"[DRY-RUN] sudo {UFW_DOCKER_BIN} {rule}")
+                    else:
+                        self.sys.sudo([UFW_DOCKER_BIN, *shlex.split(rule)], check=True)
+
+        self.log.success("All ufw rules applied.")
+        self.log.warn("Run `sudo systemctl restart ufw` to apply. Ensure ssh is allowed via plain ufw!")

scripts/engine/overlay.py

@@ -1,4 +1,4 @@
-# engine/overlay.py
+# scripts/engine/overlay.py
 """Resolve action names across commons + proprietaries.
 
 proprietaries first (override), commons second. Listing uses static ast scanning
@@ -9,10 +9,12 @@ from __future__ import annotations
 import ast
 import importlib.util
 import os
+import re
 from dataclasses import dataclass
 from typing import List, Optional
 
 ACTION_EXT = ".py"
+ACTION_NAME = re.compile(r"[A-Za-z0-9_-]+/[A-Za-z0-9_-]+")
 
 
 @dataclass
@@ -40,11 +42,15 @@ def _rel_names(root: Optional[str]):
                 continue
             full = os.path.join(dirpath, fn)
             rel = os.path.relpath(full, root)[: -len(ACTION_EXT)]
-            out[rel.replace(os.sep, "/")] = full
+            name = rel.replace(os.sep, "/")
+            if ACTION_NAME.fullmatch(name):
+                out[name] = full
     return out
 
 
 def resolve(name: str, commons_root: str, prop_root: Optional[str]) -> Optional[Resolved]:
+    if not ACTION_NAME.fullmatch(name):
+        return None
     rel = name + ACTION_EXT
     commons_path = os.path.join(commons_root, rel)
     c = os.path.isfile(commons_path)

scripts/githooks/pre-commit

@@ -6,4 +6,4 @@ else
     APEX_CMD="$(git rev-parse --show-toplevel)/commons/apex"
     [[ -x "$APEX_CMD" ]] || APEX_CMD="$(git rev-parse --show-toplevel)/apex"  # standalone commons repo
 fi
-"$APEX_CMD" utils/lint-docker-compose --hook || { echo "[pre-commit] Aborting: apex utils/lint-docker-compose failed."; exit 1; }
+"$APEX_CMD" util/lint-docker-compose --hook || { echo "[pre-commit] Aborting: apex util/lint-docker-compose failed."; exit 1; }

scripts/provisioning/__init__.py

@@ -0,0 +1 @@
+"""Host-side Docker installation and SSH handover."""

scripts/provisioning/access.py

@@ -0,0 +1,597 @@
+"""Protected SSH handover with a separately locked, persistent rollback timer."""
+from __future__ import annotations
+
+import argparse
+import base64
+import fcntl
+import fnmatch
+import hashlib
+import ipaddress
+import json
+import os
+from pathlib import Path
+import pwd
+import re
+import shlex
+import stat
+import subprocess
+import sys
+import tempfile
+import time
+
+
+class AccessError(RuntimeError):
+    pass
+
+
+def include_arguments(value):
+    arguments, word, quote, started = [], [], None, False
+    for character in value:
+        if character == '\\':
+            raise AccessError('Escaped SSH Include paths require manual reconciliation')
+        if quote:
+            if character == quote:
+                quote = None
+            else:
+                word.append(character)
+        elif character in ('"', "'"):
+            quote, started = character, True
+        elif character in ' \t\r':
+            if started:
+                arguments.append(''.join(word))
+                word, started = [], False
+        elif character == '#' and not started:
+            break
+        else:
+            word.append(character)
+            started = True
+    if quote:
+        raise AccessError('Unclosed SSH Include quote requires manual reconciliation')
+    if started:
+        arguments.append(''.join(word))
+    if not arguments or len(arguments) > 128 or any(not item or len(item) > 4096 for item in arguments):
+        raise AccessError('Unsupported SSH Include paths require manual reconciliation')
+    return arguments
+
+
+MANUAL_PROFILES = ('debian', 'proxmox')
+OPERATOR_MATCH = """Match User adam
+AuthenticationMethods publickey
+PubkeyAuthentication yes
+PasswordAuthentication no
+KbdInteractiveAuthentication no
+PermitEmptyPasswords no
+X11Forwarding no""".splitlines()
+
+
+class Host:
+    def __init__(self, root=Path('/'), runner=subprocess.run, profile=None):
+        if profile is not None and profile not in MANUAL_PROFILES:
+            raise AccessError('Unknown SSH access profile')
+        self.profile = profile
+        self.root = Path(root).resolve()
+        self.runner = runner
+
+    def path(self, path):
+        return self.root / path.lstrip('/')
+
+    def command(self, args, check=True):
+        result = self.runner(args, capture_output=True, text=True, timeout=60)
+        if check and result.returncode:
+            raise AccessError('Access command failed: ' + args[0])
+        return result
+
+    def boot_id(self):
+        return self.path('/proc/sys/kernel/random/boot_id').read_text().strip()
+
+    def fingerprint(self):
+        digest = hashlib.sha256()
+        paths = [self.path('/etc/ssh/sshd_config'), self.path('/etc/default/ssh')]
+        for directory in ('/etc/ssh/sshd_config.d', '/etc/systemd/system/ssh.service.d',
+                          '/run/systemd/system/ssh.service.d', '/usr/lib/systemd/system/ssh.service.d',
+                          '/lib/systemd/system/ssh.service.d'):
+            paths += sorted(self.path(directory).glob('*.conf'))
+        for path in paths:
+            digest.update(str(path.relative_to(self.root)).encode())
+            digest.update(path.read_bytes() if path.exists() else b'<absent>')
+        return digest.hexdigest()
+
+    def unconditional_configuration(self):
+        remaining, directory_entries, visited = 1024 * 1024, 0, set()
+
+        def expand(pattern):
+            nonlocal directory_entries
+            if any(token in pattern for token in ('[:', '[.', '[=', '[^')):
+                raise AccessError('Extended SSH Include glob syntax requires manual reconciliation')
+            absolute = pattern if pattern.startswith('/') else '/etc/ssh/' + pattern
+            mapped = self.path(absolute)
+            if len(mapped.parts) > 64:
+                raise AccessError('SSH Include path exceeds supported limits; manual reconciliation required')
+            candidates = [Path(mapped.anchor)]
+            for component in mapped.parts[1:]:
+                expanded = []
+                for parent in candidates:
+                    if not any(character in component for character in '*?['):
+                        candidate = parent / component
+                        if os.path.lexists(candidate):
+                            expanded.append(candidate)
+                        continue
+                    try:
+                        with os.scandir(parent) as entries:
+                            for entry in entries:
+                                directory_entries += 1
+                                if directory_entries > 4096:
+                                    raise AccessError('SSH Include traversal exceeds supported limits; manual reconciliation required')
+                                if entry.name.startswith('.') and not component.startswith('.'):
+                                    continue
+                                if fnmatch.fnmatchcase(entry.name, component):
+                                    expanded.append(Path(entry.path))
+                    except (FileNotFoundError, NotADirectoryError):
+                        continue
+                if len(expanded) > 128:
+                    raise AccessError('SSH Include expansion exceeds supported limits; manual reconciliation required')
+                candidates = expanded
+            return sorted(candidates)
+
+        def inspect(path, ancestors):
+            nonlocal remaining
+            information = path.stat()
+            identity = (information.st_dev, information.st_ino)
+            if identity in ancestors or len(ancestors) >= 16:
+                raise AccessError('Recursive SSH Include configuration requires manual reconciliation')
+            if identity in visited:
+                return
+            if len(visited) >= 128 or not stat.S_ISREG(information.st_mode) or information.st_size > remaining:
+                raise AccessError('SSH configuration exceeds supported inspection limits; manual reconciliation required')
+            visited.add(identity)
+            with path.open('rb') as stream:
+                content = stream.read(remaining + 1)
+            remaining -= len(content)
+            if remaining < 0:
+                raise AccessError('SSH configuration exceeds supported inspection limits; manual reconciliation required')
+            lines = content.decode().splitlines()
+            if self.profile and path == self.path('/etc/ssh/sshd_config'):
+                # The legacy action appended this exact terminal block. Any additional
+                # directives would change its meaning and require manual inspection.
+                marker = '# APEX operator SSH policy'
+                if marker in lines:
+                    position = lines.index(marker)
+                    if [item.strip() for item in lines[position + 1:] if item.strip()] == OPERATOR_MATCH:
+                        lines = lines[:position]
+            for line in lines:
+                line = line.lstrip()
+                if not line or line.startswith('#'):
+                    continue
+                if line.startswith('"'):
+                    keyword, separator, value = line[1:].partition('"')
+                    if not separator:
+                        raise AccessError('Unsupported SSH keyword quoting requires manual reconciliation')
+                else:
+                    parsed = re.match(r'([^\s=]+)(.*)', line)
+                    if not parsed:
+                        raise AccessError('Unsupported SSH configuration syntax requires manual reconciliation')
+                    keyword, value = parsed.groups()
+                    value = value.lstrip()
+                    if value.startswith('='):
+                        value = value[1:]
+                if keyword.lower() == 'match':
+                    raise AccessError('SSH Match directives require manual reconciliation before initialization')
+                if keyword.lower() != 'include':
+                    continue
+                for pattern in include_arguments(value):
+                    for included in expand(pattern):
+                        inspect(included, ancestors | {identity})
+
+        try:
+            inspect(self.path('/etc/ssh/sshd_config'), set())
+        except (OSError, UnicodeError) as exc:
+            raise AccessError('Cannot inspect SSH configuration and includes; manual reconciliation required') from exc
+
+    def bootstrap_preflight(self):
+        for activity in ('is-active', 'is-enabled'):
+            if self.command(['systemctl', activity, '--quiet', 'ssh.socket'], False).returncode == 0:
+                raise AccessError('SSH socket activation requires manual handover; no access changes made')
+        if self.command(['systemctl', 'is-active', '--quiet', 'ssh.service'], False).returncode:
+            raise AccessError('An active ssh.service is required')
+        mode = self.command(['systemctl', 'show', 'ssh.service', '-p', 'KillMode', '--value']).stdout.strip()
+        if mode != 'process':
+            raise AccessError('ssh.service must preserve established sessions with KillMode=process')
+        arguments = self.service_arguments()
+        managed = self.path('/etc/systemd/system/ssh.service.d/apex-init.conf')
+        if managed.exists():
+            commands = [line.removeprefix('ExecStart=') for line in managed.read_text().splitlines()
+                        if line.startswith('ExecStart=') and line != 'ExecStart=']
+            if len(commands) != 1 or arguments != shlex.split(commands[0]):
+                raise AccessError('Active SSH service configuration differs from its managed invocation')
+        elif arguments != ['/usr/sbin/sshd', '-D']:
+            raise AccessError('Custom SSH service invocation requires manual access reconciliation')
+        defaults = self.path('/etc/default/ssh')
+        if defaults.exists():
+            for line in defaults.read_text().splitlines():
+                if re.match(r'^\s*SSHD_OPTS\s*=', line):
+                    value = line.split('=', 1)[1].strip().strip('\"\'')
+                    if value:
+                        raise AccessError('Custom SSHD_OPTS require manual access reconciliation')
+        self.unconditional_configuration()
+        self.command(['/usr/sbin/sshd', '-t'])
+        self.live_arguments()
+
+    def preflight(self):
+        self.bootstrap_preflight()
+        self.command(['visudo', '-c'])
+        self.command(['runuser', '-u', 'adam', '--', 'sudo', '-n', 'true'])
+
+    def service_arguments(self):
+        output = self.command(['systemctl', 'show', 'ssh.service', '-p', 'ExecStart', '--value']).stdout
+        matches = re.findall(r'argv\[\]=([^;]+)', output)
+        if len(matches) != 1:
+            raise AccessError('Unable to verify the SSH service invocation')
+        arguments = shlex.split(matches[0])
+        # Debian's stock unit has this optional variable. Its actual expansion is
+        # independently checked against /proc below, so a nonempty value fails.
+        if arguments == ['/usr/sbin/sshd', '-D', '$SSHD_OPTS']:
+            arguments.pop()
+        if not arguments or arguments[0] != '/usr/sbin/sshd':
+            raise AccessError('SSH service does not invoke the supported daemon')
+        return arguments
+
+    def live_arguments(self):
+        expected = self.service_arguments()
+        pid = self.command(['systemctl', 'show', 'ssh.service', '-p', 'MainPID', '--value']).stdout.strip()
+        if not pid.isdecimal() or int(pid) <= 1:
+            raise AccessError('SSH service has no verifiable active listener process')
+        process = self.path('/proc/' + pid)
+        try:
+            if os.readlink(process / 'exe') != '/usr/sbin/sshd':
+                raise AccessError('Active SSH process does not run the supported daemon')
+            parts = [part for part in (process / 'cmdline').read_bytes().decode().split('\0') if part]
+        except (OSError, UnicodeError):
+            raise AccessError('Unable to inspect the active SSH listener invocation') from None
+        if len(parts) == 1 and parts[0].startswith('sshd: ') and ' [listener]' in parts[0]:
+            actual = shlex.split(parts[0][6:].split(' [listener]', 1)[0])
+        else:
+            actual = parts
+        if actual != expected:
+            raise AccessError('Running SSH listener differs from the active service invocation')
+        return actual, pid
+
+    def verify_active(self, ports, final, address):
+        args, pid = self.live_arguments()
+        self.verify_policy([arg for arg in args if arg != '-D'], ports, final, address)
+        if self.listener_ports(pid) != set(ports):
+            raise AccessError('Active SSH listening ports differ from the declared policy')
+
+    def listener_ports(self, pid):
+        output = self.command(['ss', '-H', '-ltnp']).stdout
+        actual_ports = set()
+        for line in output.splitlines():
+            if '("sshd",' not in line:
+                continue
+            if set(re.findall(r'pid=(\d+)', line)) != {pid}:
+                raise AccessError('An additional SSH listener exists outside the managed service')
+            fields = line.split()
+            if len(fields) < 5 or not fields[3].rsplit(':', 1)[-1].isdecimal():
+                raise AccessError('Unable to verify the active SSH listening sockets')
+            actual_ports.add(int(fields[3].rsplit(':', 1)[-1]))
+        return actual_ports
+
+    @property
+    def final_ports(self):
+        return [22, 2222] if self.profile == 'proxmox' else [2222]
+
+    def authentication_options(self):
+        proxmox = self.profile == 'proxmox'
+        options = {'PermitRootLogin': 'prohibit-password' if proxmox else 'no',
+                   'PasswordAuthentication': 'no', 'KbdInteractiveAuthentication': 'no',
+                   'PubkeyAuthentication': 'yes', 'AuthenticationMethods': 'publickey'}
+        if not proxmox:
+            options['AllowUsers'] = 'adam'
+        options.update(X11Forwarding='no', PermitEmptyPasswords='no')
+        return options
+
+    def policy(self, ports, final, address):
+        args = ['/usr/sbin/sshd']
+        for port in sorted(set(ports)):
+            args += ['-p', str(port)]
+        if final:
+            for key, value in self.authentication_options().items():
+                args += ['-o', f'{key}={value}']
+        elif self.profile:
+            # Keep existing managed authentication options when adding the probe port.
+            current = iter(self.service_arguments()[1:])
+            for flag in current:
+                if flag == '-D':
+                    continue
+                value = next(current, '')
+                if flag == '-p' and value.isdecimal():
+                    continue
+                if flag != '-o' or not re.fullmatch(r'[A-Za-z]+=[A-Za-z0-9,-]+', value):
+                    raise AccessError('Cannot safely extend the existing SSH invocation')
+                args += [flag, value]
+        self.verify_policy(args, ports, final, address)
+        return args
+
+    def verify_policy(self, args, ports, final, address):
+        self.unconditional_configuration()
+        self.command(args + ['-t'])
+        users = {'adam', 'root'}
+        if final:
+            users.update(account.pw_name for account in pwd.getpwall())
+        for user in sorted(users):
+            output = self.command(args + ['-T', '-C', f'user={user},host=localhost,addr={address}']).stdout
+            values = {}
+            for line in output.splitlines():
+                key, _, value = line.partition(' ')
+                values.setdefault(key, []).append(value)
+            if set(values.get('port', [])) != {str(p) for p in ports}:
+                raise AccessError('Effective SSH ports do not match the handover policy')
+            if final:
+                expected = {key.lower(): [value] for key, value in self.authentication_options().items()}
+                if self.profile == 'proxmox':
+                    if values.get('permitrootlogin') == ['without-password']:
+                        values['permitrootlogin'] = ['prohibit-password']
+                    if any(key in values for key in ('allowusers', 'denyusers', 'allowgroups', 'denygroups')):
+                        raise AccessError('SSH account restrictions conflict with Proxmox root-key access')
+                if any(values.get(key) != value for key, value in expected.items()):
+                    raise AccessError('Effective SSH authentication policy is inconsistent')
+
+    def install_policy(self, ports, final, address):
+        args = self.policy(ports, final, address)
+        path = self.path('/etc/systemd/system/ssh.service.d/apex-init.conf')
+        atomic(path, ('[Service]\nExecStart=\nExecStart=' + ' '.join(args + ['-D']) + '\n').encode(), 0o644)
+        self.command(['systemctl', 'daemon-reload'])
+        actual = self.command(['systemctl', 'show', 'ssh.service', '-p', 'ExecStart', '--value']).stdout
+        match = re.search(r'argv\[\]=([^;]+)', actual)
+        if not match or shlex.split(match[1]) != args + ['-D']:
+            raise AccessError('Another service override prevents the managed SSH policy')
+        self.command(['systemctl', 'restart', 'ssh.service'])
+        self.verify_active(ports, final, address)
+
+    def arm(self):
+        bundle = Path(__file__).resolve().parents[1]
+        if not re.fullmatch(r'/usr/local/lib/apex-provisioner/[a-f0-9]{64}', str(bundle)):
+            raise AccessError('Access runner must execute from the installed root-owned bundle')
+        service = ('[Unit]\nDescription=APEX pending access rollback\nAfter=local-fs.target\n'
+                   '[Service]\nType=oneshot\nWorkingDirectory=' + str(bundle) + '\n'
+                   'ExecStart=/usr/bin/python3 -m provisioning.access tick\n')
+        timer = ('[Unit]\nDescription=APEX access rollback deadline and boot recovery\n'
+                 '[Timer]\nOnBootSec=5s\nOnUnitActiveSec=10s\nAccuracySec=1s\n'
+                 '[Install]\nWantedBy=timers.target\n')
+        atomic(self.path('/etc/systemd/system/apex-access-rollback.service'), service.encode(), 0o644)
+        atomic(self.path('/etc/systemd/system/apex-access-rollback.timer'), timer.encode(), 0o644)
+        self.command(['systemctl', 'daemon-reload'])
+        self.command(['systemctl', 'enable', '--now', 'apex-access-rollback.timer'])
+
+    def disarm(self):
+        self.command(['systemctl', 'disable', '--now', 'apex-access-rollback.timer'])
+
+    def restore(self, state):
+        path = self.path('/etc/systemd/system/ssh.service.d/apex-init.conf')
+        if state['previous'] is None:
+            path.unlink(missing_ok=True)
+        else:
+            atomic(path, base64.b64decode(state['previous']), 0o644)
+        self.command(['/usr/sbin/sshd', '-t'])
+        self.command(['systemctl', 'daemon-reload'])
+        self.command(['systemctl', 'restart', 'ssh.service'])
+
+
+def atomic(path, data, mode=0o600):
+    path = Path(path)
+    for parent in [path, *path.parents]:
+        if parent.is_symlink():
+            raise AccessError('Refusing a symlink in a managed access path')
+    path.parent.mkdir(parents=True, exist_ok=True)
+    fd, temporary = tempfile.mkstemp(prefix='.apex-access-', dir=path.parent)
+    try:
+        with os.fdopen(fd, 'wb') as stream:
+            os.fchmod(stream.fileno(), mode)
+            stream.write(data)
+            stream.flush()
+            os.fsync(stream.fileno())
+        os.replace(temporary, path)
+    finally:
+        if os.path.exists(temporary):
+            os.unlink(temporary)
+    directory = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY)
+    try:
+        os.fsync(directory)
+    finally:
+        os.close(directory)
+
+
+class Transition:
+    def __init__(self, host=None, now=time.monotonic, profile=None):
+        if profile is not None and profile not in MANUAL_PROFILES:
+            raise AccessError('Unknown SSH access profile')
+        self.profile = profile
+        self.host = host or Host()
+        self.host.profile = profile
+        self.now = now
+        self.directory = self.host.path('/var/lib/apex-init/access')
+
+    def load(self):
+        path = self.directory / 'manual-state.json'
+        return json.loads(path.read_text()) if path.exists() else None
+
+    def save(self, state):
+        atomic(self.directory / 'manual-state.json', json.dumps(state, sort_keys=True).encode())
+
+    def execute(self, operation, token=None, original_port=None, client_address=None):
+        # The watchdog must recover SSH even when another APEX action holds init.lock.
+        if operation in ('tick', 'rollback'):
+            return self._execute(operation, token, original_port, client_address)
+        from .state import ProvisionError, exclusive_lock
+        try:
+            with exclusive_lock(self.host.path('/var/lib/apex-init/init.lock')):
+                return self._execute(operation, token, original_port, client_address)
+        except ProvisionError as error:
+            raise AccessError(str(error)) from None
+
+    def _execute(self, operation, token=None, original_port=None, client_address=None):
+        if token is not None and not re.fullmatch(r'[a-f0-9]{32}', token):
+            raise AccessError('Invalid access transition token')
+        for path in [self.directory, *self.directory.parents]:
+            if path.is_symlink():
+                raise AccessError('Invalid access state directory')
+        self.directory.mkdir(parents=True, exist_ok=True, mode=0o700)
+        if self.directory.stat().st_uid != os.geteuid():
+            raise AccessError('Access state directory is not owned by the provisioner user')
+        os.chmod(self.directory, 0o700)
+        descriptor = os.open(self.directory / 'lock', os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW, 0o600)
+        with os.fdopen(descriptor, 'w') as lock:
+            fcntl.flock(lock, fcntl.LOCK_EX)
+            state = self.load()
+            active = ('transition', 'final', 'recovery_pending')
+            # A historical bundle owns its pending rollback and the shared timer.
+            legacy_path = self.directory / 'state.json'
+            if legacy_path.exists():
+                legacy = json.loads(legacy_path.read_text())
+                if legacy['status'] in active or legacy.get('disarm_pending'):
+                    raise AccessError('Another access workflow is pending; wait for rollback before retrying')
+            if state and self.profile and state.get('profile') != self.profile and operation != 'tick':
+                if operation != 'begin' or state['status'] in active:
+                    raise AccessError('Access transition belongs to a different platform')
+                state = None
+            if state and state.get('disarm_pending') and state['status'] in ('committed', 'rolled_back'):
+                self._disarm(state)
+            if (state and token and token == state.get('token')
+                    and (operation, state['status']) in (('commit', 'committed'), ('rollback', 'rolled_back'))):
+                return {'status': state['status']}
+            if state and state['status'] == 'recovery_pending':
+                self._rollback(state)
+                if operation != 'tick':
+                    raise AccessError('Pending recovery completed; retry initialization')
+                return {'status': 'rolled_back'}
+            pending = state and state['status'] in ('transition', 'final')
+            if pending and (self.now() >= state['deadline'] or self.host.boot_id() != state['boot_id']):
+                self._rollback(state)
+                if operation != 'tick':
+                    raise AccessError('Uncommitted access transition expired or rebooted; previous access restored')
+                return {'status': 'rolled_back'}
+            if operation == 'tick':
+                return {'status': state['status'] if state else 'absent'}
+            if operation == 'begin':
+                if pending:
+                    raise AccessError('A pending access transition exists; wait for rollback before retrying')
+                if not token or not original_port or not 1 <= original_port <= 65535:
+                    raise AccessError('Access transition requires an original port and token')
+                try:
+                    ipaddress.ip_address(client_address)
+                except ValueError:
+                    raise AccessError('Invalid SSH client address') from None
+                self.host.preflight()
+                path = self.host.path('/etc/systemd/system/ssh.service.d/apex-init.conf')
+                return self._begin_manual(token, original_port, client_address, path)
+            if not pending or token != state['token']:
+                raise AccessError('No matching pending access transition')
+            if operation == 'rollback':
+                self._rollback(state)
+                return {'status': 'rolled_back'}
+            if state.get('fingerprint') != self.host.fingerprint():
+                raise AccessError('External SSH configuration changed; finalization stopped, rollback remains armed')
+            if operation == 'finalize':
+                if state['status'] != 'transition':
+                    raise AccessError('Access transition is not ready for final policy')
+                try:
+                    if not state.get('preserve_policy'):
+                        state['ssh_changed'] = True
+                        self.save(state)
+                        self.host.install_policy(self.host.final_ports, True, state['client_address'])
+                    state['authenticated_after_us'] = time.monotonic_ns() // 1000
+                    state['status'] = 'final'
+                    state['fingerprint'] = self.host.fingerprint()
+                    self.save(state)
+                except Exception:
+                    self._rollback(state)
+                    raise
+                return {'status': 'final'}
+            if operation == 'commit' and state['status'] == 'final':
+                self.host.verify_active(self.host.final_ports, True, state['client_address'])
+                state['status'] = 'committed'
+                state['disarm_pending'] = True
+                self.save(state)  # Timer observes this before it can restore anything.
+                self._disarm(state)
+                return {'status': 'committed'}
+            raise AccessError('Invalid access transition operation')
+
+    def _begin_manual(self, token, original_port, address, path):
+        _, pid = self.host.live_arguments()
+        ports = self.host.listener_ports(pid)
+        if original_port not in ports:
+            raise AccessError('Original SSH port is not active on the managed daemon')
+        preserve_policy = False
+        if ports == set(self.host.final_ports):
+            try:
+                self.host.verify_active(self.host.final_ports, True, address)
+                preserve_policy = True
+            except AccessError:
+                pass
+        previous = base64.b64encode(path.read_bytes()).decode() if path.exists() else None
+        state = {'status': 'transition', 'profile': self.profile, 'token': token,
+                 'deadline': self.now() + 300, 'boot_id': self.host.boot_id(),
+                 'bundle': str(Path(__file__).resolve().parents[1]),
+                 'client_address': address, 'original_port': original_port, 'previous': previous,
+                 'preserve_policy': preserve_policy, 'ssh_changed': 2222 not in ports}
+        self.host.arm()
+        self.save(state)
+        try:
+            if 2222 not in ports:
+                self.host.install_policy(sorted(ports | {2222}), False, address)
+            state['fingerprint'] = self.host.fingerprint()
+            state['authenticated_after_us'] = time.monotonic_ns() // 1000
+            self.save(state)
+        except Exception:
+            self._rollback(state)
+            raise
+        return {'status': 'transition'}
+
+    def _rollback(self, state):
+        errors = []
+        if not state.get('ssh_restored') and state.get('ssh_changed', True):
+            try:
+                self.host.restore(state)
+                state['ssh_restored'] = True
+            except Exception:
+                errors.append('ssh')
+        if errors:
+            state['status'] = 'recovery_pending'
+            state['recovery_errors'] = errors
+            self.save(state)
+            raise AccessError('Recovery remains pending for ' + ', '.join(errors) +
+                              '; watchdog remains armed; inspect the root-owned access record')
+        state['status'] = 'rolled_back'
+        state.pop('recovery_errors', None)
+        state['disarm_pending'] = True
+        self.save(state)
+        self._disarm(state)
+
+    def _disarm(self, state):
+        self.host.disarm()
+        state.pop('disarm_pending', None)
+        self.save(state)
+
+
+def main(argv=None):
+    parser = argparse.ArgumentParser()
+    parser.add_argument('operation', choices=['begin', 'finalize', 'commit', 'rollback', 'tick'])
+    parser.add_argument('--profile', choices=MANUAL_PROFILES)
+    parser.add_argument('--token')
+    parser.add_argument('--original-port', type=int)
+    parser.add_argument('--client-address')
+    args = parser.parse_args(argv)
+    if os.geteuid() != 0:
+        parser.error('Access handover requires root')
+    try:
+        if args.operation != 'tick' and not args.profile:
+            parser.error('A host profile is required')
+        result = Transition(profile=args.profile).execute(args.operation, args.token, args.original_port, args.client_address)
+        print(json.dumps(result))
+        return 0
+    except AccessError as exc:
+        print(str(exc), file=sys.stderr)
+        return 1
+
+
+if __name__ == '__main__':
+    sys.exit(main())

scripts/provisioning/docker.py

@@ -0,0 +1,312 @@
+"""Verify or install a local Docker Engine and CLI plugins."""
+from __future__ import annotations
+
+from contextlib import contextmanager
+import fcntl
+from functools import wraps
+import hashlib
+import json
+import os
+from pathlib import Path
+import re
+import shutil
+import subprocess
+import tempfile
+import urllib.request
+
+
+class DockerError(RuntimeError):
+    """A required Docker postcondition could not be established."""
+
+
+def expected_errors(operation):
+    @wraps(operation)
+    def guarded(*args, **kwargs):
+        try:
+            return operation(*args, **kwargs)
+        except (OSError, ValueError) as exc:
+            raise DockerError("Docker filesystem or saved state is invalid; inspect the target before retrying.") from exc
+    return guarded
+
+
+DOCKER_PACKAGES = ("docker-ce", "docker-ce-cli", "containerd.io",
+                   "docker-buildx-plugin", "docker-compose-plugin")
+DOCKER_CONFLICTS = ("docker.io", "docker-doc", "docker-compose", "docker-buildx",
+                    "podman-docker", "containerd", "runc")
+DOCKER_CONFIG = {"data-root": "/srv/docker", "log-driver": "json-file",
+                 "log-opts": {"max-size": "20m", "max-file": "3"}}
+POLICY = Path("/usr/sbin/policy-rc.d")
+POLICY_STATE = Path("/usr/sbin/.apex-policy-rc.d")
+APT = ["apt-get", "-o", "DPkg::Lock::Timeout=60", "-o",
+       "Acquire::Retries=2", "-o", "Acquire::http::Timeout=30", "-o",
+       "Acquire::https::Timeout=30"]
+
+
+def run(args, *, check=True, timeout=300):
+    try:
+        result = subprocess.run(args, text=True, capture_output=True, timeout=timeout,
+                                env={**os.environ, "LC_ALL": "C", "DEBIAN_FRONTEND": "noninteractive"})
+    except (OSError, subprocess.TimeoutExpired) as exc:
+        raise DockerError(f"Cannot complete {args[0]}: {exc}") from exc
+    if check and result.returncode:
+        raise DockerError(f"{args[0]} failed with exit status {result.returncode}; inspect target system logs.")
+    return result
+
+
+def package_versions():
+    architecture = run(["dpkg", "--print-architecture"]).stdout.strip()
+    result = run(["dpkg-query", "-W", "-f=${binary:Package}\t${Status}\t${Version}\n"])
+    installed = {}
+    for line in result.stdout.splitlines():
+        fields = line.split("\t")
+        if len(fields) == 3 and fields[1].split()[1:] == ["ok", "installed"]:
+            installed[fields[0].removesuffix(":" + architecture)] = fields[2]
+    return installed
+
+
+def write_managed(path, content, mode=0o644):
+    path = Path(path)
+    for parent in (path, *path.parents):
+        if parent.is_symlink():
+            raise DockerError(f"Refusing managed symlink: {parent}")
+    payload = content.encode() if isinstance(content, str) else content
+    if path.exists() and path.read_bytes() == payload:
+        if path.stat().st_mode & 0o777 != mode:
+            path.chmod(mode)
+        return False
+    path.parent.mkdir(parents=True, exist_ok=True)
+    fd, temporary = tempfile.mkstemp(prefix=".apex-", dir=path.parent)
+    try:
+        with os.fdopen(fd, "wb") as stream:
+            stream.write(payload)
+            stream.flush()
+            os.fsync(stream.fileno())
+        os.chmod(temporary, mode)
+        os.replace(temporary, path)
+    finally:
+        if os.path.exists(temporary):
+            os.unlink(temporary)
+    return True
+
+
+def restore_service_policy():
+    if not POLICY_STATE.exists():
+        return
+    backup = POLICY_STATE / "original"
+    if (POLICY_STATE.is_symlink() or POLICY_STATE.stat().st_uid != os.geteuid()):
+        raise DockerError("Refusing untrusted service policy recovery state.")
+    ready = POLICY_STATE / "ready"
+    if not ready.exists():
+        # The live policy is untouched until the backup is complete.
+        shutil.rmtree(POLICY_STATE)
+        return
+    if POLICY.exists() or POLICY.is_symlink():
+        original_matches = (backup.is_file() and not backup.is_symlink() and not POLICY.is_symlink() and
+                            POLICY.read_bytes() == backup.read_bytes())
+        symlink_matches = (backup.is_symlink() and POLICY.is_symlink() and
+                           os.readlink(backup) == os.readlink(POLICY))
+        guard_matches = not POLICY.is_symlink() and POLICY.read_bytes() == b"#!/bin/sh\nexit 101\n"
+        if not (original_matches or symlink_matches or guard_matches):
+            raise DockerError("Service policy changed outside APEX; recovery requires explicit reconciliation.")
+    if backup.exists() or backup.is_symlink():
+        # Copy first: an interrupted restoration must retain its recovery copy.
+        temporary = POLICY_STATE / "restoring"
+        if temporary.exists() or temporary.is_symlink():
+            temporary.unlink()
+        shutil.copy2(backup, temporary, follow_symlinks=False)
+        os.replace(temporary, POLICY)
+    elif (POLICY_STATE / "absent").exists():
+        POLICY.unlink(missing_ok=True)
+    else:
+        raise DockerError("Service policy recovery is incomplete; inspect /usr/sbin/.apex-policy-rc.d.")
+    ready.unlink()
+    shutil.rmtree(POLICY_STATE)
+
+
+@contextmanager
+def package_policy_lock():
+    lock_path = Path("/run/lock/apex-package-policy.lock")
+    lock_path.parent.mkdir(parents=True, exist_ok=True)
+    with lock_path.open("a") as lock:
+        try:
+            fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
+        except BlockingIOError:
+            raise DockerError("Another APEX package operation is running.") from None
+        restore_service_policy()
+        yield
+
+
+@contextmanager
+def service_start_guard():
+    """Temporarily inhibit service starts while the caller holds the package lock."""
+    POLICY_STATE.mkdir(mode=0o700)
+    try:
+        if POLICY.exists() or POLICY.is_symlink():
+            shutil.copy2(POLICY, POLICY_STATE / "original", follow_symlinks=False)
+        else:
+            (POLICY_STATE / "absent").touch()
+        (POLICY_STATE / "ready").touch()
+        # Replace the path itself, preserving any original symlink in the backup.
+        temporary = POLICY_STATE / "guard"
+        temporary.write_text("#!/bin/sh\nexit 101\n")
+        temporary.chmod(0o755)
+        os.replace(temporary, POLICY)
+        yield
+    finally:
+        restore_service_policy()
+
+
+def install_missing(packages):
+    installed = package_versions()
+    missing = [package for package in packages if package not in installed]
+    if not missing:
+        return []
+    run([*APT, "update", "--error-on=any"])
+    command = [*APT, "install", "-y", "--no-install-recommends", "--no-upgrade", "--no-remove", *missing]
+    plan = run([*command, "--simulate"]).stdout
+    if any(line.startswith("Remv ") or re.match(r"^Inst \S+ \[", line)
+           for line in plan.splitlines()):
+        raise DockerError("Missing packages require an upgrade/removal; reconcile packages explicitly before retrying.")
+    # Let dpkg finish before restoring the service-start policy.
+    run(command, timeout=None)
+    remaining = set(missing) - package_versions().keys()
+    if remaining:
+        raise DockerError("Required packages not installed: " + ", ".join(sorted(remaining)))
+    return missing
+
+
+def docker_preflight(installed):
+    conflicts = sorted(set(DOCKER_CONFLICTS) & installed.keys())
+    if conflicts:
+        raise DockerError("Conflicting Docker packages require explicit reconciliation: " + ", ".join(conflicts))
+    path = Path("/etc/docker/daemon.json")
+    try:
+        config = json.loads(path.read_text()) if path.exists() else {}
+    except (OSError, ValueError) as exc:
+        raise DockerError("Cannot validate existing Docker daemon configuration.") from exc
+    if not isinstance(config, dict):
+        raise DockerError("Docker daemon configuration must be an object.")
+    for name, value in DOCKER_CONFIG.items():
+        if name in config and config[name] != value:
+            raise DockerError(f"Conflicting Docker {name}; reconcile explicitly before initialization.")
+    old_data = Path("/var/lib/docker")
+    if old_data.exists() and any(old_data.iterdir()):
+        raise DockerError("Existing /var/lib/docker data must be reconciled explicitly; no data migration is automatic.")
+    data = Path("/srv/docker")
+    if data.is_symlink() or data.parent.is_symlink():
+        raise DockerError("Refusing a redirected Docker data-root.")
+    if data.exists() and any(data.iterdir()) and config.get("data-root") != "/srv/docker":
+        raise DockerError("Unmanaged /srv/docker data requires explicit reconciliation.")
+    if "docker-ce" in installed:
+        active = run(["systemctl", "is-active", "--quiet", "docker"], check=False).returncode == 0
+        if active:
+            root = run(["docker", "info", "--format", "{{.DockerRootDir}}"] ).stdout.strip()
+            if root != "/srv/docker" or any(config.get(key) != value for key, value in DOCKER_CONFIG.items()):
+                raise DockerError("Running Docker has incompatible data-root/log policy; reconcile it before retrying.")
+    return {**config, **DOCKER_CONFIG}
+
+
+DOCKER_KEY_SHA256 = "1500c1f56fa9e26b9b8f42452a553675796ade0807cdce11975eb98170b3a570"
+LOCAL_DOCKER = ["docker", "--host", "unix:///var/run/docker.sock"]
+
+
+def _docker_source(arch):
+    key = Path("/etc/apt/keyrings/apex-docker.asc")
+    source = Path("/etc/apt/sources.list.d/apex-docker.sources")
+    content = ("Types: deb\nURIs: https://download.docker.com/linux/debian\nSuites: trixie\n"
+               f"Components: stable\nArchitectures: {arch}\nSigned-By: {key}\n")
+    for path in (key, source):
+        if any(part.is_symlink() for part in (path, *path.parents)):
+            raise DockerError(f"Refusing redirected APT path: {path}")
+    if source.exists() and source.read_text() != content:
+        raise DockerError(f"Reconcile existing Docker repository configuration: {source}")
+    if key.exists():
+        data = key.read_bytes()
+    else:
+        url = "https://download.docker.com/linux/debian/gpg"
+        with urllib.request.urlopen(url, timeout=30) as response:  # noqa: S310
+            data = response.read(1024 * 1024)
+    if hashlib.sha256(data).hexdigest() != DOCKER_KEY_SHA256:
+        raise DockerError("Docker repository signing key differs from the pinned release key.")
+    write_managed(key, data)
+    write_managed(source, content)
+
+
+def _docker_capabilities(root, *, complete_plugins=False):
+    capabilities = {"docker_data_root": root}
+    plugins = {"docker-compose-plugin": ("compose", ["compose", "version", "--short"]),
+               "docker-buildx-plugin": ("buildx", ["buildx", "version"])}
+    missing = []
+    for package, (name, args) in plugins.items():
+        result = run([*LOCAL_DOCKER, *args], check=False)
+        if result.returncode or not result.stdout.strip():
+            missing.append(package)
+        else:
+            capabilities[name] = result.stdout.strip()
+    if missing:
+        if complete_plugins and "docker-ce" in package_versions():
+            with service_start_guard():
+                install_missing(missing)
+            return _docker_capabilities(root)
+        raise DockerError("Existing Docker needs functional plugins: " + ", ".join(missing))
+    if not root:
+        raise DockerError("Unable to verify the local Docker data-root.")
+    return capabilities
+
+
+@expected_errors
+def apply_docker_policy():
+    if Path("/usr/bin/pveversion").exists():
+        raise DockerError("Docker installation on a Proxmox host requires separate administration.")
+    release = dict(line.split("=", 1) for line in Path("/etc/os-release").read_text().splitlines()
+                   if "=" in line and not line.startswith("#"))
+    arch = run(["dpkg", "--print-architecture"]).stdout.strip()
+    if (release.get("ID", "").strip('"') != "debian" or
+            release.get("VERSION_ID", "").strip('"') != "13" or arch not in ("amd64", "arm64")):
+        raise DockerError("Docker installation supports stock Debian 13 amd64 or arm64.")
+    with package_policy_lock():
+        return _apply_docker_policy(arch)
+
+
+def _apply_docker_policy(arch):
+    info = [*LOCAL_DOCKER, "info", "--format", "{{.DockerRootDir}}"]
+    if shutil.which("docker"):
+        current = run(info, check=False)
+        if current.returncode == 0:
+            return _docker_capabilities(current.stdout.strip(), complete_plugins=True)
+    installed = package_versions()
+    source = Path("/etc/apt/sources.list.d/apex-docker.sources")
+    if (not source.exists() and
+            (shutil.which("docker") or set(DOCKER_PACKAGES).intersection(installed))):
+        raise DockerError("Existing Docker installation is inactive; reconcile it before retrying.")
+    config = docker_preflight(installed)
+    with service_start_guard():
+        install_missing(("ca-certificates",))
+        _docker_source(arch)
+        write_managed("/etc/docker/daemon.json", json.dumps(config, indent=2) + "\n")
+        install_missing(DOCKER_PACKAGES)
+    ensure_service("containerd")
+    ensure_service("docker")
+    root = run(info).stdout.strip()
+    if root != DOCKER_CONFIG["data-root"]:
+        raise DockerError("Docker started with an unexpected data-root; inspect its configuration.")
+    return _docker_capabilities(root)
+
+
+def ensure_service(name):
+    if run(["systemctl", "is-enabled", "--quiet", name], check=False).returncode:
+        run(["systemctl", "enable", name])
+    if run(["systemctl", "is-active", "--quiet", name], check=False).returncode:
+        run(["systemctl", "start", name])
+
+
+if __name__ == "__main__":
+    from .state import ProvisionError, exclusive_lock
+
+    try:
+        if os.geteuid() != 0 or not Path("/etc/debian_version").is_file():
+            raise DockerError("Docker setup requires root on Debian.")
+        with exclusive_lock("/var/lib/apex-init/init.lock"):
+            print(json.dumps(apply_docker_policy(), sort_keys=True))
+    except (DockerError, OSError, ProvisionError) as exc:
+        raise SystemExit(f"Docker setup failed: {exc}") from None

scripts/provisioning/manual_bundle.py

@@ -0,0 +1,78 @@
+"""Install an immutable watchdog bundle from the operator's checkout."""
+import errno
+import hashlib
+import os
+from pathlib import Path
+import stat
+import tempfile
+
+
+MODULES = ('__init__.py', 'access.py', 'state.py')
+
+
+def _trusted_directory(path):
+    info = path.lstat()
+    if not stat.S_ISDIR(info.st_mode) or info.st_uid != os.geteuid() or info.st_mode & 0o022:
+        raise RuntimeError(f'Untrusted provisioning directory: {path}')
+
+
+def _verify(destination, files):
+    _trusted_directory(destination)
+    package = destination / 'provisioning'
+    _trusted_directory(package)
+    # Python can create bytecode caches when the installed watchdog runs.
+    for path in destination.rglob('*'):
+        info = path.lstat()
+        if (info.st_uid != os.geteuid() or info.st_mode & 0o022
+                or not (stat.S_ISDIR(info.st_mode) or stat.S_ISREG(info.st_mode))):
+            raise RuntimeError(f'Untrusted installed provisioning file: {path}')
+    for name, data in files.items():
+        target = package / name
+        if not target.is_file() or target.read_bytes() != data:
+            raise RuntimeError('Installed provisioning bundle differs from the checkout')
+
+
+def install(root=Path('/')):
+    source = Path(__file__).resolve().parent
+    files = {}
+    digest = hashlib.sha256()
+    for name in MODULES:
+        path = source / name
+        if not stat.S_ISREG(path.lstat().st_mode):
+            raise RuntimeError(f'Unsafe provisioning source file: {path}')
+        data = files[name] = path.read_bytes()
+        digest.update(name.encode() + b'\0' + len(data).to_bytes(8, 'big') + data)
+
+    parent = root
+    _trusted_directory(parent)
+    for component in ('usr', 'local', 'lib', 'apex-provisioner'):
+        parent = parent / component
+        parent.mkdir(mode=0o755, exist_ok=True)
+        _trusted_directory(parent)
+    destination = parent / digest.hexdigest()
+    if destination.exists():
+        _verify(destination, files)
+        return destination
+
+    with tempfile.TemporaryDirectory(prefix='.install-', dir=parent) as temporary:
+        stage = Path(temporary) / 'bundle'
+        package = stage / 'provisioning'
+        stage.mkdir(mode=0o755)
+        package.mkdir(mode=0o755)
+        for name, data in files.items():
+            target = package / name
+            target.write_bytes(data)
+            target.chmod(0o644)
+        try:
+            stage.rename(destination)
+        except OSError as error:
+            if error.errno not in (errno.EEXIST, errno.ENOTEMPTY):
+                raise
+            _verify(destination, files)
+    return destination
+
+
+if __name__ == '__main__':
+    if os.geteuid() != 0:
+        raise SystemExit('Run bundle installation as root.')
+    print(install())

scripts/provisioning/state.py

@@ -0,0 +1,37 @@
+"""Serialize host configuration changes across provisioning commands."""
+import contextlib
+import fcntl
+import os
+from pathlib import Path
+import stat
+
+
+class ProvisionError(RuntimeError):
+    pass
+
+
+def safe_path(path):
+    path = Path(path).absolute()
+    for part in (*reversed(path.parents), path):
+        try:
+            info = part.lstat()
+        except FileNotFoundError:
+            continue
+        if stat.S_ISLNK(info.st_mode):
+            raise ProvisionError('managed path contains a symbolic link')
+    return path
+
+
+@contextlib.contextmanager
+def exclusive_lock(path):
+    path = safe_path(path)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    fd = os.open(path, os.O_CREAT | os.O_RDWR | os.O_NOFOLLOW, 0o600)
+    try:
+        try:
+            fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
+        except BlockingIOError:
+            raise ProvisionError('another initialization is running; inspect status before retrying') from None
+        yield
+    finally:
+        os.close(fd)

tests/provisioning/test_docker_host.py

@@ -0,0 +1,141 @@
+import fcntl
+import hashlib
+import io
+import json
+from pathlib import Path
+from subprocess import CompletedProcess
+import tempfile
+import unittest
+from unittest.mock import patch
+
+from provisioning import docker
+
+
+class ManualDockerTests(unittest.TestCase):
+    def setUp(self):
+        tmp = tempfile.TemporaryDirectory()
+        self.addCleanup(tmp.cleanup)
+        self.root = Path(tmp.name).resolve()
+        (self.root / "etc").mkdir()
+        (self.root / "etc/os-release").write_text("ID=debian\nVERSION_ID=13\n")
+        self.policy = self.root / "usr/sbin/policy-rc.d"
+        self.policy.parent.mkdir(parents=True)
+        self.state = self.policy.parent / ".apex-policy-rc.d"
+        self.commands = []
+        self.arch = "arm64"
+        self.running = False
+        self.data_root = "/custom/data"
+        self.missing_buildx = False
+        self.versions = {}
+        self.installations = []
+
+    def run_command(self, cmd, **kwargs):
+        self.commands.append(cmd)
+        if cmd == ["dpkg", "--print-architecture"]:
+            return CompletedProcess(cmd, 0, self.arch + "\n", "")
+        if cmd[-1] == "{{.DockerRootDir}}":
+            return CompletedProcess(cmd, 0 if self.running else 1, self.data_root + "\n", "")
+        if cmd[-2:] in (["version", "--short"], ["buildx", "version"]):
+            if self.missing_buildx and cmd[-2:] == ["buildx", "version"]:
+                if ["docker-buildx-plugin"] not in self.installations:
+                    return CompletedProcess(cmd, 1, "", "plugin missing")
+            return CompletedProcess(cmd, 0, "installed\n", "")
+        raise AssertionError(cmd)
+
+    def apply(self):
+        def target(path):
+            path = Path(path)
+            return path if self.root in path.parents else self.root / str(path).lstrip("/")
+
+        with (patch.object(docker, "Path", side_effect=target),
+              patch.object(docker, "POLICY", self.policy),
+              patch.object(docker, "POLICY_STATE", self.state),
+              patch.object(docker, "run", side_effect=self.run_command),
+              patch.object(docker, "package_versions", return_value=self.versions),
+              patch.object(docker.shutil, "which", return_value="/usr/bin/docker" if self.running else None),
+              patch.object(docker, "install_missing", side_effect=self.installations.append)):
+            return docker.apply_docker_policy()
+
+    def test_working_engine_keeps_local_data_root_without_writes(self):
+        self.running = True
+        self.versions = {"docker.io": "1"}
+        self.assertEqual(self.apply()["docker_data_root"], "/custom/data")
+        self.assertEqual(self.installations, [])
+        self.assertFalse((self.root / "etc/docker").exists())
+
+    def test_working_engine_restores_interrupted_policy(self):
+        self.running = True
+        self.state.mkdir()
+        (self.state / "original").write_text("provider policy")
+        (self.state / "ready").touch()
+        self.policy.write_text("#!/bin/sh\nexit 101\n")
+        self.assertEqual(self.apply()["docker_data_root"], "/custom/data")
+        self.assertEqual(self.policy.read_text(), "provider policy")
+        self.assertFalse(self.state.exists())
+
+    def test_working_engine_refuses_an_active_package_operation(self):
+        self.running = True
+        lock = self.root / "run/lock/apex-package-policy.lock"
+        lock.parent.mkdir(parents=True)
+        with lock.open("a") as other:
+            fcntl.flock(other, fcntl.LOCK_EX)
+            with self.assertRaisesRegex(docker.DockerError, "Another APEX"):
+                self.apply()
+
+    def test_pve_refused_before_packages_or_daemon(self):
+        binary = self.root / "usr/bin/pveversion"
+        binary.parent.mkdir(parents=True)
+        binary.touch()
+        with self.assertRaisesRegex(docker.DockerError, "Proxmox"):
+            self.apply()
+        self.assertEqual(self.commands, [])
+
+    def test_unsupported_architecture_refused(self):
+        self.arch = "i386"
+        with self.assertRaisesRegex(docker.DockerError, "amd64|arm64"):
+            self.apply()
+        self.assertEqual(self.installations, [])
+
+    def test_existing_runtime_packages_not_replaced(self):
+        self.versions = {"containerd": "1.7"}
+        with self.assertRaisesRegex(docker.DockerError, "containerd"):
+            self.apply()
+        self.assertEqual(self.installations, [])
+
+    def test_fresh_arm64_install_verifies_the_pinned_key_and_starts_local_engine(self):
+        key = b"test repository signing key"
+        self.data_root = "/srv/docker"
+
+        def start(service):
+            if service == "docker":
+                self.running = True
+
+        with (patch.object(docker.urllib.request, "urlopen", return_value=io.BytesIO(key)),
+              patch.object(docker, "DOCKER_KEY_SHA256", hashlib.sha256(key).hexdigest()),
+              patch.object(docker, "ensure_service", side_effect=start)):
+            self.assertEqual(self.apply()["docker_data_root"], "/srv/docker")
+        source = (self.root / "etc/apt/sources.list.d/apex-docker.sources").read_text()
+        self.assertIn("Architectures: arm64", source)
+        self.assertIn("Signed-By: ", source)
+        self.assertEqual((self.root / "etc/apt/keyrings/apex-docker.asc").read_bytes(), key)
+        self.assertEqual(json.loads((self.root / "etc/docker/daemon.json").read_text())["data-root"],
+                         "/srv/docker")
+        self.assertIn("docker-compose-plugin", self.installations[-1])
+        self.assertIn("docker-buildx-plugin", self.installations[-1])
+        self.assertNotIn("ufw", self.installations[-1])
+
+    def test_untrusted_signing_key_cannot_publish_apt_source_or_install_docker(self):
+        with patch.object(docker.urllib.request, "urlopen", return_value=io.BytesIO(b"wrong key")):
+            with self.assertRaisesRegex(docker.DockerError, "signing key"):
+                self.apply()
+        self.assertFalse((self.root / "etc/apt/sources.list.d/apex-docker.sources").exists())
+        self.assertFalse((self.root / "etc/docker/daemon.json").exists())
+        self.assertEqual(self.installations, [("ca-certificates",)])
+
+    def test_working_ce_engine_only_installs_the_missing_plugin(self):
+        self.running = True
+        self.missing_buildx = True
+        self.versions = {"docker-ce": "1"}
+        self.assertEqual(self.apply()["docker_data_root"], "/custom/data")
+        self.assertEqual(self.installations, [["docker-buildx-plugin"]])
+        self.assertFalse((self.root / "etc/docker").exists())

tests/provisioning/test_docker_packages.py

@@ -0,0 +1,146 @@
+from pathlib import Path
+import subprocess
+import tempfile
+import unittest
+from unittest.mock import patch
+
+from provisioning import docker
+
+
+class Host:
+    def __init__(self, root):
+        self.root = root
+        self.commands = []
+        self.installed = {}
+        self.partial = {}
+        self.plan = ''
+        self.path('/usr/sbin').mkdir(parents=True)
+
+    def path(self, path):
+        path = Path(path)
+        if str(path).startswith(str(self.root)):
+            return path
+        return self.root / str(path).lstrip('/')
+
+    def command(self, args, **kwargs):
+        self.commands.append(args)
+        code, output = 0, ''
+        if args[:2] == ['dpkg', '--print-architecture']:
+            output = 'amd64\n'
+        elif args[0] == 'dpkg-query':
+            output = ''.join(f'{name}\tinstall ok installed\t{version}\n' for name, version in self.installed.items())
+            output += ''.join(f'{name}\t{status}\t1.0\n' for name, status in self.partial.items())
+        elif args[0] == 'apt-get' and 'install' in args:
+            if '--simulate' in args:
+                output = self.plan
+            else:
+                for name in args[args.index('--no-remove') + 1:]:
+                    self.installed[name] = '1.0'
+                    self.partial.pop(name, None)
+        return subprocess.CompletedProcess(args, code, output, '')
+
+
+class DockerPackageTests(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.temp.cleanup)
+        self.host = Host(Path(self.temp.name).resolve())
+        for item in (
+            patch.object(docker, 'Path', self.host.path),
+            patch.object(docker, 'POLICY', self.host.path('/usr/sbin/policy-rc.d')),
+            patch.object(docker, 'POLICY_STATE', self.host.path('/usr/sbin/.apex-policy-rc.d')),
+            patch.object(docker.subprocess, 'run', side_effect=self.host.command),
+        ):
+            item.start()
+            self.addCleanup(item.stop)
+
+    def test_package_database_drives_installation_and_repeat_does_not_upgrade(self):
+        self.host.installed['curl'] = '7.0'
+        self.host.partial['rsyslog'] = 'deinstall ok config-files'
+        self.assertEqual(docker.install_missing(['curl', 'rsyslog']), ['rsyslog'])
+        self.assertEqual(self.host.installed['curl'], '7.0')
+        self.assertEqual(self.host.installed['rsyslog'], '1.0')
+        count = len(self.host.commands)
+        self.assertEqual(docker.install_missing(['curl', 'rsyslog']), [])
+        self.assertFalse(any(command[0] == 'apt-get' for command in self.host.commands[count:]))
+
+    def test_dependency_upgrade_is_rejected_before_install(self):
+        self.host.plan = 'Inst libc6 [2.0] (3.0 Debian:13)\n'
+        with self.assertRaisesRegex(docker.DockerError, 'upgrade/removal'):
+            docker.install_missing(['git'])
+        self.assertNotIn('git', self.host.installed)
+
+    def test_held_installed_packages_are_preserved(self):
+        self.host.partial['git'] = 'hold ok installed'
+        self.assertEqual(docker.install_missing(['git']), [])
+        self.assertFalse(any(command[0] == 'apt-get' for command in self.host.commands))
+
+    def test_conflicting_docker_config_is_rejected_without_host_mutation(self):
+        path = self.host.path('/etc/docker/daemon.json')
+        path.parent.mkdir(parents=True)
+        path.write_text('{"data-root": "/data/docker"}')
+        before = sorted(str(path.relative_to(self.host.root)) for path in self.host.root.rglob('*'))
+        with self.assertRaisesRegex(docker.DockerError, 'Conflicting Docker'):
+            docker.docker_preflight(self.host.installed)
+        self.assertEqual(before, sorted(str(path.relative_to(self.host.root)) for path in self.host.root.rglob('*')))
+        self.assertFalse(any(command[0] in ('apt-get', 'systemctl') for command in self.host.commands))
+
+    def test_existing_docker_data_is_preserved(self):
+        data = self.host.path('/var/lib/docker/volumes/user-data')
+        data.parent.mkdir(parents=True)
+        data.write_text('valuable')
+        with self.assertRaisesRegex(docker.DockerError, 'Existing /var/lib/docker'):
+            docker.docker_preflight(self.host.installed)
+        self.assertEqual(data.read_text(), 'valuable')
+
+    def test_conflicting_distribution_docker_package_is_never_removed(self):
+        self.host.installed['docker.io'] = '26.1.5'
+        with self.assertRaisesRegex(docker.DockerError, 'Conflicting Docker packages'):
+            docker.docker_preflight(self.host.installed)
+        self.assertEqual(self.host.installed['docker.io'], '26.1.5')
+        self.assertFalse(any(command[0] == 'apt-get' for command in self.host.commands))
+
+    def test_unmanaged_destination_data_is_not_adopted(self):
+        data = self.host.path('/srv/docker/private-data')
+        data.parent.mkdir(parents=True)
+        data.write_text('preserve me')
+        with self.assertRaisesRegex(docker.DockerError, 'Unmanaged /srv/docker'):
+            docker.docker_preflight(self.host.installed)
+        self.assertEqual(data.read_text(), 'preserve me')
+
+    def test_service_policy_symlink_is_restored_after_failure(self):
+        policy = self.host.path('/usr/sbin/policy-rc.d')
+        original = self.host.path('/usr/sbin/provider-policy')
+        original.write_text('provider policy')
+        policy.symlink_to(original)
+        with self.assertRaisesRegex(RuntimeError, 'boom'):
+            with docker.package_policy_lock(), docker.service_start_guard():
+                self.assertFalse(policy.is_symlink())
+                raise RuntimeError('boom')
+        self.assertTrue(policy.is_symlink())
+        self.assertEqual(original.read_text(), 'provider policy')
+
+    def test_unfinished_backup_is_discarded_without_replacing_live_policy(self):
+        docker.POLICY_STATE.mkdir()
+        (docker.POLICY_STATE / 'original').write_text('partial')
+        docker.POLICY.write_text('complete provider policy')
+        with docker.package_policy_lock(), docker.service_start_guard():
+            pass
+        self.assertEqual(docker.POLICY.read_text(), 'complete provider policy')
+
+    def test_interrupted_guard_is_recovered_before_next_attempt(self):
+        state = self.host.path('/usr/sbin/.apex-policy-rc.d')
+        state.mkdir()
+        (state / 'original').write_text('original policy')
+        (state / 'ready').touch()
+        self.host.path('/usr/sbin/policy-rc.d').write_text('#!/bin/sh\nexit 101\n')
+        with docker.package_policy_lock(), docker.service_start_guard():
+            pass
+        self.assertEqual(self.host.path('/usr/sbin/policy-rc.d').read_text(), 'original policy')
+
+    def test_external_policy_change_is_not_overwritten(self):
+        policy = self.host.path('/usr/sbin/policy-rc.d')
+        with self.assertRaisesRegex(docker.DockerError, 'changed outside APEX'):
+            with docker.package_policy_lock(), docker.service_start_guard():
+                policy.write_text('external policy')
+        self.assertEqual(policy.read_text(), 'external policy')

tests/provisioning/test_provision_access.py

@@ -0,0 +1,539 @@
+import json
+from pathlib import Path
+import pwd
+import shutil
+import subprocess
+import tempfile
+import unittest
+from unittest.mock import patch
+
+from provisioning.access import AccessError, Host, Transition, atomic
+
+TOKEN = 'a' * 32
+
+
+class FakeHost(Host):
+    """Replace system service effects; keep snapshots and transition files real."""
+    def __init__(self, root):
+        super().__init__(root)
+        self.boot = 'first-boot'
+        self.armed = False
+        self.restored = 0
+        self.ports = [22]
+        self.bad_sudo = False
+        self.fail_final = False
+        atomic(self.path('/etc/ssh/sshd_config'), b'Port 22\n')
+
+    def boot_id(self):
+        return self.boot
+
+    def preflight(self):
+        if self.bad_sudo:
+            raise AccessError('sudo failed')
+
+    def arm(self):
+        self.armed = True
+
+    def disarm(self):
+        self.armed = False
+
+    def policy(self, ports, final, address):
+        if final and self.fail_final:
+            raise AccessError('policy failed')
+
+    def verify_active(self, ports, final, address):
+        self.policy(ports, final, address)
+        if self.ports != sorted(set(ports)):
+            raise AccessError('Live listener ports changed')
+
+    def install_policy(self, ports, final, address):
+        self.policy(ports, final, address)
+        self.ports = sorted(set(ports))
+        atomic(self.path('/etc/systemd/system/ssh.service.d/apex-init.conf'),
+               json.dumps({'ports': self.ports, 'final': final}).encode())
+
+    def restore(self, state):
+        self.restored += 1
+        self.ports = [state['original_port']]
+        path = self.path('/etc/systemd/system/ssh.service.d/apex-init.conf')
+        if state['previous'] is None:
+            path.unlink(missing_ok=True)
+        else:
+            import base64
+            atomic(path, base64.b64decode(state['previous']))
+
+
+class AccessTests(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.temp.cleanup)
+        self.host = FakeHost(Path(self.temp.name))
+        self.clock = 1000
+        self.transition = self.manual('debian')
+
+    def begin(self):
+        return self.transition.execute('begin', TOKEN, 22, '192.0.2.10')
+
+    def test_transition_keeps_original_until_final_and_commit_stops_rollback(self):
+        self.begin()
+        self.assertEqual(self.host.ports, [22, 2222])
+        self.assertTrue(self.host.armed)
+        self.transition.execute('finalize', TOKEN)
+        self.assertEqual(self.host.ports, [2222])
+        self.assertTrue(self.host.armed)
+        self.transition.execute('commit', TOKEN)
+        self.clock += 400
+        self.host.boot = 'new-boot'
+        self.transition.execute('tick')
+        self.assertEqual(self.host.ports, [2222])
+        self.assertEqual(self.host.restored, 0)
+        self.assertFalse(self.host.armed)
+
+    def test_expiry_restores_original_even_after_final_policy(self):
+        self.begin()
+        self.transition.execute('finalize', TOKEN)
+        self.clock += 301
+        self.transition.execute('tick')
+        self.assertEqual(self.host.ports, [22])
+        self.assertEqual(self.transition.load()['status'], 'rolled_back')
+
+    def test_watchdog_restores_access_while_enrollment_lock_is_held(self):
+        from provisioning.state import exclusive_lock
+        self.begin()
+        self.clock += 301
+        with exclusive_lock(self.host.path('/var/lib/apex-init/init.lock')):
+            self.transition.execute('tick')
+        self.assertEqual(self.host.ports, [22])
+        self.assertEqual(self.host.restored, 1)
+
+    def test_external_edit_prevents_finalize_and_watchdog_preserves_edit(self):
+        self.begin()
+        path = self.host.path('/etc/ssh/sshd_config')
+        path.write_text('Port 22\nBanner /etc/banner\n')
+        with self.assertRaises(AccessError):
+            self.transition.execute('finalize', TOKEN)
+        self.assertTrue(self.host.armed)
+        self.clock += 301
+        self.transition.execute('tick')
+        self.assertIn('Banner', path.read_text())
+
+    def test_invalid_sudo_never_changes_listeners(self):
+        self.host.bad_sudo = True
+        with self.assertRaises(AccessError):
+            self.begin()
+        self.assertEqual(self.host.ports, [22])
+        self.assertFalse(self.host.armed)
+
+    def test_failed_final_policy_restores_original(self):
+        self.begin()
+        self.host.fail_final = True
+        with self.assertRaises(AccessError):
+            self.transition.execute('finalize', TOKEN)
+        self.assertEqual(self.host.ports, [22])
+        self.assertFalse(self.host.armed)
+
+    def test_other_coordinator_cannot_commit(self):
+        self.begin()
+        self.transition.execute('finalize', TOKEN)
+        with self.assertRaises(AccessError):
+            self.transition.execute('commit', 'b' * 32)
+        self.assertTrue(self.host.armed)
+
+    def manual(self, profile):
+        self.host.live_arguments = lambda: (['/usr/sbin/sshd', '-D'], '755')
+        self.host.listener_ports = lambda pid: set(self.host.ports)
+        return Transition(self.host, now=lambda: self.clock, profile=profile)
+
+    def test_manual_debian_keeps_all_old_ports_then_limits_to_operator_port(self):
+        self.host.ports = [22, 2200]
+        manual = self.manual('debian')
+        manual.execute('begin', TOKEN, 2200, '192.0.2.1')
+        self.assertEqual(self.host.ports, [22, 2200, 2222])
+        boundary = manual.load()['authenticated_after_us']
+        manual.execute('finalize', TOKEN)
+        self.assertEqual(self.host.ports, [2222])
+        self.assertGreaterEqual(manual.load()['authenticated_after_us'], boundary)
+        manual.execute('commit', TOKEN)
+        self.assertEqual(manual.load()['status'], 'committed')
+
+    def test_manual_proxmox_installed_policy_never_restarts_and_preserves_image_state(self):
+        self.host.ports = [22, 2222]
+        legacy = self.host.path('/var/lib/apex-init/access/state.json')
+        manual = self.manual('proxmox')
+        with patch.object(self.host, 'install_policy', side_effect=AssertionError('restarted sshd')):
+            for round_number, status in enumerate(('committed', 'existing')):
+                record = {'status': status}
+                atomic(legacy, json.dumps(record).encode())
+                token = str(round_number) * 32
+                manual.execute('begin', token, 22, '192.0.2.1')
+                manual.execute('finalize', token)
+                manual.execute('commit', token)
+                self.assertEqual(json.loads(legacy.read_text()), record)
+        self.assertEqual(self.host.ports, [22, 2222])
+
+    def test_inflight_legacy_handover_retains_its_timer_and_state(self):
+        legacy = self.host.path('/var/lib/apex-init/access/state.json')
+        for record in ({'status': 'final'}, {'status': 'committed', 'disarm_pending': True}):
+            atomic(legacy, json.dumps(record).encode())
+            for operation in ('begin', 'tick'):
+                with self.assertRaisesRegex(AccessError, 'workflow is pending'):
+                    self.transition.execute(operation, TOKEN, 22, '192.0.2.1')
+            self.assertEqual(json.loads(legacy.read_text()), record)
+            self.assertIsNone(self.transition.load())
+
+    def test_plain_watchdog_restores_manual_override_on_reboot(self):
+        override = self.host.path('/etc/systemd/system/ssh.service.d/apex-init.conf')
+        atomic(override, b'original managed invocation')
+        manual = self.manual('proxmox')
+        manual.execute('begin', TOKEN, 22, '192.0.2.1')
+        manual.execute('finalize', TOKEN)
+        self.assertEqual(self.host.ports, [22, 2222])
+        self.host.boot = 'second-boot'
+        Transition(self.host, now=lambda: 0).execute('tick')
+        self.assertEqual(override.read_bytes(), b'original managed invocation')
+        self.assertEqual(manual.load()['status'], 'rolled_back')
+        self.assertFalse(self.host.armed)
+
+    def test_failed_disarm_is_retried_after_durable_commit_without_rollback(self):
+        self.begin()
+        self.transition.execute('finalize', TOKEN)
+        with patch.object(self.host, 'disarm', side_effect=AccessError('unavailable')), self.assertRaises(AccessError):
+            self.transition.execute('commit', TOKEN)
+        self.clock += 400
+        self.transition.execute('tick')
+        self.assertEqual(self.transition.load()['status'], 'committed')
+        self.assertFalse(self.host.armed)
+        self.assertEqual(self.host.restored, 0)
+
+    def test_reject_symlink_state_ancestor(self):
+        outside = self.host.path('/outside')
+        outside.mkdir()
+        self.host.path('/var').mkdir()
+        self.host.path('/var/lib').symlink_to(outside)
+        with self.assertRaises(AccessError):
+            self.begin()
+        self.assertEqual(list(outside.iterdir()), [])
+
+    def test_policy_rejects_additive_ports(self):
+        def runner(args, **kwargs):
+            text = 'port 22\nport 2222\n' if '-T' in args else ''
+            return subprocess.CompletedProcess(args, 0, text, '')
+        with self.assertRaises(AccessError):
+            Host(self.host.root, runner).policy([2222], True, '192.0.2.10')
+
+    def test_atomic_snapshot_does_not_follow_predictable_temporary_link(self):
+        destination = self.host.root / 'snapshot'
+        victim = self.host.root / 'victim'
+        victim.write_bytes(b'preserved')
+        destination.with_name('snapshot.new').symlink_to(victim)
+        atomic(destination, b'new snapshot')
+        self.assertEqual(destination.read_bytes(), b'new snapshot')
+        self.assertEqual(victim.read_bytes(), b'preserved')
+
+
+class LivePolicyTests(unittest.TestCase):
+    def setUp(self):
+        temporary = tempfile.TemporaryDirectory()
+        self.addCleanup(temporary.cleanup)
+        self.root = Path(temporary.name).resolve()
+        self.arguments = ['/usr/sbin/sshd', '-D', '-p', '2222']
+        self.running_arguments = self.arguments[:]
+        self.effective = ('port 2222\npermitrootlogin no\npasswordauthentication no\n'
+                          'kbdinteractiveauthentication no\npubkeyauthentication yes\n'
+                          'authenticationmethods publickey\nallowusers adam\n'
+                          'x11forwarding no\npermitemptypasswords no\n')
+        self.listeners = 'LISTEN 0 128 0.0.0.0:2222 0.0.0.0:* users:(("sshd",pid=755,fd=6))\n'
+        self.calls = []
+        self.host = Host(self.root, self.runner)
+        atomic(self.host.path('/etc/ssh/sshd_config'), b'# fixture\n')
+        process = self.root / 'proc/755'
+        process.mkdir(parents=True)
+        (process / 'exe').symlink_to('/usr/sbin/sshd')
+        self.cmdline()
+
+    def cmdline(self, title=False):
+        data = ('sshd: ' + ' '.join(self.running_arguments) + ' [listener] 0 of 10-100 startups\0'
+                if title else '\0'.join(self.running_arguments) + '\0')
+        (self.root / 'proc/755/cmdline').write_bytes(data.encode())
+
+    def runner(self, args, **kwargs):
+        self.calls.append(args)
+        if args[:2] in (['systemctl', 'is-active'], ['systemctl', 'is-enabled']) and args[-1] == 'ssh.socket':
+            return subprocess.CompletedProcess(args, 3, '', '')
+        if 'ExecStart' in args:
+            output = '{ path=/usr/sbin/sshd ; argv[]=' + ' '.join(self.arguments) + ' ; ignore_errors=no ; }'
+        elif 'MainPID' in args:
+            output = '755\n'
+        elif 'KillMode' in args:
+            output = 'process\n'
+        elif args[0] == 'ss':
+            output = self.listeners
+        elif '-T' in args:
+            output = self.effective
+        else:
+            output = ''
+        return subprocess.CompletedProcess(args, 0, output, '')
+
+    def test_verifies_exact_live_arguments_without_inventing_auth_options(self):
+        self.cmdline(title=True)
+        self.host.verify_active([2222], True, '192.0.2.1')
+        effective_call = next(args for args in self.calls if '-T' in args)
+        self.assertEqual(effective_call[:3], ['/usr/sbin/sshd', '-p', '2222'])
+        self.assertNotIn('PermitRootLogin=no', effective_call)
+
+    def test_password_or_root_enabled_on_live_daemon_is_rejected(self):
+        for setting in ('passwordauthentication', 'permitrootlogin'):
+            with self.subTest(setting=setting):
+                previous = self.effective
+                self.effective = previous.replace(setting + ' no', setting + ' yes')
+                with self.assertRaises(AccessError):
+                    self.host.verify_active([2222], True, '192.0.2.1')
+                self.effective = previous
+
+    def test_inherited_final_restrictions_cannot_silently_allow_other_accounts(self):
+        for previous, conflicting in (('allowusers adam', 'allowusers adam\nallowusers provider'),
+                                      ('x11forwarding no', 'x11forwarding yes'),
+                                      ('permitemptypasswords no', 'permitemptypasswords yes')):
+            with self.subTest(conflicting=conflicting):
+                original = self.effective
+                self.effective = original.replace(previous, conflicting)
+                with self.assertRaises(AccessError):
+                    self.host.verify_active([2222], True, '192.0.2.1')
+                self.effective = original
+
+    def test_bootstrap_prerequisites_reject_socket_and_live_invocation_drift(self):
+        self.arguments = self.running_arguments = ['/usr/sbin/sshd', '-D']
+        self.cmdline()
+        for failure in ('socket', 'invocation'):
+            with self.subTest(failure=failure):
+                def incompatible(args, failure=failure, **kwargs):
+                    if failure == 'socket' and args == ['systemctl', 'is-enabled', '--quiet', 'ssh.socket']:
+                        return subprocess.CompletedProcess(args, 0, '', '')
+                    return self.runner(args, **kwargs)
+                self.host.runner = incompatible
+                self.running_arguments = ['/usr/sbin/sshd', '-D', '-p', '22'] if failure == 'invocation' else self.arguments
+                self.cmdline()
+                with self.assertRaises(AccessError):
+                    self.host.bootstrap_preflight()
+
+    def test_later_preflight_still_checks_administrator_sudo(self):
+        self.arguments = self.running_arguments = ['/usr/sbin/sshd', '-D']
+        self.cmdline()
+        def bad_sudo(args, **kwargs):
+            if args[:3] == ['runuser', '-u', 'adam']:
+                return subprocess.CompletedProcess(args, 1, '', 'sudo denied')
+            return self.runner(args, **kwargs)
+        self.host.runner = bad_sudo
+        with self.assertRaises(AccessError):
+            self.host.preflight()
+
+    def test_bootstrap_follows_recursive_relative_and_quoted_include_paths(self):
+        self.arguments = self.running_arguments = ['/usr/sbin/sshd', '-D']
+        self.cmdline()
+        main_config = self.host.path('/etc/ssh/sshd_config')
+        first = self.host.path('/etc/ssh/sshd_config.d/10-first.conf')
+        second = self.host.path('/etc/ssh/provider # policy.conf')
+        atomic(main_config, b'Include sshd_config.*/[0-9]*.conf /etc/ssh/missing/*.conf\n')
+        atomic(first, b'iNcLuDe = "provider # policy.conf" # relative to /etc/ssh\n')
+        atomic(second, b'# Match Address 203.0.113.0/24\nBanner "/etc/Match banner"\n')
+        self.host.bootstrap_preflight()
+        second.write_text('"mAtCh" Address 203.0.113.0/24\n PasswordAuthentication yes\n')
+        before = {path: path.read_bytes() for path in (main_config, first, second)}
+        with self.assertRaisesRegex(AccessError, 'Match.*manual reconciliation'):
+            self.host.bootstrap_preflight()
+        self.assertEqual({path: path.read_bytes() for path in before}, before)
+
+    def test_bootstrap_rejects_recursive_or_oversized_config_without_mutation(self):
+        self.arguments = self.running_arguments = ['/usr/sbin/sshd', '-D']
+        self.cmdline()
+        configuration = self.host.path('/etc/ssh/sshd_config')
+        for content in ('Include sshd_config\n', '#' + 'x' * (1024 * 1024 + 1)):
+            with self.subTest(recursive=content.startswith('Include')):
+                configuration.write_text(content)
+                with self.assertRaisesRegex(AccessError, 'manual reconciliation'):
+                    self.host.bootstrap_preflight()
+                self.assertEqual(configuration.read_text(), content)
+
+    def test_extended_include_glob_tokens_are_rejected_even_without_matches(self):
+        self.arguments = self.running_arguments = ['/usr/sbin/sshd', '-D']
+        self.cmdline()
+        configuration = self.host.path('/etc/ssh/sshd_config')
+        for pattern in ('provider[a[:digit:]].conf', 'provider[a[.x.]].conf',
+                        'provider[a[=x=]].conf', 'provider[:digit:].conf'):
+            with self.subTest(pattern=pattern):
+                content = 'Include ' + pattern + '\n'
+                configuration.write_text(content)
+                with self.assertRaisesRegex(AccessError, 'glob syntax.*manual reconciliation'):
+                    self.host.bootstrap_preflight()
+                self.assertEqual(configuration.read_text(), content)
+
+    def test_unrestarted_daemon_with_old_arguments_is_rejected(self):
+        self.running_arguments = ['/usr/sbin/sshd', '-D', '-p', '22']
+        self.cmdline()
+        with self.assertRaises(AccessError):
+            self.host.verify_active([2222], True, '192.0.2.1')
+
+    def test_extra_live_listener_port_is_rejected(self):
+        self.listeners += 'LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=755,fd=7))\n'
+        with self.assertRaises(AccessError):
+            self.host.verify_active([2222], True, '192.0.2.1')
+
+    def test_additional_sshd_service_listener_is_rejected(self):
+        self.listeners += 'LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=999,fd=7))\n'
+        with self.assertRaises(AccessError):
+            self.host.verify_active([2222], True, '192.0.2.1')
+
+    def test_runtime_override_changes_fingerprint(self):
+        before = self.host.fingerprint()
+        atomic(self.host.path('/run/systemd/system/ssh.service.d/override.conf'), b'[Service]\nExecStart=/usr/sbin/sshd -D\n')
+        self.assertNotEqual(before, self.host.fingerprint())
+
+    def test_managed_file_does_not_exempt_drifted_service_from_preflight(self):
+        atomic(self.host.path('/etc/systemd/system/ssh.service.d/apex-init.conf'),
+               b'[Service]\nExecStart=\nExecStart=/usr/sbin/sshd -p 2222 -o PermitRootLogin=no -D\n')
+        with self.assertRaisesRegex(AccessError, 'differs from its managed invocation'):
+            self.host.preflight()
+
+
+@unittest.skipUnless(shutil.which('sshd') and shutil.which('ssh-keygen'), 'OpenSSH tools unavailable')
+class OpenSSHPolicyTests(unittest.TestCase):
+    """Evaluate the managed options with a real daemon without opening sockets."""
+    def setUp(self):
+        temporary = tempfile.TemporaryDirectory()
+        self.addCleanup(temporary.cleanup)
+        self.root = Path(temporary.name)
+        self.key = self.root / 'host-key'
+        self.config = self.root / 'etc/ssh/sshd_config'
+        self.config.parent.mkdir(parents=True)
+        self.config.write_text('PasswordAuthentication yes\nPermitRootLogin yes\n'
+                               'X11Forwarding yes\nPermitEmptyPasswords yes\n')
+        subprocess.run(['ssh-keygen', '-q', '-t', 'ed25519', '-N', '', '-f', str(self.key)],
+                       check=True, capture_output=True, timeout=30)
+        self.host = Host(self.root, self.runner)
+        original_path = self.host.path
+        self.host.path = lambda path: Path(path) if Path(path).is_relative_to(self.root) else original_path(path)
+
+    def runner(self, args, **kwargs):
+        if args[0] != '/usr/sbin/sshd':
+            code = 3 if args[-1] == 'ssh.socket' else 0
+            output = 'process\n' if 'KillMode' in args else ''
+            return subprocess.CompletedProcess(args, code, output, '')
+        return subprocess.run([shutil.which('sshd'), '-f', str(self.config), '-h', str(self.key), *args[1:]],
+                              **kwargs)
+
+    def effective(self, arguments, user):
+        result = self.runner(arguments + ['-T', '-C', 'user=' + user + ',host=localhost,addr=192.0.2.1'],
+                             capture_output=True, text=True, timeout=30)
+        self.assertEqual(result.returncode, 0, result.stderr)
+        return result.stdout.splitlines()
+
+    def test_transition_preserves_provider_access_and_only_final_policy_excludes_it(self):
+        transition = self.host.policy([22, 2222], False, '192.0.2.1')
+        transitional = self.effective(transition, 'provider')
+        self.assertFalse(any(line.startswith('allowusers ') for line in transitional))
+        self.assertIn('passwordauthentication yes', transitional)
+        final = self.host.policy([2222], True, '192.0.2.1')
+        for user in ('adam', 'provider'):
+            with self.subTest(user=user):
+                effective = self.effective(final, user)
+                allowed = [line.removeprefix('allowusers ') for line in effective if line.startswith('allowusers ')]
+                self.assertEqual(allowed, ['adam'])
+                self.assertNotIn('provider', allowed)
+                self.assertIn('x11forwarding no', effective)
+                self.assertIn('permitemptypasswords no', effective)
+
+    def test_proxmox_effective_root_key_auth_and_both_ports_with_legacy_operator_block(self):
+        self.host.profile = 'proxmox'
+        with self.config.open('a') as stream:
+            stream.write('# APEX operator SSH policy\nMatch User adam\n'
+                         '  AuthenticationMethods publickey\n  PubkeyAuthentication yes\n'
+                         '  PasswordAuthentication no\n  KbdInteractiveAuthentication no\n'
+                         '  PermitEmptyPasswords no\n  X11Forwarding no\n')
+        final = self.host.policy([22, 2222], True, '192.0.2.1')
+        for user in ('root', 'adam'):
+            effective = self.effective(final, user)
+            self.assertEqual({item for item in effective if item.startswith('port ')}, {'port 22', 'port 2222'})
+            self.assertTrue({'permitrootlogin without-password', 'permitrootlogin prohibit-password'}.intersection(effective))
+            self.assertIn('authenticationmethods publickey', effective)
+            self.assertFalse(any(item.startswith('allowusers ') for item in effective))
+        with self.config.open('a') as stream:
+            stream.write('  PasswordAuthentication yes\n')
+        with self.assertRaisesRegex(AccessError, 'Match'):
+            self.host.policy([22, 2222], True, '192.0.2.1')
+
+    def test_adding_probe_port_preserves_authentication_from_the_existing_service(self):
+        self.host.profile = 'debian'
+        args = ['/usr/sbin/sshd', '-p', '2200', '-o', 'PermitRootLogin=no', '-o', 'PasswordAuthentication=no', '-D']
+        with patch.object(self.host, 'service_arguments', return_value=args):
+            transition = self.host.policy([2200, 2222], False, '192.0.2.1')
+        effective = self.effective(transition, 'root')
+        self.assertIn('permitrootlogin no', effective)
+        self.assertIn('passwordauthentication no', effective)
+        self.assertIn('port 2200', effective)
+
+    def test_additive_inherited_allowusers_stops_finalization(self):
+        with self.config.open('a') as stream:
+            stream.write('AllowUsers provider\n')
+        self.host.policy([22, 2222], False, '192.0.2.1')
+        with self.assertRaises(AccessError):
+            self.host.policy([2222], True, '192.0.2.1')
+
+    def test_provider_account_match_block_cannot_bypass_final_restrictions(self):
+        provider = pwd.struct_passwd(('provider', 'x', 1001, 1001, '', '/home/provider', '/bin/bash'))
+        for setting in ('AllowUsers provider', 'X11Forwarding yes', 'PermitEmptyPasswords yes'):
+            with self.subTest(setting=setting):
+                self.config.write_text('Match User provider\n ' + setting + '\n')
+                with patch('pwd.getpwall', return_value=[provider]), self.assertRaises(AccessError):
+                    self.host.policy([2222], True, '192.0.2.1')
+
+    def test_bootstrap_rejects_match_address_in_main_or_recursively_included_file(self):
+        conditional = ('\n AllowUsers root provider\n PasswordAuthentication yes\n'
+                       ' PermitRootLogin yes\n AuthenticationMethods any\n'
+                       ' X11Forwarding yes\n PermitEmptyPasswords yes\n')
+        first = self.root / 'first # include.conf'
+        second = self.root / 'second include.conf'
+        for directive, included in (('Match Address', False), ('mAtCh=Address', False),
+                                    ('"Match" Address', False), ('"mAtCh" Address', True)):
+            with self.subTest(directive=directive, included=included):
+                payload = directive + ' 203.0.113.0/24' + conditional
+                if included:
+                    second.write_text(payload)
+                    first.write_text('iNcLuDe "' + str(second) + '"\n')
+                    self.config.write_text('"Include" "' + str(first) + '" # provider policy\n')
+                else:
+                    self.config.write_text(payload)
+                arguments = ['/usr/sbin/sshd', '-T', '-p', '2222', '-o', 'AllowUsers=adam',
+                             '-o', 'PermitRootLogin=no', '-o', 'PasswordAuthentication=no',
+                             '-o', 'AuthenticationMethods=publickey', '-o', 'X11Forwarding=no',
+                             '-o', 'PermitEmptyPasswords=no']
+                other_address = self.runner(arguments + ['-C', 'user=root,host=localhost,addr=203.0.113.25'],
+                                            capture_output=True, text=True, timeout=30)
+                self.assertEqual(other_address.returncode, 0, other_address.stderr)
+                self.assertIn('permitrootlogin yes', other_address.stdout.splitlines())
+                self.assertIn('passwordauthentication yes', other_address.stdout.splitlines())
+                before = self.config.read_bytes()
+                with patch.object(self.host, 'service_arguments', return_value=['/usr/sbin/sshd', '-D']), \
+                        patch.object(self.host, 'live_arguments'), \
+                        self.assertRaisesRegex(AccessError, 'Match.*manual reconciliation'):
+                    self.host.bootstrap_preflight()
+                self.assertEqual(self.config.read_bytes(), before)
+
+    def test_posix_class_include_cannot_hide_a_match_from_preflight(self):
+        included = self.root / 'provider1.conf'
+        included.write_text('Match Address 203.0.113.0/24\n PermitRootLogin yes\n')
+        for pattern in ('provider[[:digit:]].conf', 'provider[a[:digit:]].conf'):
+            with self.subTest(pattern=pattern):
+                self.config.write_text('Include ' + str(self.root / pattern) + '\n')
+                effective = self.runner(['/usr/sbin/sshd', '-T', '-o', 'PermitRootLogin=no',
+                                         '-C', 'user=root,host=localhost,addr=203.0.113.25'],
+                                        capture_output=True, text=True, timeout=30)
+                self.assertEqual(effective.returncode, 0, effective.stderr)
+                self.assertIn('permitrootlogin yes', effective.stdout.splitlines())
+                with patch.object(self.host, 'service_arguments', return_value=['/usr/sbin/sshd', '-D']), \
+                        patch.object(self.host, 'live_arguments'), \
+                        self.assertRaisesRegex(AccessError, 'manual reconciliation'):
+                    self.host.bootstrap_preflight()
+
+
+if __name__ == '__main__':
+    unittest.main()

tests/run.sh

@@ -1,5 +1,7 @@
 #!/usr/bin/env bash
-# Run the full engine test suite (stdlib unittest — no pytest).
+# Run CLI, bootstrap, and host provisioning checks with the project modules on the import path.
 set -euo pipefail
 cd "$(dirname "$0")/.."
-exec python3 -m unittest discover -s tests -p 'test_*.py' -v
+export PYTHONPATH="$PWD/scripts${PYTHONPATH:+:$PYTHONPATH}"
+python3 -m unittest discover -s tests -p 'test_*.py' -v
+python3 -m unittest discover -s tests/provisioning -p 'test_*.py' -v

tests/test_backup_run.py

@@ -0,0 +1,49 @@
+import json
+from pathlib import Path
+from types import SimpleNamespace
+import unittest
+import tempfile
+from unittest.mock import Mock
+
+from engine import overlay
+
+backup = overlay.load_module(str(Path(__file__).resolve().parents[1] / 'actions/tier/backup.py'),
+                             'test_backup_action')
+
+
+class BackupRecovery(unittest.TestCase):
+    def test_failed_attempt_preserves_last_success_and_records_new_failure(self):
+        with tempfile.TemporaryDirectory() as directory:
+            path = Path(directory) / 'status.json'
+            backup.write_backup_status(path, 'success', 100)
+            backup.write_backup_status(path, 'running', 200)
+            backup.write_backup_status(path, 'failed', 210)
+            status = json.loads(path.read_text())
+            self.assertEqual(status['last_success'], 100)
+            self.assertEqual(status['state'], 'failed')
+            self.assertEqual(status['updated_at'], 210)
+
+    def test_recovers_namespace_owner_before_dependent_without_starting_other_containers(self):
+        running = {'komf': False, 'komga': False, 'intentionally-stopped': False}
+        def docker(command, **kwargs):
+            if command[1] == 'inspect':
+                return SimpleNamespace(returncode=0, stdout=json.dumps([
+                    {'Id': name, 'State': {'Running': running[name]}} for name in command[2:]]))
+            name = command[2]
+            if name == 'komf' and not running['komga']:
+                return SimpleNamespace(returncode=1)
+            running[name] = True
+            return SimpleNamespace(returncode=0)
+        ctx = SimpleNamespace(sys=SimpleNamespace(run=docker), log=Mock())
+        backup.restore_containers(ctx, ['komf', 'komga'])
+        self.assertTrue(running['komf'])
+        self.assertTrue(running['komga'])
+        self.assertFalse(running['intentionally-stopped'])
+
+    def test_unrecoverable_container_prevents_success(self):
+        def docker(command, **kwargs):
+            return SimpleNamespace(returncode=0 if command[1] == 'inspect' else 1,
+                                   stdout='[{"Id":"broken","State":{"Running":false}}]')
+        ctx = SimpleNamespace(sys=SimpleNamespace(run=docker), log=Mock())
+        with self.assertRaisesRegex(RuntimeError, 'broken'):
+            backup.restore_containers(ctx, ['broken'])

tests/test_bootstrap_client.py

@@ -0,0 +1,61 @@
+import json
+import os
+import shutil
+import subprocess
+import tempfile
+import unittest
+from pathlib import Path
+from unittest import mock
+
+
+class BootstrapClientTest(unittest.TestCase):
+    def setUp(self):
+        temporary = tempfile.TemporaryDirectory()
+        self.addCleanup(temporary.cleanup)
+        self.root = Path(temporary.name)
+        source = Path(__file__).resolve().parents[1] / "scripts/bootstrap-node"
+        self.launcher = self.root / "bootstrap-node"
+        shutil.copy(source, self.launcher)
+        (self.root / "bootstrap-node-remote.py").write_text("pass\n")
+        for name in ("operator", "deploy"):
+            subprocess.run(["ssh-keygen", "-qt", "ed25519", "-N", "", "-f", str(self.root / name)],
+                           check=True)
+        self.public = (self.root / "operator.pub").read_text()
+        (self.root / "known").write_text("[git.example]:2222 " + self.public)
+        ssh = self.root / "ssh"
+        ssh.write_text('#!/bin/sh\nprintf "%s\\n" "$@" > args\ncat > payload\n')
+        ssh.chmod(0o700)
+
+    def launch(self, *extra):
+        with mock.patch.dict(os.environ, PATH=str(self.root) + os.pathsep + os.environ["PATH"]):
+            return subprocess.run(
+                [str(self.launcher), "--target", "admin@node", "--repo",
+                 "ssh://git@git.example:2222/fleet/example.git", "--operator-key", "operator.pub",
+                 "--deploy-key", "deploy", "--known-hosts", "known", *extra],
+                cwd=self.root, capture_output=True, text=True,
+            )
+
+    def test_custom_git_port_and_secrets_delivered_via_stdin(self):
+        run = self.launch()
+        self.assertEqual(run.returncode, 0, run.stderr)
+        payload = json.loads((self.root / "payload").read_text())
+        self.assertEqual(payload["node_name"], "example")
+        secret = (self.root / "deploy").read_text()
+        self.assertEqual(payload["deploy_key"], secret)
+        self.assertNotIn(secret.strip().splitlines()[1], (self.root / "args").read_text()
+                         + run.stdout + run.stderr)
+
+    def test_git_host_pin_for_default_port_rejected(self):
+        (self.root / "known").write_text("git.example " + self.public)
+        self.assertNotEqual(self.launch().returncode, 0)
+        self.assertFalse((self.root / "payload").exists())
+
+    def test_reused_operator_key_rejected(self):
+        self.assertNotEqual(self.launch("--deploy-key", "operator").returncode, 0)
+        self.assertFalse((self.root / "payload").exists())
+
+    def test_encrypted_deploy_key_rejected(self):
+        subprocess.run(["ssh-keygen", "-pq", "-P", "", "-N", "test", "-f",
+                        str(self.root / "deploy")], check=True, capture_output=True)
+        self.assertNotEqual(self.launch().returncode, 0)
+        self.assertFalse((self.root / "payload").exists())

tests/test_bootstrap_remote.py

@@ -0,0 +1,164 @@
+"""Remote bootstrap checks using disposable repositories, keys, and home directories."""
+import importlib.util
+import os
+from pathlib import Path
+import subprocess
+import tempfile
+import unittest
+from types import SimpleNamespace
+from unittest.mock import Mock, patch
+
+SCRIPT = Path(__file__).resolve().parents[1] / "scripts/bootstrap-node-remote.py"
+loader = importlib.util.spec_from_file_location("bootstrap_remote", SCRIPT)
+remote = importlib.util.module_from_spec(loader)
+loader.loader.exec_module(remote)
+
+
+def git(*args):
+    return subprocess.run(["git", "-c", "protocol.file.allow=always", *map(str, args)],
+                          text=True, capture_output=True, check=True).stdout
+
+
+class OperatorAccount(unittest.TestCase):
+    def test_new_debian_account_is_ready_for_sudo_and_later_docker_install(self):
+        with tempfile.TemporaryDirectory() as temp:
+            home = Path(temp).resolve() / "adam"
+            account = SimpleNamespace(pw_uid=1001, pw_gid=1001, pw_dir=str(home))
+            commands = []
+
+            def provision(command):
+                commands.append(command)
+                if command[0] == "useradd":
+                    home.mkdir(mode=0o700)
+
+            actual_stat = Path.stat
+
+            def stat(path, *args, **kwargs):
+                info = actual_stat(path, *args, **kwargs)
+                if path != home:
+                    return info
+                fields = list(info)
+                fields[4] = account.pw_uid
+                return os.stat_result(fields)
+
+            with (patch.object(remote, "HOME", home),
+                  patch.object(Path, "stat", stat),
+                  patch.object(Path, "exists", return_value=False),
+                  patch.object(remote.pwd, "getpwnam", side_effect=[KeyError, account]),
+                  patch.object(remote, "checked", side_effect=provision)):
+                self.assertEqual(remote.prepare_account(), account)
+            self.assertEqual(sum(command[0] == "useradd" for command in commands), 1)
+            self.assertIn("sudo,docker", commands[-1])
+
+    def test_existing_system_account_cannot_be_promoted(self):
+        command = Mock()
+        account = SimpleNamespace(pw_uid=998, pw_gid=998, pw_dir="/home/adam")
+        with (patch.object(remote.pwd, "getpwnam", return_value=account),
+              patch.object(remote, "checked", command)):
+            with self.assertRaisesRegex(RuntimeError, "account or home"):
+                remote.prepare_account()
+        command.assert_not_called()
+
+
+class Credentials(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.temp.cleanup)
+        self.root = Path(self.temp.name).resolve()
+        self.home = self.root / "adam"
+        self.home.mkdir()
+        for name in ("operator", "deploy"):
+            subprocess.run(["ssh-keygen", "-q", "-N", "", "-t", "ed25519", "-f",
+                            str(self.root / name)], check=True)
+        self.spec = dict(operator_key=(self.root / "operator.pub").read_text(),
+                         deploy_key=(self.root / "deploy").read_text(), node_name="web",
+                         known_hosts="[example.test]:2222 " + (self.root / "deploy.pub").read_text())
+        self.authorized = self.home / ".ssh/authorized_keys"
+
+    def test_credentials_validated_against_port_and_operator_identity(self):
+        remote.validate_credentials(self.spec, "[example.test]:2222")
+        with self.assertRaises(RuntimeError):
+            remote.validate_credentials(self.spec, "example.test")
+        self.spec["deploy_key"] = (self.root / "operator").read_text()
+        with self.assertRaisesRegex(ValueError, "separate"):
+            remote.validate_credentials(self.spec, "[example.test]:2222")
+
+    def test_repeat_and_second_checkout_preserve_operator_keys_and_git_identities(self):
+        self.authorized.parent.mkdir()
+        self.authorized.write_text((self.root / "deploy.pub").read_text())
+        remote.configure_ssh(self.home, self.spec, "example.test")
+        first_config = (self.home / ".gitconfig").read_bytes()
+        remote.configure_ssh(self.home, self.spec, "example.test")
+        self.assertEqual((self.home / ".gitconfig").read_bytes(), first_config)
+        self.assertEqual(self.authorized.read_text().splitlines(), [
+            (self.root / "deploy.pub").read_text().strip(), self.spec["operator_key"].strip()])
+        remote.configure_ssh(self.home, dict(self.spec, node_name="db", deploy_key="second"), "db.test")
+        self.assertEqual((self.authorized.parent / "apex-web.key").read_text(), self.spec["deploy_key"])
+        work = self.root / "checkout"
+        git("init", "-q", work)
+        config = self.home / ".gitconfig"
+        config.write_text(config.read_text().replace("/srv/web", str(work)))
+        env = dict(os.environ, HOME=str(self.home), GIT_CONFIG_NOSYSTEM="1")
+        self.assertIn("apex-web.sshconf", subprocess.check_output(
+            ["git", "-C", str(work), "config", "core.sshCommand"], text=True, env=env))
+        self.assertEqual(subprocess.run(["git", "config", "--get", "core.sshCommand"],
+                         cwd=self.root, env=env, capture_output=True).returncode, 1)
+
+    def test_restricted_key_and_redirected_authorization_are_preserved(self):
+        self.authorized.parent.mkdir()
+        restriction = 'from="192.0.2.1" ' + self.spec["operator_key"]
+        self.authorized.write_text(restriction)
+        with self.assertRaisesRegex(ValueError, "restrictions"):
+            remote.configure_ssh(self.home, self.spec, "example.test")
+        self.assertEqual(self.authorized.read_text(), restriction)
+        victim = self.root / "preserved"
+        self.authorized.rename(victim)
+        for link in (lambda: self.authorized.symlink_to(victim),
+                     lambda: os.link(victim, self.authorized)):
+            link()
+            with self.assertRaisesRegex(ValueError, "Unsafe file"):
+                remote.configure_ssh(self.home, self.spec, "example.test")
+            self.assertEqual(victim.read_text(), restriction)
+            self.authorized.unlink()
+
+
+class Checkout(unittest.TestCase):
+    def test_failed_clone_retries_recursively_and_preserves_local_work(self):
+        with tempfile.TemporaryDirectory() as temp:
+            root = Path(temp).resolve()
+            srv = root / "srv"
+            srv.mkdir()
+            source = root / "repo"
+            account = SimpleNamespace(pw_uid=os.getuid(), pw_gid=os.getgid())
+            with self.assertRaises(subprocess.CalledProcessError):
+                remote.checkout(srv, "node", str(source), account, git)
+            self.assertEqual(list(srv.iterdir()), [])
+            for repo in (source, root / "common", root / "leaf"):
+                git("init", "-q", repo)
+                (repo / "sample").write_text("initial")
+                git("-C", repo, "add", "sample")
+                git("-C", repo, "-c", "user.name=T", "-c", "user.email=t@test.example",
+                    "commit", "-qm", "initial")
+            git("-C", root / "common", "submodule", "add", str(root / "leaf"), "nested")
+            git("-C", root / "common", "-c", "user.name=T", "-c", "user.email=t@test.example",
+                "commit", "-qm", "pin nested submodule")
+            git("-C", source, "submodule", "add", str(root / "common"), "commons")
+            git("-C", source, "-c", "user.name=T", "-c", "user.email=t@test.example",
+                "commit", "-qm", "add submodule")
+            remote.checkout(srv, "node", str(source), account, git)
+            self.assertEqual((srv / "node/commons/sample").read_text(), "initial")
+            dirty = srv / "node/sample"
+            dirty.write_text("local edits")
+            remote.checkout(srv, "node", str(source), account, git)
+            with self.assertRaisesRegex(RuntimeError, "does not match"):
+                remote.checkout(srv, "node", "another-origin", account, git)
+            self.assertEqual(dirty.read_text(), "local edits")
+            nested = srv / "node/commons/nested"
+            git("-C", nested, "-c", "user.name=T", "-c", "user.email=t@test.example",
+                "commit", "--allow-empty", "-qm", "local commit")
+            local_head = git("-C", nested, "rev-parse", "HEAD")
+            (nested / "sample").write_text("nested edits")
+            with self.assertRaisesRegex(RuntimeError, "commons/nested"):
+                remote.checkout(srv, "node", str(source), account, git)
+            self.assertEqual(git("-C", nested, "rev-parse", "HEAD"), local_head)
+            self.assertEqual((nested / "sample").read_text(), "nested edits")

tests/test_cli.py

@@ -13,6 +13,22 @@ class TestCli(unittest.TestCase):
         self.assertEqual(r.returncode, 0)
         self.assertIn("Available actions", r.stdout)
 
+    def test_compose_alias_uses_canonical_action(self):
+        canonical = run("composition/run", "--help")
+        self.assertEqual(canonical.returncode, 0)
+        alias = run("compose", "--help")
+        self.assertEqual(alias.returncode, 0)
+        self.assertEqual(alias.stdout, canonical.stdout)
+
+    def test_configuration_must_be_selected_explicitly(self):
+        listing = run("--help")
+        self.assertIn("config/hardening", listing.stdout)
+        self.assertIn("repository/chown", listing.stdout)
+        for command in ("configure", "config"):
+            result = run(command)
+            self.assertEqual(result.returncode, 1)
+            self.assertIn("Unknown action", result.stderr)
+
     def test_unknown_action_exits_one(self):
         r = run("no/such")
         self.assertEqual(r.returncode, 1)

tests/test_compose.py

@@ -0,0 +1,123 @@
+import contextlib
+import io
+import os
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import MagicMock
+
+from actions.composition import run as compose
+from engine.descriptor import build_parser
+
+
+class TestCompose(unittest.TestCase):
+    def setUp(self):
+        self.tmpdir = tempfile.TemporaryDirectory()
+        self.addCleanup(self.tmpdir.cleanup)
+        self.comp_dir = Path(self.tmpdir.name) / 'compositions'
+        for name in ('apex', 'sso', 'arr'):
+            directory = self.comp_dir / name
+            directory.mkdir(parents=True)
+            (directory / 'docker-compose.yml').write_text(f'name: {name}\n')
+        self.ctx = MagicMock()
+        self.ctx.paths.compositions = str(self.comp_dir)
+        self.ctx.paths.stier3 = self.tmpdir.name
+        self.ctx.vars.return_value = {'APEX_NODE_HOST': 'test-host', 'OTHER': 'ignored'}
+
+    def run_compose(self, *argv):
+        args = build_parser(compose.METADATA, 'apex composition/run').parse_args(argv)
+        compose.run(self.ctx, args)
+
+    def visited(self):
+        return [os.path.basename(call.kwargs['cwd'])
+                for call in self.ctx.sys.run.call_args_list]
+
+    def test_sequence_all_and_down_reverse(self):
+        for argv, expected in [(('restart',), ['apex', 'arr', 'sso']),
+                               (('-p', 'all', 'up'), ['apex', 'arr', 'sso']),
+                               (('down', 'all'), ['sso', 'arr', 'apex'])]:
+            with self.subTest(argv=argv):
+                self.ctx.sys.run.reset_mock()
+                self.run_compose(*argv)
+                self.assertEqual(self.visited(), expected)
+
+    def test_explicit_target_preserves_service_named_like_composition(self):
+        self.run_compose('-p', 'apex', 'restart', 'sso')
+        self.assertEqual(self.visited(), ['apex'])
+        call = self.ctx.sys.run.call_args
+        self.assertEqual(call.args[0], ['docker', 'compose', 'restart', 'sso'])
+        self.assertEqual(call.kwargs['env']['APEX_NODE_HOST'], 'test-host')
+        self.assertTrue(call.kwargs['check'])
+
+    def test_immediate_positional_composition_compatibility(self):
+        self.run_compose('restart', 'apex', 'traefik')
+        self.assertEqual(self.visited(), ['apex'])
+        self.assertEqual(self.ctx.sys.run.call_args.args[0],
+                         ['docker', 'compose', 'restart', 'traefik'])
+
+    def test_up_detached_compatibility_and_default_detached(self):
+        for argv in [('up', '-d', 'apex'), ('-p', 'apex', 'up'),
+                     ('up', 'apex', '-d')]:
+            with self.subTest(argv=argv):
+                self.ctx.sys.run.reset_mock()
+                self.run_compose(*argv)
+                self.assertEqual(self.visited(), ['apex'])
+                self.assertEqual(self.ctx.sys.run.call_args.args[0],
+                                 ['docker', 'compose', 'up', '-d'])
+
+    def test_flag_values_and_unknown_services_do_not_retarget(self):
+        for argv in [('logs', '--tail', 'sso'), ('restart', 'traefik'),
+                     ('up', '--scale', 'sso'), ('logs', '--', 'apex')]:
+            with self.subTest(argv=argv):
+                self.ctx.sys.run.reset_mock()
+                self.run_compose(*argv)
+                self.assertEqual(self.visited(), ['apex', 'arr', 'sso'])
+                prefix = ['docker', 'compose', argv[0]]
+                if argv[0] == 'up':
+                    prefix += ['-d']
+                self.assertEqual(self.ctx.sys.run.call_args.args[0], prefix + list(argv[1:]))
+
+    def test_invalid_wrapper_options_fail_before_execution(self):
+        for argv, code in [(('-p', 'unknown', 'up'), 1), (('-p', '', 'up'), 1),
+                           (('-p',), 64),
+                           (('up', '-p', 'apex'), 64),
+                           (('up', '--dry-run'), 64), ((), 64),
+                           (('up', '-d', 'apex', 'traefik'), 64)]:
+            with self.subTest(argv=argv), contextlib.redirect_stdout(io.StringIO()), \
+                    contextlib.redirect_stderr(io.StringIO()):
+                with self.assertRaises(SystemExit) as error:
+                    self.run_compose(*argv)
+                self.assertEqual(error.exception.code, code)
+                self.ctx.sys.run.assert_not_called()
+
+    def test_separator_preserves_wrapper_looking_passthrough_tokens(self):
+        self.run_compose('-p', 'apex', 'exec', '--', 'probe', '--dry-run', '-p', 'sso')
+        self.assertEqual(self.ctx.sys.run.call_args.args[0],
+                         ['docker', 'compose', 'exec', '--', 'probe', '--dry-run', '-p', 'sso'])
+
+    def test_env_files_layer_before_action(self):
+        for filename in ('.env', 'apex.env'):
+            (self.comp_dir / 'apex' / filename).write_text('PORT=1234\n')
+        self.run_compose('-p', 'apex', 'config')
+        self.assertEqual(self.ctx.sys.run.call_args.args[0],
+                         ['docker', 'compose', '--env-file', '.env', '--env-file',
+                          'apex.env', 'config'])
+
+    def test_socket_survives_restart_up_and_dry_run(self):
+        shm = Path(self.tmpdir.name) / 'apex' / 'xray' / 'shm'
+        shm.mkdir(parents=True)
+        path = shm / 'live.sock'
+        path.write_bytes(b"socket path must survive")
+        for argv in [('restart', 'apex'), ('-p', 'apex', 'restart', 'traefik'),
+                     ('up',), ('--dry-run', '-p', 'apex', 'logs', '--tail', '20', 'xray')]:
+            with self.subTest(argv=argv):
+                self.ctx.sys.run.reset_mock()
+                self.run_compose(*argv)
+                self.assertTrue(path.exists())
+                if argv[0] == '--dry-run':
+                    self.ctx.sys.run.assert_not_called()
+                    self.assertIn('logs --tail 20 xray', self.ctx.log.info.call_args.args[0])
+
+
+if __name__ == '__main__':
+    unittest.main()

tests/test_compose_hook.py

@@ -0,0 +1,37 @@
+import importlib.util
+from pathlib import Path
+import subprocess
+import tempfile
+from types import SimpleNamespace
+import unittest
+from unittest.mock import Mock, patch
+from engine.lib.sys import Sys
+
+spec = importlib.util.spec_from_file_location('compose_lint', Path(__file__).resolve().parents[1] / 'actions/util/lint-docker-compose.py')
+lint = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(lint)
+
+
+class HookTests(unittest.TestCase):
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.tmp.cleanup)
+        self.repo = Path(self.tmp.name)
+        subprocess.run(['git', 'init', '-q', str(self.repo)], check=True)
+        self.log = Mock()
+        self.ctx = SimpleNamespace(log=self.log, sys=Sys(self.log), node=None,
+                                   paths=SimpleNamespace(repo_root=str(self.repo), compositions=str(self.repo / 'compositions')))
+
+    def stage(self, name, content):
+        (self.repo / name).write_text(content)
+        subprocess.run(['git', '-C', str(self.repo), 'add', name], check=True)
+
+    def test_documentation_commit_does_not_require_docker(self):
+        self.stage('README.md', 'Documentation\n')
+        with patch.object(lint.shutil, 'which', return_value=None):
+            self.assertEqual(lint.run(self.ctx, SimpleNamespace(hook=True)), 0)
+
+    def test_compose_change_still_requires_docker(self):
+        self.stage('docker-compose.yml', 'services: {}\n')
+        with patch.object(lint.shutil, 'which', return_value=None):
+            self.assertNotEqual(lint.run(self.ctx, SimpleNamespace(hook=True)), 0)

tests/test_configure_alloy.py

@@ -0,0 +1,52 @@
+"""Native Alloy opt-in and validation boundaries."""
+import json
+from pathlib import Path
+import tempfile
+from types import SimpleNamespace
+import unittest
+from unittest.mock import Mock
+
+from engine import overlay
+
+ROOT = Path(__file__).resolve().parents[1]
+alloy = overlay.load_module(str(ROOT / 'actions/config/alloy.py'), 'test_alloy_action')
+
+
+class NativeAlloy(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.temp.cleanup)
+        self.configs = Path(self.temp.name)
+        self.ctx = SimpleNamespace(paths=SimpleNamespace(configs=str(self.configs), commons=str(ROOT)),
+                                   sys=Mock(), log=Mock())
+
+    def configure(self, **settings):
+        (self.configs / 'alloy').mkdir(exist_ok=True)
+        (self.configs / 'alloy/config.json').write_text(json.dumps(settings))
+
+    def test_unconfigured_node_is_untouched(self):
+        alloy.run(self.ctx, SimpleNamespace(dry_run=False))
+        self.ctx.sys.sudo.assert_not_called()
+
+    def test_invalid_role_or_environment_path_cannot_change_service(self):
+        for role, path in [('unknown', '/etc/alloy/secrets'), ('host', '/etc/alloy/invalid\nUser=root')]:
+            self.configure(role=role, environment_file=path)
+            with self.assertRaises(ValueError):
+                alloy.run(self.ctx, SimpleNamespace(dry_run=False))
+            self.ctx.sys.sudo.assert_not_called()
+
+    def test_dry_run_never_installs_or_starts(self):
+        self.configure(role='guest', environment_file='/etc/alloy/secrets')
+        alloy.run(self.ctx, SimpleNamespace(dry_run=True))
+        self.ctx.sys.sudo.assert_not_called()
+
+    def test_failed_config_validation_leaves_existing_service_untouched(self):
+        self.configure(role='host', environment_file='/etc/alloy/secrets')
+        def execute(command):
+            if command[0] == 'python3':
+                raise RuntimeError('invalid Alloy configuration')
+            self.assertEqual(command, ['test', '-f', '/etc/alloy/secrets'])
+        self.ctx.sys.sudo.side_effect = execute
+        with self.assertRaisesRegex(RuntimeError, 'invalid Alloy'):
+            alloy.run(self.ctx, SimpleNamespace(dry_run=False))
+        self.ctx.sys.daemon_reload.assert_not_called()

tests/test_configure_base.py

@@ -0,0 +1,287 @@
+"""Local baseline policy: fake only package, network, and root privilege boundaries."""
+import hashlib
+import io
+import json
+import os
+from pathlib import Path
+import stat
+import subprocess
+import sys
+import tempfile
+from types import SimpleNamespace
+import unittest
+from unittest.mock import Mock, patch
+
+from engine import overlay
+from engine.lib.host import Host
+from engine.lib.sys import Sys
+
+packages = overlay.load_module(
+    str(Path(__file__).resolve().parents[1] / "actions/config/packages.py"),
+    "apex_test_configure_base",
+)
+
+
+helper = overlay.load_module(
+    str(Path(__file__).resolve().parents[1] / "actions/config/ufw-docker.py"),
+    "apex_test_ufw_docker",
+)
+
+
+class TestBasePackages(unittest.TestCase):
+    def setUp(self):
+        self.system = Mock()
+        self.installed = set(packages.BASE_PACKAGES)
+        self.system.run.side_effect = self.query
+        self.system.sudo.return_value = subprocess.CompletedProcess([], 0, stdout="")
+
+    def query(self, command, **kwargs):
+        self.assertEqual(command[:3], ["dpkg-query", "-W", "-f=${Status}"])
+        status = "install ok installed" if command[-1] in self.installed else "unknown ok not-installed"
+        return subprocess.CompletedProcess(command, 0, stdout=status)
+
+    def test_installed_rsyslog_and_ufw_do_not_require_executable_or_apt_update(self):
+        packages._install_missing(self.system, packages.BASE_PACKAGES)
+        self.assertEqual({call.args[0][-1] for call in self.system.run.call_args_list},
+                         {"curl", "wget", "git", "rsyslog", "ufw"})
+        self.system.sudo.assert_not_called()
+
+    def test_only_missing_packages_are_installed_after_safe_simulation(self):
+        self.installed.remove("ufw")
+        self.system.sudo.return_value = subprocess.CompletedProcess(
+            [], 0, stdout="Inst ufw (0.36.2 Debian:13 [all])\nConf ufw (0.36.2 Debian:13 [all])\n",
+        )
+        packages._install_missing(self.system, packages.BASE_PACKAGES)
+        commands = [call.args[0] for call in self.system.sudo.call_args_list]
+        self.assertEqual(len(commands), 3)
+        self.assertIn("update", commands[0])
+        self.assertEqual(commands[1], [*commands[2], "--simulate"])
+        install = commands[2]
+        self.assertEqual(install[install.index("install"):], [
+            "install", "-y", "--no-install-recommends", "--no-upgrade", "--no-remove", "ufw",
+        ])
+
+    def test_privileged_apt_environment_survives_sudo_filtering(self):
+        execute = subprocess.run
+        operations = []
+
+        def filtered_sudo(command, **kwargs):
+            if command[0] == "dpkg-query":
+                return subprocess.CompletedProcess(command, 1, stdout="")
+            self.assertEqual(command[0], "sudo")
+            incoming = kwargs.get("env") or os.environ
+            filtered = {"PATH": os.defpath}
+            if "LC_ALL" in incoming:
+                filtered["LC_ALL"] = incoming["LC_ALL"]
+            apt_index = command.index("apt-get")
+            probe = execute([
+                *command[1:apt_index], sys.executable, "-I", "-c",
+                "import json, os; print(json.dumps([os.getenv('LC_ALL'), "
+                "os.getenv('DEBIAN_FRONTEND')]))",
+            ], env=filtered, check=True, text=True, capture_output=True)
+            self.assertEqual(json.loads(probe.stdout), ["C", "noninteractive"])
+            operations.append(command[apt_index:])
+            plan = "Inst ufw (0.36.2 Debian:13 [all])\n" if "--simulate" in command else ""
+            return subprocess.CompletedProcess(command, 0, stdout=plan)
+
+        with patch("engine.lib.sys.subprocess.run", side_effect=filtered_sudo):
+            packages._install_missing(Sys(Mock()), ["ufw"])
+        self.assertEqual(len(operations), 3)
+        self.assertIn("update", operations[0])
+        self.assertIn("--simulate", operations[1])
+        self.assertIn("install", operations[2])
+        self.assertNotIn("--simulate", operations[2])
+
+    def test_dependency_upgrades_and_removals_abort_before_installation(self):
+        self.installed.remove("curl")
+        for plan in ("Inst libcurl4 [7.0] (8.0 Debian:13 [amd64])\n",
+                     "Remv libcurl4 [7.0]\n"):
+            with self.subTest(plan=plan):
+                self.system.sudo.reset_mock()
+                self.system.sudo.return_value = subprocess.CompletedProcess([], 0, stdout=plan)
+                with self.assertRaisesRegex(RuntimeError, "upgrades/removals"):
+                    packages._install_missing(self.system, packages.BASE_PACKAGES)
+                self.assertEqual(self.system.sudo.call_count, 2)
+
+    def test_failed_package_query_is_not_treated_as_installed(self):
+        self.system.run.return_value = subprocess.CompletedProcess([], 1, stdout="")
+        self.system.run.side_effect = None
+        packages._install_missing(self.system, ["rsyslog"])
+        self.assertEqual(self.system.sudo.call_args.args[0][-1], "rsyslog")
+
+
+class TestPinnedHelper(unittest.TestCase):
+    def setUp(self):
+        temporary = tempfile.TemporaryDirectory()
+        self.addCleanup(temporary.cleanup)
+        self.root = Path(temporary.name).resolve()
+        self.path = self.root / "bin" / "ufw-docker"
+        self.path.parent.mkdir()
+        self.payload = b"#!/bin/sh\nprintf 'verified fixture\\n'\n"
+        self.wrong_owner = None
+        self.wrong_group = None
+        self.system = Mock()
+        self.system.sudo.side_effect = self.sudo
+        self.ctx = SimpleNamespace(log=Mock(), sys=self.system)
+        self.ctx.host = Host(self.ctx.log, self.system)
+        real_lstat = Path.lstat
+
+        def root_lstat(path, *args, **kwargs):
+            info = list(real_lstat(path, *args, **kwargs))
+            info[4] = 1001 if path == self.wrong_owner else 0
+            info[5] = 1001 if path == self.wrong_group else 0
+            if path not in (self.path, self.path.parent, self.root):
+                info[0] &= ~0o022
+            return os.stat_result(info)
+
+        # Model root-owned ancestry without sudo/chown; content, modes and links stay real.
+        self.addCleanup(patch.stopall)
+        patch.object(Path, "lstat", autospec=True, side_effect=root_lstat).start()
+        patch.object(helper, "UFW_DOCKER_PATH", self.path).start()
+        patch.object(helper, "UFW_DOCKER_SHA256", hashlib.sha256(self.payload).hexdigest()).start()
+        self.download = patch.object(helper.urllib.request, "urlopen").start()
+        self.download.return_value = io.BytesIO(self.payload)
+
+    def sudo(self, command, **kwargs):
+        if command[:3] == ["python3", "-I", "-c"]:
+            return subprocess.run([sys.executable, *command[1:]], check=True, text=True, **kwargs)
+        self.assertEqual(command, [str(self.path), "install"])
+        self.assertEqual(self.path.read_bytes(), self.payload)
+        self.assertEqual(stat.S_IMODE(self.path.stat().st_mode), 0o755)
+        return subprocess.CompletedProcess(command, 0)
+
+    def write_helper(self, payload=None, mode=0o755):
+        self.path.write_bytes(self.payload if payload is None else payload)
+        self.path.chmod(mode)
+
+    def test_verified_existing_helper_executes_without_download(self):
+        self.write_helper()
+        helper._install_helper(self.ctx)
+        self.download.assert_not_called()
+        self.assertEqual(self.system.sudo.call_args.args[0], [str(self.path), "install"])
+        self.assertEqual(self.system.sudo.call_count, 1)
+
+    def test_wrong_executable_content_is_replaced_only_with_verified_download(self):
+        self.write_helper(b"#!/bin/sh\necho wrong\n")
+        helper._install_helper(self.ctx)
+        self.assertEqual(self.path.read_bytes(), self.payload)
+        self.assertEqual(self.system.sudo.call_count, 2)
+        self.download.assert_called_once_with(helper.UFW_DOCKER_URL, timeout=30)
+
+    def test_missing_helper_is_published_with_executable_mode(self):
+        helper._install_helper(self.ctx)
+        self.assertEqual(self.path.read_bytes(), self.payload)
+        self.assertEqual(stat.S_IMODE(self.path.stat().st_mode), 0o755)
+        self.assertEqual(list(self.path.parent.iterdir()), [self.path])
+
+    def test_verified_nonexecutable_content_is_reused_and_mode_repaired(self):
+        self.write_helper(mode=0o600)
+        helper._install_helper(self.ctx)
+        self.download.assert_not_called()
+        self.assertEqual(stat.S_IMODE(self.path.stat().st_mode), 0o755)
+        self.assertEqual(self.system.sudo.call_count, 2)
+
+    def test_bad_checksum_never_publishes_or_executes(self):
+        self.download.return_value = io.BytesIO(b"not the pinned helper")
+        with self.assertRaisesRegex(RuntimeError, "checksum mismatch"):
+            helper._install_helper(self.ctx)
+        self.system.sudo.assert_not_called()
+        self.assertFalse(self.path.exists())
+
+    def test_bad_download_preserves_existing_wrong_helper_without_executing_it(self):
+        self.write_helper(b"old unknown executable")
+        self.download.return_value = io.BytesIO(b"bad download")
+        with self.assertRaisesRegex(RuntimeError, "checksum mismatch"):
+            helper._install_helper(self.ctx)
+        self.system.sudo.assert_not_called()
+        self.assertEqual(self.path.read_bytes(), b"old unknown executable")
+
+    def assert_rejected_before_network_or_sudo(self):
+        with self.assertRaisesRegex(RuntimeError, "Unsafe ufw-docker"):
+            helper._install_helper(self.ctx)
+        self.download.assert_not_called()
+        self.system.sudo.assert_not_called()
+
+    def test_symlink_target_is_rejected(self):
+        target = self.root / "untrusted"
+        target.write_bytes(self.payload)
+        self.path.symlink_to(target)
+        self.assert_rejected_before_network_or_sudo()
+
+    def test_dangling_symlink_target_is_rejected(self):
+        self.path.symlink_to(self.root / "missing")
+        self.assert_rejected_before_network_or_sudo()
+
+    def test_symlink_ancestor_is_rejected(self):
+        self.path.parent.rmdir()
+        target = self.root / "real-bin"
+        target.mkdir()
+        self.path.parent.symlink_to(target)
+        self.assert_rejected_before_network_or_sudo()
+
+    def test_writable_ancestor_is_rejected(self):
+        self.path.parent.chmod(0o777)
+        self.assert_rejected_before_network_or_sudo()
+
+    def test_unsafe_helper_preflight_precedes_package_network_or_mutation(self):
+        self.path.parent.chmod(0o777)
+        with patch.object(helper.os.path, "isfile", return_value=True):
+            with self.assertRaisesRegex(RuntimeError, "Unsafe ufw-docker"):
+                helper.run(self.ctx, None)
+        self.download.assert_not_called()
+        self.system.run.assert_not_called()
+        self.system.sudo.assert_not_called()
+
+    def test_untrusted_owner_ancestor_is_rejected(self):
+        self.wrong_owner = self.path.parent
+        self.assert_rejected_before_network_or_sudo()
+
+    def test_untrusted_file_owner_is_rejected(self):
+        self.write_helper()
+        self.wrong_owner = self.path
+        self.assert_rejected_before_network_or_sudo()
+
+    def test_untrusted_file_group_is_rejected(self):
+        self.write_helper()
+        self.wrong_group = self.path
+        self.assert_rejected_before_network_or_sudo()
+
+    def test_writable_helper_is_rejected(self):
+        self.write_helper(mode=0o775)
+        self.assert_rejected_before_network_or_sudo()
+
+    def test_hardlinked_helper_is_rejected(self):
+        self.write_helper()
+        os.link(self.path, self.root / "other-link")
+        self.assert_rejected_before_network_or_sudo()
+
+    def test_nonregular_helper_is_rejected(self):
+        self.path.mkdir()
+        self.assert_rejected_before_network_or_sudo()
+
+    def test_path_changed_during_download_is_rejected_before_publication(self):
+        def download(*args, **kwargs):
+            self.path.symlink_to(self.root / "untrusted")
+            return io.BytesIO(self.payload)
+
+        self.download.side_effect = download
+        with self.assertRaisesRegex(RuntimeError, "Unsafe ufw-docker"):
+            helper._install_helper(self.ctx)
+        self.system.sudo.assert_not_called()
+
+    def test_changed_content_after_publication_is_not_executed(self):
+        real_sudo = self.sudo
+
+        def tamper(command, **kwargs):
+            result = real_sudo(command, **kwargs)
+            self.path.write_bytes(b"changed after publication")
+            return result
+
+        self.system.sudo.side_effect = tamper
+        with self.assertRaisesRegex(RuntimeError, "verification failed"):
+            helper._install_helper(self.ctx)
+        self.assertEqual(self.system.sudo.call_count, 1)
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_env.py

@@ -0,0 +1,182 @@
+import contextlib
+import io
+import json
+import os
+import shutil
+import stat
+import subprocess
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import MagicMock
+
+from actions.config import env
+from engine.descriptor import build_parser
+from engine.lib.host import Host
+
+
+class TestEnv(unittest.TestCase):
+    def setUp(self):
+        self.tmpdir = tempfile.TemporaryDirectory()
+        self.addCleanup(self.tmpdir.cleanup)
+        self.core = Path(self.tmpdir.name) / 'compositions' / 'apex'
+        self.core.mkdir(parents=True)
+        self.ctx = MagicMock()
+        self.ctx.paths.compositions = str(self.core.parent)
+        self.ctx.vars.return_value = {
+            'APEX_SUBNET': '198.18.0.0/24',
+            'APEX_NODE_HOST': 'test-host',
+            'OTHER_VAR': 'ignore_me',
+        }
+        self.ctx.host = Host(self.ctx.log, self.ctx.sys)
+
+    def run_env(self, *argv):
+        args = build_parser(env.METADATA, 'apex config/env').parse_args(argv)
+        with contextlib.redirect_stdout(io.StringIO()) as out:
+            env.run(self.ctx, args)
+        return out.getvalue()
+
+    def test_shell_exports_round_trip_literal_values(self):
+        value = "a$b # 'quote' \"double\" \\backslash\r\nnext"
+        self.ctx.vars.return_value['APEX_NODE_HOST'] = value
+        exports = self.run_env('--export')
+        self.assertEqual(self.run_env(), exports)
+        script = exports + '\nprintf %s "$APEX_NODE_HOST"'
+        result = subprocess.run(['/bin/sh', '-c', script], capture_output=True, check=True,
+                                timeout=30, env={})
+        self.assertEqual(result.stdout.decode(), value)
+        self.assertNotIn('OTHER_VAR', script)
+
+    def test_dotenv_does_not_disclose_other_values_or_existing_secrets(self):
+        (self.core / '.env').write_text('SECRET=synthetic-private\n')
+        output = self.run_env('--dotenv')
+        self.assertIn('APEX_NODE_HOST=', output)
+        self.assertNotIn('export ', output)
+        self.assertNotIn('SECRET', output)
+        self.assertNotIn('OTHER_VAR', output)
+
+    def test_repeat_sync_preserves_credentials_bytes_and_updates_managed_values(self):
+        original = b'# user credentials\r\nTOKEN=\'a$#"\\literal\'\r\nUID=1000\r\nGID=1000'
+        target = self.core / '.env'
+        target.write_bytes(original)
+        target.chmod(0o600)
+        (self.core / 'apex.env').write_bytes(b'COMPOSE_PROFILES=core\r\nPORT=1234\r\n')
+        self.run_env('--sync')
+        first = target.read_bytes()
+        self.assertTrue(first.startswith(original + b'\n'))
+        self.assertIn(b'COMPOSE_PROFILES=core\r\nPORT=1234\r\n', first)
+        self.run_env('--sync')
+        self.assertEqual(first, target.read_bytes())
+        self.ctx.vars.return_value['APEX_NODE_HOST'] = 'new-host'
+        self.run_env('--sync')
+        self.assertTrue(target.read_bytes().startswith(original + b'\n'))
+        self.assertNotIn(b'test-host', target.read_bytes())
+        self.assertEqual(stat.S_IMODE(target.stat().st_mode), 0o600)
+
+    def test_sync_preserves_symlink_and_private_target(self):
+        target = Path(self.tmpdir.name) / 'private.env'
+        target.write_text('TOKEN=synthetic\n')
+        link = self.core / '.env'
+        link.symlink_to(target)
+        self.run_env('--sync')
+        self.assertTrue(link.is_symlink())
+        self.assertIn('TOKEN=synthetic', target.read_text())
+        self.assertIn('APEX_NODE_HOST=', target.read_text())
+        self.assertEqual(stat.S_IMODE(target.stat().st_mode), 0o600)
+
+    def test_write_basename_is_standalone_and_private(self):
+        previous = os.getcwd()
+        self.addCleanup(os.chdir, previous)
+        os.chdir(self.tmpdir.name)
+        Path('generated.env').write_text('OLD=removed\n')
+        self.run_env('--write', 'generated.env')
+        content = Path('generated.env').read_text()
+        self.assertIn('APEX_NODE_HOST=', content)
+        self.assertNotIn('OLD=', content)
+        self.assertNotIn('BEGIN', content)
+        self.assertEqual(stat.S_IMODE(Path('generated.env').stat().st_mode), 0o600)
+
+    def test_incompatible_modes_fail_without_writing(self):
+        for argv in [('--sync', '--dotenv'), ('--sync', '--write', 'file'),
+                     ('--export', '--dotenv'), ('--export', '--write', 'file')]:
+            with self.subTest(argv=argv):
+                with self.assertRaises(SystemExit) as error:
+                    self.run_env(*argv)
+                self.assertEqual(error.exception.code, 64)
+        self.assertFalse((self.core / '.env').exists())
+
+    def test_uncertain_managed_markers_fail_without_changing_file(self):
+        start = '# BEGIN APEX MANAGED ENV'
+        end = '# END APEX MANAGED ENV'
+        for content in [start+'\n', end+'\n', end+'\n'+start+'\n',
+                        start+'\n'+start+'\n'+end+'\n',
+                        start+'\n'+end+'\n'+start+'\n'+end+'\n',
+                        'TOKEN="'+start+'"\n']:
+            target = self.core / '.env'
+            target.write_text(content)
+            with self.subTest(content=content):
+                with self.assertRaises(SystemExit) as error:
+                    self.run_env('--sync')
+                self.assertEqual(error.exception.code, 65)
+                self.assertEqual(target.read_text(), content)
+
+    def test_source_marker_collision_fails_without_writing(self):
+        (self.core / 'apex.env').write_text('# BEGIN APEX MANAGED ENV\n')
+        with self.assertRaises(SystemExit) as error:
+            self.run_env('--sync')
+        self.assertEqual(error.exception.code, 65)
+        self.assertFalse((self.core / '.env').exists())
+        (self.core / 'apex.env').unlink()
+        self.ctx.vars.return_value['APEX_NODE_HOST'] = '# END APEX MANAGED ENV'
+        with self.assertRaises(SystemExit) as error:
+            self.run_env('--sync')
+        self.assertEqual(error.exception.code, 65)
+        self.assertFalse((self.core / '.env').exists())
+
+    def test_bare_compose_interprets_literals_precedence_and_profiles(self):
+        binary = shutil.which('docker-compose')
+        if binary is None:
+            self.skipTest('Standalone docker-compose required for daemon-free config check')
+        values = {
+            'APEX_LITERAL': "a$b ${UNSET} $$ # 'quote' \"double\" \\backslash",
+            'APEX_TRAILING': 'last\\',
+            'APEX_LINES': 'one\r\ntwo\nthree\tfour',
+            'APEX_EMPTY': '',
+            'APEX_ESCAPES': "\\n \\\\ \\' \\\"",
+            'APEX_NODE_HOST': 'synthesized',
+        }
+        self.ctx.vars.return_value = values
+        (self.core / '.env').write_text('UID=1000\nGID=1001\nPORT=1\nAPEX_NODE_HOST=old\n'
+                                       "TOKEN='literal$token#hash'\n")
+        (self.core / 'apex.env').write_text('PORT=2\nAPEX_NODE_HOST=public\n'
+                                           'COMPOSE_PROFILES=chosen\n')
+        keys = [*values, 'UID', 'GID', 'PORT', 'TOKEN']
+        (self.core / 'docker-compose.yml').write_text(
+            'services:\n  probe:\n    image: busybox\n    profiles: [chosen]\n'
+            '    environment:\n' + ''.join(f'      {key}: ${{{key}}}\n' for key in keys))
+        self.run_env('--sync')
+        result = subprocess.run([binary, 'config', '--format', 'json'], cwd=self.core,
+                                capture_output=True, check=True, timeout=30, env={})
+        actual = json.loads(result.stdout)['services']['probe']['environment']
+        expected = {**values, 'UID': '1000', 'GID': '1001', 'PORT': '2',
+                    'TOKEN': 'literal$token#hash'}
+        # Compose re-escapes literal dollars when serializing a reusable configuration.
+        self.assertEqual(actual, {k: v.replace('$', '$$') for k, v in expected.items()})
+        result = subprocess.run([binary, 'config', '--environment'], cwd=self.core,
+                                capture_output=True, check=True, timeout=30, env={})
+        for key, value in {**expected, 'COMPOSE_PROFILES': 'chosen'}.items():
+            self.assertIn(f'{key}={value}\n'.encode(), result.stdout)
+        with (self.core / '.env').open('a') as target:
+            target.write('# later user addition\nPORT=999\n')
+        self.run_env('--sync')
+        self.assertIn('# later user addition\nPORT=999\n', (self.core / '.env').read_text())
+        result = subprocess.run([binary, 'config', '--format', 'json'], cwd=self.core,
+                                capture_output=True, check=True, timeout=30, env={})
+        self.assertEqual(json.loads(result.stdout)['services']['probe']['environment']['PORT'], '2')
+        result = subprocess.run([binary, 'config', '--services'], cwd=self.core,
+                                capture_output=True, check=True, timeout=30, env={})
+        self.assertEqual(result.stdout.strip(), b'probe')
+
+
+if __name__ == '__main__':
+    unittest.main()

tests/test_host.py

@@ -1,6 +1,211 @@
 # tests/test_host.py
+import inspect
+import os
+from pathlib import Path
+import stat
+import subprocess
+import sys
+import tempfile
 import unittest
-from engine.lib.host import apply_block
+from unittest.mock import Mock, patch
+
+from engine.lib.host import Host, apply_block
+from engine.lib.sys import Sys
+
+
+def _check_backup_chmod(chmod, expected_group):
+    import os
+    import stat
+
+    def checked(path, mode, *args, **kwargs):
+        result = chmod(path, mode, *args, **kwargs)
+        info = os.stat(path)
+        with open(path, "rb") as stream:
+            content = stream.read()
+        if content and stat.S_IMODE(info.st_mode) & 0o077 and info.st_gid != expected_group:
+            raise AssertionError("Backup content exposed to the inherited staging group")
+        return result
+
+    return checked
+
+
+class TestHostWrites(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.temp.cleanup)
+        self.root = Path(self.temp.name).resolve()
+        self.log = Mock()
+        self.host = Host(self.log, Mock())
+
+    def test_existing_private_modes_and_owner_survive_replacement(self):
+        for mode in (0o600, 0o640):
+            with self.subTest(mode=mode):
+                path = self.root / "secret"
+                path.write_text("old")
+                path.chmod(mode)
+                previous = path.stat()
+                self.host.write_file(str(path), "new", backup=False)
+                self.assertEqual(path.read_text(), "new")
+                self.assertEqual(stat.S_IMODE(path.stat().st_mode), mode)
+                self.assertEqual((path.stat().st_uid, path.stat().st_gid),
+                                 (previous.st_uid, previous.st_gid))
+
+    def test_new_basename_defaults_to_0644(self):
+        previous = Path.cwd()
+        try:
+            os.chdir(self.root)
+            self.host.write_file("settings", "value", backup=False)
+        finally:
+            os.chdir(previous)
+        path = self.root / "settings"
+        self.assertEqual(path.read_text(), "value")
+        self.assertEqual(stat.S_IMODE(path.stat().st_mode), 0o644)
+
+    def test_existing_supplementary_group_survives_replacement(self):
+        path = self.root / "group-owned"
+        path.write_text("old")
+        groups = set(os.getgroups()) - {path.stat().st_gid}
+        if not groups:
+            self.skipTest("No alternate supplementary group available")
+        group = next(iter(groups))
+        try:
+            os.chown(path, -1, group)
+        except PermissionError:
+            self.skipTest("Filesystem does not permit changing to a supplementary group")
+        self.host.write_file(str(path), "new", backup=False)
+        self.assertEqual(path.read_text(), "new")
+        self.assertEqual(path.stat().st_gid, group)
+
+    def test_explicit_mode_is_applied_before_atomic_publication(self):
+        path = self.root / "secret"
+        path.write_text("old")
+        replace = os.replace
+        published = []
+
+        def observe(source, destination):
+            self.assertEqual(path.read_text(), "old")
+            self.assertEqual(Path(source).parent, self.root)
+            self.assertEqual(stat.S_IMODE(os.stat(source).st_mode), 0o600)
+            self.assertEqual(Path(source).read_text(), "new secret")
+            published.append(destination)
+            replace(source, destination)
+
+        with patch("os.replace", side_effect=observe):
+            self.host.write_file(str(path), "new secret", mode="0600", backup=False)
+        self.assertEqual(len(published), 1)
+        self.assertEqual(path.read_text(), "new secret")
+
+    def test_symlink_remains_and_target_receives_new_content(self):
+        target = self.root / "tier" / "secret"
+        target.parent.mkdir()
+        target.write_text("old")
+        target.chmod(0o600)
+        link = self.root / ".env"
+        link.symlink_to("tier/secret")
+        self.host.write_file(str(link), "new", backup=False)
+        self.assertTrue(link.is_symlink())
+        self.assertEqual(os.readlink(link), "tier/secret")
+        self.assertEqual(target.read_text(), "new")
+        self.assertEqual(stat.S_IMODE(target.stat().st_mode), 0o600)
+
+    def test_dry_run_does_not_create_parents_or_backups(self):
+        self.host.write_file(str(self.root / "missing" / "secret"), "new", dry_run=True)
+        self.assertEqual(list(self.root.iterdir()), [])
+
+    def test_backup_preserves_old_private_content(self):
+        path = self.root / "secret"
+        path.write_text("old")
+        path.chmod(0o600)
+        self.host.write_file(str(path), "new")
+        backups = list(self.root.glob("secret.*.bak"))
+        self.assertEqual(len(backups), 1)
+        self.assertEqual(backups[0].read_text(), "old")
+        self.assertEqual(stat.S_IMODE(backups[0].stat().st_mode), 0o600)
+        self.assertEqual(path.read_text(), "new")
+
+    def test_root_can_perform_privileged_file_operations_before_sudo_is_installed(self):
+        path = self.root / "bootstrap"
+        path.write_text("first")
+        host = Host(self.log, Sys(self.log))
+        with patch("os.geteuid", return_value=0), patch("shutil.which", return_value=None):
+            host.write_file(str(path), "second", sudo=True)
+            self.assertEqual(host._read(str(path), sudo=True), "second")
+        self.assertEqual(next(self.root.glob("bootstrap.*.bak")).read_text(), "first")
+
+    def test_backup_sets_owner_before_exposing_group_readable_content(self):
+        path = self.root / "group-secret"
+        path.write_text("synthetic group-only secret")
+        groups = set(os.getgroups()) - {path.stat().st_gid}
+        if not groups:
+            self.skipTest("No alternate supplementary group available")
+        group = next(iter(groups))
+        try:
+            os.chown(path, -1, group)
+        except PermissionError:
+            self.skipTest("Filesystem does not permit changing to a supplementary group")
+        path.chmod(0o640)
+        os.utime(path, (1600000000, 1600000000))
+
+        def local_sudo(command, **kwargs):
+            if command[:2] == ["test", "-f"]:
+                return subprocess.run(command, **kwargs)
+            self.assertEqual(command[:3], ["python3", "-I", "-c"])
+            probe = ("import os\n" + inspect.getsource(_check_backup_chmod) +
+                     f"\nos.chmod = _check_backup_chmod(os.chmod, {group})\n")
+            result = subprocess.run([sys.executable, "-I", "-c", probe + command[3]],
+                                    check=False, text=True, capture_output=True, **kwargs)
+            self.assertEqual(result.returncode, 0, result.stderr)
+            return result
+
+        self.host.sys.sudo.side_effect = local_sudo
+        for sudo in (False, True):
+            with self.subTest(sudo=sudo):
+                with patch("os.chmod", side_effect=_check_backup_chmod(os.chmod, group)):
+                    backup = Path(self.host.backup(str(path), sudo=sudo))
+                self.assertEqual(backup.read_bytes(), path.read_bytes())
+                self.assertEqual(backup.stat().st_gid, group)
+                self.assertEqual(stat.S_IMODE(backup.stat().st_mode), 0o640)
+                self.assertEqual(backup.stat().st_mtime_ns, path.stat().st_mtime_ns)
+                backup.unlink()
+
+    def test_publication_failure_keeps_original_and_cleans_staging(self):
+        path = self.root / "secret"
+        path.write_text("old")
+        with patch("os.replace", side_effect=OSError("publication failed")):
+            with self.assertRaisesRegex(OSError, "publication failed"):
+                self.host.write_file(str(path), "new", backup=False)
+        self.assertEqual(path.read_text(), "old")
+        self.assertEqual(list(self.root.iterdir()), [path])
+        self.log.success.assert_not_called()
+
+    def test_write_failure_keeps_original_and_cleans_staging(self):
+        path = self.root / "secret"
+        path.write_text("old")
+        with patch("os.fsync", side_effect=OSError("disk full")):
+            with self.assertRaisesRegex(OSError, "disk full"):
+                self.host.write_file(str(path), "new", backup=False)
+        self.assertEqual(path.read_text(), "old")
+        self.assertEqual(list(self.root.iterdir()), [path])
+        self.log.success.assert_not_called()
+
+    def test_sudo_runs_same_writer_without_copying_over_live_target(self):
+        target = self.root / "secret"
+        target.write_text("old")
+        target.chmod(0o640)
+        link = self.root / ".env"
+        link.symlink_to("secret")
+
+        def local_sudo(command, **kwargs):
+            self.assertEqual(command[:3], ["python3", "-I", "-c"])
+            return subprocess.run([sys.executable, *command[1:]], check=True,
+                                  text=True, **kwargs)
+
+        self.host.sys.sudo.side_effect = local_sudo
+        self.host.write_file(str(link), "new", sudo=True, backup=False)
+        self.assertTrue(link.is_symlink())
+        self.assertEqual(target.read_text(), "new")
+        self.assertEqual(stat.S_IMODE(target.stat().st_mode), 0o640)
+        self.assertEqual(set(self.root.iterdir()), {link, target})
 
 class TestApplyBlock(unittest.TestCase):
     def test_insert_after_anchor(self):

tests/test_identity.py

@@ -1,42 +1,39 @@
 # tests/test_identity.py
 import os, tempfile, unittest
-from engine.identity import resolve, Node
+from engine.identity import resolve
 
 class TestIdentity(unittest.TestCase):
-    def _envfile(self, text, dirname=None):
+    def _envfile(self, text):
         d = tempfile.mkdtemp()
-        if dirname:
-            d = os.path.join(d, dirname)
-            os.makedirs(d)
         with open(os.path.join(d, "node.env"), "w") as f:
             f.write(text)
         return d
 
-    def test_fqdn_is_source_of_truth(self):
+    def test_explicit_host_and_fqdn(self):
-        repo = self._envfile("APEX_CLUSTER=a1\nAPEX_SUBNET=198.18.11.0/24\n")
+        repo = self._envfile("APEX_NODE_FQDN=node1.dc1.example.com\n"
-        n, warns = resolve("daedalus.a1.apex.ermnvldmr.com", repo)
+                             "APEX_NODE_HOST=node1.example.com\n"
-        self.assertEqual((n.name, n.cluster, n.subnet), ("daedalus", "a1", "198.18.11.0/24"))
+                             "APEX_SUBNET=10.0.0.0/24\n")
+        n, warns = resolve("build-laptop", repo)   # hostname ignored when node.env has FQDN
+        self.assertEqual((n.name, n.host, n.fqdn, n.subnet),
+                         ("node1", "node1.example.com", "node1.dc1.example.com", "10.0.0.0/24"))
         self.assertEqual(warns, [])
 
-    def test_cluster_drift_warns_hostname_wins(self):
+    def test_host_defaults_to_fqdn(self):
-        repo = self._envfile("APEX_CLUSTER=a9\nAPEX_SUBNET=198.18.11.0/24\n")
+        repo = self._envfile("APEX_NODE_FQDN=n1.example.com\nAPEX_SUBNET=10.0.0.0/24\n")
-        n, warns = resolve("daedalus.a1.apex.ermnvldmr.com", repo)
+        n, warns = resolve("ignored", repo)
-        self.assertEqual(n.cluster, "a1")           # hostname wins
+        self.assertEqual((n.name, n.host, n.fqdn), ("n1", "n1.example.com", "n1.example.com"))
-        self.assertTrue(any("drift" in w.lower() for w in warns))
+        self.assertEqual(warns, [])
 
-    def test_fallback_when_fqdn_not_apex_shaped(self):
+    def test_fqdn_falls_back_to_hostname(self):
-        # repo dir basename carries the node name in fallback (real host layout:
+        repo = self._envfile("APEX_SUBNET=10.0.0.0/24\n")
-        # $HOME/morpheus.apex.ermnvldmr.com)
+        n, warns = resolve("box.local", repo)
-        repo = self._envfile("APEX_CLUSTER=a2\nAPEX_SUBNET=198.18.13.0/24\n",
+        self.assertEqual((n.name, n.host, n.fqdn), ("box", "box.local", "box.local"))
-                             dirname="morpheus.apex.ermnvldmr.com")
+        self.assertTrue(any("APEX_NODE_FQDN" in w for w in warns))
-        n, warns = resolve("build-laptop", repo)
-        self.assertEqual((n.name, n.cluster, n.subnet), ("morpheus", "a2", "198.18.13.0/24"))
-        self.assertTrue(any("fqdn" in w.lower() for w in warns))
 
     def test_missing_subnet_is_error(self):
-        repo = self._envfile("APEX_CLUSTER=a1\n")
+        repo = self._envfile("APEX_NODE_FQDN=n1.example.com\n")
         with self.assertRaises(SystemExit):
-            resolve("daedalus.a1.apex.ermnvldmr.com", repo)
+            resolve("n1.example.com", repo)
 
 if __name__ == "__main__":
     unittest.main()

tests/test_manual_host_setup.py

@@ -0,0 +1,233 @@
+import json
+import os
+from pathlib import Path
+import subprocess
+import sys
+import tempfile
+from types import SimpleNamespace
+import unittest
+from unittest.mock import Mock, patch
+
+from engine import overlay
+
+
+def action(name):
+    return overlay.load_module(str(Path(__file__).resolve().parents[1] / f"actions/{name}.py"),
+                               "test_manual_" + name.replace("/", "_"))
+
+
+class HostSetup(unittest.TestCase):
+    def test_packages_preserve_pve_firewall(self):
+        packages = action("config/packages")
+        ctx = SimpleNamespace(sys=Mock())
+        with (patch.object(Path, "is_file", return_value=True),
+              patch.object(Path, "exists", return_value=True),
+              patch.object(packages, "_install_missing") as install):
+            packages.run(ctx, None)
+        requested = install.call_args.args[1]
+        self.assertIn("sudo", requested)
+        self.assertNotIn("ufw", requested)
+
+    def test_fresh_key_proof_gates_each_destructive_phase(self):
+        hardening = action("config/hardening")
+        for phase, operation in (("transition", "finalize"), ("final", "commit")):
+            state = {"status": phase, "authenticated_after_us": 123,
+                     "bundle": "/usr/local/lib/apex-provisioner/" + "a" * 64, "token": "a" * 32}
+            system = Mock()
+            with (patch.object(hardening, "_state", return_value=state),
+                  patch.object(hardening, "_connection", side_effect=RuntimeError("fresh login required")) as proof):
+                with self.assertRaisesRegex(RuntimeError, "fresh login required"):
+                    hardening.run(SimpleNamespace(sys=system), SimpleNamespace(operation=operation))
+                proof.assert_called_once_with(unittest.mock.ANY, after=123)
+            self.assertFalse(system.mock_calls)
+
+    def test_begin_installs_bundle_before_dispatch_with_real_endpoint_and_profile(self):
+        hardening = action("config/hardening")
+        ctx = SimpleNamespace(sys=Mock())
+        bundle = "/usr/local/lib/apex-provisioner/" + "a" * 64
+        ctx.sys.sudo.return_value.stdout = bundle
+        with (patch.object(hardening, "_operator_access"),
+              patch.object(hardening, "_state", return_value={}),
+              patch.object(Path, "is_file", return_value=True),
+              patch.object(hardening, "_original_endpoint", return_value=(2022, "192.0.2.8"))):
+            hardening.run(ctx, None)
+        install, begin = [call.args[0] for call in ctx.sys.sudo.call_args_list]
+        self.assertTrue(install[-1].endswith("/manual_bundle.py"))
+        self.assertIn(bundle, begin)
+        self.assertEqual(begin[begin.index("begin"): -2],
+                         ["begin", "--profile", "proxmox", "--original-port", "2022", "--client-address", "192.0.2.8"])
+
+    def test_installed_watchdog_survives_checkout_and_dispatches_cli_arguments(self):
+        hardening = action("config/hardening")
+        root = Path(__file__).resolve().parents[1]
+        installer = overlay.load_module(str(root / "scripts/provisioning/manual_bundle.py"),
+                                        "test_manual_bundle")
+        with tempfile.TemporaryDirectory() as temp, patch.object(sys, "path", list(sys.path)):
+            destination = installer.install(Path(temp).resolve())
+            self.assertEqual(installer.install(Path(temp).resolve()), destination)
+            state = {"status": "final", "authenticated_after_us": 120, "profile": "debian",
+                     "bundle": "/usr/local/lib/apex-provisioner/" + destination.name, "token": "a" * 32}
+            system = Mock()
+            with (patch.object(hardening, "_state", return_value=state),
+                  patch.object(hardening, "_connection")):
+                hardening.run(SimpleNamespace(sys=system), SimpleNamespace(operation="commit"))
+            # Run the generated CLI in isolation. Stop at its root check after argument parsing.
+            command = system.sudo.call_args.args[0]
+            command[0] = sys.executable
+            command[3] = "import os; os.geteuid = lambda: 1000; " + command[3]
+            command[4] = str(destination)
+            result = subprocess.run(command, cwd=temp, capture_output=True, text=True, timeout=30)
+            self.assertIn("Access handover requires root", result.stderr)
+            source = destination / "provisioning/access.py"
+            source.chmod(0o666)
+            with self.assertRaisesRegex(RuntimeError, "Untrusted installed"):
+                installer.install(Path(temp).resolve())
+            source.chmod(0o644)
+            source.write_text("changed")
+            with self.assertRaisesRegex(RuntimeError, "differs from the checkout"):
+                installer.install(Path(temp).resolve())
+
+    def test_new_bundle_preserves_an_existing_rollback_installation(self):
+        installer_path = Path(__file__).resolve().parents[1] / "scripts/provisioning/manual_bundle.py"
+        installer = overlay.load_module(str(installer_path), "test_bundle_upgrade")
+        with tempfile.TemporaryDirectory() as temp:
+            root = Path(temp).resolve()
+            checkout = root / "checkout"
+            checkout.mkdir()
+            for name in ("__init__.py", "access.py", "state.py"):
+                (checkout / name).write_text("# original")
+            with patch.object(installer, "__file__", str(checkout / "manual_bundle.py")):
+                original = installer.install(root)
+                (checkout / "access.py").write_text("# updated")
+                updated = installer.install(root)
+            self.assertNotEqual(original, updated)
+            self.assertEqual((original / "provisioning/access.py").read_text(), "# original")
+            self.assertEqual((updated / "provisioning/access.py").read_text(), "# updated")
+
+    def test_other_linux_distributions_are_rejected_before_installing_the_watchdog(self):
+        hardening = action("config/hardening")
+        ctx = SimpleNamespace(sys=Mock())
+        with (patch.object(hardening, "_operator_access"),
+              patch.object(hardening, "_state", return_value={}),
+              patch.object(Path, "is_file", return_value=False),
+              patch.object(hardening.platform, "freedesktop_os_release", return_value={"ID": "ubuntu"})):
+            with self.assertRaisesRegex(RuntimeError, "supports Debian"):
+                hardening.run(ctx, None)
+        self.assertFalse(ctx.sys.mock_calls)
+
+    def test_current_endpoint_must_have_authenticated_with_a_key(self):
+        hardening = action("config/hardening")
+        ctx = SimpleNamespace(sys=Mock())
+        sockets = SimpleNamespace(stdout='0 0 192.0.2.9:22 192.0.2.8:3002 users:(("sshd",pid=60,fd=4))')
+        accepted = {"_PID": "60", "__MONOTONIC_TIMESTAMP": "100",
+                    "MESSAGE": "Accepted publickey for adam from 192.0.2.8 port 3002 ssh2: ED25519 SHA256:example"}
+        with (patch.dict(os.environ, {"SSH_CONNECTION": "192.0.2.100 4321 192.0.2.9 1234",
+                                      "SSH_USER_AUTH": "/arbitrary/file"}),
+              patch.object(hardening, "_ancestors", return_value={60: 50}),
+              patch.object(hardening.pwd, "getpwuid", return_value=SimpleNamespace(pw_name="adam"))):
+            for changes in ({"_PID": "42"}, {"__MONOTONIC_TIMESTAMP": "49"},
+                            {"MESSAGE": accepted["MESSAGE"].replace("3002", "4002")}):
+                with self.subTest(changes=changes):
+                    ctx.sys.sudo.side_effect = [sockets, SimpleNamespace(
+                        returncode=0, stdout=json.dumps({**accepted, **changes}))]
+                    with self.assertRaisesRegex(RuntimeError, "Cannot verify"):
+                        hardening._connection(ctx)
+            ctx.sys.sudo.side_effect = [sockets, SimpleNamespace(returncode=0, stdout=json.dumps(accepted))]
+            self.assertEqual(hardening._connection(ctx),
+                             "user=adam,host=192.0.2.8,addr=192.0.2.8,laddr=192.0.2.9,lport=22")
+
+    def test_reused_transport_and_wrong_port_cannot_finalize(self):
+        hardening = action("config/hardening")
+        ctx = SimpleNamespace(sys=Mock())
+        entry = {"_PID": "60", "__MONOTONIC_TIMESTAMP": "100",
+                 "MESSAGE": "Accepted publickey for adam from 192.0.2.8 port 3002 ssh2: ED25519"}
+        ctx.sys.sudo.return_value = SimpleNamespace(returncode=0, stdout=json.dumps(entry))
+        with (patch.object(hardening.pwd, "getpwuid", return_value=SimpleNamespace(pw_name="adam")),
+              patch.object(hardening, "_ancestors", return_value={60: 50})):
+            for port, after in ((22, 49), (2222, 50), (2222, 99), (2222, 101)):
+                with (self.subTest(port=port, after=after),
+                      patch.object(hardening, "_socket", return_value=("192.0.2.9", port, "192.0.2.8", 3002)),
+                      self.assertRaises(RuntimeError)):
+                    hardening._connection(ctx, after=after)
+            with patch.object(hardening, "_socket", return_value=("192.0.2.9", 2222, "192.0.2.8", 3002)):
+                self.assertIn("lport=2222", hardening._connection(ctx, after=49))
+
+    def test_journald_restart_failure_restores_previous_config(self):
+        base = action("config/base")
+        with tempfile.TemporaryDirectory() as temporary:
+            root = Path(temporary)
+            target = root / "99-apex.conf"
+            legacy = root / "maxuse.conf"
+            target.write_text("previous\n")
+            legacy.write_text("legacy\n")
+            backup = root / "legacy-backup"
+            host = Mock()
+            host.backup.side_effect = lambda path, **kwargs: (backup.write_bytes(Path(path).read_bytes())
+                                                             and str(backup))
+            host.write_file.side_effect = lambda path, content, **kwargs: Path(path).write_text(content)
+            def sudo(command):
+                if command[0] == "systemctl":
+                    raise RuntimeError("restart failed")
+                if command[0] == "cp":
+                    Path(command[-1]).write_bytes(Path(command[-2]).read_bytes())
+                else:
+                    Path(command[-1]).unlink()
+            ctx = SimpleNamespace(sys=SimpleNamespace(sudo=sudo), host=host)
+            with self.assertRaisesRegex(RuntimeError, "restart failed"):
+                base._configure(ctx, target, legacy, "updated\n", restart=True)
+            self.assertEqual(target.read_text(), "previous\n")
+            self.assertEqual(legacy.read_text(), "legacy\n")
+
+    def test_unverified_session_makes_no_host_changes(self):
+        hardening = action("config/hardening")
+        ctx = SimpleNamespace(sys=Mock(), host=Mock())
+        with (patch.object(hardening.pwd, "getpwnam", return_value=SimpleNamespace(
+                  pw_uid=1001, pw_dir="/home/adam")),
+              patch.object(hardening.pwd, "getpwuid", return_value=SimpleNamespace(pw_name="root")),
+              patch.object(Path, "read_text", return_value="1001")):
+            with self.assertRaisesRegex(RuntimeError, "independent administrator"):
+                hardening.run(ctx, None)
+        self.assertFalse(ctx.host.mock_calls)
+        self.assertFalse(ctx.sys.mock_calls)
+
+    def test_independent_administrator_can_harden_only_with_an_operator_key(self):
+        hardening = action("config/hardening")
+        account = SimpleNamespace(pw_uid=1001, pw_dir="/home/adam")
+        system = Mock()
+        def privileged(command, **kwargs):
+            output = ""
+            if "stat" in command:
+                output = "1001 600 regular file" if command[-1].endswith("authorized_keys") else "1001 700 directory"
+            elif "cat" in command:
+                output = "ssh-ed25519 fixture"
+            elif "-T" in command:
+                output = "authorizedkeysfile .ssh/authorized_keys"
+            return SimpleNamespace(stdout=output, returncode=0)
+        system.sudo.side_effect = privileged
+        ctx = SimpleNamespace(sys=system)
+        with (patch.object(hardening.pwd, "getpwnam", return_value=account),
+              patch.object(hardening.pwd, "getpwuid", return_value=SimpleNamespace(pw_name="root")),
+              patch.object(hardening.os, "getuid", return_value=0),
+              patch.object(Path, "read_text", return_value="0")):
+            system.run.return_value.returncode = 1
+            with self.assertRaisesRegex(RuntimeError, "valid unrestricted operator public key"):
+                hardening._operator_access(ctx)
+            system.run.return_value.returncode = 0
+            self.assertEqual(hardening._operator_access(ctx), "user=adam,host=localhost,addr=127.0.0.1")
+
+    def test_chown_limits_changes_to_checkout_and_skips_mounts_and_hardlinks(self):
+        chown = action("repository/chown")
+        with tempfile.TemporaryDirectory() as temporary:
+            root = Path(temporary).resolve()
+            for name in (".git", "configs", "volumes"):
+                (root / name).mkdir()
+            for name in ("configs/node", ".git/HEAD", "private", "volumes/data"):
+                (root / name).write_text("fixture")
+            os.link(root / "private", root / "configs/key")
+            (root / "configs/other").symlink_to(root / "volumes", target_is_directory=True)
+            with patch("os.chown") as ownership, patch("os.fchown"):
+                chown._repair(str(root), {"configs/node", "configs/key", "configs/other", "volumes/data"},
+                              {str(root / "volumes")}, os.getuid(), os.getgid())
+            self.assertEqual({call.args[0] for call in ownership.call_args_list},
+                             {"HEAD", "node", "other"})
+            self.assertTrue(all(not call.kwargs["follow_symlinks"] for call in ownership.call_args_list))

tests/test_notify.py

@@ -1,15 +1,17 @@
 # tests/test_notify.py
+import os
+import tempfile
 import unittest
-from engine.lib.notify import build_text
+from engine.lib.notify import build_text, resolve_bot_url
 
 class TestNotify(unittest.TestCase):
-    def test_instance_uses_apex_prefix(self):
+    def test_instance_is_the_host(self):
-        t = build_text("Title", "msg", "daedalus", "INFO", "2026-07-04 00:00:00 MSK")
+        t = build_text("Title", "msg", "node1.example.com", "INFO", "2026-07-04 00:00:00 UTC")
-        self.assertIn("com.ermnvldmr.apex.daedalus", t)
+        self.assertIn("node1.example.com", t)
-        self.assertIn("#instance_com_ermnvldmr_apex_daedalus", t)
+        self.assertIn("#instance_node1_example_com", t)   # dots -> underscores in the hashtag
 
     def test_html_escaped_and_truncated(self):
-        t = build_text("A & B", "<script>" + "x" * 300, "d", "ERROR", "T")
+        t = build_text("A & B", "<script>" + "x" * 4000, "d", "ERROR", "T")
         self.assertIn("A &amp; B", t)
         self.assertIn("&lt;script&gt;", t)
         self.assertIn("...", t)
@@ -20,5 +22,19 @@ class TestNotify(unittest.TestCase):
         self.assertIn("Resolved", t)
         self.assertIn("✅", t)
 
+    def test_literal_bot_url_is_unchanged(self):
+        self.assertEqual(resolve_bot_url("https://example.test/bot"),
+                         "https://example.test/bot")
+
+    def test_bot_url_file_reference(self):
+        fd, path = tempfile.mkstemp()
+        try:
+            with os.fdopen(fd, "w") as f:
+                f.write("https://example.test/secret\n")
+            self.assertEqual(resolve_bot_url("@" + path),
+                             "https://example.test/secret")
+        finally:
+            os.unlink(path)
+
 if __name__ == "__main__":
     unittest.main()

tests/test_overlay.py

@@ -17,15 +17,15 @@ class TestOverlay(unittest.TestCase):
         self.prop = tempfile.mkdtemp()
 
     def test_commons_only(self):
-        _write(self.commons, "configure/ufw.py", ACT.format(s="commons ufw", r="C"))
+        _write(self.commons, "config/ufw.py", ACT.format(s="commons ufw", r="C"))
-        r = overlay.resolve("configure/ufw", self.commons, self.prop)
+        r = overlay.resolve("config/ufw", self.commons, self.prop)
         self.assertIsNotNone(r); self.assertIsNone(r.shadowed)
         self.assertTrue(r.path.startswith(self.commons))
 
     def test_override_exposes_shadowed(self):
-        _write(self.commons, "configure/ufw.py", ACT.format(s="commons", r="C"))
+        _write(self.commons, "config/ufw.py", ACT.format(s="commons", r="C"))
-        _write(self.prop, "configure/ufw.py", ACT.format(s="local", r="L"))
+        _write(self.prop, "config/ufw.py", ACT.format(s="local", r="L"))
-        r = overlay.resolve("configure/ufw", self.commons, self.prop)
+        r = overlay.resolve("config/ufw", self.commons, self.prop)
         self.assertTrue(r.path.startswith(self.prop))     # local wins
         self.assertTrue(r.shadowed.startswith(self.commons))
 
@@ -33,17 +33,25 @@ class TestOverlay(unittest.TestCase):
         self.assertIsNone(overlay.resolve("no/such", self.commons, self.prop))
 
     def test_discover_marks_sources_and_disabled(self):
-        _write(self.commons, "configure/ufw.py", ACT.format(s="commons ufw", r="C"))
+        _write(self.commons, "config/ufw.py", ACT.format(s="commons ufw", r="C"))
-        _write(self.commons, "sync/repository.py", ACT.format(s="sync repo", r="C"))
+        _write(self.commons, "repository/sync.py", ACT.format(s="sync repo", r="C"))
-        _write(self.prop, "configure/ufw.py", ACT.format(s="local ufw", r="L"))
+        _write(self.prop, "config/ufw.py", ACT.format(s="local ufw", r="L"))
         _write(self.prop, "sync/minecraft.py",
                'from engine.descriptor import Meta\nMETADATA = Meta(summary="mc")\nDISABLED = "wip"\ndef run(ctx, args): pass\n')
         infos = {i.name: i for i in overlay.discover(self.commons, self.prop)}
-        self.assertEqual(infos["configure/ufw"].source, "override")
+        self.assertEqual(infos["config/ufw"].source, "override")
-        self.assertEqual(infos["sync/repository"].source, "commons")
+        self.assertEqual(infos["repository/sync"].source, "commons")
         self.assertEqual(infos["sync/minecraft"].source, "local")
         self.assertEqual(infos["sync/minecraft"].disabled, "wip")
-        self.assertEqual(infos["configure/ufw"].summary, "local ufw")
+        self.assertEqual(infos["config/ufw"].summary, "local ufw")
+
+    def test_only_entity_action_paths_can_be_listed_and_dispatched(self):
+        for name in ("configure", "config/nested/tool", "config/tool"):
+            _write(self.prop, name + ".py", ACT.format(s="test", r="ok"))
+        names = [item.name for item in overlay.discover(self.commons, self.prop)]
+        self.assertEqual(names, ["config/tool"])
+        for name in ("configure", "config/nested/tool", "config/../configure"):
+            self.assertIsNone(overlay.resolve(name, self.commons, self.prop))
 
     def test_load_module_runs(self):
         p = _write(self.commons, "x/y.py", ACT.format(s="s", r="Z"))

tests/test_sync_packages.py

@@ -0,0 +1,320 @@
+import json
+import subprocess
+import tempfile
+from pathlib import Path
+import unittest
+from types import SimpleNamespace
+from unittest.mock import Mock, patch
+
+from importlib import import_module
+
+packages = import_module("actions.util.check-updates")
+
+
+def digest(char):
+    return 'sha256:' + char * 64
+
+
+MANIFEST = 'application/vnd.oci.image.manifest.v1+json'
+INDEX = 'application/vnd.oci.image.index.v1+json'
+
+
+class Commands:
+    def __init__(self):
+        self.containers = [{'Image': digest('a'), 'Reference': 'alpine:3.21'}]
+        self.local = {digest('a'): {'Id': digest('a'), 'Os': 'linux',
+                                  'Architecture': 'amd64', 'RepoDigests': ['alpine@' + digest('b')]}}
+        self.remote = {'alpine:3.21': {'mediaType': MANIFEST,
+                                     'digest': digest('c')}}
+        self.raw = {'alpine@' + digest('c'): {'schemaVersion': 2, 'mediaType': MANIFEST,
+                                            'config': {'digest': digest('a')}, 'layers': []}}
+        self.config = {'alpine@' + digest('c'): {'os': 'linux', 'architecture': 'amd64'}}
+        self.calls = []
+        self.fail = set()
+        self.apt = ''
+        self.compose = {"name": "apex", "services": {}}
+        self.compose_options = []
+
+    def ok(self, cmd):
+        self.calls.append(cmd)
+        return tuple(cmd) not in self.fail
+
+    def run(self, cmd, check=False, capture=False, **kwargs):
+        self.calls.append(cmd)
+        if cmd[:1] == ['timeout']:
+            cmd = cmd[4:]
+        if tuple(cmd) in self.fail:
+            return subprocess.CompletedProcess(cmd, 1, '', 'registry unavailable')
+        if cmd[:2] == ['docker', 'compose']:
+            self.compose_options.append(kwargs)
+            data = self.compose if isinstance(self.compose, str) else json.dumps(self.compose)
+        elif cmd[:2] == ['bash', '-c']:
+            if 'apt-get --just-print' not in cmd[2]:
+                raise AssertionError('Image references must never reach a shell')
+            data = self.apt
+        elif cmd == ['docker', 'ps', '--quiet', '--no-trunc']:
+            data = '\n'.join(str(i) for i in range(len(self.containers)))
+        elif cmd[:3] == ['docker', 'container', 'inspect']:
+            data = '\n'.join(json.dumps(c) for c in self.containers)
+        elif cmd[:3] == ['docker', 'image', 'inspect']:
+            data = json.dumps([self.local[cmd[3]]])
+        elif cmd == ['docker', 'buildx', 'version']:
+            data = 'github.com/docker/buildx v0.37.1'
+        elif cmd[:4] == ['docker', 'buildx', 'imagetools', 'inspect']:
+            table = self.raw if cmd[4] == '--raw' else (
+                self.remote if cmd[5] == '{{json .Manifest}}' else self.config)
+            value = table[cmd[-1]]
+            data = value if isinstance(value, str) else json.dumps(value)
+        else:
+            raise AssertionError(f'Unexpected command: {cmd}')
+        return subprocess.CompletedProcess(cmd, 0, data, '')
+
+
+class TestSyncPackages(unittest.TestCase):
+    def setUp(self):
+        self.commands = Commands()
+        self.ctx = SimpleNamespace(sys=self.commands, log=Mock(), node=SimpleNamespace(name='node'),
+                                   notify=Mock())
+        self.ctx.notify.info.return_value = True
+        self.tmp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.tmp.cleanup)
+        self.ctx.paths = SimpleNamespace(compositions=self.tmp.name)
+        self.ctx.vars = lambda: {'APEX_NODE_HOST': 'test-node'}
+        self.args = SimpleNamespace(telegram_bot_url='test-notifier')
+
+    def report(self):
+        with patch.object(packages.os.path, 'isfile', return_value=True), \
+                patch.object(packages.shutil, 'which', return_value='/usr/bin/docker'):
+            packages.run(self.ctx, self.args)
+        return self.ctx.notify.info.call_args.args[2]
+
+    def containerd(self, manifest='c'):
+        local = self.commands.local.pop(digest('a'))
+        local.update(Id=digest('b'), Descriptor={'mediaType': INDEX, 'digest': digest('b')})
+        self.commands.local[digest('b')] = local
+        self.commands.containers[0].update(Image=digest('b'), ImageManifestDescriptor={
+            'mediaType': MANIFEST, 'digest': digest(manifest),
+            'platform': {'os': 'linux', 'architecture': 'amd64'}})
+
+    def test_containerd_index_id_compares_actual_platform_manifest(self):
+        self.containerd()
+        self.index()
+        self.assertIn('1 current', self.report())
+
+    def test_containerd_single_manifest_is_current_despite_different_config_digest(self):
+        self.containerd()
+        self.assertIn('1 current', self.report())
+
+    def test_containerd_changed_platform_manifest_is_update(self):
+        self.containerd('d')
+        self.index()
+        self.assertIn('1 updates', self.report())
+
+    def test_containerd_without_platform_descriptor_is_unchecked(self):
+        self.containerd()
+        del self.commands.containers[0]['ImageManifestDescriptor']
+        self.assertIn('1 unchecked', self.report())
+
+    def test_two_containerd_platforms_of_one_index_are_checked_separately(self):
+        self.containerd()
+        self.index()
+        self.commands.containers.append({
+            'Image': digest('b'), 'Reference': 'alpine:3.21', 'ImageManifestDescriptor': {
+                'mediaType': MANIFEST, 'digest': digest('d'),
+                'platform': {'os': 'linux', 'architecture': 'arm64'}}})
+        self.commands.raw['alpine@' + digest('d')] = {
+            'schemaVersion': 2, 'mediaType': MANIFEST, 'config': {'digest': digest('f')}}
+        self.assertIn('2 current', self.report())
+
+    def test_invalid_container_platform_descriptor_is_unchecked(self):
+        self.containerd()
+        self.commands.containers[0]['ImageManifestDescriptor']['digest'] = 'invalid'
+        self.assertIn('1 unchecked', self.report())
+
+    def build_service(self, image='alpine:3.21'):
+        core = Path(self.tmp.name) / 'apex'
+        core.mkdir()
+        for filename in ('docker-compose.yml', '.env', 'apex.env'):
+            (core / filename).touch()
+        self.commands.compose = {'name': 'apex', 'services': {
+            'custom': {'build': {'context': '.'}, 'environment': {'TOKEN': 'private-test-value'}}}}
+        if image is not None:
+            self.commands.compose['services']['custom']['image'] = image
+        self.commands.containers[0]['Labels'] = {
+            'com.docker.compose.project': 'apex', 'com.docker.compose.service': 'custom'}
+
+    def test_declared_compose_build_with_synthetic_repo_digest_is_local(self):
+        self.containerd()
+        self.build_service()
+        self.assertIn('1 local/untracked', self.report())
+        self.assertFalse(any('imagetools' in c for c in self.commands.calls))
+        self.assertIn(['timeout', '--signal=TERM', '--kill-after=5s', '30s', 'docker',
+                       'compose', '--env-file', '.env', '--env-file', 'apex.env',
+                       'config', '--format', 'json'], self.commands.calls)
+        self.assertEqual(self.commands.compose_options[0]['env']['APEX_NODE_HOST'], 'test-node')
+        self.assertNotIn('private-test-value', str(self.ctx.log.mock_calls))
+
+    def test_implicit_compose_build_image_uses_service_provenance(self):
+        self.containerd()
+        self.build_service(image=None)
+        self.assertIn('1 local/untracked', self.report())
+
+    def test_changed_build_declaration_does_not_hide_old_registry_image(self):
+        self.build_service(image='different:latest')
+        self.assertIn('1 current', self.report())
+
+    def test_same_image_name_without_build_service_provenance_is_registry_checked(self):
+        self.build_service()
+        self.commands.containers[0]['Labels']['com.docker.compose.service'] = 'other'
+        self.assertIn('1 current', self.report())
+
+    def test_failed_compose_resolution_is_unchecked_without_secret_output(self):
+        self.build_service()
+        self.commands.compose = 'private-test-value'
+        self.assertIn('Images: unchecked (Compose inspection failed)', self.report())
+        self.assertNotIn('private-test-value', str(self.ctx.log.mock_calls))
+
+    def test_matching_config_digest_is_current_without_comparing_index_digest(self):
+        self.assertIn('1 current', self.report())
+        self.assertIn(['docker', 'image', 'inspect', digest('a')], self.commands.calls)
+
+    def test_moved_tag_compares_running_image_not_new_local_tag(self):
+        self.commands.raw['alpine@' + digest('c')]['config']['digest'] = digest('d')
+        self.assertIn('1 updates', self.report())
+        self.assertNotIn(['docker', 'image', 'inspect', 'alpine:3.21'], self.commands.calls)
+
+    def index(self, variants=None):
+        self.commands.remote['alpine:3.21'] = {'schemaVersion': 2, 'mediaType': INDEX,
+            'digest': digest('e'), 'manifests': variants or [
+                {'mediaType': MANIFEST, 'digest': digest('d'),
+                 'platform': {'os': 'linux', 'architecture': 'arm64'}},
+                {'mediaType': MANIFEST, 'digest': digest('c'),
+                 'platform': {'os': 'linux', 'architecture': 'amd64'}},
+                {'mediaType': MANIFEST, 'digest': digest('f'),
+                 'platform': {'os': 'unknown', 'architecture': 'unknown'}}]}
+
+    def test_index_only_other_architecture_changed_is_current(self):
+        self.index()
+        self.assertIn('1 current', self.report())
+        self.assertFalse(any(c[-1] == 'alpine@' + digest('d') for c in self.commands.calls))
+
+    def test_index_matching_platform_changed_is_update(self):
+        self.index()
+        self.commands.raw['alpine@' + digest('c')]['config']['digest'] = digest('d')
+        self.assertIn('1 updates', self.report())
+
+    def test_ambiguous_or_missing_platform_is_unchecked(self):
+        for platforms in ([{'os': 'linux', 'architecture': 'arm64'}],
+                          [{'os': 'linux', 'architecture': 'amd64'}] * 2):
+            with self.subTest(platforms=platforms):
+                self.index([{'mediaType': MANIFEST, 'digest': digest('c'), 'platform': p}
+                            for p in platforms])
+                self.assertIn('1 unchecked', self.report())
+
+    def test_arm64_default_variant_matches_v8(self):
+        self.commands.local[digest('a')]['Architecture'] = 'arm64'
+        self.index([{'mediaType': MANIFEST, 'digest': digest('c'),
+                     'platform': {'os': 'linux', 'architecture': 'arm64', 'variant': 'v8'}}])
+        self.assertIn('1 current', self.report())
+
+    def test_single_manifest_different_platform_is_unchecked(self):
+        self.commands.raw['alpine@' + digest('c')]['config']['digest'] = digest('d')
+        self.commands.config['alpine@' + digest('c')]['architecture'] = 'arm64'
+        self.assertIn('1 unchecked', self.report())
+
+    def test_registry_port_and_tag_are_preserved(self):
+        ref = 'registry.example:5000/team/app:stable'
+        self.commands.containers[0]['Reference'] = ref
+        self.commands.remote[ref] = self.commands.remote['alpine:3.21']
+        self.commands.raw['registry.example:5000/team/app@' + digest('c')] = self.commands.raw['alpine@' + digest('c')]
+        self.assertIn('1 current', self.report())
+        self.assertIn(['timeout', '--signal=TERM', '--kill-after=5s', '30s',
+                       'docker', 'buildx', 'imagetools', 'inspect', '--raw',
+                       'registry.example:5000/team/app@' + digest('c')], self.commands.calls)
+
+    def test_registry_port_without_tag_uses_the_whole_repository(self):
+        ref = 'localhost:5000/team/app'
+        self.commands.containers[0]['Reference'] = ref
+        self.commands.remote[ref] = self.commands.remote['alpine:3.21']
+        self.commands.raw[ref + '@' + digest('c')] = self.commands.raw['alpine@' + digest('c')]
+        self.assertIn('1 current', self.report())
+
+    def test_two_running_versions_of_one_tag_are_both_checked(self):
+        self.commands.containers.append({'Image': digest('d'), 'Reference': 'alpine:3.21'})
+        self.commands.local[digest('d')] = {**self.commands.local[digest('a')], 'Id': digest('d')}
+        message = self.report()
+        self.assertIn('1 current', message)
+        self.assertIn('1 updates', message)
+
+    def test_malformed_local_image_is_unchecked_not_a_local_build(self):
+        del self.commands.local[digest('a')]['RepoDigests']
+        self.assertIn('1 unchecked', self.report())
+
+    def test_untrusted_reference_is_argv_not_shell_input(self):
+        self.commands.containers[0]['Reference'] = "registry.example/app:$(touch /tmp/nope)"
+        self.assertIn('1 unchecked', self.report())
+        self.assertTrue(all('touch' not in c[-1] for c in self.commands.calls if c[0] == 'bash'))
+
+    def test_apt_failure_is_explicit_and_images_still_checked(self):
+        self.commands.fail.add(('sudo', 'apt-get', 'update'))
+        message = self.report()
+        self.assertIn('APT: failed to check', message)
+        self.assertIn('1 current', message)
+
+    def test_pinned_and_local_images_are_explicit_and_not_queried(self):
+        self.commands.containers = [{'Image': digest('a'), 'Reference': 'alpine@' + digest('b')},
+                                    {'Image': digest('d'), 'Reference': 'project-local:latest'}]
+        self.commands.local[digest('d')] = {**self.commands.local[digest('a')],
+                                          'Id': digest('d'), 'RepoDigests': []}
+        message = self.report()
+        self.assertIn('1 pinned', message)
+        self.assertIn('1 local/untracked', message)
+        self.assertFalse(any('imagetools' in c for c in self.commands.calls))
+
+    def test_registry_timeout_is_bounded_and_reported_unchecked(self):
+        original = self.commands.run
+
+        def time_out(cmd, **kwargs):
+            if 'imagetools' in cmd:
+                self.assertEqual(cmd[:4], ['timeout', '--signal=TERM', '--kill-after=5s', '30s'])
+                return subprocess.CompletedProcess(cmd, 124, '', '')
+            return original(cmd, **kwargs)
+
+        self.commands.run = time_out
+        self.assertIn('1 unchecked', self.report())
+
+    def test_failed_registry_and_malformed_results_are_unchecked(self):
+        for value in ('not-json', '{}', '[]', '{"mediaType":"unknown"}'):
+            with self.subTest(value=value):
+                self.commands.remote['alpine:3.21'] = value
+                self.assertIn('1 unchecked', self.report())
+        self.commands.fail.add(('docker', 'buildx', 'imagetools', 'inspect', '--format',
+                                '{{json .Manifest}}', 'alpine:3.21'))
+        self.assertIn('1 unchecked', self.report())
+
+    def test_bad_config_digest_is_unchecked(self):
+        self.commands.raw['alpine@' + digest('c')]['config'] = {}
+        self.assertIn('1 unchecked', self.report())
+
+    def test_docker_listing_failure_is_not_empty_success(self):
+        self.commands.fail.add(('docker', 'ps', '--quiet', '--no-trunc'))
+        self.assertIn('unchecked', self.report())
+
+    def test_buildx_missing_still_reports_unchecked(self):
+        self.commands.fail.add(('docker', 'buildx', 'version'))
+        self.assertIn('1 unchecked', self.report())
+
+    def test_image_counts_precede_long_package_details_and_notification_failure_exits(self):
+        self.commands.apt = '\n'.join('package-' + str(i) for i in range(100))
+        message = self.report()
+        self.assertIn('Images:', message[:200])
+        self.assertIn('1 current', message[:200])
+        self.assertIn('100 pending', message[:200])
+        self.assertNotIn('Skopeo', message)
+        self.ctx.notify.info.return_value = False
+        with self.assertRaises(SystemExit):
+            self.report()
+
+
+if __name__ == '__main__':
+    unittest.main()

tests/test_sync_repository.py

@@ -0,0 +1,133 @@
+import os
+import subprocess
+import tempfile
+import unittest
+from pathlib import Path
+from types import SimpleNamespace
+from unittest.mock import Mock
+from actions.repository import sync as repository
+from engine.lib.sys import Sys
+
+
+class RepositorySyncTests(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.temp.cleanup)
+        self.root = Path(self.temp.name)
+        self.remote = self.root / "remote.git"
+        self.repo = self.root / "node"
+        self.env = dict(os.environ, GIT_AUTHOR_NAME="Test", GIT_AUTHOR_EMAIL="test@example.invalid",
+                        GIT_COMMITTER_NAME="Test", GIT_COMMITTER_EMAIL="test@example.invalid",
+                        GIT_ALLOW_PROTOCOL="file")
+        self.git("init", "--bare", str(self.remote))
+        self.git("clone", str(self.remote), str(self.repo))
+        self.git("checkout", "-b", "main", cwd=self.repo)
+        (self.repo / "compositions").mkdir()
+        (self.repo / "compositions/service").write_text("base\n")
+        self.commons_source = self.root / "shared"
+        self.git("init", str(self.commons_source))
+        (self.commons_source / "file").write_text("shared\n")
+        self.git("add", ".", cwd=self.commons_source)
+        self.git("commit", "-m", "shared", cwd=self.commons_source)
+        self.git("submodule", "add", str(self.commons_source), "commons", cwd=self.repo)
+        self.git("add", ".", cwd=self.repo)
+        self.git("commit", "-m", "base", cwd=self.repo)
+        self.git("push", "origin", "main", cwd=self.repo)
+        self.log = Mock()
+        class Commands(Sys):
+            def run(inner, cmd, **kw):
+                kw["env"] = dict(self.env, **kw.get("env", {}))
+                kw["env"]["GIT_ALLOW_PROTOCOL"] = "file"
+                return super().run(cmd, **kw)
+        self.ctx = SimpleNamespace(log=self.log, sys=Commands(self.log), notify=Mock(),
+                                   node=SimpleNamespace(name="node"),
+                                   paths=SimpleNamespace(repo_root=str(self.repo), commons=str(self.repo / "commons"),
+                                                         compositions=str(self.repo / "compositions")))
+        self.args = SimpleNamespace(telegram_bot_url=None)
+
+    def git(self, *args, cwd=None):
+        return subprocess.check_output(["git", *args], cwd=cwd, env=self.env, stderr=subprocess.PIPE, text=True).strip()
+
+    def sync(self):
+        repository.run(self.ctx, self.args)
+
+    def test_local_change_publishes_main_without_sync_branch_or_notification(self):
+        (self.repo / "compositions/service").write_text("changed\n")
+        self.sync()
+        self.assertEqual(self.git("--git-dir=" + str(self.remote), "show", "main:compositions/service"), "changed")
+        self.assertEqual(self.git("--git-dir=" + str(self.remote), "for-each-ref", "--format=%(refname)", "refs/heads"), "refs/heads/main")
+        self.ctx.notify.success.assert_not_called()
+        self.ctx.notify.error.assert_not_called()
+
+    def test_noop_keeps_head(self):
+        before = self.git("rev-parse", "HEAD", cwd=self.repo)
+        self.sync()
+        self.assertEqual(before, self.git("rev-parse", "HEAD", cwd=self.repo))
+
+    def test_other_branch_does_not_stage_changes(self):
+        self.git("checkout", "-b", "work", cwd=self.repo)
+        (self.repo / "compositions/service").write_text("changed\n")
+        with self.assertRaises(SystemExit):
+            self.sync()
+        self.assertEqual(self.git("diff", "--cached", cwd=self.repo), "")
+
+    def test_dirty_commons_blocks_parent_commit(self):
+        before = self.git("rev-parse", "HEAD", cwd=self.repo)
+        (self.repo / "commons/file").write_text("dirty\n")
+        with self.assertRaises(SystemExit):
+            self.sync()
+        self.assertEqual(before, self.git("rev-parse", "HEAD", cwd=self.repo))
+
+    def test_stale_submodule_after_failed_update_cannot_rollback_pin(self):
+        old = self.git("rev-parse", "HEAD", cwd=self.repo / "commons")
+        (self.commons_source / "file").write_text("new shared\n")
+        self.git("add", ".", cwd=self.commons_source)
+        self.git("commit", "-m", "new shared", cwd=self.commons_source)
+        new = self.git("rev-parse", "HEAD", cwd=self.commons_source)
+        self.git("-C", str(self.repo / "commons"), "fetch", "origin")
+        self.git("update-index", "--cacheinfo", "160000," + new + ",commons", cwd=self.repo)
+        self.git("commit", "-m", "advance pin", cwd=self.repo)
+        before = self.git("rev-parse", "HEAD", cwd=self.repo)
+        with self.assertRaises(SystemExit):
+            self.sync()
+        self.assertEqual(before, self.git("rev-parse", "HEAD", cwd=self.repo))
+        self.assertEqual(old, self.git("rev-parse", "HEAD", cwd=self.repo / "commons"))
+        self.git("submodule", "update", "--init", cwd=self.repo)
+        self.sync()
+        self.assertEqual(new, self.git("--git-dir=" + str(self.remote), "rev-parse", "main:commons"))
+
+    def test_concurrent_main_advance_rejects_push_without_overwriting(self):
+        original = self.ctx.sys.run
+        other = self.root / "other"
+        self.git("clone", "--branch", "main", str(self.remote), str(other))
+        def race(cmd, **kw):
+            if "push" in cmd:
+                (other / "compositions/service").write_text("concurrent\n")
+                self.git("add", ".", cwd=other)
+                self.git("commit", "-m", "concurrent", cwd=other)
+                self.git("push", "origin", "main", cwd=other)
+            return original(cmd, **kw)
+        self.ctx.sys.run = race
+        (self.repo / "compositions/service").write_text("local\n")
+        with self.assertRaises(SystemExit):
+            self.sync()
+        self.assertEqual("concurrent", self.git("--git-dir=" + str(self.remote), "show", "main:compositions/service"))
+        self.assertEqual("local", self.git("show", "HEAD:compositions/service", cwd=self.repo))
+
+    def test_conflict_preserves_local_commit_and_remote(self):
+        other = self.root / "other"
+        self.git("clone", "--branch", "main", str(self.remote), str(other))
+        (other / "compositions/service").write_text("remote\n")
+        self.git("add", ".", cwd=other)
+        self.git("commit", "-m", "remote", cwd=other)
+        self.git("push", "origin", "main", cwd=other)
+        (self.repo / "compositions/service").write_text("local\n")
+        with self.assertRaises(SystemExit):
+            self.sync()
+        self.assertEqual(self.git("show", "HEAD:compositions/service", cwd=self.repo), "local")
+        self.assertEqual(self.git("--git-dir=" + str(self.remote), "show", "main:compositions/service"), "remote")
+        self.assertFalse((self.repo / ".git/rebase-merge").exists())
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_timezone.py

@@ -0,0 +1,72 @@
+import importlib.util
+from pathlib import Path
+import tempfile
+from types import SimpleNamespace
+import unittest
+from unittest.mock import Mock
+
+ROOT = Path(__file__).resolve().parents[1]
+spec = importlib.util.spec_from_file_location('timezone_action', ROOT / 'actions/config/timezone.py')
+action = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(action)
+
+
+class TestTimezone(unittest.TestCase):
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.tmp.cleanup)
+        self.source = Path(self.tmp.name) / 'timezone'
+        self.zone = 'UTC'
+        self.cron_refreshes = []
+        def read(*args, **kwargs):
+            return SimpleNamespace(stdout=self.zone+'\n')
+        def apply(args):
+            self.zone = args[-1]
+        self.system = Mock()
+        self.system.run.side_effect = read
+        self.system.sudo.side_effect = apply
+        self.system.service_is_active.return_value = True
+        def restart(name):
+            self.cron_refreshes.append(name)
+            return True
+        self.system.restart.side_effect = restart
+        self.ctx = SimpleNamespace(paths=SimpleNamespace(configs=self.tmp.name), sys=self.system, log=Mock())
+
+    def test_timezone_transition_refreshes_cron_once_and_repeat_is_noop(self):
+        self.source.write_text('Europe/Moscow\n')
+        action.run(self.ctx, SimpleNamespace(dry_run=False))
+        self.assertEqual(self.zone, 'Europe/Moscow')
+        self.assertEqual(self.cron_refreshes, ['cron'])
+        action.run(self.ctx, SimpleNamespace(dry_run=False))
+        self.assertEqual(self.cron_refreshes, ['cron'])
+
+    def test_preview_and_absent_config_preserve_host(self):
+        action.run(self.ctx, SimpleNamespace(dry_run=False))
+        self.source.write_text('Europe/Moscow\n')
+        action.run(self.ctx, SimpleNamespace(dry_run=True))
+        self.assertEqual(self.zone, 'UTC')
+        self.assertEqual(self.cron_refreshes, [])
+        self.system.sudo.assert_not_called()
+
+    def test_failed_cron_restart_is_reported_as_failure(self):
+        self.source.write_text('Europe/Moscow\n')
+        self.system.restart.side_effect = None
+        self.system.restart.return_value = False
+        with self.assertRaises(SystemExit):
+            action.run(self.ctx, SimpleNamespace(dry_run=False))
+        self.ctx.log.success.assert_not_called()
+        self.assertEqual(self.zone, 'Europe/Moscow')
+
+    def test_inactive_cron_is_not_started(self):
+        self.source.write_text('Europe/Moscow\n')
+        self.system.service_is_active.return_value = False
+        action.run(self.ctx, SimpleNamespace(dry_run=False))
+        self.assertEqual(self.zone, 'Europe/Moscow')
+        self.assertEqual(self.cron_refreshes, [])
+
+    def test_invalid_zone_does_not_mutate_host(self):
+        self.source.write_text('../not-a-zone\n')
+        with self.assertRaises(SystemExit):
+            action.run(self.ctx, SimpleNamespace(dry_run=False))
+        self.system.sudo.assert_not_called()
+        self.assertEqual(self.cron_refreshes, [])

tests/test_tooling.py

@@ -0,0 +1,76 @@
+"""Exercise verification entrypoints in disposable source trees."""
+import os
+from pathlib import Path
+import shutil
+import subprocess
+import sys
+import tempfile
+import unittest
+
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+class ToolingTests(unittest.TestCase):
+    def test_lint_fallback_recurses_and_propagates_each_root_failure(self):
+        with tempfile.TemporaryDirectory() as directory:
+            root = Path(directory)
+            shutil.copyfile(ROOT / 'Makefile', root / 'Makefile')
+            binaries = root / 'bin'
+            binaries.mkdir()
+            make = shutil.which('make')
+            self.assertIsNotNone(make, 'make is required for the declared verification workflow')
+            (binaries / 'make').symlink_to(make)
+            (binaries / 'python3').symlink_to(sys.executable)
+            environment = dict(os.environ, PATH=str(binaries))
+            sources = []
+            (root / 'scripts').mkdir()
+            (root / 'scripts/bootstrap-node').write_text('print(1)\n')
+            for name in ('actions', 'tests', 'scripts'):
+                source = root / name / 'nested' / 'case.py'
+                source.parent.mkdir(parents=True)
+                source.write_text('value = 1\n')
+                sources.append(source)
+            for source in sources:
+                with self.subTest(root=source.parent.parent):
+                    invalid = source.with_name('invalid.py')
+                    invalid.write_text('def broken(:\n')
+                    result = subprocess.run([str(binaries / 'make'), 'lint'], cwd=root,
+                                            env=environment, capture_output=True, text=True,
+                                            timeout=30)
+                    self.assertNotEqual(result.returncode, 0, result.stdout + result.stderr)
+                    self.assertNotIn('verified python bytecode compilation', result.stdout)
+                    invalid.unlink()
+            result = subprocess.run([str(binaries / 'make'), 'lint'], cwd=root,
+                                    env=environment, capture_output=True, text=True, timeout=30)
+            self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+            self.assertIn('verified python bytecode compilation', result.stdout)
+
+    def test_suite_runner_discovers_both_suites_and_propagates_failures(self):
+        with tempfile.TemporaryDirectory() as directory:
+            root = Path(directory)
+            suites = (root / 'tests', root / 'tests/provisioning')
+            for suite in suites:
+                suite.mkdir(parents=True)
+            shutil.copyfile(ROOT / 'tests/run.sh', suites[0] / 'run.sh')
+            binaries = root / 'bin'
+            binaries.mkdir()
+            (binaries / 'python3').symlink_to(sys.executable)
+            environment = dict(os.environ, PATH=str(binaries) + os.pathsep + os.environ['PATH'])
+            for failure in (None, 0, 1):
+                with self.subTest(failing_suite=failure):
+                    for index, suite in enumerate(suites):
+                        (suite / 'test_case.py').write_text(
+                            'import unittest\nclass Case(unittest.TestCase):\n'
+                            '    def test_suite_' + str(index) + '(self):\n'
+                            '        self.assertTrue(' + repr(index != failure) + ')\n')
+                    result = subprocess.run(['/bin/bash', 'tests/run.sh'], cwd=root,
+                                            env=environment, capture_output=True, text=True,
+                                            timeout=30)
+                    if failure is None:
+                        self.assertEqual(result.returncode, 0, result.stderr)
+                        self.assertIn('test_suite_0', result.stderr)
+                        self.assertIn('test_suite_1', result.stderr)
+                    else:
+                        self.assertNotEqual(result.returncode, 0, result.stderr)
+                        self.assertIn('FAIL: test_suite_' + str(failure), result.stderr)

tests/test_tpl.py

@@ -1,22 +1,31 @@
 # tests/test_tpl.py
-import os, tempfile, unittest
+import os
-from engine.lib.tpl import Tpl
+import shutil
+import tempfile
+import unittest
+
 from engine.lib.log import Log
+from engine.lib.tpl import Tpl
+
 
 class TestTpl(unittest.TestCase):
     def setUp(self):
         self.cfg = tempfile.mkdtemp()
         self.tpl = Tpl(self.cfg, Log())
 
+    def tearDown(self):
+        shutil.rmtree(self.cfg, ignore_errors=True)
+
     def _w(self, rel, body):
         p = os.path.join(self.cfg, rel)
         os.makedirs(os.path.dirname(p), exist_ok=True)
-        open(p, "w").write(body)
+        with open(p, "w", encoding="utf-8") as f:
+            f.write(body)
 
     def test_render_substitutes_known_vars(self):
         self._w("cron/crontab", "root=${APEX_TIER1} node=$APEX_NODE\n")
-        out = self.tpl.render("cron/crontab", {"APEX_TIER1": "/mnt/t1", "APEX_NODE": "daedalus"})
+        out = self.tpl.render("cron/crontab", {"APEX_TIER1": "/mnt/t1", "APEX_NODE": "node1"})
-        self.assertEqual(out, "root=/mnt/t1 node=daedalus\n")
+        self.assertEqual(out, "root=/mnt/t1 node=node1\n")
 
     def test_render_leaves_shell_command_substitution(self):
         self._w("c", "x $(cat /f) ${APEX_NODE}\n")
@@ -29,5 +38,6 @@ class TestTpl(unittest.TestCase):
         out = self.tpl.render_dir("fail2ban", {"X": "1", "Y": "2"})
         self.assertEqual(out, {"jail.local": "a=1\n", "sub/extra.conf": "b=2\n"})
 
+
 if __name__ == "__main__":
     unittest.main()

tests/test_ufw.py

@@ -0,0 +1,27 @@
+import unittest
+import os
+import shutil
+from engine.lib.ufw import resolve_ufw_docker_bin
+
+class TestUfwDockerResolution(unittest.TestCase):
+    def test_prefers_path_or_usr_local_bin_when_present(self):
+        bin_path = resolve_ufw_docker_bin()
+        # Should return a valid string path
+        self.assertIsInstance(bin_path, str)
+        self.assertTrue(len(bin_path) > 0)
+
+    def test_fallback_path(self):
+        # If not in PATH, should fall back to user local bin
+        original_which = shutil.which
+        original_isfile = os.path.isfile
+        try:
+            shutil.which = lambda cmd: None
+            os.path.isfile = lambda p: False if p == "/usr/local/bin/ufw-docker" else original_isfile(p)
+            expected = os.path.expanduser("~/.local/bin/ufw-docker")
+            self.assertEqual(resolve_ufw_docker_bin(), expected)
+        finally:
+            shutil.which = original_which
+            os.path.isfile = original_isfile
+
+if __name__ == "__main__":
+    unittest.main()

vendor/VENDORED.md

@@ -1,13 +0,0 @@
-# Vendored third-party libraries
-
-Runtime is **git + python3 stdlib only**. This directory is currently **empty**.
-
-If a pure-python library ever becomes truly necessary:
-1. It must be **pure python** (no C extensions — there is no build step).
-2. Check it into `vendor/<pkg>/`. The `apex` launcher adds `vendor/` to `sys.path`.
-3. Add one record row below: `name, version, source URL, git SHA, sha256`.
-4. Never `pip install` at runtime. Updates are a manual re-vendor + record bump.
-
-| name | version | source | git sha | sha256 |
-|------|---------|--------|---------|--------|
-| _(none)_ | | | | |