fix: allow non-Compose commits on Docker-free hosts [skip ci]

- Purge misplaced RFCs, scratch tokens, and node-proprietary assets
- Decouple remote provisioning to ci/scripts/init-node.py
- Add targeted composition orchestration in actions/compose.py
- Add host-side Xray stale socket cleanup in /dev/shm
- Add actions/env.py for shell exports and .env synchronization
- Harden UFW, backup retention, host tempfiles, and notify dispatcher
- Add pyproject.toml and Makefile with 78 passing unit tests

.agents/skills/update-compositions/SKILL.md

@@ -0,0 +1,234 @@
+---
+name: update-compositions
+description: Use when updating, upgrading, or bumping Docker container images, tags, composition services, and OS packages across one or more APEX fleet nodes.
+---
+
+# Updating APEX Compositions & System Packages
+
+This skill guides the safe, structured update of container images, compose definitions, and base operating system packages across nodes in an APEX fleet.
+
+## Core Rules
+
+1. **Verify access first**: Never execute commands against a node without confirming SSH connectivity and credentials.
+2. **Hold critical services**: Always ask the user which services or compositions must NOT be touched before discovering or updating images.
+3. **Classify update scope**: Separate updates into **Minor/Patch** (non-breaking), **Major** (breaking / schema migration risks), and **Unstable SemVer / CalVer** (requiring changelog scrutiny).
+4. **Gate major and breaking updates**: Never apply a Major version bump, SemVer `0.x` minor change, or CalVer jump without migration research and explicit user approval.
+5. **Preserve APEX engine context**: Always orchestrate compose actions through `./commons/apex compose` (or with derived `APEX_*` variables and layered env files) rather than bare `docker compose`.
+6. **Safeguard in-tree patches**: Do not bump images for containers with host bind mounts that patch application source code without explicit review.
+7. **Verify fleet observability post-rollout**: An update is not complete until containers reach `Up` and `healthy`, logs are clean, Prometheus scrape targets report `UP`, blackbox probes pass, and remote write streams are established.
+
+---
+
+## Workflow Steps
+
+### Step 1: Confirm Node Access
+
+1. Identify the target nodes from the user request.
+2. If access credentials, custom SSH ports, or key paths are not declared in the environment's SSH configuration, confirm:
+   - SSH hostname / IP
+   - SSH port (APEX default: `2222`)
+   - SSH user
+   - Identity file / key
+3. Test connectivity to each node before proceeding:
+   ```bash
+   ssh -p <port> <user>@<node_fqdn> "hostname && uptime"
+   ```
+4. **Completion check**: All target nodes return exit code 0 on the connectivity probe.
+
+---
+
+### Step 2: Inquire About Critical Services & Holds
+
+1. Prompt the user explicitly before inspecting or modifying any services:
+   > "Which critical services or compositions on these nodes should **NOT** be updated during this run (e.g. databases, SSO/Identity providers, Mail, Vaults, core routing)?"
+2. Record the freeze list (held services/compositions).
+3. Ensure all subsequent steps skip services on the freeze list.
+4. **Completion check**: User response received, and the list of excluded services is confirmed.
+
+---
+
+### Step 3: Inventory Running Services, Images, and Custom Patches
+
+For each target node:
+
+1. List active containers, current images, and status:
+   ```bash
+   ssh <node> "docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}'"
+   ```
+2. Locate the compose files on the node:
+   - Shared core: `/srv/<node_fqdn>/compositions/apex/docker-compose.yml`
+   - Node applications: `/srv/<node_fqdn>/compositions/*/docker-compose.yml`
+   - Overrides: `docker-compose.override.yml` in each composition directory
+3. Map every running container to its compose declaration:
+   - Image repository
+   - Pinned tag / SHA digest
+   - Whether the image is locally built (`build:` context present)
+4. **Inspect for in-tree patches & mounts**:
+   - Check container volume declarations for bind mounts into application code paths (e.g. `./patches/...:/app/...`, monkeypatched modules, or custom entrypoint scripts).
+   - If a service mounts patches into internal software paths, flag it: upstream image updates may alter internal APIs and break the patch. Add to hold or require explicit user confirmation.
+5. Exclude any services identified on the freeze list from Step 2.
+6. **Completion check**: A complete table of candidate compositions, running tags, compose file paths, and in-tree patch status is constructed.
+
+---
+
+### Step 4: Seek Latest Tags and Classify Update Types
+
+For each candidate service:
+
+1. Query the container registry (Docker Hub, GitHub Container Registry `ghcr.io`, Quay, etc.) for available release tags. Refer to [references/registry-queries.md](references/registry-queries.md) for registry API patterns.
+2. Determine the latest stable versioned tag:
+   - Prefer semantic versioned tags matching the existing packaging flavor (e.g. `-alpine`, `-bookworm`).
+   - Ignore pre-release, alpha, beta, or release-candidate (`-rc`) tags unless the running container already pins one.
+   - Do not replace a pinned version tag with bare `:latest`.
+3. Compare the current tag with the latest tag:
+
+| Tag Transition | Category | Classification | Action Path |
+| :--- | :--- | :--- | :--- |
+| `1.37.1` -> `1.37.2` | SemVer `1.x+` | **Patch** | Safe -> Proceed to Step 6 |
+| `1.37.1` -> `1.38.0` | SemVer `1.x+` | **Minor** | Safe -> Proceed to Step 6 |
+| `0.4.1` -> `0.4.2` | SemVer `0.x` | **0.x Patch** | Safe -> Proceed to Step 6 |
+| `0.4.1` -> `0.7.0` | SemVer `0.x` | **0.x Minor** | **Route to Step 5 (Strict Gate)** |
+| `YY.M.D` -> `YY.M.D` | CalVer / Date | **CalVer Jump** | Multi-month jump -> **Route to Step 5** |
+| Floating tag (e.g. `:4`, `:16`) | Floating | **In-place Digest Bump** | Safe -> Proceed to Step 6 |
+| `1.x` -> `2.x` or `16` -> `17` | SemVer `1.x+` | **Major** | **Route to Step 5 (Strict Gate)** |
+| Locally built image (`build:`) | Local Build | **Local Build** | Git update + build (Step 6) |
+
+> [!IMPORTANT]
+> **SemVer 0.x Rule**: Per SemVer specification Clause 4, version `0.x` software has no public API stability guarantees. Minor version bumps (e.g. `0.4` -> `0.7`) often contain breaking changes, renamed environment variables, or schema changes. Treat them with the same scrutiny as Major updates.
+>
+> **CalVer Rule**: Projects using date-based versioning (e.g. Xray, Ubuntu base images) do not follow SemVer breaking change rules. Releases spanning several months must be vetted against upstream changelogs for breaking configuration or default behavior shifts.
+
+4. **Completion check**: Every eligible service has a determined target tag and update classification.
+
+---
+
+### Step 5: Major & Breaking Update Protocol (Strict Gating)
+
+For each service classified as **Major**, **0.x Minor**, or **CalVer Jump**:
+
+1. **Research Upstream Changes**:
+   - Check upstream release notes, migration guides, and changelogs.
+   - Identify:
+     - Breaking configuration changes (renamed environment variables, CLI flags, config format alterations).
+     - Altered default security behaviors (e.g. new access control rules, loopback/private IP blocking, required authentication headers).
+     - Storage schema migrations (e.g., PostgreSQL major upgrade requirements).
+2. **Formulate Migration & Rollback Plan**:
+   - Outline pre-upgrade steps (e.g. explicit logical database dump `pg_dump` / `mysqldump` beyond normal snapshots).
+   - Outline required compose file changes (volumes, environment variables).
+   - Outline fallback/rollback steps if the update fails.
+3. **Request Explicit User Approval**:
+   - Present findings to the user:
+     - Target service and node.
+     - Current version vs New version.
+     - Summary of breaking changes and migration strategy.
+   - Ask: *"Do you approve proceeding with the update for `<service>` on `<node>` under this strategy?"*
+4. **Gate**:
+   - If approved: Proceed to Step 6 for this service.
+   - If rejected or deferred: Add service to the freeze list and leave it untouched.
+5. **Completion check**: Every breaking or major update has documented strategy + user approval or is explicitly skipped.
+
+---
+
+### Step 6: Apply Container Updates via APEX Engine
+
+Follow node and service rollout order to minimize blast radius:
+1. **Node ordering**:
+   - Application & worker nodes first.
+   - Communications, data, and storage nodes next.
+   - Core edge / gateway / SSO nodes last, ensuring routing and auth remain stable during earlier rollouts.
+2. **Service dependency sequencing**:
+   - Infrastructure providers first (DNS resolvers, database servers, mail exchangers).
+   - Verify provider health before updating or restarting dependent consumer services (e.g. SSO forwardAuth, telemetry agents).
+
+For each node:
+
+1. **Pre-upgrade backup**:
+   - Trigger a backup of stateful volumes:
+     ```bash
+     ssh <node> "apex backup/run"
+     ```
+   - For database services being updated, execute an in-container logical export to host storage.
+2. **Update image tags in compose definitions**:
+   - Modify the target `docker-compose.yml`, `docker-compose.override.yml`, or node configuration repository.
+3. **Pull and recreate services via APEX CLI**:
+   - Always run through the node's APEX CLI to preserve derived `APEX_*` variables and layered env files:
+     ```bash
+     ssh <node> "cd /srv/<node_fqdn> && ./commons/apex compose pull"
+     ssh <node> "cd /srv/<node_fqdn> && ./commons/apex compose up -d"
+     ```
+   - *Note on `--force-recreate`*: When updating containers using floating tags, recovering from host reboots, or handling services with `tmpfs` state (e.g. mail spools), pass `--force-recreate` to ensure the container entrypoint runs a clean initialization:
+     ```bash
+     ssh <node> "cd /srv/<node_fqdn> && ./commons/apex compose up -d --force-recreate"
+     ```
+4. **Completion check**: Compose commands finish with exit code 0.
+
+---
+
+### Step 7: Apply System Packages and OS Updates
+
+When the update scope includes host operating system packages:
+
+1. **Pre-upgrade check**:
+   - Check available disk space: `df -h /`
+   - Check for broken package states: `sudo dpkg --audit`
+2. **Handle unattended package configuration (debconf)**:
+   - Prevent interactive prompts (e.g. GRUB drive selection on VPS instances) from blocking non-interactive execution:
+     ```bash
+     # Example for grub-pc on virtualized hosts using /dev/sda:
+     echo 'grub-pc grub-pc/install_devices multiselect /dev/sda' | sudo debconf-set-selections
+     ```
+3. **Execute system package upgrade**:
+   ```bash
+   ssh <node> "sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade -y"
+   ```
+4. **Clean package cache**:
+   ```bash
+   ssh <node> "sudo apt-get autoremove -y && sudo apt-get clean"
+   ```
+5. **Reboot handling & coordination**:
+   - Check if a system reboot is required:
+     ```bash
+     ssh <node> "test -f /var/run/reboot-required && echo 'REBOOT_REQUIRED' || echo 'OK'"
+     ```
+   - If reboot is required:
+     1. Notify the user before initiating the reboot.
+     2. Reboot node: `ssh <node> "sudo reboot"`
+     3. Wait for SSH availability: poll connectivity with timeout.
+     4. After boot, verify systemd state (`systemctl is-system-running`) and check that Docker containers auto-started.
+     5. If services with `tmpfs` mounts fail startup checks (e.g. Postfix spool directories in mail containers), run `./commons/apex compose up -d --force-recreate`.
+6. **Completion check**: Host packages are up to date and node is online.
+
+---
+
+### Step 8: Verify Service Health & Fleet Observability
+
+Immediately after recreation and host reboots on each node:
+
+1. **Inspect container state**:
+   ```bash
+   ssh <node> "docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'"
+   ```
+   - Verify status is `Up`.
+   - Verify healthchecks transition to `(healthy)` and do not hang in `(unhealthy)` or `(starting)`.
+2. **Inspect recent logs**:
+   ```bash
+   ssh <node> "docker logs --tail 50 <container_name>"
+   ```
+   - Check for crash loops, panic traces, database connection errors, or authentication failures.
+3. **Verify HTTP/network response (edge/web services)**:
+   ```bash
+   curl -IfsS -k https://<service_fqdn> || true
+   ```
+4. **Observability & Telemetry Verification**:
+   - If the fleet runs Prometheus:
+     - Check scrape targets: `curl -s http://<prom_internal_ip>:9090/prometheus/api/v1/targets | jq -r '.data.activeTargets[] | "\(.labels.job) | \(.health)"'` — verify all targets report `up`.
+     - Check blackbox probes: `curl -s 'http://<prom_internal_ip>:9090/prometheus/api/v1/query?query=probe_success==0'` — verify 0 failing probes.
+     - Check active alerts: `curl -s http://<prom_internal_ip>:9090/prometheus/api/v1/alerts | jq '.data.alerts[] | select(.state=="firing")'` — verify only expected watchdogs fire.
+     - Check remote write: confirm telemetry agent (e.g. Alloy) is actively shipping host/container metrics without push retry errors.
+5. **Rollback protocol**:
+   - If a container enters a crash loop (`Restarting`, `Exit 1`) or breaks ingress:
+     1. Capture failure logs.
+     2. Revert the image tag in the compose file to the prior tag.
+     3. Re-run `./commons/apex compose up -d --force-recreate`.
+     4. Notify user immediately with logs and root cause.
+6. **Completion check**: All containers are confirmed `Up` and `healthy`, external ingress functions normally, and telemetry metrics/targets are green.

.agents/skills/update-compositions/references/registry-queries.md

@@ -0,0 +1,99 @@
+# Container Registry Tag Queries
+
+This reference details how to discover available image tags across common container registries without requiring external tooling.
+
+## 1. Docker Hub (`hub.docker.com`)
+
+Docker Hub provides an unauthenticated v2 API for public repositories.
+
+### Official Images (Library)
+For top-level images like `postgres`, `redis`, `traefik`, `mariadb`:
+
+```bash
+IMAGE="postgres"
+curl -s "https://registry.hub.docker.com/v2/repositories/library/${IMAGE}/tags?page_size=100" \
+  | python3 -c '
+import sys, json
+data = json.load(sys.stdin)
+tags = [t["name"] for t in data.get("results", [])]
+print("\n".join(tags[:30]))
+'
+```
+
+### Community / User Images
+For namespace images like `adguard/adguardhome`, `prom/prometheus`, `grafana/alloy`:
+
+```bash
+ORG="adguard"
+IMAGE="adguardhome"
+curl -s "https://registry.hub.docker.com/v2/repositories/${ORG}/${IMAGE}/tags?page_size=100" \
+  | python3 -c '
+import sys, json
+data = json.load(sys.stdin)
+tags = [t["name"] for t in data.get("results", [])]
+print("\n".join(tags[:30]))
+'
+```
+
+---
+
+## 2. GitHub Container Registry (`ghcr.io`)
+
+GHCR implements OCI distribution spec. Public image tags can be queried with an anonymous bearer token.
+
+```bash
+IMAGE="xtls/xray-core"
+TOKEN=$(curl -s "https://ghcr.io/token?scope=repository:${IMAGE}:pull" | python3 -c 'import sys, json; print(json.load(sys.stdin)["token"])')
+curl -s -H "Authorization: Bearer ${TOKEN}" "https://ghcr.io/v2/${IMAGE}/tags/list" \
+  | python3 -c '
+import sys, json
+data = json.load(sys.stdin)
+tags = data.get("tags", [])
+print("\n".join(sorted(tags, reverse=True)[:30]))
+'
+```
+
+---
+
+## 3. Quay.io (`quay.io`)
+
+Quay provides a REST API:
+
+```bash
+REPO="repository_name"
+curl -s "https://quay.io/api/v1/repository/${REPO}/tag/?limit=50&onlyActiveTags=true" \
+  | python3 -c '
+import sys, json
+data = json.load(sys.stdin)
+tags = [t["name"] for t in data.get("tags", [])]
+print("\n".join(tags[:30]))
+'
+```
+
+---
+
+## 4. SemVer Matching & Sorting
+
+When filtering tags to find the latest stable version matching the current container's flavor:
+
+```python
+import re
+
+def parse_semver(tag: str):
+    # Match patterns like v1.2.3, 1.2.3-alpine, 16.4
+    m = re.match(r"^v?(\d+)(?:\.(\d+))?(?:\.(\d+))?(?:-(.+))?$", tag)
+    if not m:
+        return None
+    major = int(m.group(1))
+    minor = int(m.group(2) or 0)
+    patch = int(m.group(3) or 0)
+    flavor = m.group(4) or ""
+    return (major, minor, patch, flavor)
+```
+
+Rules for candidate selection:
+1. Preserve flavor: If currently running `16-alpine`, match only tags with suffix `-alpine`.
+2. Ignore pre-releases: Filter out tags containing `alpha`, `beta`, `rc`, `dev`, `test`, or git SHAs.
+3. Classify:
+   - Same `major`, higher `minor` or `patch` -> **Minor / Patch**.
+   - Higher `major` -> **Major** (requires Step 5 approval).

.gitea/actions/download-artifact/LICENSE

@@ -0,0 +1,22 @@
+
+The MIT License (MIT)
+
+Copyright (c) 2018 GitHub, Inc. and contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

.gitea/actions/download-artifact/UPSTREAM.json

@@ -0,0 +1,8 @@
+{
+  "name": "ci-action-gitea-download-artifact",
+  "source": "https://github.com/christopherhx/gitea-download-artifact.git",
+  "requested_ref": "v4",
+  "commit": "75635f32b4c1c41c4b3d64e8f85210112ed4c9c7",
+  "path": "/private/tmp/gitea-migration-20261105/mirrors/ci-action-gitea-download-artifact.git",
+  "destination": "https://git.ermnvldmr.com/deytenit/ci-action-gitea-download-artifact"
+}

.gitea/actions/download-artifact/action.yml

@@ -0,0 +1,40 @@
+name: 'Download a Build Artifact'
+description: 'Download a build artifact that was previously uploaded in the workflow by the upload-artifact action'
+author: 'GitHub'
+inputs:
+  name:
+    description: 'Name of the artifact to download. If unspecified, all artifacts for the run are downloaded.'
+    required: false
+  path:
+    description: 'Destination path. Supports basic tilde expansion. Defaults to $GITHUB_WORKSPACE'
+    required: false
+  pattern:
+    description: 'A glob pattern matching the artifacts that should be downloaded. Ignored if name is specified.'
+    required: false
+  merge-multiple:
+    description: 'When multiple artifacts are matched, this changes the behavior of the destination directories.
+      If true, the downloaded artifacts will be in the same directory specified by path.
+      If false, the downloaded artifacts will be extracted into individual named directories within the specified path.'
+    required: false
+    default: 'false'
+  github-token:
+    description: 'The GitHub token used to authenticate with the GitHub API.
+      This is required when downloading artifacts from a different repository or from a different workflow run.
+      If this is not specified, the action will attempt to download artifacts from the current repository and the current workflow run.'
+    required: false
+  repository:
+    description: 'The repository owner and the repository name joined together by "/".
+      If github-token is specified, this is the repository that artifacts will be downloaded from.'
+    required: false
+    default: ${{ github.repository }}
+  run-id:
+    description: 'The id of the workflow run where the desired download artifact was uploaded from.
+      If github-token is specified, this is the run that artifacts will be downloaded from.'
+    required: false
+    default: ${{ github.run_id }}
+outputs:
+  download-path:
+    description: 'Path of artifact download'
+runs:
+  using: 'node20'
+  main: 'dist/index.js'

.gitea/actions/download-artifact/package.json

@@ -0,0 +1,47 @@
+{
+  "name": "download-artifact",
+  "version": "4.0.1",
+  "description": "Download an Actions Artifact from a workflow run",
+  "main": "dist/index.js",
+  "scripts": {
+    "build": "tsc",
+    "release": "ncc build src/download-artifact.ts && git add -f dist/",
+    "check-all": "concurrently \"npm:format-check\" \"npm:lint\" \"npm:build\"",
+    "format": "prettier --write **/*.ts",
+    "format-check": "prettier --check **/*.ts",
+    "lint": "eslint **/*.ts"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/actions/download-artifact.git"
+  },
+  "keywords": [
+    "Actions",
+    "GitHub",
+    "Artifacts",
+    "Download"
+  ],
+  "author": "GitHub",
+  "license": "MIT",
+  "bugs": {
+    "url": "https://github.com/actions/download-artifact/issues"
+  },
+  "homepage": "https://github.com/actions/download-artifact#readme",
+  "dependencies": {
+    "@actions/artifact": "^2.0.1",
+    "@actions/core": "^1.10.0",
+    "@actions/github": "^5.1.1",
+    "minimatch": "^9.0.3"
+  },
+  "devDependencies": {
+    "@types/node": "^12.12.6",
+    "@typescript-eslint/eslint-plugin": "^6.14.0",
+    "@vercel/ncc": "^0.33.4",
+    "concurrently": "^5.2.0",
+    "eslint": "^8.55.0",
+    "eslint-plugin-github": "^4.10.1",
+    "eslint-plugin-prettier": "^5.0.1",
+    "prettier": "^3.1.1",
+    "typescript": "^5.3.3"
+  }
+}

.gitea/actions/upload-artifact/LICENSE

@@ -0,0 +1,22 @@
+
+The MIT License (MIT)
+
+Copyright (c) 2018 GitHub, Inc. and contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

.gitea/actions/upload-artifact/UPSTREAM.json

@@ -0,0 +1,8 @@
+{
+  "name": "ci-action-gitea-upload-artifact",
+  "source": "https://github.com/christopherhx/gitea-upload-artifact.git",
+  "requested_ref": "v4",
+  "commit": "81f940d004763f986ba3582c007fd842dd5cb0d7",
+  "path": "/private/tmp/gitea-migration-20261105/mirrors/ci-action-gitea-upload-artifact.git",
+  "destination": "https://git.ermnvldmr.com/deytenit/ci-action-gitea-upload-artifact"
+}

.gitea/actions/upload-artifact/action.yml

@@ -0,0 +1,61 @@
+name: 'Upload a Build Artifact'
+description: 'Upload a build artifact that can be used by subsequent workflow steps'
+author: 'GitHub'
+inputs:
+  name:
+    description: 'Artifact name'
+    default: 'artifact'
+  path:
+    description: 'A file, directory or wildcard pattern that describes what to upload'
+    required: true
+  if-no-files-found:
+    description: >
+      The desired behavior if no files are found using the provided path.
+
+      Available Options:
+        warn: Output a warning but do not fail the action
+        error: Fail the action with an error message
+        ignore: Do not output any warnings or errors, the action does not fail
+    default: 'warn'
+  retention-days:
+    description: >
+      Duration after which artifact will expire in days. 0 means using default retention.
+
+      Minimum 1 day.
+      Maximum 90 days unless changed from the repository settings page.
+  compression-level:
+    description: >
+      The level of compression for Zlib to be applied to the artifact archive.
+      The value can range from 0 to 9:
+      - 0: No compression
+      - 1: Best speed
+      - 6: Default compression (same as GNU Gzip)
+      - 9: Best compression
+      Higher levels will result in better compression, but will take longer to complete.
+      For large files that are not easily compressed, a value of 0 is recommended for significantly faster uploads.
+    default: '6'
+  overwrite:
+    description: >
+      If true, an artifact with a matching name will be deleted before a new one is uploaded.
+      If false, the action will fail if an artifact for the given name already exists.
+      Does not fail if the artifact does not exist.
+    default: 'false'
+
+outputs:
+  artifact-id:
+    description: >
+      A unique identifier for the artifact that was just uploaded. Empty if the artifact upload failed.
+
+      This ID can be used as input to other APIs to download, delete or get more information about an artifact: https://docs.github.com/en/rest/actions/artifacts
+  artifact-url:
+    description: >
+      A download URL for the artifact that was just uploaded. Empty if the artifact upload failed.
+
+      This download URL only works for requests Authenticated with GitHub. Anonymous downloads will be prompted to first login. 
+      If an anonymous download URL is needed than a short time restricted URL can be generated using the download artifact API: https://docs.github.com/en/rest/actions/artifacts#download-an-artifact    
+
+      This URL will be valid for as long as the artifact exists and the workflow run and repository exists. Once an artifact has expired this URL will no longer work.
+      Common uses cases for such a download URL can be adding download links to artifacts in descriptions or comments on pull requests or issues.
+runs:
+  using: 'node20'
+  main: 'dist/index.js'

.gitea/actions/upload-artifact/package.json

@@ -0,0 +1,54 @@
+{
+  "name": "upload-artifact",
+  "version": "4.2.0",
+  "description": "Upload an Actions Artifact in a workflow run",
+  "main": "dist/index.js",
+  "scripts": {
+    "build": "tsc",
+    "release": "ncc build src/upload-artifact.ts && git add -f dist/index.js",
+    "check-all": "concurrently \"npm:format-check\" \"npm:lint\" \"npm:test\" \"npm:build\"",
+    "format": "prettier --write **/*.ts",
+    "format-check": "prettier --check **/*.ts",
+    "lint": "eslint **/*.ts",
+    "test": "jest --testTimeout 10000"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/actions/upload-artifact.git"
+  },
+  "keywords": [
+    "Actions",
+    "GitHub",
+    "Artifacts",
+    "Upload"
+  ],
+  "author": "GitHub",
+  "license": "MIT",
+  "bugs": {
+    "url": "https://github.com/actions/upload-artifact/issues"
+  },
+  "homepage": "https://github.com/actions/upload-artifact#readme",
+  "dependencies": {
+    "@actions/artifact": "^2.1.0",
+    "@actions/core": "^1.10.0",
+    "@actions/github": "^6.0.0",
+    "@actions/glob": "^0.3.0",
+    "@actions/io": "^1.1.2"
+  },
+  "devDependencies": {
+    "@types/jest": "^29.2.5",
+    "@types/node": "^18.11.18",
+    "@typescript-eslint/parser": "^5.48.0",
+    "@vercel/ncc": "^0.36.0",
+    "concurrently": "^7.6.0",
+    "eslint": "^8.31.0",
+    "eslint-plugin-github": "^4.6.0",
+    "eslint-plugin-jest": "^27.2.0",
+    "glob": "^8.0.3",
+    "jest": "^29.3.1",
+    "jest-circus": "^29.3.1",
+    "prettier": "^2.8.1",
+    "ts-jest": "^29.0.3",
+    "typescript": "^4.9.4"
+  }
+}

.gitea/workflows/ci-packer-main-manual-build-golden-image.yml

@@ -0,0 +1,179 @@
+name: ci/packer - main/manual-build-golden-image
+'on':
+  workflow_dispatch:
+    inputs:
+      upload_s3:
+        description: Upload built image to specified static S3 Object Storage
+        type: boolean
+        default: true
+concurrency:
+  group: ci-golden-image
+  cancel-in-progress: false
+permissions:
+  contents: read
+jobs:
+  validate-scripts:
+    name: 'Validate: Provisioner Scripts'
+    runs-on: apex-debian13
+    steps:
+    - name: Checkout repository
+      run: "set -euo pipefail\ncase \"$CHECKOUT_REF\" in -*) exit 1;; esac\nGITHUB_SERVER_URL=${GITHUB_SERVER_URL%/}\n\
+        case \"$GITHUB_SERVER_URL\" in https://git.ermnvldmr.com) ;; *) exit 1;; esac\n\
+        AUTH=$(printf 'deytenit:%s' \"$CHECKOUT_TOKEN\" | base64 -w0)\nHEADER=\"http.${GITHUB_SERVER_URL}/.extraheader=AUTHORIZATION:\
+        \ basic $AUTH\"\ngit init \"$GITHUB_WORKSPACE\"\ncd \"$GITHUB_WORKSPACE\"\n\
+        git remote add origin \"$GITHUB_SERVER_URL/$GITHUB_REPOSITORY.git\"\ngit -c\
+        \ \"$HEADER\" fetch --tags origin '+refs/heads/*:refs/remotes/origin/*'\n\
+        git -c \"$HEADER\" fetch origin \"$CHECKOUT_REF\"\ngit checkout --detach FETCH_HEAD\n\
+        test \"$(git rev-parse HEAD)\" = \"$GITHUB_SHA\"\nif [ \"$CHECKOUT_SUBMODULES\"\
+        \ = recursive ]; then\n    key_file=$(mktemp \"${RUNNER_TEMP:-/tmp}/hextra-key.XXXXXX\"\
+        )\n    trap 'rm -f \"$key_file\"' EXIT\n    umask 077\n    printf '%s\\n'\
+        \ \"$HEXTRA_READ_KEY\" > \"$key_file\"\n    chmod 600 \"$key_file\"\n    export\
+        \ GIT_SSH_COMMAND=\"ssh -T -i $key_file -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes\
+        \ -o HostKeyAlgorithms=ssh-ed25519 -o UserKnownHostsFile=$GITHUB_WORKSPACE/.gitea/known_hosts\"\
+        \n    git submodule update --init --recursive\nfi\nif [ \"$PERSIST_CREDENTIALS\"\
+        \ = true ]; then\n    git config \"http.${GITHUB_SERVER_URL}/.extraheader\"\
+        \ \"AUTHORIZATION: basic $AUTH\"\nfi\n"
+      env:
+        CHECKOUT_REF: ${{ github.sha }}
+        CHECKOUT_SUBMODULES: 'false'
+        CHECKOUT_TOKEN: ${{ github.token }}
+        PERSIST_CREDENTIALS: 'false'
+        HEXTRA_READ_KEY: ''
+    - name: Setup Python
+      run: python3 -c "import sys; assert sys.version_info >= (3, 10)"
+    - name: Install Developer Dependencies
+      run: python3 -m pip install ruff==0.17.2 "PyYAML>=6.0,<7"
+    - name: Validate Shell Script Syntax
+      run: "for script in ci/packer/scripts/*.sh ci/scripts/init-node/provisioning/seed.sh;\
+        \ do\n  bash -n \"$script\"\ndone\nmake check\n"
+    - name: Setup Packer
+      run: 'packer version
+
+        packer plugins installed | grep qemu'
+    - name: Validate Packer Template
+      run: 'packer plugins installed | grep qemu
+
+        packer validate ci/packer/apex-debian-13.pkr.hcl
+
+        '
+  build-image:
+    name: 'Build: Golden Image'
+    needs: validate-scripts
+    runs-on: apex-debian13
+    outputs:
+      image_name: ${{ steps.vars.outputs.image_name }}
+    steps:
+    - name: Checkout repository
+      run: "set -euo pipefail\ncase \"$CHECKOUT_REF\" in -*) exit 1;; esac\nGITHUB_SERVER_URL=${GITHUB_SERVER_URL%/}\n\
+        case \"$GITHUB_SERVER_URL\" in https://git.ermnvldmr.com) ;; *) exit 1;; esac\n\
+        AUTH=$(printf 'deytenit:%s' \"$CHECKOUT_TOKEN\" | base64 -w0)\nHEADER=\"http.${GITHUB_SERVER_URL}/.extraheader=AUTHORIZATION:\
+        \ basic $AUTH\"\ngit init \"$GITHUB_WORKSPACE\"\ncd \"$GITHUB_WORKSPACE\"\n\
+        git remote add origin \"$GITHUB_SERVER_URL/$GITHUB_REPOSITORY.git\"\ngit -c\
+        \ \"$HEADER\" fetch --tags origin '+refs/heads/*:refs/remotes/origin/*'\n\
+        git -c \"$HEADER\" fetch origin \"$CHECKOUT_REF\"\ngit checkout --detach FETCH_HEAD\n\
+        test \"$(git rev-parse HEAD)\" = \"$GITHUB_SHA\"\nif [ \"$CHECKOUT_SUBMODULES\"\
+        \ = recursive ]; then\n    key_file=$(mktemp \"${RUNNER_TEMP:-/tmp}/hextra-key.XXXXXX\"\
+        )\n    trap 'rm -f \"$key_file\"' EXIT\n    umask 077\n    printf '%s\\n'\
+        \ \"$HEXTRA_READ_KEY\" > \"$key_file\"\n    chmod 600 \"$key_file\"\n    export\
+        \ GIT_SSH_COMMAND=\"ssh -T -i $key_file -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes\
+        \ -o HostKeyAlgorithms=ssh-ed25519 -o UserKnownHostsFile=$GITHUB_WORKSPACE/.gitea/known_hosts\"\
+        \n    git submodule update --init --recursive\nfi\nif [ \"$PERSIST_CREDENTIALS\"\
+        \ = true ]; then\n    git config \"http.${GITHUB_SERVER_URL}/.extraheader\"\
+        \ \"AUTHORIZATION: basic $AUTH\"\nfi\n"
+      env:
+        CHECKOUT_REF: ${{ github.sha }}
+        CHECKOUT_SUBMODULES: 'false'
+        CHECKOUT_TOKEN: ${{ github.token }}
+        PERSIST_CREDENTIALS: 'false'
+        HEXTRA_READ_KEY: ''
+    - name: Compute Image Metadata
+      id: vars
+      run: 'SHORT_SHA="$(echo ${{ github.sha }} | cut -c1-7)"
+
+        IMAGE_NAME="com-ermnvldmr-apex-debian-13-amd64-${SHORT_SHA}.qcow2"
+
+        echo "image_name=${IMAGE_NAME}" >> "$GITHUB_OUTPUT"
+
+        '
+    - name: Install QEMU & Build Tools
+      run: 'sudo apt-get update
+
+        sudo apt-get install -y qemu-system-x86 qemu-utils xorriso
+
+        sudo chmod 666 /dev/kvm 2>/dev/null || true
+
+        '
+    - name: Setup Packer
+      run: 'packer version
+
+        packer plugins installed | grep qemu'
+    - name: Build Debian 13 Image with Packer
+      run: 'packer plugins installed | grep qemu
+
+        packer build -var "vm_name=${{ steps.vars.outputs.image_name }}" -var "provision_revision=${{
+        github.sha }}" ci/packer/apex-debian-13.pkr.hcl
+
+        '
+    - name: Upload Image Artifact
+      uses: ./.gitea/actions/upload-artifact
+      with:
+        name: golden-image
+        path: output-qemu/${{ steps.vars.outputs.image_name }}
+        retention-days: 30
+  upload-s3:
+    name: 'Upload: S3 Storage'
+    needs: build-image
+    if: inputs.upload_s3 == true
+    runs-on: apex-debian13
+    steps:
+    - run: "set -euo pipefail\ncase \"$CHECKOUT_REF\" in -*) exit 1;; esac\nGITHUB_SERVER_URL=${GITHUB_SERVER_URL%/}\n\
+        case \"$GITHUB_SERVER_URL\" in https://git.ermnvldmr.com) ;; *) exit 1;; esac\n\
+        AUTH=$(printf 'deytenit:%s' \"$CHECKOUT_TOKEN\" | base64 -w0)\nHEADER=\"http.${GITHUB_SERVER_URL}/.extraheader=AUTHORIZATION:\
+        \ basic $AUTH\"\ngit init \"$GITHUB_WORKSPACE\"\ncd \"$GITHUB_WORKSPACE\"\n\
+        git remote add origin \"$GITHUB_SERVER_URL/$GITHUB_REPOSITORY.git\"\ngit -c\
+        \ \"$HEADER\" fetch --tags origin '+refs/heads/*:refs/remotes/origin/*'\n\
+        git -c \"$HEADER\" fetch origin \"$CHECKOUT_REF\"\ngit checkout --detach FETCH_HEAD\n\
+        test \"$(git rev-parse HEAD)\" = \"$GITHUB_SHA\"\nif [ \"$CHECKOUT_SUBMODULES\"\
+        \ = recursive ]; then\n    key_file=$(mktemp \"${RUNNER_TEMP:-/tmp}/hextra-key.XXXXXX\"\
+        )\n    trap 'rm -f \"$key_file\"' EXIT\n    umask 077\n    printf '%s\\n'\
+        \ \"$HEXTRA_READ_KEY\" > \"$key_file\"\n    chmod 600 \"$key_file\"\n    export\
+        \ GIT_SSH_COMMAND=\"ssh -T -i $key_file -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes\
+        \ -o HostKeyAlgorithms=ssh-ed25519 -o UserKnownHostsFile=$GITHUB_WORKSPACE/.gitea/known_hosts\"\
+        \n    git submodule update --init --recursive\nfi\nif [ \"$PERSIST_CREDENTIALS\"\
+        \ = true ]; then\n    git config \"http.${GITHUB_SERVER_URL}/.extraheader\"\
+        \ \"AUTHORIZATION: basic $AUTH\"\nfi\n"
+      env:
+        CHECKOUT_REF: ${{ github.sha }}
+        CHECKOUT_SUBMODULES: 'false'
+        CHECKOUT_TOKEN: ${{ github.token }}
+        PERSIST_CREDENTIALS: 'false'
+        HEXTRA_READ_KEY: ''
+    - name: Download Image Artifact
+      uses: ./.gitea/actions/download-artifact
+      with:
+        name: golden-image
+        path: output-qemu
+    - name: Upload to Static S3 (/dev/images/)
+      env:
+        AWS_ACCESS_KEY_ID: ${{ secrets.STATIC_AWS_ACCESS_KEY_ID || secrets.COM_ERMNVLDMR_STATIC_AWS_ACCESS_KEY_ID
+          }}
+        AWS_SECRET_ACCESS_KEY: ${{ secrets.STATIC_AWS_SECRET_ACCESS_KEY || secrets.COM_ERMNVLDMR_STATIC_AWS_SECRET_ACCESS_KEY
+          }}
+        AWS_DEFAULT_REGION: ${{ secrets.STATIC_AWS_REGION || secrets.COM_ERMNVLDMR_STATIC_S3_REGION
+          || 'auto' }}
+        S3_ENDPOINT: ${{ secrets.STATIC_S3_ENDPOINT_URL || secrets.COM_ERMNVLDMR_STATIC_S3_ENDPOINT_URL
+          }}
+        S3_BUCKET: ${{ secrets.STATIC_S3_BUCKET || secrets.COM_ERMNVLDMR_STATIC_S3_BUCKET
+          }}
+        IMAGE_NAME: ${{ needs.build-image.outputs.image_name }}
+      run: "if [ -n \"${S3_BUCKET}\" ] && [ -n \"${AWS_ACCESS_KEY_ID}\" ]; then\n\
+        \  ENDPOINT_FLAG=\"\"\n  if [ -n \"${S3_ENDPOINT}\" ]; then\n    ENDPOINT_FLAG=\"\
+        --endpoint-url=${S3_ENDPOINT}\"\n  fi\n  echo \"Cleaning up older builds in\
+        \ dev/images/com-ermnvldmr-apex-debian-13-amd64-*.qcow2 to maintain quota...\"\
+        \n  aws ${ENDPOINT_FLAG} s3 rm \"s3://${S3_BUCKET}/dev/images/\" --recursive\
+        \ --exclude \"*\" --include \"com-ermnvldmr-apex-debian-13-amd64-*.qcow2\"\
+        \ || true\n  echo \"Uploading ${IMAGE_NAME}...\"\n  aws ${ENDPOINT_FLAG} s3\
+        \ cp \"output-qemu/${IMAGE_NAME}\" \"s3://${S3_BUCKET}/dev/images/${IMAGE_NAME}\"\
+        \n  echo \"Golden image uploaded successfully to s3://${S3_BUCKET}/dev/images/${IMAGE_NAME}\"\
+        \nelse\n  echo \"S3 credentials not configured; skipping upload. Image artifact\
+        \ preserved in GitHub Actions run.\"\nfi\n"

.gitignore

@@ -9,9 +9,12 @@
 __pycache__/
 *.pyc
 .pytest_cache/
+.venv/
 
 # Runtime / secrets (never in commons)
 *.log
 *.env
 .env
 *.bak
+._*
+.DS_Store

LICENSE

@@ -1,21 +1,24 @@
-MIT License
+BSD 2-Clause License
 
-Copyright (c) 2026-present Vladimir Eremin
+Copyright (c) 2026, Vladimir Eremin
 
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
 
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
 
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Makefile

@@ -0,0 +1,21 @@
+.PHONY: test lint fmt check
+
+test:
+	bash tests/run.sh
+
+lint:
+	@if command -v ruff >/dev/null 2>&1; then \
+		ruff check .; \
+	else \
+		python3 -m compileall -q engine actions tests ci/scripts && \
+		echo "ruff not installed in PATH; verified python bytecode compilation."; \
+	fi
+
+fmt:
+	@if command -v ruff >/dev/null 2>&1; then \
+		ruff format .; \
+	else \
+		echo "ruff not installed in PATH; skipping fmt."; \
+	fi
+
+check: lint test

PARITY.md

@@ -1,76 +0,0 @@
-# Port parity map (bash -> python)
-
-Behavioral parity is validated on a scratch host in Phase 3. This maps each ported
-action to its bash source and the behaviors that must match.
-
-| apex action | bash source (root commons) | must-match behaviors |
-|---|---|---|
-| configure/base | actions/configure/base | apt update+install curl/wget/git/rsyslog; rsyslog running; ufw + ufw-docker at ~/.local/bin |
-| configure/ufw | actions/configure/ufw + root_template_ufw | host.rules via plain ufw; before/after.rules block inject (APEX markers, self-heal ROOT); docker rules via ufw-docker |
-| configure/crowdsec | actions/configure/crowdsec | install if cscli absent; render configs->/etc/crowdsec; collections linux+traefik; firewall bouncer if bouncer file present |
-| configure/cron | actions/configure/cron + root_template_crontab | render crontab; backup+install; rollback on empty/failure |
-| configure/systemd | actions/configure/systemd | render each unit -> /etc/systemd/system; daemon-reload; enable+start .service |
-| configure/routing | actions/configure/routing | ip_forward=1; fwmark 0x1->table100 rule; local default route table100 (idempotent) |
-| configure | (new umbrella) | run base,ufw,crowdsec,cron,systemd,routing in order |
-| sync/repository | actions/sync/repository | sync/<node> branch; add -A + commons submodule; commit; rebase origin/main (was trunk in the monorepo); force-with-lease; telegram |
-| sync/packages | actions/sync/packages | apt --just-print upgrade Inst list; docker+skopeo digest compare; telegram info |
-| backup/run | actions/sync/tiers (restic-backup) | init-if-needed; label-driven backup via resticontainer (discovers `restic.*`-labelled services, runs pre-hooks/stops, snapshots the resolved host paths) --host <node>.apex.ermnvldmr.com --tag biweekly --compression max; guard on a fresh snapshot; forget --prune 7/4/12; telegram |
-| tiers/link | actions/tiers/link | node + per-project @tierN symlinks; shared/ links; .env symlink |
-| tiers/useradd | actions/tiers/useradd | noroot-<proj> system users; noroot-shared group; APEX_UID/GID block in .env |
-| tiers/chown | actions/tiers/chown | {tier}/{proj}=uid:gid; {tier}/shared/{proj}=uid:sgid+sticky; base=root:root |
-| utils/extract-traefik-certs | actions/utils/extract-traefik-certs | traefik-certs-dumper v3; 600 perms; chown caller |
-| utils/generate-happ-subscriptions | actions/utils/generate-happ-subscriptions | MOVED to daedalus proprietaries (node-specific tooling; commons keeps only shared stuff) |
-| utils/lint-docker-compose | actions/utils/lint-docker-compose | dclint --fix via docker; --hook staged-only + restage; + anchor-IP check |
-| compose | actions/compose | apex core first then services alpha; reverse on down; --dry-run; extra passthrough |
-
-## Deliberate deltas (rename + fixes)
-- root->apex everywhere (CLI, container names apex-*, telegram instance com.ermnvldmr.apex.*, ufw markers APEX.*, networks direct/enclave/socket).
-- No <node> arg; identity from FQDN + node.env.
-- backup superseded by the resticontainer migration (see below): the old whole-tier-root
-  restic snapshot is replaced by per-service, `restic.*`-label-driven backups.
-- ufw markers self-heal (delete ROOT + APEX blocks before inject).
-- envsubst -> string.Template.safe_substitute; wget/curl download -> urllib. Unset template
-  vars stay LITERAL (envsubst emptied them) — typos surface instead of silently blanking.
-- crowdsec install auto-confirms (`apt-get install -y`; bash ran promptful `apt install`,
-  which blocked on a tty and aborted non-interactive runs).
-- pre-commit staged-compose pathspec widened to `*docker-compose.yml` — the old
-  `*.docker-compose.yml`/root-anchored patterns never matched nested compose files.
-- anchor-IP validation in utils/lint-docker-compose is a planned addition (SP3 §6.2).
-- new opt-in `--dry-run` flags on configure/* have no bash counterpart; the default
-  (flag absent) path is the ported behavior.
-- ufw/crowdsec deploys keep a timestamped `.bak` next to overwritten system files
-  (bash: none / `/tmp`) as a rollback net; tiers/useradd `.env` rewrites deliberately do
-  NOT (no secret-bearing copies).
-- non-conventional tool exit codes normalize to 1 (bash sometimes propagated e.g.
-  apt-get's 100); 64/65/66 conventions preserved.
-
-## Core-compose consolidation deltas (beyond the rename)
-- icarus: traefik no longer publishes 53/tcp+udp (AdGuard DNS) from the shared file —
-  restored via an icarus-only override layered in the node repo's include (Phase 2).
-- icarus: ACME propagation-tuning flags and DEBUG log level drop to the shared defaults;
-  traefik now runs noroot (APEX_UID) with file provider + HTTP/3 like daedalus.
-- xray is profile-gated (`xray` in COMPOSE_PROFILES): daedalus + icarus opt in,
-  morpheus does not (it never ran xray — owner decision 2026-07-05).
-- loki (profile `loki`, no node opts in yet) is exposed via traefik at
-  `https://<node-fqdn>/loki` behind basicauth (`APEX_LOKI_BASIC_AUTH` in the node .env).
-- morpheus: dashboard auth changes basicauth (TRAEFIK_INTERNAL_API_AUTH) -> enclave IP
-  allowlist; the adguard router labels exist without a backend (502, was 404, on /adguard).
-- restic: per-node selective tier2 mounts replaced by whole-tier-root RO mounts
-  (superset; snapshot paths preserved); RESTIC_COMPRESSION per-node via
-  `APEX_RESTIC_COMPRESSION` (default auto).
-
-## resticontainer migration (SP4 realized)
-- The `restic` core service is replaced by `resticontainer`
-  (`ghcr.io/deytenit/resticontainer`), a restic wrapper that reads per-service
-  `restic.*` compose labels: `restic.enable`, `restic.backup.paths` (comma-separated
-  container mount destinations — matched exactly, whole mount dir backed up),
-  `restic.backup.stop`, and `restic.hooks.pre-backup`/`post-backup` (run via docker exec).
-- It mounts the docker socket (list/inspect/exec/stop/start) and `/srv:/srv:ro`
-  (`RESTIC_HOSTFS=/`); apex data + checkouts live under /srv and the `@tierN` mounts
-  are absolute symlinks into /srv, so mounting /srv at the same path lets restic
-  follow them and record the real host paths, and snapshots the union in one run.
-- The whole-tier-root RO mounts and the explicit `/data/@tier1,2` backup args are gone;
-  a service is backed up only if it carries `restic.enable=true`. DB consistency is
-  per-service: a pre-backup hook dumps into a dedicated backed-up mount (e.g. postgres
-  `pg_dump … > /dumps/dump.sql`), never the live data dir. `backup/run` now guards on a
-  fresh snapshot landing for the host (a no-label repo would otherwise be a silent no-op).

README.md

@@ -1,31 +1,150 @@
-# apex.ermnvldmr.com
+<p align="center">
+  <img alt="apex.ermnvldmr.com logo" src="./assets/logo.png" height="128" />
+</p>
 
-The **commons** of the apex fleet: a stock-python engine (`apex <group>/<action>`) and
-the shared core docker-compose, consumed by every node as the `commons/` git submodule.
+# [apex.ermnvldmr.com](https://apex.ermnvldmr.com)
 
-Full documentation: [docs.ermnvldmr.com/en/apex](https://docs.ermnvldmr.com/en/apex/).
+Local infrastructure operations for independently owned nodes, with a Python standard-library core.
 
-## Layout
-- `apex` — launcher (python3, stdlib only); node `init.sh` symlinks it onto `PATH`.
-- `engine/` — the runner: CLI, identity, overlay, context, and the `ctx.*` helper library.
-- `actions/` — grouped actions; the path IS the name (`configure/ufw`, `tiers/link`, …).
-- `compositions/apex/docker-compose.yml` — the shared, `${env}`-parameterized core compose.
-- `vendor/` — pinned pure-python third-party (empty; see `vendor/VENDORED.md`).
-- `tests/` — stdlib `unittest` suite (`tests/run.sh`).
+## Quick Start
 
-Node-specific tooling lives in each node repo's `proprietaries/` — commons holds only
-what every node shares.
+- **Live Documentation**: <https://docs.ermnvldmr.com/en/apex/> (also available in [Russian](https://docs.ermnvldmr.com/ru/apex/))
+- **Getting Started Guide**: <https://docs.ermnvldmr.com/en/apex/getting-started/>
+- **Report an Issue**: <https://git.ermnvldmr.com/deytenit/apex.ermnvldmr.com/issues>
+- **License**: [BSD-2-Clause](#copyright--license)
 
-## Usage (on a node)
-`apex` acts on the **local checkout** — no `<node>` argument. Identity comes from the
-host FQDN (`<node>.a<x>.apex.ermnvldmr.com`) and `node.env`.
-- `apex` / `apex --help` — list actions
-- `apex configure/ufw` — render + apply this node's ufw config
-- `apex configure` — run every `configure/*` in order
-- `apex compose up` — bring up the node's compositions
+```sh
+# Clone your node's repository (which pins apex as commons/)
+git clone --recurse-submodules ssh://git@git.example.com:22222/infrastructure/selfhost.example.com.git
+cd selfhost.example.com && ./init.sh && source ~/.bashrc
 
-## Constraints
-git + python3 stdlib only. No pip/Node/Go/CUE. Composes are hand-authored (no codegen).
+# Configure node identity
+cat > node.env <<'EOF'
+APEX_NODE_FQDN=selfhost.example.com
+APEX_NODE_HOST=selfhost.example.com
+APEX_SUBNET=198.18.0.0/24
+EOF
 
-## License
-MIT (see `LICENSE`).
+# Configure host baseline and bring up compositions
+apex configure        # firewall, crowdsec, cron, systemd, routing
+apex compose up       # bring node services online
+```
+
+## Essential Information
+
+You should understand the following invariants before exploring the codebase:
+
+### Core Principles
+
+- **No Control Plane**: No central orchestrator, agent daemon, or cluster controller. Commands act locally on the node you run them on.
+- **Core Runtime**: Python 3.10 or newer and Git, with no `pip` dependencies or build step. Standalone host provisioning and workstation checks have separate requirements below.
+- **Node-Owned Repositories**: Each machine is an independent Git repository pinning `apex` as a `commons/` submodule. It declares its own identity, compositions, and local overrides.
+- **Overlay Architecture**: Node repositories can extend or override any shared action by placing an executable in `proprietaries/actions/`.
+
+### Repository Structure
+
+- **[`actions/`](./actions)**: Built-in engine actions (`compose.py`, `configure/`, `sync/`, `tiers/`, `backup/`, `utils/`).
+- **[`compositions/`](./compositions)**: The shared core Compose stack (`traefik`, `socket-proxy`, `alloy`, `adguard`, `xray`, `restic`).
+- **[`engine/`](./engine)**: Framework engine — action discovery, overlay resolution, context injection, and logging.
+- **[`etc/`](./etc)**: Shared static configuration, including Alloy.
+- **[`ci/`](./ci)**: Host image build recipes, cloud-init templates, and image enrollment tooling.
+- **[`assets/`](./assets)**: Repository branding assets (`logo.png`).
+
+---
+
+## Who Are You?
+
+### Node Operator
+
+Running and maintaining services on an existing host.
+
+- **Requirements**: Debian-based host with Docker, Docker Compose v2, Git, and Python 3.
+- **Workflow**:
+  - Update services: `apex compose pull && apex compose up`
+  - Run backups: `apex backup/run`
+  - Sync repository state: `apex sync/repository`
+
+Target a composition explicitly when operating on one application:
+
+```sh
+apex compose -p apex restart traefik
+apex compose -p app up -d
+apex compose --dry-run -p apex logs --tail 20 xray
+```
+
+Wrapper options precede the action; remaining arguments go to Docker Compose.
+The positional forms `apex compose restart apex traefik` and `apex compose up -d apex`
+remain supported. Use `-p` when service names or option values could be mistaken
+for composition names. Without a target, APEX visits core first, then other
+compositions alphabetically; `down` reverses that order.
+
+Replace `app` with a composition defined in your repository. Use targeted `up -d`
+after changing a container definition; `restart` retains the existing definition.
+
+For direct Docker Compose use in the core composition:
+
+```sh
+apex env --sync
+cd compositions/apex
+docker compose config --quiet
+docker compose restart traefik
+```
+
+Sync preserves existing `.env` credentials and UID/GID entries, including tier
+symlinks. It updates a managed block containing `apex.env` settings (including
+profiles) followed by synthesized node/path values, and writes with mode `0600`.
+Resync after changing `apex.env`, node identity, or tier paths. Shell environment
+values still take precedence in Docker Compose. Do not edit the generated block.
+
+`eval "$(apex env)"` exports synthesized APEX values to the current shell;
+`apex env --dotenv` prints dotenv syntax, and `apex env --write generated.env`
+writes those values to a separate file. These three modes do not merge the
+composition's secrets or `apex.env`; use `--sync` for the complete core bridge.
+Choose one output mode per invocation.
+
+### Infrastructure Administrator
+
+Build Debian 13 amd64 golden images with Packer, then supply node identity,
+operator keys, and the node repository through
+[`ci/cloud-init/user-data.template.yaml`](./ci/cloud-init/user-data.template.yaml)
+at first boot. Keep filled user-data and repository credentials outside Git.
+
+```sh
+packer init ci/packer/apex-debian-13.pkr.hcl
+packer validate ci/packer/apex-debian-13.pkr.hcl
+packer build ci/packer/apex-debian-13.pkr.hcl
+```
+
+Cloud-init runs the image's bundled enrollment helper. After enrollment, run
+`apex configure` and `apex compose up` on the node.
+
+### Developer & Contributor
+
+Adding shared actions or improving the engine.
+
+- **Conventions**: Maintain strict Python standard library compliance. No external runtime packages.
+- **Public documentation**: Describe supported interfaces using synthetic examples. Keep deployment inventories, internal plans, operational runbooks, and host-specific configuration in their owning private repositories.
+- **Setup**:
+  ```sh
+  python3 -m venv .venv
+  . .venv/bin/activate
+  python3 -m pip install ruff==0.17.2 'PyYAML>=6.0,<7'
+  ```
+- **Verification**: `make check` runs lint and both the core and image provisioning suites. `make test` (or `bash tests/run.sh`) runs both suites; discovery limited to `tests/` omits provisioning. PyYAML is needed for the provisioning tests, not core execution. Environment integration tests use `docker compose` or standalone `docker-compose` with `config`; no daemon is needed.
+- **Linting**: Ruff checks correctness and security rules. Existing import-order, line-length, compact-statement, and annotation-style debt is explicitly deferred in `pyproject.toml` to avoid unrelated formatting changes. Without Ruff, `make lint` recursively checks Python compilation and fails on syntax errors; that fallback is not equivalent to Ruff. `make fmt` formats files and is a separate, deliberate operation.
+- **Packer**: Run `packer init ci/packer/apex-debian-13.pkr.hcl` and `packer validate ci/packer/apex-debian-13.pkr.hcl` for template validation. A VM build or first-boot enrollment is a separate operation; unit tests and template validation do not demonstrate a deployed host.
+
+---
+
+## Communication & Support
+
+- **Issues**: <https://git.ermnvldmr.com/deytenit/apex.ermnvldmr.com/issues>
+- **Email**: <personal@ermnvldmr.com>
+
+## Copyright & License
+
+Licensed under the [BSD-2-Clause License](./LICENSE).
+
+---
+
+**AI Assistance**: Portions of this project's source code and configuration may have been developed with the assistance of AI code-generation tools.

actions/backup/run.py

@@ -1,5 +1,5 @@
 # actions/backup/run.py
-"""Label-driven restic backup via resticontainer (github.com/deytenit/resticontainer).
+"""Label-driven restic backup via resticontainer (git.ermnvldmr.com/deytenit/resticontainer).
 Runs the resticontainer core service one-shot: init-if-needed -> backup (discovers
 `restic.*`-labelled services, runs their pre-hooks/stops, snapshots the union of the
 resolved host paths in a single restic run) -> forget --prune -> Telegram. Every
@@ -7,16 +7,48 @@ non-`backup` subcommand passes straight through to restic, so init/snapshots/for
 behave exactly as before."""
 import json
 import os
+from pathlib import Path
+import time
 from engine.descriptor import Meta, Arg
 
 METADATA = Meta(summary="Run the resticontainer label-driven backup (init-if-needed, backup, forget --prune).",
                 args=[Arg("telegram_bot_url", "Telegram Bot URL for notifications")])
 TITLE = "Restic Host Backup"
 
+def write_backup_status(path, state, timestamp):
+    path = Path(path)
+    previous = json.loads(path.read_text()) if path.exists() else {}
+    status = {"state": state, "updated_at": timestamp,
+              "last_success": timestamp if state == "success" else previous.get("last_success", 0)}
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_name(path.name + f".{os.getpid()}.tmp")
+    temporary.write_text(json.dumps(status) + "\n")
+    temporary.chmod(0o644)
+    temporary.replace(path)
+
+def restore_containers(ctx, container_ids):
+    """Recover previously running backup targets, retrying namespace dependents."""
+    pending = list(container_ids)
+    if not pending:
+        return
+    for _ in range(len(pending) + 1):
+        cp = ctx.sys.run(["docker", "inspect", *pending], capture=True)
+        stopped = [c["Id"] for c in json.loads(cp.stdout) if not c["State"]["Running"]]
+        if not stopped:
+            return
+        progress = False
+        for container_id in stopped:
+            cp = ctx.sys.run(["docker", "start", container_id], check=False, capture=True)
+            progress = progress or cp.returncode == 0
+        if not progress:
+            raise RuntimeError("Backup left containers stopped: " + ", ".join(stopped))
+        pending = stopped
+    raise RuntimeError("Backup containers did not stay running: " + ", ".join(pending))
+
 def run(ctx, args):
     log, node, url = ctx.log, ctx.node.name, args.telegram_bot_url
     core = ctx.paths.core
-    host = f"{node}.apex.ermnvldmr.com"         # restic --host: the per-node public name (matches the bucket)
+    host = ctx.node.host                        # restic --host: the node's public host name (APEX_NODE_HOST)
     tag = "biweekly"                            # restic --tag: snapshot cadence (this cron fires on the 5th & 20th)
     if not os.path.isfile(os.path.join(core, "docker-compose.yml")):
         log.error(f"Core compose not found in {core}"); raise SystemExit(1)
@@ -42,17 +74,27 @@ def run(ctx, args):
         except (ValueError, TypeError):
             return None
 
+    status_path = Path("/var/lib/apex/backup-status.json")
+    write_backup_status(status_path, "running", time.time())
     try:
         log.info("Checking restic repository availability...")
         if restic("snapshots", check=False).returncode == 0:
             log.info("Restic repository reachable.")
         elif restic("init", check=False).returncode != 0:
-            ctx.notify.error(TITLE, url, "restic init failed", node); raise SystemExit(1)
+            ctx.notify.error(TITLE, url, "restic init failed"); raise SystemExit(1)
         before = host_snapshot_ids()
         log.info("Backing up `restic.*`-labelled services (resticontainer discovers paths)...")
-        if restic("backup", "--host", host, "--tag", tag, "--compression", "max",
-                  check=False).returncode != 0:
-            ctx.notify.error(TITLE, url, "restic backup failed", node); raise SystemExit(1)
+        targets = ctx.sys.run(["docker", "ps", "-q", "--no-trunc",
+                               "--filter", "label=restic.enable=true",
+                               "--filter", "label=restic.backup.stop=true"], capture=True).stdout.split()
+        try:
+            result = restic("backup", "--host", host, "--tag", tag, "--compression", "max",
+                            check=False)
+        finally:
+            # Resticontainer may start a network-namespace dependent before its owner.
+            restore_containers(ctx, targets)
+        if result.returncode != 0:
+            ctx.notify.error(TITLE, url, "restic backup failed"); raise SystemExit(1)
         # Guard: the label-driven backup is a silent no-op if no service carries the
         # `restic.enable` label (resticontainer finds zero paths and skips restic).
         # Prove a fresh snapshot for this host actually landed before we prune/notify.
@@ -60,18 +102,22 @@ def run(ctx, args):
         if before is not None and after is not None and not (after - before):
             log.error("No new snapshot produced — refusing to report success.")
             ctx.notify.error(TITLE, url,
-                             "backup produced no new snapshot (no restic.* labels discovered?)", node)
+                             "backup produced no new snapshot (no restic.* labels discovered?)")
             raise SystemExit(1)
         log.info("Applying retention (forget --prune)...")
-        if restic("forget", "--prune", "--keep-daily", "7", "--keep-weekly", "4",
+        if restic("forget", "--prune", "--host", host, "--tag", tag,
+                  "--keep-daily", "7", "--keep-weekly", "4",
                   "--keep-monthly", "12", check=False).returncode != 0:
-            ctx.notify.error(TITLE, url, "restic forget/prune failed", node); raise SystemExit(1)
+            ctx.notify.error(TITLE, url, "restic forget/prune failed"); raise SystemExit(1)
         log.success("Restic backup completed.")
-        if not ctx.notify.success(TITLE, url, f"Restic backup + retention succeeded for {node}", node):
+        if not ctx.notify.success(TITLE, url, f"Restic backup + retention succeeded for {node}"):
             raise SystemExit(1)   # bash exited 1 when the success telegram failed to send
+        write_backup_status(status_path, "success", time.time())
     except SystemExit:
+        write_backup_status(status_path, "failed", time.time())
         raise
     except Exception as e:
+        write_backup_status(status_path, "failed", time.time())
         log.error(f"{type(e).__name__}: {e}")
-        ctx.notify.error(TITLE, url, f"backup failed: {e}", node)
-        raise SystemExit(1)
+        ctx.notify.error(TITLE, url, f"backup failed: {e}")
+        raise SystemExit(1) from e

actions/compose.py

@@ -1,32 +1,78 @@
 # actions/compose.py
-"""Port of compose — orchestrate docker compose across the node. apex core first
-(owns networks), then services alpha; reverse on down. Exports derived APEX_* vars and
-layers --env-file (.env secrets + apex.env committed scalars)."""
+"""Orchestrate docker compose across the node.
+
+Supports targeting all compositions in sequence (default: apex core first, services next;
+reverse on down) or targeting one with `apex compose -p apex restart traefik`.
+Wrapper options precede the action. Exports derived APEX_* vars and layers env files.
+"""
 import os
-from engine.descriptor import Meta, Arg, Flag, Rest
+import shlex
+
+from engine.descriptor import Flag, Meta, Opt, Rest
+
+METADATA = Meta(
+    summary="Orchestrate docker compose across the node or on a specific composition.",
+    args=[
+        Opt("-p", "Target composition name or 'all'.", default=None),
+        Flag("--dry-run", "Print commands only."),
+        Rest("command", "ACTION [COMPOSE_ARGS...]; wrapper options must precede ACTION."),
+    ],
+)
 
-METADATA = Meta(summary="Orchestrate docker compose across the node (apex core first, services next).",
-                args=[Arg("action", "up | down | restart | ..."),
-                      Flag("--dry-run", "Print commands only."),
-                      Rest("extra", "Extra args passed through to docker compose.")])
 
 def run(ctx, args):
-    log, comp, action = ctx.log, ctx.paths.compositions, args.action
-    # argparse.REMAINDER swallows flags placed after the action, so strip --dry-run
-    # out of the passthrough args like bash did ("apex compose up --dry-run" must
-    # preview, not hand docker compose its own --dry-run).
-    extra = [a for a in (args.extra or []) if a != "--dry-run"]
-    dry_run = args.dry_run or len(extra) != len(args.extra or [])
+    log, comp = ctx.log, ctx.paths.compositions
+    command = list(args.command)
+    if command and command[0] == "--":
+        command.pop(0)
+    if not command:
+        log.error("ACTION is required (up, down, restart, logs, ...).")
+        raise SystemExit(64)
+    action, extra = command[0], command[1:]
+    dry_run = args.dry_run
+    before_separator = extra[:extra.index("--")] if "--" in extra else extra
+    if any(arg in ("-p", "--dry-run") or arg.startswith("-p=")
+           or arg.startswith("--dry-run=") for arg in before_separator):
+        log.error("Wrapper options -p and --dry-run must precede ACTION; use -- for passthrough.")
+        raise SystemExit(64)
     env = dict(os.environ)
     env.update({k: v for k, v in ctx.vars().items() if k.startswith("APEX_")})
 
+    available = []
+    if os.path.isdir(comp):
+        available = [
+            d for d in os.listdir(comp)
+            if os.path.isdir(os.path.join(comp, d))
+            and not d.startswith((".", "@"))
+            and os.path.isfile(os.path.join(comp, d, "docker-compose.yml"))
+        ]
+
+    target = getattr(args, "p", None)
+    if target is None and extra:
+        if extra[0] in ["all", *available]:
+            target, extra = extra[0], extra[1:]
+        elif action == "up" and len(extra) >= 2 and extra[0] == "-d":
+            # Preserve the documented `up -d apex` spelling without scanning flag values.
+            if extra[1] in ["all", *available]:
+                if len(extra) != 2:
+                    log.error("Use -p COMPOSITION before up when adding service arguments.")
+                    raise SystemExit(64)
+                target, extra = extra[1], ["-d"]
+
+    if target is None:
+        target = "all"
+
+    if target != "all" and target not in available:
+        names = ", ".join(sorted(available)) if available else "none"
+        log.error(f"Unknown composition '{target}'. Available: {names}")
+        raise SystemExit(1)
+
     def sequence():
+        if target != "all":
+            return [os.path.join(comp, target)]
         core = os.path.join(comp, "apex")
-        others = sorted(
-            os.path.join(comp, d) for d in os.listdir(comp)
-            if os.path.isdir(os.path.join(comp, d)) and not d.startswith((".", "@")) and d != "apex"
-            and os.path.isfile(os.path.join(comp, d, "docker-compose.yml")))
-        seq = ([core] if os.path.isfile(os.path.join(core, "docker-compose.yml")) else []) + others
+        others = sorted(os.path.join(comp, d) for d in available if d != "apex")
+        seq = ([core] if "apex" in available else []) + others
         return list(reversed(seq)) if action == "down" else seq
 
     def run_one(dirpath):
@@ -35,13 +81,18 @@ def run(ctx, args):
         for ef in (".env", "apex.env"):
             if os.path.isfile(os.path.join(dirpath, ef)):
                 base += ["--env-file", ef]
-        cmd = base + (["up", "-d"] if action == "up" else [action]) + extra
+        if action == "up":
+            up_flags = [] if any(flag in extra for flag in ("-d", "--detach")) else ["-d"]
+            cmd = base + ["up"] + up_flags + extra
+        else:
+            cmd = base + [action] + extra
+
         if dry_run:
-            log.info(f"[DRY-RUN] (cd {dirpath} && {' '.join(cmd)})")
+            log.info(f"[DRY-RUN] (cd {shlex.quote(dirpath)} && {shlex.join(cmd)})")
         else:
             log.info(f"{action} {name}...")
             ctx.sys.run(cmd, cwd=dirpath, env=env, check=True)
 
     for d in sequence():
         run_one(d)
-    log.success(f"Orchestrated {action}.")
+    log.success(f"Orchestrated {action} ({target}).")

actions/configure.py

@@ -1,12 +1,12 @@
 # actions/configure.py
-"""Umbrella: run every configure/* action in a stable order (SP2 §3.1)."""
+"""Run every configure/* action in a stable order."""
 import os
 from engine.descriptor import Meta, Flag, build_parser
 from engine import overlay
 
 METADATA = Meta(summary="Run every configure/* action in stable order.",
                 args=[Flag("--dry-run", "Pass --dry-run to sub-actions that support it.")])
-ORDER = ["configure/base", "configure/ufw", "configure/crowdsec",
+ORDER = ["configure/base", "configure/timezone", "configure/ufw", "configure/crowdsec",
          "configure/cron", "configure/systemd", "configure/routing"]
 
 def run(ctx, args):
@@ -21,7 +21,7 @@ def run(ctx, args):
         mod = overlay.load_module(r.path, "apex_umbrella_" + name.replace("/", "_"))
         if getattr(mod, "DISABLED", None):
             ctx.log.warn(f"Skipping disabled {name}"); continue
-        meta = getattr(mod, "METADATA")
+        meta = mod.METADATA
         has_dry = any(getattr(a, "name", None) == "--dry-run" for a in meta.args)
         argv = ["--dry-run"] if (args.dry_run and has_dry) else []
         sub_args = build_parser(meta, f"apex {name}").parse_args(argv)

actions/configure/alloy.py

@@ -0,0 +1,77 @@
+"""Deploy the opt-in native Alloy service using shared host/guest configs."""
+import json
+from pathlib import Path
+import re
+import tempfile
+
+from engine.descriptor import Flag, Meta
+
+METADATA = Meta(summary="Configure native Alloy from configs/alloy/config.json.",
+                args=[Flag("--dry-run", "Validate the deployment plan without changes.")])
+
+
+def deployment(ctx):
+    settings = Path(ctx.paths.configs) / "alloy/config.json"
+    if not settings.exists():
+        return None
+    values = json.loads(settings.read_text())
+    role = values.get("role")
+    if role not in ("host", "guest"):
+        raise ValueError("Alloy role must be host or guest")
+    env_file = values.get("environment_file", "")
+    if not re.fullmatch(r"/[A-Za-z0-9_./-]+", env_file):
+        raise ValueError("Alloy environment_file must be an absolute path without whitespace")
+    source = Path(ctx.paths.commons) / "etc/alloy" / (
+        "native-host.alloy" if role == "host" else "native.alloy")
+    config = source.read_text()
+    extra = Path(ctx.paths.configs) / "alloy/extra.alloy"
+    if extra.is_file():
+        config += "\n" + extra.read_text()
+    unit = ("[Service]\nEnvironmentFile=" + env_file + "\n"
+            'ExecStart=\nExecStart=/usr/bin/alloy run --storage.path=/var/lib/alloy --server.http.listen-addr=127.0.0.1:12346 /etc/alloy/config.alloy\n'
+            + ("User=root\nGroup=root\n" if role == "guest" else "") +
+            "SupplementaryGroups=systemd-journal" + (" docker" if role == "guest" else "") + "\n")
+    return config, unit, env_file
+
+
+def run(ctx, args):
+    plan = deployment(ctx)
+    if plan is None:
+        ctx.log.info("Native Alloy is not configured for this node; skipping.")
+        return
+    config, unit, env_file = plan
+    if args.dry_run:
+        ctx.log.info("Native Alloy deployment plan validated; no changes made.")
+        return
+    ctx.sys.sudo(["test", "-f", env_file])
+    # Validate with the actual service environment without printing credentials.
+    validator = '''import os, stat, subprocess, sys
+from pathlib import Path
+# Environment files deliberately use plain KEY=value, with no shell evaluation.
+secret = Path(sys.argv[1])
+info = secret.stat()
+if info.st_uid != 0 or stat.S_IMODE(info.st_mode) & 0o077:
+    raise RuntimeError('Alloy environment file must be root-owned with mode 0600 or stricter')
+for line in secret.read_text().splitlines():
+    if not line.strip() or line.lstrip().startswith('#'): continue
+    key, value = line.split('=', 1)
+    os.environ[key] = value
+subprocess.run(['alloy', 'validate', sys.argv[2]], check=True)
+'''
+    with tempfile.TemporaryDirectory(prefix="apex-alloy-") as directory:
+        config_path = Path(directory) / "config.alloy"
+        unit_path = Path(directory) / "apex.conf"
+        config_path.write_text(config)
+        unit_path.write_text(unit)
+        ctx.sys.sudo(["python3", "-c", validator, env_file, str(config_path)])
+        ctx.sys.sudo(["install", "-d", "-m", "0755", "/etc/alloy",
+                      "/etc/systemd/system/alloy.service.d"])
+        ctx.sys.sudo(["install", "-m", "0644", str(config_path), "/etc/alloy/config.alloy"])
+        ctx.sys.sudo(["install", "-m", "0644", str(unit_path),
+                      "/etc/systemd/system/alloy.service.d/apex.conf"])
+    ctx.sys.sudo(["systemctl", "unmask", "alloy.service"])
+    ctx.sys.daemon_reload()
+    ctx.sys.sudo(["systemctl", "enable", "alloy.service"])
+    ctx.sys.sudo(["systemctl", "restart", "alloy.service"])
+    ctx.sys.sudo(["systemctl", "is-active", "--quiet", "alloy.service"])
+    ctx.log.success("Configured native Alloy.")

actions/configure/base.py

@@ -1,31 +1,131 @@
 # actions/configure/base.py
 """Port of configure/base — base packages + ufw + ufw-docker helper."""
-import os, shutil, stat, urllib.request
+import hashlib
+import os
+from pathlib import Path
+import re
+import stat
+import urllib.request
 from engine.descriptor import Meta
 
 METADATA = Meta(summary="Configure base system packages (curl, wget, git, rsyslog, ufw, ufw-docker).")
-UFW_DOCKER_URL = "https://github.com/chaifeng/ufw-docker/raw/master/ufw-docker"
+UFW_DOCKER_REVISION = "020a8699f95592561f254d8d4ad1bb40d401dfc7"
+UFW_DOCKER_SHA256 = "643e56b080567c567b4aa28650196849b2a2da5dd0473fd3e5216b0886035ab0"
+UFW_DOCKER_URL = (
+    "https://raw.githubusercontent.com/chaifeng/ufw-docker/"
+    + UFW_DOCKER_REVISION + "/ufw-docker"
+)
+UFW_DOCKER_PATH = Path("/usr/local/bin/ufw-docker")
+BASE_PACKAGES = ("curl", "wget", "git", "rsyslog", "ufw")
+
+
+def _install_missing(s, packages):
+    missing = []
+    for package in packages:
+        result = s.run(["dpkg-query", "-W", "-f=${Status}", package],
+                       check=False, capture=True)
+        if result.returncode or result.stdout.split()[1:] != ["ok", "installed"]:
+            missing.append(package)
+    if not missing:
+        return
+    apt = ["env", "LC_ALL=C", "DEBIAN_FRONTEND=noninteractive", "apt-get",
+           "-o", "DPkg::Lock::Timeout=60", "-o", "Acquire::Retries=2",
+           "-o", "Acquire::http::Timeout=30", "-o", "Acquire::https::Timeout=30"]
+    s.sudo([*apt, "update", "--error-on=any"])
+    command = [*apt, "install", "-y", "--no-install-recommends", "--no-upgrade",
+               "--no-remove", *missing]
+    plan = s.sudo([*command, "--simulate"], capture=True)
+    if any(line.startswith("Remv ") or re.match(r"^Inst \S+ \[", line)
+           for line in plan.stdout.splitlines()):
+        raise RuntimeError("Missing packages require upgrades/removals; reconcile packages first.")
+    s.sudo(command)
+
+
+def _check_helper_path(path):
+    for parent in reversed(path.parents):
+        info = parent.lstat()
+        if (not stat.S_ISDIR(info.st_mode) or info.st_uid != 0 or
+                info.st_mode & 0o022):
+            raise RuntimeError(f"Unsafe ufw-docker ancestor: {parent}")
+    try:
+        info = path.lstat()
+    except FileNotFoundError:
+        return False
+    if (not stat.S_ISREG(info.st_mode) or info.st_uid != 0 or info.st_gid != 0 or
+            info.st_mode & 0o022 or info.st_nlink != 1):
+        raise RuntimeError(f"Unsafe ufw-docker target: {path}")
+    return True
+
+
+def _install_helper(ctx):
+    path = UFW_DOCKER_PATH
+    exists = _check_helper_path(path)
+    payload = path.read_bytes() if exists else None
+    if payload is None or hashlib.sha256(payload).hexdigest() != UFW_DOCKER_SHA256:
+        ctx.log.info(f"Downloading pinned ufw-docker helper to {path}...")
+        # The immutable HTTPS URL is authenticated by the digest below.
+        with urllib.request.urlopen(UFW_DOCKER_URL, timeout=30) as response:  # noqa: S310
+            payload = response.read(1024 * 1024 + 1)
+        if len(payload) > 1024 * 1024 or hashlib.sha256(payload).hexdigest() != UFW_DOCKER_SHA256:
+            raise RuntimeError("ufw-docker checksum mismatch; refusing installation.")
+        _check_helper_path(path)
+        ctx.host.write_file(str(path), payload.decode("utf-8"), mode="0755",
+                            backup=False, sudo=True)
+    elif stat.S_IMODE(path.stat().st_mode) != 0o755:
+        ctx.host.write_file(str(path), payload.decode("utf-8"), mode="0755",
+                            backup=False, sudo=True)
+    if (not _check_helper_path(path) or
+            hashlib.sha256(path.read_bytes()).hexdigest() != UFW_DOCKER_SHA256 or
+            stat.S_IMODE(path.stat().st_mode) != 0o755):
+        raise RuntimeError("ufw-docker verification failed; refusing execution.")
+    ctx.log.info("Ensuring ufw-docker routing rules are installed in UFW...")
+    ctx.sys.sudo([str(path), "install"])
+
 
 def run(ctx, args):
     log, s = ctx.log, ctx.sys
     log.info("Starting base system initialization...")
     if not os.path.isfile("/etc/debian_version"):
-        log.error("Requires a Debian-based system (Debian/Ubuntu)."); raise SystemExit(1)
-    log.info("Updating package lists..."); s.sudo(["apt-get", "update", "-y"])
-    log.info("Installing essential packages...")
-    s.sudo(["apt-get", "install", "-y", "curl", "wget", "git", "rsyslog"])
+        log.error("Requires a Debian-based system (Debian/Ubuntu).")
+        raise SystemExit(1)
+    _check_helper_path(UFW_DOCKER_PATH)
+    _install_missing(s, BASE_PACKAGES)
     s.ensure_running("rsyslog")
-    if not shutil.which("ufw"):
-        log.info("Installing UFW..."); s.sudo(["apt-get", "install", "-y", "ufw"])
-    else:
-        log.info("UFW already installed.")
-    binp = os.path.expanduser("~/.local/bin/ufw-docker")
-    if not (os.path.isfile(binp) and os.access(binp, os.X_OK)):
-        log.info("Downloading ufw-docker helper...")
-        os.makedirs(os.path.dirname(binp), exist_ok=True)
-        urllib.request.urlretrieve(UFW_DOCKER_URL, binp)       # stdlib; replaces wget/curl
-        os.chmod(binp, os.stat(binp).st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
-        log.success(f"ufw-docker downloaded to {binp}.")
-    else:
-        log.info(f"ufw-docker already exists at {binp}.")
+
+    # Configure APT to discard downloaded archives
+    apt_clean_path = "/etc/apt/apt.conf.d/99-apex"
+    apt_clean_content = 'APT::Keep-Downloaded-Packages "0";\n'
+    if os.path.exists("/etc/apt/apt.conf.d/01clean"):
+        s.sudo(["rm", "-f", "/etc/apt/apt.conf.d/01clean"])
+    current_apt = None
+    if os.path.exists(apt_clean_path):
+        try:
+            with open(apt_clean_path, encoding="utf-8") as f:
+                current_apt = f.read()
+        except OSError:
+            current_apt = None
+    if current_apt != apt_clean_content:
+        log.info("Configuring APT to discard downloaded archives...")
+        s.sudo(["tee", apt_clean_path], input=apt_clean_content)
+
+    # Limit systemd-journald max disk usage
+    journal_d = "/etc/systemd/journald.conf.d"
+    journal_conf = os.path.join(journal_d, "99-apex.conf")
+    journal_content = "[Journal]\nSystemMaxUse=200M\n"
+    if os.path.exists(os.path.join(journal_d, "maxuse.conf")):
+        s.sudo(["rm", "-f", os.path.join(journal_d, "maxuse.conf")])
+    current_journal = None
+    if os.path.exists(journal_conf):
+        try:
+            with open(journal_conf, encoding="utf-8") as f:
+                current_journal = f.read()
+        except OSError:
+            current_journal = None
+    if current_journal != journal_content:
+        log.info("Configuring systemd-journald SystemMaxUse=200M...")
+        s.sudo(["mkdir", "-p", journal_d])
+        s.sudo(["tee", journal_conf], input=journal_content)
+        s.sudo(["systemctl", "restart", "systemd-journald"])
+
+    _install_helper(ctx)
     log.success("Base system initialization completed.")

actions/configure/timezone.py

@@ -0,0 +1,33 @@
+"""Apply the optional node timezone before installing scheduled jobs."""
+from pathlib import Path
+from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
+
+from engine.descriptor import Meta, Flag
+
+METADATA = Meta(summary="Apply configs/timezone and refresh cron when the timezone changes.",
+                args=[Flag("--dry-run", "Validate and preview without changing the host.")])
+
+
+def run(ctx, args):
+    source = Path(ctx.paths.configs) / "timezone"
+    if not source.is_file():
+        ctx.log.info("No configs/timezone declaration; preserving host timezone.")
+        return
+    zone = source.read_text().strip()
+    try:
+        ZoneInfo(zone)
+    except (ValueError, ZoneInfoNotFoundError):
+        ctx.log.error("configs/timezone must name an installed IANA timezone.")
+        raise SystemExit(1) from None
+    current = ctx.sys.run(["timedatectl", "show", "--property=Timezone", "--value"], capture=True).stdout.strip()
+    if current == zone:
+        ctx.log.info(f"Timezone already set to {zone}.")
+        return
+    if args.dry_run:
+        ctx.log.info(f"[DRY-RUN] would change timezone from {current} to {zone} and refresh active cron.")
+        return
+    ctx.sys.sudo(["timedatectl", "set-timezone", zone])
+    if ctx.sys.service_is_active("cron") and not ctx.sys.restart("cron"):
+        ctx.log.error("Timezone changed, but cron did not become active after restart.")
+        raise SystemExit(1)
+    ctx.log.success(f"Timezone set to {zone}.")

actions/configure/ufw.py

@@ -13,12 +13,12 @@ def run(ctx, args):
     docker_dir = os.path.join(cfg, "docker")
     host_dir = os.path.join(cfg, "host")
     ctx.log.info("Configuring UFW rules...")
-    # Mirrors the bash branch structure: host dir is passed unconditionally (a missing
-    # one exits 66 inside apply); both dirs missing = warn + success, like bash.
-    if os.path.isdir(docker_dir):
-        ctx.ufw.apply(docker_dir, host_dir, dry_run=args.dry_run)
-    else:
+    has_docker = os.path.isdir(docker_dir)
+    has_host = os.path.isdir(host_dir)
+    if not has_docker:
         ctx.log.warn(f"Docker rules dir not found at {docker_dir}. Skipping ufw-docker rules.")
-        if os.path.isdir(host_dir):
-            ctx.ufw.apply(docker_dir, host_dir, dry_run=args.dry_run)
+    if not has_host:
+        ctx.log.warn(f"Host rules dir not found at {host_dir}. Skipping host rules.")
+    if has_docker or has_host:
+        ctx.ufw.apply(docker_dir if has_docker else None, host_dir if has_host else None, dry_run=args.dry_run)
     ctx.log.success("Configured ufw.")

actions/env.py

@@ -0,0 +1,82 @@
+"""Export synthesized APEX values or bridge a composition's environment for Compose."""
+import os
+import shlex
+
+from engine.descriptor import Flag, Meta, Opt
+
+METADATA = Meta(
+    summary="Export or sync synthesized APEX_* variables for shell or docker compose.",
+    args=[
+        Flag("--export", "Format as shell export statements (default)."),
+        Flag("--dotenv", "Print quoted generated APEX_* dotenv values only."),
+        Opt("--write", "Write a standalone generated dotenv file (0600).", default=None),
+        Flag("--sync", "Preserve user .env text and refresh the core Compose managed block."),
+    ],
+)
+
+_START = "# BEGIN APEX MANAGED ENV"
+_END = "# END APEX MANAGED ENV"
+
+
+def _dotenv_value(value):
+    """Quote a literal using Compose dotenv escapes, including interpolation dollars."""
+    value = str(value)
+    for old, new in (("\\", "\\\\"), ('"', '\\"'), ("$", "$$"),
+                     ("\r", "\\r"), ("\n", "\\n"), ("\t", "\\t")):
+        value = value.replace(old, new)
+    return f'"{value}"'
+
+
+def _read_optional(path):
+    try:
+        with open(path, encoding="utf-8", newline="") as source:
+            return source.read()
+    except FileNotFoundError:
+        return ""
+
+
+def _sync_body(existing, public, generated):
+    if any(marker in public or marker in generated for marker in (_START, _END)):
+        raise ValueError("Managed environment marker collides with source content.")
+    lines = existing.splitlines(keepends=True)
+    markers = [(index, line.rstrip("\r\n")) for index, line in enumerate(lines)
+               if _START in line or _END in line]
+    if markers:
+        if (len(markers) != 2 or markers[0][1] != _START or markers[1][1] != _END):
+            raise ValueError("Malformed or duplicate managed environment markers; .env unchanged.")
+        start, end = markers[0][0], markers[1][0]
+        existing = "".join(lines[:start] + lines[end + 1:])
+    # Keep user bytes in order, but place managed values last to preserve precedence.
+    separator = "" if not existing or existing.endswith("\n") else "\n"
+    public_separator = "" if not public or public.endswith("\n") else "\n"
+    return (existing + separator + _START + "\n" + public + public_separator
+            + generated + _END + "\n")
+
+
+def run(ctx, args):
+    modes = (args.export, args.dotenv, args.write is not None, args.sync)
+    if sum(bool(mode) for mode in modes) > 1:
+        ctx.log.error("Choose only one of --export, --dotenv, --write, or --sync.")
+        raise SystemExit(64)
+    variables = {k: v for k, v in ctx.vars().items() if k.startswith("APEX_")}
+    generated = "".join(f"{key}={_dotenv_value(value)}\n"
+                        for key, value in sorted(variables.items()))
+    target = args.write
+    if args.sync:
+        core = os.path.join(ctx.paths.compositions, "apex")
+        target = os.path.join(core, ".env")
+        try:
+            generated = _sync_body(_read_optional(target),
+                                   _read_optional(os.path.join(core, "apex.env")), generated)
+        except ValueError as error:
+            ctx.log.error(str(error))
+            raise SystemExit(65) from error
+    if target is not None:
+        ctx.host.write_file(target, generated, mode="0600")
+        ctx.log.success(f"Wrote {len(variables)} synthesized variables to {target}.")
+    elif args.dotenv:
+        print(generated, end="")
+    else:
+        for key, value in sorted(variables.items()):
+            print(f"export {key}={shlex.quote(str(value))}")
+    return 0

actions/sync/packages.py

@@ -1,12 +1,216 @@
 # actions/sync/packages.py
-"""Port of sync/packages — apt upgradable + docker image (skopeo) update check; Telegram."""
-import os, re, shutil
+"""Check pending APT and running-image updates without pulling images."""
+import json
+import os
+import re
+import shutil
 from engine.descriptor import Meta, Arg
 
 METADATA = Meta(summary="Check pending apt updates + docker image updates; Telegram notify.",
                 args=[Arg("telegram_bot_url", "Telegram Bot URL for notifications")])
 TITLE = "Node Security Updates"
 
+MANIFEST_TYPES = {"application/vnd.oci.image.manifest.v1+json",
+                  "application/vnd.docker.distribution.manifest.v2+json"}
+INDEX_TYPES = {"application/vnd.oci.image.index.v1+json",
+               "application/vnd.docker.distribution.manifest.list.v2+json"}
+
+
+def _output(s, command):
+    result = s.run(command, check=False, capture=True)
+    if result.returncode:
+        # Registry errors can contain authentication details; don't relay stderr.
+        raise ValueError("inspection command failed")
+    return result.stdout or ""
+
+
+def _digest(value):
+    if not isinstance(value, str) or not re.fullmatch(r"sha256:[0-9a-f]{64}", value):
+        raise ValueError("missing or unsupported image digest")
+    return value
+
+
+def _platform(os_name, architecture, variant=""):
+    if not isinstance(os_name, str) or not os_name or not isinstance(architecture, str) or not architecture:
+        raise ValueError("missing image platform")
+    variant = variant or ""
+    if architecture == "arm64" and variant == "v8":
+        variant = ""
+    return os_name, architecture, variant
+
+
+def _repository(reference):
+    name = reference.split("@", 1)[0]
+    # Only a colon in the final path component is a tag separator.
+    return name.rsplit(":", 1)[0] if ":" in name.rsplit("/", 1)[-1] else name
+
+
+def _remote_digests(s, reference, local, cache, running_manifest=None):
+    def inspect(ref, raw=False, field="Manifest"):
+        command = ["docker", "buildx", "imagetools", "inspect"]
+        command += ["--raw"] if raw else ["--format", "{{json ." + field + "}}"]
+        key = tuple(command + [ref])
+        if key not in cache:
+            value = json.loads(_output(s, ["timeout", "--signal=TERM", "--kill-after=5s",
+                                          "30s", *key]))
+            if not isinstance(value, dict):
+                raise ValueError("invalid registry JSON object")
+            cache[key] = value
+        return cache[key]
+
+    platform = _platform(local.get("Os"), local.get("Architecture"), local.get("Variant"))
+    descriptor = inspect(reference)
+    media_type = descriptor.get("mediaType")
+    selected = _digest(descriptor.get("digest"))
+    if media_type in INDEX_TYPES:
+        manifests = descriptor.get("manifests")
+        if not isinstance(manifests, list):
+            raise ValueError("invalid image index")
+        matches = []
+        for item in manifests:
+            p = item.get("platform", {})
+            if _platform(p.get("os"), p.get("architecture"), p.get("variant")) == platform:
+                if item.get("mediaType") not in MANIFEST_TYPES:
+                    raise ValueError("unsupported nested image index")
+                matches.append(_digest(item.get("digest")))
+        if len(matches) != 1:
+            raise ValueError("missing or ambiguous matching image platform")
+        selected = matches[0]
+    elif media_type not in MANIFEST_TYPES:
+        raise ValueError("unsupported registry manifest")
+
+    immutable = _repository(reference) + "@" + selected
+    manifest = inspect(immutable, raw=True)
+    if manifest.get("schemaVersion") != 2 or manifest.get("mediaType") not in MANIFEST_TYPES:
+        raise ValueError("invalid platform manifest")
+    remote_id = _digest(manifest.get("config", {}).get("digest"))
+    if media_type in MANIFEST_TYPES and selected != running_manifest and remote_id != local["Id"]:
+        config = inspect(immutable, field="Image")
+        if _platform(config.get("os"), config.get("architecture"), config.get("variant")) != platform:
+            raise ValueError("registry image platform changed")
+    return selected, remote_id
+
+
+def _compose_builds(ctx):
+    builds = {}
+    env = dict(os.environ)
+    env.update({k: v for k, v in ctx.vars().items() if k.startswith("APEX_")})
+    for name in sorted(os.listdir(ctx.paths.compositions)):
+        directory = os.path.join(ctx.paths.compositions, name)
+        if name.startswith((".", "@")) or not os.path.isfile(os.path.join(directory, "docker-compose.yml")):
+            continue
+        command = ["docker", "compose"]
+        for filename in (".env", "apex.env"):
+            if os.path.isfile(os.path.join(directory, filename)):
+                command += ["--env-file", filename]
+        # Resolved Compose includes secrets: capture privately and never log it.
+        result = ctx.sys.run(["timeout", "--signal=TERM", "--kill-after=5s", "30s",
+                              *command, "config", "--format", "json"],
+                             cwd=directory, env=env, check=False, capture=True)
+        if result.returncode:
+            raise ValueError("Compose inspection failed")
+        config = json.loads(result.stdout)
+        project, services = config["name"], config["services"]
+        if not isinstance(project, str) or not project or not isinstance(services, dict):
+            raise ValueError("invalid Compose model")
+        for service, settings in services.items():
+            if settings.get("build") is not None:
+                reference = settings.get("image")
+                if reference is not None and not isinstance(reference, str):
+                    raise ValueError("invalid Compose image")
+                builds[(project, service)] = reference
+    return builds
+
+
+def _images(s, log, ctx=None):
+    if not shutil.which("docker"):
+        return "Images: unchecked (Docker unavailable)"
+    if not shutil.which("timeout"):
+        return "Images: unchecked (coreutils timeout unavailable)"
+    try:
+        ids = _output(s, ["docker", "ps", "--quiet", "--no-trunc"]).split()
+        if not ids:
+            return "Images: no running containers"
+        output = _output(s, ["docker", "container", "inspect", "--format",
+                            '{"Image":{{json .Image}},"Reference":{{json .Config.Image}},'
+                            '"ImageManifestDescriptor":{{json (index . "ImageManifestDescriptor")}},'
+                            '"Labels":{{json .Config.Labels}}}', *ids])
+        containers = [json.loads(line) for line in output.splitlines()]
+        if len(containers) != len(ids):
+            raise ValueError("incomplete running-container inspection")
+        if any(not isinstance(c["Reference"], str) for c in containers):
+            raise ValueError("invalid running-image reference")
+    except (ValueError, KeyError, TypeError, OSError) as error:
+        log.warn(f"Images unchecked: {type(error).__name__}")
+        return "Images: unchecked (container inspection failed)"
+
+    try:
+        builds = _compose_builds(ctx) if ctx is not None else {}
+    except (ValueError, KeyError, TypeError, AttributeError, OSError):
+        log.warn("Images unchecked: Compose build provenance could not be resolved")
+        return "Images: unchecked (Compose inspection failed)"
+
+    buildx = s.run(["docker", "buildx", "version"], check=False, capture=True).returncode == 0
+    counts = dict.fromkeys(("updates", "current", "pinned", "local/untracked", "unchecked"), 0)
+    cache, images, seen = {}, {}, set()
+    for container in containers:
+        reference = container["Reference"]
+        reason = ""
+        try:
+            image_id = _digest(container["Image"])
+            descriptor = container.get("ImageManifestDescriptor")
+            labels = container.get("Labels") or {}
+            build_key = (labels.get("com.docker.compose.project"), labels.get("com.docker.compose.service"))
+            declared_build = build_key in builds and builds[build_key] in (None, reference)
+            key = (reference, image_id, json.dumps(descriptor, sort_keys=True), declared_build)
+            if key in seen:
+                continue
+            seen.add(key)
+            if not isinstance(reference, str) or not reference or reference.startswith("-"):
+                raise ValueError("invalid image reference")
+            if "@" in reference or re.fullmatch(r"(?:sha256:)?[0-9a-f]{64}", reference):
+                status = "pinned"
+            elif declared_build:
+                status = "local/untracked"
+            else:
+                if image_id not in images:
+                    data = json.loads(_output(s, ["docker", "image", "inspect", image_id]))
+                    if not isinstance(data, list) or len(data) != 1 or data[0].get("Id") != image_id:
+                        raise ValueError("invalid running-image inspection")
+                    images[image_id] = data[0]
+                local = images[image_id]
+                repo_digests = local["RepoDigests"]
+                if repo_digests is None or repo_digests == []:
+                    status = "local/untracked"
+                elif not isinstance(repo_digests, list):
+                    raise ValueError("invalid repository digests")
+                elif not buildx:
+                    raise ValueError("Buildx unavailable")
+                else:
+                    running_manifest = None
+                    if descriptor is not None:
+                        if descriptor.get("mediaType") not in MANIFEST_TYPES:
+                            raise ValueError("invalid running platform manifest")
+                        running_manifest = _digest(descriptor.get("digest"))
+                        platform = descriptor.get("platform", {})
+                        os_name, architecture, variant = _platform(
+                            platform.get("os"), platform.get("architecture"), platform.get("variant"))
+                        local = dict(local, Os=os_name, Architecture=architecture, Variant=variant)
+                    elif local.get("Descriptor") is not None:
+                        raise ValueError("containerd running platform manifest unavailable")
+                    remote_manifest, remote_config = _remote_digests(
+                        s, reference, local, cache, running_manifest)
+                    # Classic Docker IDs are configs; containerd IDs may be indices.
+                    current = (remote_manifest == running_manifest if running_manifest
+                               else remote_config == image_id)
+                    status = "current" if current else "updates"
+        except (ValueError, KeyError, TypeError, AttributeError, OSError) as error:
+            status, reason = "unchecked", str(error) if isinstance(error, ValueError) else type(error).__name__
+        counts[status] += 1
+        log.info(f"Image {reference}: {status}" + (f" ({reason})" if reason else ""))
+    return "Images: " + ", ".join(f"{count} {status}" for status, count in counts.items())
+
+
 def run(ctx, args):
     log, node, url, s = ctx.log, ctx.node.name, args.telegram_bot_url, ctx.sys
     try:
@@ -15,13 +219,13 @@ def run(ctx, args):
         raise
     except Exception as e:   # bash ERR trap parity: unhandled failures page via Telegram
         log.error(f"{type(e).__name__}: {e}")
-        ctx.notify.error(TITLE, url, f"Critical error: {e}", node)
-        raise SystemExit(1)
+        ctx.notify.error(TITLE, url, f"Critical error: {e}")
+        raise SystemExit(1) from e
 
 def _check(ctx, args, log, node, url, s):
     if not os.path.isfile("/etc/debian_version"):
         log.error("Requires a Debian-based system.")
-        ctx.notify.error(TITLE, url, "Requires a Debian-based system.", node)
+        ctx.notify.error(TITLE, url, "Requires a Debian-based system.")
         raise SystemExit(1)
 
     log.info("Updating package lists...")
@@ -31,41 +235,15 @@ def _check(ctx, args, log, node, url, s):
                    check=False, capture=True)
         upd = [l for l in (cp.stdout or "").splitlines() if l.strip()]
         packages = (f"*Packages ({len(upd)}):* " + ", ".join(f"`{u}`" for u in upd)) if upd else "*Packages:* _none_"
+        apt_summary = f"APT: {len(upd)} pending"
     else:
         packages = "*Packages:* _Failed to check_"
+        apt_summary = "APT: failed to check"
 
-    if not shutil.which("docker"):
-        images = "*Images:* _Docker not available_"
-    elif not shutil.which("skopeo"):
-        images = "*Images:* _skopeo not available_"
-    else:
-        cp = s.run(["bash", "-c", "docker ps --format '{{.Image}}' | sort -u"], check=False, capture=True)
-        outdated = []
-        for image in [l for l in (cp.stdout or "").splitlines() if l.strip()]:
-            name, sep, tag = image.partition(":")
-            tag = tag or "latest"
-            ld = s.run(["bash", "-c",
-                        f"docker image inspect '{image}' --format '{{{{index .RepoDigests 0}}}}' 2>/dev/null | cut -d'@' -f2"],
-                       check=False, capture=True).stdout.strip()
-            if not ld:
-                continue
-            if "/" not in name:
-                reg = f"docker.io/library/{name}"
-            elif not re.match(r".*\..*/", name):
-                reg = f"docker.io/{name}"
-            else:
-                reg = name
-            rd = s.run(["bash", "-c",
-                        f"skopeo inspect 'docker://{reg}:{tag}' 2>/dev/null | "
-                        f"grep -o '\"Digest\"[[:space:]]*:[[:space:]]*\"[^\"]*\"' | cut -d'\"' -f4"],
-                       check=False, capture=True).stdout.strip()
-            if rd and ld != rd:
-                disp = name[len("docker.io/"):] if name.startswith("docker.io/") else name
-                disp = disp[len("library/"):] if disp.startswith("library/") else disp
-                outdated.append(disp)
-        images = (f"*Images ({len(outdated)}):* " + ", ".join(f"`{i}`" for i in outdated)) if outdated else "*Images:* _none_"
+    images = _images(s, log, ctx)
+    log.info(packages)
 
     # A failed send exits 1 (bash: telegram_info's return 1 tripped set -e).
-    if not ctx.notify.info(TITLE, url, f"*APT:*\n{packages}\n\n*Skopeo:*\n{images}", node):
+    if not ctx.notify.info(TITLE, url, f"{apt_summary}\n{images}\n\n{packages}"):
         raise SystemExit(1)
     log.success("Security update check completed.")

actions/sync/repository.py

@@ -1,60 +1,59 @@
-# actions/sync/repository.py
-"""Port of sync/repository — capture-up commit + rebase onto origin/main + force-push."""
+"""Commit node configuration and synchronize main directly with its remote."""
 import os
 from datetime import datetime
 from engine.descriptor import Meta, Arg
+from engine.identity import read_env
+
+METADATA = Meta(summary="Commit node changes, rebase onto origin/main, and push main.",
+                args=[Arg("telegram_bot_url", "Optional notification URL", required=False)])
+TITLE = "Repository Sync"
 
-METADATA = Meta(summary="Capture-up: commit node changes, rebase onto origin/main, force-push sync/<node>.",
-                args=[Arg("telegram_bot_url", "Telegram Bot URL for notifications")])
-TITLE = "Annual Remote Sync"
 
 def run(ctx, args):
-    log, node, url = ctx.log, ctx.node.name, args.telegram_bot_url
+    log, node = ctx.log, ctx.node.name
+    url = args.telegram_bot_url
     repo, commons = ctx.paths.repo_root, ctx.paths.commons
-    env = dict(os.environ,
-               GIT_AUTHOR_NAME="Adam Jensen [bot]", GIT_AUTHOR_EMAIL="adam@ermnvldmr.com",
-               GIT_COMMITTER_NAME="Adam Jensen [bot]", GIT_COMMITTER_EMAIL="adam@ermnvldmr.com")
+    ne = read_env(os.path.join(repo, "node.env"))
+    author = ne.get("APEX_GIT_AUTHOR_NAME", "apex [bot]")
+    email = ne.get("APEX_GIT_AUTHOR_EMAIL", "apex@localhost")
+    env = dict(os.environ, GIT_AUTHOR_NAME=author, GIT_AUTHOR_EMAIL=email,
+               GIT_COMMITTER_NAME=author, GIT_COMMITTER_EMAIL=email)
 
-    def git(*a, check=True):
-        return ctx.sys.run(["git", "-C", repo, *a], check=check, env=env)
+    def git(*a, check=True, capture=False):
+        return ctx.sys.run(["git", "-C", repo, *a], check=check, capture=capture, env=env)
 
     try:
+        branch = git("symbolic-ref", "--quiet", "--short", "HEAD", capture=True).stdout.strip()
+        if branch != "main":
+            raise RuntimeError("Repository sync requires main; switch branches explicitly first.")
         cp = ctx.sys.run(["git", "-C", commons, "status", "--porcelain"], capture=True)
         if (cp.stdout or "").strip():
-            log.error(f"Commons submodule at {commons} has uncommitted changes. Aborting.")
-            raise SystemExit(1)
+            raise RuntimeError(f"Commons submodule at {commons} has uncommitted changes.")
         if not os.path.isdir(ctx.paths.compositions):
-            # Without this, `git add -A` would stage a deleted node tree and an
-            # unattended cron run would commit + force-push the deletion.
-            log.error(f"Directory '{ctx.paths.compositions}' does not exist. Cannot sync.")
-            raise SystemExit(1)
-        branch = f"sync/{node}"
-        ts = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
-        if ctx.sys.ok(["git", "-C", repo, "rev-parse", "--verify", branch]):
-            git("checkout", branch)
-        else:
-            git("checkout", "-b", branch)
+            raise RuntimeError(f"Directory '{ctx.paths.compositions}' does not exist.")
+        expected = git("rev-parse", "HEAD:commons", capture=True).stdout.strip()
+        actual = ctx.sys.run(["git", "-C", commons, "rev-parse", "HEAD"], capture=True).stdout.strip()
+        if actual != expected:
+            raise RuntimeError("Commons checkout differs from the committed pin; update the submodule before retrying.")
         git("add", "-A")
-        git("submodule", "update", "--remote", "--recursive")
-        git("add", "commons")
         if git("diff", "--cached", "--quiet", check=False).returncode != 0:
+            ts = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
             git("commit", "-m", f"[{node}] Auto-commit: {ts}")
         else:
             log.info("No changes to commit.")
         git("fetch", "origin", "main")
         if git("rebase", "origin/main", check=False).returncode != 0:
-            log.error("Rebase conflict — aborting rebase; manual intervention required.")
             git("rebase", "--abort", check=False)
-            ctx.notify.error(TITLE, url, "Rebase conflict. Manual intervention required.", node)
-            raise SystemExit(1)
+            raise RuntimeError("Rebase conflict; local commits retained for manual resolution.")
         git("submodule", "update", "--init", "--recursive")
-        git("push", "--force-with-lease", "origin", branch)
-        log.success(f"Pushed {branch}.")
-        if not ctx.notify.success(TITLE, url, f"Sync to '{branch}' completed.", node):
+        git("push", "origin", "HEAD:refs/heads/main")
+        log.success("Synchronized main.")
+        if url and not ctx.notify.success(TITLE, url, "Sync to main completed."):
             raise SystemExit(1)
     except SystemExit:
         raise
     except Exception as e:
         log.error(f"{type(e).__name__}: {e}")
-        ctx.notify.error(TITLE, url, f"Sync failed: {e}", node)
-        raise SystemExit(1)
+        if url:
+            ctx.notify.error(TITLE, url, f"Sync failed: {e}")
+        raise SystemExit(1) from e

actions/tiers/chown.py

@@ -13,10 +13,11 @@ SHARED_GROUP = "noroot-shared"
 def _env_val(path, key):
     if not os.path.exists(path):
         return ""
-    for line in open(os.path.realpath(path)):
-        line = line.strip()
-        if line.startswith(key + "="):
-            return line.split("=", 1)[1].strip()
+    with open(os.path.realpath(path), encoding="utf-8") as f:
+        for line in f:
+            line = line.strip()
+            if line.startswith(key + "="):
+                return line.split("=", 1)[1].strip()
     return ""
 
 def run(ctx, args):

actions/utils/extract-traefik-certs.py

@@ -21,15 +21,16 @@ def run(ctx, args):
         abs_dest = os.path.realpath(args.dest_dir)
         log.info(f"Extracting certs: {abs_src} -> {abs_dest}")
         ctx.sys.run(["docker", "run", "--rm", "-v", f"{src_dir}:/data", "-v", f"{abs_dest}:/output",
-                     "ldez/traefik-certs-dumper:v2.9.3", "file", "--version", "v3",
+                     "ldez/traefik-certs-dumper:v2.11.4", "file", "--version", "v3",
                      "--source", f"/data/{src_file}", "--dest", "/output", "--domain-subdir=true",
                      "--crt-name=fullchain", "--crt-ext=.pem", "--key-name=privkey", "--key-ext=.pem"],
                     check=True)
-        log.info("Setting secure permissions (600) on extracted keys...")
-        ctx.sys.sudo(["find", abs_dest, "-type", "f", "-name", "*.pem", "-exec", "chmod", "600", "{}", "+"])
+        log.info("Setting secure permissions on extracted certs and keys...")
+        ctx.sys.sudo(["find", abs_dest, "-type", "f", "-name", "*key*.pem", "-exec", "chmod", "600", "{}", "+"])
+        ctx.sys.sudo(["find", abs_dest, "-type", "f", "-name", "*chain*.pem", "-exec", "chmod", "644", "{}", "+"])
         ctx.sys.sudo(["chown", "-R", f"{os.getuid()}:{os.getgid()}", abs_dest])
         log.success("Certificates extracted.")
-        ctx.notify.success(TITLE, url, f"Certificates extracted for {node}", node)
+        ctx.notify.success(TITLE, url, f"Certificates extracted for {node}")
     except Exception as e:
-        log.error(str(e)); ctx.notify.error(TITLE, url, f"cert extraction failed: {e}", node)
-        raise SystemExit(1)
+        log.error(str(e)); ctx.notify.error(TITLE, url, f"cert extraction failed: {e}")
+        raise SystemExit(1) from e

actions/utils/lint-docker-compose.py

@@ -1,7 +1,7 @@
 # actions/utils/lint-docker-compose.py
-"""Port of utils/lint-docker-compose (dclint via docker) + stdlib anchor-IP validation:
+"""Run dclint through Docker and validate anchor IPs:
 every ipv4_address in compositions/*/docker-compose.yml must be unique and inside the
-node's /24 (SP3 §6.2). No new deps — ipaddress + re are stdlib."""
+node's /24."""
 import ipaddress, os, re, shutil
 from engine.descriptor import Meta, Flag
 
@@ -12,8 +12,6 @@ def run(ctx, args):
     log, s = ctx.log, ctx.sys
     repo = ctx.paths.repo_root
     rc_ip = _validate_anchor_ips(ctx)
-    if not shutil.which("docker"):
-        log.error("docker is not installed or not in PATH."); return rc_ip or 1
 
     files_string, targets = "", ["."]
     if args.hook:
@@ -25,12 +23,19 @@ def run(ctx, args):
             log.info("No staged docker-compose files to lint."); return rc_ip
         targets = [f"/workdir/{f}" for f in files_string.splitlines()]
 
+    if not shutil.which("docker"):
+        log.error("docker is not installed or not in PATH."); return rc_ip or 1
+
     log.info("Running dclint --fix (via docker)...")
+    env_args = []
+    for k in ("HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy", "NO_PROXY", "no_proxy"):
+        if k in os.environ:
+            env_args.extend(["-e", f"{k}={os.environ[k]}"])
     # "-e commons": never --fix inside the pinned submodule — a dirtied commons makes
     # sync/repository abort the nightly capture-up.
-    r = s.run(["docker", "run", "--rm", "-v", f"{repo}:/workdir", "-w", "/workdir", "node:lts-alpine",
+    r = s.run(["docker", "run", "--rm", "--network", "host", *env_args, "-v", f"{repo}:/workdir", "-w", "/workdir", "node:lts-alpine",
                "npx", "--yes", "dclint", "-e", "@tier1", "-e", "@tier2", "-e", "@tier3", "-e", ".env",
-               "-e", "commons", "-r", "--fix", *targets], check=False)
+               "-e", "commons", "--disable-rule", "no-build-and-image", "-r", "--fix", *targets], check=False)
     if r.returncode != 0:
         log.error(f"dclint issues (exit {r.returncode})."); return r.returncode or rc_ip
     if args.hook and files_string:
@@ -53,7 +58,8 @@ def _validate_anchor_ips(ctx) -> int:
         if not os.path.isfile(f):
             continue
         try:
-            text = open(f).read()
+            with open(f, encoding="utf-8") as fh:
+                text = fh.read()
         except OSError as e:
             ctx.log.error(f"Cannot read {f}: {e}"); rc = 1
             continue

assets/logo.svg

@@ -1,35 +0,0 @@
-<svg width="1410" height="293" viewBox="0 0 1410 293" fill="none" xmlns="http://www.w3.org/2000/svg">
-<mask id="path-1-outside-1_1_35" maskUnits="userSpaceOnUse" x="1" y="77.5" width="1080" height="129" fill="black">
-<rect fill="white" x="1" y="77.5" width="1080" height="129"/>
-<path d="M62.64 150.02C62.64 147.46 62.2667 145.06 61.52 142.82C60.8267 140.527 59.76 138.527 58.32 136.82C56.88 135.113 55.04 133.78 52.8 132.82C50.6133 131.807 48.0533 131.3 45.12 131.3C39.4133 131.3 34.9067 132.927 31.6 136.18C28.3467 139.433 26.2667 144.047 25.36 150.02H62.64ZM24.96 161.94C25.28 166.153 26.0267 169.807 27.2 172.9C28.3733 175.94 29.92 178.473 31.84 180.5C33.76 182.473 36.0267 183.967 38.64 184.98C41.3067 185.94 44.24 186.42 47.44 186.42C50.64 186.42 53.3867 186.047 55.68 185.3C58.0267 184.553 60.0533 183.727 61.76 182.82C63.52 181.913 65.04 181.087 66.32 180.34C67.6533 179.593 68.9333 179.22 70.16 179.22C71.8133 179.22 73.04 179.833 73.84 181.06L79.52 188.26C77.3333 190.82 74.88 192.98 72.16 194.74C69.44 196.447 66.5867 197.833 63.6 198.9C60.6667 199.913 57.6533 200.633 54.56 201.06C51.52 201.487 48.56 201.7 45.68 201.7C39.9733 201.7 34.6667 200.767 29.76 198.9C24.8533 196.98 20.5867 194.18 16.96 190.5C13.3333 186.767 10.48 182.18 8.4 176.74C6.32 171.247 5.28 164.9 5.28 157.7C5.28 152.1 6.18667 146.847 8 141.94C9.81333 136.98 12.4 132.687 15.76 129.06C19.1733 125.38 23.3067 122.473 28.16 120.34C33.0667 118.207 38.5867 117.14 44.72 117.14C49.8933 117.14 54.6667 117.967 59.04 119.62C63.4133 121.273 67.1733 123.7 70.32 126.9C73.4667 130.047 75.92 133.94 77.68 138.58C79.4933 143.167 80.4 148.42 80.4 154.34C80.4 157.327 80.08 159.353 79.44 160.42C78.8 161.433 77.5733 161.94 75.76 161.94H24.96ZM111.869 132.66C114.429 127.753 117.469 123.913 120.989 121.14C124.509 118.313 128.669 116.9 133.469 116.9C137.255 116.9 140.295 117.727 142.589 119.38L141.309 134.18C141.042 135.14 140.642 135.833 140.109 136.26C139.629 136.633 138.962 136.82 138.109 136.82C137.309 136.82 136.109 136.687 134.509 136.42C132.962 136.153 131.442 136.02 129.949 136.02C127.762 136.02 125.815 136.34 124.109 136.98C122.402 137.62 120.855 138.553 119.469 139.78C118.135 140.953 116.935 142.393 115.869 144.1C114.855 145.807 113.895 147.753 112.989 149.94V200.5H93.2288V118.42H104.829C106.855 118.42 108.269 118.793 109.069 119.54C109.869 120.233 110.402 121.513 110.669 123.38L111.869 132.66ZM150.916 200.5V118.42H162.996C165.556 118.42 167.236 119.62 168.036 122.02L169.316 128.1C170.756 126.5 172.25 125.033 173.796 123.7C175.396 122.367 177.076 121.22 178.836 120.26C180.65 119.3 182.57 118.553 184.596 118.02C186.676 117.433 188.943 117.14 191.396 117.14C196.57 117.14 200.81 118.553 204.116 121.38C207.476 124.153 209.983 127.86 211.636 132.5C212.916 129.78 214.516 127.46 216.436 125.54C218.356 123.567 220.463 121.967 222.756 120.74C225.05 119.513 227.476 118.607 230.036 118.02C232.65 117.433 235.263 117.14 237.876 117.14C242.41 117.14 246.436 117.833 249.956 119.22C253.476 120.607 256.436 122.633 258.836 125.3C261.236 127.967 263.05 131.22 264.276 135.06C265.556 138.9 266.196 143.3 266.196 148.26V200.5H246.436V148.26C246.436 143.033 245.29 139.113 242.996 136.5C240.703 133.833 237.343 132.5 232.916 132.5C230.89 132.5 228.996 132.847 227.236 133.54C225.53 134.233 224.01 135.247 222.676 136.58C221.396 137.86 220.383 139.487 219.636 141.46C218.89 143.433 218.516 145.7 218.516 148.26V200.5H198.676V148.26C198.676 142.767 197.556 138.767 195.316 136.26C193.13 133.753 189.903 132.5 185.636 132.5C182.756 132.5 180.063 133.22 177.556 134.66C175.103 136.047 172.81 137.967 170.676 140.42V200.5H150.916ZM301.459 128.5C303.112 126.847 304.845 125.327 306.659 123.94C308.525 122.5 310.472 121.3 312.499 120.34C314.579 119.327 316.792 118.553 319.139 118.02C321.485 117.433 324.045 117.14 326.819 117.14C331.299 117.14 335.272 117.913 338.739 119.46C342.205 120.953 345.085 123.087 347.379 125.86C349.725 128.58 351.485 131.86 352.659 135.7C353.885 139.487 354.499 143.673 354.499 148.26V200.5H334.739V148.26C334.739 143.247 333.592 139.38 331.299 136.66C329.005 133.887 325.512 132.5 320.819 132.5C317.405 132.5 314.205 133.273 311.219 134.82C308.232 136.367 305.405 138.473 302.739 141.14V200.5H282.979V118.42H295.059C297.619 118.42 299.299 119.62 300.099 122.02L301.459 128.5ZM443.416 118.42L410.856 200.5H392.936L360.376 118.42H376.696C378.19 118.42 379.416 118.767 380.376 119.46C381.39 120.153 382.056 121.033 382.376 122.1L398.216 165.94C399.07 168.5 399.816 171.007 400.456 173.46C401.096 175.913 401.656 178.367 402.136 180.82C402.67 178.367 403.256 175.913 403.896 173.46C404.536 171.007 405.31 168.5 406.216 165.94L422.376 122.1C422.75 121.033 423.416 120.153 424.376 119.46C425.336 118.767 426.483 118.42 427.816 118.42H443.416ZM471.948 81.62V200.5H452.188V81.62H471.948ZM541.155 139.38C538.915 136.66 536.462 134.74 533.795 133.62C531.182 132.5 528.355 131.94 525.315 131.94C522.328 131.94 519.635 132.5 517.235 133.62C514.835 134.74 512.782 136.447 511.075 138.74C509.368 140.98 508.062 143.86 507.155 147.38C506.248 150.847 505.795 154.953 505.795 159.7C505.795 164.5 506.168 168.58 506.915 171.94C507.715 175.247 508.835 177.967 510.275 180.1C511.715 182.18 513.475 183.7 515.555 184.66C517.635 185.567 519.955 186.02 522.515 186.02C526.622 186.02 530.115 185.167 Line truncated
-</mask>
-<path d="M62.64 150.02C62.64 147.46 62.2667 145.06 61.52 142.82C60.8267 140.527 59.76 138.527 58.32 136.82C56.88 135.113 55.04 133.78 52.8 132.82C50.6133 131.807 48.0533 131.3 45.12 131.3C39.4133 131.3 34.9067 132.927 31.6 136.18C28.3467 139.433 26.2667 144.047 25.36 150.02H62.64ZM24.96 161.94C25.28 166.153 26.0267 169.807 27.2 172.9C28.3733 175.94 29.92 178.473 31.84 180.5C33.76 182.473 36.0267 183.967 38.64 184.98C41.3067 185.94 44.24 186.42 47.44 186.42C50.64 186.42 53.3867 186.047 55.68 185.3C58.0267 184.553 60.0533 183.727 61.76 182.82C63.52 181.913 65.04 181.087 66.32 180.34C67.6533 179.593 68.9333 179.22 70.16 179.22C71.8133 179.22 73.04 179.833 73.84 181.06L79.52 188.26C77.3333 190.82 74.88 192.98 72.16 194.74C69.44 196.447 66.5867 197.833 63.6 198.9C60.6667 199.913 57.6533 200.633 54.56 201.06C51.52 201.487 48.56 201.7 45.68 201.7C39.9733 201.7 34.6667 200.767 29.76 198.9C24.8533 196.98 20.5867 194.18 16.96 190.5C13.3333 186.767 10.48 182.18 8.4 176.74C6.32 171.247 5.28 164.9 5.28 157.7C5.28 152.1 6.18667 146.847 8 141.94C9.81333 136.98 12.4 132.687 15.76 129.06C19.1733 125.38 23.3067 122.473 28.16 120.34C33.0667 118.207 38.5867 117.14 44.72 117.14C49.8933 117.14 54.6667 117.967 59.04 119.62C63.4133 121.273 67.1733 123.7 70.32 126.9C73.4667 130.047 75.92 133.94 77.68 138.58C79.4933 143.167 80.4 148.42 80.4 154.34C80.4 157.327 80.08 159.353 79.44 160.42C78.8 161.433 77.5733 161.94 75.76 161.94H24.96ZM111.869 132.66C114.429 127.753 117.469 123.913 120.989 121.14C124.509 118.313 128.669 116.9 133.469 116.9C137.255 116.9 140.295 117.727 142.589 119.38L141.309 134.18C141.042 135.14 140.642 135.833 140.109 136.26C139.629 136.633 138.962 136.82 138.109 136.82C137.309 136.82 136.109 136.687 134.509 136.42C132.962 136.153 131.442 136.02 129.949 136.02C127.762 136.02 125.815 136.34 124.109 136.98C122.402 137.62 120.855 138.553 119.469 139.78C118.135 140.953 116.935 142.393 115.869 144.1C114.855 145.807 113.895 147.753 112.989 149.94V200.5H93.2288V118.42H104.829C106.855 118.42 108.269 118.793 109.069 119.54C109.869 120.233 110.402 121.513 110.669 123.38L111.869 132.66ZM150.916 200.5V118.42H162.996C165.556 118.42 167.236 119.62 168.036 122.02L169.316 128.1C170.756 126.5 172.25 125.033 173.796 123.7C175.396 122.367 177.076 121.22 178.836 120.26C180.65 119.3 182.57 118.553 184.596 118.02C186.676 117.433 188.943 117.14 191.396 117.14C196.57 117.14 200.81 118.553 204.116 121.38C207.476 124.153 209.983 127.86 211.636 132.5C212.916 129.78 214.516 127.46 216.436 125.54C218.356 123.567 220.463 121.967 222.756 120.74C225.05 119.513 227.476 118.607 230.036 118.02C232.65 117.433 235.263 117.14 237.876 117.14C242.41 117.14 246.436 117.833 249.956 119.22C253.476 120.607 256.436 122.633 258.836 125.3C261.236 127.967 263.05 131.22 264.276 135.06C265.556 138.9 266.196 143.3 266.196 148.26V200.5H246.436V148.26C246.436 143.033 245.29 139.113 242.996 136.5C240.703 133.833 237.343 132.5 232.916 132.5C230.89 132.5 228.996 132.847 227.236 133.54C225.53 134.233 224.01 135.247 222.676 136.58C221.396 137.86 220.383 139.487 219.636 141.46C218.89 143.433 218.516 145.7 218.516 148.26V200.5H198.676V148.26C198.676 142.767 197.556 138.767 195.316 136.26C193.13 133.753 189.903 132.5 185.636 132.5C182.756 132.5 180.063 133.22 177.556 134.66C175.103 136.047 172.81 137.967 170.676 140.42V200.5H150.916ZM301.459 128.5C303.112 126.847 304.845 125.327 306.659 123.94C308.525 122.5 310.472 121.3 312.499 120.34C314.579 119.327 316.792 118.553 319.139 118.02C321.485 117.433 324.045 117.14 326.819 117.14C331.299 117.14 335.272 117.913 338.739 119.46C342.205 120.953 345.085 123.087 347.379 125.86C349.725 128.58 351.485 131.86 352.659 135.7C353.885 139.487 354.499 143.673 354.499 148.26V200.5H334.739V148.26C334.739 143.247 333.592 139.38 331.299 136.66C329.005 133.887 325.512 132.5 320.819 132.5C317.405 132.5 314.205 133.273 311.219 134.82C308.232 136.367 305.405 138.473 302.739 141.14V200.5H282.979V118.42H295.059C297.619 118.42 299.299 119.62 300.099 122.02L301.459 128.5ZM443.416 118.42L410.856 200.5H392.936L360.376 118.42H376.696C378.19 118.42 379.416 118.767 380.376 119.46C381.39 120.153 382.056 121.033 382.376 122.1L398.216 165.94C399.07 168.5 399.816 171.007 400.456 173.46C401.096 175.913 401.656 178.367 402.136 180.82C402.67 178.367 403.256 175.913 403.896 173.46C404.536 171.007 405.31 168.5 406.216 165.94L422.376 122.1C422.75 121.033 423.416 120.153 424.376 119.46C425.336 118.767 426.483 118.42 427.816 118.42H443.416ZM471.948 81.62V200.5H452.188V81.62H471.948ZM541.155 139.38C538.915 136.66 536.462 134.74 533.795 133.62C531.182 132.5 528.355 131.94 525.315 131.94C522.328 131.94 519.635 132.5 517.235 133.62C514.835 134.74 512.782 136.447 511.075 138.74C509.368 140.98 508.062 143.86 507.155 147.38C506.248 150.847 505.795 154.953 505.795 159.7C505.795 164.5 506.168 168.58 506.915 171.94C507.715 175.247 508.835 177.967 510.275 180.1C511.715 182.18 513.475 183.7 515.555 184.66C517.635 185.567 519.955 186.02 522.515 186.02C526.622 186.02 530.115 185.167 Line truncated
-<path d="M62.64 150.02V154.02H66.64V150.02H62.64ZM61.52 142.82L57.6912 143.978L57.7075 144.031L57.7253 144.085L61.52 142.82ZM52.8 132.82L51.1182 136.449L51.1709 136.474L51.2243 136.497L52.8 132.82ZM31.6 136.18L28.7947 133.329L28.7831 133.34L28.7716 133.352L31.6 136.18ZM25.36 150.02L21.4053 149.42L20.707 154.02H25.36V150.02ZM24.96 161.94V157.94H20.6447L20.9715 162.243L24.96 161.94ZM27.2 172.9L23.46 174.319L23.4641 174.329L23.4683 174.34L27.2 172.9ZM31.84 180.5L28.9362 183.251L28.9545 183.27L28.9731 183.289L31.84 180.5ZM38.64 184.98L37.1939 188.709L37.2393 188.727L37.2851 188.744L38.64 184.98ZM55.68 185.3L54.4672 181.488L54.4544 181.492L54.4417 181.497L55.68 185.3ZM61.76 182.82L59.9282 179.264L59.9057 179.276L59.8834 179.288L61.76 182.82ZM66.32 180.34L64.3656 176.85L64.3349 176.867L64.3045 176.885L66.32 180.34ZM73.84 181.06L70.4896 183.245L70.588 183.396L70.6996 183.537L73.84 181.06ZM79.52 188.26L82.5615 190.858L84.6944 188.361L82.6604 185.783L79.52 188.26ZM72.16 194.74L74.286 198.128L74.3096 198.113L74.333 198.098L72.16 194.74ZM63.6 198.9L64.9061 202.681L64.9258 202.674L64.9454 202.667L63.6 198.9ZM54.56 201.06L54.0135 197.098L54.004 197.099L54.56 201.06ZM29.76 198.9L28.3024 202.625L28.32 202.632L28.3377 202.639L29.76 198.9ZM16.96 190.5L14.0909 193.287L14.1009 193.297L14.111 193.308L16.96 190.5ZM8.4 176.74L4.65916 178.156L4.66379 178.169L8.4 176.74ZM8 141.94L11.752 143.327L11.7568 143.313L8 141.94ZM15.76 129.06L12.8273 126.34L12.8258 126.342L15.76 129.06ZM28.16 120.34L26.5651 116.672L26.5577 116.675L26.5504 116.678L28.16 120.34ZM59.04 119.62L60.4545 115.878L59.04 119.62ZM70.32 126.9L67.4679 129.705L67.4797 129.717L67.4916 129.728L70.32 126.9ZM77.68 138.58L73.94 139.999L73.9499 140.025L73.9602 140.051L77.68 138.58ZM79.44 160.42L82.822 162.556L82.8464 162.517L82.87 162.478L79.44 160.42ZM62.64 150.02H66.64C66.64 147.066 66.2083 144.236 65.3147 141.555L61.52 142.82L57.7253 144.085C58.325 145.884 58.64 147.854 58.64 150.02H62.64ZM61.52 142.82L65.3488 141.662C64.5095 138.886 63.1936 136.393 61.3772 134.241L58.32 136.82L55.2628 139.399C56.3264 140.66 57.1439 142.167 57.6912 143.978L61.52 142.82ZM58.32 136.82L61.3772 134.241C59.4997 132.015 57.1349 130.326 54.3757 129.143L52.8 132.82L51.2243 136.497C52.9451 137.234 54.2603 138.211 55.2628 139.399L58.32 136.82ZM52.8 132.82L54.4818 129.191C51.6664 127.886 48.5106 127.3 45.12 127.3V131.3V135.3C47.5961 135.3 49.5602 135.727 51.1182 136.449L52.8 132.82ZM45.12 131.3V127.3C38.6197 127.3 33.014 129.177 28.7947 133.329L31.6 136.18L34.4053 139.031C36.7993 136.676 40.207 135.3 45.12 135.3V131.3ZM31.6 136.18L28.7716 133.352C24.7503 137.373 22.4001 142.866 21.4053 149.42L25.36 150.02L29.3147 150.62C30.1332 145.228 31.9431 141.494 34.4284 139.008L31.6 136.18ZM25.36 150.02V154.02H62.64V150.02V146.02H25.36V150.02ZM24.96 161.94L20.9715 162.243C21.3129 166.738 22.1193 170.784 23.46 174.319L27.2 172.9L30.94 171.481C29.934 168.829 29.2471 165.569 28.9485 161.637L24.96 161.94ZM27.2 172.9L23.4683 174.34C24.7985 177.787 26.6026 180.788 28.9362 183.251L31.84 180.5L34.7438 177.749C33.2374 176.159 31.9482 174.093 30.9317 171.46L27.2 172.9ZM31.84 180.5L28.9731 183.289C31.3011 185.682 34.0549 187.492 37.1939 188.709L38.64 184.98L40.0861 181.251C37.9984 180.441 36.2189 179.265 34.7069 177.711L31.84 180.5ZM38.64 184.98L37.2851 188.744C40.4386 189.879 43.8371 190.42 47.44 190.42V186.42V182.42C44.6429 182.42 42.1747 182.001 39.9949 181.216L38.64 184.98ZM47.44 186.42V190.42C50.9318 190.42 54.1187 190.015 56.9183 189.103L55.68 185.3L54.4417 181.497C52.6546 182.078 50.3482 182.42 47.44 182.42V186.42ZM55.68 185.3L56.8928 189.112C59.4097 188.311 61.6691 187.398 63.6366 186.352L61.76 182.82L59.8834 179.288C58.4376 180.056 56.6436 180.796 54.4672 181.488L55.68 185.3ZM61.76 182.82L63.5918 186.376C65.3948 185.447 66.979 184.586 68.3355 183.795L66.32 180.34L64.3045 176.885C63.101 177.587 61.6452 178.38 59.9282 179.264L61.76 182.82ZM66.32 180.34L68.2744 183.83C69.1554 183.337 69.7664 183.22 70.16 183.22V179.22V175.22C68.1002 175.22 66.1512 175.85 64.3656 176.85L66.32 180.34ZM70.16 179.22V183.22C70.4918 183.22 70.576 183.28 70.5312 183.258C70.5118 183.248 70.4889 183.233 70.4713 183.217C70.4548 183.202 70.4631 183.204 70.4896 183.245L73.84 181.06L77.1904 178.875C75.5467 176.354 72.9601 175.22 70.16 175.22V179.22ZM73.84 181.06L70.6996 183.537L76.3796 190.737L79.52 188.26L82.6604 185.783L76.9804 178.583L73.84 181.06ZM79.52 188.26L76.4785 185.662C74.5331 187.94 72.3697 189.84 69.987 191.382L72.16 194.74L74.333 198.098C77.3903 196.12 80.1335 193.7 82.5615 190.858L79.52 188.26ZM72.16 194.74L70.034 191.352C67.5567 192.906 64.9647 194.165 62.2547 195.133L63.6 198.9L64.9454 202.667C68.2086 201.502 71.3233 199.987 74.286 198.128L72.16 194.74ZM63.6 198.9L62.2939 195.119C59.6088 196.047 56.8499 196.706 54.0135 197.098L54.56 201.06L55.1066 205.022C58.4568 204.56 61.7246 203.78 64.9061 202.681L63.6 198.9ZM54.56 201.06L54.004 197.099C51.1363 197.501 48.3623 197.7 45.68 197.7V201.7V205.7C48.7577 205.7 51.9037 205.472 55.116Line truncated
-<rect width="443" height="64" transform="translate(0 226.5)" fill="white"/>
-<path d="M25.92 257.58C26.944 255.361 28.2027 253.633 29.696 252.396C31.1893 251.137 33.0133 250.508 35.168 250.508C35.8507 250.508 36.5013 250.583 37.12 250.732C37.76 250.881 38.3253 251.116 38.816 251.436L38.4 255.692C38.272 256.225 37.952 256.492 37.44 256.492C37.1413 256.492 36.704 256.428 36.128 256.3C35.552 256.172 34.9013 256.108 34.176 256.108C33.152 256.108 32.2347 256.257 31.424 256.556C30.6347 256.855 29.92 257.303 29.28 257.9C28.6613 258.476 28.096 259.201 27.584 260.076C27.0933 260.929 26.6453 261.911 26.24 263.02V283.5H20.512V251.084H23.776C24.3947 251.084 24.8213 251.201 25.056 251.436C25.2907 251.671 25.4507 252.076 25.536 252.652L25.92 257.58ZM56.676 250.572C59.044 250.572 61.1773 250.967 63.076 251.756C64.9747 252.545 66.596 253.665 67.94 255.116C69.284 256.567 70.308 258.327 71.012 260.396C71.7373 262.444 72.1 264.737 72.1 267.276C72.1 269.836 71.7373 272.14 71.012 274.188C70.308 276.236 69.284 277.985 67.94 279.436C66.596 280.887 64.9747 282.007 63.076 282.796C61.1773 283.564 59.044 283.948 56.676 283.948C54.2867 283.948 52.132 283.564 50.212 282.796C48.3133 282.007 46.692 280.887 45.348 279.436C44.004 277.985 42.9693 276.236 42.244 274.188C41.54 272.14 41.188 269.836 41.188 267.276C41.188 264.737 41.54 262.444 42.244 260.396C42.9693 258.327 44.004 256.567 45.348 255.116C46.692 253.665 48.3133 252.545 50.212 251.756C52.132 250.967 54.2867 250.572 56.676 250.572ZM56.676 279.5C59.876 279.5 62.2653 278.433 63.844 276.3C65.4227 274.145 66.212 271.148 66.212 267.308C66.212 263.447 65.4227 260.439 63.844 258.284C62.2653 256.129 59.876 255.052 56.676 255.052C55.0547 255.052 53.636 255.329 52.42 255.884C51.2253 256.439 50.2227 257.239 49.412 258.284C48.6227 259.329 48.0253 260.62 47.62 262.156C47.236 263.671 47.044 265.388 47.044 267.308C47.044 271.148 47.8333 274.145 49.412 276.3C51.012 278.433 53.4333 279.5 56.676 279.5ZM93.4885 250.572C95.8565 250.572 97.9898 250.967 99.8885 251.756C101.787 252.545 103.409 253.665 104.753 255.116C106.097 256.567 107.121 258.327 107.825 260.396C108.55 262.444 108.913 264.737 108.913 267.276C108.913 269.836 108.55 272.14 107.825 274.188C107.121 276.236 106.097 277.985 104.753 279.436C103.409 280.887 101.787 282.007 99.8885 282.796C97.9898 283.564 95.8565 283.948 93.4885 283.948C91.0992 283.948 88.9445 283.564 87.0245 282.796C85.1258 282.007 83.5045 280.887 82.1605 279.436C80.8165 277.985 79.7818 276.236 79.0565 274.188C78.3525 272.14 78.0005 269.836 78.0005 267.276C78.0005 264.737 78.3525 262.444 79.0565 260.396C79.7818 258.327 80.8165 256.567 82.1605 255.116C83.5045 253.665 85.1258 252.545 87.0245 251.756C88.9445 250.967 91.0992 250.572 93.4885 250.572ZM93.4885 279.5C96.6885 279.5 99.0778 278.433 100.657 276.3C102.235 274.145 103.025 271.148 103.025 267.308C103.025 263.447 102.235 260.439 100.657 258.284C99.0778 256.129 96.6885 255.052 93.4885 255.052C91.8672 255.052 90.4485 255.329 89.2325 255.884C88.0378 256.439 87.0352 257.239 86.2245 258.284C85.4352 259.329 84.8378 260.62 84.4325 262.156C84.0485 263.671 83.8565 265.388 83.8565 267.308C83.8565 271.148 84.6458 274.145 86.2245 276.3C87.8245 278.433 90.2458 279.5 93.4885 279.5ZM126.027 284.012C123.467 284.012 121.493 283.297 120.107 281.868C118.741 280.439 118.059 278.38 118.059 275.692V255.852H114.155C113.813 255.852 113.525 255.756 113.291 255.564C113.056 255.351 112.939 255.031 112.939 254.604V252.332L118.251 251.66L119.562 241.644C119.627 241.324 119.765 241.068 119.979 240.876C120.213 240.663 120.512 240.556 120.875 240.556H123.755V251.724H133.131V255.852H123.755V275.308C123.755 276.673 124.085 277.687 124.747 278.348C125.408 279.009 126.261 279.34 127.307 279.34C127.904 279.34 128.416 279.265 128.843 279.116C129.291 278.945 129.675 278.764 129.995 278.572C130.315 278.38 130.581 278.209 130.795 278.06C131.029 277.889 131.232 277.804 131.403 277.804C131.701 277.804 131.968 277.985 132.203 278.348L133.867 281.068C132.885 281.985 131.701 282.711 130.315 283.244C128.928 283.756 127.499 284.012 126.027 284.012ZM178.947 256.844C178.776 257.079 178.606 257.26 178.435 257.388C178.264 257.516 178.03 257.58 177.731 257.58C177.411 257.58 177.059 257.452 176.675 257.196C176.291 256.919 175.811 256.62 175.235 256.3C174.68 255.98 173.987 255.692 173.155 255.436C172.344 255.159 171.342 255.02 170.147 255.02C168.547 255.02 167.139 255.308 165.923 255.884C164.707 256.439 163.683 257.249 162.851 258.316C162.04 259.383 161.422 260.673 160.995 262.188C160.59 263.703 160.387 265.399 160.387 267.276C160.387 269.239 160.611 270.988 161.059 272.524C161.507 274.039 162.136 275.319 162.947 276.364C163.779 277.388 164.771 278.177 165.923 278.732C167.096 279.265 168.408 279.532 169.859 279.532C171.246 279.532 172.387 279.372 173.283 279.052C174.179 278.711 174.915 278.337 175.491 277.932C176.088 277.527 176.579 277.164 176.963 276.844C177.368 276.503 177.763 276.332 178.147 276.332C178.616 276.332 178.979 276.513 179.235 276.876L180.835 278.956C180.131 279.831 179.331 280.577 178.435 281.196C177.539 281.815 176.568 282.337 175.523 Line truncated
-<path d="M1200 218.5H1260L1242 206.5H1182L1200 218.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1200 218.5V278.5L1182 266.5V206.5L1200 218.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1260 218.5H1200V278.5H1260V218.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1200 158.5H1260L1242 146.5H1182L1200 158.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1200 158.5V218.5L1182 206.5V146.5L1200 158.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1260 158.5H1200V218.5H1260V158.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1347 14.5H1407L1389 2.5H1329L1347 14.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1347 14.5V74.5L1329 62.5V2.5L1347 14.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1407 14.5H1347V74.5H1407V14.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1242 86.5H1302L1284 74.5H1224L1242 86.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1242 86.5V146.5L1224 134.5V74.5L1242 86.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1302 86.5H1242V146.5H1302V86.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1218 230.5H1278L1260 218.5H1200L1218 230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1218 230.5V290.5L1200 278.5V218.5L1218 230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1278 230.5H1218V290.5H1278V230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1158 230.5H1218L1200 218.5H1140L1158 230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1158 230.5V290.5L1140 278.5V218.5L1158 230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1218 230.5H1158V290.5H1218V230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1158 230.5H1218L1200 218.5H1140L1158 230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1158 230.5V290.5L1140 278.5V218.5L1158 230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1218 230.5H1158V290.5H1218V230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1098 230.5V290.5L1080 278.5V218.5L1098 230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1038 230.5H1098L1080 218.5H1020L1038 230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1038 230.5V290.5L1020 278.5V218.5L1038 230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-<path d="M1098 230.5H1038V290.5H1098V230.5Z" fill="black" stroke="white" stroke-width="5" stroke-linejoin="round"/>
-</svg>

ci/cloud-init/user-data.template.yaml

@@ -0,0 +1,33 @@
+#cloud-config
+# Supply this file to cloud-init when booting the golden image.
+# Replace the hostname, repository and public key. Keep filled files outside Git.
+# SSH repositories additionally require approved deferred id_ed25519 (0600)
+# and known_hosts (0644) writes under /home/adam/.ssh, owned by adam:adam.
+# Supply trusted Git host keys; do not infer trust from ssh-keyscan alone.
+preserve_hostname: false
+hostname: node1.example.com
+users:
+  - name: adam
+    sudo: ALL=(ALL) NOPASSWD:ALL
+    groups: [sudo, docker]
+    shell: /bin/bash
+    ssh_authorized_keys:
+      - "ssh-ed25519 REPLACE_WITH_OPERATOR_PUBLIC_KEY operator@workstation"
+write_files:
+  - path: /etc/apex/bootstrap.env
+    owner: root:root
+    permissions: '0600'
+    content: |
+      APEX_REPO_NAME="node1.example.com"
+      APEX_REPO_URL="https://git.example.com/infrastructure/node1.example.com.git"
+      APEX_TIER1_MODE="directory"
+      APEX_TIER1_DEVICE=""
+      APEX_TIER2_MODE="directory"
+      APEX_TIER2_DEVICE=""
+      APEX_TIER3_MODE="directory"
+      APEX_TIER3_DEVICE=""
+# For a verified empty disk, explicitly select MODE=create and DEVICE=/dev/disk/by-id/...
+# For an existing single-device pool, select MODE=import, DEVICE and POOL_GUID.
+# Occupied devices are never erased; missing devices never fall back to directories.
+runcmd:
+  - /usr/local/bin/apex-bootstrap

ci/packer/apex-debian-13.pkr.hcl

@@ -0,0 +1,95 @@
+packer {
+  required_plugins {
+    qemu = {
+      version = ">= 1.1.0"
+      source  = "github.com/hashicorp/qemu"
+    }
+  }
+}
+
+variable "debian_cloud_image_url" {
+  type    = string
+  default = "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-amd64-daily.qcow2"
+}
+
+variable "debian_cloud_image_checksum" {
+  type    = string
+  default = "none"
+}
+
+variable "vm_name" {
+  type    = string
+  default = "com-ermnvldmr-apex-debian-13-amd64.qcow2"
+}
+
+variable "accelerator" {
+  type    = string
+  default = "kvm"
+}
+
+variable "provision_revision" {
+  type    = string
+  default = "working-tree"
+}
+
+source "qemu" "apex_debian" {
+  iso_url          = var.debian_cloud_image_url
+  iso_checksum     = var.debian_cloud_image_checksum
+  disk_image       = true
+  disk_size        = "10G"
+  format           = "qcow2"
+  output_directory = "output-qemu"
+  shutdown_command = "sudo -n shutdown -P now"
+  accelerator      = var.accelerator
+  cd_files         = ["ci/packer/cidata/user-data", "ci/packer/cidata/meta-data"]
+  cd_label         = "cidata"
+  ssh_username     = "debian"
+  ssh_password     = "packer"
+  ssh_timeout      = "10m"
+  vm_name          = var.vm_name
+  net_device       = "virtio-net"
+  disk_interface   = "virtio"
+  headless         = true
+
+  qemuargs = [
+    ["-m", "2048M"],
+    ["-smp", "2"]
+  ]
+}
+
+build {
+  sources = ["source.qemu.apex_debian"]
+
+  provisioner "file" {
+    source      = "ci/scripts/init-node/provisioning"
+    destination = "/tmp/"
+  }
+
+  provisioner "shell" {
+    execute_command = "sudo -n sh -c '{{ .Vars }} {{ .Path }}'"
+    environment_vars = ["APEX_PROVISION_REVISION=${var.provision_revision}"]
+    inline = [
+      "bash /tmp/provisioning/seed.sh",
+      "cd /tmp && python3 -m provisioning.image install"
+    ]
+  }
+
+  provisioner "shell" {
+    execute_command = "sudo -n sh -c '{{ .Vars }} {{ .Path }}'"
+    scripts = [
+      "ci/packer/scripts/01-base-system.sh",
+      "ci/packer/scripts/02-storage-docker.sh",
+      "ci/packer/scripts/02-alloy-package.sh",
+      "ci/packer/scripts/03-bootstrap-helper.sh",
+      "ci/packer/scripts/04-cleanup.sh"
+    ]
+  }
+
+  post-processor "shell-local" {
+    inline = [
+      "echo 'Compressing and optimizing qcow2 image with 2M cluster size...'",
+      "qemu-img convert -f qcow2 -O qcow2 -o cluster_size=2M -c output-qemu/${var.vm_name} output-qemu/com-ermnvldmr-apex-debian-13-optimized.qcow2",
+      "mv output-qemu/com-ermnvldmr-apex-debian-13-optimized.qcow2 output-qemu/${var.vm_name}"
+    ]
+  }
+}

ci/packer/cidata/meta-data

@@ -0,0 +1,2 @@
+instance-id: apex-golden-image-build
+local-hostname: apex-builder

ci/packer/cidata/user-data

@@ -0,0 +1,14 @@
+#cloud-config
+ssh_pwauth: true
+disable_root: false
+chpasswd:
+  list: |
+    debian:packer
+    root:packer
+  expire: false
+users:
+  - default
+  - name: debian
+    sudo: ALL=(ALL) NOPASSWD:ALL
+    shell: /bin/bash
+    lock_passwd: false

ci/packer/scripts/01-base-system.sh

@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+set -euo pipefail
+export DEBIAN_FRONTEND=noninteractive
+apt-get -o DPkg::Lock::Timeout=60 update
+apt-get -o DPkg::Lock::Timeout=60 install -y --no-upgrade --no-install-recommends qemu-guest-agent cloud-init
+
+echo "=== [3/6] Configuring VirtIO Kernel Modules ==="
+cat << 'EOF_VIRTIO' > /etc/modules-load.d/virtio.conf
+virtio
+virtio_net
+virtio_blk
+virtio_pci
+virtio_scsi
+virtiofs
+EOF_VIRTIO
+
+echo "=== [4/6] Configuring GRUB Serial Console & Predictable Interfaces ==="
+sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,115200n8 net.ifnames=0 biosdevname=0"/' /etc/default/grub
+sed -i 's/^#GRUB_TERMINAL=.*/GRUB_TERMINAL="console serial"/' /etc/default/grub
+if ! grep -q "GRUB_SERIAL_COMMAND" /etc/default/grub; then
+  echo 'GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"' >> /etc/default/grub
+fi
+update-grub
+
+mkdir -p /etc/systemd/system/serial-getty@ttyS0.service.d
+systemctl enable serial-getty@ttyS0.service || true
+systemctl enable qemu-guest-agent 2>/dev/null || true
+
+echo "=== [5/6] Configuring Cloud-Init Datasources & Service Enablement ==="
+mkdir -p /etc/cloud /etc/cloud/cloud.cfg.d
+cat << 'EOF_DS' > /etc/cloud/ds-identify.cfg
+policy: enabled
+EOF_DS
+
+cat << 'EOF_CLOUD_CFG' > /etc/cloud/cloud.cfg.d/99-cloud-init.cfg
+datasource_list: [ NoCloud, ConfigDrive, OpenStack, Ec2, None ]
+disable_vm_template: false
+EOF_CLOUD_CFG
+
+# Cloud-init services are static and enabled by generator via ds-identify
+systemctl enable cloud-init-local.service cloud-init.service cloud-config.service cloud-final.service 2>/dev/null || true
+
+echo "=== [6/6] Configuring Predictable Network DHCP ==="
+mkdir -p /etc/systemd/network
+cat << 'EOF_NET' > /etc/systemd/network/10-eth.network
+[Match]
+Name=eth* en*
+
+[Network]
+DHCP=yes
+EOF_NET
+systemctl enable systemd-networkd systemd-resolved || true
+
+mkdir -p /etc/network/interfaces.d
+cat << 'EOF_IF' > /etc/network/interfaces.d/eth0
+auto eth0
+iface eth0 inet dhcp
+EOF_IF
+
+echo "=== Base System Configuration Complete ==="

ci/packer/scripts/02-alloy-package.sh

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Service configuration and credentials are deployed later by apex configure/alloy.
+alloy_version=1.21.3
+alloy_sha256=dbbd705de8ab37dec0cbd6361a3fc7e754d6b2854b57ce672a83e6366199ba73
+[[ "$(dpkg --print-architecture)" == amd64 ]]
+alloy_deb=$(mktemp --suffix=.deb)
+trap 'rm -f "$alloy_deb"' EXIT
+curl --fail --location --retry 3 --output "$alloy_deb" \
+  "https://github.com/grafana/alloy/releases/download/v${alloy_version}/alloy-${alloy_version}-1.amd64.deb"
+printf '%s  %s\n' "$alloy_sha256" "$alloy_deb" | sha256sum --check --status
+systemctl mask alloy.service
+DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends "$alloy_deb"
+systemctl disable --now alloy.service
+systemctl unmask alloy.service
+! systemctl is-active --quiet alloy.service
+! systemctl is-enabled --quiet alloy.service

ci/packer/scripts/02-storage-docker.sh

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+set -euo pipefail
+/usr/local/bin/apex-provision-image baseline

ci/packer/scripts/03-bootstrap-helper.sh

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+set -euo pipefail
+/usr/local/bin/apex-provision-image install-runner

ci/packer/scripts/04-cleanup.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+set -euo pipefail
+/usr/local/bin/apex-provision-image sanitize
+apt-get clean
+cloud-init clean --logs --seed
+sync

ci/scripts/init-node/provisioning/__init__.py

@@ -0,0 +1,3 @@
+"""Shared initialization primitives; importing this package performs no host changes."""
+
+ENROLLMENT_INTERFACE = 1

ci/scripts/init-node/provisioning/access.py

@@ -0,0 +1,558 @@
+"""Protected SSH handover with a separately locked, persistent rollback timer."""
+from __future__ import annotations
+
+import argparse
+import base64
+import fcntl
+import fnmatch
+import hashlib
+import ipaddress
+import json
+import os
+from pathlib import Path
+import pwd
+import re
+import shlex
+import stat
+import subprocess
+import sys
+import tempfile
+import time
+
+
+class AccessError(RuntimeError):
+    pass
+
+
+def include_arguments(value):
+    arguments, word, quote, started = [], [], None, False
+    for character in value:
+        if character == '\\':
+            raise AccessError('Escaped SSH Include paths require manual reconciliation')
+        if quote:
+            if character == quote:
+                quote = None
+            else:
+                word.append(character)
+        elif character in ('"', "'"):
+            quote, started = character, True
+        elif character in ' \t\r':
+            if started:
+                arguments.append(''.join(word))
+                word, started = [], False
+        elif character == '#' and not started:
+            break
+        else:
+            word.append(character)
+            started = True
+    if quote:
+        raise AccessError('Unclosed SSH Include quote requires manual reconciliation')
+    if started:
+        arguments.append(''.join(word))
+    if not arguments or len(arguments) > 128 or any(not item or len(item) > 4096 for item in arguments):
+        raise AccessError('Unsupported SSH Include paths require manual reconciliation')
+    return arguments
+
+
+class Host:
+    def __init__(self, root=Path('/'), runner=subprocess.run):
+        self.root = Path(root).resolve()
+        self.runner = runner
+
+    def path(self, path):
+        return self.root / path.lstrip('/')
+
+    def command(self, args, check=True):
+        result = self.runner(args, capture_output=True, text=True, timeout=60)
+        if check and result.returncode:
+            raise AccessError('Access command failed: ' + args[0])
+        return result
+
+    def boot_id(self):
+        return self.path('/proc/sys/kernel/random/boot_id').read_text().strip()
+
+    def fingerprint(self):
+        digest = hashlib.sha256()
+        paths = [self.path('/etc/ssh/sshd_config'), self.path('/etc/default/ssh')]
+        for directory in ('/etc/ssh/sshd_config.d', '/etc/systemd/system/ssh.service.d',
+                          '/run/systemd/system/ssh.service.d', '/usr/lib/systemd/system/ssh.service.d',
+                          '/lib/systemd/system/ssh.service.d'):
+            paths += sorted(self.path(directory).glob('*.conf'))
+        for path in paths:
+            digest.update(str(path.relative_to(self.root)).encode())
+            digest.update(path.read_bytes() if path.exists() else b'<absent>')
+        return digest.hexdigest()
+
+    def unconditional_configuration(self):
+        remaining, directory_entries, visited = 1024 * 1024, 0, set()
+
+        def expand(pattern):
+            nonlocal directory_entries
+            if any(token in pattern for token in ('[:', '[.', '[=', '[^')):
+                raise AccessError('Extended SSH Include glob syntax requires manual reconciliation')
+            absolute = pattern if pattern.startswith('/') else '/etc/ssh/' + pattern
+            mapped = self.path(absolute)
+            if len(mapped.parts) > 64:
+                raise AccessError('SSH Include path exceeds supported limits; manual reconciliation required')
+            candidates = [Path(mapped.anchor)]
+            for component in mapped.parts[1:]:
+                expanded = []
+                for parent in candidates:
+                    if not any(character in component for character in '*?['):
+                        candidate = parent / component
+                        if os.path.lexists(candidate):
+                            expanded.append(candidate)
+                        continue
+                    try:
+                        with os.scandir(parent) as entries:
+                            for entry in entries:
+                                directory_entries += 1
+                                if directory_entries > 4096:
+                                    raise AccessError('SSH Include traversal exceeds supported limits; manual reconciliation required')
+                                if entry.name.startswith('.') and not component.startswith('.'):
+                                    continue
+                                if fnmatch.fnmatchcase(entry.name, component):
+                                    expanded.append(Path(entry.path))
+                    except (FileNotFoundError, NotADirectoryError):
+                        continue
+                if len(expanded) > 128:
+                    raise AccessError('SSH Include expansion exceeds supported limits; manual reconciliation required')
+                candidates = expanded
+            return sorted(candidates)
+
+        def inspect(path, ancestors):
+            nonlocal remaining
+            information = path.stat()
+            identity = (information.st_dev, information.st_ino)
+            if identity in ancestors or len(ancestors) >= 16:
+                raise AccessError('Recursive SSH Include configuration requires manual reconciliation')
+            if identity in visited:
+                return
+            if len(visited) >= 128 or not stat.S_ISREG(information.st_mode) or information.st_size > remaining:
+                raise AccessError('SSH configuration exceeds supported inspection limits; manual reconciliation required')
+            visited.add(identity)
+            with path.open('rb') as stream:
+                content = stream.read(remaining + 1)
+            remaining -= len(content)
+            if remaining < 0:
+                raise AccessError('SSH configuration exceeds supported inspection limits; manual reconciliation required')
+            for line in content.decode().splitlines():
+                line = line.lstrip()
+                if not line or line.startswith('#'):
+                    continue
+                if line.startswith('"'):
+                    keyword, separator, value = line[1:].partition('"')
+                    if not separator:
+                        raise AccessError('Unsupported SSH keyword quoting requires manual reconciliation')
+                else:
+                    parsed = re.match(r'([^\s=]+)(.*)', line)
+                    if not parsed:
+                        raise AccessError('Unsupported SSH configuration syntax requires manual reconciliation')
+                    keyword, value = parsed.groups()
+                    value = value.lstrip()
+                    if value.startswith('='):
+                        value = value[1:]
+                if keyword.lower() == 'match':
+                    raise AccessError('SSH Match directives require manual reconciliation before initialization')
+                if keyword.lower() != 'include':
+                    continue
+                for pattern in include_arguments(value):
+                    for included in expand(pattern):
+                        inspect(included, ancestors | {identity})
+
+        try:
+            inspect(self.path('/etc/ssh/sshd_config'), set())
+        except (OSError, UnicodeError) as exc:
+            raise AccessError('Cannot inspect SSH configuration and includes; manual reconciliation required') from exc
+
+    def bootstrap_preflight(self):
+        for activity in ('is-active', 'is-enabled'):
+            if self.command(['systemctl', activity, '--quiet', 'ssh.socket'], False).returncode == 0:
+                raise AccessError('SSH socket activation requires manual handover; no access changes made')
+        if self.command(['systemctl', 'is-active', '--quiet', 'ssh.service'], False).returncode:
+            raise AccessError('An active ssh.service is required')
+        mode = self.command(['systemctl', 'show', 'ssh.service', '-p', 'KillMode', '--value']).stdout.strip()
+        if mode != 'process':
+            raise AccessError('ssh.service must preserve established sessions with KillMode=process')
+        arguments = self.service_arguments()
+        managed = self.path('/etc/systemd/system/ssh.service.d/apex-init.conf')
+        if managed.exists():
+            commands = [line.removeprefix('ExecStart=') for line in managed.read_text().splitlines()
+                        if line.startswith('ExecStart=') and line != 'ExecStart=']
+            if len(commands) != 1 or arguments != shlex.split(commands[0]):
+                raise AccessError('Active SSH service configuration differs from its managed invocation')
+        elif arguments != ['/usr/sbin/sshd', '-D']:
+            raise AccessError('Custom SSH service invocation requires manual access reconciliation')
+        defaults = self.path('/etc/default/ssh')
+        if defaults.exists():
+            for line in defaults.read_text().splitlines():
+                if re.match(r'^\s*SSHD_OPTS\s*=', line):
+                    value = line.split('=', 1)[1].strip().strip('\"\'')
+                    if value:
+                        raise AccessError('Custom SSHD_OPTS require manual access reconciliation')
+        self.unconditional_configuration()
+        self.command(['/usr/sbin/sshd', '-t'])
+        self.live_arguments()
+
+    def preflight(self):
+        self.bootstrap_preflight()
+        self.command(['visudo', '-c'])
+        self.command(['runuser', '-u', 'adam', '--', 'sudo', '-n', 'true'])
+
+    def service_arguments(self):
+        output = self.command(['systemctl', 'show', 'ssh.service', '-p', 'ExecStart', '--value']).stdout
+        matches = re.findall(r'argv\[\]=([^;]+)', output)
+        if len(matches) != 1:
+            raise AccessError('Unable to verify the SSH service invocation')
+        arguments = shlex.split(matches[0])
+        # Debian's stock unit has this optional variable. Its actual expansion is
+        # independently checked against /proc below, so a nonempty value fails.
+        if arguments == ['/usr/sbin/sshd', '-D', '$SSHD_OPTS']:
+            arguments.pop()
+        if not arguments or arguments[0] != '/usr/sbin/sshd':
+            raise AccessError('SSH service does not invoke the supported daemon')
+        return arguments
+
+    def live_arguments(self):
+        expected = self.service_arguments()
+        pid = self.command(['systemctl', 'show', 'ssh.service', '-p', 'MainPID', '--value']).stdout.strip()
+        if not pid.isdecimal() or int(pid) <= 1:
+            raise AccessError('SSH service has no verifiable active listener process')
+        process = self.path('/proc/' + pid)
+        try:
+            if os.readlink(process / 'exe') != '/usr/sbin/sshd':
+                raise AccessError('Active SSH process does not run the supported daemon')
+            parts = [part for part in (process / 'cmdline').read_bytes().decode().split('\0') if part]
+        except (OSError, UnicodeError):
+            raise AccessError('Unable to inspect the active SSH listener invocation') from None
+        if len(parts) == 1 and parts[0].startswith('sshd: ') and ' [listener]' in parts[0]:
+            actual = shlex.split(parts[0][6:].split(' [listener]', 1)[0])
+        else:
+            actual = parts
+        if actual != expected:
+            raise AccessError('Running SSH listener differs from the active service invocation')
+        return actual, pid
+
+    def verify_active(self, ports, final, address):
+        args, pid = self.live_arguments()
+        self.verify_policy([arg for arg in args if arg != '-D'], ports, final, address)
+        output = self.command(['ss', '-H', '-ltnp']).stdout
+        actual_ports = set()
+        for line in output.splitlines():
+            if '("sshd",' not in line:
+                continue
+            if set(re.findall(r'pid=(\d+)', line)) != {pid}:
+                raise AccessError('An additional SSH listener exists outside the managed service')
+            fields = line.split()
+            if len(fields) < 5 or not fields[3].rsplit(':', 1)[-1].isdecimal():
+                raise AccessError('Unable to verify the active SSH listening sockets')
+            actual_ports.add(int(fields[3].rsplit(':', 1)[-1]))
+        if actual_ports != set(ports):
+            raise AccessError('Active SSH listening ports differ from the declared policy')
+
+    def policy(self, ports, final, address):
+        args = ['/usr/sbin/sshd']
+        for port in sorted(set(ports)):
+            args += ['-p', str(port)]
+        if final:
+            for value in ('PermitRootLogin=no', 'PasswordAuthentication=no',
+                          'KbdInteractiveAuthentication=no', 'PubkeyAuthentication=yes',
+                          'AuthenticationMethods=publickey', 'AllowUsers=adam',
+                          'X11Forwarding=no', 'PermitEmptyPasswords=no'):
+                args += ['-o', value]
+        self.verify_policy(args, ports, final, address)
+        return args
+
+    def verify_policy(self, args, ports, final, address):
+        self.unconditional_configuration()
+        self.command(args + ['-t'])
+        users = {'adam', 'root'}
+        if final:
+            users.update(account.pw_name for account in pwd.getpwall())
+        for user in sorted(users):
+            output = self.command(args + ['-T', '-C', f'user={user},host=localhost,addr={address}']).stdout
+            values = {}
+            for line in output.splitlines():
+                key, _, value = line.partition(' ')
+                values.setdefault(key, []).append(value)
+            if set(values.get('port', [])) != {str(p) for p in ports}:
+                raise AccessError('Effective SSH ports do not match the handover policy')
+            if final and any(values.get(k) != [v] for k, v in {
+                    'permitrootlogin': 'no', 'passwordauthentication': 'no',
+                    'kbdinteractiveauthentication': 'no', 'pubkeyauthentication': 'yes',
+                    'authenticationmethods': 'publickey', 'allowusers': 'adam',
+                    'x11forwarding': 'no', 'permitemptypasswords': 'no'}.items()):
+                raise AccessError('Effective SSH authentication policy is inconsistent')
+
+    def current_final(self, address):
+        try:
+            self.verify_active([2222], True, address)
+            return True
+        except AccessError:
+            return False
+
+    def install_policy(self, ports, final, address):
+        args = self.policy(ports, final, address)
+        path = self.path('/etc/systemd/system/ssh.service.d/apex-init.conf')
+        atomic(path, ('[Service]\nExecStart=\nExecStart=' + ' '.join(args + ['-D']) + '\n').encode(), 0o644)
+        self.command(['systemctl', 'daemon-reload'])
+        actual = self.command(['systemctl', 'show', 'ssh.service', '-p', 'ExecStart', '--value']).stdout
+        match = re.search(r'argv\[\]=([^;]+)', actual)
+        if not match or shlex.split(match[1]) != args + ['-D']:
+            raise AccessError('Another service override prevents the managed SSH policy')
+        self.command(['systemctl', 'restart', 'ssh.service'])
+        self.verify_active(ports, final, address)
+
+    def firewall_needed(self):
+        result = self.command(['ufw', 'status'], False)
+        if result.returncode or 'Status: active' not in result.stdout:
+            return False
+        if re.search(r'^2222/tcp(?:\s|\(v6\)).*ALLOW', result.stdout, re.MULTILINE):
+            return False
+        return True
+
+    def open_firewall(self):
+        # Existing allows remain owned by their administrator. Only our commented rule is removed.
+        self.command(['ufw', 'allow', '2222/tcp', 'comment', 'apex-init-access'])
+
+    def close_firewall(self):
+        self.command(['ufw', '--force', 'delete', 'allow', '2222/tcp', 'comment', 'apex-init-access'])
+
+    def arm(self):
+        bundle = Path(__file__).resolve().parents[1]
+        if not re.fullmatch(r'/usr/local/lib/apex-provisioner/[a-f0-9]{64}', str(bundle)):
+            raise AccessError('Access runner must execute from the installed root-owned bundle')
+        service = ('[Unit]\nDescription=APEX pending access rollback\nAfter=local-fs.target\n'
+                   '[Service]\nType=oneshot\nWorkingDirectory=' + str(bundle) + '\n'
+                   'ExecStart=/usr/bin/python3 -m provisioning.access tick\n')
+        timer = ('[Unit]\nDescription=APEX access rollback deadline and boot recovery\n'
+                 '[Timer]\nOnBootSec=5s\nOnUnitActiveSec=10s\nAccuracySec=1s\n'
+                 '[Install]\nWantedBy=timers.target\n')
+        atomic(self.path('/etc/systemd/system/apex-access-rollback.service'), service.encode(), 0o644)
+        atomic(self.path('/etc/systemd/system/apex-access-rollback.timer'), timer.encode(), 0o644)
+        self.command(['systemctl', 'daemon-reload'])
+        self.command(['systemctl', 'enable', '--now', 'apex-access-rollback.timer'])
+
+    def disarm(self):
+        self.command(['systemctl', 'disable', '--now', 'apex-access-rollback.timer'])
+
+    def restore(self, state):
+        path = self.path('/etc/systemd/system/ssh.service.d/apex-init.conf')
+        if state['previous'] is None:
+            path.unlink(missing_ok=True)
+        else:
+            atomic(path, base64.b64decode(state['previous']), 0o644)
+        self.command(['/usr/sbin/sshd', '-t'])
+        self.command(['systemctl', 'daemon-reload'])
+        self.command(['systemctl', 'restart', 'ssh.service'])
+        if state.get('firewall_added'):
+            self.close_firewall()
+
+
+def atomic(path, data, mode=0o600):
+    path = Path(path)
+    for parent in [path, *path.parents]:
+        if parent.is_symlink():
+            raise AccessError('Refusing a symlink in a managed access path')
+    path.parent.mkdir(parents=True, exist_ok=True)
+    fd, temporary = tempfile.mkstemp(prefix='.apex-access-', dir=path.parent)
+    try:
+        with os.fdopen(fd, 'wb') as stream:
+            os.fchmod(stream.fileno(), mode)
+            stream.write(data)
+            stream.flush()
+            os.fsync(stream.fileno())
+        os.replace(temporary, path)
+    finally:
+        if os.path.exists(temporary):
+            os.unlink(temporary)
+    directory = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY)
+    try:
+        os.fsync(directory)
+    finally:
+        os.close(directory)
+
+
+class Transition:
+    def __init__(self, host=None, now=time.monotonic, security=None):
+        from .security import Security
+        self.host = host or Host()
+        self.security = security if security is not None else Security(self.host)
+        self.now = now
+        self.directory = self.host.path('/var/lib/apex-init/access')
+
+    def load(self):
+        path = self.directory / 'state.json'
+        return json.loads(path.read_text()) if path.exists() else None
+
+    def save(self, state):
+        atomic(self.directory / 'state.json', json.dumps(state, sort_keys=True).encode())
+
+    def execute(self, operation, token=None, original_port=None, client_address=None):
+        # Rollback must never wait for a stalled enrollment stage holding init.lock.
+        if operation in ('tick', 'rollback'):
+            return self._execute(operation, token, original_port, client_address)
+        from .state import ProvisionError, exclusive_lock
+        try:
+            with exclusive_lock(self.host.path('/var/lib/apex-init/init.lock')):
+                return self._execute(operation, token, original_port, client_address)
+        except ProvisionError as error:
+            raise AccessError(str(error)) from None
+
+    def _execute(self, operation, token=None, original_port=None, client_address=None):
+        if token is not None and not re.fullmatch(r'[a-f0-9]{32}', token):
+            raise AccessError('Invalid access transition token')
+        for path in [self.directory, *self.directory.parents]:
+            if path.is_symlink():
+                raise AccessError('Invalid access state directory')
+        self.directory.mkdir(parents=True, exist_ok=True, mode=0o700)
+        if self.directory.stat().st_uid != os.geteuid():
+            raise AccessError('Access state directory is not owned by the provisioner user')
+        os.chmod(self.directory, 0o700)
+        descriptor = os.open(self.directory / 'lock', os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW, 0o600)
+        with os.fdopen(descriptor, 'w') as lock:
+            fcntl.flock(lock, fcntl.LOCK_EX)
+            state = self.load()
+            if state and state['status'] == 'recovery_pending':
+                self._rollback(state)
+                if operation != 'tick':
+                    raise AccessError('Pending recovery completed; retry initialization')
+                return {'status': 'rolled_back'}
+            pending = state and state['status'] in ('transition', 'final')
+            if pending and (self.now() >= state['deadline'] or self.host.boot_id() != state['boot_id']):
+                self._rollback(state)
+                if operation != 'tick':
+                    raise AccessError('Uncommitted access transition expired or rebooted; previous access restored')
+                return {'status': 'rolled_back'}
+            if operation == 'tick':
+                return {'status': state['status'] if state else 'absent'}
+            if operation == 'begin':
+                if pending:
+                    raise AccessError('A pending access transition exists; wait for rollback before retrying')
+                if not token or not original_port or not 1 <= original_port <= 65535:
+                    raise AccessError('Access transition requires an original port and token')
+                try:
+                    ipaddress.ip_address(client_address)
+                except ValueError:
+                    raise AccessError('Invalid SSH client address') from None
+                self.host.preflight()
+                path = self.host.path('/etc/systemd/system/ssh.service.d/apex-init.conf')
+                preserve_policy = False
+                if state and state['status'] == 'committed':
+                    if state['fingerprint'] != self.host.fingerprint():
+                        raise AccessError('Committed access configuration changed; reconcile before retrying')
+                    self.host.verify_active([2222], True, client_address)
+                    if state.get('security'):
+                        self.security.verify(state['security'])
+                        return {'status': 'committed'}
+                    preserve_policy = True
+                if original_port == 2222 and not path.exists() and self.host.current_final(client_address):
+                    preserve_policy = True
+                security_state = self.security.snapshot(self.security.preflight(), original_port, client_address)
+                self.host.arm()
+                state = {'status': 'transition', 'token': token, 'deadline': self.now() + 300,
+                         'boot_id': self.host.boot_id(), 'client_address': client_address,
+                         'original_port': original_port, 'firewall_added': False,
+                         'preserve_policy': preserve_policy, 'security': security_state,
+                         'previous': base64.b64encode(path.read_bytes()).decode() if path.exists() else None}
+                self.save(state)
+                try:
+                    self.security.activate(state['security'], lambda value: self._save_security(state, value))
+                    if not preserve_policy:
+                        self.host.install_policy([original_port, 2222], False, client_address)
+                    state['fingerprint'] = self.host.fingerprint()
+                    self.save(state)
+                except Exception:
+                    self._rollback(state)
+                    raise
+                return {'status': 'transition'}
+            if not state or (not pending and state['status'] != 'existing') or token != state['token']:
+                raise AccessError('No matching pending access transition')
+            if state['status'] == 'existing':
+                if operation == 'rollback':
+                    state['status'] = 'rolled_back'
+                    self.save(state)
+                    return {'status': 'rolled_back'}
+                raise AccessError('Legacy access verification lacks security activation; retry initialization')
+            if operation == 'rollback':
+                self._rollback(state)
+                return {'status': 'rolled_back'}
+            if state.get('fingerprint') != self.host.fingerprint():
+                raise AccessError('External SSH configuration changed; finalization stopped, rollback remains armed')
+            if operation == 'finalize':
+                if state['status'] != 'transition':
+                    raise AccessError('Access transition is not ready for final policy')
+                try:
+                    if not state.get('preserve_policy'):
+                        self.host.install_policy([2222], True, state['client_address'])
+                    self.security.finalize(state['security'], lambda value: self._save_security(state, value))
+                    state['status'] = 'final'
+                    state['fingerprint'] = self.host.fingerprint()
+                    self.save(state)
+                except Exception:
+                    self._rollback(state)
+                    raise
+                return {'status': 'final'}
+            if operation == 'commit' and state['status'] == 'final':
+                self.host.verify_active([2222], True, state['client_address'])
+                self.security.verify(state['security'])
+                state['status'] = 'committed'
+                self.save(state)  # Timer observes this before it can restore anything.
+                self.host.disarm()
+                return {'status': 'committed'}
+            raise AccessError('Invalid access transition operation')
+
+    def _save_security(self, state, value):
+        state['security'] = value
+        self.save(state)
+
+    def _rollback(self, state):
+        errors = []
+        if state.get('security'):
+            try:
+                self.security.restore(state['security'], lambda value: self._save_security(state, value))
+            except Exception:
+                errors.append('security')
+        if not state.get('ssh_restored'):
+            try:
+                self.host.restore(state)
+                state['ssh_restored'] = True
+            except Exception:
+                errors.append('ssh')
+        if errors:
+            state['status'] = 'recovery_pending'
+            state['recovery_errors'] = errors
+            self.save(state)
+            raise AccessError('Recovery remains pending for ' + ', '.join(errors) +
+                              '; watchdog remains armed; inspect the root-owned access record')
+        state['status'] = 'rolled_back'
+        state.pop('recovery_errors', None)
+        self.save(state)
+        self.host.disarm()
+
+
+def main(argv=None):
+    from .security import SecurityError
+    parser = argparse.ArgumentParser()
+    parser.add_argument('operation', choices=['preflight', 'begin', 'finalize', 'commit', 'rollback', 'tick'])
+    parser.add_argument('--token')
+    parser.add_argument('--original-port', type=int)
+    parser.add_argument('--client-address')
+    args = parser.parse_args(argv)
+    if os.geteuid() != 0:
+        parser.error('Access handover requires root')
+    try:
+        if args.operation == 'preflight':
+            Host().bootstrap_preflight()
+            result = {'status': 'prerequisites-verified'}
+        else:
+            result = Transition().execute(args.operation, args.token, args.original_port, args.client_address)
+        print(json.dumps(result))
+        return 0
+    except (AccessError, SecurityError) as exc:
+        print(str(exc), file=sys.stderr)
+        return 1
+
+
+if __name__ == '__main__':
+    sys.exit(main())

ci/scripts/init-node/provisioning/baseline.py

@@ -0,0 +1,488 @@
+"""Shared Debian baseline. Callers own the exclusive initialization lock."""
+from __future__ import annotations
+
+from contextlib import contextmanager
+import fcntl
+from functools import wraps
+import hashlib
+import json
+import os
+from pathlib import Path
+import re
+import shutil
+import stat
+import subprocess
+import tempfile
+import urllib.request
+import uuid
+
+
+class BaselineError(RuntimeError):
+    """A required baseline postcondition could not be established."""
+
+
+def expected_errors(operation):
+    @wraps(operation)
+    def guarded(*args, **kwargs):
+        try:
+            return operation(*args, **kwargs)
+        except (OSError, ValueError) as exc:
+            raise BaselineError("Baseline filesystem or saved state is invalid; inspect the target before retrying.") from exc
+    return guarded
+
+
+BASE_PACKAGES = ("curl", "wget", "git", "rsyslog", "ufw")
+ADMIN_PACKAGES = (
+    "tcpdump", "socat", "rsync", "bind9-dnsutils", "iproute2", "net-tools",
+    "traceroute", "ca-certificates", "jq", "sed", "gawk", "tar", "gzip",
+    "bzip2", "xz-utils", "unzip", "htop", "iotop", "strace", "lsof",
+    "sysstat", "parted", "fdisk", "pciutils", "usbutils", "smartmontools",
+    "tmux", "vim", "less", "tree", "bash-completion", "python3",
+    "python3-minimal", "python3-yaml", "sudo", "cron", "iptables", "ipset",
+    "debian-archive-keyring", "kmod",
+)
+DOCKER_PACKAGES = ("docker-ce", "docker-ce-cli", "containerd.io",
+                   "docker-buildx-plugin", "docker-compose-plugin")
+SECURITY_PACKAGES = ("crowdsec", "crowdsec-firewall-bouncer-iptables")
+DOCKER_CONFLICTS = ("docker.io", "docker-doc", "docker-compose", "docker-buildx",
+                    "podman-docker", "containerd", "runc")
+DOCKER_CONFIG = {"data-root": "/srv/docker", "log-driver": "json-file",
+                 "log-opts": {"max-size": "20m", "max-file": "3"}}
+HELPER_REVISION = "020a8699f95592561f254d8d4ad1bb40d401dfc7"
+HELPER_URL = f"https://raw.githubusercontent.com/chaifeng/ufw-docker/{HELPER_REVISION}/ufw-docker"
+HELPER_SHA256 = "643e56b080567c567b4aa28650196849b2a2da5dd0473fd3e5216b0886035ab0"
+POLICY = Path("/usr/sbin/policy-rc.d")
+POLICY_STATE = Path("/usr/sbin/.apex-policy-rc.d")
+SECURITY_STATE = Path("/var/lib/apex/baseline-security.json")
+SECURITY_SERVICES = {"ufw": "ufw", "crowdsec": "crowdsec",
+                     "crowdsec-firewall-bouncer-iptables": "crowdsec-firewall-bouncer"}
+SECURITY_CONDITION = "[Unit]\nConditionPathExists=/dev/null/apex-security-start-allowed\n"
+APT = ["apt-get", "-o", "DPkg::Lock::Timeout=60", "-o",
+       "Acquire::Retries=2", "-o", "Acquire::http::Timeout=30", "-o",
+       "Acquire::https::Timeout=30"]
+
+
+def run(args, *, check=True, timeout=300):
+    try:
+        result = subprocess.run(args, text=True, capture_output=True, timeout=timeout,
+                                env={**os.environ, "LC_ALL": "C", "DEBIAN_FRONTEND": "noninteractive"})
+    except (OSError, subprocess.TimeoutExpired) as exc:
+        raise BaselineError(f"Cannot complete {args[0]}: {exc}") from exc
+    if check and result.returncode:
+        raise BaselineError(f"{args[0]} failed with exit status {result.returncode}; inspect target system logs.")
+    return result
+
+
+def preflight(*, strict=True):
+    if os.geteuid() != 0:
+        raise BaselineError("Baseline requires root.")
+    if not strict:
+        if not Path("/etc/debian_version").is_file():
+            raise BaselineError("Base policy requires a Debian-based system.")
+        return
+    if not Path("/etc/os-release").is_file():
+        raise BaselineError("Supported target is Debian 13 amd64 only.")
+    release = dict(line.split("=", 1) for line in Path("/etc/os-release").read_text().splitlines()
+                   if "=" in line and not line.startswith("#"))
+    if (release.get("ID", "").strip('"') != "debian" or
+            release.get("VERSION_ID", "").strip('"') != "13" or
+            run(["dpkg", "--print-architecture"]).stdout.strip() != "amd64"):
+        raise BaselineError("Supported target is Debian 13 amd64 only.")
+
+
+def package_versions():
+    result = run(["dpkg-query", "-W", "-f=${binary:Package}\t${Status}\t${Version}\n"])
+    installed = {}
+    for line in result.stdout.splitlines():
+        fields = line.split("\t")
+        if len(fields) == 3 and fields[1].split()[1:] == ["ok", "installed"]:
+            installed[fields[0].removesuffix(":amd64")] = fields[2]
+    return installed
+
+
+def write_managed(path, content, mode=0o644):
+    path = Path(path)
+    for parent in (path, *path.parents):
+        if parent.is_symlink():
+            raise BaselineError(f"Refusing managed symlink: {parent}")
+    payload = content.encode() if isinstance(content, str) else content
+    if path.exists() and path.read_bytes() == payload:
+        if path.stat().st_mode & 0o777 != mode:
+            path.chmod(mode)
+        return False
+    path.parent.mkdir(parents=True, exist_ok=True)
+    fd, temporary = tempfile.mkstemp(prefix=".apex-", dir=path.parent)
+    try:
+        with os.fdopen(fd, "wb") as stream:
+            stream.write(payload)
+            stream.flush()
+            os.fsync(stream.fileno())
+        os.chmod(temporary, mode)
+        os.replace(temporary, path)
+    finally:
+        if os.path.exists(temporary):
+            os.unlink(temporary)
+    return True
+
+
+def restore_service_policy():
+    if not POLICY_STATE.exists():
+        return
+    backup = POLICY_STATE / "original"
+    if (POLICY_STATE.is_symlink() or POLICY_STATE.stat().st_uid != os.geteuid()):
+        raise BaselineError("Refusing untrusted service policy recovery state.")
+    if (POLICY_STATE / "ready").exists() and POLICY.exists():
+        original_matches = (backup.exists() and not POLICY.is_symlink() and
+                            POLICY.read_bytes() == backup.read_bytes())
+        symlink_matches = (backup.is_symlink() and POLICY.is_symlink() and
+                           os.readlink(backup) == os.readlink(POLICY))
+        guard_matches = not POLICY.is_symlink() and POLICY.read_bytes() == b"#!/bin/sh\nexit 101\n"
+        if not (original_matches or symlink_matches or guard_matches):
+            raise BaselineError("Service policy changed outside APEX; recovery requires explicit reconciliation.")
+    if backup.exists() or backup.is_symlink():
+        # Copy first: an interrupted restoration must retain its recovery copy.
+        temporary = POLICY_STATE / "restoring"
+        if temporary.exists() or temporary.is_symlink():
+            temporary.unlink()
+        shutil.copy2(backup, temporary, follow_symlinks=False)
+        os.replace(temporary, POLICY)
+    elif (POLICY_STATE / "absent").exists():
+        POLICY.unlink(missing_ok=True)
+    elif (POLICY_STATE / "ready").exists():
+        raise BaselineError("Service policy recovery is incomplete; inspect /usr/sbin/.apex-policy-rc.d.")
+    shutil.rmtree(POLICY_STATE)
+
+
+@contextmanager
+def service_start_guard():
+    lock_path = Path("/run/lock/apex-package-policy.lock")
+    lock_path.parent.mkdir(parents=True, exist_ok=True)
+    with lock_path.open("a") as lock:
+        try:
+            fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
+        except BlockingIOError:
+            raise BaselineError("Another APEX package operation is running.") from None
+        with _service_start_guard():
+            yield
+
+
+@contextmanager
+def _service_start_guard():
+    restore_service_policy()
+    POLICY_STATE.mkdir(mode=0o700)
+    try:
+        if POLICY.exists() or POLICY.is_symlink():
+            shutil.copy2(POLICY, POLICY_STATE / "original", follow_symlinks=False)
+        else:
+            (POLICY_STATE / "absent").touch()
+        (POLICY_STATE / "ready").touch()
+        # Replace the path itself, preserving any original symlink in the backup.
+        temporary = POLICY_STATE / "guard"
+        temporary.write_text("#!/bin/sh\nexit 101\n")
+        temporary.chmod(0o755)
+        os.replace(temporary, POLICY)
+        yield
+    finally:
+        restore_service_policy()
+
+
+@contextmanager
+def security_service_guard(packages):
+    """Guard new security units without occupying package-owned unit paths."""
+    record = json.loads(SECURITY_STATE.read_text()) if SECURITY_STATE.exists() else {"version": 2, "services": {}}
+    if not isinstance(record, dict) or record.get("version") != 2:
+        raise BaselineError("Legacy security service guard state requires explicit reconciliation.")
+    managed = record.get("services")
+    if not isinstance(managed, dict) or any(SECURITY_SERVICES.get(key) != value for key, value in managed.items()):
+        raise BaselineError("Invalid baseline security service recovery state.")
+    installed = package_versions()
+    for package, service in SECURITY_SERVICES.items():
+        if package in packages and package not in installed and package not in managed:
+            if run(["systemctl", "is-enabled", service], check=False).stdout.strip() == "masked":
+                continue
+            managed[package] = service
+            write_managed(SECURITY_STATE, json.dumps({"version": 2, "services": managed}), 0o600)
+    for service in managed.values():
+        dropin = Path(f"/etc/systemd/system/{service}.service.d/99-apex-bootstrap-guard.conf")
+        if dropin.exists() and dropin.read_text() != SECURITY_CONDITION:
+            raise BaselineError("Security service guard changed outside APEX; reconcile it explicitly.")
+        # /dev/null cannot have children: this condition stays false after reboot.
+        write_managed(dropin, SECURITY_CONDITION)
+    if managed:
+        run(["systemctl", "daemon-reload"])
+    try:
+        yield
+    finally:
+        installed = package_versions()
+        for package, service in managed.items():
+            unit_exists = any(Path(f"{directory}/{service}.service").exists() for directory in
+                              ("/etc/systemd/system", "/usr/lib/systemd/system", "/lib/systemd/system"))
+            run(["systemctl", "disable", service], check=package in installed or unit_exists)
+            dropin = Path(f"/etc/systemd/system/{service}.service.d/99-apex-bootstrap-guard.conf")
+            if dropin.exists():
+                if dropin.is_symlink() or dropin.read_text() != SECURITY_CONDITION:
+                    raise BaselineError("Security service guard changed outside APEX; reconcile it explicitly.")
+                dropin.unlink()
+        if managed:
+            run(["systemctl", "daemon-reload"])
+        SECURITY_STATE.unlink(missing_ok=True)
+
+
+def install_missing(packages):
+    installed = package_versions()
+    missing = [package for package in packages if package not in installed]
+    if not missing:
+        return []
+    run([*APT, "update", "--error-on=any"])
+    command = [*APT, "install", "-y", "--no-install-recommends", "--no-upgrade", "--no-remove", *missing]
+    plan = run([*command, "--simulate"]).stdout
+    if any(line.startswith("Remv ") or re.match(r"^Inst \S+ \[", line)
+           for line in plan.splitlines()):
+        raise BaselineError("Missing packages require an upgrade/removal; reconcile packages explicitly before retrying.")
+    # Package writes must settle even after coordinator loss; stages gate later work.
+    run(command, timeout=None)
+    remaining = set(missing) - package_versions().keys()
+    if remaining:
+        raise BaselineError("Required packages not installed: " + ", ".join(sorted(remaining)))
+    return missing
+
+
+def base_policy():
+    return {"packages": list(BASE_PACKAGES), "files": {
+        "/etc/apt/apt.conf.d/99-apex": 'APT::Keep-Downloaded-Packages "0";\n',
+        "/etc/systemd/journald.conf.d/99-apex.conf": "[Journal]\nSystemMaxUse=200M\n",
+    }, "helper": {"url": HELPER_URL, "sha256": HELPER_SHA256}}
+
+
+def configure_common():
+    policy = base_policy()
+    for legacy in ("/etc/apt/apt.conf.d/01clean", "/etc/systemd/journald.conf.d/maxuse.conf"):
+        path = Path(legacy)
+        if path.exists() or path.is_symlink():
+            path.unlink()
+    for path, content in policy["files"].items():
+        changed = write_managed(path, content)
+        if changed and path.endswith("99-apex.conf"):
+            run(["systemctl", "restart", "systemd-journald"])
+    target = Path("/usr/local/bin/ufw-docker")
+    if (not target.is_file() or hashlib.sha256(target.read_bytes()).hexdigest() != HELPER_SHA256):
+        try:
+            with urllib.request.urlopen(HELPER_URL, timeout=30) as response:
+                data = response.read()
+        except OSError as exc:
+            raise BaselineError("Unable to download pinned ufw-docker helper.") from exc
+        if hashlib.sha256(data).hexdigest() != HELPER_SHA256:
+            raise BaselineError("Pinned ufw-docker checksum mismatch.")
+        write_managed(target, data, 0o755)
+    elif target.is_symlink():
+        raise BaselineError("Refusing a symlinked ufw-docker helper.")
+    else:
+        target.chmod(0o755)
+
+
+def docker_preflight(installed):
+    conflicts = sorted(set(DOCKER_CONFLICTS) & installed.keys())
+    if conflicts:
+        raise BaselineError("Conflicting Docker packages require explicit reconciliation: " + ", ".join(conflicts))
+    path = Path("/etc/docker/daemon.json")
+    try:
+        config = json.loads(path.read_text()) if path.exists() else {}
+    except (OSError, ValueError) as exc:
+        raise BaselineError("Cannot validate existing Docker daemon configuration.") from exc
+    if not isinstance(config, dict):
+        raise BaselineError("Docker daemon configuration must be an object.")
+    for name, value in DOCKER_CONFIG.items():
+        if name in config and config[name] != value:
+            raise BaselineError(f"Conflicting Docker {name}; reconcile explicitly before initialization.")
+    old_data = Path("/var/lib/docker")
+    if old_data.exists() and any(old_data.iterdir()):
+        raise BaselineError("Existing /var/lib/docker data must be reconciled explicitly; no data migration is automatic.")
+    data = Path("/srv/docker")
+    if data.is_symlink() or data.parent.is_symlink():
+        raise BaselineError("Refusing a redirected Docker data-root.")
+    if data.exists() and any(data.iterdir()) and config.get("data-root") != "/srv/docker":
+        raise BaselineError("Unmanaged /srv/docker data requires explicit reconciliation.")
+    if "docker-ce" in installed:
+        active = run(["systemctl", "is-active", "--quiet", "docker"], check=False).returncode == 0
+        if active:
+            root = run(["docker", "info", "--format", "{{.DockerRootDir}}"] ).stdout.strip()
+            if root != "/srv/docker" or any(config.get(key) != value for key, value in DOCKER_CONFIG.items()):
+                raise BaselineError("Running Docker has incompatible data-root/log policy; reconcile it before retrying.")
+    return {**config, **DOCKER_CONFIG}
+
+
+def configure_sources():
+    write_managed("/etc/apt/sources.list.d/apex-zfs.sources",
+                  "Types: deb\nURIs: https://deb.debian.org/debian\nSuites: trixie\n"
+                  "Components: contrib\nArchitectures: amd64\n"
+                  "Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg\n")
+    for name, key_url, uri, suite, component in (
+        ("docker", "https://download.docker.com/linux/debian/gpg",
+         "https://download.docker.com/linux/debian", "trixie", "stable"),
+        ("crowdsec", "https://packagecloud.io/crowdsec/crowdsec/gpgkey",
+         "https://packagecloud.io/crowdsec/crowdsec/any", "any", "main"),
+    ):
+        key = Path(f"/etc/apt/keyrings/apex-{name}.asc")
+        if not key.exists():
+            try:
+                # Both repository key URLs above are fixed HTTPS endpoints.
+                with urllib.request.urlopen(key_url, timeout=30) as response:  # noqa: S310
+                    data = response.read()
+            except OSError as exc:
+                raise BaselineError(f"Cannot download {name} repository signing key.") from exc
+            if not data.startswith(b"-----BEGIN PGP PUBLIC KEY BLOCK-----"):
+                raise BaselineError(f"Invalid {name} repository signing key.")
+            write_managed(key, data)
+        write_managed(f"/etc/apt/sources.list.d/apex-{name}.sources",
+                      f"Types: deb\nURIs: {uri}\nSuites: {suite}\nComponents: {component}\n"
+                      f"Architectures: amd64\nSigned-By: {key}\n")
+
+
+def ensure_service(name):
+    if run(["systemctl", "is-enabled", "--quiet", name], check=False).returncode:
+        run(["systemctl", "enable", name])
+    if run(["systemctl", "is-active", "--quiet", name], check=False).returncode:
+        run(["systemctl", "start", name])
+
+
+def repair_bouncer_credentials():
+    """Repair package placeholders without rewriting administrator YAML settings."""
+    config = Path("/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml")
+    identity = config.with_suffix(".yaml.id")
+    for path in (config, identity):
+        if any(parent.is_symlink() for parent in (path, *path.parents)):
+            raise BaselineError("Refusing redirected CrowdSec bouncer credential paths.")
+        if path.exists():
+            metadata = path.stat()
+            if (not stat.S_ISREG(metadata.st_mode) or metadata.st_nlink != 1 or
+                    metadata.st_uid != os.geteuid()):
+                raise BaselineError("Refusing untrusted CrowdSec bouncer credential files.")
+    if not config.exists():
+        raise BaselineError("CrowdSec bouncer configuration is missing; reconcile the package installation.")
+    content = config.read_text()
+    entries = list(re.finditer(r"^(?:api_key|'api_key'|\"api_key\"):[^\r\n]*", content, re.MULTILINE))
+    if len(entries) != 1:
+        raise BaselineError("CrowdSec bouncer requires one explicit api_key; reconcile its configuration.")
+    entry = entries[0]
+    for following in content[entry.end():].splitlines():
+        if not following.strip() or following.lstrip().startswith("#"):
+            continue
+        if following[:1].isspace():
+            raise BaselineError("Multiline CrowdSec bouncer api_key requires explicit reconciliation.")
+        break
+    scalar = re.fullmatch(r"(?:api_key|'api_key'|\"api_key\"):[ \t]*(?P<value>\"[^\"]*\"|'[^']*'|[^ \t#'\"]*)(?:[ \t]+(?:#.*)?)?", entry.group())
+    if scalar is None:
+        raise BaselineError("Unsupported CrowdSec bouncer api_key; reconcile its configuration.")
+    value = scalar.group("value")
+    uninitialized = value in ("null", "Null", "NULL", "~")
+    if value[:1] in ("'", '"'):
+        value = value[1:-1]
+    if not uninitialized and value not in ("", "<API_KEY>", "$API_KEY", "${API_KEY}"):
+        return
+    metadata = config.stat()
+    registration = "cs-firewall-bouncer-apex-" + uuid.uuid4().hex
+    key = run(["cscli", "-oraw", "bouncers", "add", registration]).stdout.strip()
+    if not re.fullmatch(r"[A-Za-z0-9_+/=-]{24,256}", key):
+        raise BaselineError("CrowdSec bouncer registration did not return a usable API key.")
+    start, end = scalar.span("value")
+    separator = " " if entry.group()[start - 1] == ":" else ""
+    updated = content[:entry.start() + start] + separator + key + content[entry.start() + end:]
+
+    def publish(path, payload, mode, owner):
+        fd, temporary = tempfile.mkstemp(prefix=".apex-", dir=path.parent)
+        try:
+            with os.fdopen(fd, "w") as stream:
+                os.fchown(stream.fileno(), *owner)
+                os.fchmod(stream.fileno(), mode)
+                stream.write(payload)
+                stream.flush()
+                os.fsync(stream.fileno())
+            os.replace(temporary, path)
+            directory = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY)
+            try:
+                os.fsync(directory)
+            finally:
+                os.close(directory)
+        finally:
+            if os.path.exists(temporary):
+                os.unlink(temporary)
+
+    # Publish the key last: interruption can orphan a registration, but retries
+    # replace the placeholder and its package bookkeeping together in this order.
+    owner = (metadata.st_uid, metadata.st_gid)
+    publish(identity, registration + "\n", 0o600, owner)
+    publish(config, updated, (stat.S_IMODE(metadata.st_mode) & 0o640) | 0o600, owner)
+    if config.read_text() != updated or identity.read_text().strip() != registration:
+        raise BaselineError("CrowdSec bouncer credential publication did not converge.")
+
+
+@expected_errors
+def apply_base_policy():
+    """The bounded configure/base policy: no Docker, storage or enrollment."""
+    preflight(strict=False)
+    with service_start_guard(), security_service_guard(BASE_PACKAGES):
+        install_missing(BASE_PACKAGES)
+    configure_common()
+    ensure_service("rsyslog")
+    run(["/usr/local/bin/ufw-docker", "install"])
+    return {"packages": {key: value for key, value in package_versions().items() if key in BASE_PACKAGES}}
+
+
+@expected_errors
+def apply_baseline():
+    """Converge baseline and return verified package versions and capabilities."""
+    preflight()
+    installed = package_versions()
+    docker_config = docker_preflight(installed)
+    kernel = run(["uname", "-r"]).stdout.strip()
+    if not re.fullmatch(r"[A-Za-z0-9.+_-]+", kernel):
+        raise BaselineError("Unsupported running kernel release.")
+    packages = (*BASE_PACKAGES, *ADMIN_PACKAGES, *DOCKER_PACKAGES,
+                *SECURITY_PACKAGES, "linux-headers-amd64", f"linux-headers-{kernel}",
+                "zfsutils-linux", "zfs-dkms")
+    with service_start_guard(), security_service_guard(packages):
+        install_missing((*BASE_PACKAGES, *ADMIN_PACKAGES))
+        configure_sources()
+        install_missing(("crowdsec",))
+        install_missing(packages)
+        repair_bouncer_credentials()
+    configure_common()
+    write_managed("/etc/docker/daemon.json", json.dumps(docker_config, indent=2) + "\n")
+    Path("/srv/docker").mkdir(parents=True, exist_ok=True)
+    ensure_service("containerd")
+    ensure_service("docker")
+    ensure_service("rsyslog")
+    ensure_service("cron")
+    root = run(["docker", "info", "--format", "{{.DockerRootDir}}"] ).stdout.strip()
+    if root != "/srv/docker":
+        raise BaselineError("Docker did not activate the required /srv/docker data-root.")
+    compose = run(["docker", "compose", "version", "--short"]).stdout.strip()
+    if not compose:
+        raise BaselineError("Docker Compose capability is unavailable.")
+    module = run(["modprobe", "zfs"], check=False)
+    if module.returncode:
+        raise BaselineError(f"ZFS unavailable for running kernel {kernel}; install matching headers/module or reboot into the supported kernel, then retry. No automatic reboot was performed.")
+    zfs = run(["zfs", "version"]).stdout.strip()
+    versions = package_versions()
+    if set(packages) - versions.keys():
+        raise BaselineError("Baseline package verification failed.")
+    return {"packages": {name: versions[name] for name in packages},
+            "capabilities": {"docker_data_root": root, "compose": compose, "zfs": zfs,
+                             "kernel": kernel, "ufw_docker_sha256": HELPER_SHA256},
+            "sources": {"docker": "docker.com/debian trixie stable",
+                        "crowdsec": "packagecloud.io/crowdsec/crowdsec/any any main",
+                        "zfs": "Debian trixie contrib"}}
+
+
+if __name__ == "__main__":
+    import argparse
+    from .state import ProvisionError, exclusive_lock
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--base-policy", action="store_true")
+    args = parser.parse_args()
+    try:
+        preflight(strict=not args.base_policy)
+        with exclusive_lock("/var/lib/apex-init/init.lock"):
+            print(json.dumps(apply_base_policy() if args.base_policy else apply_baseline(), sort_keys=True))
+    except (BaselineError, OSError, ProvisionError) as exc:
+        parser.exit(1, f"Baseline failed: {exc}\n")

ci/scripts/init-node/provisioning/bundle.py

@@ -0,0 +1,51 @@
+"""Deterministic provisioning payloads for golden-image builds."""
+import hashlib
+import io
+from pathlib import Path
+import re
+import shutil
+import subprocess
+import tarfile
+
+from .state import ProvisionError
+
+
+def _bundle_from_source(root):
+    root = Path(root).resolve()
+    package = root / 'provisioning'
+    files = sorted(package.rglob('*.py'))
+    seed = package / 'seed.sh'
+    guard = package / 'profiles.sh'
+    if not seed.is_file() or not (package / '__init__.py').is_file():
+        raise ProvisionError('Provisioning package or seed is missing')
+    files.append(seed)
+    if guard.is_file():
+        files.append(guard)
+    output = io.BytesIO()
+    with tarfile.open(fileobj=output, mode='w') as archive:
+        for path in files:
+            if (not path.is_file() or '__pycache__' in path.parts
+                    or any(part.is_symlink() for part in (path, *path.parents))):
+                raise ProvisionError('Provisioning bundle contains an unsafe source path')
+            data = path.read_bytes()
+            info = tarfile.TarInfo(path.relative_to(root).as_posix())
+            info.size, info.mode, info.mtime = len(data), (0o755 if path == seed else 0o644), 0
+            info.uid = info.gid = 0
+            archive.addfile(info, io.BytesIO(data))
+    data = output.getvalue()
+    return data, hashlib.sha256(data).hexdigest()
+
+
+def build_bundle(commons):
+    root = Path(commons).resolve()
+    data, digest = _bundle_from_source(root / 'ci/scripts/init-node')
+    revision = 'unversioned-modified'
+    if shutil.which('git'):
+        result = subprocess.run(['git', '-C', str(root), 'rev-parse', 'HEAD'], capture_output=True, text=True, timeout=30)
+        if result.returncode == 0 and re.fullmatch(r'[a-f0-9]{40,64}\n?', result.stdout):
+            revision = result.stdout.strip()
+            dirty = subprocess.run(['git', '-C', str(root), 'status', '--porcelain', '--', 'ci/scripts/init-node'], capture_output=True, text=True, timeout=30)
+            if dirty.returncode or dirty.stdout.strip():
+                revision += '-modified'
+    return data, digest, revision
+

ci/scripts/init-node/provisioning/contract.py

@@ -0,0 +1,236 @@
+"""Pure validation for the supported APEX enrollment contract."""
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass, field
+from pathlib import PurePosixPath
+from typing import Any, Dict, Mapping, Optional, Tuple
+from urllib.parse import urlsplit
+
+
+class ContractError(ValueError):
+    """A safe diagnostic that never includes a supplied value."""
+
+
+@dataclass(frozen=True)
+class Repository:
+    name: str
+    url: str
+    ref: Optional[str]
+
+
+@dataclass(frozen=True)
+class Tier:
+    number: int
+    mode: str
+    device: Optional[str]
+    pool_guid: Optional[str]
+
+
+@dataclass(frozen=True)
+class ManagedFile:
+    path: str
+    owner: str
+    mode: int
+    content: str = field(repr=False)
+
+
+@dataclass(frozen=True)
+class Enrollment:
+    hostname: str
+    repository: Repository
+    tiers: Tuple[Tier, ...]
+    authorized_keys: Tuple[str, ...] = field(repr=False)
+    files: Tuple[ManagedFile, ...] = field(repr=False)
+
+    def identity(self) -> Dict[str, Any]:
+        return {
+            'hostname': self.hostname,
+            'repository': {
+                'name': self.repository.name,
+                'url': self.repository.url,
+                'ref': self.repository.ref,
+            },
+            'tiers': [
+                {'number': t.number, 'mode': t.mode,
+                 'device': t.device, 'pool_guid': t.pool_guid}
+                for t in self.tiers
+            ],
+        }
+
+
+_ENV_KEYS = {'APEX_REPO_NAME', 'APEX_REPO_URL', 'APEX_REPO_REF'} | {
+    f'APEX_TIER{number}_{suffix}'
+    for number in (1, 2, 3)
+    for suffix in ('DEVICE', 'MODE', 'POOL_GUID')
+}
+_ASSIGNMENT = re.compile(
+    r'''\s*([A-Z][A-Z0-9_]*)\s*=\s*(?:"([^"\\]*)"|'([^'\\]*)'|([^\s'"\\()#]*))\s*'''
+)
+_FILE_POLICY = {
+    '/etc/apex/bootstrap.env': ('root:root', 0o600, False),
+    '/home/adam/.ssh/id_ed25519': ('adam:adam', 0o600, True),
+    '/home/adam/.ssh/known_hosts': ('adam:adam', 0o644, True),
+}
+
+
+def _require(condition: bool, diagnostic: str) -> None:
+    if not condition:
+        raise ContractError(diagnostic)
+
+
+def _mapping(value: Any, required: set, optional: set, diagnostic: str) -> Mapping:
+    _require(isinstance(value, Mapping), diagnostic)
+    _require(set(value) >= required and set(value) <= required | optional, diagnostic)
+    return value
+
+
+def _plain(value: Any, diagnostic: str, allow_empty: bool = False) -> str:
+    _require(type(value) is str, diagnostic)
+    _require(allow_empty or bool(value), diagnostic)
+    _require(not any(ord(c) < 32 or ord(c) == 127 for c in value), diagnostic)
+    return value
+
+
+def parse_bootstrap_env(text: str) -> Dict[str, str]:
+    _require(type(text) is str, 'bootstrap.env must contain literal assignments')
+    result = {}
+    for line in text.splitlines():
+        if not line.strip() or line.lstrip().startswith('#'):
+            continue
+        match = _ASSIGNMENT.fullmatch(line)
+        _require(match is not None, 'bootstrap.env must contain literal assignments')
+        key = match.group(1)
+        _require(key in _ENV_KEYS, 'bootstrap.env contains an unsupported assignment')
+        _require(key not in result, 'bootstrap.env contains a duplicate assignment')
+        value = next(part for part in match.groups()[1:] if part is not None)
+        _require(not any(c in value for c in '$`;|&<>'),
+                 'bootstrap.env contains unsupported shell syntax')
+        result[key] = _plain(value, 'bootstrap.env contains an invalid literal', True)
+    return result
+
+
+def _repository(values: Mapping[str, str]) -> Repository:
+    name = values.get('APEX_REPO_NAME', '')
+    _require(bool(re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9._-]*', name))
+             and name not in ('.', '..'), 'repository name must be one safe directory name')
+    url = values.get('APEX_REPO_URL', '')
+    _require(bool(url) and not any(c.isspace() for c in url), 'repository URL is invalid')
+    if '://' in url:
+        try:
+            parsed = urlsplit(url)
+            port = parsed.port
+        except ValueError:
+            raise ContractError('repository URL is invalid') from None
+        _require(parsed.scheme in ('https', 'ssh') and bool(parsed.hostname)
+                 and parsed.password is None and not parsed.query and not parsed.fragment
+                 and bool(parsed.path.strip('/')) and (port is None or port > 0),
+                 'repository URL is invalid or contains credentials')
+        _require(parsed.scheme != 'https' or parsed.username is None,
+                 'HTTPS repository URL must not contain credentials')
+    else:
+        _require(bool(re.fullmatch(
+            r'[A-Za-z0-9._-]+@[A-Za-z0-9.-]+:[A-Za-z0-9_./-]+', url)),
+            'repository URL must use HTTPS or SSH')
+        remote_path = url.split(':', 1)[1]
+        _require(not remote_path.startswith('-') and remote_path not in ('.', '..'),
+                 'repository URL is invalid')
+    ref = values.get('APEX_REPO_REF') or None
+    if ref is not None:
+        _require(not ref.startswith('-') and not any(c.isspace() for c in ref),
+                 'repository ref is invalid')
+    return Repository(name, url, ref)
+
+
+def _tiers(values: Mapping[str, str]) -> Tuple[Tier, ...]:
+    result = []
+    devices = set()
+    guids = set()
+    for number in (1, 2, 3):
+        prefix = f'APEX_TIER{number}_'
+        device = values.get(prefix + 'DEVICE') or None
+        mode = values.get(prefix + 'MODE')
+        guid = values.get(prefix + 'POOL_GUID') or None
+        _require(mode is not None or device is None,
+                 'a mapped tier device requires an explicit mode')
+        mode = mode or 'directory'
+        _require(mode in ('directory', 'create', 'import'), 'tier mode is invalid')
+        if mode == 'directory':
+            _require(device is None and guid is None,
+                     'directory tier must not select a device or pool')
+        else:
+            _require(device is not None and device.startswith('/dev/')
+                     and not any(c.isspace() for c in device)
+                     and '..' not in PurePosixPath(device).parts
+                     and str(PurePosixPath(device)) == device,
+                     'block-backed tier requires a normalized device path')
+            _require(device not in devices, 'a device is selected by multiple tiers')
+            devices.add(device)
+            if mode == 'create':
+                _require(guid is None, 'create tier must not name an existing pool')
+            else:
+                _require(guid is not None and bool(re.fullmatch(r'[1-9][0-9]{0,19}', guid))
+                         and int(guid) < 2**64, 'import tier requires a valid pool GUID')
+                _require(guid not in guids, 'a pool is selected by multiple tiers')
+                guids.add(guid)
+        result.append(Tier(number, mode, device, guid))
+    return tuple(result)
+
+
+def normalize_user_data(document: Any) -> Enrollment:
+    doc = _mapping(document,
+                   {'preserve_hostname', 'hostname', 'users', 'write_files', 'runcmd'},
+                   set(), 'unsupported cloud-config structure')
+    _require(doc['preserve_hostname'] is False, 'preserve_hostname must be false')
+    hostname = _plain(doc['hostname'], 'hostname is invalid')
+    labels = hostname.split('.')
+    _require(len(hostname) <= 253 and len(labels) >= 2
+             and all(re.fullmatch(r'[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?', x)
+                     for x in labels), 'hostname must be a valid FQDN')
+    users = doc['users']
+    _require(type(users) is list and len(users) == 1, 'exactly one administrator is required')
+    user = _mapping(users[0],
+                    {'name', 'sudo', 'groups', 'shell', 'ssh_authorized_keys'},
+                    set(), 'unsupported administrator configuration')
+    _require(user['name'] == 'adam' and user['shell'] == '/bin/bash',
+             'administrator must be adam with Bash')
+    _require(type(user['sudo']) is str and ''.join(user['sudo'].split()) in
+             ('ALL=(ALL)NOPASSWD:ALL', 'ALL=(ALL:ALL)NOPASSWD:ALL'),
+             'administrator requires passwordless sudo')
+    groups = user['groups']
+    _require(type(groups) is list and len(groups) == 2
+             and all(type(g) is str for g in groups) and set(groups) == {'sudo', 'docker'},
+             'administrator requires sudo and docker groups')
+    keys = user['ssh_authorized_keys']
+    _require(type(keys) is list and len(keys) > 0, 'administrator requires public SSH keys')
+    authorized_keys = tuple(_plain(k, 'administrator SSH key is invalid') for k in keys)
+    _require(all(k.strip() for k in authorized_keys), 'administrator SSH key is empty')
+    commands = doc['runcmd']
+    _require(commands in (['/usr/local/bin/apex-bootstrap'],
+                          [['/usr/local/bin/apex-bootstrap']]),
+             'only the standard APEX bootstrap command is supported')
+    writes = doc['write_files']
+    _require(type(writes) is list, 'write_files must be a list')
+    files = []
+    seen = set()
+    for write in writes:
+        item = _mapping(write, {'path', 'owner', 'permissions', 'content'}, {'defer'},
+                        'unsupported write_files entry')
+        path = _plain(item['path'], 'managed file path is invalid')
+        _require(path in _FILE_POLICY and path not in seen,
+                 'managed file path is unsupported or repeated')
+        seen.add(path)
+        owner, mode, deferred = _FILE_POLICY[path]
+        permission = item['permissions']
+        _require(type(permission) is str and bool(re.fullmatch(r'0?[0-7]{3}', permission))
+                 and int(permission, 8) == mode and item['owner'] == owner,
+                 'managed file ownership or permissions are invalid')
+        _require(item.get('defer', False) is deferred, 'managed file defer setting is invalid')
+        content = item['content']
+        _require(type(content) is str and bool(content.strip()) and '\x00' not in content,
+                 'managed file content is invalid')
+        files.append(ManagedFile(path, owner, mode, content))
+    bootstrap = next((f for f in files if f.path == '/etc/apex/bootstrap.env'), None)
+    _require(bootstrap is not None, 'bootstrap.env is required')
+    values = parse_bootstrap_env(bootstrap.content)
+    return Enrollment(hostname, _repository(values), _tiers(values), authorized_keys, tuple(files))

ci/scripts/init-node/provisioning/image.py

@@ -0,0 +1,124 @@
+"""Image-only installation and sanitization; for golden-image builds."""
+from __future__ import annotations
+
+import argparse
+import io
+import json
+import os
+from pathlib import Path
+import shutil
+import tarfile
+import uuid
+
+from . import ENROLLMENT_INTERFACE
+from .state import ProvisionError, atomic_write, exclusive_lock, safe_path
+from .target import preflight, run
+
+
+def install():
+    from .bundle import _bundle_from_source
+    source = Path(__file__).resolve().parents[1]
+    data, digest = _bundle_from_source(source)
+    destination = safe_path(Path('/usr/local/lib/apex-provisioner') / digest)
+    destination.mkdir(mode=0o755, parents=True, exist_ok=False)
+    with tarfile.open(fileobj=io.BytesIO(data)) as archive:
+        archive.extractall(destination, filter='data')
+    revision = os.environ.get('APEX_PROVISION_REVISION', 'unversioned-modified')
+    metadata = {'bundle_digest': digest, 'revision': revision, 'path': str(destination),
+                'baseline_revision': revision, 'enrollment_interface': ENROLLMENT_INTERFACE}
+    atomic_write('/etc/apex/provisioner.json', json.dumps(metadata) + '\n')
+    for executable, operation in (('apex-provision-image', ''), ('apex-bootstrap', 'cloud-enroll ')):
+        content = ('#!/bin/sh\nset -eu\ncd ' + str(destination) + '\n'
+                   'exec /usr/bin/python3 -m provisioning.image ' + operation + '"$@"\n')
+        atomic_write('/usr/local/bin/' + executable, content, 0o755)
+    return metadata
+
+
+def cloud_enroll():
+    from .access import Transition
+    from .target import enroll
+    from .userdata import decode_user_data
+    metadata = json.loads(Path('/etc/apex/provisioner.json').read_text())
+    if metadata.get('enrollment_interface') != ENROLLMENT_INTERFACE:
+        raise ProvisionError('image enrollment interface is missing or unsupported; rebuild the image')
+    instance = Path('/var/lib/cloud/instance').resolve(strict=True)
+    try:
+        instance.relative_to('/var/lib/cloud/instances')
+    except ValueError:
+        raise RuntimeError('cloud-init instance path is outside its managed directory') from None
+    path = safe_path(instance / 'user-data.txt')
+    if path.stat().st_uid != 0 or path.stat().st_size > 1024 * 1024:
+        raise RuntimeError('cloud-init user-data is not a supported root-owned file')
+    spec = decode_user_data(path.read_bytes())
+    result = enroll(spec, metadata['revision'], metadata['bundle_digest'])
+    token = uuid.uuid4().hex
+    transition = Transition()
+    try:
+        current = transition.execute('begin', token, 22, '127.0.0.1')
+        run(['runuser', '-u', 'adam', '--', 'sudo', '-n', 'true'])
+        if current['status'] not in ('existing', 'committed'):
+            transition.execute('finalize', token)
+        if current['status'] != 'committed':
+            transition.execute('commit', token)
+    except Exception:
+        transition.execute('rollback', token)
+        raise
+    result['external_access'] = 'not-verified-cloud-init'
+    atomic_write('/var/lib/apex-init/enrollment.json', json.dumps(result, sort_keys=True) + '\n')
+    return result
+
+
+def sanitize_files(root=Path('/')):
+    root = Path(root)
+    for home in ('root', 'home/debian'):
+        for name in ('.ssh/authorized_keys', '.ssh/authorized_keys2', '.bash_history'):
+            path = root / home / name
+            if path.exists() or path.is_symlink():
+                path.unlink()
+    for path in (root / 'etc/ssh').glob('ssh_host_*'):
+        if path.is_file() or path.is_symlink():
+            path.unlink()
+    atomic_write(root / 'etc/machine-id', '', 0o444)
+    dbus = root / 'var/lib/dbus/machine-id'
+    if dbus.exists() or dbus.is_symlink():
+        dbus.unlink()
+    dbus.parent.mkdir(parents=True, exist_ok=True)
+    dbus.symlink_to('/etc/machine-id')
+    for name in ('var/lib/apex-init', 'tmp/provisioning', 'var/lib/cloud'):
+        path = root / name
+        if path.is_symlink():
+            path.unlink()
+        elif path.exists():
+            shutil.rmtree(path)
+    autologin = root / 'etc/systemd/system/serial-getty@ttyS0.service.d/autologin.conf'
+    autologin.unlink(missing_ok=True)
+    atomic_write(root / 'etc/systemd/system/ssh.service.d/10-generate-host-keys.conf',
+                 '[Service]\nExecStartPre=\nExecStartPre=/usr/bin/ssh-keygen -A\n'
+                 'ExecStartPre=/usr/sbin/sshd -t\n', 0o644)
+
+
+def main(argv=None):
+    parser = argparse.ArgumentParser()
+    parser.add_argument('operation', choices=('install', 'baseline', 'install-runner', 'sanitize', 'cloud-enroll'))
+    args = parser.parse_args(argv)
+    preflight()
+    if args.operation == 'install':
+        print(json.dumps(install()))
+    elif args.operation == 'baseline':
+        from .baseline import apply_baseline
+        with exclusive_lock('/var/lib/apex-init/init.lock'):
+            result = apply_baseline()
+            atomic_write('/etc/apex/image-baseline.json', json.dumps(result, sort_keys=True) + '\n')
+    elif args.operation == 'install-runner':
+        run(['test', '-x', '/usr/local/bin/apex-bootstrap'])
+    elif args.operation == 'sanitize':
+        run(['usermod', '--password', '!', 'root'])
+        run(['usermod', '--password', '!', 'debian'])
+        sanitize_files()
+    else:
+        print(json.dumps(cloud_enroll(), sort_keys=True))
+    return 0
+
+
+if __name__ == '__main__':
+    raise SystemExit(main())

ci/scripts/init-node/provisioning/repository.py

@@ -0,0 +1,121 @@
+"""Pinned, non-destructive checkout enrollment as the node administrator."""
+from __future__ import annotations
+
+import os
+from pathlib import Path
+import pwd
+import subprocess
+
+from engine.identity import read_env
+from . import ENROLLMENT_INTERFACE
+from .state import ProvisionError, safe_path
+
+
+def as_adam(args, cwd=None):
+    environment = dict(os.environ, GIT_TERMINAL_PROMPT='0',
+                       GIT_SSH_COMMAND='ssh -o BatchMode=yes -o StrictHostKeyChecking=yes',
+                       LC_ALL='C')
+    result = subprocess.run(['runuser', '-u', 'adam', '--', *args], cwd=cwd,
+                            env=environment, text=True, capture_output=True, timeout=300)
+    if result.returncode:
+        # Git/hook stderr may contain credential files or remote URL credentials.
+        raise ProvisionError('repository operation failed; check administrator Git access and the pinned checkout')
+    return result.stdout.strip()
+
+
+def verify_checkout_setup(destination, account):
+    if as_adam(['git', 'config', '--get', 'core.hooksPath'], destination) != 'commons/githooks':
+        raise ProvisionError('checkout hook configuration is missing or incompatible')
+    home = safe_path(account.pw_dir)
+    bashrc = safe_path(home / '.bashrc')
+    entry = 'export PATH="' + str(destination / 'commons') + ':$PATH"'
+    if not bashrc.is_file() or bashrc.read_text().splitlines().count(entry) != 1:
+        raise ProvisionError('checkout PATH setup is missing or duplicated; reconcile init.sh before retrying')
+    vim = home / '.vimrc'
+    if not vim.is_symlink() or vim.resolve() != (destination / 'commons/etc/.vimrc').resolve():
+        raise ProvisionError('checkout editor link is missing or incompatible')
+
+
+def reconcile_repository(spec, state, save, srv=Path('/srv')):
+    repository = spec.repository
+    destination = safe_path(Path(srv) / repository.name)
+    staging = safe_path(Path(srv) / ('.apex-clone-' + repository.name))
+    expected = {'name': repository.name, 'url': repository.url, 'ref': repository.ref}
+    account = pwd.getpwnam('adam')
+    record = state.setdefault('repository', {})
+    if record and record.get('request') != expected:
+        raise ProvisionError('repository enrollment conflicts with its recorded request')
+    if not record:
+        if destination.exists() or staging.exists():
+            raise ProvisionError('unrecorded repository path exists; explicit adoption is required')
+        record.update(request=expected, clone_intent=True)
+        save()
+    if not destination.exists():
+        if staging.exists():
+            # An interrupted clone may be incomplete: never erase or guess its state.
+            raise ProvisionError('an interrupted clone remains; inspect the staging checkout before retrying')
+        staging.mkdir(mode=0o755)
+        os.chown(staging, account.pw_uid, account.pw_gid)
+        as_adam(['git', 'clone', '--no-checkout', '--', repository.url, str(staging)])
+        os.rename(staging, destination)
+    safe_path(destination / '.git')
+    origin = as_adam(['git', 'remote', 'get-url', 'origin'], destination)
+    if origin != repository.url:
+        raise ProvisionError('existing checkout has a different origin; it was preserved')
+    if 'commit' not in record:
+        reference = repository.ref or 'HEAD'
+        if repository.ref:
+            as_adam(['git', 'fetch', '--no-tags', 'origin', reference], destination)
+            reference = 'FETCH_HEAD'
+        commit = as_adam(['git', 'rev-parse', '--verify', reference + '^{commit}'], destination)
+        record['commit'] = commit
+        record['checkout_intent'] = True
+        save()
+    commit = record['commit']
+    if record.get('checkout_intent'):
+        # --no-checkout index intentionally looks dirty before the first checkout.
+        if any(p.name != '.git' for p in destination.iterdir()):
+            head = as_adam(['git', 'rev-parse', 'HEAD'], destination)
+            dirty = as_adam(['git', 'status', '--porcelain'], destination)
+            if head != commit or dirty:
+                raise ProvisionError('interrupted checkout has unexpected contents; it was preserved')
+        else:
+            as_adam(['git', 'checkout', '--detach', commit], destination)
+        record.pop('checkout_intent')
+        save()
+    if as_adam(['git', 'rev-parse', 'HEAD'], destination) != commit:
+        raise ProvisionError('checkout revision changed; initialization will not reset it')
+    if as_adam(['git', 'status', '--porcelain'], destination):
+        raise ProvisionError('checkout contains local changes; initialization will not reset it')
+    node_env = safe_path(destination / 'node.env')
+    node_identity = read_env(str(node_env))
+    if node_identity.get('APEX_NODE_FQDN') != spec.hostname:
+        raise ProvisionError('repository node.env does not declare the requested hostname')
+    if node_identity.get('APEX_ENROLLMENT_INTERFACE') != str(ENROLLMENT_INTERFACE):
+        raise ProvisionError('node checkout enrollment interface is missing or unsupported; update its init.sh and node.env together')
+    as_adam(['git', 'submodule', 'update', '--init', '--recursive'], destination)
+    commons = safe_path(destination / 'commons')
+    if not commons.is_dir():
+        raise ProvisionError('node checkout does not contain its pinned commons submodule')
+    commons_commit = as_adam(['git', 'rev-parse', 'HEAD'], commons)
+    gitlink = as_adam(['git', 'ls-tree', 'HEAD', '--', 'commons'], destination).split()
+    if gitlink != ['160000', 'commit', commons_commit, 'commons']:
+        raise ProvisionError('commons is not the exact pinned Git submodule')
+    if Path(as_adam(['git', 'rev-parse', '--show-toplevel'], commons)).resolve() != commons.resolve():
+        raise ProvisionError('commons does not have its own Git worktree')
+    if record.get('commons_commit', commons_commit) != commons_commit:
+        raise ProvisionError('pinned commons revision changed unexpectedly')
+    hook = safe_path(destination / 'init.sh')
+    if not hook.is_file():
+        raise ProvisionError('node repository is missing its checkout initializer')
+    if not record.get('hook_complete'):
+        as_adam(['bash', './init.sh'], destination)
+        verify_checkout_setup(destination, account)
+        record['hook_complete'] = True
+    else:
+        verify_checkout_setup(destination, account)
+    if as_adam(['git', 'status', '--porcelain'], destination):
+        raise ProvisionError('checkout initializer changed tracked files; inspect before resuming')
+    record['commons_commit'] = commons_commit
+    save()
+    return {'path': str(destination), 'commit': commit, 'commons_commit': commons_commit}

ci/scripts/init-node/provisioning/security.py

@@ -0,0 +1,591 @@
+"""Security activation owned by the protected access transaction."""
+from __future__ import annotations
+
+import base64
+import hashlib
+import ipaddress
+import json
+import os
+import re
+import shlex
+import tempfile
+import uuid
+
+from .access import atomic
+from .baseline import HELPER_SHA256
+
+
+SERVICES = ('ufw', 'crowdsec', 'crowdsec-firewall-bouncer')
+FILES = ('/etc/default/ufw', '/etc/ufw/ufw.conf', '/etc/ufw/before.rules',
+         '/etc/ufw/before6.rules', '/etc/ufw/after.rules', '/etc/ufw/after6.rules',
+         '/etc/ufw/user.rules', '/etc/ufw/user6.rules')
+HELPER = '/usr/local/bin/ufw-docker'
+BLOCK = re.compile(r'^# BEGIN UFW AND DOCKER\n.*?^# END UFW AND DOCKER\n?', re.M | re.S)
+
+
+def _helper_lines(ipv6):
+    prefix = 'ufw6' if ipv6 else 'ufw'
+    subnets = ('fd00::/8',) if ipv6 else ('10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16')
+    return ['# BEGIN UFW AND DOCKER', '*filter', ':' + prefix + '-user-forward - [0:0]',
+            ':' + prefix + '-docker-logging-deny - [0:0]', ':DOCKER-USER - [0:0]',
+            '-A DOCKER-USER -j ' + prefix + '-user-forward',
+            '-A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j RETURN',
+            '-A DOCKER-USER -m conntrack --ctstate INVALID -j DROP',
+            '-A DOCKER-USER -i docker0 -o docker0 -j ACCEPT',
+            *['-A DOCKER-USER -j RETURN -s ' + subnet for subnet in subnets],
+            *['-A DOCKER-USER -j ' + prefix + '-docker-logging-deny -m conntrack --ctstate NEW -d ' + subnet
+              for subnet in subnets],
+            '-A DOCKER-USER -j RETURN',
+            '-A ' + prefix + '-docker-logging-deny -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW DOCKER BLOCK] "',
+            '-A ' + prefix + '-docker-logging-deny -j DROP', 'COMMIT', '# END UFW AND DOCKER']
+
+
+def _runtime_rule(line):
+    tokens = shlex.split(line)
+    if len(tokens) % 2:
+        raise SecurityError('Unsupported runtime firewall rule')
+    pairs = []
+    for key, value in zip(tokens[::2], tokens[1::2], strict=True):
+        if key in ('-s', '-d'):
+            value = str(ipaddress.ip_network(value, strict=False))
+        elif key == '--ctstate':
+            value = ','.join(sorted(value.split(',')))
+        pairs.append((key, value))
+    return sorted(pairs)
+
+
+class SecurityError(RuntimeError):
+    pass
+
+
+class Security:
+    def __init__(self, host):
+        self.host = host
+
+    def preflight(self):
+        services = {name: self._service(name) for name in SERVICES}
+        for name, state in services.items():
+            if (state.get('LoadState') != 'loaded' or state.get('UnitFileState') not in ('enabled', 'disabled')
+                    or state.get('ActiveState') not in ('active', 'inactive')
+                    or state.get('DropInPaths') or not self._supported_unit(name, state.get('FragmentPath'))):
+                raise SecurityError('Unsupported or masked security service: ' + name)
+        if self._digest(self._path(HELPER).read_bytes()) != HELPER_SHA256:
+            raise SecurityError('The ufw-docker helper does not match the pinned baseline')
+        files = self._files()
+        active = self._active()
+        pristine = False
+        if not active:
+            output = self.host.command(['dpkg-query', '-W', '-f=${Conffiles}', 'ufw']).stdout
+            conffiles = dict(re.findall(r'^\s*(/\S+)\s+([a-f0-9]{32})(?:\s|$)', output, re.M))
+            pristine = (hashlib.md5(base64.b64decode(files['/etc/default/ufw']['data']), usedforsecurity=False).hexdigest()
+                        == conffiles.get('/etc/default/ufw'))
+            manifest = dict((name, digest) for digest, name in re.findall(
+                r'^([a-f0-9]{32})\s+(\S+)\s*$', self._text('/var/lib/dpkg/info/ufw.md5sums'), re.M))
+            # Debian's postinst copies/ucf-installs these files; they are not dpkg conffiles.
+            for name in FILES[1:]:
+                filename = name.rsplit('/', 1)[1]
+                template = '/usr/share/ufw/' + ('' if filename == 'ufw.conf' else 'iptables/') + filename
+                data = self._path(template).read_bytes()
+                pristine = pristine and (hashlib.md5(data, usedforsecurity=False).hexdigest() == manifest.get(template.lstrip('/'))
+                                         and data == base64.b64decode(files[name]['data']))
+            if not pristine:
+                raise SecurityError('Inactive UFW is not provably pristine; reconcile its existing policy first')
+        texts = [self._text('/etc/ufw/' + name) for name in ('after.rules', 'after6.rules')]
+        present = any('UFW AND DOCKER' in text for text in texts)
+        if present:
+            self._check_helper_files()
+            runtime = None
+        else:
+            runtime = {executable: self._helper_runtime(executable) for executable in ('iptables', 'ip6tables')}
+            for chains in runtime.values():
+                self._validate_helper_before(chains)
+        return {'version': 1, 'pristine': pristine, 'active': active, 'services': services,
+                'files': files, 'helper_present': present, 'helper_runtime_before': runtime,
+                'ipv6': bool(re.search(r'^IPV6\s*=\s*[\"\x27]?yes', self._text('/etc/default/ufw'), re.M))}
+
+    def _supported_unit(self, name, path):
+        if path in ('/usr/lib/systemd/system/' + name + '.service', '/lib/systemd/system/' + name + '.service'):
+            return True
+        if name != 'crowdsec-firewall-bouncer' or path != '/etc/systemd/system/crowdsec-firewall-bouncer.service':
+            return False
+        output = self.host.command(['dpkg-query', '-W', '-f=${Conffiles}',
+                                    'crowdsec-firewall-bouncer-iptables']).stdout
+        conffiles = dict(re.findall(r'^\s*(/\S+)\s+([a-f0-9]{32})(?:\s|$)', output, re.M))
+        return hashlib.md5(self._path(path).read_bytes(), usedforsecurity=False).hexdigest() == conffiles.get(path)
+
+    def _path(self, name):
+        path = self.host.path(name)
+        for parent in (path, *path.parents):
+            if parent.is_symlink():
+                raise SecurityError('Refusing redirected security path: ' + name)
+        if not path.is_file():
+            raise SecurityError('Required security file is missing: ' + name)
+        return path
+
+    def _text(self, name):
+        return self._path(name).read_text()
+
+    @staticmethod
+    def _digest(data):
+        return hashlib.sha256(data).hexdigest()
+
+    def _files(self):
+        return {name: self._file(name) for name in FILES}
+
+    def _file(self, name):
+        path = self._path(name)
+        data, metadata = path.read_bytes(), path.stat()
+        result = {'data': base64.b64encode(data).decode(), 'digest': self._digest(data),
+                  'mode': metadata.st_mode & 0o777, 'uid': metadata.st_uid, 'gid': metadata.st_gid}
+        result['fingerprint'] = self._digest(json.dumps(result, sort_keys=True).encode())
+        return result
+
+    def _service(self, name):
+        output = self.host.command(['systemctl', 'show', name, '-p',
+                                   'LoadState,ActiveState,UnitFileState,FragmentPath,DropInPaths']).stdout
+        return dict(line.split('=', 1) for line in output.splitlines() if '=' in line)
+
+    def _active(self):
+        output = self.host.command(['ufw', 'status']).stdout
+        if 'Status: inactive' in output:
+            return False
+        if 'Status: active' in output:
+            return True
+        raise SecurityError('Cannot determine UFW runtime state')
+
+    def snapshot(self, plan, original_port, client_address):
+        if not isinstance(original_port, int) or not 1 <= original_port <= 65535:
+            raise SecurityError('Invalid bootstrap port')
+        try:
+            address = str(ipaddress.ip_address(client_address))
+        except ValueError:
+            raise SecurityError('Invalid bootstrap peer address') from None
+        if plan != self.preflight():
+            raise SecurityError('Security configuration changed after preflight')
+        if ipaddress.ip_address(address).version == 6 and not plan['ipv6']:
+            raise SecurityError('IPv6 bootstrap access requires enabled UFW IPv6 support')
+        if plan['services']['crowdsec']['ActiveState'] == 'active':
+            self._check_decisions(address)
+        return {'version': 1, 'plan': plan, 'original_port': original_port,
+                'client_address': address, 'token': uuid.uuid4().hex, 'rules': [],
+                'expected': {name: value['fingerprint'] for name, value in plan['files'].items()},
+                'service_changes': [], 'inflight': None, 'phase': 'prepared',
+                'access_recovered': False, 'cleanup': 'not-started', 'helper_blocks': {},
+                'helper_runtime': {'introduced': False, 'restored': []}}
+
+    def activate(self, state, persist):
+        if state['phase'] != 'prepared' or state['inflight']:
+            raise SecurityError('Security activation requires a fresh prepared snapshot')
+        self._assert_files(state)
+        state['phase'] = 'activating'
+        persist(state)
+        self._add_rule(state, state['original_port'], state['client_address'], True, persist)
+        self._add_rule(state, 2222, '0.0.0.0/0', False, persist)
+        if state['plan']['ipv6']:
+            self._add_rule(state, 2222, '::/0', False, persist)
+        if state['plan']['pristine']:
+            for policy, direction in (('deny', 'incoming'), ('allow', 'outgoing'), ('allow', 'routed')):
+                self._mutate(state, ['ufw', 'default', policy, direction], ['/etc/default/ufw'], persist)
+        self._mutate(state, ['ufw', '--force', 'enable'],
+                     ['/etc/ufw/ufw.conf', '/etc/ufw/user.rules', '/etc/ufw/user6.rules'], persist)
+        if not state['plan']['helper_present']:
+            if any(self._helper_runtime(executable) != before for executable, before in
+                   state['plan']['helper_runtime_before'].items()):
+                raise SecurityError('Docker firewall rules changed before helper installation')
+            state['helper_runtime']['introduced'] = True
+            persist(state)
+            names = ['/etc/ufw/after.rules', '/etc/ufw/after6.rules']
+            self._mutate(state, [HELPER, 'install'], names, persist)
+            for name in names:
+                blocks = BLOCK.findall(self._text(name))
+                if len(blocks) != 1:
+                    raise SecurityError('ufw-docker did not install a verifiable integration block')
+                state['helper_blocks'][name] = blocks[0]
+            persist(state)
+        self._check_helper_files()
+        self._mutate(state, ['ufw', 'reload'], [], persist)
+        self._enable(state, 'ufw', persist)
+        self.host.command(['crowdsec', '-t', '-c', '/etc/crowdsec/config.yaml'])
+        self._enable(state, 'crowdsec', persist)
+        self.host.command(['cscli', 'lapi', 'status'])
+        self._check_decisions(state['client_address'])
+        self._enable(state, 'crowdsec-firewall-bouncer', persist)
+        state['phase'] = 'active'
+        persist(state)
+        self.verify(state)
+
+    def finalize(self, state, persist):
+        self.verify(state)
+        for rule in state['rules']:
+            if rule['temporary'] and rule['owned']:
+                self._delete_rule(state, rule, persist)
+        state['phase'] = 'final'
+        persist(state)
+        self.verify(state)
+
+    def verify(self, state):
+        if state['phase'] not in ('active', 'final') or state['inflight']:
+            raise SecurityError('Security activation is incomplete')
+        self._assert_files(state)
+        if not self._active():
+            raise SecurityError('UFW is not active')
+        for name in SERVICES:
+            service = self._service(name)
+            if service.get('ActiveState') != 'active' or service.get('UnitFileState') != 'enabled':
+                raise SecurityError('Security service is not active and enabled: ' + name)
+        self.host.command(['cscli', 'lapi', 'status'])
+        self._check_decisions(state['client_address'])
+        self._check_helper_files()
+        self._check_helper_runtime(state['plan']['ipv6'])
+        for rule in state['rules']:
+            if not (rule['temporary'] and state['phase'] == 'final') and self._rule_comment(rule) is None:
+                raise SecurityError('Required SSH firewall allowance is missing')
+
+    def _check_helper_files(self):
+        for ipv6, name in ((False, 'after.rules'), (True, 'after6.rules')):
+            text = self._text('/etc/ufw/' + name)
+            blocks = BLOCK.findall(text)
+            if (len(blocks) != 1 or text.count('UFW AND DOCKER') != 2
+                    or [line.strip() for line in blocks[0].splitlines() if line.strip()] != _helper_lines(ipv6)):
+                raise SecurityError('Existing ufw-docker integration is custom or incomplete')
+
+    def _check_helper_runtime(self, ipv6):
+        for version in (False, True) if ipv6 else (False,):
+            executable = 'ip6tables' if version else 'iptables'
+            prefix = 'ufw6' if version else 'ufw'
+            for chain in ('DOCKER-USER', prefix + '-docker-logging-deny'):
+                result = self.host.command([executable, '-S', chain], False)
+                expected = ['-N ' + chain] + [line for line in _helper_lines(version)
+                                              if line.startswith('-A ' + chain + ' ')]
+                try:
+                    matches = (not result.returncode and [_runtime_rule(line) for line in result.stdout.splitlines()]
+                               == [_runtime_rule(line) for line in expected])
+                except ValueError:
+                    matches = False
+                if not matches:
+                    raise SecurityError('ufw-docker runtime chain is missing or changed: ' + executable + ' ' + chain)
+            if self.host.command([executable, '-C', 'FORWARD', '-j', 'DOCKER-USER'], False).returncode:
+                raise SecurityError('Docker forwarding hook is missing: ' + executable)
+
+    def _helper_runtime(self, executable):
+        logging = ('ufw6' if executable == 'ip6tables' else 'ufw') + '-docker-logging-deny'
+        chains = {'DOCKER-USER': None, logging: None}
+        for line in self.host.command([executable, '-S']).stdout.splitlines():
+            tokens = shlex.split(line)
+            if len(tokens) < 2 or tokens[1] not in chains:
+                continue
+            if tokens[0] == '-N' and len(tokens) == 2 and chains[tokens[1]] is None:
+                chains[tokens[1]] = [line]
+            elif tokens[0] == '-A' and chains[tokens[1]] is not None:
+                chains[tokens[1]].append(line)
+            else:
+                raise SecurityError('Unsupported Docker firewall chain declaration')
+        return chains
+
+    @staticmethod
+    def _validate_helper_before(chains):
+        if (not isinstance(chains, dict) or len(chains) != 2 or 'DOCKER-USER' not in chains
+                or chains['DOCKER-USER'] not in (None, ['-N DOCKER-USER'],
+                                                ['-N DOCKER-USER', '-A DOCKER-USER -j RETURN'])
+                or any(value is not None for name, value in chains.items() if name != 'DOCKER-USER')):
+            raise SecurityError('Unmanaged Docker firewall rules require reconciliation')
+
+    def _restore_helper_runtime(self, state, persist):
+        if state['plan']['helper_present']:
+            return
+        before = state['plan'].get('helper_runtime_before')
+        progress = state.get('helper_runtime')
+        if (not isinstance(before, dict) or set(before) != {'iptables', 'ip6tables'}
+                or not isinstance(progress, dict) or not isinstance(progress.get('introduced'), bool)
+                or not isinstance(progress.get('restored'), list)):
+            raise SecurityError('Missing Docker firewall recovery snapshot; reconcile explicitly')
+        for executable, original in before.items():
+            self._validate_helper_before(original)
+            current = self._helper_runtime(executable)
+            if set(original) != set(current):
+                raise SecurityError('Unsupported Docker firewall recovery snapshot')
+            if current != original:
+                ipv6 = executable == 'ip6tables'
+                expected = {name: ['-N ' + name] + [line for line in _helper_lines(ipv6)
+                            if line.startswith('-A ' + name + ' ')] for name in current}
+                try:
+                    owned = (progress['introduced'] and (not ipv6 or state['plan']['ipv6'])
+                             and all(current[name] is not None and
+                                     [_runtime_rule(line) for line in current[name]] ==
+                                     [_runtime_rule(line) for line in expected[name]] for name in current))
+                except (ValueError, SecurityError):
+                    owned = False
+                if not owned:
+                    raise SecurityError('Docker firewall runtime changed; preserve it for reconciliation')
+                logging = ('ufw6' if ipv6 else 'ufw') + '-docker-logging-deny'
+                commands = ['*filter', '-F DOCKER-USER', *(original['DOCKER-USER'] or [])[1:],
+                            '-F ' + logging, '-X ' + logging]
+                if original['DOCKER-USER'] is None:
+                    commands.append('-X DOCKER-USER')
+                commands.append('COMMIT')
+                progress['restoring'] = executable
+                persist(state)
+                with tempfile.NamedTemporaryFile(mode='w', prefix='apex-firewall-', suffix='.rules') as stream:
+                    stream.write('\n'.join(commands) + '\n')
+                    stream.flush()
+                    if self._helper_runtime(executable) != current:
+                        raise SecurityError('Docker firewall runtime changed immediately before restoration')
+                    self.host.command([executable + '-restore', '--wait', '10', '--noflush', stream.name])
+                if self._helper_runtime(executable) != original:
+                    raise SecurityError('Docker firewall restoration did not converge')
+            if executable not in progress['restored']:
+                progress['restored'].append(executable)
+            progress.pop('restoring', None)
+            persist(state)
+
+    def restore(self, state, persist):
+        state['cleanup'] = 'pending'
+        state['access_recovered'] = False
+        persist(state)
+        errors = []
+        for change in reversed(state['service_changes']):
+            name, field = change['service'], change['field']
+            if name == 'ufw' and field == 'ActiveState' and state['plan']['active']:
+                continue
+            try:
+                current = self._service(name)
+                if any(current.get(key) != change['metadata'].get(key) for key in change['metadata']):
+                    raise SecurityError('Service unit changed during activation')
+                owned_start = (field == 'ActiveState' and change['before'] == 'inactive'
+                               and change['after'] == 'active' and current[field] in
+                               ('active', 'inactive', 'activating', 'deactivating', 'failed', 'reloading'))
+                # Stop even when inactive: systemd may still have a queued start job.
+                if current[field] == change['after'] or owned_start:
+                    actions = {'active': 'start', 'inactive': 'stop', 'enabled': 'enable', 'disabled': 'disable'}
+                    change['status'] = 'restoring'
+                    persist(state)
+                    self.host.command(['systemctl', actions[change['before']], name])
+                    stopped = self._service(name)
+                    if any(stopped.get(key) != change['metadata'].get(key) for key in change['metadata']):
+                        raise SecurityError('Service unit changed during restoration')
+                    if owned_start and stopped[field] == 'failed':
+                        self.host.command(['systemctl', 'reset-failed', name])
+                elif current[field] != change['before']:
+                    raise SecurityError('Service state changed outside this transition')
+                if self._service(name)[field] != change['before']:
+                    raise SecurityError('Service state restoration did not converge')
+                change['status'] = 'restored'
+                persist(state)
+            except Exception:
+                errors.append('service:' + name + ':' + field)
+        ambiguous = []
+        for name in FILES:
+            try:
+                if self._file(name)['fingerprint'] not in (state['plan']['files'][name]['fingerprint'], state['expected'][name]):
+                    ambiguous.append(name)
+            except Exception:
+                ambiguous.append(name)
+        pending = bool(ambiguous or state['inflight'] or errors)
+        original_active = state['plan']['active']
+        # Recover runtime access even when an unrelated edit prevents file cleanup.
+        try:
+            if not original_active:
+                try:
+                    self._path('/etc/ufw/ufw.conf')
+                except SecurityError:
+                    # A service stop removes runtime rules without following the replaced enable file.
+                    self.host.command(['systemctl', 'stop', 'ufw'])
+                else:
+                    self.host.command(['ufw', 'disable'])
+                state['access_recovered'] = not self._active()
+            elif pending:
+                self._retain_bootstrap(state, persist)
+                if not self._active():
+                    self.host.command(['ufw', '--force', 'enable'])
+                state['access_recovered'] = self._active()
+        except Exception:
+            errors.append('bootstrap-firewall')
+        for name, previous in state['plan']['files'].items():
+            try:
+                path = self._path(name)
+                content = path.read_bytes()
+                fingerprint = self._file(name)['fingerprint']
+                if fingerprint == previous['fingerprint']:
+                    continue
+                if fingerprint == state['expected'][name] and not (original_active and pending and name.endswith(('user.rules', 'user6.rules'))):
+                    atomic(path, base64.b64decode(previous['data']), previous['mode'])
+                    os.chown(path, previous['uid'], previous['gid'])
+                    continue
+                block = state['helper_blocks'].get(name)
+                if block and content.decode().count(block) == 1:
+                    metadata = path.stat()
+                    atomic(path, content.replace(block.encode(), b'', 1), metadata.st_mode & 0o777)
+                    os.chown(path, metadata.st_uid, metadata.st_gid)
+                else:
+                    errors.append('file:' + name)
+            except Exception:
+                errors.append('file:' + name)
+        # Rule removal is ownership-checked independently of whole-file snapshots.
+        for rule in state['rules']:
+            if rule['owned'] and not (original_active and pending and rule['temporary']):
+                try:
+                    if self._rule_comment(rule) == rule['comment']:
+                        self.host.command(self._rule_command(rule, delete=True))
+                    elif self._rule_comment(rule) is not None:
+                        errors.append('rule-ownership')
+                except Exception:
+                    errors.append('rule-cleanup')
+        try:
+            self._restore_helper_runtime(state, persist)
+        except Exception:
+            errors.append('docker-firewall-runtime')
+        try:
+            if original_active:
+                if errors or pending:
+                    self._retain_bootstrap(state, persist)
+                else:
+                    self.host.command(['ufw', 'reload'])
+                state['access_recovered'] = self._active()
+            else:
+                # Restoring the former ufw.conf must not reactivate the runtime.
+                state['access_recovered'] = not self._active()
+        except Exception:
+            errors.append('bootstrap-firewall')
+        if not state['access_recovered']:
+            errors.append('bootstrap-firewall')
+        if not state['plan']['helper_present']:
+            try:
+                if any(self._helper_runtime(executable) != before for executable, before in
+                       state['plan']['helper_runtime_before'].items()):
+                    raise SecurityError('Docker firewall restoration changed before cleanup completed')
+            except Exception:
+                errors.append('docker-firewall-runtime')
+        for change in state['service_changes']:
+            name, field = change['service'], change['field']
+            if name == 'ufw' and field == 'ActiveState' and original_active:
+                continue
+            try:
+                current = self._service(name)
+                if (current[field] != change['before'] or any(current.get(key) != value
+                                                            for key, value in change['metadata'].items())):
+                    raise SecurityError('Service restoration changed before cleanup completed')
+            except Exception:
+                errors.append('service:' + name + ':' + field)
+        if errors or pending:
+            state['diagnostics'] = sorted(set(errors + ['file:' + name for name in ambiguous]
+                                              + (['interrupted-mutation'] if state['inflight'] else [])))
+            persist(state)
+            raise SecurityError('Security cleanup pending; bootstrap firewall recovery=' + str(state['access_recovered'])
+                                + '; inspect the root-owned recovery record')
+        state.update(cleanup='complete', phase='restored', inflight=None, diagnostics=[])
+        persist(state)
+
+    def _assert_files(self, state):
+        if any(value['fingerprint'] != state['expected'][name] for name, value in self._files().items()):
+            raise SecurityError('Security files changed outside this transition; recovery remains required')
+
+    def _mutate(self, state, args, changed, persist):
+        self._assert_files(state)
+        state['inflight'] = {'command': args[:2], 'files': changed}
+        persist(state)
+        self.host.command(args)
+        current = self._files()
+        for name in changed:
+            state['expected'][name] = current[name]['fingerprint']
+        if any(value['fingerprint'] != state['expected'][name] for name, value in current.items()):
+            raise SecurityError('Concurrent security file change during activation')
+        state['inflight'] = None
+        persist(state)
+
+    def _enable(self, state, name, persist):
+        for field, before, after, operation in (
+                ('UnitFileState', 'disabled', 'enabled', 'enable'),
+                ('ActiveState', 'inactive', 'active', 'start')):
+            current = self._service(name)
+            if current[field] == after:
+                continue
+            if current[field] != before:
+                raise SecurityError('Security service changed before activation: ' + name)
+            metadata = {key: current.get(key) for key in ('LoadState', 'FragmentPath', 'DropInPaths')}
+            if any(value != state['plan']['services'][name].get(key) for key, value in metadata.items()):
+                raise SecurityError('Security service unit changed before activation: ' + name)
+            change = {'service': name, 'field': field, 'before': before, 'after': after,
+                      'metadata': metadata, 'status': 'pending'}
+            state['service_changes'].append(change)
+            persist(state)
+            self.host.command(['systemctl', operation, name])
+            if self._service(name)[field] != after:
+                raise SecurityError('Security service did not activate: ' + name)
+            change['status'] = 'applied'
+            persist(state)
+
+    def _check_decisions(self, peer):
+        self.host.command(['cscli', 'lapi', 'status'])
+        try:
+            alerts = json.loads(self.host.command(
+                ['cscli', 'decisions', 'list', '--all', '--limit', '0', '--no-simu', '-o', 'json']).stdout)
+            if not isinstance(alerts, list):
+                raise ValueError()
+            address = ipaddress.ip_address(peer)
+            for alert in alerts:
+                if not isinstance(alert, dict) or not isinstance(alert.get('decisions'), list):
+                    raise ValueError()
+                for decision in alert['decisions']:
+                    if (not isinstance(decision, dict) or any(not isinstance(decision.get(key), str)
+                            or not decision[key] for key in ('scope', 'value', 'type'))
+                            or decision.get('simulated', False) is not False):
+                        raise ValueError()
+                    if decision['scope'].lower() not in ('ip', 'range'):
+                        continue
+                    network = ipaddress.ip_network(decision['value'], strict=False)
+                    if address in network:
+                        raise SecurityError('An existing CrowdSec decision covers the bootstrap peer')
+        except (ValueError, TypeError, KeyError):
+            raise SecurityError('Cannot verify CrowdSec decisions safely') from None
+
+    @staticmethod
+    def _rule_file(rule):
+        return '/etc/ufw/user6.rules' if ipaddress.ip_network(rule['source'], strict=False).version == 6 else '/etc/ufw/user.rules'
+
+    def _rule_comment(self, rule):
+        source = ipaddress.ip_network(rule['source'], strict=False)
+        any_address = '::/0' if source.version == 6 else '0.0.0.0/0'
+        for line in self._text(self._rule_file(rule)).splitlines():
+            if not line.startswith('### tuple ### '):
+                continue
+            parts = line[len('### tuple ### '):].split()
+            if len(parts) < 7 or parts[:5] != ['allow', 'tcp', str(rule['port']), any_address, 'any'] or parts[6] != 'in':
+                continue
+            try:
+                if ipaddress.ip_network(parts[5], strict=False) != source:
+                    continue
+                comment = next((p[8:] for p in parts[7:] if p.startswith('comment=')), '')
+                return bytes.fromhex(comment).decode() if comment else ''
+            except (ValueError, UnicodeError):
+                raise SecurityError('Cannot prove ownership of an existing SSH firewall rule') from None
+        return None
+
+    @staticmethod
+    def _rule_command(rule, delete=False):
+        command = ['ufw', '--force', 'delete'] if delete else (['ufw', 'prepend'] if rule['temporary'] else ['ufw'])
+        return command + ['allow', 'proto', 'tcp', 'from', rule['source'], 'to', 'any', 'port', str(rule['port']),
+                          'comment', rule['comment']]
+
+    def _add_rule(self, state, port, source, temporary, persist):
+        rule = {'port': port, 'source': source, 'temporary': temporary,
+                'comment': 'apex-init-' + state['token'] + ('-bootstrap' if temporary else '-final')}
+        rule['owned'] = self._rule_comment(rule) is None
+        state['rules'].append(rule)
+        persist(state)
+        if rule['owned']:
+            self._mutate(state, self._rule_command(rule), ['/etc/ufw/user.rules', '/etc/ufw/user6.rules'], persist)
+
+    def _delete_rule(self, state, rule, persist):
+        if self._rule_comment(rule) != rule['comment']:
+            raise SecurityError('Owned SSH rule changed; preserve it for explicit reconciliation')
+        self._mutate(state, self._rule_command(rule, delete=True), ['/etc/ufw/user.rules', '/etc/ufw/user6.rules'], persist)
+
+    def _retain_bootstrap(self, state, persist):
+        rule = next((rule for rule in state['rules'] if rule['temporary']), None)
+        if rule is None:
+            raise SecurityError('No recorded bootstrap allowance exists')
+        if self._rule_comment(rule) is None:
+            rule['owned'] = True
+            state['bootstrap_retained'] = True
+            persist(state)
+            self.host.command(self._rule_command(rule))

ci/scripts/init-node/provisioning/seed.sh

@@ -0,0 +1,87 @@
+#!/usr/bin/env bash
+# Only the parser runtime may be installed before semantic input validation.
+set -euo pipefail
+export LC_ALL=C DEBIAN_FRONTEND=noninteractive
+[[ $(id -u) == 0 ]] || { echo 'APEX seed requires root.' >&2; exit 1; }
+[[ -r /etc/os-release ]] || { echo 'APEX seed requires Debian 13 amd64.' >&2; exit 1; }
+. /etc/os-release
+[[ ${ID:-} == debian && ${VERSION_ID:-} == 13 && $(dpkg --print-architecture) == amd64 ]] || {
+    echo 'APEX seed requires Debian 13 amd64.' >&2; exit 1;
+}
+
+policy=/usr/sbin/policy-rc.d
+state=/usr/sbin/.apex-policy-rc.d
+for path in /var /var/lib /var/lib/apex-init; do
+    [[ ! -L $path ]] || { echo 'Refusing redirected initialization state.' >&2; exit 1; }
+done
+mkdir -p -m 0700 /var/lib/apex-init
+[[ $(stat -c %u /var/lib/apex-init) == 0 ]] || {
+    echo 'Initialization state must be root-owned.' >&2; exit 1;
+}
+(( (8#$(stat -c %a /var/lib/apex-init) & 022) == 0 )) || {
+    echo 'Initialization state must not be writable by other users.' >&2; exit 1;
+}
+[[ ! -L /var/lib/apex-init/init.lock ]] || { echo 'Refusing redirected initialization lock.' >&2; exit 1; }
+exec 8>/var/lib/apex-init/init.lock
+flock -n 8 || { echo 'Another APEX initialization is running.' >&2; exit 1; }
+mkdir -p /run/lock
+exec 9>/run/lock/apex-package-policy.lock
+flock -n 9 || { echo 'Another APEX package operation is running.' >&2; exit 1; }
+restore_policy() {
+    [[ -d $state ]] || return 0
+    [[ ! -L $state && $(stat -c %u "$state") == 0 ]] || {
+        echo 'Refusing untrusted service policy recovery state.' >&2; return 1;
+    }
+    if [[ -e $state/ready && ( -e $policy || -L $policy ) ]]; then
+        if ! cmp -s "$policy" <(printf '#!/bin/sh\nexit 101\n') &&
+           ! cmp -s "$policy" "$state/original"; then
+            echo 'Service policy changed outside APEX; reconcile it explicitly.' >&2
+            return 1
+        fi
+    fi
+    if [[ -e $state/original || -L $state/original ]]; then
+        rm -f -- "$state/restoring"
+        cp -a -- "$state/original" "$state/restoring"
+        mv -Tf -- "$state/restoring" "$policy"
+    elif [[ -e $state/absent ]]; then
+        rm -f -- "$policy"
+    elif [[ -e $state/ready ]]; then
+        echo 'APEX service policy recovery is incomplete; inspect /usr/sbin/.apex-policy-rc.d.' >&2
+        return 1
+    fi
+    rm -rf -- "$state"
+}
+
+# Release the shared host lock on exit; the target runner reacquires it later.
+restore_policy
+missing=()
+for package in python3 python3-yaml; do
+    status=$(dpkg-query -W -f='${Status}' "$package" 2>/dev/null || true)
+    [[ $status == *' ok installed' ]] || missing+=("$package")
+done
+if (( ${#missing[@]} )); then
+    mkdir -m 0700 -- "$state"
+    trap restore_policy EXIT
+    trap 'exit 130' INT
+    trap 'exit 143' TERM
+    if [[ -e $policy || -L $policy ]]; then
+        cp -a -- "$policy" "$state/original"
+    else
+        touch "$state/absent"
+    fi
+    touch "$state/ready"
+    printf '#!/bin/sh\nexit 101\n' > "$state/guard"
+    chmod 0755 "$state/guard"
+    mv -Tf -- "$state/guard" "$policy"
+    apt=(apt-get -o DPkg::Lock::Timeout=60 -o Acquire::Retries=2
+         -o Acquire::http::Timeout=30 -o Acquire::https::Timeout=30)
+    timeout 300 "${apt[@]}" update --error-on=any
+    install=("${apt[@]}" install -y --no-install-recommends --no-upgrade --no-remove "${missing[@]}")
+    plan=$(timeout 300 "${install[@]}" --simulate)
+    if grep -Eq '^Remv |^Inst [^ ]+ \[' <<< "$plan"; then
+        echo 'Parser installation requires a package upgrade/removal; reconcile it explicitly.' >&2
+        exit 1
+    fi
+    "${install[@]}"
+fi
+python3 -c 'import yaml'

ci/scripts/init-node/provisioning/state.py

@@ -0,0 +1,128 @@
+"""Durable enrollment records and writes that refuse redirected paths."""
+from __future__ import annotations
+
+import contextlib
+import fcntl
+import json
+import os
+from pathlib import Path
+import stat
+import tempfile
+import time
+
+from . import ENROLLMENT_INTERFACE
+
+
+class ProvisionError(RuntimeError):
+    pass
+
+
+def safe_path(path):
+    path = Path(path).absolute()
+    for part in (*reversed(path.parents), path):
+        try:
+            info = part.lstat()
+        except FileNotFoundError:
+            continue
+        if stat.S_ISLNK(info.st_mode):
+            raise ProvisionError('managed path contains a symbolic link')
+    return path
+
+
+def atomic_write(path, content, mode=0o600, uid=None, gid=None):
+    path = safe_path(path)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    if path.exists() and (not path.is_file() or path.stat().st_nlink != 1):
+        raise ProvisionError('managed file is not an ordinary unlinked file')
+    data = content.encode() if isinstance(content, str) else content
+    fd, temporary = tempfile.mkstemp(prefix='.apex-', dir=path.parent)
+    try:
+        with os.fdopen(fd, 'wb') as stream:
+            os.fchmod(stream.fileno(), mode)
+            if uid is not None:
+                os.fchown(stream.fileno(), uid, gid)
+            stream.write(data)
+            stream.flush()
+            os.fsync(stream.fileno())
+        os.replace(temporary, path)
+        directory = os.open(path.parent, os.O_RDONLY)
+        try:
+            os.fsync(directory)
+        finally:
+            os.close(directory)
+    finally:
+        if os.path.exists(temporary):
+            os.unlink(temporary)
+
+
+@contextlib.contextmanager
+def exclusive_lock(path):
+    path = safe_path(path)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    fd = os.open(path, os.O_CREAT | os.O_RDWR | os.O_NOFOLLOW, 0o600)
+    try:
+        try:
+            fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
+        except BlockingIOError:
+            raise ProvisionError('another initialization is running; inspect status before retrying') from None
+        yield
+    finally:
+        os.close(fd)
+
+
+def check_heartbeat(path, now=None):
+    if path is None:
+        return
+    try:
+        age = (time.time() if now is None else now) - safe_path(path).stat().st_mtime
+    except FileNotFoundError:
+        raise ProvisionError('coordinator heartbeat is missing; stopping before the next stage') from None
+    if not 0 <= age < 90:
+        raise ProvisionError('coordinator heartbeat expired; stopping before the next stage')
+
+
+class Journal:
+    def __init__(self, path, identity, revision, bundle_digest):
+        self.path = safe_path(path)
+        if self.path.exists():
+            try:
+                self.data = json.loads(self.path.read_text())
+            except (ValueError, OSError):
+                raise ProvisionError('invalid enrollment journal; reconcile it before retrying') from None
+            if self.data.get('enrollment_interface') != ENROLLMENT_INTERFACE:
+                raise ProvisionError('recorded enrollment interface is missing or unsupported')
+            if self.data.get('identity') != identity:
+                raise ProvisionError('enrollment identity conflicts with the recorded target')
+            if self.data.get('bundle_digest') != bundle_digest:
+                raise ProvisionError('provisioner bundle changed; resume using the recorded bundle')
+        else:
+            self.data = dict(identity=identity, revision=revision, bundle_digest=bundle_digest,
+                             baseline_revision=revision, enrollment_interface=ENROLLMENT_INTERFACE,
+                             status='running', stages={})
+
+    def save(self):
+        atomic_write(self.path, json.dumps(self.data, sort_keys=True, indent=2) + '\n')
+
+    def stage(self, name, operation, heartbeat=None):
+        try:
+            check_heartbeat(heartbeat)
+        except ProvisionError:
+            self.data['status'] = 'stopped'
+            self.data['failed_stage'] = name
+            self.save()
+            raise
+        self.data['status'] = 'running'
+        self.data['stages'][name] = {'status': 'running'}
+        self.save()
+        try:
+            result = operation()
+        except Exception:
+            self.data['status'] = 'failed'
+            self.data['failed_stage'] = name
+            self.data['stages'][name] = {'status': 'failed'}
+            self.save()
+            raise
+        self.data['stages'][name] = {'status': 'done', 'result': result}
+        self.data.pop('failed_stage', None)
+        self.save()
+        return result

ci/scripts/init-node/provisioning/storage.py

@@ -0,0 +1,509 @@
+"""Conservative target-side tier and optional swap reconciliation.
+
+The caller holds the initialization lock and supplies a zero-argument atomic
+journal writer. Ambiguous device or pool ownership always requires remediation.
+"""
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+from pathlib import Path
+import re
+import stat
+import subprocess
+import tempfile
+
+from .contract import Enrollment
+
+
+SRV = Path('/srv')
+SWAPFILE = Path('/swapfile')
+FSTAB = Path('/etc/fstab')
+MEMINFO = Path('/proc/meminfo')
+SWAPS = Path('/proc/swaps')
+_OWNER = 'org.apex:enrollment'
+_GIB = 1024 ** 3
+
+
+class StorageError(RuntimeError):
+    """Storage could not be positively identified or safely reconciled."""
+
+
+def _require(condition, message):
+    if not condition:
+        raise StorageError(message)
+
+
+def _run(argv, *, full_vdev_paths=False):
+    environment = {**os.environ, 'LC_ALL': 'C'}
+    if full_vdev_paths:
+        environment.update(ZPOOL_VDEV_NAME_PATH='1', ZPOOL_VDEV_NAME_FOLLOW_LINKS='1')
+        environment.pop('ZPOOL_VDEV_NAME_GUID', None)
+    try:
+        result = subprocess.run(argv, check=False, capture_output=True, text=True,
+                                timeout=300, env=environment)
+    except (OSError, subprocess.TimeoutExpired):
+        raise StorageError('required storage tool is unavailable or timed out: ' + argv[0]) from None
+    _require(result.returncode == 0,
+             'storage inspection or operation failed: ' + ' '.join(argv[:2]))
+    return result.stdout
+
+
+def _json(argv, key):
+    try:
+        value = json.loads(_run(argv))[key]
+        _require(isinstance(value, list), 'storage inspection returned an invalid list')
+        return value
+    except (ValueError, KeyError, TypeError):
+        raise StorageError('storage inspection returned unsupported JSON') from None
+
+
+def _flatten(items):
+    for item in items:
+        yield item
+        yield from _flatten(item.get('children', []))
+
+
+def _mounts():
+    return list(_flatten(_json(['findmnt', '--json', '--output', 'TARGET,SOURCE,FSTYPE,MAJ:MIN'],
+                               'filesystems')))
+
+
+def _device_number(path):
+    try:
+        info = os.stat(path)
+    except OSError:
+        raise StorageError('selected device is missing or is not a block device') from None
+    _require(stat.S_ISBLK(info.st_mode), 'selected device is not a block device')
+    return f'{os.major(info.st_rdev)}:{os.minor(info.st_rdev)}'
+
+
+def _inventory():
+    roots = _json(['lsblk', '--json', '--bytes', '--paths', '--output',
+                   'NAME,MAJ:MIN,TYPE,PKNAME,MOUNTPOINTS,FSTYPE,WWN,SERIAL,SIZE,PARTUUID'], 'blockdevices')
+    nodes, edges = {}, {}
+
+    def visit(items, parent=None):
+        for item in items:
+            number = item['maj:min']
+            nodes[number] = item
+            edges.setdefault(number, set())
+            if parent:
+                edges[number].add(parent)
+                edges[parent].add(number)
+            visit(item.get('children', []), number)
+
+    try:
+        visit(roots)
+        by_name = {item['name']: number for number, item in nodes.items()}
+        for number, item in nodes.items():
+            parent = by_name.get(item.get('pkname'))
+            if parent:
+                edges[number].add(parent)
+                edges[parent].add(number)
+    except (KeyError, TypeError):
+        raise StorageError('block-device topology could not be established') from None
+    return nodes, edges
+
+
+def _connected(number, edges):
+    found, pending = set(), [number]
+    while pending:
+        item = pending.pop()
+        if item not in found:
+            found.add(item)
+            pending.extend(edges.get(item, ()))
+    return found
+
+
+def _pools():
+    result = {}
+    for line in _run(['zpool', 'list', '-H', '-o', 'name,guid,health']).splitlines():
+        fields = line.split('\t')
+        _require(len(fields) == 3 and fields[1].isdigit(), 'active pool identity is ambiguous')
+        result[fields[0]] = {'guid': fields[1], 'health': fields[2]}
+    return result
+
+
+def _members(text, name, imported=True):
+    """Accept only a plain single-device ONLINE pool, never infer topology."""
+    marker = re.search(r'^\s*config:\s*$', text, re.M)
+    _require(marker is not None, 'pool membership is unavailable')
+    lines = text[marker.end():].splitlines()
+    rows = []
+    for line in lines:
+        fields = line.split()
+        if not fields or fields[0] == 'NAME':
+            continue
+        if fields[0] == 'errors:':
+            _require(line.strip() == 'errors: No known data errors', 'pool reports data errors')
+            break
+        _require(len(fields) >= 2 and fields[1] == 'ONLINE',
+                 'pool is not a healthy single-device topology')
+        if imported:
+            _require(len(fields) == 5 and fields[2:] == ['0', '0', '0'],
+                     'pool reports device errors')
+        rows.append(fields[0])
+    _require(len(rows) == 2 and rows[0] == name,
+             'pool must contain exactly one identifiable device')
+    leaf = rows[1]
+    if not imported and re.fullmatch(r'[A-Za-z0-9_.-]+', leaf):
+        leaf = '/dev/' + leaf
+    _require(leaf.startswith('/dev/'), 'pool device path cannot be positively resolved')
+    return leaf
+
+
+def _pool_device(name):
+    return _members(_run(['zpool', 'status', '-P', name], full_vdev_paths=True), name)
+
+
+def _inspect_device(device, mounts, allowed_pool=None, allowed_path=None):
+    number = _device_number(device)
+    nodes, edges = _inventory()
+    _require(number in nodes, 'selected device is absent from the block topology')
+    node = nodes[number]
+    hardware = node.get('wwn') or node.get('serial')
+    partition = None
+    if node.get('type') == 'part':
+        parents = [item for item in nodes.values() if item['name'] == node.get('pkname')]
+        _require(len(parents) == 1 and parents[0]['type'] == 'disk' and node.get('partuuid'),
+                 'selected partition needs a stable partition UUID and identifiable parent disk')
+        hardware = parents[0].get('wwn') or parents[0].get('serial')
+        partition = node['partuuid']
+    _require(isinstance(hardware, str) and bool(hardware.strip()),
+             'selected device needs a stable WWN or hardware serial')
+    _require(node.get('type') in ('disk', 'part') and int(node.get('size') or 0) > 0,
+             'selected device type or capacity is unsupported')
+    related = _connected(number, edges)
+    protected = set()
+    for other, data in nodes.items():
+        for mount in data.get('mountpoints') or []:
+            if mount and mount != allowed_path:
+                protected.add(other)
+    for mount in mounts:
+        if mount['target'] != allowed_path:
+            protected.add(mount.get('maj:min'))
+    active = _pools()
+    for pool in active:
+        leaf = _device_number(_pool_device(pool))
+        if pool != allowed_pool or any(
+                m['source'].split('/')[0] == pool and m['target'] != allowed_path for m in mounts):
+            protected.add(leaf)
+    _require(not (related & protected),
+             'selected device intersects mounted, root, boot, swap, or active pool storage')
+    identity = {'hardware': hardware, 'size': int(node['size']), 'type': node['type']}
+    if partition:
+        identity['partition'] = partition
+    return {'number': number, 'identity': identity, 'related': related, 'node': node, 'pools': active}
+
+
+def _safe_path(path):
+    for parent in reversed((path, *path.parents)):
+        try:
+            info = parent.lstat()
+        except FileNotFoundError:
+            _require(parent == path, 'storage parent directory is missing')
+            continue
+        _require(stat.S_ISDIR(info.st_mode), 'storage path contains a symlink or non-directory')
+
+
+def _directory(path, mounts, pool=None):
+    _safe_path(path)
+    exact = [m for m in mounts if m['target'] == str(path)]
+    _require(not exact or (len(exact) == 1 and pool is not None and
+                          exact[0]['fstype'] == 'zfs' and exact[0]['source'] == pool),
+             'tier path is occupied by an unexpected mount')
+    _require(not any(m['target'].startswith(str(path) + '/') for m in mounts),
+             'tier path contains an unexpected nested mount')
+    created = not path.exists()
+    if created:
+        path.mkdir(mode=0o755)
+        path.chmod(0o755)
+    return created
+
+
+def _directory_identity(path):
+    _safe_path(path)
+    try:
+        info = path.lstat()
+    except FileNotFoundError:
+        raise StorageError('managed directory identity is missing') from None
+    return {'device': info.st_dev, 'inode': info.st_ino}
+
+
+def _verify_ownership_identity(record, path):
+    ownership = record.get('ownership')
+    if ownership:
+        _require(ownership['identity'] == _directory_identity(path),
+                 'managed directory identity changed before ownership handoff')
+    return ownership
+
+
+def _record_pending_ownership(record, path):
+    if not _verify_ownership_identity(record, path):
+        record['ownership'] = {'status': 'pending', 'identity': _directory_identity(path)}
+
+
+def _verify_pool(name, guid, device_info, path, owner, require_owner=True):
+    pools = _pools()
+    _require(name in pools and pools[name] == {'guid': guid, 'health': 'ONLINE'},
+             'pool name, GUID, or health does not match enrollment')
+    _require(_member_matches(_pool_device(name), device_info),
+             'pool device membership does not match the selected device')
+    rows = [line.split('\t') for line in _run(
+        ['zfs', 'list', '-H', '-r', '-o', 'name,mountpoint,type,mounted', name]).splitlines()]
+    _require(len(rows) == 1 and len(rows[0]) == 4 and rows[0][:3] == [name, str(path), 'filesystem'],
+             'pool dataset or mountpoint requires explicit reconciliation')
+    if require_owner:
+        actual = _run(['zfs', 'get', '-H', '-o', 'value', _OWNER, name]).strip()
+        _require(actual == owner, 'pool lacks matching enrollment ownership')
+    return rows[0][3] == 'yes'
+
+
+def _member_matches(member, info):
+    number = _device_number(member)
+    return number == info['number'] or (info['node']['type'] == 'disk' and number in info['related'])
+
+
+def _import_candidate(device, guid):
+    text = _run(['zpool', 'import', '-d', device], full_vdev_paths=True)
+    names = re.findall(r'^\s*pool:\s*(\S+)\s*$', text, re.M)
+    guids = re.findall(r'^\s*id:\s*(\d+)\s*$', text, re.M)
+    states = re.findall(r'^\s*state:\s*(\S+)\s*$', text, re.M)
+    _require(len(names) == 1 and guids == [guid] and states == ['ONLINE'],
+             'import discovery does not match the declared healthy pool GUID')
+    actions = re.findall(r'^\s*action:\s*(.+)$', text, re.M)
+    _require(not re.search(r'^\s*status:', text, re.M) and
+             all(action == 'The pool can be imported using its name or numeric identifier.' for action in actions),
+             'pool import reports ownership or recovery conditions; reconcile manually')
+    return names[0], _members(text, names[0], imported=False)
+
+
+def _block_tier(tier, path, journal, owner, save):
+    record = journal.get(str(tier.number))
+    created = False
+    pool = record.get('pool') if record else None
+    info = _inspect_device(tier.device, _mounts(), pool, str(path) if pool else None)
+    if record:
+        _require(record['device'] == info['identity'], 'selected device identity changed since enrollment')
+        _require(record['status'] in ('intent', 'complete'), 'tier journal status is unsupported')
+    if pool and pool in info['pools']:
+        guid = record.get('guid') or info['pools'][pool]['guid']
+        _verify_pool(pool, guid, info, path, owner)
+    else:
+        _require(not record or record['status'] == 'intent',
+                 'previously enrolled pool is not active; explicit reconciliation is required')
+        if tier.mode == 'create':
+            pool = f'tier-{tier.number}'
+            _require(pool not in info['pools'], 'existing pool name does not prove enrollment ownership')
+            _require(info['node']['type'] == 'disk', 'pool creation requires an unused whole disk')
+            _require(len(info['related']) == 1, 'selected disk has partitions or dependent devices')
+            signatures = _json(['wipefs', '--json', '--output', 'TYPE,UUID,LABEL', tier.device], 'signatures')
+            _require(not signatures and not info['node'].get('fstype'), 'selected device contains existing signatures')
+        else:
+            pool, member = _import_candidate(tier.device, tier.pool_guid)
+            _require(_member_matches(member, info), 'import membership differs from the selected device')
+            _require(pool not in info['pools'], 'pool is already active without matching enrollment ownership')
+        created = _directory(path, _mounts())
+        _require(not path.exists() or not any(path.iterdir()), 'pool mountpoint contains existing data')
+        record = {'status': 'intent', 'device': info['identity'], 'pool': pool,
+                  'ownership_intent': tier.mode == 'create'}
+        journal[str(tier.number)] = record
+        save()
+        fresh = _inspect_device(tier.device, _mounts())
+        _require(fresh['identity'] == info['identity'] and fresh['number'] == info['number'],
+                 'selected device identity changed before pool operation')
+        _directory(path, _mounts())
+        _require(not any(path.iterdir()), 'pool mountpoint acquired existing data before operation')
+        if tier.mode == 'create':
+            _require(not _json(['wipefs', '--json', '--output', 'TYPE,UUID,LABEL', tier.device], 'signatures'),
+                     'selected device acquired signatures before creation')
+            _run(['zpool', 'create', '-o', 'ashift=12', '-O', f'mountpoint={path}',
+                  '-O', f'{_OWNER}={owner}', pool, tier.device])
+        else:
+            _require(_import_candidate(tier.device, tier.pool_guid) == (pool, member),
+                     'pool identity changed before import')
+            _run(['zpool', 'import', '-N', '-d', tier.device, tier.pool_guid])
+        pools = _pools()
+        _require(pool in pools, 'pool operation did not establish an active pool')
+        guid = pools[pool]['guid']
+        _require(tier.mode != 'import' or guid == tier.pool_guid, 'imported pool GUID differs from intent')
+        refreshed = _inspect_device(tier.device, _mounts(), pool, str(path))
+        _require(refreshed['identity'] == info['identity'], 'selected device changed during pool operation')
+        info = refreshed
+        _verify_pool(pool, guid, info, path, owner, require_owner=tier.mode == 'create')
+        if tier.mode == 'import':
+            previous_owner = _run(['zfs', 'get', '-H', '-o', 'value', _OWNER, pool]).strip()
+            _require(previous_owner in ('-', owner), 'imported pool belongs to a different enrollment')
+            _run(['zfs', 'set', f'{_OWNER}={owner}', pool])
+    created = _directory(path, _mounts(), pool) or created
+    if not _verify_pool(pool, guid, info, path, owner):
+        _require(not any(path.iterdir()), 'refusing to hide existing files beneath a pool mount')
+        _run(['zfs', 'mount', pool])
+    _require(any(m['target'] == str(path) and m['source'] == pool and m['fstype'] == 'zfs'
+                 for m in _mounts()), 'pool is not mounted at its enrolled path')
+    record.update(status='complete', guid=guid)
+    if tier.mode == 'create' and record.get('ownership_intent'):
+        _record_pending_ownership(record, path)
+    save()
+    return {'path': str(path), 'mode': tier.mode, 'created': created and tier.mode == 'create',
+            'pool': pool, 'guid': guid}
+
+
+def _persist_swap():
+    info = FSTAB.lstat()
+    _require(stat.S_ISREG(info.st_mode) and info.st_uid == 0, 'fstab must be a root-owned regular file')
+    original = FSTAB.read_text()
+    lines = [line.split() for line in original.splitlines() if line.strip() and not line.lstrip().startswith('#')]
+    matches = [line for line in lines if line[0] == str(SWAPFILE)]
+    if matches:
+        _require(len(matches) == 1 and len(matches[0]) >= 4 and matches[0][2] == 'swap'
+                 and 'noauto' not in matches[0][3].split(','),
+                 'existing swap fstab entry is incompatible')
+        return
+    fd, temporary = tempfile.mkstemp(prefix='.apex-swap-', dir=FSTAB.parent)
+    try:
+        with os.fdopen(fd, 'w') as output:
+            os.fchmod(output.fileno(), stat.S_IMODE(info.st_mode))
+            os.fchown(output.fileno(), info.st_uid, info.st_gid)
+            output.write(original + ('' if original.endswith('\n') else '\n') +
+                         f'{SWAPFILE} none swap sw 0 0\n')
+            output.flush()
+            os.fsync(output.fileno())
+        current = FSTAB.lstat()
+        _require(all(getattr(current, key) == getattr(info, key) for key in
+                     ('st_dev', 'st_ino', 'st_size', 'st_mtime_ns', 'st_ctime_ns')),
+                 'fstab changed during swap setup')
+        os.replace(temporary, FSTAB)
+    finally:
+        if os.path.exists(temporary):
+            os.unlink(temporary)
+
+
+def _active_swaps():
+    lines = SWAPS.read_bytes().splitlines()
+    _require(bool(lines) and lines[0].split() == [b'Filename', b'Type', b'Size', b'Used', b'Priority'],
+             'kernel swap inventory header is invalid')
+    paths = []
+    for line in lines[1:]:
+        fields = line.split()
+        _require(len(fields) == 5 and fields[0].startswith(b'/') and fields[1] in (b'file', b'partition')
+                 and fields[2].isdigit() and fields[3].isdigit()
+                 and re.fullmatch(rb'-?[0-9]+', fields[4]) is not None,
+                 'kernel swap inventory entry is invalid')
+        _require(re.search(rb'\\(?![0-3][0-7]{2})', fields[0]) is None,
+                 'kernel swap inventory path escaping is invalid')
+        decoded = re.sub(rb'\\([0-3][0-7]{2})', lambda match: bytes([int(match.group(1), 8)]), fields[0])
+        paths.append(os.fsdecode(decoded))
+    return paths
+
+
+def _swap(journal, save):
+    try:
+        active = _active_swaps()
+        if active:
+            return {'status': 'skipped', 'reason': 'existing active swap preserved'}
+        if os.path.lexists(SWAPFILE):
+            return {'status': 'skipped', 'reason': 'existing swapfile preserved; no overwrite or resize'}
+        mounts = _mounts()
+        covering = [m for m in mounts if str(SWAPFILE).startswith(m['target'].rstrip('/') + '/')]
+        _require(bool(covering), 'swap filesystem could not be identified')
+        filesystem = max(covering, key=lambda m: len(m['target']))['fstype']
+        _require(filesystem in ('ext4', 'xfs'), 'optional swap requires a supported ext4 or XFS filesystem')
+        match = re.search(r'^MemTotal:\s+(\d+)\s+kB$', MEMINFO.read_text(), re.M)
+        _require(match is not None, 'RAM size could not be established')
+        size = (4 if int(match.group(1)) * 1024 > 4 * _GIB else 2) * _GIB
+        space = os.statvfs(SWAPFILE.parent)
+        _require(space.f_bavail * space.f_frsize >= size + _GIB,
+                 'insufficient free space for optional swap and reserve')
+        _require(hasattr(os, 'posix_fallocate'), 'safe swap allocation is unavailable')
+        journal['swap'] = {'status': 'intent', 'size': size}
+        save()
+        fd = os.open(SWAPFILE, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600)
+        try:
+            os.posix_fallocate(fd, 0, size)
+            os.fsync(fd)
+        finally:
+            os.close(fd)
+        _run(['mkswap', str(SWAPFILE)])
+        _run(['swapon', str(SWAPFILE)])
+        active = _active_swaps()
+        _require(str(SWAPFILE) in active,
+                 'optional swap activation did not become observable')
+        _persist_swap()
+        return {'status': 'complete', 'size': size}
+    except (StorageError, OSError) as error:
+        return {'status': 'skipped', 'reason': str(error) if isinstance(error, StorageError)
+                else 'optional swap allocation or persistence failed; existing resources preserved'}
+
+
+def _enrollment_identity(spec):
+    return hashlib.sha256(json.dumps(spec.identity(), sort_keys=True,
+                                     separators=(',', ':')).encode()).hexdigest()
+
+
+def acknowledge_storage_ownership(spec: Enrollment, state: dict, paths, save) -> None:
+    """Persist the caller's successful, nonrecursive ownership handoff for paths."""
+    journal = state.get('storage', {})
+    _require(journal.get('identity') == _enrollment_identity(spec),
+             'storage enrollment identity conflicts with ownership acknowledgement')
+    allowed = {str(SRV / f'tier-{tier.number}.{spec.repository.name}'): tier for tier in spec.tiers}
+    acknowledged = []
+    for path in paths:
+        _require(path in allowed, 'ownership acknowledgement path is outside enrollment')
+        tier = allowed[path]
+        record = journal.get('tiers', {}).get(str(tier.number), {})
+        ownership = _verify_ownership_identity(record, Path(path))
+        _require(tier.mode != 'import' and ownership is not None,
+                 'ownership acknowledgement requires a directory created by enrollment')
+        _require(ownership['status'] in ('pending', 'applied'), 'ownership journal status is unsupported')
+        acknowledged.append(ownership)
+    for ownership in acknowledged:
+        ownership['status'] = 'applied'
+    save()
+
+
+def reconcile_storage(spec: Enrollment, state: dict, save) -> dict:
+    """Return ownership_pending paths until the caller acknowledges their chown."""
+    identity = _enrollment_identity(spec)
+    journal = state.setdefault('storage', {'identity': identity, 'tiers': {}})
+    _require(journal.get('identity') == identity, 'storage enrollment identity conflicts with journal')
+    _safe_path(SRV)
+    _require(SRV.exists(), 'storage parent directory is missing')
+    selected = set()
+    for tier in spec.tiers:
+        if tier.device:
+            number = _device_number(tier.device)
+            _require(number not in selected, 'multiple tiers select aliases of the same device')
+            selected.add(number)
+    result = {'tiers': [], 'ownership_pending': []}
+    for tier in spec.tiers:
+        path = SRV / f'tier-{tier.number}.{spec.repository.name}'
+        if tier.mode == 'directory':
+            record = journal['tiers'].get(str(tier.number), {})
+            _verify_ownership_identity(record, path)
+            _safe_path(path)
+            _require(not (record.get('ownership_intent') and not record.get('ownership') and path.exists()),
+                     'directory creation was interrupted before identity was recorded; reconcile manually')
+            if not path.exists():
+                record = {'status': 'intent', 'path': str(path), 'ownership_intent': True}
+                journal['tiers'][str(tier.number)] = record
+                save()
+            created = _directory(path, _mounts())
+            if created:
+                _record_pending_ownership(record, path)
+            entry = {'path': str(path), 'mode': tier.mode, 'created': created}
+            record.update(status='complete', path=str(path))
+            journal['tiers'][str(tier.number)] = record
+            save()
+        else:
+            entry = _block_tier(tier, path, journal['tiers'], identity, save)
+        ownership = journal['tiers'][str(tier.number)].get('ownership')
+        if ownership and ownership['status'] == 'pending':
+            result['ownership_pending'].append(str(path))
+        result['tiers'].append(entry)
+    result['swap'] = _swap(journal, save)
+    journal['swap'] = result['swap']
+    save()
+    return result

ci/scripts/init-node/provisioning/target.py

@@ -0,0 +1,170 @@
+"""Shared root runner for remote and cloud-init enrollment."""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+from pathlib import Path
+import platform
+import pwd
+import subprocess
+import sys
+import tempfile
+
+from .state import Journal, ProvisionError, atomic_write, exclusive_lock, safe_path
+from .userdata import decode_user_data
+
+STATE = Path('/var/lib/apex-init')
+
+
+def run(args, **kwargs):
+    result = subprocess.run(args, text=True, capture_output=True, timeout=300, **kwargs)
+    if result.returncode:
+        raise ProvisionError('required target operation failed: ' + args[0])
+    return result.stdout.strip()
+
+
+def preflight():
+    if os.geteuid() != 0:
+        raise ProvisionError('target enrollment requires root privileges')
+    release = {}
+    for line in Path('/etc/os-release').read_text().splitlines():
+        key, separator, value = line.partition('=')
+        if separator:
+            release[key] = value.strip('"')
+    if release.get('ID') != 'debian' or release.get('VERSION_ID') != '13' or platform.machine() != 'x86_64':
+        raise ProvisionError('initialization supports Debian 13 amd64 only')
+
+
+def validate_keys(spec):
+    for key in spec.authorized_keys:
+        with tempfile.NamedTemporaryFile(mode='w') as stream:
+            stream.write(key + '\n')
+            stream.flush()
+            try:
+                run(['ssh-keygen', '-l', '-f', stream.name])
+            except ProvisionError:
+                raise ProvisionError('administrator public key failed cryptographic validation') from None
+
+
+def administrator(spec):
+    try:
+        account = pwd.getpwnam('adam')
+    except KeyError:
+        run(['useradd', '--create-home', '--shell', '/bin/bash', '--groups', 'sudo,docker', 'adam'])
+        account = pwd.getpwnam('adam')
+    if account.pw_dir != '/home/adam' or account.pw_uid == 0:
+        raise ProvisionError('existing administrator has incompatible home or UID')
+    safe_path(account.pw_dir)
+    if account.pw_shell != '/bin/bash':
+        run(['usermod', '--shell', '/bin/bash', 'adam'])
+    run(['usermod', '--append', '--groups', 'sudo,docker', 'adam'])
+    directory = safe_path(Path(account.pw_dir) / '.ssh')
+    directory.mkdir(mode=0o700, exist_ok=True)
+    os.chown(directory, account.pw_uid, account.pw_gid)
+    os.chmod(directory, 0o700)
+    authorized = safe_path(directory / 'authorized_keys')
+    existing = authorized.read_text().splitlines() if authorized.exists() else []
+    keys = existing + [key for key in spec.authorized_keys if key not in existing]
+    atomic_write(authorized, '\n'.join(keys) + '\n', uid=account.pw_uid, gid=account.pw_gid)
+    policy = 'adam ALL=(ALL:ALL) NOPASSWD: ALL\n'
+    with tempfile.NamedTemporaryFile(mode='w') as stream:
+        stream.write(policy)
+        stream.flush()
+        run(['visudo', '-cf', stream.name])
+    atomic_write('/etc/sudoers.d/99-apex-adam', policy, 0o440)
+    run(['visudo', '-c'])
+    run(['runuser', '-u', 'adam', '--', 'sudo', '-n', 'true'])
+    for item in spec.files:
+        uid, gid = (0, 0) if item.owner == 'root:root' else (account.pw_uid, account.pw_gid)
+        atomic_write(item.path, item.content, item.mode, uid=uid, gid=gid)
+    if run(['hostname']) != spec.hostname:
+        run(['hostnamectl', 'set-hostname', spec.hostname])
+    groups = run(['id', '-nG', 'adam']).split()
+    if not {'sudo', 'docker'} <= set(groups):
+        raise ProvisionError('administrator groups did not converge')
+    return {'administrator': 'adam', 'hostname': spec.hostname, 'sudo': 'verified'}
+
+
+def enroll(spec, revision, bundle_digest, heartbeat=None):
+    from .baseline import apply_baseline
+    from .storage import acknowledge_storage_ownership, reconcile_storage
+    from .repository import reconcile_repository
+
+    preflight()
+    validate_keys(spec)
+    safe_path(STATE).mkdir(mode=0o700, exist_ok=True)
+    if STATE.stat().st_uid != 0:
+        raise ProvisionError('enrollment state directory must be root-owned')
+    os.chmod(STATE, 0o700)
+    with exclusive_lock(STATE / 'init.lock'):
+        journal = Journal(STATE / 'enrollment.json', spec.identity(), revision, bundle_digest)
+        journal.save()
+        journal.stage('baseline', apply_baseline, heartbeat)
+        journal.stage('administrator', lambda: administrator(spec), heartbeat)
+        result = journal.stage('storage', lambda: reconcile_storage(spec, journal.data, journal.save), heartbeat)
+        account = pwd.getpwnam('adam')
+        for path in result['ownership_pending']:
+            os.chown(safe_path(path), account.pw_uid, account.pw_gid)
+            acknowledge_storage_ownership(spec, journal.data, [path], journal.save)
+        journal.stage('repository', lambda: reconcile_repository(spec, journal.data, journal.save), heartbeat)
+        journal.data['status'] = 'enrolled'
+        journal.data['external_access'] = 'pending'
+        journal.save()
+        return journal.data
+
+
+def main(argv=None):
+    parser = argparse.ArgumentParser()
+    parser.add_argument('operation', choices=('validate', 'enroll', 'status', 'complete'))
+    parser.add_argument('--user-data')
+    parser.add_argument('--revision', default='unknown')
+    parser.add_argument('--bundle-digest', default='unknown')
+    parser.add_argument('--heartbeat')
+    args = parser.parse_args(argv)
+    try:
+        preflight()
+        if args.operation == 'status':
+            path = safe_path(STATE / 'enrollment.json')
+            print(path.read_text() if path.exists() else json.dumps({'status': 'not-started'}))
+            return 0
+        if args.operation == 'complete':
+            with exclusive_lock(STATE / 'init.lock'):
+                access = json.loads(safe_path(STATE / 'access/state.json').read_text())
+                path = safe_path(STATE / 'enrollment.json')
+                record = json.loads(path.read_text())
+                if access.get('status') != 'committed' or record.get('status') not in ('enrolled', 'complete'):
+                    raise ProvisionError('enrollment and access must be verified before completion')
+                record['external_access'] = 'verified'
+                record['status'] = 'complete'
+                atomic_write(path, json.dumps(record, sort_keys=True) + '\n')
+                print(json.dumps(record, sort_keys=True))
+            return 0
+        if not args.user_data:
+            raise ProvisionError('--user-data is required')
+        path = safe_path(args.user_data)
+        if path.stat().st_uid != 0 or path.stat().st_mode & 0o077:
+            raise ProvisionError('user-data must be root-owned and private')
+        if not path.is_file() or path.stat().st_size > 1024 * 1024:
+            raise ProvisionError('user-data must be a regular file no larger than 1 MiB')
+        spec = decode_user_data(path.read_bytes())
+        validate_keys(spec)
+        if args.operation == 'validate':
+            print(json.dumps({'status': 'valid', 'hostname': spec.hostname}))
+        else:
+            print(json.dumps(enroll(spec, args.revision, args.bundle_digest, args.heartbeat), sort_keys=True))
+        return 0
+    except Exception as error:
+        from .contract import ContractError
+        from .baseline import BaselineError
+        from .storage import StorageError
+        if isinstance(error, (ProvisionError, ContractError, BaselineError, StorageError)):
+            print(str(error), file=sys.stderr)
+        else:
+            # Unexpected dependency failures must not render raw credential input.
+            print('initialization failed (' + type(error).__name__ + '); inspect stage status', file=sys.stderr)
+        return 1
+
+
+if __name__ == '__main__':
+    raise SystemExit(main())

ci/scripts/init-node/provisioning/userdata.py

@@ -0,0 +1,30 @@
+"""Target-only decoding; importing the CLI never requires a YAML dependency."""
+from .contract import ContractError, normalize_user_data
+
+
+def decode_user_data(content):
+    import yaml
+
+    class Loader(yaml.SafeLoader):
+        def compose_node(self, parent, index):
+            if self.check_event(yaml.AliasEvent):
+                raise ContractError('YAML aliases are not supported')
+            return super().compose_node(parent, index)
+
+        def construct_mapping(self, node, deep=False):
+            result = {}
+            for key_node, value_node in node.value:
+                key = self.construct_object(key_node, deep=deep)
+                if not isinstance(key, str) or key in result:
+                    raise ContractError('YAML mapping keys must be unique strings')
+                result[key] = self.construct_object(value_node, deep=deep)
+            return result
+
+    if len(content) > 1024 * 1024:
+        raise ContractError('user-data exceeds the supported size')
+    try:
+        # Loader extends SafeLoader only to reject aliases and duplicate keys.
+        document = yaml.load(content, Loader=Loader)  # noqa: S506
+    except (yaml.YAMLError, UnicodeError, RecursionError):
+        raise ContractError('invalid or unsupported YAML user-data') from None
+    return normalize_user_data(document)

ci/scripts/init-node/tests/provision_fakes.py

@@ -0,0 +1,38 @@
+from provisioning.security import SecurityError
+
+
+class SecurityBoundary:
+    """Isolate firewall/service effects while exercising the durable coordinator."""
+    def __init__(self):
+        self.active = False
+        self.fail_activation = False
+        self.fail_restore = False
+
+    def preflight(self):
+        return {}
+
+    def snapshot(self, plan, port, address):
+        return {'phase': 'prepared', 'was_active': self.active}
+
+    def activate(self, state, persist):
+        self.active = True
+        state['phase'] = 'active'
+        persist(state)
+        if self.fail_activation:
+            raise SecurityError('activation failed')
+
+    def finalize(self, state, persist):
+        self.verify(state)
+        state['phase'] = 'final'
+        persist(state)
+
+    def verify(self, state):
+        if not self.active:
+            raise SecurityError('security stopped')
+
+    def restore(self, state, persist):
+        self.active = state['was_active']
+        state['cleanup'] = 'pending' if self.fail_restore else 'complete'
+        persist(state)
+        if self.fail_restore:
+            raise SecurityError('cleanup pending')

ci/scripts/init-node/tests/test_provision_access.py

@@ -0,0 +1,513 @@
+import json
+from pathlib import Path
+import pwd
+import shutil
+import subprocess
+import tempfile
+import unittest
+from unittest.mock import patch
+
+from provisioning.access import AccessError, Host, Transition, atomic, main
+from provision_fakes import SecurityBoundary
+
+
+TOKEN = 'a' * 32
+
+
+class FakeHost(Host):
+    """Replace system service effects; keep snapshots and transition files real."""
+    def __init__(self, root):
+        super().__init__(root)
+        self.security = SecurityBoundary()
+        self.boot = 'first-boot'
+        self.armed = False
+        self.restored = 0
+        self.ports = [22]
+        self.firewall = False
+        self.bad_sudo = False
+        self.fail_final = False
+        self.existing_final = False
+        atomic(self.path('/etc/ssh/sshd_config'), b'Port 22\n')
+
+    def boot_id(self):
+        return self.boot
+
+    def preflight(self):
+        if self.bad_sudo:
+            raise AccessError('sudo failed')
+
+    def arm(self):
+        self.armed = True
+
+    def disarm(self):
+        self.armed = False
+
+    def firewall_needed(self):
+        return True
+
+    def open_firewall(self):
+        self.firewall = True
+
+    def policy(self, ports, final, address):
+        if final and self.fail_final:
+            raise AccessError('policy failed')
+
+    def current_final(self, address):
+        return self.existing_final
+
+    def verify_active(self, ports, final, address):
+        self.policy(ports, final, address)
+        if self.ports != sorted(set(ports)):
+            raise AccessError('Live listener ports changed')
+
+    def install_policy(self, ports, final, address):
+        self.policy(ports, final, address)
+        self.ports = sorted(set(ports))
+        atomic(self.path('/etc/systemd/system/ssh.service.d/apex-init.conf'),
+               json.dumps({'ports': self.ports, 'final': final}).encode())
+
+    def restore(self, state):
+        self.restored += 1
+        self.ports = [state['original_port']]
+        self.firewall = False
+        path = self.path('/etc/systemd/system/ssh.service.d/apex-init.conf')
+        if state['previous'] is None:
+            path.unlink(missing_ok=True)
+        else:
+            import base64
+            atomic(path, base64.b64decode(state['previous']))
+
+
+class AccessTests(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.temp.cleanup)
+        self.host = FakeHost(Path(self.temp.name))
+        self.clock = 1000
+        self.transition = Transition(self.host, now=lambda: self.clock, security=self.host.security)
+
+    def begin(self):
+        return self.transition.execute('begin', TOKEN, 22, '192.0.2.10')
+
+    def test_transition_keeps_original_until_final_and_commit_stops_rollback(self):
+        self.begin()
+        self.assertEqual(self.host.ports, [22, 2222])
+        self.assertTrue(self.host.armed)
+        self.transition.execute('finalize', TOKEN)
+        self.assertEqual(self.host.ports, [2222])
+        self.assertTrue(self.host.armed)
+        self.transition.execute('commit', TOKEN)
+        self.clock += 400
+        self.host.boot = 'new-boot'
+        self.transition.execute('tick')
+        self.assertEqual(self.host.ports, [2222])
+        self.assertEqual(self.host.restored, 0)
+        self.assertFalse(self.host.armed)
+
+    def test_expiry_restores_original_even_after_final_policy(self):
+        self.begin()
+        self.transition.execute('finalize', TOKEN)
+        self.clock += 301
+        self.transition.execute('tick')
+        self.assertEqual(self.host.ports, [22])
+        self.assertFalse(self.host.firewall)
+        self.assertEqual(self.transition.load()['status'], 'rolled_back')
+
+    def test_reboot_recovers_persisted_transition_with_fresh_runner(self):
+        self.begin()
+        self.host.boot = 'second-boot'
+        fresh = Transition(self.host, now=lambda: 0, security=self.host.security)
+        fresh.execute('tick')
+        self.assertEqual(self.host.ports, [22])
+        self.assertEqual(self.host.restored, 1)
+
+    def test_watchdog_restores_access_while_enrollment_lock_is_held(self):
+        from provisioning.state import exclusive_lock
+        self.begin()
+        self.clock += 301
+        with exclusive_lock(self.host.path('/var/lib/apex-init/init.lock')):
+            self.transition.execute('tick')
+        self.assertEqual(self.host.ports, [22])
+        self.assertEqual(self.host.restored, 1)
+
+    def test_external_edit_prevents_finalize_and_watchdog_preserves_edit(self):
+        self.begin()
+        path = self.host.path('/etc/ssh/sshd_config')
+        path.write_text('Port 22\nBanner /etc/banner\n')
+        with self.assertRaises(AccessError):
+            self.transition.execute('finalize', TOKEN)
+        self.assertTrue(self.host.armed)
+        self.clock += 301
+        self.transition.execute('tick')
+        self.assertIn('Banner', path.read_text())
+
+    def test_invalid_sudo_never_changes_listeners(self):
+        self.host.bad_sudo = True
+        with self.assertRaises(AccessError):
+            self.begin()
+        self.assertEqual(self.host.ports, [22])
+        self.assertFalse(self.host.armed)
+
+    def test_failed_final_policy_restores_original(self):
+        self.begin()
+        self.host.fail_final = True
+        with self.assertRaises(AccessError):
+            self.transition.execute('finalize', TOKEN)
+        self.assertEqual(self.host.ports, [22])
+        self.assertFalse(self.host.armed)
+
+    def test_other_coordinator_cannot_commit(self):
+        self.begin()
+        self.transition.execute('finalize', TOKEN)
+        with self.assertRaises(AccessError):
+            self.transition.execute('commit', 'b' * 32)
+        self.assertTrue(self.host.armed)
+
+    def test_committed_retry_does_not_reopen_root_listener(self):
+        self.begin()
+        self.transition.execute('finalize', TOKEN)
+        self.transition.execute('commit', TOKEN)
+        result = self.transition.execute('begin', 'b' * 32, 2222, '192.0.2.10')
+        self.assertEqual(result['status'], 'committed')
+        self.assertEqual(self.host.ports, [2222])
+        self.assertFalse(self.host.armed)
+
+    def test_committed_retry_rejects_live_listener_drift(self):
+        self.begin()
+        self.transition.execute('finalize', TOKEN)
+        self.transition.execute('commit', TOKEN)
+        self.host.ports = [22, 2222]
+        with self.assertRaises(AccessError):
+            self.transition.execute('begin', 'b' * 32, 2222, '192.0.2.10')
+
+    def test_preexisting_final_policy_is_preserved_during_security_activation(self):
+        self.host.existing_final = True
+        self.host.ports = [2222]
+        result = self.transition.execute('begin', TOKEN, 2222, '192.0.2.10')
+        self.assertEqual(result['status'], 'transition')
+        self.assertTrue(self.host.armed)
+        self.assertFalse(self.host.firewall)
+        self.transition.execute('finalize', TOKEN)
+        self.transition.execute('commit', TOKEN)
+        self.assertEqual(self.transition.load()['status'], 'committed')
+        self.assertFalse(self.host.path('/etc/systemd/system/ssh.service.d/apex-init.conf').exists())
+
+    def test_reject_symlink_state_ancestor(self):
+        outside = self.host.path('/outside')
+        outside.mkdir()
+        self.host.path('/var').mkdir()
+        self.host.path('/var/lib').symlink_to(outside)
+        with self.assertRaises(AccessError):
+            self.begin()
+        self.assertEqual(list(outside.iterdir()), [])
+
+    def test_policy_rejects_additive_ports(self):
+        def runner(args, **kwargs):
+            text = 'port 22\nport 2222\n' if '-T' in args else ''
+            return subprocess.CompletedProcess(args, 0, text, '')
+        with self.assertRaises(AccessError):
+            Host(self.host.root, runner).policy([2222], True, '192.0.2.10')
+
+    def test_atomic_snapshot_does_not_follow_predictable_temporary_link(self):
+        destination = self.host.root / 'snapshot'
+        victim = self.host.root / 'victim'
+        victim.write_bytes(b'preserved')
+        destination.with_name('snapshot.new').symlink_to(victim)
+        atomic(destination, b'new snapshot')
+        self.assertEqual(destination.read_bytes(), b'new snapshot')
+        self.assertEqual(victim.read_bytes(), b'preserved')
+
+
+class LivePolicyTests(unittest.TestCase):
+    def setUp(self):
+        temporary = tempfile.TemporaryDirectory()
+        self.addCleanup(temporary.cleanup)
+        self.root = Path(temporary.name).resolve()
+        self.arguments = ['/usr/sbin/sshd', '-D', '-p', '2222']
+        self.running_arguments = self.arguments[:]
+        self.effective = ('port 2222\npermitrootlogin no\npasswordauthentication no\n'
+                          'kbdinteractiveauthentication no\npubkeyauthentication yes\n'
+                          'authenticationmethods publickey\nallowusers adam\n'
+                          'x11forwarding no\npermitemptypasswords no\n')
+        self.listeners = 'LISTEN 0 128 0.0.0.0:2222 0.0.0.0:* users:(("sshd",pid=755,fd=6))\n'
+        self.calls = []
+        self.host = Host(self.root, self.runner)
+        atomic(self.host.path('/etc/ssh/sshd_config'), b'# fixture\n')
+        process = self.root / 'proc/755'
+        process.mkdir(parents=True)
+        (process / 'exe').symlink_to('/usr/sbin/sshd')
+        self.cmdline()
+
+    def cmdline(self, title=False):
+        data = ('sshd: ' + ' '.join(self.running_arguments) + ' [listener] 0 of 10-100 startups\0'
+                if title else '\0'.join(self.running_arguments) + '\0')
+        (self.root / 'proc/755/cmdline').write_bytes(data.encode())
+
+    def runner(self, args, **kwargs):
+        self.calls.append(args)
+        if args[:2] in (['systemctl', 'is-active'], ['systemctl', 'is-enabled']) and args[-1] == 'ssh.socket':
+            return subprocess.CompletedProcess(args, 3, '', '')
+        if 'ExecStart' in args:
+            output = '{ path=/usr/sbin/sshd ; argv[]=' + ' '.join(self.arguments) + ' ; ignore_errors=no ; }'
+        elif 'MainPID' in args:
+            output = '755\n'
+        elif 'KillMode' in args:
+            output = 'process\n'
+        elif args[0] == 'ss':
+            output = self.listeners
+        elif '-T' in args:
+            output = self.effective
+        else:
+            output = ''
+        return subprocess.CompletedProcess(args, 0, output, '')
+
+    def test_verifies_exact_live_arguments_without_inventing_auth_options(self):
+        self.cmdline(title=True)
+        self.host.verify_active([2222], True, '192.0.2.1')
+        effective_call = next(args for args in self.calls if '-T' in args)
+        self.assertEqual(effective_call[:3], ['/usr/sbin/sshd', '-p', '2222'])
+        self.assertNotIn('PermitRootLogin=no', effective_call)
+
+    def test_password_or_root_enabled_on_live_daemon_is_rejected(self):
+        for setting in ('passwordauthentication', 'permitrootlogin'):
+            with self.subTest(setting=setting):
+                previous = self.effective
+                self.effective = previous.replace(setting + ' no', setting + ' yes')
+                with self.assertRaises(AccessError):
+                    self.host.verify_active([2222], True, '192.0.2.1')
+                self.effective = previous
+
+    def test_inherited_final_restrictions_cannot_silently_allow_other_accounts(self):
+        for previous, conflicting in (('allowusers adam', 'allowusers adam\nallowusers provider'),
+                                      ('x11forwarding no', 'x11forwarding yes'),
+                                      ('permitemptypasswords no', 'permitemptypasswords yes')):
+            with self.subTest(conflicting=conflicting):
+                original = self.effective
+                self.effective = original.replace(previous, conflicting)
+                with self.assertRaises(AccessError):
+                    self.host.verify_active([2222], True, '192.0.2.1')
+                self.effective = original
+
+    def test_bootstrap_prerequisites_do_not_require_adam_or_sudo_or_write_state(self):
+        self.arguments = self.running_arguments = ['/usr/sbin/sshd', '-D']
+        self.cmdline()
+        def unseeded(args, **kwargs):
+            if args[0] in ('visudo', 'runuser', 'sudo'):
+                raise AssertionError('administrator tools do not exist before baseline')
+            return self.runner(args, **kwargs)
+        self.host.runner = unseeded
+        before = sorted(self.root.rglob('*'))
+        with patch('provisioning.access.Host', return_value=self.host), \
+                patch('provisioning.access.os.geteuid', return_value=0):
+            self.assertEqual(main(['preflight']), 0)
+        self.assertEqual(sorted(self.root.rglob('*')), before)
+        self.assertIn(['/usr/sbin/sshd', '-t'], self.calls)
+        self.assertTrue(any('MainPID' in args for args in self.calls))
+
+    def test_bootstrap_prerequisites_reject_socket_and_live_invocation_drift(self):
+        self.arguments = self.running_arguments = ['/usr/sbin/sshd', '-D']
+        self.cmdline()
+        for failure in ('socket', 'invocation'):
+            with self.subTest(failure=failure):
+                def incompatible(args, failure=failure, **kwargs):
+                    if failure == 'socket' and args == ['systemctl', 'is-enabled', '--quiet', 'ssh.socket']:
+                        return subprocess.CompletedProcess(args, 0, '', '')
+                    return self.runner(args, **kwargs)
+                self.host.runner = incompatible
+                self.running_arguments = ['/usr/sbin/sshd', '-D', '-p', '22'] if failure == 'invocation' else self.arguments
+                self.cmdline()
+                with self.assertRaises(AccessError):
+                    self.host.bootstrap_preflight()
+
+    def test_later_preflight_still_checks_administrator_sudo(self):
+        self.arguments = self.running_arguments = ['/usr/sbin/sshd', '-D']
+        self.cmdline()
+        def bad_sudo(args, **kwargs):
+            if args[:3] == ['runuser', '-u', 'adam']:
+                return subprocess.CompletedProcess(args, 1, '', 'sudo denied')
+            return self.runner(args, **kwargs)
+        self.host.runner = bad_sudo
+        with self.assertRaises(AccessError):
+            self.host.preflight()
+
+    def test_bootstrap_follows_recursive_relative_and_quoted_include_paths(self):
+        self.arguments = self.running_arguments = ['/usr/sbin/sshd', '-D']
+        self.cmdline()
+        main_config = self.host.path('/etc/ssh/sshd_config')
+        first = self.host.path('/etc/ssh/sshd_config.d/10-first.conf')
+        second = self.host.path('/etc/ssh/provider # policy.conf')
+        atomic(main_config, b'Include sshd_config.*/[0-9]*.conf /etc/ssh/missing/*.conf\n')
+        atomic(first, b'iNcLuDe = "provider # policy.conf" # relative to /etc/ssh\n')
+        atomic(second, b'# Match Address 203.0.113.0/24\nBanner "/etc/Match banner"\n')
+        self.host.bootstrap_preflight()
+        second.write_text('"mAtCh" Address 203.0.113.0/24\n PasswordAuthentication yes\n')
+        before = {path: path.read_bytes() for path in (main_config, first, second)}
+        with self.assertRaisesRegex(AccessError, 'Match.*manual reconciliation'):
+            self.host.bootstrap_preflight()
+        self.assertEqual({path: path.read_bytes() for path in before}, before)
+
+    def test_bootstrap_rejects_recursive_or_oversized_config_without_mutation(self):
+        self.arguments = self.running_arguments = ['/usr/sbin/sshd', '-D']
+        self.cmdline()
+        configuration = self.host.path('/etc/ssh/sshd_config')
+        for content in ('Include sshd_config\n', '#' + 'x' * (1024 * 1024 + 1)):
+            with self.subTest(recursive=content.startswith('Include')):
+                configuration.write_text(content)
+                with self.assertRaisesRegex(AccessError, 'manual reconciliation'):
+                    self.host.bootstrap_preflight()
+                self.assertEqual(configuration.read_text(), content)
+
+    def test_extended_include_glob_tokens_are_rejected_even_without_matches(self):
+        self.arguments = self.running_arguments = ['/usr/sbin/sshd', '-D']
+        self.cmdline()
+        configuration = self.host.path('/etc/ssh/sshd_config')
+        for pattern in ('provider[a[:digit:]].conf', 'provider[a[.x.]].conf',
+                        'provider[a[=x=]].conf', 'provider[:digit:].conf'):
+            with self.subTest(pattern=pattern):
+                content = 'Include ' + pattern + '\n'
+                configuration.write_text(content)
+                with self.assertRaisesRegex(AccessError, 'glob syntax.*manual reconciliation'):
+                    self.host.bootstrap_preflight()
+                self.assertEqual(configuration.read_text(), content)
+
+    def test_unrestarted_daemon_with_old_arguments_is_rejected(self):
+        self.running_arguments = ['/usr/sbin/sshd', '-D', '-p', '22']
+        self.cmdline()
+        with self.assertRaises(AccessError):
+            self.host.verify_active([2222], True, '192.0.2.1')
+
+    def test_extra_live_listener_port_is_rejected(self):
+        self.listeners += 'LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=755,fd=7))\n'
+        with self.assertRaises(AccessError):
+            self.host.verify_active([2222], True, '192.0.2.1')
+
+    def test_additional_sshd_service_listener_is_rejected(self):
+        self.listeners += 'LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=999,fd=7))\n'
+        with self.assertRaises(AccessError):
+            self.host.verify_active([2222], True, '192.0.2.1')
+
+    def test_runtime_override_changes_fingerprint(self):
+        before = self.host.fingerprint()
+        atomic(self.host.path('/run/systemd/system/ssh.service.d/override.conf'), b'[Service]\nExecStart=/usr/sbin/sshd -D\n')
+        self.assertNotEqual(before, self.host.fingerprint())
+
+    def test_managed_file_does_not_exempt_drifted_service_from_preflight(self):
+        atomic(self.host.path('/etc/systemd/system/ssh.service.d/apex-init.conf'),
+               b'[Service]\nExecStart=\nExecStart=/usr/sbin/sshd -p 2222 -o PermitRootLogin=no -D\n')
+        with self.assertRaisesRegex(AccessError, 'differs from its managed invocation'):
+            self.host.preflight()
+
+
+@unittest.skipUnless(shutil.which('sshd') and shutil.which('ssh-keygen'), 'OpenSSH tools unavailable')
+class OpenSSHPolicyTests(unittest.TestCase):
+    """Evaluate the managed options with a real daemon without opening sockets."""
+    def setUp(self):
+        temporary = tempfile.TemporaryDirectory()
+        self.addCleanup(temporary.cleanup)
+        self.root = Path(temporary.name)
+        self.key = self.root / 'host-key'
+        self.config = self.root / 'etc/ssh/sshd_config'
+        self.config.parent.mkdir(parents=True)
+        self.config.write_text('PasswordAuthentication yes\nPermitRootLogin yes\n'
+                               'X11Forwarding yes\nPermitEmptyPasswords yes\n')
+        subprocess.run(['ssh-keygen', '-q', '-t', 'ed25519', '-N', '', '-f', str(self.key)],
+                       check=True, capture_output=True, timeout=30)
+        self.host = Host(self.root, self.runner)
+        original_path = self.host.path
+        self.host.path = lambda path: Path(path) if Path(path).is_relative_to(self.root) else original_path(path)
+
+    def runner(self, args, **kwargs):
+        if args[0] != '/usr/sbin/sshd':
+            code = 3 if args[-1] == 'ssh.socket' else 0
+            output = 'process\n' if 'KillMode' in args else ''
+            return subprocess.CompletedProcess(args, code, output, '')
+        return subprocess.run([shutil.which('sshd'), '-f', str(self.config), '-h', str(self.key), *args[1:]],
+                              **kwargs)
+
+    def effective(self, arguments, user):
+        result = self.runner(arguments + ['-T', '-C', 'user=' + user + ',host=localhost,addr=192.0.2.1'],
+                             capture_output=True, text=True, timeout=30)
+        self.assertEqual(result.returncode, 0, result.stderr)
+        return result.stdout.splitlines()
+
+    def test_transition_preserves_provider_access_and_only_final_policy_excludes_it(self):
+        transition = self.host.policy([22, 2222], False, '192.0.2.1')
+        transitional = self.effective(transition, 'provider')
+        self.assertFalse(any(line.startswith('allowusers ') for line in transitional))
+        self.assertIn('passwordauthentication yes', transitional)
+        final = self.host.policy([2222], True, '192.0.2.1')
+        for user in ('adam', 'provider'):
+            with self.subTest(user=user):
+                effective = self.effective(final, user)
+                allowed = [line.removeprefix('allowusers ') for line in effective if line.startswith('allowusers ')]
+                self.assertEqual(allowed, ['adam'])
+                self.assertNotIn('provider', allowed)
+                self.assertIn('x11forwarding no', effective)
+                self.assertIn('permitemptypasswords no', effective)
+
+    def test_additive_inherited_allowusers_stops_finalization(self):
+        with self.config.open('a') as stream:
+            stream.write('AllowUsers provider\n')
+        self.host.policy([22, 2222], False, '192.0.2.1')
+        with self.assertRaises(AccessError):
+            self.host.policy([2222], True, '192.0.2.1')
+
+    def test_provider_account_match_block_cannot_bypass_final_restrictions(self):
+        provider = pwd.struct_passwd(('provider', 'x', 1001, 1001, '', '/home/provider', '/bin/bash'))
+        for setting in ('AllowUsers provider', 'X11Forwarding yes', 'PermitEmptyPasswords yes'):
+            with self.subTest(setting=setting):
+                self.config.write_text('Match User provider\n ' + setting + '\n')
+                with patch('pwd.getpwall', return_value=[provider]), self.assertRaises(AccessError):
+                    self.host.policy([2222], True, '192.0.2.1')
+
+    def test_bootstrap_rejects_match_address_in_main_or_recursively_included_file(self):
+        conditional = ('\n AllowUsers root provider\n PasswordAuthentication yes\n'
+                       ' PermitRootLogin yes\n AuthenticationMethods any\n'
+                       ' X11Forwarding yes\n PermitEmptyPasswords yes\n')
+        first = self.root / 'first # include.conf'
+        second = self.root / 'second include.conf'
+        for directive, included in (('Match Address', False), ('mAtCh=Address', False),
+                                    ('"Match" Address', False), ('"mAtCh" Address', True)):
+            with self.subTest(directive=directive, included=included):
+                payload = directive + ' 203.0.113.0/24' + conditional
+                if included:
+                    second.write_text(payload)
+                    first.write_text('iNcLuDe "' + str(second) + '"\n')
+                    self.config.write_text('"Include" "' + str(first) + '" # provider policy\n')
+                else:
+                    self.config.write_text(payload)
+                arguments = ['/usr/sbin/sshd', '-T', '-p', '2222', '-o', 'AllowUsers=adam',
+                             '-o', 'PermitRootLogin=no', '-o', 'PasswordAuthentication=no',
+                             '-o', 'AuthenticationMethods=publickey', '-o', 'X11Forwarding=no',
+                             '-o', 'PermitEmptyPasswords=no']
+                other_address = self.runner(arguments + ['-C', 'user=root,host=localhost,addr=203.0.113.25'],
+                                            capture_output=True, text=True, timeout=30)
+                self.assertEqual(other_address.returncode, 0, other_address.stderr)
+                self.assertIn('permitrootlogin yes', other_address.stdout.splitlines())
+                self.assertIn('passwordauthentication yes', other_address.stdout.splitlines())
+                before = self.config.read_bytes()
+                with patch.object(self.host, 'service_arguments', return_value=['/usr/sbin/sshd', '-D']), \
+                        patch.object(self.host, 'live_arguments'), \
+                        self.assertRaisesRegex(AccessError, 'Match.*manual reconciliation'):
+                    self.host.bootstrap_preflight()
+                self.assertEqual(self.config.read_bytes(), before)
+
+    def test_posix_class_include_cannot_hide_a_match_from_preflight(self):
+        included = self.root / 'provider1.conf'
+        included.write_text('Match Address 203.0.113.0/24\n PermitRootLogin yes\n')
+        for pattern in ('provider[[:digit:]].conf', 'provider[a[:digit:]].conf'):
+            with self.subTest(pattern=pattern):
+                self.config.write_text('Include ' + str(self.root / pattern) + '\n')
+                effective = self.runner(['/usr/sbin/sshd', '-T', '-o', 'PermitRootLogin=no',
+                                         '-C', 'user=root,host=localhost,addr=203.0.113.25'],
+                                        capture_output=True, text=True, timeout=30)
+                self.assertEqual(effective.returncode, 0, effective.stderr)
+                self.assertIn('permitrootlogin yes', effective.stdout.splitlines())
+                with patch.object(self.host, 'service_arguments', return_value=['/usr/sbin/sshd', '-D']), \
+                        patch.object(self.host, 'live_arguments'), \
+                        self.assertRaisesRegex(AccessError, 'manual reconciliation'):
+                    self.host.bootstrap_preflight()
+
+
+if __name__ == '__main__':
+    unittest.main()

ci/scripts/init-node/tests/test_provision_baseline.py

@@ -0,0 +1,510 @@
+"""Exercise baseline decisions with a temporary host filesystem and fake commands."""
+import hashlib
+import importlib.util
+import io
+from pathlib import Path
+import subprocess
+import tempfile
+import unittest
+from unittest.mock import patch
+
+from provisioning import baseline
+
+
+class Host:
+    def __init__(self, root):
+        self.root = root
+        self.commands = []
+        self.installed = {}
+        self.partial = {}
+        self.active = set()
+        self.enabled = set()
+        self.masked = set()
+        self.fail_install = False
+        self.plan = ''
+        self.zfs_available = True
+        self.docker_root = '/srv/docker'
+        self.bouncer_key = 'testOnlyGeneratedCredential1234567890'
+        self.registration_failure = False
+        self.registrations = {}
+        self.path('/etc/os-release').parent.mkdir(parents=True)
+        self.path('/etc/os-release').write_text('ID=debian\nVERSION_ID="13"\n')
+        self.path('/etc/debian_version').touch()
+        self.path('/usr/sbin').mkdir(parents=True)
+
+    def path(self, path):
+        path = Path(path)
+        if str(path).startswith(str(self.root)):
+            return path
+        return self.root / str(path).lstrip('/')
+
+    def command(self, args, **kwargs):
+        self.commands.append(args)
+        code, output = 0, ''
+        if args[:2] == ['dpkg', '--print-architecture']:
+            output = 'amd64\n'
+        elif args[0] == 'dpkg-query':
+            output = ''.join(f'{name}\tinstall ok installed\t{version}\n' for name, version in self.installed.items())
+            output += ''.join(f'{name}\t{status}\t1.0\n' for name, status in self.partial.items())
+        elif args == ['uname', '-r']:
+            output = '6.12.0-amd64\n'
+        elif args[0] == 'apt-get' and 'install' in args:
+            if '--simulate' in args:
+                output = self.plan
+            elif self.fail_install:
+                code = 100
+            else:
+                start = args.index('--no-remove') + 1
+                # Reproduce dpkg configuring the bouncer before CrowdSec in one batch.
+                if 'crowdsec-firewall-bouncer-iptables' in args[start:]:
+                    config = self.path('/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml')
+                    config.parent.mkdir(parents=True, exist_ok=True)
+                    if not config.exists():
+                        key = 'packageGeneratedCredential123456789' if 'crowdsec' in self.installed else '<API_KEY>'
+                        config.write_text(f'api_url: http://127.0.0.1:8080/\napi_key: {key}\nmode: iptables\n')
+                        config.chmod(0o640)
+                for name in args[start:]:
+                    self.installed[name] = '1.0'
+                    self.partial.pop(name, None)
+        elif args[:4] == ['cscli', '-oraw', 'bouncers', 'add']:
+            if self.registration_failure:
+                code, output = 1, 'sensitive diagnostic'
+            else:
+                output = self.bouncer_key + '\n'
+                self.registrations[args[4]] = self.bouncer_key
+        elif args[0] == 'systemctl':
+            service = args[-1]
+            operation = args[1]
+            if operation == 'is-active':
+                code = 0 if service in self.active else 3
+            elif operation == 'is-enabled':
+                code = 0 if service in self.enabled else 1
+                output = 'masked\n' if service in self.masked else ''
+            elif operation == 'start':
+                conditions = []
+                for dropin in self.path(f'/etc/systemd/system/{service}.service.d').glob('*.conf'):
+                    conditions.extend(line.partition('=')[2] for line in dropin.read_text().splitlines()
+                                      if line.startswith('ConditionPathExists='))
+                if service not in self.masked and all(self.path(path).exists() for path in conditions):
+                    self.active.add(service)
+            elif operation == 'enable':
+                self.enabled.add(service)
+            elif operation == 'disable':
+                self.enabled.discard(service)
+            elif operation == 'mask':
+                self.masked.add(service)
+                unit = self.path(f'/etc/systemd/system/{service}.service')
+                unit.parent.mkdir(parents=True, exist_ok=True)
+                unit.symlink_to('/dev/null')
+            elif operation == 'unmask':
+                self.masked.discard(service)
+                unit = self.path(f'/etc/systemd/system/{service}.service')
+                if unit.is_symlink():
+                    unit.unlink()
+        elif args[:2] == ['docker', 'info']:
+            output = self.docker_root + '\n'
+        elif args[:3] == ['docker', 'compose', 'version']:
+            output = '2.39.2\n'
+        elif args == ['modprobe', 'zfs']:
+            code = 0 if self.zfs_available else 1
+        elif args == ['zfs', 'version']:
+            output = 'zfs-2.3.2\nzfs-kmod-2.3.2\n'
+        return subprocess.CompletedProcess(args, code, output, 'installation failed' if code else '')
+
+
+class BaselineTests(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.temp.cleanup)
+        self.host = Host(Path(self.temp.name).resolve())
+        self.helper = b'#!/bin/bash\necho pinned test helper\n'
+        patches = [
+            patch.object(baseline, 'Path', self.host.path),
+            patch.object(baseline, 'POLICY', self.host.path('/usr/sbin/policy-rc.d')),
+            patch.object(baseline, 'POLICY_STATE', self.host.path('/usr/sbin/.apex-policy-rc.d')),
+            patch.object(baseline, 'SECURITY_STATE', self.host.path('/var/lib/apex/baseline-security.json')),
+            patch.object(baseline.os, 'geteuid', return_value=Path(self.temp.name).stat().st_uid),
+            patch.object(baseline.subprocess, 'run', side_effect=self.host.command),
+            patch.object(baseline.urllib.request, 'urlopen', side_effect=self.download),
+            patch.object(baseline, 'HELPER_SHA256', hashlib.sha256(self.helper).hexdigest()),
+        ]
+        for item in patches:
+            item.start()
+            self.addCleanup(item.stop)
+
+    def download(self, url, **kwargs):
+        if url == baseline.HELPER_URL:
+            return io.BytesIO(self.helper)
+        return io.BytesIO(b'-----BEGIN PGP PUBLIC KEY BLOCK-----\nfixture\n')
+
+    def apply(self):
+        # The filesystem remains owned by the test uid; only root preflight is isolated.
+        with patch.object(baseline, 'preflight'):
+            return baseline.apply_baseline()
+
+    def test_package_database_drives_installation_and_repeat_does_not_upgrade(self):
+        self.host.installed['curl'] = '7.0'
+        self.host.partial['rsyslog'] = 'deinstall ok config-files'
+        self.assertEqual(baseline.install_missing(['curl', 'rsyslog']), ['rsyslog'])
+        self.assertEqual(self.host.installed['curl'], '7.0')
+        self.assertEqual(self.host.installed['rsyslog'], '1.0')
+        count = len(self.host.commands)
+        self.assertEqual(baseline.install_missing(['curl', 'rsyslog']), [])
+        self.assertFalse(any(command[0] == 'apt-get' for command in self.host.commands[count:]))
+
+    def test_dependency_upgrade_is_rejected_before_install(self):
+        self.host.plan = 'Inst libc6 [2.0] (3.0 Debian:13)\n'
+        with self.assertRaisesRegex(baseline.BaselineError, 'upgrade/removal'):
+            baseline.install_missing(['git'])
+        self.assertNotIn('git', self.host.installed)
+
+    def test_held_installed_packages_are_preserved(self):
+        self.host.partial['git'] = 'hold ok installed'
+        self.assertEqual(baseline.install_missing(['git']), [])
+        self.assertFalse(any(command[0] == 'apt-get' for command in self.host.commands))
+
+    def test_conflicting_docker_config_is_rejected_without_host_mutation(self):
+        path = self.host.path('/etc/docker/daemon.json')
+        path.parent.mkdir(parents=True)
+        path.write_text('{"data-root": "/data/docker"}')
+        before = sorted(str(path.relative_to(self.host.root)) for path in self.host.root.rglob('*'))
+        with self.assertRaisesRegex(baseline.BaselineError, 'Conflicting Docker'):
+            self.apply()
+        self.assertEqual(before, sorted(str(path.relative_to(self.host.root)) for path in self.host.root.rglob('*')))
+        self.assertFalse(any(command[0] in ('apt-get', 'systemctl') for command in self.host.commands))
+
+    def test_existing_docker_data_is_preserved(self):
+        data = self.host.path('/var/lib/docker/volumes/user-data')
+        data.parent.mkdir(parents=True)
+        data.write_text('valuable')
+        with self.assertRaisesRegex(baseline.BaselineError, 'Existing /var/lib/docker'):
+            self.apply()
+        self.assertEqual(data.read_text(), 'valuable')
+
+    def test_conflicting_distribution_docker_package_is_never_removed(self):
+        self.host.installed['docker.io'] = '26.1.5'
+        with self.assertRaisesRegex(baseline.BaselineError, 'Conflicting Docker packages'):
+            self.apply()
+        self.assertEqual(self.host.installed['docker.io'], '26.1.5')
+        self.assertFalse(any(command[0] == 'apt-get' for command in self.host.commands))
+
+    def test_unmanaged_destination_data_is_not_adopted(self):
+        data = self.host.path('/srv/docker/private-data')
+        data.parent.mkdir(parents=True)
+        data.write_text('preserve me')
+        with self.assertRaisesRegex(baseline.BaselineError, 'Unmanaged /srv/docker'):
+            self.apply()
+        self.assertEqual(data.read_text(), 'preserve me')
+
+    def test_package_failure_restores_existing_service_policy(self):
+        policy = self.host.path('/usr/sbin/policy-rc.d')
+        policy.write_text('#!/bin/sh\nexit 42\n')
+        policy.chmod(0o751)
+        self.host.fail_install = True
+        with self.assertRaises(baseline.BaselineError):
+            self.apply()
+        self.assertEqual(policy.read_text(), '#!/bin/sh\nexit 42\n')
+        self.assertEqual(policy.stat().st_mode & 0o777, 0o751)
+        self.assertFalse(self.host.path('/usr/sbin/.apex-policy-rc.d').exists())
+        self.assertFalse(self.host.masked)
+
+    def test_service_policy_symlink_is_restored_after_failure(self):
+        policy = self.host.path('/usr/sbin/policy-rc.d')
+        original = self.host.path('/usr/sbin/provider-policy')
+        original.write_text('provider policy')
+        policy.symlink_to(original)
+        with self.assertRaisesRegex(RuntimeError, 'boom'):
+            with baseline.service_start_guard():
+                self.assertFalse(policy.is_symlink())
+                raise RuntimeError('boom')
+        self.assertTrue(policy.is_symlink())
+        self.assertEqual(original.read_text(), 'provider policy')
+
+    def test_interrupted_guard_is_recovered_before_next_attempt(self):
+        state = self.host.path('/usr/sbin/.apex-policy-rc.d')
+        state.mkdir()
+        (state / 'original').write_text('original policy')
+        (state / 'ready').touch()
+        self.host.path('/usr/sbin/policy-rc.d').write_text('#!/bin/sh\nexit 101\n')
+        with baseline.service_start_guard():
+            pass
+        self.assertEqual(self.host.path('/usr/sbin/policy-rc.d').read_text(), 'original policy')
+
+    def test_external_policy_change_is_not_overwritten(self):
+        policy = self.host.path('/usr/sbin/policy-rc.d')
+        with self.assertRaisesRegex(baseline.BaselineError, 'changed outside APEX'):
+            with baseline.service_start_guard():
+                policy.write_text('external policy')
+        self.assertEqual(policy.read_text(), 'external policy')
+
+    def test_baseline_reports_capabilities_and_repeats_without_restarts(self):
+        result = self.apply()
+        self.assertEqual(result['capabilities']['docker_data_root'], '/srv/docker')
+        self.assertEqual(result['capabilities']['compose'], '2.39.2')
+        self.assertIn('zfs-kmod-2.3.2', result['capabilities']['zfs'])
+        self.assertEqual(result['packages']['zfs-dkms'], '1.0')
+        self.assertEqual(self.host.path('/etc/apt/apt.conf.d/99-apex').read_text(), 'APT::Keep-Downloaded-Packages "0";\n')
+        self.assertFalse(self.host.path('/usr/sbin/policy-rc.d').exists())
+        self.assertFalse(self.host.masked)
+        self.assertNotIn('ufw', self.host.active)
+        self.assertNotIn('crowdsec-firewall-bouncer', self.host.enabled)
+        self.assertFalse(any(command[0] in ('ufw', '/usr/local/bin/ufw-docker') for command in self.host.commands))
+        count = len(self.host.commands)
+        again = self.apply()
+        self.assertEqual(again, result)
+        repeated = self.host.commands[count:]
+        self.assertFalse(any(command[0] == 'apt-get' or command[:2] in
+                             (['systemctl', 'restart'], ['systemctl', 'start']) for command in repeated))
+
+    def bouncer_config(self, key='<API_KEY>'):
+        self.host.installed.update({'crowdsec': '1.8.1', 'crowdsec-firewall-bouncer-iptables': '0.0.36'})
+        config = self.host.path('/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml')
+        config.parent.mkdir(parents=True, exist_ok=True)
+        config.write_text(f'# custom settings\napi_url: http://127.0.0.1:9090/\napi_key: {key} # keep comment\nmode: iptables\nblacklists_ipv4: [custom]\n')
+        config.chmod(0o640)
+        return config
+
+    def test_fresh_crowdsec_is_configured_before_bouncer_install(self):
+        self.apply()
+        installs = [args for args in self.host.commands if args[0] == 'apt-get' and 'install' in args and '--simulate' not in args]
+        crowdsec = next(index for index, args in enumerate(installs) if 'crowdsec' in args)
+        bouncer = next(index for index, args in enumerate(installs) if 'crowdsec-firewall-bouncer-iptables' in args)
+        self.assertLess(crowdsec, bouncer)
+        self.assertFalse(self.host.registrations)
+        self.assertFalse(self.host.active & {'crowdsec', 'crowdsec-firewall-bouncer'})
+
+    def test_broken_installed_bouncer_is_repaired_and_repeat_preserves_registration(self):
+        config = self.bouncer_config()
+        before = config.read_text()
+        owner = (config.stat().st_uid, config.stat().st_gid)
+        self.apply()
+        self.assertEqual(config.read_text(), before.replace('<API_KEY>', self.host.bouncer_key))
+        registration = config.with_suffix('.yaml.id').read_text().strip()
+        self.assertEqual(self.host.registrations[registration], self.host.bouncer_key)
+        self.assertEqual((config.stat().st_uid, config.stat().st_gid), owner)
+        self.assertEqual(config.stat().st_mode & 0o777, 0o640)
+        count = len(self.host.registrations)
+        self.apply()
+        self.assertEqual(len(self.host.registrations), count)
+        self.assertFalse(self.host.active & {'crowdsec', 'crowdsec-firewall-bouncer'})
+
+    def test_existing_bouncer_key_and_custom_settings_are_preserved(self):
+        config = self.bouncer_config('existing-custom-key')
+        before = config.read_bytes()
+        self.apply()
+        self.assertEqual(config.read_bytes(), before)
+        self.assertFalse(self.host.registrations)
+
+    def test_registration_failure_leaves_placeholder_without_exposing_output(self):
+        config = self.bouncer_config()
+        before = config.read_bytes()
+        self.host.registration_failure = True
+        with self.assertRaises(baseline.BaselineError) as failure:
+            self.apply()
+        self.assertEqual(config.read_bytes(), before)
+        self.assertFalse(config.with_suffix('.yaml.id').exists())
+        self.assertNotIn('sensitive diagnostic', str(failure.exception))
+
+    def test_invalid_registered_key_is_not_published(self):
+        for key in ('', '<API_KEY>', 'not a key\nsensitive diagnostic'):
+            with self.subTest(key=key):
+                config = self.bouncer_config()
+                before = config.read_bytes()
+                self.host.bouncer_key = key
+                with self.assertRaises(baseline.BaselineError):
+                    self.apply()
+                self.assertEqual(config.read_bytes(), before)
+                self.assertFalse(config.with_suffix('.yaml.id').exists())
+
+    def test_bouncer_config_symlink_is_rejected_before_registration(self):
+        config = self.bouncer_config()
+        destination = config.with_suffix('.provider')
+        config.rename(destination)
+        config.symlink_to(destination)
+        with self.assertRaises(baseline.BaselineError):
+            self.apply()
+        self.assertIn('<API_KEY>', destination.read_text())
+        self.assertFalse(self.host.registrations)
+
+    def test_quoted_null_is_an_existing_key_not_an_uninitialized_yaml_value(self):
+        config = self.bouncer_config('"null"')
+        before = config.read_bytes()
+        self.apply()
+        self.assertEqual(config.read_bytes(), before)
+        self.assertFalse(self.host.registrations)
+
+    def test_package_placeholder_variants_are_repaired(self):
+        for placeholder in ('"<API_KEY>"', "'$API_KEY'", '${API_KEY}', 'null', '~', '', '""'):
+            with self.subTest(placeholder=placeholder):
+                config = self.bouncer_config(placeholder)
+                self.apply()
+                self.assertIn('api_key: ' + self.host.bouncer_key, config.read_text())
+
+    def test_ambiguous_or_missing_api_key_is_not_mutated(self):
+        for content in ('api_key: <API_KEY>\n"api_key": existing-key\n', 'api_url: http://localhost:8080/\n',
+                        'api_key: "<API_KEY>x\n'):
+            with self.subTest(content=content):
+                config = self.bouncer_config()
+                config.write_text(content)
+                with self.assertRaises(baseline.BaselineError):
+                    self.apply()
+                self.assertEqual(config.read_text(), content)
+                self.assertFalse(self.host.registrations)
+
+    def test_multiline_custom_credentials_are_preserved_without_registration(self):
+        for credential in ('\n  existing-custom-credential', ' # comment\n\n  existing-custom-credential',
+                           ' <API_KEY>\n  existing-custom-credential', '\n  # comment\n  existing-custom-credential'):
+            with self.subTest(credential=credential):
+                config = self.bouncer_config()
+                config.write_text('api_key:' + credential + '\napi_url: http://localhost:9090/\n')
+                identity = config.with_suffix('.yaml.id')
+                identity.write_text('existing-registration\n')
+                before = config.read_bytes()
+                with self.assertRaises(baseline.BaselineError):
+                    self.apply()
+                self.assertEqual(config.read_bytes(), before)
+                self.assertEqual(identity.read_text(), 'existing-registration\n')
+                self.assertFalse(self.host.registrations)
+
+    @unittest.skipUnless(importlib.util.find_spec('yaml'), 'target-only python3-yaml is unavailable')
+    def test_empty_credential_replacement_parses_as_the_generated_key(self):
+        import yaml
+        for entry in ('api_key:', 'api_key: # comment', 'api_key:    ',
+                      'api_key:\n  # next-line comment'):
+            with self.subTest(entry=entry):
+                config = self.bouncer_config()
+                config.write_text(entry + '\napi_url: http://localhost:9090/\n')
+                self.assertIsNone(yaml.safe_load(config.read_text())['api_key'])
+                self.apply()
+                parsed = yaml.safe_load(config.read_text())
+                self.assertEqual(parsed['api_key'], self.host.bouncer_key)
+                self.assertEqual(parsed['api_url'], 'http://localhost:9090/')
+                if '# ' in entry:
+                    self.assertIn(entry[entry.index('# '):], config.read_text())
+
+    def test_redirected_bookkeeping_is_rejected_before_registration(self):
+        config = self.bouncer_config()
+        destination = config.with_suffix('.provider')
+        destination.write_text('provider-data')
+        config.with_suffix('.yaml.id').symlink_to(destination)
+        with self.assertRaises(baseline.BaselineError):
+            self.apply()
+        self.assertEqual(destination.read_text(), 'provider-data')
+        self.assertFalse(self.host.registrations)
+
+    def test_interrupted_credential_publication_can_retry_without_reusing_wrong_identity(self):
+        config = self.bouncer_config()
+        replace = baseline.os.replace
+        def interrupt_config(source, destination):
+            if destination == config:
+                raise OSError('interrupted publication')
+            return replace(source, destination)
+        with patch.object(baseline.os, 'replace', side_effect=interrupt_config):
+            with self.assertRaises(baseline.BaselineError):
+                self.apply()
+        self.assertIn('<API_KEY>', config.read_text())
+        orphan = config.with_suffix('.yaml.id').read_text().strip()
+        self.assertIn(orphan, self.host.registrations)
+        self.apply()
+        current = config.with_suffix('.yaml.id').read_text().strip()
+        self.assertNotEqual(orphan, current)
+        self.assertIn(self.host.registrations[current], config.read_text())
+        self.assertFalse(list(config.parent.glob('.apex-*')))
+
+    def test_zfs_missing_for_running_kernel_is_an_explicit_incomplete_result(self):
+        self.host.zfs_available = False
+        with self.assertRaisesRegex(baseline.BaselineError, 'reboot.*retry'):
+            self.apply()
+
+    def test_narrow_base_policy_preserves_scope_and_shared_configuration(self):
+        with patch.object(baseline, 'preflight'):
+            result = baseline.apply_base_policy()
+        self.assertEqual(set(result['packages']), {'curl', 'wget', 'git', 'rsyslog', 'ufw'})
+        self.assertFalse(self.host.path('/etc/docker/daemon.json').exists())
+        self.assertEqual(self.host.path('/etc/systemd/journald.conf.d/99-apex.conf').read_text(),
+                         '[Journal]\nSystemMaxUse=200M\n')
+        self.assertIn(['/usr/local/bin/ufw-docker', 'install'], self.host.commands)
+
+    def test_unverified_download_is_not_installed_or_executed(self):
+        with patch.object(baseline.urllib.request, 'urlopen', return_value=io.BytesIO(b'tampered')):
+            with self.assertRaisesRegex(baseline.BaselineError, 'checksum mismatch'):
+                baseline.configure_common()
+        self.assertFalse(self.host.path('/usr/local/bin/ufw-docker').exists())
+        self.assertFalse(any(command[0] == '/usr/local/bin/ufw-docker' for command in self.host.commands))
+
+    def test_recovery_does_not_remove_preexisting_service_masks(self):
+        self.host.masked.add('ufw')
+        with baseline.security_service_guard(['ufw']):
+            self.host.installed['ufw'] = '0.36'
+        self.assertIn('ufw', self.host.masked)
+
+    def test_package_owned_etc_service_file_can_be_unpacked_under_guard(self):
+        unit = self.host.path('/etc/systemd/system/crowdsec-firewall-bouncer.service')
+        with baseline.security_service_guard(['crowdsec-firewall-bouncer-iptables']):
+            self.assertFalse(unit.is_symlink(), 'dpkg must be able to install its own conffile')
+            unit.parent.mkdir(parents=True, exist_ok=True)
+            unit.write_text('[Service]\nExecStart=/usr/bin/crowdsec-firewall-bouncer\n')
+            self.host.installed['crowdsec-firewall-bouncer-iptables'] = '0.0.36'
+            self.host.enabled.add('crowdsec-firewall-bouncer')
+            baseline.run(['systemctl', 'start', 'crowdsec-firewall-bouncer'])
+            self.assertNotIn('crowdsec-firewall-bouncer', self.host.active)
+        self.assertTrue(unit.is_file())
+        self.assertIn('ExecStart=/usr/bin/crowdsec-firewall-bouncer', unit.read_text())
+        self.assertNotIn('crowdsec-firewall-bouncer', self.host.enabled)
+        baseline.run(['systemctl', 'start', 'crowdsec-firewall-bouncer'])
+        self.assertIn('crowdsec-firewall-bouncer', self.host.active)
+
+    def test_disable_failure_retains_guard_and_retry_recovers_it(self):
+        original_command = self.host.command
+        def fail_disable(args, **kwargs):
+            if args == ['systemctl', 'disable', 'crowdsec-firewall-bouncer']:
+                return subprocess.CompletedProcess(args, 1, '', '')
+            return original_command(args, **kwargs)
+        with patch.object(baseline.subprocess, 'run', side_effect=fail_disable):
+            with self.assertRaises(baseline.BaselineError):
+                with baseline.security_service_guard(['crowdsec-firewall-bouncer-iptables']):
+                    self.host.installed['crowdsec-firewall-bouncer-iptables'] = '0.0.36'
+                    self.host.enabled.add('crowdsec-firewall-bouncer')
+        baseline.run(['systemctl', 'start', 'crowdsec-firewall-bouncer'])
+        self.assertNotIn('crowdsec-firewall-bouncer', self.host.active)
+        with baseline.security_service_guard(['crowdsec-firewall-bouncer-iptables']):
+            pass
+        self.assertNotIn('crowdsec-firewall-bouncer', self.host.enabled)
+        baseline.run(['systemctl', 'start', 'crowdsec-firewall-bouncer'])
+        self.assertIn('crowdsec-firewall-bouncer', self.host.active)
+
+    def test_generic_package_failure_does_not_suggest_zfs_reboot(self):
+        original_command = self.host.command
+        def fail_security_packages(args, **kwargs):
+            if args[0] == 'apt-get' and 'crowdsec-firewall-bouncer-iptables' in args and '--simulate' not in args:
+                return subprocess.CompletedProcess(args, 100, '', 'secret source diagnostic')
+            return original_command(args, **kwargs)
+        with patch.object(baseline.subprocess, 'run', side_effect=fail_security_packages):
+            with self.assertRaises(baseline.BaselineError) as failure:
+                self.apply()
+        self.assertNotIn('reboot', str(failure.exception))
+        self.assertNotIn('secret source diagnostic', str(failure.exception))
+
+    def test_unsupported_os_fails_before_package_commands(self):
+        self.host.path('/etc/os-release').write_text('ID=ubuntu\nVERSION_ID=24.04\n')
+        with patch.object(baseline.os, 'geteuid', return_value=0):
+            with self.assertRaisesRegex(baseline.BaselineError, 'Debian 13 amd64'):
+                baseline.apply_baseline()
+        self.assertEqual(self.host.commands, [])
+
+    def test_non_root_fails_before_any_command(self):
+        with patch.object(baseline.os, 'geteuid', return_value=1000):
+            with self.assertRaisesRegex(baseline.BaselineError, 'root'):
+                baseline.apply_baseline()
+        self.assertEqual(self.host.commands, [])
+
+    def test_unsupported_architecture_fails_before_apt_mutation(self):
+        with patch.object(baseline.os, 'geteuid', return_value=0), patch.object(
+                baseline.subprocess, 'run', return_value=subprocess.CompletedProcess([], 0, 'arm64\n', '')) as command:
+            with self.assertRaisesRegex(baseline.BaselineError, 'Debian 13 amd64'):
+                baseline.apply_baseline()
+        self.assertEqual([call.args[0] for call in command.call_args_list], [['dpkg', '--print-architecture']])
+
+
+if __name__ == '__main__':
+    unittest.main()

ci/scripts/init-node/tests/test_provision_contract.py

@@ -0,0 +1,230 @@
+import copy
+import unittest
+
+from provisioning.contract import ContractError, normalize_user_data, parse_bootstrap_env
+
+
+def example():
+    return {
+        'preserve_hostname': False,
+        'hostname': 'node1.a1.example.com',
+        'users': [{
+            'name': 'adam',
+            'sudo': 'ALL=(ALL) NOPASSWD:ALL',
+            'groups': ['sudo', 'docker'],
+            'shell': '/bin/bash',
+            'ssh_authorized_keys': ['ssh-ed25519 PUBLIC_TEST_KEY operator'],
+        }],
+        'write_files': [{
+            'path': '/etc/apex/bootstrap.env',
+            'owner': 'root:root',
+            'permissions': '0600',
+            'content': 'APEX_REPO_NAME="node1.example.com"\n'
+                       'APEX_REPO_URL="https://example.com/team/node1.git"\n',
+        }],
+        'runcmd': ['/usr/local/bin/apex-bootstrap'],
+    }
+
+
+def add_env(doc, text):
+    doc['write_files'][0]['content'] += text
+    return doc
+
+
+class TestBootstrapAssignments(unittest.TestCase):
+    def test_literal_quotes_comments_and_empty_values(self):
+        values = parse_bootstrap_env(
+            '# enrollment\nAPEX_REPO_NAME=node1.example.com\n'
+            "APEX_REPO_REF='release-1'\nAPEX_TIER1_DEVICE=\n")
+        self.assertEqual(values['APEX_REPO_REF'], 'release-1')
+        self.assertEqual(values['APEX_TIER1_DEVICE'], '')
+
+    def test_rejects_shell_forms_without_repeating_input(self):
+        payloads = [
+            'APEX_REPO_NAME=$(touch /tmp/SECRET_SENTINEL)',
+            'APEX_REPO_NAME="${SECRET_SENTINEL}"',
+            'APEX_REPO_NAME=`SECRET_SENTINEL`',
+            'APEX_REPO_NAME=x;SECRET_SENTINEL',
+            'export APEX_REPO_NAME=SECRET_SENTINEL',
+        ]
+        for payload in payloads:
+            with self.subTest(payload=payload):
+                with self.assertRaises(ContractError) as caught:
+                    parse_bootstrap_env(payload)
+                self.assertNotIn('SECRET_SENTINEL', str(caught.exception))
+
+    def test_rejects_duplicate_and_unknown_assignments(self):
+        for text in ('APEX_REPO_NAME=a\nAPEX_REPO_NAME=b\n', 'OTHER_SECRET=value\n'):
+            with self.subTest(text=text):
+                with self.assertRaises(ContractError):
+                    parse_bootstrap_env(text)
+
+    def test_rejects_unquoted_parentheses_and_inline_comments(self):
+        for value in ('(main)', 'main(', 'main)', ' #comment', 'main #comment'):
+            with self.subTest(value=value):
+                with self.assertRaises(ContractError):
+                    parse_bootstrap_env('APEX_REPO_REF=' + value)
+
+    def test_parentheses_and_hashes_remain_quoted_literals(self):
+        for quote in ('"', "'"):
+            for value in ('(main)', '#comment', 'main #comment'):
+                with self.subTest(quote=quote, value=value):
+                    values = parse_bootstrap_env('APEX_REPO_REF=' + quote + value + quote)
+                    self.assertEqual(values['APEX_REPO_REF'], value)
+
+
+class TestUserDataContract(unittest.TestCase):
+    def test_public_repository_and_directory_defaults(self):
+        spec = normalize_user_data(example())
+        self.assertEqual(spec.hostname, 'node1.a1.example.com')
+        self.assertEqual(spec.repository.name, 'node1.example.com')
+        self.assertIsNone(spec.repository.ref)
+        self.assertEqual([tier.mode for tier in spec.tiers], ['directory'] * 3)
+        self.assertEqual(spec.tiers[0].number, 1)
+
+    def test_ref_and_explicit_storage_intent(self):
+        doc = add_env(example(),
+            'APEX_REPO_REF="refs/tags/v1"\n'
+            'APEX_TIER1_MODE=create\nAPEX_TIER1_DEVICE=/dev/disk/by-id/disk-a\n'
+            'APEX_TIER2_MODE=import\nAPEX_TIER2_DEVICE=/dev/disk/by-id/disk-b\n'
+            'APEX_TIER2_POOL_GUID=1234567\n')
+        spec = normalize_user_data(doc)
+        self.assertEqual(spec.repository.ref, 'refs/tags/v1')
+        self.assertEqual(spec.tiers[0].mode, 'create')
+        self.assertEqual(spec.tiers[1].pool_guid, '1234567')
+
+    def test_legacy_device_requires_explicit_mode(self):
+        with self.assertRaisesRegex(ContractError, 'explicit mode'):
+            normalize_user_data(add_env(example(), 'APEX_TIER1_DEVICE=/dev/vdb\n'))
+
+    def test_storage_conflicts_are_rejected(self):
+        cases = [
+            'APEX_TIER1_MODE=create\n',
+            'APEX_TIER1_MODE=import\nAPEX_TIER1_DEVICE=/dev/vdb\n',
+            'APEX_TIER1_MODE=directory\nAPEX_TIER1_DEVICE=/dev/vdb\n',
+            'APEX_TIER1_MODE=create\nAPEX_TIER1_DEVICE=/dev/vdb\nAPEX_TIER1_POOL_GUID=1\n',
+            'APEX_TIER1_MODE=erase\n',
+            'APEX_TIER1_MODE=create\nAPEX_TIER1_DEVICE=/dev/vdb\n'
+            'APEX_TIER2_MODE=create\nAPEX_TIER2_DEVICE=/dev/vdb\n',
+        ]
+        for assignment in cases:
+            with self.subTest(assignment=assignment):
+                with self.assertRaises(ContractError):
+                    normalize_user_data(add_env(example(), assignment))
+
+    def test_arbitrary_cloud_init_operations_are_rejected(self):
+        for field in ('packages', 'bootcmd', 'disk_setup'):
+            doc = example()
+            doc[field] = []
+            with self.subTest(field=field):
+                with self.assertRaises(ContractError):
+                    normalize_user_data(doc)
+        doc = example()
+        doc['runcmd'] = ['touch /tmp/unapproved']
+        with self.assertRaises(ContractError):
+            normalize_user_data(doc)
+
+    def test_standard_list_form_of_bootstrap_command(self):
+        doc = example()
+        doc['runcmd'] = [['/usr/local/bin/apex-bootstrap']]
+        self.assertEqual(normalize_user_data(doc).hostname, doc['hostname'])
+
+    def test_invalid_administrator_and_missing_keys(self):
+        changes = [
+            ('name', 'root'), ('shell', '/bin/sh'), ('groups', ['sudo']),
+            ('sudo', 'ALL=(ALL) ALL'), ('ssh_authorized_keys', []),
+        ]
+        for key, value in changes:
+            doc = example()
+            doc['users'][0][key] = value
+            with self.subTest(key=key):
+                with self.assertRaises(ContractError):
+                    normalize_user_data(doc)
+
+    def test_unapproved_or_duplicate_file_writes(self):
+        for mutate in ('path', 'owner', 'permissions', 'append', 'duplicate'):
+            doc = example()
+            item = doc['write_files'][0]
+            if mutate == 'path':
+                item['path'] = '/etc/shadow'
+            elif mutate == 'owner':
+                item['owner'] = 'adam:adam'
+            elif mutate == 'permissions':
+                item['permissions'] = '0644'
+            elif mutate == 'append':
+                item['append'] = True
+            else:
+                doc['write_files'].append(copy.deepcopy(item))
+            with self.subTest(mutate=mutate):
+                with self.assertRaises(ContractError):
+                    normalize_user_data(doc)
+
+    def test_rejects_wrong_types_and_missing_required_fields(self):
+        cases = [None, [], {'hostname': 'x.example.com'}]
+        for value in cases:
+            with self.subTest(value=value):
+                with self.assertRaises(ContractError):
+                    normalize_user_data(value)
+        doc = example()
+        doc['preserve_hostname'] = 'false'
+        with self.assertRaises(ContractError):
+            normalize_user_data(doc)
+
+    def test_repository_paths_and_embedded_passwords(self):
+        for name, url in [
+            ('../escape', 'https://example.com/repo.git'),
+            ('node1', 'https://user:SECRET_SENTINEL@example.com/repo.git'),
+            ('node1', 'file:///etc/shadow'),
+            ('node1', '-uSECRET_SENTINEL'),
+        ]:
+            doc = example()
+            doc['write_files'][0]['content'] = (
+                f'APEX_REPO_NAME="{name}"\nAPEX_REPO_URL="{url}"\n')
+            with self.subTest(name=name, url=url):
+                with self.assertRaises(ContractError) as caught:
+                    normalize_user_data(doc)
+                self.assertNotIn('SECRET_SENTINEL', str(caught.exception))
+
+    def test_runtime_secrets_are_excluded_from_repr_and_identity(self):
+        doc = example()
+        for path, content, permission in [
+            ('/home/adam/.ssh/id_ed25519', 'SECRET_SENTINEL', '0600'),
+            ('/home/adam/.ssh/known_hosts', 'HOST_TRUST_SENTINEL', '0644'),
+        ]:
+            doc['write_files'].append({
+                'path': path, 'owner': 'adam:adam', 'permissions': permission,
+                'defer': True, 'content': content,
+            })
+        spec = normalize_user_data(doc)
+        text = repr(spec) + repr(spec.identity())
+        self.assertNotIn('SECRET_SENTINEL', text)
+        self.assertNotIn('PUBLIC_TEST_KEY', text)
+        self.assertNotIn('HOST_TRUST_SENTINEL', text)
+        self.assertEqual(spec.files[1].content, 'SECRET_SENTINEL')
+
+    def test_administrator_files_require_defer(self):
+        doc = example()
+        doc['write_files'].append({
+            'path': '/home/adam/.ssh/known_hosts', 'owner': 'adam:adam',
+            'permissions': '0644', 'content': 'example.com ssh-ed25519 VALUE',
+        })
+        with self.assertRaises(ContractError):
+            normalize_user_data(doc)
+
+    def test_contract_does_not_mutate_caller_input(self):
+        doc = example()
+        before = copy.deepcopy(doc)
+        normalize_user_data(doc)
+        self.assertEqual(doc, before)
+
+    def test_identity_changes_for_target_but_not_credentials(self):
+        doc = example()
+        before = normalize_user_data(doc).identity()
+        doc['users'][0]['ssh_authorized_keys'] = ['ssh-ed25519 ANOTHER_PUBLIC_KEY operator']
+        self.assertEqual(normalize_user_data(doc).identity(), before)
+        doc['hostname'] = 'other.a1.example.com'
+        self.assertNotEqual(normalize_user_data(doc).identity(), before)
+
+
+if __name__ == '__main__':
+    unittest.main()

ci/scripts/init-node/tests/test_provision_image.py

@@ -0,0 +1,112 @@
+import io
+import os
+from pathlib import Path
+import shlex
+import subprocess
+import sys
+import tarfile
+import tempfile
+import unittest
+from unittest.mock import patch
+
+from provisioning import access, image
+from provisioning.image import cloud_enroll, sanitize_files
+from provisioning.bundle import build_bundle
+from provisioning.state import atomic_write, safe_path
+from provisioning.state import ProvisionError
+
+
+class ImageTests(unittest.TestCase):
+    def test_uploaded_package_installs_runnable_wrappers_and_watchdog(self):
+        commons = Path(__file__).resolve().parents[4]
+        data, digest, _ = build_bundle(commons)
+        with tempfile.TemporaryDirectory() as temporary:
+            root = Path(temporary).resolve()
+            upload = root / 'tmp'
+            upload.mkdir()
+            with tarfile.open(fileobj=io.BytesIO(data)) as archive:
+                archive.extractall(upload, filter='data')
+
+            def mapped(path):
+                return safe_path(root / str(path).lstrip('/'))
+
+            def write(path, content, mode=0o600):
+                return atomic_write(mapped(path), content, mode)
+
+            with patch.object(image, '__file__', str(upload / 'provisioning/image.py')), \
+                    patch.object(image, 'safe_path', side_effect=mapped), \
+                    patch.object(image, 'atomic_write', side_effect=write):
+                metadata = image.install()
+            installed = root / 'usr/local/lib/apex-provisioner' / digest
+            self.assertEqual(metadata['path'], str(installed))
+            self.assertTrue((installed / 'provisioning/seed.sh').is_file())
+            environment = dict(os.environ)
+            environment.pop('PYTHONPATH', None)
+            for executable in ('apex-provision-image', 'apex-bootstrap'):
+                wrapper = root / 'usr/local/bin' / executable
+                self.assertEqual(wrapper.stat().st_mode & 0o777, 0o755)
+                script = wrapper.read_text().replace('/usr/bin/python3', shlex.quote(sys.executable) + ' -S')
+                result = subprocess.run(['/bin/sh', '-c', script, executable, '--help'],
+                                        cwd=root, env=environment, capture_output=True,
+                                        text=True, timeout=30)
+                self.assertEqual(result.returncode, 0, result.stderr)
+                self.assertIn('usage:', result.stdout)
+
+            commands = []
+            def runner(args, **kwargs):
+                commands.append(args)
+                return subprocess.CompletedProcess(args, 0, '', '')
+
+            canonical = '/usr/local/lib/apex-provisioner/' + digest
+            with patch.object(access, '__file__', canonical + '/provisioning/access.py'):
+                access.Host(root, runner=runner).arm()
+            service = (root / 'etc/systemd/system/apex-access-rollback.service').read_text()
+            fields = dict(line.split('=', 1) for line in service.splitlines() if '=' in line)
+            self.assertEqual(fields['WorkingDirectory'], canonical)
+            argv = shlex.split(fields['ExecStart'])
+            self.assertEqual(argv[-1], 'tick')
+            result = subprocess.run([sys.executable, '-S', *argv[1:-1], '--help'],
+                                    cwd=installed, env=environment, capture_output=True,
+                                    text=True, timeout=30)
+            self.assertEqual(result.returncode, 0, result.stderr)
+            self.assertIn(['systemctl', 'enable', '--now', 'apex-access-rollback.timer'], commands)
+            sanitize_files(root)
+            self.assertFalse((upload / 'provisioning').exists())
+            self.assertTrue((installed / 'provisioning/image.py').is_file())
+
+    def test_watchdog_rejects_uninstalled_package_before_writing_units(self):
+        with tempfile.TemporaryDirectory() as temporary:
+            root = Path(temporary).resolve()
+            with self.assertRaisesRegex(access.AccessError, 'installed root-owned bundle'):
+                access.Host(root).arm()
+            self.assertEqual(list(root.iterdir()), [])
+
+    def test_incompatible_image_stops_before_any_enrollment(self):
+        for metadata in ('{}', '{"enrollment_interface":999}'):
+            with self.subTest(metadata=metadata), patch.object(Path, 'read_text', return_value=metadata), \
+                    patch('provisioning.target.enroll') as enroll:
+                with self.assertRaisesRegex(ProvisionError, 'image enrollment interface'):
+                    cloud_enroll()
+                enroll.assert_not_called()
+
+    def test_sanitization_removes_cloned_identity_and_enrollment_but_preserves_runner(self):
+        with tempfile.TemporaryDirectory() as temporary:
+            root = Path(temporary).resolve()
+            for relative in ('etc/ssh/ssh_host_ed25519_key', 'etc/ssh/ssh_host_ed25519_key.pub',
+                             'root/.ssh/authorized_keys', 'home/debian/.ssh/authorized_keys',
+                             'etc/machine-id', 'var/lib/dbus/machine-id',
+                             'var/lib/apex-init/enrollment.json', 'var/lib/cloud/instance/user-data.txt',
+                             'usr/local/lib/apex-provisioner/digest/provisioning/target.py',
+                             'tmp/provisioning/image.py'):
+                path = root / relative
+                path.parent.mkdir(parents=True, exist_ok=True)
+                path.write_text('build-specific')
+            sanitize_files(root)
+            self.assertEqual(list((root / 'etc/ssh').glob('ssh_host_*')), [])
+            self.assertEqual((root / 'etc/machine-id').read_text(), '')
+            self.assertFalse((root / 'root/.ssh/authorized_keys').exists())
+            self.assertFalse((root / 'var/lib/apex-init').exists())
+            self.assertFalse((root / 'var/lib/cloud').exists())
+            self.assertFalse((root / 'tmp/provisioning').exists())
+            self.assertTrue((root / 'usr/local/lib/apex-provisioner/digest/provisioning/target.py').is_file())
+            self.assertTrue((root / 'etc/systemd/system/ssh.service.d/10-generate-host-keys.conf').is_file())

ci/scripts/init-node/tests/test_provision_repository.py

@@ -0,0 +1,132 @@
+import os
+from pathlib import Path
+from types import SimpleNamespace
+import subprocess
+import tempfile
+import unittest
+from unittest.mock import patch
+
+from provisioning.contract import Enrollment, Repository
+from provisioning.repository import reconcile_repository
+from provisioning.state import ProvisionError
+
+
+class RepositoryTests(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.temp.cleanup)
+        self.root = Path(self.temp.name).resolve()
+        self.upstream = self.root / 'upstream'
+        self.commons = self.root / 'commons'
+        self.srv = self.root / 'srv'
+        self.srv.mkdir()
+        self.home = self.root / 'adam'
+        self.home.mkdir()
+        for path in (self.upstream, self.commons):
+            path.mkdir()
+            self.git(['init', '-q'], path)
+            self.git(['config', 'user.name', 'Test'], path)
+            self.git(['config', 'user.email', 'test@example.invalid'], path)
+        (self.commons / 'marker').write_text('commons')
+        (self.commons / 'etc').mkdir()
+        (self.commons / 'etc/.vimrc').write_text('set number\n')
+        self.git(['add', '.'], self.commons)
+        self.git(['commit', '-qm', 'initial'], self.commons)
+        (self.upstream / 'node.env').write_text('APEX_NODE_FQDN=node1.example.com\nAPEX_ENROLLMENT_INTERFACE=1\n')
+        (self.upstream / 'init.sh').write_text(
+            '#!/bin/sh\nset -eu\ngit config core.hooksPath commons/githooks\n'
+            'printf \'export PATH="%s/commons:$PATH"\\n\' "$PWD" > "$HOME/.bashrc"\n'
+            'ln -sf "$PWD/commons/etc/.vimrc" "$HOME/.vimrc"\n')
+        self.git(['-c', 'protocol.file.allow=always', 'submodule', 'add', str(self.commons), 'commons'], self.upstream)
+        self.git(['add', '.'], self.upstream)
+        self.git(['commit', '-qm', 'initial'], self.upstream)
+        self.spec = Enrollment('node1.example.com', Repository('node1', str(self.upstream), None), (), (), ())
+        self.state = {}
+        self.saved = []
+        self.account = patch('provisioning.repository.pwd.getpwnam',
+                             return_value=SimpleNamespace(pw_uid=os.getuid(), pw_gid=os.getgid(), pw_dir=str(self.home)))
+        self.account.start()
+        self.addCleanup(self.account.stop)
+        self.runner = patch('provisioning.repository.as_adam', side_effect=self.local_run)
+        self.runner.start()
+        self.addCleanup(self.runner.stop)
+
+    @staticmethod
+    def git(args, path):
+        return subprocess.run(['git', *args], cwd=path, text=True, capture_output=True, check=True).stdout.strip()
+
+    def local_run(self, args, cwd=None):
+        if args[0] == 'git':
+            args = ['git', '-c', 'protocol.file.allow=always', *args[1:]]
+        return subprocess.run(args, cwd=cwd, text=True, capture_output=True, check=True,
+                              env={**os.environ, 'HOME': str(self.home)}).stdout.strip()
+
+    def enroll(self):
+        return reconcile_repository(self.spec, self.state, lambda: self.saved.append(True), self.srv)
+
+    def test_repeat_does_not_follow_new_upstream_commit(self):
+        first = self.enroll()
+        (self.upstream / 'new').write_text('upstream advanced')
+        self.git(['add', '.'], self.upstream)
+        self.git(['commit', '-qm', 'advance'], self.upstream)
+        second = self.enroll()
+        self.assertEqual(first['commit'], second['commit'])
+        self.assertFalse((self.srv / 'node1' / 'new').exists())
+        self.assertTrue(self.state['repository']['hook_complete'])
+
+    def test_local_changes_are_preserved_and_stop_initialization(self):
+        self.enroll()
+        path = self.srv / 'node1' / 'node.env'
+        path.write_text('local changes')
+        with self.assertRaisesRegex(ProvisionError, 'local changes'):
+            self.enroll()
+        self.assertEqual(path.read_text(), 'local changes')
+
+    def test_unrecorded_checkout_is_not_adopted(self):
+        (self.srv / 'node1').mkdir()
+        with self.assertRaisesRegex(ProvisionError, 'adoption'):
+            self.enroll()
+
+    def test_noop_hook_cannot_claim_success(self):
+        (self.upstream / 'init.sh').write_text('true\n')
+        self.git(['add', '.'], self.upstream)
+        self.git(['commit', '-qm', 'noop'], self.upstream)
+        with self.assertRaises((ProvisionError, subprocess.CalledProcessError)):
+            self.enroll()
+        self.assertFalse(self.state['repository'].get('hook_complete', False))
+
+    def test_missing_or_unsupported_interface_stops_before_hook(self):
+        for version in ('', 'APEX_ENROLLMENT_INTERFACE=999\n'):
+            with self.subTest(version=version):
+                (self.upstream / 'node.env').write_text('APEX_NODE_FQDN=node1.example.com\n' + version)
+                self.git(['add', '.'], self.upstream)
+                self.git(['commit', '-qm', 'unsupported interface'], self.upstream)
+                self.spec = Enrollment('node1.example.com', Repository('missing-' + str(len(version)), str(self.upstream), None), (), (), ())
+                self.state = {}
+                with self.assertRaisesRegex(ProvisionError, 'enrollment interface'):
+                    self.enroll()
+                self.assertFalse((self.home / '.bashrc').exists())
+                self.assertNotIn('hook_complete', self.state['repository'])
+
+    def test_retry_checks_hook_state_instead_of_completion_marker(self):
+        self.enroll()
+        self.git(['config', 'core.hooksPath', 'wrong-path'], self.srv / 'node1')
+        with self.assertRaisesRegex(ProvisionError, 'hook configuration'):
+            self.enroll()
+
+    def test_ordinary_commons_directory_is_not_a_pinned_submodule(self):
+        self.git(['rm', '-f', 'commons'], self.upstream)
+        (self.upstream / 'commons').mkdir()
+        (self.upstream / 'commons/ordinary').write_text('no submodule')
+        self.git(['add', '.'], self.upstream)
+        self.git(['commit', '-qm', 'ordinary directory'], self.upstream)
+        with self.assertRaisesRegex(ProvisionError, 'pinned Git submodule'):
+            self.enroll()
+
+    def test_mismatched_node_hostname_stops_before_hook(self):
+        (self.upstream / 'node.env').write_text('APEX_NODE_FQDN=wrong.example.com\n')
+        self.git(['add', '.'], self.upstream)
+        self.git(['commit', '-qm', 'wrong'], self.upstream)
+        with self.assertRaisesRegex(ProvisionError, 'hostname'):
+            self.enroll()
+        self.assertNotIn('hook_complete', self.state['repository'])

ci/scripts/init-node/tests/test_provision_security.py

@@ -0,0 +1,878 @@
+import base64
+import copy
+import hashlib
+import json
+import ipaddress
+import re
+from pathlib import Path
+import subprocess
+import tempfile
+import unittest
+from unittest.mock import patch
+
+from provisioning.access import Host
+from provisioning import security
+
+
+# Pinned helper default output, kept independent of the adapter's validation code.
+HELPER_V4 = """# BEGIN UFW AND DOCKER
+*filter
+:ufw-user-forward - [0:0]
+:ufw-docker-logging-deny - [0:0]
+:DOCKER-USER - [0:0]
+-A DOCKER-USER -j ufw-user-forward
+-A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j RETURN
+-A DOCKER-USER -m conntrack --ctstate INVALID -j DROP
+-A DOCKER-USER -i docker0 -o docker0 -j ACCEPT
+-A DOCKER-USER -j RETURN -s 10.0.0.0/8
+-A DOCKER-USER -j RETURN -s 172.16.0.0/12
+-A DOCKER-USER -j RETURN -s 192.168.0.0/16
+-A DOCKER-USER -j ufw-docker-logging-deny -m conntrack --ctstate NEW -d 10.0.0.0/8
+-A DOCKER-USER -j ufw-docker-logging-deny -m conntrack --ctstate NEW -d 172.16.0.0/12
+-A DOCKER-USER -j ufw-docker-logging-deny -m conntrack --ctstate NEW -d 192.168.0.0/16
+-A DOCKER-USER -j RETURN
+-A ufw-docker-logging-deny -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW DOCKER BLOCK] "
+-A ufw-docker-logging-deny -j DROP
+COMMIT
+# END UFW AND DOCKER
+"""
+HELPER_V6 = """# BEGIN UFW AND DOCKER
+*filter
+:ufw6-user-forward - [0:0]
+:ufw6-docker-logging-deny - [0:0]
+:DOCKER-USER - [0:0]
+-A DOCKER-USER -j ufw6-user-forward
+-A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j RETURN
+-A DOCKER-USER -m conntrack --ctstate INVALID -j DROP
+-A DOCKER-USER -i docker0 -o docker0 -j ACCEPT
+-A DOCKER-USER -j RETURN -s fd00::/8
+-A DOCKER-USER -j ufw6-docker-logging-deny -m conntrack --ctstate NEW -d fd00::/8
+-A DOCKER-USER -j RETURN
+-A ufw6-docker-logging-deny -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW DOCKER BLOCK] "
+-A ufw6-docker-logging-deny -j DROP
+COMMIT
+# END UFW AND DOCKER
+"""
+
+
+def native_alert(value, scope='Ip', origin='cscli', identifier=1):
+    return {'id': identifier, 'created_at': '2026-10-10T12:00:00Z', 'events_count': 1,
+            'events': [], 'message': 'manual decision', 'scenario': 'manual', 'simulated': False,
+            'source': {'scope': scope, 'value': value},
+            'decisions': [{'id': identifier, 'duration': '4h', 'origin': origin, 'scenario': 'manual',
+                           'scope': scope, 'simulated': False, 'type': 'ban', 'value': value}]}
+
+
+class SecurityHost(Host):
+    """Keep filesystem effects real; substitute Debian commands and services."""
+    def __init__(self, root):
+        super().__init__(root)
+        self.active = False
+        self.services = {name: {'LoadState': 'loaded', 'ActiveState': 'inactive',
+                               'UnitFileState': 'disabled', 'DropInPaths': '',
+                               'FragmentPath': '/usr/lib/systemd/system/' + name + '.service'}
+                         for name in ('ufw', 'crowdsec', 'crowdsec-firewall-bouncer')}
+        self.commands = []
+        self.decisions = []
+        self.runtime_overrides = {}
+        self.runtime_chains = {name: {'DOCKER-USER': ['-N DOCKER-USER'],
+                                      'ADMIN-SENTINEL': ['-N ADMIN-SENTINEL', '-A ADMIN-SENTINEL -j RETURN']}
+                               for name in ('iptables', 'ip6tables')}
+        self.forward_hook = True
+        self.fail = None
+        self.after_command = None
+        self.pending_starts = set()
+        contents = {
+            '/etc/default/ufw': 'IPV6=yes\nDEFAULT_INPUT_POLICY="ACCEPT"\nDEFAULT_OUTPUT_POLICY="ACCEPT"\nDEFAULT_FORWARD_POLICY="DROP"\n',
+            '/etc/ufw/ufw.conf': 'ENABLED=no\n',
+            '/etc/ufw/user.rules': '*filter\n-A ufw-after-logging-forward -j LOG --log-prefix "[UFW BLOCK] " -m limit --limit 3/min --limit-burst 10\nCOMMIT\n',
+            '/etc/ufw/user6.rules': '*filter\n-A ufw6-after-logging-forward -j LOG --log-prefix "[UFW BLOCK] " -m limit --limit 3/min --limit-burst 10\nCOMMIT\n',
+            '/etc/ufw/before.rules': '*filter\nCOMMIT\n',
+            '/etc/ufw/before6.rules': '*filter\nCOMMIT\n',
+            '/etc/ufw/after.rules': '*filter\nCOMMIT\n',
+            '/etc/ufw/after6.rules': '*filter\nCOMMIT\n',
+            '/usr/local/bin/ufw-docker': '#!/bin/sh\n',
+        }
+        for name, content in contents.items():
+            path = self.path(name)
+            path.parent.mkdir(parents=True, exist_ok=True)
+            path.write_text(content)
+        self.conffiles = {'/etc/default/ufw': hashlib.md5(contents['/etc/default/ufw'].encode(), usedforsecurity=False).hexdigest()}
+        self.bouncer_conffiles = {}
+        checksums = []
+        for name, content in contents.items():
+            if not name.startswith('/etc/ufw/'):
+                continue
+            filename = name.rsplit('/', 1)[1]
+            template = '/usr/share/ufw/' + ('' if filename == 'ufw.conf' else 'iptables/') + filename
+            path = self.path(template)
+            path.parent.mkdir(parents=True, exist_ok=True)
+            path.write_text(content)
+            checksums.append(hashlib.md5(content.encode(), usedforsecurity=False).hexdigest() + '  ' + template.lstrip('/'))
+        manifest = self.path('/var/lib/dpkg/info/ufw.md5sums')
+        manifest.parent.mkdir(parents=True, exist_ok=True)
+        manifest.write_text('\n'.join(checksums) + '\n')
+
+    def load_helper_runtime(self):
+        for executable, filename, fixture in (('iptables', 'after.rules', HELPER_V4),
+                                              ('ip6tables', 'after6.rules', HELPER_V6)):
+            if executable == 'ip6tables' and 'IPV6=yes' not in self.path('/etc/default/ufw').read_text():
+                continue
+            if '# BEGIN UFW AND DOCKER' not in self.path('/etc/ufw/' + filename).read_text():
+                continue
+            logging_chain = 'ufw6-docker-logging-deny' if executable == 'ip6tables' else 'ufw-docker-logging-deny'
+            for chain in ('DOCKER-USER', logging_chain):
+                self.runtime_chains[executable][chain] = ['-N ' + chain] + [
+                    line for line in fixture.splitlines() if line.startswith('-A ' + chain + ' ')]
+
+    def assert_restore_scope(self, args):
+        if '--noflush' not in args:
+            raise AssertionError('whole-table restore is forbidden')
+        if Path(args[-1]).stat().st_mode & 0o777 != 0o600:
+            raise AssertionError('restore input must be private')
+
+
+    def command(self, args, check=True):
+        self.commands.append(args)
+        output, code = '', 0
+        if args[:2] == ['systemctl', 'show']:
+            output = '\n'.join(k + '=' + v for k, v in self.services[args[2]].items())
+        elif args[:2] == ['ufw', 'status']:
+            output = 'Status: ' + ('active' if self.active else 'inactive')
+        elif args[0] == 'dpkg-query':
+            conffiles = self.bouncer_conffiles if args[-1] == 'crowdsec-firewall-bouncer-iptables' else self.conffiles
+            output = '\n'.join(' ' + name + ' ' + digest for name, digest in conffiles.items())
+        elif args[0] in ('iptables', 'ip6tables'):
+            if args[1] == '-C':
+                code = 0 if self.forward_hook else 1
+            elif args[1] == '-S':
+                chains = self.runtime_chains[args[0]]
+                if len(args) == 3:
+                    chain = args[2]
+                    code = 0 if chain in chains else 1
+                    output = self.runtime_overrides.get((args[0], chain), '\n'.join(chains.get(chain, [])) + '\n')
+                else:
+                    output = '-P FORWARD ACCEPT\n' + ('-A FORWARD -j DOCKER-USER\n' if self.forward_hook else '')
+                    output += ''.join(self.runtime_overrides.get((args[0], chain), '\n'.join(lines) + '\n')
+                                      for chain, lines in chains.items())
+            else:
+                raise AssertionError(args)
+        elif args[0] in ('iptables-restore', 'ip6tables-restore'):
+            self.assert_restore_scope(args)
+            executable = args[0].removesuffix('-restore')
+            chains = copy.deepcopy(self.runtime_chains[executable])
+            for line in Path(args[-1]).read_text().splitlines():
+                if line in ('*filter', 'COMMIT'):
+                    continue
+                operation, chain, *rest = line.split()
+                if operation == '-F':
+                    chains[chain] = ['-N ' + chain]
+                elif operation == '-A':
+                    chains[chain].append(line)
+                elif operation == '-X':
+                    del chains[chain]
+                else:
+                    raise AssertionError(line)
+            self.runtime_chains[executable] = chains
+        elif args[0] == '/usr/local/bin/ufw-docker':
+            if args[1] == 'check':
+                code = 0 if '# BEGIN UFW AND DOCKER' in self.path('/etc/ufw/after.rules').read_text() else 1
+            elif args[1] == 'install':
+                if not self.active:
+                    raise RuntimeError('UFW is disabled')
+                for name in ('after.rules', 'after6.rules'):
+                    path = self.path('/etc/ufw/' + name)
+                    path.write_text(path.read_text() + (HELPER_V6 if name == 'after6.rules' else HELPER_V4))
+        elif args[0] == 'ufw':
+            if 'allow' in args and 'proto' in args:
+                if args[1:3] == ['insert', '1'] and not any(
+                        '### tuple ###' in self.path('/etc/ufw/' + name).read_text()
+                        for name in ('user.rules', 'user6.rules')):
+                    if check:
+                        raise RuntimeError("ERROR: Invalid position '1'")
+                    return subprocess.CompletedProcess(args, 1, '', "ERROR: Invalid position '1'")
+                source, port = args[args.index('from') + 1], args[args.index('port') + 1]
+                v6 = ipaddress.ip_network(source, strict=False).version == 6
+                path = self.path('/etc/ufw/user6.rules' if v6 else '/etc/ufw/user.rules')
+                line = '### tuple ### allow tcp ' + port + (' ::/0' if v6 else ' 0.0.0.0/0') + ' any ' + source + ' in comment=' + args[-1].encode().hex() + '\n'
+                path.write_text(path.read_text().replace(line, '') if 'delete' in args else path.read_text() + line)
+            elif args[1] == 'default':
+                key = {'incoming': 'INPUT', 'outgoing': 'OUTPUT', 'routed': 'FORWARD'}[args[3]]
+                path = self.path('/etc/default/ufw')
+                path.write_text(re.sub('DEFAULT_' + key + '_POLICY="[A-Z]+"', 'DEFAULT_' + key + '_POLICY="' + ('DROP' if args[2] == 'deny' else 'ACCEPT') + '"', path.read_text()))
+            elif args[-1] in ('enable', 'disable'):
+                self.active = args[-1] == 'enable'
+                self.path('/etc/ufw/ufw.conf').write_text('ENABLED=' + ('yes' if self.active else 'no') + '\n')
+                if self.active and 'DEFAULT_FORWARD_POLICY="ACCEPT"' in self.path('/etc/default/ufw').read_text():
+                    for name, prefix in (('user.rules', 'ufw'), ('user6.rules', 'ufw6')):
+                        path = self.path('/etc/ufw/' + name)
+                        path.write_text(''.join(line for line in path.read_text().splitlines(keepends=True)
+                                                if not line.startswith('-A ' + prefix + '-after-logging-forward -j LOG ')))
+            elif args[-1] != 'reload':
+                raise AssertionError(args)
+            if self.active and args[-1] in ('enable', 'reload'):
+                self.load_helper_runtime()
+        elif args[0] == 'systemctl':
+            field, value = {'enable': ('UnitFileState', 'enabled'), 'disable': ('UnitFileState', 'disabled'),
+                            'start': ('ActiveState', 'active'), 'stop': ('ActiveState', 'inactive'),
+                            'reset-failed': ('ActiveState', 'inactive')}[args[1]]
+            if args[1] == 'reset-failed' and self.services[args[2]]['ActiveState'] != 'failed':
+                raise AssertionError('reset-failed must only follow an observed failure')
+            self.services[args[2]][field] = value
+            if args[1] == 'stop':
+                self.pending_starts.discard(args[2])
+            if args[1:3] == ['stop', 'ufw']:
+                self.active = False
+        elif args[0] == 'crowdsec':
+            pass
+        elif args[:2] == ['cscli', 'lapi']:
+            code = 0 if self.services['crowdsec']['ActiveState'] == 'active' else 1
+        elif args[:2] == ['cscli', 'decisions']:
+            alerts = self.decisions
+            if isinstance(alerts, list) and all(isinstance(a, dict) and isinstance(a.get('decisions'), list) for a in alerts):
+                if '--all' not in args:
+                    alerts = [a for a in alerts if all(d['origin'] != 'CAPI' for d in a['decisions'])]
+                limit = int(args[args.index('--limit') + 1]) if '--limit' in args else 100
+                if limit:
+                    alerts = alerts[:limit]
+            output = json.dumps(alerts)
+        else:
+            raise AssertionError('Unexpected command: ' + repr(args))
+        result = subprocess.CompletedProcess(args, code, output, '')
+        if self.after_command:
+            self.after_command(args)
+        if check and code:
+            raise RuntimeError('simulated command failure')
+        return result
+
+
+class SecurityTests(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.temp.cleanup)
+        self.host = SecurityHost(Path(self.temp.name))
+        self.adapter = security.Security(self.host)
+        self.addCleanup(patch.stopall)
+        patch.object(security, 'HELPER_SHA256', hashlib.sha256(b'#!/bin/sh\n').hexdigest(), create=True).start()
+
+    def test_masked_security_service_refuses_before_mutation(self):
+        self.host.services['crowdsec']['UnitFileState'] = 'masked'
+        before = self.host.path('/etc/default/ufw').read_bytes()
+        with self.assertRaises(security.SecurityError):
+            self.adapter.preflight()
+        self.assertEqual(self.host.path('/etc/default/ufw').read_bytes(), before)
+        self.assertFalse(self.host.active)
+
+    def test_inactive_custom_policy_is_not_mistaken_for_pristine(self):
+        path = self.host.path('/etc/default/ufw')
+        path.write_text(path.read_text() + 'MANAGED_BY_ADMIN=yes\n')
+        with self.assertRaises(security.SecurityError):
+            self.adapter.preflight()
+        self.assertFalse(self.host.active)
+
+    def test_packaged_bouncer_unit_in_etc_can_activate_and_restore(self):
+        name = '/etc/systemd/system/crowdsec-firewall-bouncer.service'
+        path = self.host.path(name)
+        path.parent.mkdir(parents=True)
+        path.write_text('[Service]\nExecStart=/usr/bin/crowdsec-firewall-bouncer\n')
+        self.host.bouncer_conffiles[name] = hashlib.md5(path.read_bytes(), usedforsecurity=False).hexdigest()
+        self.host.services['crowdsec-firewall-bouncer']['FragmentPath'] = name
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        self.assertEqual(self.host.services['crowdsec-firewall-bouncer']['ActiveState'], 'active')
+        self.adapter.restore(state, self.persist)
+        self.assertEqual(self.host.services['crowdsec-firewall-bouncer']['ActiveState'], 'inactive')
+        self.assertEqual(state['cleanup'], 'complete')
+
+    def test_modified_or_unmanaged_bouncer_unit_in_etc_refuses_before_activation(self):
+        name = '/etc/systemd/system/crowdsec-firewall-bouncer.service'
+        path = self.host.path(name)
+        path.parent.mkdir(parents=True)
+        path.write_text('[Service]\nExecStart=/usr/bin/crowdsec-firewall-bouncer\n')
+        self.host.services['crowdsec-firewall-bouncer']['FragmentPath'] = name
+        for conffiles in ({}, {name: '0' * 32}):
+            with self.subTest(conffiles=conffiles):
+                self.host.bouncer_conffiles = conffiles
+                with self.assertRaises(security.SecurityError):
+                    self.adapter.preflight()
+                self.assertFalse(self.host.active)
+
+    def test_native_generated_ufw_files_are_verified_against_package_templates(self):
+        self.assertEqual(set(self.host.conffiles), {'/etc/default/ufw'})
+        self.assertTrue(self.adapter.preflight()['pristine'])
+
+    def test_modified_generated_file_or_package_template_is_not_pristine(self):
+        for name in ('/etc/ufw/user.rules', '/usr/share/ufw/iptables/after.rules'):
+            with self.subTest(path=name):
+                path = self.host.path(name)
+                before = path.read_bytes()
+                path.write_bytes(before + b'# modified\n')
+                with self.assertRaises(security.SecurityError):
+                    self.adapter.preflight()
+                path.write_bytes(before)
+
+    def test_active_custom_policy_and_ipv6_rules_are_preserved_in_plan(self):
+        self.host.active = True
+        self.host.path('/etc/default/ufw').write_text('IPV6=yes\nDEFAULT_FORWARD_POLICY="DROP"\n')
+        self.host.path('/etc/ufw/user6.rules').write_text('# existing administrator sentinel\n')
+        plan = self.adapter.preflight()
+        self.assertFalse(plan['pristine'])
+        self.assertTrue(plan['ipv6'])
+        self.assertIn('sentinel', base64.b64decode(plan['files']['/etc/ufw/user6.rules']['data']).decode())
+
+    def test_incomplete_helper_is_rejected(self):
+        self.host.active = True
+        self.host.path('/etc/ufw/after.rules').write_text('# BEGIN UFW AND DOCKER\ncustom\n# END UFW AND DOCKER\n')
+        with self.assertRaises(security.SecurityError):
+            self.adapter.preflight()
+
+    def prepared(self):
+        self.saved = []
+        state = self.adapter.snapshot(self.adapter.preflight(), 22, '192.0.2.10')
+        self.saved.append(copy.deepcopy(state))
+        return state
+
+    def persist(self, state):
+        self.saved.append(copy.deepcopy(state))
+
+    def test_pristine_activation_establishes_security_without_losing_bootstrap(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        self.assertTrue(self.host.active)
+        self.assertIn('DEFAULT_INPUT_POLICY="DROP"', self.host.path('/etc/default/ufw').read_text())
+        self.assertIn('DEFAULT_FORWARD_POLICY="ACCEPT"', self.host.path('/etc/default/ufw').read_text())
+        rules = self.host.path('/etc/ufw/user.rules').read_text()
+        self.assertIn('192.0.2.10', rules)
+        self.assertIn('2222', rules)
+        for service in self.host.services.values():
+            self.assertEqual(service['ActiveState'], 'active')
+            self.assertEqual(service['UnitFileState'], 'enabled')
+        self.adapter.verify(state)
+
+    def test_bootstrap_allowance_exists_when_pristine_firewall_first_activates(self):
+        enabled = []
+        def inspect_enable(args):
+            if args == ['ufw', '--force', 'enable']:
+                rules = self.host.path('/etc/ufw/user.rules').read_text()
+                self.assertIn('192.0.2.10', rules)
+                self.assertIn('2222', rules)
+                self.assertIn('2222', self.host.path('/etc/ufw/user6.rules').read_text())
+                enabled.append(True)
+        self.host.after_command = inspect_enable
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        self.assertEqual(enabled, [True])
+
+    def test_active_policy_survives_activation_and_rollback(self):
+        self.host.active = True
+        self.host.services['ufw'].update(ActiveState='active', UnitFileState='enabled')
+        self.host.path('/etc/ufw/user6.rules').write_text('# IPv6 administrator sentinel\n')
+        before = {name: self.host.path(name).read_bytes() for name in security.FILES}
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        self.assertEqual(self.host.path('/etc/default/ufw').read_bytes(), before['/etc/default/ufw'])
+        self.adapter.restore(state, self.persist)
+        self.assertTrue(self.host.active)
+        for name, content in before.items():
+            self.assertEqual(self.host.path(name).read_bytes(), content)
+
+    def test_peer_range_ban_refuses_bouncer_and_can_restore_access(self):
+        self.host.decisions = [native_alert('192.0.2.0/24', 'Range')]
+        state = self.prepared()
+        with self.assertRaises(security.SecurityError):
+            self.adapter.activate(state, self.persist)
+        self.assertEqual(self.host.services['crowdsec-firewall-bouncer']['ActiveState'], 'inactive')
+        self.adapter.restore(state, self.persist)
+        self.assertFalse(self.host.active)
+        self.assertEqual(self.host.services['crowdsec']['ActiveState'], 'inactive')
+
+    def test_concurrent_edit_is_preserved_and_inactive_access_recovers(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        path = self.host.path('/etc/default/ufw')
+        path.write_text(path.read_text() + 'ADMIN_SENTINEL=yes\n')
+        with self.assertRaises(security.SecurityError):
+            self.adapter.restore(state, self.persist)
+        self.assertFalse(self.host.active)
+        self.assertIn('ADMIN_SENTINEL=yes', path.read_text())
+        self.assertTrue(state['access_recovered'])
+        self.assertEqual(state['cleanup'], 'pending')
+        self.assertEqual(self.host.services['crowdsec-firewall-bouncer']['ActiveState'], 'inactive')
+
+    def test_finalization_removes_only_temporary_bootstrap_allowance(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        self.adapter.finalize(state, self.persist)
+        rules = self.host.path('/etc/ufw/user.rules').read_text()
+        self.assertNotIn('192.0.2.10', rules)
+        self.assertIn('2222', rules)
+        self.adapter.verify(state)
+
+    def test_clean_rollback_restores_inactive_service_states(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        self.adapter.restore(state, self.persist)
+        for service in self.host.services.values():
+            self.assertEqual(service['ActiveState'], 'inactive')
+            self.assertEqual(service['UnitFileState'], 'disabled')
+        self.assertEqual(state['cleanup'], 'complete')
+
+    def test_rollback_restores_live_helper_chains_and_allows_next_preflight(self):
+        for original in (['-N DOCKER-USER'], ['-N DOCKER-USER', '-A DOCKER-USER -j RETURN']):
+            with self.subTest(original=original):
+                for executable in ('iptables', 'ip6tables'):
+                    self.host.runtime_chains[executable]['DOCKER-USER'] = original[:]
+                before = copy.deepcopy(self.host.runtime_chains)
+                state = self.prepared()
+                self.adapter.activate(state, self.persist)
+                self.adapter.restore(state, self.persist)
+                self.assertEqual(self.host.runtime_chains, before)
+                self.assertTrue(self.host.forward_hook)
+                self.assertTrue(self.adapter.preflight()['pristine'])
+
+    def test_preexisting_helper_specific_chain_is_rejected_before_mutation(self):
+        self.host.runtime_chains['ip6tables']['ufw6-docker-logging-deny'] = ['-N ufw6-docker-logging-deny']
+        before = copy.deepcopy(self.host.runtime_chains)
+        with self.assertRaises(security.SecurityError):
+            self.adapter.preflight()
+        self.assertEqual(self.host.runtime_chains, before)
+        self.assertFalse(self.host.active)
+
+    def test_docker_runtime_must_match_preflight_at_capture(self):
+        plan = self.adapter.preflight()
+        self.host.runtime_chains['iptables']['DOCKER-USER'].append('-A DOCKER-USER -j RETURN')
+        with self.assertRaises(security.SecurityError):
+            self.adapter.snapshot(plan, 22, '192.0.2.10')
+        self.assertFalse(self.host.active)
+
+    def test_changed_helper_runtime_is_preserved_while_access_recovers(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        self.host.runtime_chains['iptables']['DOCKER-USER'].append('-A DOCKER-USER -j ACCEPT')
+        before = copy.deepcopy(self.host.runtime_chains)
+        with self.assertRaises(security.SecurityError):
+            self.adapter.restore(state, self.persist)
+        self.assertEqual(self.host.runtime_chains, before)
+        self.assertTrue(state['access_recovered'])
+        self.assertEqual(state['cleanup'], 'pending')
+        self.assertIn('docker-firewall-runtime', state['diagnostics'])
+
+    def test_interrupted_helper_runtime_cleanup_resumes_each_family(self):
+        state = self.prepared()
+        before = copy.deepcopy(self.host.runtime_chains)
+        self.adapter.activate(state, self.persist)
+        def interrupt(args):
+            if args[0] == 'iptables-restore':
+                raise KeyboardInterrupt()
+        self.host.after_command = interrupt
+        with self.assertRaises(KeyboardInterrupt):
+            self.adapter.restore(state, self.persist)
+        self.host.after_command = None
+        self.assertEqual(self.host.runtime_chains['iptables'], before['iptables'])
+        self.assertNotEqual(self.host.runtime_chains['ip6tables'], before['ip6tables'])
+        resumed = copy.deepcopy(self.saved[-1])
+        self.adapter.restore(resumed, self.persist)
+        self.assertEqual(self.host.runtime_chains, before)
+        self.assertEqual(resumed['cleanup'], 'complete')
+        self.assertEqual(sum(args[0] == 'iptables-restore' for args in self.host.commands), 1)
+
+    def test_successful_restore_command_requires_runtime_postcondition(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        native = self.host.command
+        def no_change(args, check=True):
+            if args[0].endswith('tables-restore'):
+                return subprocess.CompletedProcess(args, 0, '', '')
+            return native(args, check)
+        with patch.object(self.host, 'command', side_effect=no_change):
+            with self.assertRaises(security.SecurityError):
+                self.adapter.restore(state, self.persist)
+        self.assertTrue(state['access_recovered'])
+        self.assertEqual(state['cleanup'], 'pending')
+
+    def test_runtime_drift_after_cleanup_persistence_is_not_overwritten(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        changed = []
+        def change_during_persist(value):
+            self.persist(value)
+            if value['helper_runtime'].get('restoring') == 'iptables' and not changed:
+                self.host.runtime_chains['iptables']['DOCKER-USER'].append('-A DOCKER-USER -j ACCEPT')
+                changed.append(True)
+        with self.assertRaises(security.SecurityError):
+            self.adapter.restore(state, change_during_persist)
+        self.assertEqual(self.host.runtime_chains['iptables']['DOCKER-USER'][-1], '-A DOCKER-USER -j ACCEPT')
+        self.assertTrue(state['access_recovered'])
+        self.assertFalse(any(args[0].endswith('tables-restore') for args in self.host.commands))
+
+    def test_runtime_drift_during_final_reload_prevents_false_complete_cleanup(self):
+        self.host.active = True
+        self.host.services['ufw'].update(ActiveState='active', UnitFileState='enabled')
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        def change_after_reload(args):
+            if args == ['ufw', 'reload']:
+                self.host.runtime_chains['iptables']['DOCKER-USER'].append('-A DOCKER-USER -j ACCEPT')
+        self.host.after_command = change_after_reload
+        with self.assertRaises(security.SecurityError):
+            self.adapter.restore(state, self.persist)
+        self.assertEqual(self.host.runtime_chains['iptables']['DOCKER-USER'][-1], '-A DOCKER-USER -j ACCEPT')
+        self.assertEqual(state['cleanup'], 'pending')
+        self.assertTrue(state['access_recovered'])
+
+    def test_legacy_runtime_snapshot_requires_reconciliation_but_recovers_access(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        del state['plan']['helper_runtime_before']
+        del state['helper_runtime']
+        before = copy.deepcopy(self.host.runtime_chains)
+        with self.assertRaises(security.SecurityError):
+            self.adapter.restore(state, self.persist)
+        self.assertEqual(self.host.runtime_chains, before)
+        self.assertTrue(state['access_recovered'])
+        self.assertEqual(state['cleanup'], 'pending')
+
+    def test_disabled_ipv6_runtime_is_preserved_during_helper_cleanup(self):
+        path = self.host.path('/etc/default/ufw')
+        path.write_text(path.read_text().replace('IPV6=yes', 'IPV6=no'))
+        self.host.conffiles['/etc/default/ufw'] = hashlib.md5(path.read_bytes(), usedforsecurity=False).hexdigest()
+        before = copy.deepcopy(self.host.runtime_chains)
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        self.adapter.restore(state, self.persist)
+        self.assertEqual(self.host.runtime_chains, before)
+        self.assertFalse(any(args[0] == 'ip6tables-restore' for args in self.host.commands))
+
+    def test_rollback_does_not_restart_preexisting_service_stopped_by_administrator(self):
+        self.host.services['crowdsec'].update(ActiveState='active', UnitFileState='enabled')
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        self.host.services['crowdsec']['ActiveState'] = 'inactive'
+        self.adapter.restore(state, self.persist)
+        self.assertEqual(self.host.services['crowdsec']['ActiveState'], 'inactive')
+        self.assertEqual(self.host.services['crowdsec']['UnitFileState'], 'enabled')
+        self.assertNotIn(['systemctl', 'start', 'crowdsec'], self.host.commands)
+
+    def test_owned_enable_does_not_claim_preexisting_activity_field(self):
+        self.host.services['crowdsec']['ActiveState'] = 'active'
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        self.host.services['crowdsec']['ActiveState'] = 'inactive'
+        self.adapter.restore(state, self.persist)
+        self.assertEqual(self.host.services['crowdsec']['ActiveState'], 'inactive')
+        self.assertEqual(self.host.services['crowdsec']['UnitFileState'], 'disabled')
+
+    def test_rollback_stops_owned_services_that_are_restarting_or_failed(self):
+        for activity in ('activating', 'deactivating', 'failed', 'reloading'):
+            with self.subTest(activity=activity):
+                self.host = SecurityHost(Path(self.temp.name) / activity)
+                self.adapter = security.Security(self.host)
+                state = self.prepared()
+                self.adapter.activate(state, self.persist)
+                self.host.services['crowdsec-firewall-bouncer']['ActiveState'] = activity
+                self.adapter.restore(state, self.persist)
+                self.assertEqual(self.host.services['crowdsec-firewall-bouncer']['ActiveState'], 'inactive')
+                self.assertEqual(state['cleanup'], 'complete')
+
+    def test_owned_inactive_service_stop_cancels_queued_start(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        self.host.services['crowdsec-firewall-bouncer']['ActiveState'] = 'inactive'
+        self.host.pending_starts.add('crowdsec-firewall-bouncer')
+        self.adapter.restore(state, self.persist)
+        self.assertFalse(self.host.pending_starts)
+        self.assertEqual(state['cleanup'], 'complete')
+
+    def test_failed_owned_stop_is_reset_and_verified_inactive(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        def failed_stop(args):
+            if args == ['systemctl', 'stop', 'crowdsec']:
+                self.host.services['crowdsec']['ActiveState'] = 'failed'
+        self.host.after_command = failed_stop
+        self.adapter.restore(state, self.persist)
+        self.assertEqual(self.host.services['crowdsec']['ActiveState'], 'inactive')
+        self.assertIn(['systemctl', 'reset-failed', 'crowdsec'], self.host.commands)
+        self.assertNotIn(['systemctl', 'reset-failed', 'crowdsec-firewall-bouncer'], self.host.commands)
+        self.assertEqual(state['cleanup'], 'complete')
+
+    def test_previously_restored_owned_service_is_rechecked_and_stopped_again(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        self.adapter.restore(state, self.persist)
+        self.host.services['crowdsec-firewall-bouncer']['ActiveState'] = 'activating'
+        self.adapter.restore(state, self.persist)
+        self.assertEqual(self.host.services['crowdsec-firewall-bouncer']['ActiveState'], 'inactive')
+        self.assertEqual(state['cleanup'], 'complete')
+
+    def test_late_owned_service_start_prevents_false_complete_cleanup(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        def late_start(args):
+            if args == ['ufw', 'disable']:
+                self.host.services['crowdsec-firewall-bouncer']['ActiveState'] = 'activating'
+        self.host.after_command = late_start
+        with self.assertRaises(security.SecurityError):
+            self.adapter.restore(state, self.persist)
+        self.assertEqual(state['cleanup'], 'pending')
+        self.assertTrue(state['access_recovered'])
+        self.host.after_command = None
+        self.adapter.restore(state, self.persist)
+        self.assertEqual(self.host.services['crowdsec-firewall-bouncer']['ActiveState'], 'inactive')
+        self.assertEqual(state['cleanup'], 'complete')
+
+    def test_failed_preexisting_service_is_not_stopped_or_reset(self):
+        self.host.services['crowdsec'].update(ActiveState='active', UnitFileState='enabled')
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        self.host.services['crowdsec']['ActiveState'] = 'failed'
+        self.adapter.restore(state, self.persist)
+        self.assertEqual(self.host.services['crowdsec']['ActiveState'], 'failed')
+        self.assertNotIn(['systemctl', 'stop', 'crowdsec'], self.host.commands)
+        self.assertNotIn(['systemctl', 'reset-failed', 'crowdsec'], self.host.commands)
+
+    def test_failed_reset_does_not_claim_service_recovery(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        def remains_failed(args):
+            if args in (['systemctl', 'stop', 'crowdsec'], ['systemctl', 'reset-failed', 'crowdsec']):
+                self.host.services['crowdsec']['ActiveState'] = 'failed'
+        self.host.after_command = remains_failed
+        with self.assertRaises(security.SecurityError):
+            self.adapter.restore(state, self.persist)
+        self.assertEqual(state['cleanup'], 'pending')
+        self.assertTrue(state['access_recovered'])
+
+    def test_changed_previously_restored_service_unit_is_preserved(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        self.adapter.restore(state, self.persist)
+        self.host.services['crowdsec-firewall-bouncer'].update(
+            ActiveState='activating', DropInPaths='/etc/systemd/system/bouncer.service.d/custom.conf')
+        count = len(self.host.commands)
+        with self.assertRaises(security.SecurityError):
+            self.adapter.restore(state, self.persist)
+        self.assertEqual(self.host.services['crowdsec-firewall-bouncer']['ActiveState'], 'activating')
+        self.assertNotIn(['systemctl', 'stop', 'crowdsec-firewall-bouncer'], self.host.commands[count:])
+        self.assertTrue(state['access_recovered'])
+
+    def test_changed_service_unit_is_preserved_and_reported_pending(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        self.host.services['crowdsec']['ActiveState'] = 'failed'
+        self.host.services['crowdsec']['DropInPaths'] = '/etc/systemd/system/crowdsec.service.d/administrator.conf'
+        with self.assertRaises(security.SecurityError):
+            self.adapter.restore(state, self.persist)
+        self.assertEqual(self.host.services['crowdsec']['ActiveState'], 'failed')
+        self.assertTrue(state['access_recovered'])
+        self.assertIn('service:crowdsec:ActiveState', state['diagnostics'])
+
+    def test_redirected_file_does_not_prevent_runtime_access_recovery(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        target = self.host.path('/etc/ufw/after.rules')
+        sentinel = self.host.path('/administrator-owned')
+        sentinel.write_text('do not touch\n')
+        target.unlink()
+        target.symlink_to(sentinel)
+        with self.assertRaises(security.SecurityError):
+            self.adapter.restore(state, self.persist)
+        self.assertFalse(self.host.active)
+        self.assertTrue(state['access_recovered'])
+        self.assertEqual(sentinel.read_text(), 'do not touch\n')
+
+    def test_metadata_change_is_not_silently_overwritten(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        path = self.host.path('/etc/default/ufw')
+        path.chmod(0o600)
+        with self.assertRaises(security.SecurityError):
+            self.adapter.verify(state)
+        with self.assertRaises(security.SecurityError):
+            self.adapter.restore(state, self.persist)
+        self.assertEqual(path.stat().st_mode & 0o777, 0o600)
+        self.assertFalse(self.host.active)
+
+    def test_process_death_after_native_write_preserves_unknown_bytes(self):
+        state = self.prepared()
+        def die(args):
+            if args[:2] == ['ufw', 'default']:
+                raise KeyboardInterrupt()
+        self.host.after_command = die
+        with self.assertRaises(KeyboardInterrupt):
+            self.adapter.activate(state, self.persist)
+        self.host.after_command = None
+        recovered = copy.deepcopy(self.saved[-1])
+        with self.assertRaises(security.SecurityError):
+            security.Security(self.host).restore(recovered, self.persist)
+        self.assertFalse(self.host.active)
+        self.assertTrue(recovered['access_recovered'])
+        self.assertEqual(recovered['cleanup'], 'pending')
+        self.assertIn('interrupted-mutation', recovered['diagnostics'])
+
+    def test_active_firewall_keeps_source_limited_bootstrap_on_ambiguous_recovery(self):
+        self.host.active = True
+        self.host.services['ufw'].update(ActiveState='active', UnitFileState='enabled')
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        self.adapter.finalize(state, self.persist)
+        path = self.host.path('/etc/ufw/user6.rules')
+        path.write_text(path.read_text() + '# concurrent IPv6 sentinel\n')
+        before_restore = len(self.host.commands)
+        with self.assertRaises(security.SecurityError):
+            self.adapter.restore(state, self.persist)
+        self.assertTrue(self.host.active)
+        self.assertTrue(state['access_recovered'])
+        self.assertIn('192.0.2.10', self.host.path('/etc/ufw/user.rules').read_text())
+        self.assertIn('# concurrent IPv6 sentinel', path.read_text())
+        self.assertNotIn(['ufw', 'disable'], self.host.commands)
+        self.assertNotIn(['ufw', 'reload'], self.host.commands[before_restore:])
+
+    def test_preexisting_ssh_rule_is_never_relabelled_or_deleted(self):
+        self.host.active = True
+        self.host.services['ufw'].update(ActiveState='active', UnitFileState='enabled')
+        path = self.host.path('/etc/ufw/user.rules')
+        sentinel = '### tuple ### allow tcp 2222 0.0.0.0/0 any 0.0.0.0/0 in comment=' + b'administrator'.hex() + '\n'
+        path.write_text(path.read_text() + sentinel)
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        self.adapter.finalize(state, self.persist)
+        self.assertIn(sentinel, path.read_text())
+        self.adapter.restore(state, self.persist)
+        self.assertIn(sentinel, path.read_text())
+
+    def test_retry_verification_detects_stopped_bouncer(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        self.host.services['crowdsec-firewall-bouncer']['ActiveState'] = 'inactive'
+        with self.assertRaises(security.SecurityError):
+            self.adapter.verify(state)
+
+    def test_redirected_ufw_enable_file_is_not_written_during_recovery(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        path = self.host.path('/etc/ufw/ufw.conf')
+        sentinel = self.host.path('/administrator-config')
+        sentinel.write_text('do not change\n')
+        path.unlink()
+        path.symlink_to(sentinel)
+        with self.assertRaises(security.SecurityError):
+            self.adapter.restore(state, self.persist)
+        self.assertEqual(sentinel.read_text(), 'do not change\n')
+        self.assertFalse(self.host.active)
+        self.assertTrue(state['access_recovered'])
+
+    def test_compatible_existing_helper_is_preserved(self):
+        self.host.active = True
+        for name in ('after.rules', 'after6.rules'):
+            path = self.host.path('/etc/ufw/' + name)
+            path.write_text(path.read_text() + (HELPER_V6 if name == 'after6.rules' else HELPER_V4))
+        self.host.load_helper_runtime()
+        before = copy.deepcopy(self.host.runtime_chains)
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        self.assertNotIn([security.HELPER, 'install'], self.host.commands)
+        self.adapter.restore(state, self.persist)
+        self.assertEqual(self.host.runtime_chains, before)
+        self.assertFalse(any(args[0].endswith('tables-restore') for args in self.host.commands))
+
+    def test_helper_check_success_does_not_accept_permissive_marker_block(self):
+        self.host.active = True
+        for name in ('after.rules', 'after6.rules'):
+            self.host.path('/etc/ufw/' + name).write_text(
+                '# BEGIN UFW AND DOCKER\n*filter\n-A DOCKER-USER -j ACCEPT\nCOMMIT\n# END UFW AND DOCKER\n')
+        self.assertEqual(self.host.command([security.HELPER, 'check']).returncode, 0)
+        with self.assertRaises(security.SecurityError):
+            self.adapter.preflight()
+
+    def test_valid_helper_files_require_live_rules_and_forward_hook(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        for executable, chain, contents in (
+                ('iptables', 'DOCKER-USER', '-N DOCKER-USER\n-A DOCKER-USER -j RETURN\n'),
+                ('ip6tables', 'DOCKER-USER', '-N DOCKER-USER\n-A DOCKER-USER -j ACCEPT\n'),
+                ('iptables', 'DOCKER-USER', '-N DOCKER-USER\n-A DOCKER-USER -j ACCEPT\n' + '\n'.join(
+                    line for line in HELPER_V4.splitlines() if line.startswith('-A DOCKER-USER ')) + '\n'),
+                ('iptables', 'ufw-docker-logging-deny', '-N ufw-docker-logging-deny\n')):
+            with self.subTest(executable=executable, chain=chain):
+                self.host.runtime_overrides[(executable, chain)] = contents
+                with self.assertRaises(security.SecurityError):
+                    self.adapter.verify(state)
+                self.host.runtime_overrides.clear()
+        self.host.forward_hook = False
+        with self.assertRaises(security.SecurityError):
+            self.adapter.verify(state)
+
+    def test_native_runtime_option_order_is_accepted_without_reordering_rules(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        output = self.host.command(['iptables', '-S', 'DOCKER-USER']).stdout
+        output = output.replace('RELATED,ESTABLISHED', 'ESTABLISHED,RELATED').replace(
+            '-j RETURN -s 10.0.0.0/8', '-s 10.0.0.0/8 -j RETURN')
+        self.host.runtime_overrides[('iptables', 'DOCKER-USER')] = output
+        self.adapter.verify(state)
+        lines = output.splitlines()
+        lines[1], lines[-1] = lines[-1], lines[1]
+        self.host.runtime_overrides[('iptables', 'DOCKER-USER')] = '\n'.join(lines) + '\n'
+        with self.assertRaises(security.SecurityError):
+            self.adapter.verify(state)
+
+    def test_existing_live_ban_is_rejected_before_firewall_changes(self):
+        self.host.services['crowdsec'].update(ActiveState='active', UnitFileState='enabled')
+        self.host.decisions = [native_alert('192.0.2.10')]
+        with self.assertRaises(security.SecurityError):
+            self.prepared()
+        self.assertFalse(self.host.active)
+        self.assertNotIn('2222', self.host.path('/etc/ufw/user.rules').read_text())
+
+    def test_ipv6_peer_requires_ipv6_firewall_support(self):
+        path = self.host.path('/etc/default/ufw')
+        path.write_text(path.read_text().replace('IPV6=yes', 'IPV6=no'))
+        self.host.conffiles['/etc/default/ufw'] = hashlib.md5(path.read_bytes(), usedforsecurity=False).hexdigest()
+        with self.assertRaises(security.SecurityError):
+            self.adapter.snapshot(self.adapter.preflight(), 22, '2001:db8::1')
+
+    def test_decision_check_covers_alerts_after_default_limit_and_central_api(self):
+        self.host.services['crowdsec'].update(ActiveState='active', UnitFileState='enabled')
+        harmless = [native_alert('198.51.100.' + str(i), identifier=i) for i in range(1, 101)]
+        for origin in ('cscli', 'CAPI'):
+            with self.subTest(origin=origin):
+                self.host.decisions = harmless + [native_alert('192.0.2.0/24', 'Range', origin, 101)]
+                with self.assertRaises(security.SecurityError):
+                    self.prepared()
+        self.assertFalse(self.host.active)
+
+    def test_non_native_decision_json_is_rejected_instead_of_skipped(self):
+        self.host.services['crowdsec'].update(ActiveState='active', UnitFileState='enabled')
+        for malformed in ([{'scope': 'Ip', 'value': '192.0.2.10'}], [{}], {'decisions': []}, None):
+            with self.subTest(value=malformed):
+                self.host.decisions = malformed
+                with self.assertRaises(security.SecurityError):
+                    self.prepared()
+
+    def test_optional_simulated_field_is_not_required_for_native_decisions(self):
+        self.host.services['crowdsec'].update(ActiveState='active', UnitFileState='enabled')
+        alert = native_alert('198.51.100.3')
+        del alert['decisions'][0]['simulated']
+        self.host.decisions = [alert]
+        self.prepared()
+        alert['decisions'][0]['value'] = '192.0.2.0/24'
+        alert['decisions'][0]['scope'] = 'Range'
+        with self.assertRaises(security.SecurityError):
+            self.prepared()
+
+    def test_surgical_helper_cleanup_preserves_concurrent_metadata_and_text(self):
+        state = self.prepared()
+        self.adapter.activate(state, self.persist)
+        path = self.host.path('/etc/ufw/after.rules')
+        path.write_text(path.read_text() + '# administrator sentinel\n')
+        path.chmod(0o600)
+        with self.assertRaises(security.SecurityError):
+            self.adapter.restore(state, self.persist)
+        self.assertIn('# administrator sentinel', path.read_text())
+        self.assertNotIn('# BEGIN UFW AND DOCKER', path.read_text())
+        self.assertEqual(path.stat().st_mode & 0o777, 0o600)
+
+
+if __name__ == '__main__':
+    unittest.main()

ci/scripts/init-node/tests/test_provision_security_access.py

@@ -0,0 +1,105 @@
+import tempfile
+import unittest
+from pathlib import Path
+
+from provisioning.access import AccessError, Transition
+from provisioning.security import SecurityError
+from test_provision_access import FakeHost, TOKEN
+
+
+from provision_fakes import SecurityBoundary
+
+
+class SecurityAccessTests(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.temp.cleanup)
+        self.host = FakeHost(Path(self.temp.name))
+        self.security = SecurityBoundary()
+        self.clock = 1000
+        self.transition = Transition(self.host, now=lambda: self.clock)
+        self.transition.security = self.security
+
+    def begin(self, port=22):
+        return self.transition.execute('begin', TOKEN, port, '192.0.2.10')
+
+    def test_activation_failure_restores_original_access_and_security(self):
+        self.security.fail_activation = True
+        with self.assertRaises((AccessError, SecurityError)):
+            self.begin()
+        self.assertEqual(self.host.ports, [22])
+        self.assertFalse(self.security.active)
+        self.assertFalse(self.host.armed)
+        self.assertEqual(self.transition.load()['status'], 'rolled_back')
+
+    def test_security_cleanup_failure_does_not_prevent_ssh_recovery(self):
+        self.begin()
+        self.transition.execute('finalize', TOKEN)
+        self.security.fail_restore = True
+        self.clock += 301
+        with self.assertRaises(AccessError):
+            self.transition.execute('tick')
+        self.assertEqual(self.host.ports, [22])
+        self.assertTrue(self.host.armed)
+        self.assertEqual(self.transition.load()['status'], 'recovery_pending')
+        self.security.fail_restore = False
+        self.transition.execute('tick')
+        self.assertEqual(self.transition.load()['status'], 'rolled_back')
+        self.assertFalse(self.host.armed)
+
+    def test_committed_retry_cannot_accept_stopped_security(self):
+        self.begin()
+        self.transition.execute('finalize', TOKEN)
+        self.transition.execute('commit', TOKEN)
+        self.security.active = False
+        with self.assertRaises((AccessError, SecurityError)):
+            self.begin(2222)
+        self.assertEqual(self.host.ports, [2222])
+
+    def test_existing_final_ssh_still_activates_security_with_watchdog(self):
+        self.host.existing_final = True
+        self.host.ports = [2222]
+        self.assertEqual(self.begin(2222)['status'], 'transition')
+        self.assertTrue(self.security.active)
+        self.assertTrue(self.host.armed)
+        self.assertEqual(self.host.ports, [2222])
+        self.transition.execute('finalize', TOKEN)
+        self.transition.execute('commit', TOKEN)
+        self.assertFalse(self.host.path('/etc/systemd/system/ssh.service.d/apex-init.conf').exists())
+        self.assertFalse(self.host.armed)
+
+    def test_legacy_committed_access_gains_security_without_reopening_root(self):
+        self.begin()
+        self.transition.execute('finalize', TOKEN)
+        self.transition.execute('commit', TOKEN)
+        state = self.transition.load()
+        state.pop('security')
+        self.transition.save(state)
+        self.security.active = False
+        self.assertEqual(self.begin(2222)['status'], 'transition')
+        self.assertEqual(self.host.ports, [2222])
+        self.assertTrue(self.security.active)
+        self.assertTrue(self.host.armed)
+
+    def test_failed_ssh_restore_still_restores_security_and_retries(self):
+        self.begin()
+        restore = self.host.restore
+        self.host.restore = lambda state: (_ for _ in ()).throw(AccessError('restart failed'))
+        self.clock += 301
+        with self.assertRaises(AccessError):
+            self.transition.execute('tick')
+        self.assertFalse(self.security.active)
+        self.assertTrue(self.host.armed)
+        self.host.restore = restore
+        self.transition.execute('tick')
+        self.assertEqual(self.host.ports, [22])
+        self.assertEqual(self.transition.load()['status'], 'rolled_back')
+
+    def test_security_failure_at_commit_keeps_rollback_armed(self):
+        self.begin()
+        self.transition.execute('finalize', TOKEN)
+        self.security.active = False
+        with self.assertRaises(SecurityError):
+            self.transition.execute('commit', TOKEN)
+        self.assertTrue(self.host.armed)
+        self.assertEqual(self.transition.load()['status'], 'final')

ci/scripts/init-node/tests/test_provision_state.py

@@ -0,0 +1,80 @@
+import json
+import os
+from pathlib import Path
+import tempfile
+import unittest
+
+from provisioning.state import (Journal, ProvisionError, atomic_write,
+                                      check_heartbeat, exclusive_lock)
+
+
+class StateTests(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.temp.cleanup)
+        self.root = Path(self.temp.name).resolve()
+
+    def test_atomic_private_file_and_symlink_rejection(self):
+        path = self.root / 'state' / 'record'
+        atomic_write(path, 'first')
+        atomic_write(path, 'second')
+        self.assertEqual(path.read_text(), 'second')
+        self.assertEqual(path.stat().st_mode & 0o777, 0o600)
+        link = self.root / 'redirect'
+        link.symlink_to(path.parent)
+        with self.assertRaises(ProvisionError):
+            atomic_write(link / 'record', 'corruption')
+        self.assertEqual(path.read_text(), 'second')
+
+    def test_lock_rejects_competing_initializer_and_releases(self):
+        path = self.root / 'lock'
+        with exclusive_lock(path):
+            with self.assertRaises(ProvisionError):
+                with exclusive_lock(path):
+                    self.fail('competing lock acquired')
+        with exclusive_lock(path):
+            pass
+
+    def test_failed_stage_is_retryable_and_success_is_reverified(self):
+        path = self.root / 'journal'
+        record = Journal(path, {'hostname': 'node1'}, 'rev', 'digest')
+        with self.assertRaises(RuntimeError):
+            record.stage('package', lambda: (_ for _ in ()).throw(RuntimeError('SECRET')))
+        self.assertNotIn('SECRET', path.read_text())
+        self.assertEqual(json.loads(path.read_text())['failed_stage'], 'package')
+        record = Journal(path, {'hostname': 'node1'}, 'rev', 'digest')
+        results = iter(({'version': 'one'}, {'version': 'two'}))
+        record.stage('package', lambda: next(results))
+        record.stage('package', lambda: next(results))
+        self.assertEqual(record.data['stages']['package']['result']['version'], 'two')
+
+    def test_identity_or_bundle_change_cannot_retarget_partial_run(self):
+        path = self.root / 'journal'
+        Journal(path, {'hostname': 'node1'}, 'rev', 'digest').save()
+        for identity, digest in (({'hostname': 'other'}, 'digest'), ({'hostname': 'node1'}, 'other')):
+            with self.assertRaises(ProvisionError):
+                Journal(path, identity, 'rev', digest)
+
+    def test_unsupported_interface_cannot_resume_and_revisions_survive_retry(self):
+        path = self.root / 'journal'
+        Journal(path, {'hostname': 'node1'}, 'source-rev', 'digest').save()
+        resumed = Journal(path, {'hostname': 'node1'}, 'source-rev', 'digest')
+        self.assertEqual(resumed.data['baseline_revision'], 'source-rev')
+        for version in (None, 999):
+            resumed.data['enrollment_interface'] = version
+            resumed.save()
+            with self.assertRaisesRegex(ProvisionError, 'enrollment interface'):
+                Journal(path, {'hostname': 'node1'}, 'source-rev', 'digest')
+
+    def test_expired_heartbeat_prevents_next_operation(self):
+        beat = self.root / 'heartbeat'
+        beat.touch()
+        os.utime(beat, (100, 100))
+        check_heartbeat(beat, now=110)
+        with self.assertRaises(ProvisionError):
+            check_heartbeat(beat, now=200)
+        record = Journal(self.root / 'journal', {}, 'rev', 'digest')
+        called = []
+        with self.assertRaises(ProvisionError):
+            record.stage('next', lambda: called.append(True), beat)
+        self.assertEqual(called, [])

ci/scripts/init-node/tests/test_provision_storage.py

@@ -0,0 +1,471 @@
+import copy
+import json
+import os
+from pathlib import Path
+import subprocess
+import tempfile
+from types import SimpleNamespace
+import unittest
+from unittest.mock import patch
+
+from provisioning.contract import Enrollment, Repository, Tier
+from provisioning import storage
+from provisioning.storage import StorageError, _persist_swap, reconcile_storage
+
+
+def enrollment(*tiers):
+    return Enrollment('node.example.com', Repository('node', 'https://example.com/node', None),
+                      tuple(tiers) or (Tier(1, 'directory', None, None),), (), ())
+
+
+class TestStorage(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.temp.cleanup)
+        self.root = Path(self.temp.name).resolve()
+        self.srv = self.root / 'srv'
+        self.srv.mkdir()
+        self.swap = self.root / 'swapfile'
+        self.fstab = self.root / 'fstab'
+        self.fstab.write_text('# existing entries\n')
+        self.meminfo = self.root / 'meminfo'
+        self.meminfo.write_text('MemTotal:       8388608 kB\n')
+        self.swaps = self.root / 'swaps'
+        self.calls = []
+        self.state = {}
+        self.saved = []
+        self.mounts = [{'target': '/', 'source': '/dev/root', 'fstype': 'ext4', 'maj:min': '8:0'}]
+        self.devices = [self.device('/dev/root', '8:0', 'ROOT', ['/']),
+                        self.device('/dev/spare', '8:16', 'SPARE', [])]
+        self.signatures = []
+        self.pool = None
+        self.pool_member = '/dev/spare'
+        self.pool_mountpoint = self.srv / 'tier-1.node'
+        self.pool_owner = None
+        self.import_listing = ''
+        self.import_full_path_listing = None
+        self.set_active_swaps(['/dev/existing-swap'])
+        self.activate_swap = True
+        self.create_hook = None
+        for name, value in [('SRV', self.srv), ('SWAPFILE', self.swap),
+                            ('FSTAB', self.fstab), ('MEMINFO', self.meminfo), ('SWAPS', self.swaps)]:
+            patcher = patch('provisioning.storage.' + name, value, create=True)
+            patcher.start()
+            self.addCleanup(patcher.stop)
+        patcher = patch('provisioning.storage.subprocess.run', side_effect=self.run_command)
+        patcher.start()
+        self.addCleanup(patcher.stop)
+        patcher = patch('provisioning.storage._device_number', side_effect=self.device_number)
+        patcher.start()
+        self.addCleanup(patcher.stop)
+
+    @staticmethod
+    def device(name, number, serial, mounts):
+        return {'name': name, 'maj:min': number, 'type': 'disk', 'pkname': None,
+                'mountpoints': mounts, 'fstype': None, 'wwn': None,
+                'serial': serial, 'size': 1024 ** 4}
+
+    def device_number(self, path):
+        aliases = {'/dev/alias': '/dev/spare'}
+        path = aliases.get(str(path), str(path))
+        for item in self.devices:
+            if item['name'] == path:
+                return item['maj:min']
+        raise StorageError('selected device is missing or is not a block device')
+
+    def save(self):
+        self.saved.append(copy.deepcopy(self.state))
+
+    def set_active_swaps(self, paths):
+        rows = ['Filename\tType\tSize\tUsed\tPriority']
+        for path in paths:
+            escaped = path.replace('\\', r'\134').replace(' ', r'\040').replace('\t', r'\011').replace('\n', r'\012')
+            rows.append(f'{escaped}\tfile\t2097148\t0\t-2')
+        self.swaps.write_text('\n'.join(rows) + '\n')
+
+    def run_command(self, argv, **kwargs):
+        self.calls.append(argv)
+        output = ''
+        if argv[0] == 'findmnt':
+            output = json.dumps({'filesystems': self.mounts})
+        elif argv[0] == 'lsblk':
+            output = json.dumps({'blockdevices': self.devices})
+        elif argv[0] == 'wipefs':
+            output = json.dumps({'signatures': self.signatures})
+        elif argv[:2] == ['zpool', 'list']:
+            if self.pool:
+                output = f"tier-1\t{self.pool['guid']}\tONLINE\n"
+        elif argv[:2] == ['zpool', 'status']:
+            output = ('  pool: tier-1\n state: ONLINE\nconfig:\n\n'
+                      '\tNAME STATE READ WRITE CKSUM\n'
+                      '\ttier-1 ONLINE 0 0 0\n'
+                      f'\t  {self.pool_member} ONLINE 0 0 0\n\nerrors: No known data errors\n')
+        elif argv[:2] == ['zpool', 'create']:
+            self.assertTrue(self.saved)
+            self.assertEqual(self.saved[-1]['storage']['tiers']['1']['status'], 'intent')
+            if self.create_hook:
+                self.create_hook()
+            self.pool = {'guid': '12345'}
+        elif argv[:2] == ['zpool', 'import']:
+            if '-N' in argv:
+                self.pool = {'guid': argv[-1]}
+            else:
+                output = self.import_listing
+                if self.import_full_path_listing is not None and kwargs['env'].get('ZPOOL_VDEV_NAME_PATH') == '1' \
+                        and kwargs['env'].get('ZPOOL_VDEV_NAME_FOLLOW_LINKS') == '1' \
+                        and 'ZPOOL_VDEV_NAME_GUID' not in kwargs['env']:
+                    output = self.import_full_path_listing
+        elif argv[:2] == ['zfs', 'list']:
+            mounted = 'yes' if any(m['source'] == 'tier-1' for m in self.mounts) else 'no'
+            output = f'tier-1\t{self.pool_mountpoint}\tfilesystem\t{mounted}\n'
+        elif argv[:2] == ['zfs', 'get']:
+            output = (self.pool_owner or self.state['storage']['identity']) + '\n'
+        elif argv[:2] == ['zfs', 'set']:
+            self.pool_owner = argv[2].split('=', 1)[1]
+        elif argv[:2] == ['zfs', 'mount']:
+            self.mounts.append({'target': str(self.srv / 'tier-1.node'), 'source': 'tier-1',
+                                'fstype': 'zfs', 'maj:min': '0:77'})
+        elif argv[0] == 'mkswap':
+            self.assertTrue(self.swap.exists())
+            self.assertEqual(self.saved[-1]['storage']['swap']['status'], 'intent')
+        elif argv == ['swapon', str(self.swap)]:
+            if self.activate_swap:
+                self.set_active_swaps([str(self.swap)])
+        else:
+            self.fail('Unexpected command: ' + repr(argv))
+        return subprocess.CompletedProcess(argv, 0, output, '')
+
+    def test_directory_preserves_contents_and_only_reports_new_directory(self):
+        path = self.srv / 'tier-1.node'
+        path.mkdir()
+        (path / 'keep').write_text('existing data')
+        result = reconcile_storage(enrollment(Tier(1, 'directory', None, None),
+                                              Tier(2, 'directory', None, None)), self.state, self.save)
+        self.assertEqual((path / 'keep').read_text(), 'existing data')
+        self.assertFalse(result['tiers'][0]['created'])
+        self.assertTrue(result['tiers'][1]['created'])
+        self.assertEqual(result['swap']['status'], 'skipped')
+
+    def test_new_directory_mode_is_exact_under_restrictive_umask(self):
+        self.srv.chmod(0o710)
+        previous_umask = os.umask(0o077)
+        try:
+            reconcile_storage(enrollment(), self.state, self.save)
+        finally:
+            os.umask(previous_umask)
+        self.assertEqual((self.srv / 'tier-1.node').stat().st_mode & 0o777, 0o755)
+        self.assertEqual(self.srv.stat().st_mode & 0o777, 0o710)
+
+    def test_preexisting_directory_keeps_restrictive_mode(self):
+        path = self.srv / 'tier-1.node'
+        path.mkdir()
+        path.chmod(0o700)
+        result = reconcile_storage(enrollment(), self.state, self.save)
+        self.assertFalse(result['tiers'][0]['created'])
+        self.assertEqual(path.stat().st_mode & 0o777, 0o700)
+
+    def test_directory_symlink_and_unexpected_mount_are_rejected(self):
+        path = self.srv / 'tier-1.node'
+        path.symlink_to(self.root)
+        with self.assertRaises(StorageError):
+            reconcile_storage(enrollment(), self.state, self.save)
+        path.unlink()
+        path.mkdir()
+        self.mounts.append({'target': str(path), 'source': '/dev/other', 'fstype': 'ext4', 'maj:min': '8:32'})
+        with self.assertRaises(StorageError):
+            reconcile_storage(enrollment(), self.state, self.save)
+
+    def test_pending_directory_ownership_survives_later_tier_failure_until_ack(self):
+        spec = enrollment(Tier(1, 'directory', None, None), Tier(2, 'directory', None, None))
+        first = self.srv / 'tier-1.node'
+        second = self.srv / 'tier-2.node'
+        second.symlink_to(self.root)
+        with self.assertRaises(StorageError):
+            reconcile_storage(spec, self.state, self.save)
+        self.assertTrue(first.is_dir())
+        self.state = copy.deepcopy(self.saved[-1])
+        second.unlink()
+        second.mkdir()
+        result = reconcile_storage(spec, self.state, self.save)
+        self.assertEqual(result['ownership_pending'], [str(first)])
+        self.assertEqual(reconcile_storage(spec, self.state, self.save)['ownership_pending'], [str(first)])
+        storage.acknowledge_storage_ownership(spec, self.state, [str(first)], self.save)
+        self.state = copy.deepcopy(self.saved[-1])
+        self.assertEqual(reconcile_storage(spec, self.state, self.save)['ownership_pending'], [])
+
+    def test_replaced_pending_directory_is_never_adopted_or_acknowledged(self):
+        spec = enrollment()
+        path = self.srv / 'tier-1.node'
+        reconcile_storage(spec, self.state, self.save)
+        path.rename(self.srv / 'original')
+        path.mkdir()
+        with self.assertRaisesRegex(StorageError, 'identity'):
+            reconcile_storage(spec, self.state, self.save)
+        with self.assertRaisesRegex(StorageError, 'identity'):
+            storage.acknowledge_storage_ownership(spec, self.state, [str(path)], self.save)
+
+    def test_preexisting_directory_cannot_be_acknowledged(self):
+        path = self.srv / 'tier-1.node'
+        path.mkdir()
+        spec = enrollment()
+        result = reconcile_storage(spec, self.state, self.save)
+        self.assertEqual(result['ownership_pending'], [])
+        with self.assertRaises(StorageError):
+            storage.acknowledge_storage_ownership(spec, self.state, [str(path)], self.save)
+
+    def test_ownership_acknowledgement_rejects_retargeted_enrollment(self):
+        spec = enrollment()
+        result = reconcile_storage(spec, self.state, self.save)
+        changed = Enrollment('other.example.com', spec.repository, spec.tiers, (), ())
+        with self.assertRaisesRegex(StorageError, 'identity'):
+            storage.acknowledge_storage_ownership(changed, self.state, result['ownership_pending'], self.save)
+        self.assertEqual(reconcile_storage(spec, self.state, self.save)['ownership_pending'],
+                         result['ownership_pending'])
+
+    def test_missing_root_and_occupied_device_never_create_pool(self):
+        for device in ('/dev/missing', '/dev/root', '/dev/spare'):
+            with self.subTest(device=device):
+                self.signatures = [{'type': 'ext4'}] if device == '/dev/spare' else []
+                with self.assertRaises(StorageError):
+                    reconcile_storage(enrollment(Tier(1, 'create', device, None)), {}, lambda: None)
+        self.assertFalse(any(c[:2] == ['zpool', 'create'] for c in self.calls))
+
+    def test_aliases_selected_by_two_tiers_are_rejected_before_creation(self):
+        with self.assertRaises(StorageError):
+            reconcile_storage(enrollment(Tier(1, 'create', '/dev/spare', None),
+                                          Tier(2, 'create', '/dev/alias', None)), self.state, self.save)
+        self.assertFalse(any(c[:2] == ['zpool', 'create'] for c in self.calls))
+
+    def test_create_records_guid_and_retry_verifies_without_recreation(self):
+        spec = enrollment(Tier(1, 'create', '/dev/spare', None))
+        reconcile_storage(spec, self.state, self.save)
+        self.assertEqual(self.state['storage']['tiers']['1']['guid'], '12345')
+        reconcile_storage(spec, self.state, self.save)
+        self.assertEqual(sum(c[:2] == ['zpool', 'create'] for c in self.calls), 1)
+        self.devices[1]['serial'] = 'REPLACED'
+        with self.assertRaises(StorageError):
+            reconcile_storage(spec, self.state, self.save)
+
+    def test_existing_pool_name_without_journal_is_not_ownership(self):
+        self.pool = {'guid': '12345'}
+        with self.assertRaises(StorageError):
+            reconcile_storage(enrollment(Tier(1, 'create', '/dev/spare', None)), self.state, self.save)
+
+    def test_import_requires_exact_guid_and_single_device(self):
+        self.import_listing = ('   pool: tier-1\n     id: 54321\n  state: ONLINE\n'
+                               ' config:\n\n\ttier-1 ONLINE\n\t  /dev/spare ONLINE\n')
+        with self.assertRaises(StorageError):
+            reconcile_storage(enrollment(Tier(1, 'import', '/dev/spare', '12345')), self.state, self.save)
+        self.assertFalse(any(c[:2] == ['zpool', 'import'] and '-N' in c for c in self.calls))
+
+    def test_unknown_swapfile_is_preserved(self):
+        self.set_active_swaps([])
+        self.swap.write_text('do not overwrite')
+        result = reconcile_storage(enrollment(), self.state, self.save)
+        self.assertEqual(self.swap.read_text(), 'do not overwrite')
+        self.assertEqual(result['swap']['status'], 'skipped')
+        self.assertIn('existing', result['swap']['reason'])
+
+    def test_root_disk_descendant_is_protected(self):
+        child = self.device('/dev/root1', '8:1', 'ROOT', ['/boot'])
+        child.update(type='part', pkname='/dev/root', partuuid='root-partition')
+        self.devices.append(child)
+        for device in ('/dev/root', '/dev/root1'):
+            with self.subTest(device=device):
+                with self.assertRaisesRegex(StorageError, 'intersects'):
+                    reconcile_storage(enrollment(Tier(1, 'create', device, None)), self.state, self.save)
+                self.state.clear()
+        self.assertFalse(any(c[:2] == ['zpool', 'create'] for c in self.calls))
+
+    def test_missing_hardware_identity_stops_before_creation(self):
+        self.devices[1]['serial'] = None
+        with self.assertRaisesRegex(StorageError, 'stable'):
+            reconcile_storage(enrollment(Tier(1, 'create', '/dev/spare', None)), self.state, self.save)
+        self.assertFalse(any(c[:2] == ['zpool', 'create'] for c in self.calls))
+
+    def test_identity_rechecked_after_intent_is_saved(self):
+        def changed_device():
+            self.save()
+            self.devices[1]['serial'] = 'REPLACED'
+        with self.assertRaisesRegex(StorageError, 'changed before'):
+            reconcile_storage(enrollment(Tier(1, 'create', '/dev/spare', None)), self.state, changed_device)
+        self.assertFalse(any(c[:2] == ['zpool', 'create'] for c in self.calls))
+
+    def test_created_whole_disk_pool_can_use_a_new_partition(self):
+        def partition_created():
+            child = self.device('/dev/spare1', '8:17', 'SPARE', [])
+            child.update(type='part', pkname='/dev/spare')
+            self.devices.append(child)
+            self.pool_member = '/dev/spare1'
+        self.create_hook = partition_created
+        result = reconcile_storage(enrollment(Tier(1, 'create', '/dev/spare', None)), self.state, self.save)
+        self.assertEqual(result['tiers'][0]['guid'], '12345')
+
+    def importable(self, members='\t  spare ONLINE\n'):
+        self.import_listing = ('   pool: tier-1\n     id: 12345\n  state: ONLINE\n'
+                               ' action: The pool can be imported using its name or numeric identifier.\n'
+                               ' config:\n\n\ttier-1 ONLINE\n' + members)
+
+    def test_imports_declared_guid_without_force_or_mountpoint_rewrite(self):
+        self.importable()
+        result = reconcile_storage(enrollment(Tier(1, 'import', '/dev/spare', '12345')), self.state, self.save)
+        self.assertEqual(result['tiers'][0]['guid'], '12345')
+        self.assertFalse(result['tiers'][0]['created'])
+        self.assertEqual(result['ownership_pending'], [])
+        with self.assertRaises(StorageError):
+            storage.acknowledge_storage_ownership(
+                enrollment(Tier(1, 'import', '/dev/spare', '12345')), self.state,
+                [str(self.pool_mountpoint)], self.save)
+        self.assertIn(['zpool', 'import', '-N', '-d', '/dev/spare', '12345'], self.calls)
+        self.assertFalse(any('-f' in command for command in self.calls))
+        self.assertFalse(any('mountpoint=' in arg for command in self.calls for arg in command))
+
+    def test_multidevice_import_is_rejected(self):
+        self.importable('\t  spare ONLINE\n\t  other ONLINE\n')
+        with self.assertRaisesRegex(StorageError, 'exactly one'):
+            reconcile_storage(enrollment(Tier(1, 'import', '/dev/spare', '12345')), self.state, self.save)
+        self.assertFalse(any(c[:2] == ['zpool', 'import'] and '-N' in c for c in self.calls))
+
+    def partitioned_import_fixture(self):
+        for suffix, number in (('1', '8:17'), ('9', '8:25')):
+            child = self.device('/dev/spare' + suffix, number, None, [])
+            child.update(type='part', pkname='/dev/spare', partuuid='partition-' + suffix)
+            self.devices.append(child)
+        self.pool_member = '/dev/spare1'
+        self.importable()
+        self.import_full_path_listing = self.import_listing.replace('spare ONLINE', '/dev/spare1 ONLINE')
+
+    def test_import_discovery_preserves_partition_path_instead_of_whole_disk_display(self):
+        self.partitioned_import_fixture()
+        with patch.dict(os.environ, {'ZPOOL_VDEV_NAME_GUID': '1'}):
+            result = reconcile_storage(enrollment(Tier(1, 'import', '/dev/spare1', '12345')),
+                                       self.state, self.save)
+        self.assertEqual(result['tiers'][0]['guid'], '12345')
+        self.assertIn(['zpool', 'import', '-N', '-d', '/dev/spare1', '12345'], self.calls)
+
+    def test_import_discovery_never_accepts_a_sibling_partition(self):
+        self.partitioned_import_fixture()
+        with self.assertRaisesRegex(StorageError, 'membership differs'):
+            reconcile_storage(enrollment(Tier(1, 'import', '/dev/spare9', '12345')), self.state, self.save)
+        self.assertFalse(any(c[:2] == ['zpool', 'import'] and '-N' in c for c in self.calls))
+
+    def test_import_discovery_parent_only_output_is_not_partition_proof(self):
+        self.partitioned_import_fixture()
+        self.import_full_path_listing = self.import_listing.replace('spare ONLINE', '/dev/spare ONLINE')
+        with self.assertRaisesRegex(StorageError, 'membership differs'):
+            reconcile_storage(enrollment(Tier(1, 'import', '/dev/spare1', '12345')), self.state, self.save)
+        self.assertFalse(any(c[:2] == ['zpool', 'import'] and '-N' in c for c in self.calls))
+
+    def test_import_does_not_mount_or_reassign_unrelated_mountpoint(self):
+        self.importable()
+        self.pool_mountpoint = '/important-data'
+        with self.assertRaisesRegex(StorageError, 'mountpoint'):
+            reconcile_storage(enrollment(Tier(1, 'import', '/dev/spare', '12345')), self.state, self.save)
+        self.assertFalse(any(c[:2] in (['zfs', 'mount'], ['zfs', 'set']) for c in self.calls))
+
+    def test_swap_skip_reports_low_space_and_unsupported_filesystems(self):
+        self.set_active_swaps([])
+        with patch('provisioning.storage.os.statvfs', return_value=SimpleNamespace(f_bavail=1, f_frsize=4096)):
+            result = reconcile_storage(enrollment(), self.state, self.save)
+        self.assertIn('insufficient', result['swap']['reason'])
+        self.mounts[0]['fstype'] = 'btrfs'
+        result = reconcile_storage(enrollment(), self.state, self.save)
+        self.assertIn('supported', result['swap']['reason'])
+        self.assertFalse(self.swap.exists())
+
+    def test_swap_sizes_allocation_and_records_verified_result(self):
+        self.set_active_swaps([])
+        with patch('provisioning.storage.os.statvfs', return_value=SimpleNamespace(f_bavail=16, f_frsize=1024 ** 3)), \
+                patch('provisioning.storage.os.posix_fallocate', create=True) as allocate, \
+                patch('provisioning.storage._persist_swap') as persist:
+            result = reconcile_storage(enrollment(), self.state, self.save)
+        self.assertEqual(allocate.call_args.args[1:], (0, 4 * 1024 ** 3))
+        self.assertEqual(result['swap'], {'status': 'complete', 'size': 4 * 1024 ** 3})
+        persist.assert_called_once()
+
+    def test_existing_active_swap_preserved_without_cli_inspection(self):
+        before = self.swaps.read_text()
+        result = reconcile_storage(enrollment(), self.state, self.save)
+        self.assertEqual(result['swap'], {'status': 'skipped', 'reason': 'existing active swap preserved'})
+        self.assertEqual(self.swaps.read_text(), before)
+        self.assertFalse(any(command[0] in ('swapon', 'mkswap') for command in self.calls))
+        self.assertFalse(self.swap.exists())
+
+    def test_proc_swaps_empty_and_escaped_paths(self):
+        self.set_active_swaps([])
+        self.assertEqual(storage._active_swaps(), [])
+        self.swaps.write_text('Filename\tType\tSize\tUsed\tPriority\n'
+                              '/swap\\040file\\011name\\012with\\134slash\tfile\t2097148\t0\t-2\n')
+        self.assertEqual(storage._active_swaps(), ['/swap file\tname\nwith\\slash'])
+
+    def test_unobserved_swap_activation_is_not_persisted_or_reported_complete(self):
+        self.set_active_swaps([])
+        self.activate_swap = False
+        with patch('provisioning.storage.os.statvfs', return_value=SimpleNamespace(f_bavail=16, f_frsize=1024 ** 3)), \
+                patch('provisioning.storage.os.posix_fallocate', create=True), \
+                patch('provisioning.storage._persist_swap') as persist:
+            result = reconcile_storage(enrollment(), self.state, self.save)
+        self.assertEqual(result['swap']['status'], 'skipped')
+        self.assertIn('activation did not become observable', result['swap']['reason'])
+        persist.assert_not_called()
+
+    def test_malformed_proc_swaps_stops_optional_allocation(self):
+        for text in ('', 'wrong header\n', 'Filename Type Size Used Priority\n/swapfile file broken 0 -2\n'):
+            with self.subTest(text=text):
+                self.swaps.write_text(text)
+                result = reconcile_storage(enrollment(), self.state, self.save)
+                self.assertEqual(result['swap']['status'], 'skipped')
+                self.assertIn('swap inventory', result['swap']['reason'])
+                self.assertFalse(self.swap.exists())
+
+    def test_small_memory_swap_is_two_gib(self):
+        self.set_active_swaps([])
+        self.meminfo.write_text('MemTotal:       4194304 kB\n')
+        with patch('provisioning.storage.os.statvfs', return_value=SimpleNamespace(f_bavail=16, f_frsize=1024 ** 3)), \
+                patch('provisioning.storage.os.posix_fallocate', create=True) as allocate, \
+                patch('provisioning.storage._persist_swap'):
+            result = reconcile_storage(enrollment(), self.state, self.save)
+        self.assertEqual(allocate.call_args.args[1:], (0, 2 * 1024 ** 3))
+        self.assertEqual(result['swap']['size'], 2 * 1024 ** 3)
+
+    def test_fstab_preserves_existing_content_and_does_not_duplicate_swap(self):
+        actual_lstat = Path.lstat
+
+        def root_owned(path, *args, **kwargs):
+            info = actual_lstat(path, *args, **kwargs)
+            if path == self.fstab:
+                fields = list(info)
+                fields[4] = 0
+                return os.stat_result(fields)
+            return info
+
+        with patch('provisioning.storage.Path.lstat', root_owned), \
+                patch('provisioning.storage.os.fchown'):
+            _persist_swap()
+            _persist_swap()
+        self.assertEqual(self.fstab.read_text(), '# existing entries\n' +
+                         f'{self.swap} none swap sw 0 0\n')
+
+    def test_pool_guid_change_on_retry_is_rejected(self):
+        spec = enrollment(Tier(1, 'create', '/dev/spare', None))
+        reconcile_storage(spec, self.state, self.save)
+        self.pool['guid'] = '67890'
+        with self.assertRaisesRegex(StorageError, 'GUID'):
+            reconcile_storage(spec, self.state, self.save)
+
+    def test_interrupted_create_recovers_only_matching_pool_ownership(self):
+        spec = enrollment(Tier(1, 'create', '/dev/spare', None))
+        reconcile_storage(spec, self.state, self.save)
+        record = self.state['storage']['tiers']['1']
+        record['status'] = 'intent'
+        del record['guid']
+        reconcile_storage(spec, self.state, self.save)
+        self.assertEqual(record['guid'], '12345')
+        self.assertEqual(sum(c[:2] == ['zpool', 'create'] for c in self.calls), 1)
+        self.pool_owner = 'other-enrollment'
+        with self.assertRaisesRegex(StorageError, 'ownership'):
+            reconcile_storage(spec, self.state, self.save)
+
+
+if __name__ == '__main__':
+    unittest.main()

ci/scripts/init-node/tests/test_provision_target.py

@@ -0,0 +1,87 @@
+import contextlib
+import io
+import json
+import os
+from pathlib import Path
+from types import SimpleNamespace
+import tempfile
+import unittest
+from unittest.mock import patch
+
+from provisioning import target
+from provisioning.contract import normalize_user_data
+from provisioning.state import ProvisionError, atomic_write, safe_path
+from test_provision_contract import example
+
+
+class TargetTests(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.temp.cleanup)
+        self.root = Path(self.temp.name).resolve()
+
+    def test_invalid_public_key_rejected_before_host_enrollment(self):
+        with self.assertRaisesRegex(ProvisionError, 'cryptographic validation'):
+            target.validate_keys(normalize_user_data(example()))
+
+    def test_administrator_preserves_existing_keys_and_uses_narrow_writes(self):
+        home = self.root / 'home/adam'
+        (home / '.ssh').mkdir(parents=True)
+        (home / '.ssh/authorized_keys').write_text('existing-key\n')
+        account = SimpleNamespace(pw_dir='/home/adam', pw_uid=os.getuid(), pw_gid=os.getgid(), pw_shell='/bin/bash')
+        commands = []
+
+        def command(args, **kwargs):
+            commands.append(args)
+            if args == ['hostname']:
+                return 'provider.example.com'
+            if args == ['id', '-nG', 'adam']:
+                return 'adam sudo docker'
+            return ''
+
+        def mapped(path):
+            path = Path(path)
+            return safe_path(path if path.is_relative_to(self.root) else self.root / str(path).lstrip('/'))
+
+        def write(path, content, mode=0o600, **kwargs):
+            # Keep real file writes; privileged ownership is the isolated boundary.
+            kwargs.pop('uid', None)
+            kwargs.pop('gid', None)
+            atomic_write(mapped(path), content, mode, **kwargs)
+
+        with patch.object(target.pwd, 'getpwnam', return_value=account), \
+                patch.object(target, 'safe_path', side_effect=mapped), \
+                patch.object(target, 'atomic_write', side_effect=write), \
+                patch.object(target, 'run', side_effect=command):
+            spec = normalize_user_data(example())
+            target.administrator(spec)
+            target.administrator(spec)
+        content = (home / '.ssh/authorized_keys').read_text().splitlines()
+        self.assertEqual(content, ['existing-key', 'ssh-ed25519 PUBLIC_TEST_KEY operator'])
+        self.assertIn(['hostnamectl', 'set-hostname', spec.hostname], commands)
+        self.assertEqual((self.root / 'etc/sudoers.d/99-apex-adam').stat().st_mode & 0o777, 0o440)
+        self.assertFalse((self.root / 'etc/machine-id').exists())
+        self.assertFalse((self.root / 'etc/network').exists())
+
+    def test_completion_requires_committed_access_and_preserves_revisions(self):
+        state = self.root / 'state'
+        (state / 'access').mkdir(parents=True)
+        record = {'status': 'enrolled', 'repository': {'commit': 'nodecommit', 'commons_commit': 'commonscommit'}}
+        atomic_write(state / 'enrollment.json', json.dumps(record))
+        atomic_write(state / 'access/state.json', json.dumps({'status': 'final'}))
+        with patch.object(target, 'STATE', state), patch.object(target, 'preflight'), \
+                contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()):
+            self.assertEqual(target.main(['complete']), 1)
+            self.assertEqual(json.loads((state / 'enrollment.json').read_text())['status'], 'enrolled')
+            atomic_write(state / 'access/state.json', json.dumps({'status': 'committed'}))
+            self.assertEqual(target.main(['complete']), 0)
+        actual = json.loads((state / 'enrollment.json').read_text())
+        self.assertEqual(actual['external_access'], 'verified')
+        self.assertEqual(actual['repository'], record['repository'])
+
+    def test_preflight_rejects_wrong_os_before_any_enrollment(self):
+        with patch.object(target.os, 'geteuid', return_value=0), \
+                patch.object(target.platform, 'machine', return_value='x86_64'), \
+                patch.object(Path, 'read_text', return_value='ID=ubuntu\nVERSION_ID="24.04"\n'):
+            with self.assertRaisesRegex(ProvisionError, 'Debian 13'):
+                target.preflight()

ci/scripts/init-node/tests/test_provision_userdata.py

@@ -0,0 +1,30 @@
+import importlib.util
+import unittest
+
+from provisioning.contract import ContractError
+from provisioning.userdata import decode_user_data
+
+
+@unittest.skipUnless(importlib.util.find_spec('yaml'), 'target-only python3-yaml is unavailable')
+class UserDataTests(unittest.TestCase):
+    def test_duplicate_mapping_keys_rejected_before_normalization(self):
+        with self.assertRaisesRegex(ContractError, 'unique'):
+            decode_user_data(b'hostname: one.example.com\nhostname: two.example.com\n')
+
+    def test_alias_and_unsafe_yaml_do_not_execute_or_echo_content(self):
+        for content in (b'a: &a [*a]', b'!!python/object/apply:os.system [SECRET_SENTINEL]',
+                        b'a: [SECRET_SENTINEL'):
+            with self.subTest(content=content):
+                with self.assertRaises(ContractError) as error:
+                    decode_user_data(content)
+                self.assertNotIn('SECRET_SENTINEL', str(error.exception))
+
+    def test_normal_supported_document(self):
+        import yaml
+        from test_provision_contract import example
+        spec = decode_user_data(yaml.safe_dump(example()).encode())
+        self.assertEqual(spec.hostname, 'node1.a1.example.com')
+
+    def test_yaml_boolean_mapping_key_rejected(self):
+        with self.assertRaisesRegex(ContractError, 'unique strings'):
+            decode_user_data(b'true: value\n')

compositions/apex/docker-compose.yml

@@ -1,32 +1,75 @@
 # commons/compositions/apex/docker-compose.yml
-# The shared apex CORE composition. Consumed by each node via include: (SP3).
+# The shared apex CORE composition. Available to node compositions through include:.
 # FULLY ${env}-parameterized; NO './' host paths (relative paths in an included file
 # resolve against THIS dir — always use absolute ${APEX_*} vars). The engine exports
 # APEX_TIER*/APEX_TIER*_SHARED/APEX_TIER_ROOT*/APEX_SUBNET/APEX_NODE_FQDN at invoke.
 x-environment: &shared-environment
-  TZ: "Europe/Moscow"
+  TZ: "${TZ:-Etc/UTC}" # override per node via TZ in the env; default UTC
 
 services:
-  # --- edge (host network; profile-gated — a node opts in via COMPOSE_PROFILES) ---
-  xray:
-    image: ghcr.io/xtls/xray-core:26.3.27
-    container_name: apex-xray
-    profiles: [xray]
+  adguard:
+    image: adguard/adguardhome:v0.107.79
+    container_name: apex-adguard
     volumes:
-      - ${APEX_TIER1}/xray/data/config.jsonc:/etc/xray/config.jsonc:ro
-      - ${APEX_TIER1_SHARED}/apex/letsencrypt:/etc/letsencrypt/live:ro
-      - ${APEX_TIER3_SHARED}/apex/xray/log:/var/log/xray/
-      - ${APEX_TIER3_SHARED}/apex/xray/shm:/dev/shm/xray/
+      - ${APEX_TIER1}/adguard/data:/opt/adguardhome/work
+      - ${APEX_TIER1}/adguard/conf:/opt/adguardhome/conf
     environment:
       <<: *shared-environment
-    network_mode: "host"
-    command: ["-c", "/etc/xray/config.jsonc"]
+    network_mode: "service:traefik"
     restart: unless-stopped
-    user: "0:0"
     labels:
-      # backup (resticontainer, core/all-nodes): xray config only (logs + shm are ephemeral)
+      # backup (resticontainer): adguard config only (work data regenerable)
       - "restic.enable=true"
-      - "restic.backup.paths=/etc/xray/config.jsonc"
+      - "restic.backup.paths=/opt/adguardhome/conf"
+    cap_add:
+      - NET_BIND_SERVICE
+      - SYS_NICE
+      - CAP_CHOWN
+    profiles: [ adguard ]
+    security_opt:
+      - no-new-privileges:true
+
+  # --- optional: observability agent (profile-gated) ---
+  # Collect host/container metrics and logs for configured remote endpoints.
+  # Alloy requires outbound access to those endpoints through the direct network.
+  alloy:
+    image: grafana/alloy:v1.20.0
+    container_name: apex-alloy
+    volumes:
+      - ${APEX_COMMONS}/etc/alloy/config.alloy:/etc/alloy/config.alloy:ro
+      - ${APEX_TIER2}/alloy/data:/var/lib/alloy
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /run/containerd/containerd.sock:/run/containerd/containerd.sock:ro # cadvisor per-container stats on nodes using the containerd snapshotter (overlayfs)
+      - /proc:/host/proc:ro
+      - /sys:/host/sys:ro
+      - /:/rootfs:ro
+      - /var/log/journal:/var/log/journal:ro
+      - /run/log/journal:/run/log/journal:ro
+      - /etc/machine-id:/etc/machine-id:ro
+    environment:
+      <<: *shared-environment
+      APEX_NODE_HOST: "${APEX_NODE_HOST?error}" # node + instance metric/log labels
+      # push targets + machine auth (per-node scalars in apex.env / secret .env)
+      APEX_LOKI_PUSH_URL: "${APEX_LOKI_PUSH_URL?error}"
+      APEX_PROM_PUSH_URL: "${APEX_PROM_PUSH_URL?error}"
+      APEX_OBS_BASIC_AUTH_USER: "${APEX_OBS_BASIC_AUTH_USER?error}"
+      APEX_OBS_BASIC_AUTH_PASS: "${APEX_OBS_BASIC_AUTH_PASS?error}"
+      GOMEMLIMIT: "${ALLOY_GOMEMLIMIT:-160MiB}"
+      GOGC: "${ALLOY_GOGC:-75}"
+    networks:
+      direct: {}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    command:
+      - run
+      - --server.http.listen-addr=0.0.0.0:12345
+      - --storage.path=/var/lib/alloy
+      - /etc/alloy/config.alloy
+    restart: unless-stopped
+    user: "0:0" # root: required for host metrics, journal, and docker.sock
+    cgroup: host # host cgroup namespace: the cadvisor exporter needs it to read per-container cgroups (else machine-root only)
+    hostname: "${APEX_NODE_HOST}" # UTS hostname => node_exporter's node_uname_info `nodename` (else the container id)
+    profiles: [ alloy ]
     security_opt:
       - no-new-privileges:true
 
@@ -39,42 +82,42 @@ services:
   # container mounts to host paths (via the /srv mount), snapshots the union
   # in a single restic run, then restarts + runs post-hooks. Every other subcommand
   # (init/snapshots/forget/…) passes straight through to restic. Needs the docker
-  # socket (list/inspect/exec/stop/start) and /srv read-only (the fleet's data root)
-  # — a privileged manual one-shot, same trust surface as the old restic core.
+  # socket (list/inspect/exec/stop/start) and read-only access to /srv.
   resticontainer:
-    image: ghcr.io/deytenit/resticontainer:0.19.1
+    image: ghcr.io/deytenit/resticontainer:0.19.1@sha256:85457ee665b1c6c8438ac3494c286d267169ee023857da3a605a6fe16dab3d82
     container_name: apex-resticontainer
-    profiles: [manual]
     volumes:
       - ${APEX_TIER3}/restic/cache:/var/restic-cache
-      - /var/run/docker.sock:/var/run/docker.sock          # query + exec/stop/start
+      - /var/run/docker.sock:/var/run/docker.sock # query + exec/stop/start
       # apex tier data + checkouts live under /srv, and the @tierN bind mounts are
       # ABSOLUTE symlinks into /srv (compositions/<p>/@tierN -> /srv/com-…-tierN/<p>).
       # Mount /srv at the same path with RESTIC_HOSTFS=/ so restic follows those
       # symlinks and records the real host paths — a bare /:/hostfs cannot, because
       # the absolute symlink target /srv/… is dangling inside the container.
-      - /srv:/srv:ro                                        # read the data (resolves @tierN symlinks)
+      - /srv:/srv:ro # read the data (resolves @tierN symlinks)
     environment:
       <<: *shared-environment
-      RESTIC_HOSTFS: "/"                                    # mount sources are already absolute (/srv/…)
-      RESTIC_REPOSITORY: "${APEX_RESTIC_REPOSITORY?error}"
-      RESTIC_PASSWORD: "${APEX_RESTIC_PASSWORD?error}"
-      AWS_ACCESS_KEY_ID: "${APEX_RESTIC_AWS_ACCESS_KEY_ID?error}"
-      AWS_SECRET_ACCESS_KEY: "${APEX_RESTIC_AWS_SECRET_ACCESS_KEY?error}"
-      AWS_REGION: "ru-central1"
-      AWS_ENDPOINT_URL: "https://storage.yandexcloud.net/"
-      # max compression is fleet policy (backup/run also forces --compression max on
-      # the CLI; needs repo format v2, which a fresh restic init creates). Per-node
-      # override via APEX_RESTIC_COMPRESSION only if ever needed.
+      RESTIC_HOSTFS: "/" # mount sources are already absolute (/srv/…)
+      RESTIC_REPOSITORY: "${APEX_RESTIC_REPOSITORY:-}"
+      RESTIC_PASSWORD: "${APEX_RESTIC_PASSWORD:-}"
+      AWS_ACCESS_KEY_ID: "${APEX_RESTIC_AWS_ACCESS_KEY_ID:-}"
+      AWS_SECRET_ACCESS_KEY: "${APEX_RESTIC_AWS_SECRET_ACCESS_KEY:-}"
+      # S3 backend region + endpoint. Defaults suit AWS S3 (us-east-1, no custom
+      # endpoint); set APEX_RESTIC_AWS_REGION / APEX_RESTIC_AWS_ENDPOINT (node apex.env)
+      # for any other S3-compatible store (Yandex, Backblaze B2, MinIO, …).
+      AWS_REGION: "${APEX_RESTIC_AWS_REGION:-us-east-1}"
+      AWS_ENDPOINT_URL: "${APEX_RESTIC_AWS_ENDPOINT:-}"
+      # Compression requires repository format v2. The default is max;
+      # override it with APEX_RESTIC_COMPRESSION for direct container use.
       RESTIC_COMPRESSION: "${APEX_RESTIC_COMPRESSION:-max}"
       RESTIC_CACHE_DIR: /var/restic-cache
     network_mode: "host"
+    profiles: [ manual ]
 
   # --- docker socket proxy ---
   socket-proxy:
     image: tecnativa/docker-socket-proxy:0
     container_name: apex-socket-proxy
-    user: "0:0"
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock:ro
     environment:
@@ -84,14 +127,12 @@ services:
     networks:
       socket: {}
     restart: unless-stopped
+    user: "0:0"
 
   # --- edge reverse proxy (the core anchor on direct) ---
   traefik:
-    image: traefik:3.6
+    image: traefik:3.7
     container_name: apex-traefik
-    user: "${APEX_UID?error}:${APEX_GID?error}"
-    group_add:
-      - "${APEX_SHARED_GID?error}"
     depends_on:
       - socket-proxy
     volumes:
@@ -100,7 +141,7 @@ services:
       - ${APEX_TIER1}/traefik/dynamic:/etc/traefik/dynamic:ro
     environment:
       <<: *shared-environment
-      CF_DNS_API_TOKEN: "${APEX_TRAEFIK_CF_DNS_API_TOKEN?error}"
+      CF_DNS_API_TOKEN: "${APEX_TRAEFIK_CF_DNS_API_TOKEN:-}"
       UMASK: "0002"
     ports:
       - '0.0.0.0:80:80'
@@ -133,13 +174,13 @@ services:
       - "--entrypoints.websecure.http3"
       - "--certificatesresolvers.cloudflare.acme.dnschallenge=true"
       - "--certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare"
-      - "--certificatesresolvers.cloudflare.acme.email=${APEX_TRAEFIK_ACME_EMAIL?error}"
+      - "--certificatesresolvers.cloudflare.acme.email=${APEX_TRAEFIK_ACME_EMAIL:-admin@localhost}"
       - "--certificatesresolvers.cloudflare.acme.storage=/letsencrypt/acme.json"
     restart: unless-stopped
     labels:
       - "traefik.enable=true"
       - "traefik.http.routers.apex-traefik.rule=Host(`${APEX_NODE_FQDN}`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
-      - "traefik.http.middlewares.xray-enclave.ipallowlist.sourcerange=198.18.0.0/16"
+      - "traefik.http.middlewares.xray-enclave.ipallowlist.sourcerange=${APEX_ENCLAVE_SUBNET:-198.18.0.0/16}"
       - "traefik.http.routers.apex-traefik.middlewares=xray-enclave@docker"
       - "traefik.http.routers.apex-traefik.service=api@internal"
       # adguard route (backend only present when the adguard profile is active)
@@ -148,76 +189,34 @@ services:
       - "traefik.http.middlewares.apex-adguard-strip-prefix.stripprefix.prefixes=/adguard"
       - "traefik.http.routers.apex-adguard.service=apex-adguard"
       - "traefik.http.services.apex-adguard.loadbalancer.server.port=3000"
-      # backup (resticontainer, core/all-nodes): dynamic routing rules only
+      # backup (resticontainer): dynamic routing rules only
       # (letsencrypt certs are regenerable via ACME/Cloudflare DNS-01)
       - "restic.enable=true"
       - "restic.backup.paths=/etc/traefik/dynamic"
+    user: "${APEX_UID?error}:${APEX_GID?error}"
+    group_add:
+      - "${APEX_SHARED_GID?error}"
     security_opt:
       - no-new-privileges:true
-
-  # --- optional: adguard (shares traefik netns) ---
-  adguard:
-    image: adguard/adguardhome:v0.107.72
-    container_name: apex-adguard
-    profiles: [adguard]
+  xray:
+    image: ghcr.io/xtls/xray-core:26.9.9
+    container_name: apex-xray
     volumes:
-      - ${APEX_TIER1}/adguard/data:/opt/adguardhome/work
-      - ${APEX_TIER1}/adguard/conf:/opt/adguardhome/conf
+      - ${APEX_TIER1}/xray/data/config.jsonc:/etc/xray/config.jsonc:ro
+      - ${APEX_TIER1_SHARED}/apex/letsencrypt:/etc/letsencrypt/live:ro
+      - ${APEX_TIER3_SHARED}/apex/xray/log:/var/log/xray/
+      - ${APEX_TIER3_SHARED}/apex/xray/shm:/dev/shm/xray/
     environment:
       <<: *shared-environment
+    network_mode: "host"
+    command: [ "-c", "/etc/xray/config.jsonc" ]
+    restart: unless-stopped
     labels:
-      # backup (resticontainer, core/all-nodes): adguard config only (work data regenerable)
+      # backup (resticontainer): xray config only (logs + shm are ephemeral)
       - "restic.enable=true"
-      - "restic.backup.paths=/opt/adguardhome/conf"
-    network_mode: "service:traefik"
-    restart: unless-stopped
-    cap_add:
-      - NET_BIND_SERVICE
-      - SYS_NICE
-      - CAP_CHOWN
-    security_opt:
-      - no-new-privileges:true
-
-  # --- optional: observability agent (profile-gated) ---
-  # Grafana Alloy: one agent per node. Collects host metrics (node_exporter-
-  # equivalent) + container metrics (cadvisor) and PUSHES them via remote_write;
-  # collects systemd-journal + docker logs and PUSHES them to Loki. All targets
-  # are OUTBOUND to the central gfn stack (Prometheus + Loki live on icarus), so
-  # Alloy needs egress (direct, not enclave). Replaces the old store-only `loki`
-  # service (which had no feeder). Config is a committed, env-parameterized asset.
-  alloy:
-    image: grafana/alloy:v1.17.1
-    container_name: apex-alloy
-    profiles: [alloy]
-    user: "0:0"  # root: required for host metrics, journal, and docker.sock
-    volumes:
-      - ${APEX_COMMONS}/assets/alloy/config.alloy:/etc/alloy/config.alloy:ro
-      - ${APEX_TIER2}/alloy/data:/var/lib/alloy
-      - /var/run/docker.sock:/var/run/docker.sock:ro
-      - /proc:/host/proc:ro
-      - /sys:/host/sys:ro
-      - /:/rootfs:ro
-      - /var/log/journal:/var/log/journal:ro
-      - /run/log/journal:/run/log/journal:ro
-      - /etc/machine-id:/etc/machine-id:ro
-    environment:
-      <<: *shared-environment
-      APEX_NODE: "${APEX_NODE?error}"
-      # push targets + machine auth (per-node scalars in apex.env / secret .env)
-      APEX_LOKI_PUSH_URL: "${APEX_LOKI_PUSH_URL?error}"
-      APEX_PROM_PUSH_URL: "${APEX_PROM_PUSH_URL?error}"
-      APEX_OBS_BASIC_AUTH_USER: "${APEX_OBS_BASIC_AUTH_USER?error}"
-      APEX_OBS_BASIC_AUTH_PASS: "${APEX_OBS_BASIC_AUTH_PASS?error}"
-    command:
-      - run
-      - --server.http.listen-addr=0.0.0.0:12345
-      - --storage.path=/var/lib/alloy
-      - /etc/alloy/config.alloy
-    networks:
-      direct: {}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    restart: unless-stopped
+      - "restic.backup.paths=/etc/xray/config.jsonc"
+    user: "0:0"
+    profiles: [ xray ]
     security_opt:
       - no-new-privileges:true
 

engine/cli.py

@@ -12,17 +12,19 @@ from engine.descriptor import build_parser, Meta
 
 
 def _resolve_layout():
-    commons_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
+    commons_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
     parent = os.path.dirname(commons_dir)
-    is_node = (os.path.basename(commons_dir) == "commons"
-               and os.path.isfile(os.path.join(parent, "node.env")))
-    if is_node:
+    cwd = os.getcwd()
+
+    if os.path.isfile(os.path.join(parent, "node.env")):
         repo_root = parent
         prop_actions = os.path.join(repo_root, "proprietaries", "actions")
-    else:
-        repo_root = commons_dir          # standalone commons repo is its own root
-        prop_actions = None
-    return commons_dir, repo_root, prop_actions, is_node
+        return commons_dir, repo_root, prop_actions, True
+    if os.path.isfile(os.path.join(cwd, "node.env")):
+        repo_root = cwd
+        prop_actions = os.path.join(repo_root, "proprietaries", "actions")
+        return commons_dir, repo_root, prop_actions, True
+    return commons_dir, commons_dir, None, False
 
 
 def _print_list(commons_actions, prop_actions):

engine/context.py

@@ -1,5 +1,5 @@
 # engine/context.py
-"""Context (ctx) — the single seam between an action and the world (SP2 §2.1)."""
+"""Context and filesystem paths available to actions."""
 from __future__ import annotations
 import os
 
@@ -81,7 +81,7 @@ class Context:
         self.ufw = Ufw(self.log, self.sys, self.host)
         self.crowdsec = Crowdsec(self.log, self.sys, self.tpl, self.host)
         self.systemd = Systemd(self.log, self.sys, self.tpl, self.host)
-        self.notify = Notify(self.log, name)
+        self.notify = Notify(self.log, node.host if node else "global")
         self._shadowed = shadowed
         self._commons = None
 
@@ -98,10 +98,9 @@ class Context:
         v["APEX_REPO_ROOT"] = self.paths.repo_root
         if self.node:
             v.update({
-                "APEX_NODE": self.node.name,
-                "APEX_CLUSTER": self.node.cluster,
-                "APEX_SUBNET": self.node.subnet,
+                "APEX_NODE_HOST": self.node.host,
                 "APEX_NODE_FQDN": self.node.fqdn,
+                "APEX_SUBNET": self.node.subnet,
                 "APEX_TIER1": self.paths.tier1, "APEX_TIER2": self.paths.tier2,
                 "APEX_TIER3": self.paths.tier3,
                 "APEX_TIER1_SHARED": self.paths.stier1,

engine/identity.py

@@ -1,29 +1,29 @@
 # engine/identity.py
-"""Node identity from host FQDN + node.env (replaces bash root_require_node).
+"""Node identity — fully explicit, from node.env. The framework assumes no domain
+and no clustering: a node declares its real FQDN and its public host name, and the
+short `name` is just the first label. The OS hostname is only a fallback for the FQDN.
 
-Decision B: the FQDN <node>.a<x>.apex.ermnvldmr.com keeps the cluster; we parse
-node+cluster from it. node.env owns APEX_SUBNET and mirrors APEX_CLUSTER; hostname
-wins on drift. Falls back to the repo directory name when the FQDN is not yet
-apex-shaped (mid-cutover).
+node.env keys:
+  APEX_NODE_FQDN  real FQDN (traefik Host() routing).            default: OS hostname
+  APEX_NODE_HOST  public host name (backup --host, obs/notify    default: APEX_NODE_FQDN
+                  `instance`); often the FQDN minus a cluster
+                  segment, but that's the node's choice.
+  APEX_SUBNET     the node's /24 for the `direct` network.       required
 """
 from __future__ import annotations
 
 import os
-import re
 import sys
 from dataclasses import dataclass
 from typing import Dict, List, Tuple
 
-_FQDN = re.compile(r"^(?P<node>[^.]+)\.(?P<cluster>a\d+)\.apex\.ermnvldmr\.com$")
-_DIRNAME = re.compile(r"^(?P<node>[^.]+)\.apex\.ermnvldmr\.com")
-
 
 @dataclass
 class Node:
-    name: str
-    cluster: str
-    subnet: str
-    fqdn: str
+    name: str    # short label (first segment of the host), e.g. "node1" — logs, branch, messages
+    subnet: str  # the node's direct-network /24
+    fqdn: str    # real FQDN — traefik Host() routing
+    host: str    # public host name — backup --host, obs `instance`, notify instance
 
 
 def read_env(path: str) -> Dict[str, str]:
@@ -41,36 +41,24 @@ def read_env(path: str) -> Dict[str, str]:
     return out
 
 
-def resolve(fqdn: str, repo_root: str) -> Tuple[Node, List[str]]:
+def resolve(hostname: str, repo_root: str) -> Tuple[Node, List[str]]:
     """Return (Node, warnings). Exits 66 (EX_NOINPUT) if APEX_SUBNET is missing."""
     warns: List[str] = []
     env = read_env(os.path.join(repo_root, "node.env"))
 
-    m = _FQDN.match(fqdn or "")
-    if m:
-        name, cluster = m.group("node"), m.group("cluster")
-        env_cluster = env.get("APEX_CLUSTER")
-        if env_cluster and env_cluster != cluster:
-            warns.append(
-                f"cluster drift: hostname says {cluster!r}, node.env says "
-                f"{env_cluster!r}; hostname wins."
-            )
-    else:
-        # Fallback: derive node from repo dir name; cluster from node.env.
-        warns.append(
-            f"FQDN {fqdn!r} is not apex-shaped; falling back to repo dir + node.env."
-        )
-        base = os.path.basename(os.path.realpath(repo_root))
-        dm = _DIRNAME.match(base)
-        name = dm.group("node") if dm else base
-        cluster = env.get("APEX_CLUSTER", "")
-        if not cluster:
-            sys.stderr.write("identity: no cluster in FQDN or node.env\n")
-            raise SystemExit(66)
+    fqdn = env.get("APEX_NODE_FQDN", "") or (hostname or "").strip()
+    if not fqdn:
+        sys.stderr.write("identity: APEX_NODE_FQDN missing from node.env and no hostname\n")
+        raise SystemExit(66)
+    if not env.get("APEX_NODE_FQDN"):
+        warns.append(f"APEX_NODE_FQDN not set in node.env; using hostname {fqdn!r}.")
+
+    host = env.get("APEX_NODE_HOST", "") or fqdn      # node's public name; defaults to the FQDN
+    name = host.split(".")[0]                          # short label for logs/branch/messages
 
     subnet = env.get("APEX_SUBNET", "")
     if not subnet:
         sys.stderr.write("identity: APEX_SUBNET missing from node.env\n")
         raise SystemExit(66)
 
-    return Node(name=name, cluster=cluster, subnet=subnet, fqdn=fqdn), warns
+    return Node(name=name, subnet=subnet, fqdn=fqdn, host=host), warns

engine/lib/crowdsec.py

@@ -13,7 +13,7 @@ class Crowdsec:
         if shutil.which("cscli"):
             return
         self.log.info("Installing CrowdSec via official script...")
-        self.sys.run(["bash", "-c", "set -o pipefail; curl -s https://install.crowdsec.net | sudo sh"],
+        self.sys.run(["bash", "-c", "set -o pipefail; curl -sSfL https://install.crowdsec.net | sudo sh"],
                      check=True)
         self.sys.sudo(["apt-get", "install", "-y", "crowdsec"])
 

engine/lib/host.py

@@ -5,10 +5,11 @@ Ports root_template_crontab (install + rollback) and the ufw block-injection log
 (root_template.sh _root_template_inject_rules_to_ufw_file), generalized + self-healing.
 """
 from __future__ import annotations
+
+import inspect
+import json
 import os
-import shutil
 import subprocess
-import tempfile
 import time
 from typing import List, Optional, Sequence, Tuple
 
@@ -51,7 +52,61 @@ def apply_block(text: str, content: str, start: str, end: str,
         if not inserted and anchor in ln:
             result.extend(block)
             inserted = True
-    return "\n".join(result) + "\n", True
+    return "\n".join(result) + "\n", inserted
+
+
+def _atomic_write(path, content, mode):
+    """Self-contained so sudo executes the same operation without importing user code."""
+    import os
+    import stat
+    import tempfile
+
+    target = os.path.realpath(path)
+    parent = os.path.dirname(target)
+    try:
+        previous = os.stat(target)
+    except FileNotFoundError:
+        previous = None
+    if previous is not None and not stat.S_ISREG(previous.st_mode):
+        raise ValueError(f"Not a regular file: {path}")
+    target_mode = (int(mode, 8) if mode is not None else
+                   stat.S_IMODE(previous.st_mode) if previous is not None else 0o644)
+    os.makedirs(parent, exist_ok=True)
+    fd, temporary = tempfile.mkstemp(prefix=".apex-", dir=parent)
+    try:
+        with os.fdopen(fd, "w", encoding="utf-8", newline="") as stream:
+            stream.write(content)
+            stream.flush()
+            if previous is not None:
+                current = os.fstat(stream.fileno())
+                if (current.st_uid, current.st_gid) != (previous.st_uid, previous.st_gid):
+                    os.fchown(stream.fileno(), previous.st_uid, previous.st_gid)
+            os.fchmod(stream.fileno(), target_mode)
+            os.fsync(stream.fileno())
+        os.replace(temporary, target)
+    finally:
+        if os.path.exists(temporary):
+            os.unlink(temporary)
+
+
+def _atomic_backup(path, destination):
+    import os
+    import shutil
+    import tempfile
+
+    destination = os.path.abspath(destination)
+    fd, temporary = tempfile.mkstemp(prefix=".apex-", dir=os.path.dirname(destination))
+    os.close(fd)
+    try:
+        shutil.copyfile(path, temporary)
+        previous, current = os.stat(path), os.stat(temporary)
+        if (current.st_uid, current.st_gid) != (previous.st_uid, previous.st_gid):
+            os.chown(temporary, previous.st_uid, previous.st_gid)
+        shutil.copystat(path, temporary)
+        os.replace(temporary, destination)
+    finally:
+        if os.path.exists(temporary):
+            os.unlink(temporary)
 
 
 class Host:
@@ -59,12 +114,20 @@ class Host:
         self.log = log
         self.sys = sys_
 
+    def _file_operation(self, operation, args, sudo):
+        if sudo:
+            script = (inspect.getsource(operation) + "\nimport json, sys\n" +
+                      f"{operation.__name__}(*json.load(sys.stdin))\n")
+            self.sys.sudo(["python3", "-I", "-c", script], input=json.dumps(args))
+        else:
+            operation(*args)
+
     # --- read/write with sudo awareness ---
     def _read(self, path: str, sudo: bool = False) -> str:
         if sudo:
             cp = self.sys.run(["sudo", "cat", path], check=True, capture=True)
             return cp.stdout or ""
-        with open(path) as f:
+        with open(path, encoding="utf-8") as f:
             return f.read()
 
     def backup(self, path: str, sudo: bool = False) -> Optional[str]:
@@ -73,34 +136,18 @@ class Host:
         if not exists:
             return None
         bkp = f"{path}.{int(time.time())}.bak"
-        if sudo:
-            self.sys.sudo(["cp", "-a", path, bkp])
-        else:
-            shutil.copy2(path, bkp)
+        self._file_operation(_atomic_backup, [path, bkp], sudo)
         self.log.info(f"Backed up {path} -> {bkp}")
         return bkp
 
-    def write_file(self, path: str, content: str, backup: bool = True,
-                   dry_run: bool = False, sudo: bool = False) -> None:
+    def write_file(self, path: str, content: str, mode: Optional[str] = None,
+                   backup: bool = True, dry_run: bool = False, sudo: bool = False) -> None:
         if dry_run:
             self.log.info(f"[DRY-RUN] would write {len(content)} bytes to {path}")
             return
         if backup:
             self.backup(path, sudo=sudo)
-        tmp = tempfile.NamedTemporaryFile("w", delete=False)
-        tmp.write(content)
-        tmp.close()
-        # 0644 like the bash renders (umask-022 redirect); mkstemp's 0600 would make
-        # freshly created units/configs owner-only. Overwrites via sudo cp keep the
-        # destination's existing mode either way.
-        os.chmod(tmp.name, 0o644)
-        if sudo:
-            self.sys.sudo(["mkdir", "-p", os.path.dirname(path)])
-            self.sys.sudo(["cp", tmp.name, path])
-            os.unlink(tmp.name)
-        else:
-            os.makedirs(os.path.dirname(path), exist_ok=True)
-            shutil.move(tmp.name, path)
+        self._file_operation(_atomic_write, [path, content, mode], sudo)
         self.log.success(f"Wrote {path}")
 
     def inject_block(self, target: str, content: str, start: str, end: str,

engine/lib/notify.py

@@ -1,16 +1,30 @@
 # engine/lib/notify.py
-"""ctx.notify — Telegram notifications (port of root_core_telegram*).
+"""ctx.notify — Telegram notifications.
 
-instance renamed com.ermnvldmr.root.<node> -> com.ermnvldmr.apex.<node>. POSTs
-application/x-www-form-urlencoded via urllib (stdlib) — functionally identical to
-the bash curl multipart for Telegram sendMessage.
+The alert `instance` is the node's public host (APEX_NODE_HOST), passed in from the
+context. POSTs application/x-www-form-urlencoded via urllib (stdlib).
 """
 from __future__ import annotations
+
 import html
 import urllib.parse
 import urllib.request
 from datetime import datetime
 
+NOTIFY_TIMEOUT = 20
+MAX_MESSAGE_LEN = 3500
+
+
+def resolve_bot_url(value: str) -> str:
+    """Resolve a literal URL or an @file reference without putting secrets in argv."""
+    if not value or not value.startswith("@"):
+        return value
+    path = value[1:]
+    if not path:
+        raise ValueError("empty Telegram Bot URL file reference")
+    with open(path, encoding="utf-8") as f:
+        return f.read().strip()
+
 
 def build_text(title: str, message: str, node: str, level: str, ts: str) -> str:
     status, icon, severity = "firing", "🚨", "info"
@@ -20,8 +34,8 @@ def build_text(title: str, message: str, node: str, level: str, ts: str) -> str:
         severity = "critical"
     elif level == "WARN":
         severity = "warning"
-    instance = f"com.ermnvldmr.apex.{node}"
-    msg = message if len(message) <= 200 else message[:200] + "..."
+    instance = node
+    msg = message if len(message) <= MAX_MESSAGE_LEN else message[:MAX_MESSAGE_LEN] + "..."
     msg = html.escape(msg)
     safe_title = html.escape(title)
     return (f"{icon} <b>{safe_title}</b>\n"
@@ -40,15 +54,27 @@ class Notify:
 
     def telegram(self, title, bot_url, message, node=None, level="INFO") -> bool:
         node = node or self.node
+        try:
+            bot_url = resolve_bot_url(bot_url)
+        except (OSError, ValueError) as e:
+            self.log.error(f"Failed to read Telegram Bot URL: {e}")
+            return False
         if not bot_url:
             self.log.warn("Telegram Bot URL not provided. Skipping notification.")
             return False
+
+        parsed = urllib.parse.urlparse(bot_url)
+        if parsed.scheme != "https":
+            self.log.error(f"Telegram Bot URL must use HTTPS, got scheme: {parsed.scheme!r}")
+            return False
+
         ts = datetime.now().astimezone().strftime("%Y-%m-%d %H:%M:%S %Z")
         text = build_text(title, message, node, level, ts)
         data = urllib.parse.urlencode({"parse_mode": "HTML", "text": text}).encode()
         self.log.info(f"Sending Telegram notification ({level}): {title}")
         try:
-            with urllib.request.urlopen(urllib.request.Request(bot_url, data=data), timeout=120) as r:
+            with urllib.request.urlopen(urllib.request.Request(bot_url, data=data),  # noqa: S310 -- HTTPS validated above.
+                                        timeout=NOTIFY_TIMEOUT) as r:
                 r.read()
             return True
         except Exception as e:

engine/lib/tpl.py

@@ -1,9 +1,8 @@
 # engine/lib/tpl.py
-"""ctx.tpl — stdlib string.Template rendering (envsubst successor, SP2 decision #3).
+"""ctx.tpl — stdlib string.Template rendering.
 
-safe_substitute replaces known $VAR/${VAR} and leaves unknown ones literal (deliberate
-delta: envsubst emptied them — literal survivors surface typos instead of silently
-blanking). Shell $(...) command-substitution survives untouched.
+safe_substitute replaces known $VAR/${VAR} and leaves unknown ones literal.
+Shell $(...) command-substitution survives untouched.
 Loops/conditionals live in the python action, never in the template file.
 """
 from __future__ import annotations

engine/lib/ufw.py

@@ -1,17 +1,31 @@
 # engine/lib/ufw.py
-"""ctx.ufw — ufw + ufw-docker rule application with self-healing marker migration.
+"""Apply ufw and ufw-docker rules.
 
-Injects the APEX marker block into before/after.rules, deleting BOTH the old ROOT
-and the new APEX blocks first (so re-running after cutover migrates the marker in
-place — SP2 §6). Docker rules go through ufw-docker; host rules through plain ufw.
+Replace existing supported marker blocks before inserting current rules.
+Docker rules go through ufw-docker; host rules through plain ufw.
 """
 from __future__ import annotations
+
 import os
+import shlex
+import shutil
 
 APEX_START = "# START APEX.ERMNVLDMR.COM RULES"
 APEX_END = "# END APEX.ERMNVLDMR.COM RULES"
 OLD_PAIRS = [("# START ROOT.ERMNVLDMR.COM RULES", "# END ROOT.ERMNVLDMR.COM RULES")]
-UFW_DOCKER_BIN = os.path.expanduser("~/.local/bin/ufw-docker")
+
+
+def resolve_ufw_docker_bin() -> str:
+    """Find ufw-docker in PATH, /usr/local/bin, or fallback to ~/.local/bin/ufw-docker."""
+    found = shutil.which("ufw-docker")
+    if found:
+        return found
+    if os.path.isfile("/usr/local/bin/ufw-docker") and os.access("/usr/local/bin/ufw-docker", os.X_OK):
+        return "/usr/local/bin/ufw-docker"
+    return os.path.expanduser("~/.local/bin/ufw-docker")
+
+
+UFW_DOCKER_BIN = resolve_ufw_docker_bin()
 
 
 def clean_rules(text: str):
@@ -29,19 +43,29 @@ class Ufw:
         if not self.sys.ok(["sudo", "test", "-f", target]):
             self.log.error(f"Target file does not exist: {target}")
             raise SystemExit(1)
-        content = "\n".join(clean_rules(open(source_file).read()))
-        self.host.inject_block(target, content, APEX_START, APEX_END,
-                               anchor="# End required lines", old_pairs=OLD_PAIRS,
-                               dry_run=dry_run, sudo=True)
+        with open(source_file, encoding="utf-8") as f:
+            content = "\n".join(clean_rules(f.read()))
+        self.host.inject_block(
+            target,
+            content,
+            APEX_START,
+            APEX_END,
+            anchor="# End required lines",
+            old_pairs=OLD_PAIRS,
+            dry_run=dry_run,
+            sudo=True,
+        )
 
-    def apply(self, rules_dir: str, config_dir=None, dry_run: bool = False) -> None:
-        # EX_NOINPUT guards first, tools second — same order/codes as root_template_ufw.
-        if not os.path.isdir(rules_dir):
+    def apply(self, rules_dir: str | None = None, config_dir: str | None = None, dry_run: bool = False) -> None:
+        if rules_dir and not os.path.isdir(rules_dir):
             self.log.error(f"Rules directory does not exist: {rules_dir}")
             raise SystemExit(66)
         if config_dir and not os.path.isdir(config_dir):
             self.log.error(f"Config directory does not exist: {config_dir}")
             raise SystemExit(66)
+        if not rules_dir and not config_dir:
+            self.log.warn("Neither docker nor host ufw rules directory found; nothing to apply.")
+            return
         if not self.sys.ok(["sudo", "ufw", "--version"]):
             self.log.error("ufw not installed/working via sudo. Run configure/base first.")
             raise SystemExit(1)
@@ -52,26 +76,30 @@ class Ufw:
         if config_dir:
             host_rules = os.path.join(config_dir, "host.rules")
             if os.path.isfile(host_rules):
-                for rule in clean_rules(open(host_rules).read()):
+                with open(host_rules, encoding="utf-8") as f:
+                    rules = clean_rules(f.read())
+                for rule in rules:
                     self.log.info(f"Applying host ufw rule: {rule}")
                     if dry_run:
                         self.log.info(f"[DRY-RUN] sudo ufw {rule}")
                     else:
-                        self.sys.run(["bash", "-c", f"sudo ufw {rule}"], check=True)
+                        self.sys.sudo(["ufw", *shlex.split(rule)], check=True)
             self._inject_file("/etc/ufw/before.rules", os.path.join(config_dir, "before.rules"), dry_run)
             self._inject_file("/etc/ufw/after.rules", os.path.join(config_dir, "after.rules"), dry_run)
 
-        rule_files = [fn for fn in sorted(os.listdir(rules_dir))
-                      if fn.endswith(".rules") and not fn.startswith(".")]
-        if not rule_files:
-            self.log.warn(f"No .rules files found in {rules_dir}, nothing to apply.")
-        for fn in rule_files:
-            for rule in clean_rules(open(os.path.join(rules_dir, fn)).read()):
-                self.log.info(f"Applying ufw-docker rule ({fn}): {rule}")
-                if dry_run:
-                    self.log.info(f"[DRY-RUN] sudo {UFW_DOCKER_BIN} {rule}")
-                else:
-                    self.sys.run(["sudo", UFW_DOCKER_BIN, *rule.split()], check=True)
+        if rules_dir:
+            rule_files = [fn for fn in sorted(os.listdir(rules_dir)) if fn.endswith(".rules") and not fn.startswith(".")]
+            if not rule_files:
+                self.log.warn(f"No .rules files found in {rules_dir}, nothing to apply.")
+            for fn in rule_files:
+                with open(os.path.join(rules_dir, fn), encoding="utf-8") as f:
+                    rules = clean_rules(f.read())
+                for rule in rules:
+                    self.log.info(f"Applying ufw-docker rule ({fn}): {rule}")
+                    if dry_run:
+                        self.log.info(f"[DRY-RUN] sudo {UFW_DOCKER_BIN} {rule}")
+                    else:
+                        self.sys.sudo([UFW_DOCKER_BIN, *shlex.split(rule)], check=True)
 
         self.log.success("All ufw rules applied.")
         self.log.warn("Run `sudo systemctl restart ufw` to apply. Ensure ssh is allowed via plain ufw!")

etc/alloy/config.alloy

@@ -1,14 +1,6 @@
-// ============================================================================
-// APEX fleet observability agent — per-node Grafana Alloy config (shared).
-// Committed, env-parameterized: every node runs this SAME file; per-node values
-// come from sys.env(...) (APEX_NODE, push URLs, obs basic-auth creds).
-// Verified against grafana.com/docs/alloy/latest (Alloy v1.17.1).
-// ============================================================================
-// Model: PUSH. Metrics -> remote_write to the central Prometheus (on icarus);
-// logs -> loki.write to the central Loki (on icarus). daedalus/morpheus reach
-// them through the public gfn.ermnvldmr.com front (authelia-bypassed + basicauth
-// on /loki and /prometheus/api/v1/write); icarus resolves gfn locally.
-// ============================================================================
+// Optional shared Grafana Alloy configuration.
+// Metrics and logs use the endpoints and credentials supplied through
+// APEX_PROM_PUSH_URL, APEX_LOKI_PUSH_URL, and APEX_OBS_BASIC_AUTH_USER/PASS.
 
 // ----------------------------------------------------------------------------
 // METRICS
@@ -24,8 +16,21 @@ prometheus.exporter.unix "node" {
 
 // Container metrics (cadvisor-equivalent) via the docker socket.
 prometheus.exporter.cadvisor "containers" {
-  docker_host            = "unix:///var/run/docker.sock"
-  store_container_labels = true
+  docker_host                  = "unix:///var/run/docker.sock"
+  store_container_labels       = false
+  allowlisted_container_labels = [
+    "com.docker.compose.project",
+    "com.docker.compose.service",
+  ]
+  docker_only                  = true
+  enabled_metrics              = [
+    "cpu",
+    "cpuLoad",
+    "diskIO",
+    "memory",
+    "network",
+    "oom_event",
+  ]
 }
 
 // Stamp a static `node` label onto each exporter's targets.
@@ -34,7 +39,14 @@ discovery.relabel "node_metrics" {
   rule {
     action       = "replace"
     target_label = "node"
-    replacement  = sys.env("APEX_NODE")
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+  // Stamp a stable `instance` (the node's public host) instead of the default,
+  // which is the Alloy container id/address.
+  rule {
+    action       = "replace"
+    target_label = "instance"
+    replacement  = sys.env("APEX_NODE_HOST")
   }
 }
 discovery.relabel "container_metrics" {
@@ -42,22 +54,40 @@ discovery.relabel "container_metrics" {
   rule {
     action       = "replace"
     target_label = "node"
-    replacement  = sys.env("APEX_NODE")
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+  rule {
+    action       = "replace"
+    target_label = "instance"
+    replacement  = sys.env("APEX_NODE_HOST")
   }
 }
 
 prometheus.scrape "node" {
   targets         = discovery.relabel.node_metrics.output
   forward_to      = [prometheus.remote_write.default.receiver]
-  scrape_interval = "15s"
+  scrape_interval = "30s"
 }
 prometheus.scrape "containers" {
   targets         = discovery.relabel.container_metrics.output
   forward_to      = [prometheus.remote_write.default.receiver]
-  scrape_interval = "15s"
+  scrape_interval = "30s"
 }
 
-// Push metrics to the central Prometheus (needs --web.enable-remote-write-receiver).
+// The CrowdSec scrape requires a reachable host metrics endpoint on port 6060.
+// If no endpoint is configured, this target reports down.
+prometheus.scrape "crowdsec" {
+  targets = [{
+    __address__ = "host.docker.internal:6060",
+    instance    = sys.env("APEX_NODE_HOST"),
+    node        = sys.env("APEX_NODE_HOST"),
+    job         = "crowdsec",
+  }]
+  forward_to      = [prometheus.remote_write.default.receiver]
+  scrape_interval = "30s"
+}
+
+// The configured Prometheus-compatible endpoint must accept remote writes.
 prometheus.remote_write "default" {
   endpoint {
     url = sys.env("APEX_PROM_PUSH_URL")
@@ -65,6 +95,20 @@ prometheus.remote_write "default" {
       username = sys.env("APEX_OBS_BASIC_AUTH_USER")
       password = sys.env("APEX_OBS_BASIC_AUTH_PASS")
     }
+    queue_config {
+      capacity             = 5000
+      max_shards           = 10
+      min_shards           = 1
+      max_samples_per_send = 1000
+      batch_send_deadline  = "5s"
+      min_backoff          = "100ms"
+      max_backoff          = "10s"
+    }
+  }
+  wal {
+    truncate_frequency = "30m"
+    min_keepalive_time = "5m"
+    max_keepalive_time = "30m"
   }
 }
 
@@ -77,7 +121,7 @@ loki.source.journal "systemd" {
   path    = "/var/log/journal"
   max_age = "12h"
   labels  = {
-    node = sys.env("APEX_NODE"),
+    node = sys.env("APEX_NODE_HOST"),
     job  = "systemd-journal",
   }
   forward_to = [loki.write.default.receiver]
@@ -98,7 +142,7 @@ discovery.relabel "container_logs" {
   rule {
     action       = "replace"
     target_label = "node"
-    replacement  = sys.env("APEX_NODE")
+    replacement  = sys.env("APEX_NODE_HOST")
   }
 }
 loki.source.docker "containers" {

etc/alloy/native-host.alloy

@@ -0,0 +1,102 @@
+// Shared native Grafana Alloy configuration for the physical host.
+// Metrics and logs use the endpoints and credentials supplied through
+// APEX_PROM_PUSH_URL, APEX_LOKI_PUSH_URL, and APEX_OBS_BASIC_AUTH_USER/PASS.
+
+// ----------------------------------------------------------------------------
+// METRICS
+// ----------------------------------------------------------------------------
+
+// Native filesystem views; physical capacity is collected on the Proxmox host.
+prometheus.exporter.unix "node" {
+  enable_collectors = ["textfile"]
+  textfile {
+    directory = "/var/lib/alloy/textfile"
+  }
+  procfs_path = "/proc"
+  sysfs_path  = "/sys"
+  rootfs_path = "/"
+}
+
+// Container metrics (cadvisor-equivalent) via the docker socket.
+
+
+// Stamp a static `node` label onto each exporter's targets.
+discovery.relabel "node_metrics" {
+  targets = prometheus.exporter.unix.node.targets
+  rule {
+    action       = "replace"
+    target_label = "node"
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+  // Stamp a stable `instance` (the node's public host) instead of the default,
+  // which is the Alloy container id/address.
+  rule {
+    action       = "replace"
+    target_label = "instance"
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+}
+
+
+prometheus.scrape "node" {
+  targets         = discovery.relabel.node_metrics.output
+  forward_to      = [prometheus.remote_write.default.receiver]
+  scrape_interval = "30s"
+}
+
+
+// The configured Prometheus-compatible endpoint must accept remote writes.
+prometheus.remote_write "default" {
+  endpoint {
+    url = sys.env("APEX_PROM_PUSH_URL")
+    basic_auth {
+      username = sys.env("APEX_OBS_BASIC_AUTH_USER")
+      password = sys.env("APEX_OBS_BASIC_AUTH_PASS")
+    }
+    queue_config {
+      capacity             = 5000
+      max_shards           = 10
+      min_shards           = 1
+      max_samples_per_send = 1000
+      batch_send_deadline  = "5s"
+      min_backoff          = "100ms"
+      max_backoff          = "10s"
+    }
+  }
+  wal {
+    truncate_frequency = "30m"
+    min_keepalive_time = "5m"
+    max_keepalive_time = "30m"
+  }
+}
+
+// ----------------------------------------------------------------------------
+// LOGS
+// ----------------------------------------------------------------------------
+
+// systemd journal (host services, ufw, sshd, crowdsec, etc.).
+loki.source.journal "systemd" {
+  path    = "/var/log/journal"
+  max_age = "12h"
+  labels  = {
+    node = sys.env("APEX_NODE_HOST"),
+    job  = "systemd-journal",
+  }
+  forward_to = [loki.write.default.receiver]
+}
+
+// All docker containers' stdout/stderr.
+
+
+
+
+// Push logs to the central Loki.
+loki.write "default" {
+  endpoint {
+    url = sys.env("APEX_LOKI_PUSH_URL")
+    basic_auth {
+      username = sys.env("APEX_OBS_BASIC_AUTH_USER")
+      password = sys.env("APEX_OBS_BASIC_AUTH_PASS")
+    }
+  }
+}

etc/alloy/native.alloy

@@ -0,0 +1,128 @@
+// Shared native Grafana Alloy configuration for Docker guests.
+// Metrics and logs use the endpoints and credentials supplied through
+// APEX_PROM_PUSH_URL, APEX_LOKI_PUSH_URL, and APEX_OBS_BASIC_AUTH_USER/PASS.
+
+// ----------------------------------------------------------------------------
+// METRICS
+// ----------------------------------------------------------------------------
+
+// Native filesystem views; physical capacity is collected on the Proxmox host.
+
+
+// Container metrics (cadvisor-equivalent) via the docker socket.
+prometheus.exporter.cadvisor "containers" {
+  docker_host                  = "unix:///var/run/docker.sock"
+  store_container_labels       = false
+  allowlisted_container_labels = [
+    "com.docker.compose.project",
+    "com.docker.compose.service",
+  ]
+  docker_only                  = true
+  enabled_metrics              = [
+    "cpu",
+    "cpuLoad",
+    "diskIO",
+    "memory",
+    "network",
+    "oom_event",
+  ]
+}
+
+// Stamp a static `node` label onto each exporter's targets.
+
+discovery.relabel "container_metrics" {
+  targets = prometheus.exporter.cadvisor.containers.targets
+  rule {
+    action       = "replace"
+    target_label = "node"
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+  rule {
+    action       = "replace"
+    target_label = "instance"
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+}
+
+
+prometheus.scrape "containers" {
+  targets         = discovery.relabel.container_metrics.output
+  forward_to      = [prometheus.remote_write.default.receiver]
+  scrape_interval = "30s"
+}
+
+// The configured Prometheus-compatible endpoint must accept remote writes.
+prometheus.remote_write "default" {
+  endpoint {
+    url = sys.env("APEX_PROM_PUSH_URL")
+    basic_auth {
+      username = sys.env("APEX_OBS_BASIC_AUTH_USER")
+      password = sys.env("APEX_OBS_BASIC_AUTH_PASS")
+    }
+    queue_config {
+      capacity             = 5000
+      max_shards           = 10
+      min_shards           = 1
+      max_samples_per_send = 1000
+      batch_send_deadline  = "5s"
+      min_backoff          = "100ms"
+      max_backoff          = "10s"
+    }
+  }
+  wal {
+    truncate_frequency = "30m"
+    min_keepalive_time = "5m"
+    max_keepalive_time = "30m"
+  }
+}
+
+// ----------------------------------------------------------------------------
+// LOGS
+// ----------------------------------------------------------------------------
+
+// systemd journal (host services, ufw, sshd, crowdsec, etc.).
+loki.source.journal "systemd" {
+  path    = "/var/log/journal"
+  max_age = "12h"
+  labels  = {
+    node = sys.env("APEX_NODE_HOST"),
+    job  = "systemd-journal",
+  }
+  forward_to = [loki.write.default.receiver]
+}
+
+// All docker containers' stdout/stderr.
+discovery.docker "containers" {
+  host = "unix:///var/run/docker.sock"
+}
+discovery.relabel "container_logs" {
+  targets = discovery.docker.containers.targets
+  rule {
+    action        = "replace"
+    source_labels = ["__meta_docker_container_name"]
+    regex         = "/?(.*)"
+    target_label  = "container"
+  }
+  rule {
+    action       = "replace"
+    target_label = "node"
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+}
+loki.source.docker "containers" {
+  host          = "unix:///var/run/docker.sock"
+  targets       = discovery.relabel.container_logs.output
+  relabel_rules = discovery.relabel.container_logs.rules
+  forward_to    = [loki.write.default.receiver]
+}
+
+// Push logs to the central Loki.
+loki.write "default" {
+  endpoint {
+    url = sys.env("APEX_LOKI_PUSH_URL")
+    basic_auth {
+      username = sys.env("APEX_OBS_BASIC_AUTH_USER")
+      password = sys.env("APEX_OBS_BASIC_AUTH_PASS")
+    }
+  }
+}

pyproject.toml

@@ -0,0 +1,26 @@
+[project]
+name = "apex-commons"
+version = "2026.10.0"
+description = "Core runtime and orchestration framework for apex.ermnvldmr.com node fleet"
+readme = "README.md"
+license = { text = "BSD-2-Clause" }
+requires-python = ">=3.10"
+
+[project.optional-dependencies]
+dev = ["ruff==0.17.2", "PyYAML>=6.0,<7"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "B", "UP", "S"]
+# Existing formatting and typing modernization are outside correctness checks.
+# Subprocess arguments are constructed explicitly; no implicit shell execution.
+ignore = ["E401", "E501", "E702", "E741", "I001", "UP006", "UP035", "UP045", "S603", "S607"]
+
+[tool.mypy]
+python_version = "3.10"
+warn_return_any = true
+warn_unused_configs = true
+disallow_untyped_defs = false

tests/run.sh

@@ -1,5 +1,6 @@
 #!/usr/bin/env bash
-# Run the full engine test suite (stdlib unittest — no pytest).
+# Run the core and standalone provisioning test suites (stdlib unittest — no pytest).
 set -euo pipefail
 cd "$(dirname "$0")/.."
-exec python3 -m unittest discover -s tests -p 'test_*.py' -v
+python3 -m unittest discover -s tests -p 'test_*.py' -v
+PYTHONPATH=ci/scripts/init-node${PYTHONPATH:+:$PYTHONPATH} python3 -m unittest discover -s ci/scripts/init-node/tests -p 'test_*.py' -v

tests/test_backup_run.py

@@ -0,0 +1,49 @@
+import json
+from pathlib import Path
+from types import SimpleNamespace
+import unittest
+import tempfile
+from unittest.mock import Mock
+
+from engine import overlay
+
+backup = overlay.load_module(str(Path(__file__).resolve().parents[1] / 'actions/backup/run.py'),
+                             'test_backup_action')
+
+
+class BackupRecovery(unittest.TestCase):
+    def test_failed_attempt_preserves_last_success_and_records_new_failure(self):
+        with tempfile.TemporaryDirectory() as directory:
+            path = Path(directory) / 'status.json'
+            backup.write_backup_status(path, 'success', 100)
+            backup.write_backup_status(path, 'running', 200)
+            backup.write_backup_status(path, 'failed', 210)
+            status = json.loads(path.read_text())
+            self.assertEqual(status['last_success'], 100)
+            self.assertEqual(status['state'], 'failed')
+            self.assertEqual(status['updated_at'], 210)
+
+    def test_recovers_namespace_owner_before_dependent_without_starting_other_containers(self):
+        running = {'komf': False, 'komga': False, 'intentionally-stopped': False}
+        def docker(command, **kwargs):
+            if command[1] == 'inspect':
+                return SimpleNamespace(returncode=0, stdout=json.dumps([
+                    {'Id': name, 'State': {'Running': running[name]}} for name in command[2:]]))
+            name = command[2]
+            if name == 'komf' and not running['komga']:
+                return SimpleNamespace(returncode=1)
+            running[name] = True
+            return SimpleNamespace(returncode=0)
+        ctx = SimpleNamespace(sys=SimpleNamespace(run=docker), log=Mock())
+        backup.restore_containers(ctx, ['komf', 'komga'])
+        self.assertTrue(running['komf'])
+        self.assertTrue(running['komga'])
+        self.assertFalse(running['intentionally-stopped'])
+
+    def test_unrecoverable_container_prevents_success(self):
+        def docker(command, **kwargs):
+            return SimpleNamespace(returncode=0 if command[1] == 'inspect' else 1,
+                                   stdout='[{"Id":"broken","State":{"Running":false}}]')
+        ctx = SimpleNamespace(sys=SimpleNamespace(run=docker), log=Mock())
+        with self.assertRaisesRegex(RuntimeError, 'broken'):
+            backup.restore_containers(ctx, ['broken'])

tests/test_compose.py

@@ -0,0 +1,123 @@
+import contextlib
+import io
+import os
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import MagicMock
+
+from actions import compose
+from engine.descriptor import build_parser
+
+
+class TestCompose(unittest.TestCase):
+    def setUp(self):
+        self.tmpdir = tempfile.TemporaryDirectory()
+        self.addCleanup(self.tmpdir.cleanup)
+        self.comp_dir = Path(self.tmpdir.name) / 'compositions'
+        for name in ('apex', 'sso', 'arr'):
+            directory = self.comp_dir / name
+            directory.mkdir(parents=True)
+            (directory / 'docker-compose.yml').write_text(f'name: {name}\n')
+        self.ctx = MagicMock()
+        self.ctx.paths.compositions = str(self.comp_dir)
+        self.ctx.paths.stier3 = self.tmpdir.name
+        self.ctx.vars.return_value = {'APEX_NODE_HOST': 'test-host', 'OTHER': 'ignored'}
+
+    def run_compose(self, *argv):
+        args = build_parser(compose.METADATA, 'apex compose').parse_args(argv)
+        compose.run(self.ctx, args)
+
+    def visited(self):
+        return [os.path.basename(call.kwargs['cwd'])
+                for call in self.ctx.sys.run.call_args_list]
+
+    def test_sequence_all_and_down_reverse(self):
+        for argv, expected in [(('restart',), ['apex', 'arr', 'sso']),
+                               (('-p', 'all', 'up'), ['apex', 'arr', 'sso']),
+                               (('down', 'all'), ['sso', 'arr', 'apex'])]:
+            with self.subTest(argv=argv):
+                self.ctx.sys.run.reset_mock()
+                self.run_compose(*argv)
+                self.assertEqual(self.visited(), expected)
+
+    def test_explicit_target_preserves_service_named_like_composition(self):
+        self.run_compose('-p', 'apex', 'restart', 'sso')
+        self.assertEqual(self.visited(), ['apex'])
+        call = self.ctx.sys.run.call_args
+        self.assertEqual(call.args[0], ['docker', 'compose', 'restart', 'sso'])
+        self.assertEqual(call.kwargs['env']['APEX_NODE_HOST'], 'test-host')
+        self.assertTrue(call.kwargs['check'])
+
+    def test_immediate_positional_composition_compatibility(self):
+        self.run_compose('restart', 'apex', 'traefik')
+        self.assertEqual(self.visited(), ['apex'])
+        self.assertEqual(self.ctx.sys.run.call_args.args[0],
+                         ['docker', 'compose', 'restart', 'traefik'])
+
+    def test_up_detached_compatibility_and_default_detached(self):
+        for argv in [('up', '-d', 'apex'), ('-p', 'apex', 'up'),
+                     ('up', 'apex', '-d')]:
+            with self.subTest(argv=argv):
+                self.ctx.sys.run.reset_mock()
+                self.run_compose(*argv)
+                self.assertEqual(self.visited(), ['apex'])
+                self.assertEqual(self.ctx.sys.run.call_args.args[0],
+                                 ['docker', 'compose', 'up', '-d'])
+
+    def test_flag_values_and_unknown_services_do_not_retarget(self):
+        for argv in [('logs', '--tail', 'sso'), ('restart', 'traefik'),
+                     ('up', '--scale', 'sso'), ('logs', '--', 'apex')]:
+            with self.subTest(argv=argv):
+                self.ctx.sys.run.reset_mock()
+                self.run_compose(*argv)
+                self.assertEqual(self.visited(), ['apex', 'arr', 'sso'])
+                prefix = ['docker', 'compose', argv[0]]
+                if argv[0] == 'up':
+                    prefix += ['-d']
+                self.assertEqual(self.ctx.sys.run.call_args.args[0], prefix + list(argv[1:]))
+
+    def test_invalid_wrapper_options_fail_before_execution(self):
+        for argv, code in [(('-p', 'unknown', 'up'), 1), (('-p', '', 'up'), 1),
+                           (('-p',), 64),
+                           (('up', '-p', 'apex'), 64),
+                           (('up', '--dry-run'), 64), ((), 64),
+                           (('up', '-d', 'apex', 'traefik'), 64)]:
+            with self.subTest(argv=argv), contextlib.redirect_stdout(io.StringIO()), \
+                    contextlib.redirect_stderr(io.StringIO()):
+                with self.assertRaises(SystemExit) as error:
+                    self.run_compose(*argv)
+                self.assertEqual(error.exception.code, code)
+                self.ctx.sys.run.assert_not_called()
+
+    def test_separator_preserves_wrapper_looking_passthrough_tokens(self):
+        self.run_compose('-p', 'apex', 'exec', '--', 'probe', '--dry-run', '-p', 'sso')
+        self.assertEqual(self.ctx.sys.run.call_args.args[0],
+                         ['docker', 'compose', 'exec', '--', 'probe', '--dry-run', '-p', 'sso'])
+
+    def test_env_files_layer_before_action(self):
+        for filename in ('.env', 'apex.env'):
+            (self.comp_dir / 'apex' / filename).write_text('PORT=1234\n')
+        self.run_compose('-p', 'apex', 'config')
+        self.assertEqual(self.ctx.sys.run.call_args.args[0],
+                         ['docker', 'compose', '--env-file', '.env', '--env-file',
+                          'apex.env', 'config'])
+
+    def test_socket_survives_restart_up_and_dry_run(self):
+        shm = Path(self.tmpdir.name) / 'apex' / 'xray' / 'shm'
+        shm.mkdir(parents=True)
+        path = shm / 'live.sock'
+        path.write_bytes(b"socket path must survive")
+        for argv in [('restart', 'apex'), ('-p', 'apex', 'restart', 'traefik'),
+                     ('up',), ('--dry-run', '-p', 'apex', 'logs', '--tail', '20', 'xray')]:
+            with self.subTest(argv=argv):
+                self.ctx.sys.run.reset_mock()
+                self.run_compose(*argv)
+                self.assertTrue(path.exists())
+                if argv[0] == '--dry-run':
+                    self.ctx.sys.run.assert_not_called()
+                    self.assertIn('logs --tail 20 xray', self.ctx.log.info.call_args.args[0])
+
+
+if __name__ == '__main__':
+    unittest.main()

tests/test_compose_hook.py

@@ -0,0 +1,37 @@
+import importlib.util
+from pathlib import Path
+import subprocess
+import tempfile
+from types import SimpleNamespace
+import unittest
+from unittest.mock import Mock, patch
+from engine.lib.sys import Sys
+
+spec = importlib.util.spec_from_file_location('compose_lint', Path(__file__).resolve().parents[1] / 'actions/utils/lint-docker-compose.py')
+lint = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(lint)
+
+
+class HookTests(unittest.TestCase):
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.tmp.cleanup)
+        self.repo = Path(self.tmp.name)
+        subprocess.run(['git', 'init', '-q', str(self.repo)], check=True)
+        self.log = Mock()
+        self.ctx = SimpleNamespace(log=self.log, sys=Sys(self.log), node=None,
+                                   paths=SimpleNamespace(repo_root=str(self.repo), compositions=str(self.repo / 'compositions')))
+
+    def stage(self, name, content):
+        (self.repo / name).write_text(content)
+        subprocess.run(['git', '-C', str(self.repo), 'add', name], check=True)
+
+    def test_documentation_commit_does_not_require_docker(self):
+        self.stage('README.md', 'Documentation\n')
+        with patch.object(lint.shutil, 'which', return_value=None):
+            self.assertEqual(lint.run(self.ctx, SimpleNamespace(hook=True)), 0)
+
+    def test_compose_change_still_requires_docker(self):
+        self.stage('docker-compose.yml', 'services: {}\n')
+        with patch.object(lint.shutil, 'which', return_value=None):
+            self.assertNotEqual(lint.run(self.ctx, SimpleNamespace(hook=True)), 0)

tests/test_configure_alloy.py

@@ -0,0 +1,52 @@
+"""Native Alloy opt-in and validation boundaries."""
+import json
+from pathlib import Path
+import tempfile
+from types import SimpleNamespace
+import unittest
+from unittest.mock import Mock
+
+from engine import overlay
+
+ROOT = Path(__file__).resolve().parents[1]
+alloy = overlay.load_module(str(ROOT / 'actions/configure/alloy.py'), 'test_alloy_action')
+
+
+class NativeAlloy(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.temp.cleanup)
+        self.configs = Path(self.temp.name)
+        self.ctx = SimpleNamespace(paths=SimpleNamespace(configs=str(self.configs), commons=str(ROOT)),
+                                   sys=Mock(), log=Mock())
+
+    def configure(self, **settings):
+        (self.configs / 'alloy').mkdir(exist_ok=True)
+        (self.configs / 'alloy/config.json').write_text(json.dumps(settings))
+
+    def test_unconfigured_node_is_untouched(self):
+        alloy.run(self.ctx, SimpleNamespace(dry_run=False))
+        self.ctx.sys.sudo.assert_not_called()
+
+    def test_invalid_role_or_environment_path_cannot_change_service(self):
+        for role, path in [('unknown', '/etc/alloy/secrets'), ('host', '/etc/alloy/invalid\nUser=root')]:
+            self.configure(role=role, environment_file=path)
+            with self.assertRaises(ValueError):
+                alloy.run(self.ctx, SimpleNamespace(dry_run=False))
+            self.ctx.sys.sudo.assert_not_called()
+
+    def test_dry_run_never_installs_or_starts(self):
+        self.configure(role='guest', environment_file='/etc/alloy/secrets')
+        alloy.run(self.ctx, SimpleNamespace(dry_run=True))
+        self.ctx.sys.sudo.assert_not_called()
+
+    def test_failed_config_validation_leaves_existing_service_untouched(self):
+        self.configure(role='host', environment_file='/etc/alloy/secrets')
+        def execute(command):
+            if command[0] == 'python3':
+                raise RuntimeError('invalid Alloy configuration')
+            self.assertEqual(command, ['test', '-f', '/etc/alloy/secrets'])
+        self.ctx.sys.sudo.side_effect = execute
+        with self.assertRaisesRegex(RuntimeError, 'invalid Alloy'):
+            alloy.run(self.ctx, SimpleNamespace(dry_run=False))
+        self.ctx.sys.daemon_reload.assert_not_called()

tests/test_configure_base.py

@@ -0,0 +1,281 @@
+"""Local baseline policy: fake only package, network, and root privilege boundaries."""
+import hashlib
+import io
+import json
+import os
+from pathlib import Path
+import stat
+import subprocess
+import sys
+import tempfile
+from types import SimpleNamespace
+import unittest
+from unittest.mock import Mock, patch
+
+from engine import overlay
+from engine.lib.host import Host
+from engine.lib.sys import Sys
+
+base = overlay.load_module(
+    str(Path(__file__).resolve().parents[1] / "actions/configure/base.py"),
+    "apex_test_configure_base",
+)
+
+
+class TestBasePackages(unittest.TestCase):
+    def setUp(self):
+        self.system = Mock()
+        self.installed = set(base.BASE_PACKAGES)
+        self.system.run.side_effect = self.query
+        self.system.sudo.return_value = subprocess.CompletedProcess([], 0, stdout="")
+
+    def query(self, command, **kwargs):
+        self.assertEqual(command[:3], ["dpkg-query", "-W", "-f=${Status}"])
+        status = "install ok installed" if command[-1] in self.installed else "unknown ok not-installed"
+        return subprocess.CompletedProcess(command, 0, stdout=status)
+
+    def test_installed_rsyslog_and_ufw_do_not_require_executable_or_apt_update(self):
+        base._install_missing(self.system, base.BASE_PACKAGES)
+        self.assertEqual({call.args[0][-1] for call in self.system.run.call_args_list},
+                         {"curl", "wget", "git", "rsyslog", "ufw"})
+        self.system.sudo.assert_not_called()
+
+    def test_only_missing_packages_are_installed_after_safe_simulation(self):
+        self.installed.remove("ufw")
+        self.system.sudo.return_value = subprocess.CompletedProcess(
+            [], 0, stdout="Inst ufw (0.36.2 Debian:13 [all])\nConf ufw (0.36.2 Debian:13 [all])\n",
+        )
+        base._install_missing(self.system, base.BASE_PACKAGES)
+        commands = [call.args[0] for call in self.system.sudo.call_args_list]
+        self.assertEqual(len(commands), 3)
+        self.assertIn("update", commands[0])
+        self.assertEqual(commands[1], [*commands[2], "--simulate"])
+        install = commands[2]
+        self.assertEqual(install[install.index("install"):], [
+            "install", "-y", "--no-install-recommends", "--no-upgrade", "--no-remove", "ufw",
+        ])
+
+    def test_privileged_apt_environment_survives_sudo_filtering(self):
+        execute = subprocess.run
+        operations = []
+
+        def filtered_sudo(command, **kwargs):
+            if command[0] == "dpkg-query":
+                return subprocess.CompletedProcess(command, 1, stdout="")
+            self.assertEqual(command[0], "sudo")
+            incoming = kwargs.get("env") or os.environ
+            filtered = {"PATH": os.defpath}
+            if "LC_ALL" in incoming:
+                filtered["LC_ALL"] = incoming["LC_ALL"]
+            apt_index = command.index("apt-get")
+            probe = execute([
+                *command[1:apt_index], sys.executable, "-I", "-c",
+                "import json, os; print(json.dumps([os.getenv('LC_ALL'), "
+                "os.getenv('DEBIAN_FRONTEND')]))",
+            ], env=filtered, check=True, text=True, capture_output=True)
+            self.assertEqual(json.loads(probe.stdout), ["C", "noninteractive"])
+            operations.append(command[apt_index:])
+            plan = "Inst ufw (0.36.2 Debian:13 [all])\n" if "--simulate" in command else ""
+            return subprocess.CompletedProcess(command, 0, stdout=plan)
+
+        with patch("engine.lib.sys.subprocess.run", side_effect=filtered_sudo):
+            base._install_missing(Sys(Mock()), ["ufw"])
+        self.assertEqual(len(operations), 3)
+        self.assertIn("update", operations[0])
+        self.assertIn("--simulate", operations[1])
+        self.assertIn("install", operations[2])
+        self.assertNotIn("--simulate", operations[2])
+
+    def test_dependency_upgrades_and_removals_abort_before_installation(self):
+        self.installed.remove("curl")
+        for plan in ("Inst libcurl4 [7.0] (8.0 Debian:13 [amd64])\n",
+                     "Remv libcurl4 [7.0]\n"):
+            with self.subTest(plan=plan):
+                self.system.sudo.reset_mock()
+                self.system.sudo.return_value = subprocess.CompletedProcess([], 0, stdout=plan)
+                with self.assertRaisesRegex(RuntimeError, "upgrades/removals"):
+                    base._install_missing(self.system, base.BASE_PACKAGES)
+                self.assertEqual(self.system.sudo.call_count, 2)
+
+    def test_failed_package_query_is_not_treated_as_installed(self):
+        self.system.run.return_value = subprocess.CompletedProcess([], 1, stdout="")
+        self.system.run.side_effect = None
+        base._install_missing(self.system, ["rsyslog"])
+        self.assertEqual(self.system.sudo.call_args.args[0][-1], "rsyslog")
+
+
+class TestPinnedHelper(unittest.TestCase):
+    def setUp(self):
+        temporary = tempfile.TemporaryDirectory()
+        self.addCleanup(temporary.cleanup)
+        self.root = Path(temporary.name).resolve()
+        self.path = self.root / "bin" / "ufw-docker"
+        self.path.parent.mkdir()
+        self.payload = b"#!/bin/sh\nprintf 'verified fixture\\n'\n"
+        self.wrong_owner = None
+        self.wrong_group = None
+        self.system = Mock()
+        self.system.sudo.side_effect = self.sudo
+        self.ctx = SimpleNamespace(log=Mock(), sys=self.system)
+        self.ctx.host = Host(self.ctx.log, self.system)
+        real_lstat = Path.lstat
+
+        def root_lstat(path, *args, **kwargs):
+            info = list(real_lstat(path, *args, **kwargs))
+            info[4] = 1001 if path == self.wrong_owner else 0
+            info[5] = 1001 if path == self.wrong_group else 0
+            if path not in (self.path, self.path.parent, self.root):
+                info[0] &= ~0o022
+            return os.stat_result(info)
+
+        # Model root-owned ancestry without sudo/chown; content, modes and links stay real.
+        self.addCleanup(patch.stopall)
+        patch.object(Path, "lstat", autospec=True, side_effect=root_lstat).start()
+        patch.object(base, "UFW_DOCKER_PATH", self.path).start()
+        patch.object(base, "UFW_DOCKER_SHA256", hashlib.sha256(self.payload).hexdigest()).start()
+        self.download = patch.object(base.urllib.request, "urlopen").start()
+        self.download.return_value = io.BytesIO(self.payload)
+
+    def sudo(self, command, **kwargs):
+        if command[:3] == ["python3", "-I", "-c"]:
+            return subprocess.run([sys.executable, *command[1:]], check=True, text=True, **kwargs)
+        self.assertEqual(command, [str(self.path), "install"])
+        self.assertEqual(self.path.read_bytes(), self.payload)
+        self.assertEqual(stat.S_IMODE(self.path.stat().st_mode), 0o755)
+        return subprocess.CompletedProcess(command, 0)
+
+    def write_helper(self, payload=None, mode=0o755):
+        self.path.write_bytes(self.payload if payload is None else payload)
+        self.path.chmod(mode)
+
+    def test_verified_existing_helper_executes_without_download(self):
+        self.write_helper()
+        base._install_helper(self.ctx)
+        self.download.assert_not_called()
+        self.assertEqual(self.system.sudo.call_args.args[0], [str(self.path), "install"])
+        self.assertEqual(self.system.sudo.call_count, 1)
+
+    def test_wrong_executable_content_is_replaced_only_with_verified_download(self):
+        self.write_helper(b"#!/bin/sh\necho wrong\n")
+        base._install_helper(self.ctx)
+        self.assertEqual(self.path.read_bytes(), self.payload)
+        self.assertEqual(self.system.sudo.call_count, 2)
+        self.download.assert_called_once_with(base.UFW_DOCKER_URL, timeout=30)
+
+    def test_missing_helper_is_published_with_executable_mode(self):
+        base._install_helper(self.ctx)
+        self.assertEqual(self.path.read_bytes(), self.payload)
+        self.assertEqual(stat.S_IMODE(self.path.stat().st_mode), 0o755)
+        self.assertEqual(list(self.path.parent.iterdir()), [self.path])
+
+    def test_verified_nonexecutable_content_is_reused_and_mode_repaired(self):
+        self.write_helper(mode=0o600)
+        base._install_helper(self.ctx)
+        self.download.assert_not_called()
+        self.assertEqual(stat.S_IMODE(self.path.stat().st_mode), 0o755)
+        self.assertEqual(self.system.sudo.call_count, 2)
+
+    def test_bad_checksum_never_publishes_or_executes(self):
+        self.download.return_value = io.BytesIO(b"not the pinned helper")
+        with self.assertRaisesRegex(RuntimeError, "checksum mismatch"):
+            base._install_helper(self.ctx)
+        self.system.sudo.assert_not_called()
+        self.assertFalse(self.path.exists())
+
+    def test_bad_download_preserves_existing_wrong_helper_without_executing_it(self):
+        self.write_helper(b"old unknown executable")
+        self.download.return_value = io.BytesIO(b"bad download")
+        with self.assertRaisesRegex(RuntimeError, "checksum mismatch"):
+            base._install_helper(self.ctx)
+        self.system.sudo.assert_not_called()
+        self.assertEqual(self.path.read_bytes(), b"old unknown executable")
+
+    def assert_rejected_before_network_or_sudo(self):
+        with self.assertRaisesRegex(RuntimeError, "Unsafe ufw-docker"):
+            base._install_helper(self.ctx)
+        self.download.assert_not_called()
+        self.system.sudo.assert_not_called()
+
+    def test_symlink_target_is_rejected(self):
+        target = self.root / "untrusted"
+        target.write_bytes(self.payload)
+        self.path.symlink_to(target)
+        self.assert_rejected_before_network_or_sudo()
+
+    def test_dangling_symlink_target_is_rejected(self):
+        self.path.symlink_to(self.root / "missing")
+        self.assert_rejected_before_network_or_sudo()
+
+    def test_symlink_ancestor_is_rejected(self):
+        self.path.parent.rmdir()
+        target = self.root / "real-bin"
+        target.mkdir()
+        self.path.parent.symlink_to(target)
+        self.assert_rejected_before_network_or_sudo()
+
+    def test_writable_ancestor_is_rejected(self):
+        self.path.parent.chmod(0o777)
+        self.assert_rejected_before_network_or_sudo()
+
+    def test_unsafe_helper_preflight_precedes_package_network_or_mutation(self):
+        self.path.parent.chmod(0o777)
+        with patch.object(base.os.path, "isfile", return_value=True):
+            with self.assertRaisesRegex(RuntimeError, "Unsafe ufw-docker"):
+                base.run(self.ctx, None)
+        self.download.assert_not_called()
+        self.system.run.assert_not_called()
+        self.system.sudo.assert_not_called()
+
+    def test_untrusted_owner_ancestor_is_rejected(self):
+        self.wrong_owner = self.path.parent
+        self.assert_rejected_before_network_or_sudo()
+
+    def test_untrusted_file_owner_is_rejected(self):
+        self.write_helper()
+        self.wrong_owner = self.path
+        self.assert_rejected_before_network_or_sudo()
+
+    def test_untrusted_file_group_is_rejected(self):
+        self.write_helper()
+        self.wrong_group = self.path
+        self.assert_rejected_before_network_or_sudo()
+
+    def test_writable_helper_is_rejected(self):
+        self.write_helper(mode=0o775)
+        self.assert_rejected_before_network_or_sudo()
+
+    def test_hardlinked_helper_is_rejected(self):
+        self.write_helper()
+        os.link(self.path, self.root / "other-link")
+        self.assert_rejected_before_network_or_sudo()
+
+    def test_nonregular_helper_is_rejected(self):
+        self.path.mkdir()
+        self.assert_rejected_before_network_or_sudo()
+
+    def test_path_changed_during_download_is_rejected_before_publication(self):
+        def download(*args, **kwargs):
+            self.path.symlink_to(self.root / "untrusted")
+            return io.BytesIO(self.payload)
+
+        self.download.side_effect = download
+        with self.assertRaisesRegex(RuntimeError, "Unsafe ufw-docker"):
+            base._install_helper(self.ctx)
+        self.system.sudo.assert_not_called()
+
+    def test_changed_content_after_publication_is_not_executed(self):
+        real_sudo = self.sudo
+
+        def tamper(command, **kwargs):
+            result = real_sudo(command, **kwargs)
+            self.path.write_bytes(b"changed after publication")
+            return result
+
+        self.system.sudo.side_effect = tamper
+        with self.assertRaisesRegex(RuntimeError, "verification failed"):
+            base._install_helper(self.ctx)
+        self.assertEqual(self.system.sudo.call_count, 1)
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_env.py

@@ -0,0 +1,182 @@
+import contextlib
+import io
+import json
+import os
+import shutil
+import stat
+import subprocess
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import MagicMock
+
+from actions import env
+from engine.descriptor import build_parser
+from engine.lib.host import Host
+
+
+class TestEnv(unittest.TestCase):
+    def setUp(self):
+        self.tmpdir = tempfile.TemporaryDirectory()
+        self.addCleanup(self.tmpdir.cleanup)
+        self.core = Path(self.tmpdir.name) / 'compositions' / 'apex'
+        self.core.mkdir(parents=True)
+        self.ctx = MagicMock()
+        self.ctx.paths.compositions = str(self.core.parent)
+        self.ctx.vars.return_value = {
+            'APEX_SUBNET': '198.18.0.0/24',
+            'APEX_NODE_HOST': 'test-host',
+            'OTHER_VAR': 'ignore_me',
+        }
+        self.ctx.host = Host(self.ctx.log, self.ctx.sys)
+
+    def run_env(self, *argv):
+        args = build_parser(env.METADATA, 'apex env').parse_args(argv)
+        with contextlib.redirect_stdout(io.StringIO()) as out:
+            env.run(self.ctx, args)
+        return out.getvalue()
+
+    def test_shell_exports_round_trip_literal_values(self):
+        value = "a$b # 'quote' \"double\" \\backslash\r\nnext"
+        self.ctx.vars.return_value['APEX_NODE_HOST'] = value
+        exports = self.run_env('--export')
+        self.assertEqual(self.run_env(), exports)
+        script = exports + '\nprintf %s "$APEX_NODE_HOST"'
+        result = subprocess.run(['/bin/sh', '-c', script], capture_output=True, check=True,
+                                timeout=30, env={})
+        self.assertEqual(result.stdout.decode(), value)
+        self.assertNotIn('OTHER_VAR', script)
+
+    def test_dotenv_does_not_disclose_other_values_or_existing_secrets(self):
+        (self.core / '.env').write_text('SECRET=synthetic-private\n')
+        output = self.run_env('--dotenv')
+        self.assertIn('APEX_NODE_HOST=', output)
+        self.assertNotIn('export ', output)
+        self.assertNotIn('SECRET', output)
+        self.assertNotIn('OTHER_VAR', output)
+
+    def test_repeat_sync_preserves_credentials_bytes_and_updates_managed_values(self):
+        original = b'# user credentials\r\nTOKEN=\'a$#"\\literal\'\r\nUID=1000\r\nGID=1000'
+        target = self.core / '.env'
+        target.write_bytes(original)
+        target.chmod(0o600)
+        (self.core / 'apex.env').write_bytes(b'COMPOSE_PROFILES=core\r\nPORT=1234\r\n')
+        self.run_env('--sync')
+        first = target.read_bytes()
+        self.assertTrue(first.startswith(original + b'\n'))
+        self.assertIn(b'COMPOSE_PROFILES=core\r\nPORT=1234\r\n', first)
+        self.run_env('--sync')
+        self.assertEqual(first, target.read_bytes())
+        self.ctx.vars.return_value['APEX_NODE_HOST'] = 'new-host'
+        self.run_env('--sync')
+        self.assertTrue(target.read_bytes().startswith(original + b'\n'))
+        self.assertNotIn(b'test-host', target.read_bytes())
+        self.assertEqual(stat.S_IMODE(target.stat().st_mode), 0o600)
+
+    def test_sync_preserves_symlink_and_private_target(self):
+        target = Path(self.tmpdir.name) / 'private.env'
+        target.write_text('TOKEN=synthetic\n')
+        link = self.core / '.env'
+        link.symlink_to(target)
+        self.run_env('--sync')
+        self.assertTrue(link.is_symlink())
+        self.assertIn('TOKEN=synthetic', target.read_text())
+        self.assertIn('APEX_NODE_HOST=', target.read_text())
+        self.assertEqual(stat.S_IMODE(target.stat().st_mode), 0o600)
+
+    def test_write_basename_is_standalone_and_private(self):
+        previous = os.getcwd()
+        self.addCleanup(os.chdir, previous)
+        os.chdir(self.tmpdir.name)
+        Path('generated.env').write_text('OLD=removed\n')
+        self.run_env('--write', 'generated.env')
+        content = Path('generated.env').read_text()
+        self.assertIn('APEX_NODE_HOST=', content)
+        self.assertNotIn('OLD=', content)
+        self.assertNotIn('BEGIN', content)
+        self.assertEqual(stat.S_IMODE(Path('generated.env').stat().st_mode), 0o600)
+
+    def test_incompatible_modes_fail_without_writing(self):
+        for argv in [('--sync', '--dotenv'), ('--sync', '--write', 'file'),
+                     ('--export', '--dotenv'), ('--export', '--write', 'file')]:
+            with self.subTest(argv=argv):
+                with self.assertRaises(SystemExit) as error:
+                    self.run_env(*argv)
+                self.assertEqual(error.exception.code, 64)
+        self.assertFalse((self.core / '.env').exists())
+
+    def test_uncertain_managed_markers_fail_without_changing_file(self):
+        start = '# BEGIN APEX MANAGED ENV'
+        end = '# END APEX MANAGED ENV'
+        for content in [start+'\n', end+'\n', end+'\n'+start+'\n',
+                        start+'\n'+start+'\n'+end+'\n',
+                        start+'\n'+end+'\n'+start+'\n'+end+'\n',
+                        'TOKEN="'+start+'"\n']:
+            target = self.core / '.env'
+            target.write_text(content)
+            with self.subTest(content=content):
+                with self.assertRaises(SystemExit) as error:
+                    self.run_env('--sync')
+                self.assertEqual(error.exception.code, 65)
+                self.assertEqual(target.read_text(), content)
+
+    def test_source_marker_collision_fails_without_writing(self):
+        (self.core / 'apex.env').write_text('# BEGIN APEX MANAGED ENV\n')
+        with self.assertRaises(SystemExit) as error:
+            self.run_env('--sync')
+        self.assertEqual(error.exception.code, 65)
+        self.assertFalse((self.core / '.env').exists())
+        (self.core / 'apex.env').unlink()
+        self.ctx.vars.return_value['APEX_NODE_HOST'] = '# END APEX MANAGED ENV'
+        with self.assertRaises(SystemExit) as error:
+            self.run_env('--sync')
+        self.assertEqual(error.exception.code, 65)
+        self.assertFalse((self.core / '.env').exists())
+
+    def test_bare_compose_interprets_literals_precedence_and_profiles(self):
+        binary = shutil.which('docker-compose')
+        if binary is None:
+            self.skipTest('Standalone docker-compose required for daemon-free config check')
+        values = {
+            'APEX_LITERAL': "a$b ${UNSET} $$ # 'quote' \"double\" \\backslash",
+            'APEX_TRAILING': 'last\\',
+            'APEX_LINES': 'one\r\ntwo\nthree\tfour',
+            'APEX_EMPTY': '',
+            'APEX_ESCAPES': "\\n \\\\ \\' \\\"",
+            'APEX_NODE_HOST': 'synthesized',
+        }
+        self.ctx.vars.return_value = values
+        (self.core / '.env').write_text('UID=1000\nGID=1001\nPORT=1\nAPEX_NODE_HOST=old\n'
+                                       "TOKEN='literal$token#hash'\n")
+        (self.core / 'apex.env').write_text('PORT=2\nAPEX_NODE_HOST=public\n'
+                                           'COMPOSE_PROFILES=chosen\n')
+        keys = [*values, 'UID', 'GID', 'PORT', 'TOKEN']
+        (self.core / 'docker-compose.yml').write_text(
+            'services:\n  probe:\n    image: busybox\n    profiles: [chosen]\n'
+            '    environment:\n' + ''.join(f'      {key}: ${{{key}}}\n' for key in keys))
+        self.run_env('--sync')
+        result = subprocess.run([binary, 'config', '--format', 'json'], cwd=self.core,
+                                capture_output=True, check=True, timeout=30, env={})
+        actual = json.loads(result.stdout)['services']['probe']['environment']
+        expected = {**values, 'UID': '1000', 'GID': '1001', 'PORT': '2',
+                    'TOKEN': 'literal$token#hash'}
+        # Compose re-escapes literal dollars when serializing a reusable configuration.
+        self.assertEqual(actual, {k: v.replace('$', '$$') for k, v in expected.items()})
+        result = subprocess.run([binary, 'config', '--environment'], cwd=self.core,
+                                capture_output=True, check=True, timeout=30, env={})
+        for key, value in {**expected, 'COMPOSE_PROFILES': 'chosen'}.items():
+            self.assertIn(f'{key}={value}\n'.encode(), result.stdout)
+        with (self.core / '.env').open('a') as target:
+            target.write('# later user addition\nPORT=999\n')
+        self.run_env('--sync')
+        self.assertIn('# later user addition\nPORT=999\n', (self.core / '.env').read_text())
+        result = subprocess.run([binary, 'config', '--format', 'json'], cwd=self.core,
+                                capture_output=True, check=True, timeout=30, env={})
+        self.assertEqual(json.loads(result.stdout)['services']['probe']['environment']['PORT'], '2')
+        result = subprocess.run([binary, 'config', '--services'], cwd=self.core,
+                                capture_output=True, check=True, timeout=30, env={})
+        self.assertEqual(result.stdout.strip(), b'probe')
+
+
+if __name__ == '__main__':
+    unittest.main()

tests/test_host.py

@@ -1,6 +1,200 @@
 # tests/test_host.py
+import inspect
+import os
+from pathlib import Path
+import stat
+import subprocess
+import sys
+import tempfile
 import unittest
-from engine.lib.host import apply_block
+from unittest.mock import Mock, patch
+
+from engine.lib.host import Host, apply_block
+
+
+def _check_backup_chmod(chmod, expected_group):
+    import os
+    import stat
+
+    def checked(path, mode, *args, **kwargs):
+        result = chmod(path, mode, *args, **kwargs)
+        info = os.stat(path)
+        with open(path, "rb") as stream:
+            content = stream.read()
+        if content and stat.S_IMODE(info.st_mode) & 0o077 and info.st_gid != expected_group:
+            raise AssertionError("Backup content exposed to the inherited staging group")
+        return result
+
+    return checked
+
+
+class TestHostWrites(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.temp.cleanup)
+        self.root = Path(self.temp.name).resolve()
+        self.log = Mock()
+        self.host = Host(self.log, Mock())
+
+    def test_existing_private_modes_and_owner_survive_replacement(self):
+        for mode in (0o600, 0o640):
+            with self.subTest(mode=mode):
+                path = self.root / "secret"
+                path.write_text("old")
+                path.chmod(mode)
+                previous = path.stat()
+                self.host.write_file(str(path), "new", backup=False)
+                self.assertEqual(path.read_text(), "new")
+                self.assertEqual(stat.S_IMODE(path.stat().st_mode), mode)
+                self.assertEqual((path.stat().st_uid, path.stat().st_gid),
+                                 (previous.st_uid, previous.st_gid))
+
+    def test_new_basename_defaults_to_0644(self):
+        previous = Path.cwd()
+        try:
+            os.chdir(self.root)
+            self.host.write_file("settings", "value", backup=False)
+        finally:
+            os.chdir(previous)
+        path = self.root / "settings"
+        self.assertEqual(path.read_text(), "value")
+        self.assertEqual(stat.S_IMODE(path.stat().st_mode), 0o644)
+
+    def test_existing_supplementary_group_survives_replacement(self):
+        path = self.root / "group-owned"
+        path.write_text("old")
+        groups = set(os.getgroups()) - {path.stat().st_gid}
+        if not groups:
+            self.skipTest("No alternate supplementary group available")
+        group = next(iter(groups))
+        try:
+            os.chown(path, -1, group)
+        except PermissionError:
+            self.skipTest("Filesystem does not permit changing to a supplementary group")
+        self.host.write_file(str(path), "new", backup=False)
+        self.assertEqual(path.read_text(), "new")
+        self.assertEqual(path.stat().st_gid, group)
+
+    def test_explicit_mode_is_applied_before_atomic_publication(self):
+        path = self.root / "secret"
+        path.write_text("old")
+        replace = os.replace
+        published = []
+
+        def observe(source, destination):
+            self.assertEqual(path.read_text(), "old")
+            self.assertEqual(Path(source).parent, self.root)
+            self.assertEqual(stat.S_IMODE(os.stat(source).st_mode), 0o600)
+            self.assertEqual(Path(source).read_text(), "new secret")
+            published.append(destination)
+            replace(source, destination)
+
+        with patch("os.replace", side_effect=observe):
+            self.host.write_file(str(path), "new secret", mode="0600", backup=False)
+        self.assertEqual(len(published), 1)
+        self.assertEqual(path.read_text(), "new secret")
+
+    def test_symlink_remains_and_target_receives_new_content(self):
+        target = self.root / "tier" / "secret"
+        target.parent.mkdir()
+        target.write_text("old")
+        target.chmod(0o600)
+        link = self.root / ".env"
+        link.symlink_to("tier/secret")
+        self.host.write_file(str(link), "new", backup=False)
+        self.assertTrue(link.is_symlink())
+        self.assertEqual(os.readlink(link), "tier/secret")
+        self.assertEqual(target.read_text(), "new")
+        self.assertEqual(stat.S_IMODE(target.stat().st_mode), 0o600)
+
+    def test_dry_run_does_not_create_parents_or_backups(self):
+        self.host.write_file(str(self.root / "missing" / "secret"), "new", dry_run=True)
+        self.assertEqual(list(self.root.iterdir()), [])
+
+    def test_backup_preserves_old_private_content(self):
+        path = self.root / "secret"
+        path.write_text("old")
+        path.chmod(0o600)
+        self.host.write_file(str(path), "new")
+        backups = list(self.root.glob("secret.*.bak"))
+        self.assertEqual(len(backups), 1)
+        self.assertEqual(backups[0].read_text(), "old")
+        self.assertEqual(stat.S_IMODE(backups[0].stat().st_mode), 0o600)
+        self.assertEqual(path.read_text(), "new")
+
+    def test_backup_sets_owner_before_exposing_group_readable_content(self):
+        path = self.root / "group-secret"
+        path.write_text("synthetic group-only secret")
+        groups = set(os.getgroups()) - {path.stat().st_gid}
+        if not groups:
+            self.skipTest("No alternate supplementary group available")
+        group = next(iter(groups))
+        try:
+            os.chown(path, -1, group)
+        except PermissionError:
+            self.skipTest("Filesystem does not permit changing to a supplementary group")
+        path.chmod(0o640)
+        os.utime(path, (1600000000, 1600000000))
+
+        def local_sudo(command, **kwargs):
+            self.assertEqual(command[:3], ["python3", "-I", "-c"])
+            probe = ("import os\n" + inspect.getsource(_check_backup_chmod) +
+                     f"\nos.chmod = _check_backup_chmod(os.chmod, {group})\n")
+            result = subprocess.run([sys.executable, "-I", "-c", probe + command[3]],
+                                    check=False, text=True, capture_output=True, **kwargs)
+            self.assertEqual(result.returncode, 0, result.stderr)
+            return result
+
+        self.host.sys.ok.return_value = True
+        self.host.sys.sudo.side_effect = local_sudo
+        for sudo in (False, True):
+            with self.subTest(sudo=sudo):
+                with patch("os.chmod", side_effect=_check_backup_chmod(os.chmod, group)):
+                    backup = Path(self.host.backup(str(path), sudo=sudo))
+                self.assertEqual(backup.read_bytes(), path.read_bytes())
+                self.assertEqual(backup.stat().st_gid, group)
+                self.assertEqual(stat.S_IMODE(backup.stat().st_mode), 0o640)
+                self.assertEqual(backup.stat().st_mtime_ns, path.stat().st_mtime_ns)
+                backup.unlink()
+
+    def test_publication_failure_keeps_original_and_cleans_staging(self):
+        path = self.root / "secret"
+        path.write_text("old")
+        with patch("os.replace", side_effect=OSError("publication failed")):
+            with self.assertRaisesRegex(OSError, "publication failed"):
+                self.host.write_file(str(path), "new", backup=False)
+        self.assertEqual(path.read_text(), "old")
+        self.assertEqual(list(self.root.iterdir()), [path])
+        self.log.success.assert_not_called()
+
+    def test_write_failure_keeps_original_and_cleans_staging(self):
+        path = self.root / "secret"
+        path.write_text("old")
+        with patch("os.fsync", side_effect=OSError("disk full")):
+            with self.assertRaisesRegex(OSError, "disk full"):
+                self.host.write_file(str(path), "new", backup=False)
+        self.assertEqual(path.read_text(), "old")
+        self.assertEqual(list(self.root.iterdir()), [path])
+        self.log.success.assert_not_called()
+
+    def test_sudo_runs_same_writer_without_copying_over_live_target(self):
+        target = self.root / "secret"
+        target.write_text("old")
+        target.chmod(0o640)
+        link = self.root / ".env"
+        link.symlink_to("secret")
+
+        def local_sudo(command, **kwargs):
+            self.assertEqual(command[:3], ["python3", "-I", "-c"])
+            return subprocess.run([sys.executable, *command[1:]], check=True,
+                                  text=True, **kwargs)
+
+        self.host.sys.sudo.side_effect = local_sudo
+        self.host.write_file(str(link), "new", sudo=True, backup=False)
+        self.assertTrue(link.is_symlink())
+        self.assertEqual(target.read_text(), "new")
+        self.assertEqual(stat.S_IMODE(target.stat().st_mode), 0o640)
+        self.assertEqual(set(self.root.iterdir()), {link, target})
 
 class TestApplyBlock(unittest.TestCase):
     def test_insert_after_anchor(self):

tests/test_identity.py

@@ -1,42 +1,39 @@
 # tests/test_identity.py
 import os, tempfile, unittest
-from engine.identity import resolve, Node
+from engine.identity import resolve
 
 class TestIdentity(unittest.TestCase):
-    def _envfile(self, text, dirname=None):
+    def _envfile(self, text):
         d = tempfile.mkdtemp()
-        if dirname:
-            d = os.path.join(d, dirname)
-            os.makedirs(d)
         with open(os.path.join(d, "node.env"), "w") as f:
             f.write(text)
         return d
 
-    def test_fqdn_is_source_of_truth(self):
-        repo = self._envfile("APEX_CLUSTER=a1\nAPEX_SUBNET=198.18.11.0/24\n")
-        n, warns = resolve("daedalus.a1.apex.ermnvldmr.com", repo)
-        self.assertEqual((n.name, n.cluster, n.subnet), ("daedalus", "a1", "198.18.11.0/24"))
+    def test_explicit_host_and_fqdn(self):
+        repo = self._envfile("APEX_NODE_FQDN=node1.dc1.example.com\n"
+                             "APEX_NODE_HOST=node1.example.com\n"
+                             "APEX_SUBNET=10.0.0.0/24\n")
+        n, warns = resolve("build-laptop", repo)   # hostname ignored when node.env has FQDN
+        self.assertEqual((n.name, n.host, n.fqdn, n.subnet),
+                         ("node1", "node1.example.com", "node1.dc1.example.com", "10.0.0.0/24"))
         self.assertEqual(warns, [])
 
-    def test_cluster_drift_warns_hostname_wins(self):
-        repo = self._envfile("APEX_CLUSTER=a9\nAPEX_SUBNET=198.18.11.0/24\n")
-        n, warns = resolve("daedalus.a1.apex.ermnvldmr.com", repo)
-        self.assertEqual(n.cluster, "a1")           # hostname wins
-        self.assertTrue(any("drift" in w.lower() for w in warns))
+    def test_host_defaults_to_fqdn(self):
+        repo = self._envfile("APEX_NODE_FQDN=n1.example.com\nAPEX_SUBNET=10.0.0.0/24\n")
+        n, warns = resolve("ignored", repo)
+        self.assertEqual((n.name, n.host, n.fqdn), ("n1", "n1.example.com", "n1.example.com"))
+        self.assertEqual(warns, [])
 
-    def test_fallback_when_fqdn_not_apex_shaped(self):
-        # repo dir basename carries the node name in fallback (real host layout:
-        # $HOME/morpheus.apex.ermnvldmr.com)
-        repo = self._envfile("APEX_CLUSTER=a2\nAPEX_SUBNET=198.18.13.0/24\n",
-                             dirname="morpheus.apex.ermnvldmr.com")
-        n, warns = resolve("build-laptop", repo)
-        self.assertEqual((n.name, n.cluster, n.subnet), ("morpheus", "a2", "198.18.13.0/24"))
-        self.assertTrue(any("fqdn" in w.lower() for w in warns))
+    def test_fqdn_falls_back_to_hostname(self):
+        repo = self._envfile("APEX_SUBNET=10.0.0.0/24\n")
+        n, warns = resolve("box.local", repo)
+        self.assertEqual((n.name, n.host, n.fqdn), ("box", "box.local", "box.local"))
+        self.assertTrue(any("APEX_NODE_FQDN" in w for w in warns))
 
     def test_missing_subnet_is_error(self):
-        repo = self._envfile("APEX_CLUSTER=a1\n")
+        repo = self._envfile("APEX_NODE_FQDN=n1.example.com\n")
         with self.assertRaises(SystemExit):
-            resolve("daedalus.a1.apex.ermnvldmr.com", repo)
+            resolve("n1.example.com", repo)
 
 if __name__ == "__main__":
     unittest.main()

tests/test_notify.py

@@ -1,15 +1,17 @@
 # tests/test_notify.py
+import os
+import tempfile
 import unittest
-from engine.lib.notify import build_text
+from engine.lib.notify import build_text, resolve_bot_url
 
 class TestNotify(unittest.TestCase):
-    def test_instance_uses_apex_prefix(self):
-        t = build_text("Title", "msg", "daedalus", "INFO", "2026-07-04 00:00:00 MSK")
-        self.assertIn("com.ermnvldmr.apex.daedalus", t)
-        self.assertIn("#instance_com_ermnvldmr_apex_daedalus", t)
+    def test_instance_is_the_host(self):
+        t = build_text("Title", "msg", "node1.example.com", "INFO", "2026-07-04 00:00:00 UTC")
+        self.assertIn("node1.example.com", t)
+        self.assertIn("#instance_node1_example_com", t)   # dots -> underscores in the hashtag
 
     def test_html_escaped_and_truncated(self):
-        t = build_text("A & B", "<script>" + "x" * 300, "d", "ERROR", "T")
+        t = build_text("A & B", "<script>" + "x" * 4000, "d", "ERROR", "T")
         self.assertIn("A &amp; B", t)
         self.assertIn("&lt;script&gt;", t)
         self.assertIn("...", t)
@@ -20,5 +22,19 @@ class TestNotify(unittest.TestCase):
         self.assertIn("Resolved", t)
         self.assertIn("✅", t)
 
+    def test_literal_bot_url_is_unchanged(self):
+        self.assertEqual(resolve_bot_url("https://example.test/bot"),
+                         "https://example.test/bot")
+
+    def test_bot_url_file_reference(self):
+        fd, path = tempfile.mkstemp()
+        try:
+            with os.fdopen(fd, "w") as f:
+                f.write("https://example.test/secret\n")
+            self.assertEqual(resolve_bot_url("@" + path),
+                             "https://example.test/secret")
+        finally:
+            os.unlink(path)
+
 if __name__ == "__main__":
     unittest.main()

tests/test_sync_packages.py

@@ -0,0 +1,318 @@
+import json
+import subprocess
+import tempfile
+from pathlib import Path
+import unittest
+from types import SimpleNamespace
+from unittest.mock import Mock, patch
+
+from actions.sync import packages
+
+
+def digest(char):
+    return 'sha256:' + char * 64
+
+
+MANIFEST = 'application/vnd.oci.image.manifest.v1+json'
+INDEX = 'application/vnd.oci.image.index.v1+json'
+
+
+class Commands:
+    def __init__(self):
+        self.containers = [{'Image': digest('a'), 'Reference': 'alpine:3.21'}]
+        self.local = {digest('a'): {'Id': digest('a'), 'Os': 'linux',
+                                  'Architecture': 'amd64', 'RepoDigests': ['alpine@' + digest('b')]}}
+        self.remote = {'alpine:3.21': {'mediaType': MANIFEST,
+                                     'digest': digest('c')}}
+        self.raw = {'alpine@' + digest('c'): {'schemaVersion': 2, 'mediaType': MANIFEST,
+                                            'config': {'digest': digest('a')}, 'layers': []}}
+        self.config = {'alpine@' + digest('c'): {'os': 'linux', 'architecture': 'amd64'}}
+        self.calls = []
+        self.fail = set()
+        self.apt = ''
+        self.compose = {"name": "apex", "services": {}}
+        self.compose_options = []
+
+    def ok(self, cmd):
+        self.calls.append(cmd)
+        return tuple(cmd) not in self.fail
+
+    def run(self, cmd, check=False, capture=False, **kwargs):
+        self.calls.append(cmd)
+        if cmd[:1] == ['timeout']:
+            cmd = cmd[4:]
+        if tuple(cmd) in self.fail:
+            return subprocess.CompletedProcess(cmd, 1, '', 'registry unavailable')
+        if cmd[:2] == ['docker', 'compose']:
+            self.compose_options.append(kwargs)
+            data = self.compose if isinstance(self.compose, str) else json.dumps(self.compose)
+        elif cmd[:2] == ['bash', '-c']:
+            if 'apt-get --just-print' not in cmd[2]:
+                raise AssertionError('Image references must never reach a shell')
+            data = self.apt
+        elif cmd == ['docker', 'ps', '--quiet', '--no-trunc']:
+            data = '\n'.join(str(i) for i in range(len(self.containers)))
+        elif cmd[:3] == ['docker', 'container', 'inspect']:
+            data = '\n'.join(json.dumps(c) for c in self.containers)
+        elif cmd[:3] == ['docker', 'image', 'inspect']:
+            data = json.dumps([self.local[cmd[3]]])
+        elif cmd == ['docker', 'buildx', 'version']:
+            data = 'github.com/docker/buildx v0.37.1'
+        elif cmd[:4] == ['docker', 'buildx', 'imagetools', 'inspect']:
+            table = self.raw if cmd[4] == '--raw' else (
+                self.remote if cmd[5] == '{{json .Manifest}}' else self.config)
+            value = table[cmd[-1]]
+            data = value if isinstance(value, str) else json.dumps(value)
+        else:
+            raise AssertionError(f'Unexpected command: {cmd}')
+        return subprocess.CompletedProcess(cmd, 0, data, '')
+
+
+class TestSyncPackages(unittest.TestCase):
+    def setUp(self):
+        self.commands = Commands()
+        self.ctx = SimpleNamespace(sys=self.commands, log=Mock(), node=SimpleNamespace(name='node'),
+                                   notify=Mock())
+        self.ctx.notify.info.return_value = True
+        self.tmp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.tmp.cleanup)
+        self.ctx.paths = SimpleNamespace(compositions=self.tmp.name)
+        self.ctx.vars = lambda: {'APEX_NODE_HOST': 'test-node'}
+        self.args = SimpleNamespace(telegram_bot_url='test-notifier')
+
+    def report(self):
+        with patch.object(packages.os.path, 'isfile', return_value=True), \
+                patch.object(packages.shutil, 'which', return_value='/usr/bin/docker'):
+            packages.run(self.ctx, self.args)
+        return self.ctx.notify.info.call_args.args[2]
+
+    def containerd(self, manifest='c'):
+        local = self.commands.local.pop(digest('a'))
+        local.update(Id=digest('b'), Descriptor={'mediaType': INDEX, 'digest': digest('b')})
+        self.commands.local[digest('b')] = local
+        self.commands.containers[0].update(Image=digest('b'), ImageManifestDescriptor={
+            'mediaType': MANIFEST, 'digest': digest(manifest),
+            'platform': {'os': 'linux', 'architecture': 'amd64'}})
+
+    def test_containerd_index_id_compares_actual_platform_manifest(self):
+        self.containerd()
+        self.index()
+        self.assertIn('1 current', self.report())
+
+    def test_containerd_single_manifest_is_current_despite_different_config_digest(self):
+        self.containerd()
+        self.assertIn('1 current', self.report())
+
+    def test_containerd_changed_platform_manifest_is_update(self):
+        self.containerd('d')
+        self.index()
+        self.assertIn('1 updates', self.report())
+
+    def test_containerd_without_platform_descriptor_is_unchecked(self):
+        self.containerd()
+        del self.commands.containers[0]['ImageManifestDescriptor']
+        self.assertIn('1 unchecked', self.report())
+
+    def test_two_containerd_platforms_of_one_index_are_checked_separately(self):
+        self.containerd()
+        self.index()
+        self.commands.containers.append({
+            'Image': digest('b'), 'Reference': 'alpine:3.21', 'ImageManifestDescriptor': {
+                'mediaType': MANIFEST, 'digest': digest('d'),
+                'platform': {'os': 'linux', 'architecture': 'arm64'}}})
+        self.commands.raw['alpine@' + digest('d')] = {
+            'schemaVersion': 2, 'mediaType': MANIFEST, 'config': {'digest': digest('f')}}
+        self.assertIn('2 current', self.report())
+
+    def test_invalid_container_platform_descriptor_is_unchecked(self):
+        self.containerd()
+        self.commands.containers[0]['ImageManifestDescriptor']['digest'] = 'invalid'
+        self.assertIn('1 unchecked', self.report())
+
+    def build_service(self, image='alpine:3.21'):
+        core = Path(self.tmp.name) / 'apex'
+        core.mkdir()
+        for filename in ('docker-compose.yml', '.env', 'apex.env'):
+            (core / filename).touch()
+        self.commands.compose = {'name': 'apex', 'services': {
+            'custom': {'build': {'context': '.'}, 'environment': {'TOKEN': 'private-test-value'}}}}
+        if image is not None:
+            self.commands.compose['services']['custom']['image'] = image
+        self.commands.containers[0]['Labels'] = {
+            'com.docker.compose.project': 'apex', 'com.docker.compose.service': 'custom'}
+
+    def test_declared_compose_build_with_synthetic_repo_digest_is_local(self):
+        self.containerd()
+        self.build_service()
+        self.assertIn('1 local/untracked', self.report())
+        self.assertFalse(any('imagetools' in c for c in self.commands.calls))
+        self.assertIn(['timeout', '--signal=TERM', '--kill-after=5s', '30s', 'docker',
+                       'compose', '--env-file', '.env', '--env-file', 'apex.env',
+                       'config', '--format', 'json'], self.commands.calls)
+        self.assertEqual(self.commands.compose_options[0]['env']['APEX_NODE_HOST'], 'test-node')
+        self.assertNotIn('private-test-value', str(self.ctx.log.mock_calls))
+
+    def test_implicit_compose_build_image_uses_service_provenance(self):
+        self.containerd()
+        self.build_service(image=None)
+        self.assertIn('1 local/untracked', self.report())
+
+    def test_changed_build_declaration_does_not_hide_old_registry_image(self):
+        self.build_service(image='different:latest')
+        self.assertIn('1 current', self.report())
+
+    def test_same_image_name_without_build_service_provenance_is_registry_checked(self):
+        self.build_service()
+        self.commands.containers[0]['Labels']['com.docker.compose.service'] = 'other'
+        self.assertIn('1 current', self.report())
+
+    def test_failed_compose_resolution_is_unchecked_without_secret_output(self):
+        self.build_service()
+        self.commands.compose = 'private-test-value'
+        self.assertIn('Images: unchecked (Compose inspection failed)', self.report())
+        self.assertNotIn('private-test-value', str(self.ctx.log.mock_calls))
+
+    def test_matching_config_digest_is_current_without_comparing_index_digest(self):
+        self.assertIn('1 current', self.report())
+        self.assertIn(['docker', 'image', 'inspect', digest('a')], self.commands.calls)
+
+    def test_moved_tag_compares_running_image_not_new_local_tag(self):
+        self.commands.raw['alpine@' + digest('c')]['config']['digest'] = digest('d')
+        self.assertIn('1 updates', self.report())
+        self.assertNotIn(['docker', 'image', 'inspect', 'alpine:3.21'], self.commands.calls)
+
+    def index(self, variants=None):
+        self.commands.remote['alpine:3.21'] = {'schemaVersion': 2, 'mediaType': INDEX,
+            'digest': digest('e'), 'manifests': variants or [
+                {'mediaType': MANIFEST, 'digest': digest('d'),
+                 'platform': {'os': 'linux', 'architecture': 'arm64'}},
+                {'mediaType': MANIFEST, 'digest': digest('c'),
+                 'platform': {'os': 'linux', 'architecture': 'amd64'}},
+                {'mediaType': MANIFEST, 'digest': digest('f'),
+                 'platform': {'os': 'unknown', 'architecture': 'unknown'}}]}
+
+    def test_index_only_other_architecture_changed_is_current(self):
+        self.index()
+        self.assertIn('1 current', self.report())
+        self.assertFalse(any(c[-1] == 'alpine@' + digest('d') for c in self.commands.calls))
+
+    def test_index_matching_platform_changed_is_update(self):
+        self.index()
+        self.commands.raw['alpine@' + digest('c')]['config']['digest'] = digest('d')
+        self.assertIn('1 updates', self.report())
+
+    def test_ambiguous_or_missing_platform_is_unchecked(self):
+        for platforms in ([{'os': 'linux', 'architecture': 'arm64'}],
+                          [{'os': 'linux', 'architecture': 'amd64'}] * 2):
+            with self.subTest(platforms=platforms):
+                self.index([{'mediaType': MANIFEST, 'digest': digest('c'), 'platform': p}
+                            for p in platforms])
+                self.assertIn('1 unchecked', self.report())
+
+    def test_arm64_default_variant_matches_v8(self):
+        self.commands.local[digest('a')]['Architecture'] = 'arm64'
+        self.index([{'mediaType': MANIFEST, 'digest': digest('c'),
+                     'platform': {'os': 'linux', 'architecture': 'arm64', 'variant': 'v8'}}])
+        self.assertIn('1 current', self.report())
+
+    def test_single_manifest_different_platform_is_unchecked(self):
+        self.commands.raw['alpine@' + digest('c')]['config']['digest'] = digest('d')
+        self.commands.config['alpine@' + digest('c')]['architecture'] = 'arm64'
+        self.assertIn('1 unchecked', self.report())
+
+    def test_registry_port_and_tag_are_preserved(self):
+        ref = 'registry.example:5000/team/app:stable'
+        self.commands.containers[0]['Reference'] = ref
+        self.commands.remote[ref] = self.commands.remote['alpine:3.21']
+        self.commands.raw['registry.example:5000/team/app@' + digest('c')] = self.commands.raw['alpine@' + digest('c')]
+        self.assertIn('1 current', self.report())
+        self.assertIn(['timeout', '--signal=TERM', '--kill-after=5s', '30s',
+                       'docker', 'buildx', 'imagetools', 'inspect', '--raw',
+                       'registry.example:5000/team/app@' + digest('c')], self.commands.calls)
+
+    def test_registry_port_without_tag_uses_the_whole_repository(self):
+        ref = 'localhost:5000/team/app'
+        self.commands.containers[0]['Reference'] = ref
+        self.commands.remote[ref] = self.commands.remote['alpine:3.21']
+        self.commands.raw[ref + '@' + digest('c')] = self.commands.raw['alpine@' + digest('c')]
+        self.assertIn('1 current', self.report())
+
+    def test_two_running_versions_of_one_tag_are_both_checked(self):
+        self.commands.containers.append({'Image': digest('d'), 'Reference': 'alpine:3.21'})
+        self.commands.local[digest('d')] = {**self.commands.local[digest('a')], 'Id': digest('d')}
+        message = self.report()
+        self.assertIn('1 current', message)
+        self.assertIn('1 updates', message)
+
+    def test_malformed_local_image_is_unchecked_not_a_local_build(self):
+        del self.commands.local[digest('a')]['RepoDigests']
+        self.assertIn('1 unchecked', self.report())
+
+    def test_untrusted_reference_is_argv_not_shell_input(self):
+        self.commands.containers[0]['Reference'] = "registry.example/app:$(touch /tmp/nope)"
+        self.assertIn('1 unchecked', self.report())
+        self.assertTrue(all('touch' not in c[-1] for c in self.commands.calls if c[0] == 'bash'))
+
+    def test_apt_failure_is_explicit_and_images_still_checked(self):
+        self.commands.fail.add(('sudo', 'apt-get', 'update'))
+        message = self.report()
+        self.assertIn('APT: failed to check', message)
+        self.assertIn('1 current', message)
+
+    def test_pinned_and_local_images_are_explicit_and_not_queried(self):
+        self.commands.containers = [{'Image': digest('a'), 'Reference': 'alpine@' + digest('b')},
+                                    {'Image': digest('d'), 'Reference': 'project-local:latest'}]
+        self.commands.local[digest('d')] = {**self.commands.local[digest('a')],
+                                          'Id': digest('d'), 'RepoDigests': []}
+        message = self.report()
+        self.assertIn('1 pinned', message)
+        self.assertIn('1 local/untracked', message)
+        self.assertFalse(any('imagetools' in c for c in self.commands.calls))
+
+    def test_registry_timeout_is_bounded_and_reported_unchecked(self):
+        original = self.commands.run
+
+        def time_out(cmd, **kwargs):
+            if 'imagetools' in cmd:
+                self.assertEqual(cmd[:4], ['timeout', '--signal=TERM', '--kill-after=5s', '30s'])
+                return subprocess.CompletedProcess(cmd, 124, '', '')
+            return original(cmd, **kwargs)
+
+        self.commands.run = time_out
+        self.assertIn('1 unchecked', self.report())
+
+    def test_failed_registry_and_malformed_results_are_unchecked(self):
+        for value in ('not-json', '{}', '[]', '{"mediaType":"unknown"}'):
+            with self.subTest(value=value):
+                self.commands.remote['alpine:3.21'] = value
+                self.assertIn('1 unchecked', self.report())
+        self.commands.fail.add(('docker', 'buildx', 'imagetools', 'inspect', '--format',
+                                '{{json .Manifest}}', 'alpine:3.21'))
+        self.assertIn('1 unchecked', self.report())
+
+    def test_bad_config_digest_is_unchecked(self):
+        self.commands.raw['alpine@' + digest('c')]['config'] = {}
+        self.assertIn('1 unchecked', self.report())
+
+    def test_docker_listing_failure_is_not_empty_success(self):
+        self.commands.fail.add(('docker', 'ps', '--quiet', '--no-trunc'))
+        self.assertIn('unchecked', self.report())
+
+    def test_buildx_missing_still_reports_unchecked(self):
+        self.commands.fail.add(('docker', 'buildx', 'version'))
+        self.assertIn('1 unchecked', self.report())
+
+    def test_image_counts_precede_long_package_details_and_notification_failure_exits(self):
+        self.commands.apt = '\n'.join('package-' + str(i) for i in range(100))
+        message = self.report()
+        self.assertIn('Images:', message[:200])
+        self.assertIn('1 current', message[:200])
+        self.assertIn('100 pending', message[:200])
+        self.assertNotIn('Skopeo', message)
+        self.ctx.notify.info.return_value = False
+        with self.assertRaises(SystemExit):
+            self.report()
+
+
+if __name__ == '__main__':
+    unittest.main()

tests/test_sync_repository.py

@@ -0,0 +1,133 @@
+import os
+import subprocess
+import tempfile
+import unittest
+from pathlib import Path
+from types import SimpleNamespace
+from unittest.mock import Mock
+from actions.sync import repository
+from engine.lib.sys import Sys
+
+
+class RepositorySyncTests(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.temp.cleanup)
+        self.root = Path(self.temp.name)
+        self.remote = self.root / "remote.git"
+        self.repo = self.root / "node"
+        self.env = dict(os.environ, GIT_AUTHOR_NAME="Test", GIT_AUTHOR_EMAIL="test@example.invalid",
+                        GIT_COMMITTER_NAME="Test", GIT_COMMITTER_EMAIL="test@example.invalid",
+                        GIT_ALLOW_PROTOCOL="file")
+        self.git("init", "--bare", str(self.remote))
+        self.git("clone", str(self.remote), str(self.repo))
+        self.git("checkout", "-b", "main", cwd=self.repo)
+        (self.repo / "compositions").mkdir()
+        (self.repo / "compositions/service").write_text("base\n")
+        self.commons_source = self.root / "shared"
+        self.git("init", str(self.commons_source))
+        (self.commons_source / "file").write_text("shared\n")
+        self.git("add", ".", cwd=self.commons_source)
+        self.git("commit", "-m", "shared", cwd=self.commons_source)
+        self.git("submodule", "add", str(self.commons_source), "commons", cwd=self.repo)
+        self.git("add", ".", cwd=self.repo)
+        self.git("commit", "-m", "base", cwd=self.repo)
+        self.git("push", "origin", "main", cwd=self.repo)
+        self.log = Mock()
+        class Commands(Sys):
+            def run(inner, cmd, **kw):
+                kw["env"] = dict(self.env, **kw.get("env", {}))
+                kw["env"]["GIT_ALLOW_PROTOCOL"] = "file"
+                return super().run(cmd, **kw)
+        self.ctx = SimpleNamespace(log=self.log, sys=Commands(self.log), notify=Mock(),
+                                   node=SimpleNamespace(name="node"),
+                                   paths=SimpleNamespace(repo_root=str(self.repo), commons=str(self.repo / "commons"),
+                                                         compositions=str(self.repo / "compositions")))
+        self.args = SimpleNamespace(telegram_bot_url=None)
+
+    def git(self, *args, cwd=None):
+        return subprocess.check_output(["git", *args], cwd=cwd, env=self.env, stderr=subprocess.PIPE, text=True).strip()
+
+    def sync(self):
+        repository.run(self.ctx, self.args)
+
+    def test_local_change_publishes_main_without_sync_branch_or_notification(self):
+        (self.repo / "compositions/service").write_text("changed\n")
+        self.sync()
+        self.assertEqual(self.git("--git-dir=" + str(self.remote), "show", "main:compositions/service"), "changed")
+        self.assertEqual(self.git("--git-dir=" + str(self.remote), "for-each-ref", "--format=%(refname)", "refs/heads"), "refs/heads/main")
+        self.ctx.notify.success.assert_not_called()
+        self.ctx.notify.error.assert_not_called()
+
+    def test_noop_keeps_head(self):
+        before = self.git("rev-parse", "HEAD", cwd=self.repo)
+        self.sync()
+        self.assertEqual(before, self.git("rev-parse", "HEAD", cwd=self.repo))
+
+    def test_other_branch_does_not_stage_changes(self):
+        self.git("checkout", "-b", "work", cwd=self.repo)
+        (self.repo / "compositions/service").write_text("changed\n")
+        with self.assertRaises(SystemExit):
+            self.sync()
+        self.assertEqual(self.git("diff", "--cached", cwd=self.repo), "")
+
+    def test_dirty_commons_blocks_parent_commit(self):
+        before = self.git("rev-parse", "HEAD", cwd=self.repo)
+        (self.repo / "commons/file").write_text("dirty\n")
+        with self.assertRaises(SystemExit):
+            self.sync()
+        self.assertEqual(before, self.git("rev-parse", "HEAD", cwd=self.repo))
+
+    def test_stale_submodule_after_failed_update_cannot_rollback_pin(self):
+        old = self.git("rev-parse", "HEAD", cwd=self.repo / "commons")
+        (self.commons_source / "file").write_text("new shared\n")
+        self.git("add", ".", cwd=self.commons_source)
+        self.git("commit", "-m", "new shared", cwd=self.commons_source)
+        new = self.git("rev-parse", "HEAD", cwd=self.commons_source)
+        self.git("-C", str(self.repo / "commons"), "fetch", "origin")
+        self.git("update-index", "--cacheinfo", "160000," + new + ",commons", cwd=self.repo)
+        self.git("commit", "-m", "advance pin", cwd=self.repo)
+        before = self.git("rev-parse", "HEAD", cwd=self.repo)
+        with self.assertRaises(SystemExit):
+            self.sync()
+        self.assertEqual(before, self.git("rev-parse", "HEAD", cwd=self.repo))
+        self.assertEqual(old, self.git("rev-parse", "HEAD", cwd=self.repo / "commons"))
+        self.git("submodule", "update", "--init", cwd=self.repo)
+        self.sync()
+        self.assertEqual(new, self.git("--git-dir=" + str(self.remote), "rev-parse", "main:commons"))
+
+    def test_concurrent_main_advance_rejects_push_without_overwriting(self):
+        original = self.ctx.sys.run
+        other = self.root / "other"
+        self.git("clone", "--branch", "main", str(self.remote), str(other))
+        def race(cmd, **kw):
+            if "push" in cmd:
+                (other / "compositions/service").write_text("concurrent\n")
+                self.git("add", ".", cwd=other)
+                self.git("commit", "-m", "concurrent", cwd=other)
+                self.git("push", "origin", "main", cwd=other)
+            return original(cmd, **kw)
+        self.ctx.sys.run = race
+        (self.repo / "compositions/service").write_text("local\n")
+        with self.assertRaises(SystemExit):
+            self.sync()
+        self.assertEqual("concurrent", self.git("--git-dir=" + str(self.remote), "show", "main:compositions/service"))
+        self.assertEqual("local", self.git("show", "HEAD:compositions/service", cwd=self.repo))
+
+    def test_conflict_preserves_local_commit_and_remote(self):
+        other = self.root / "other"
+        self.git("clone", "--branch", "main", str(self.remote), str(other))
+        (other / "compositions/service").write_text("remote\n")
+        self.git("add", ".", cwd=other)
+        self.git("commit", "-m", "remote", cwd=other)
+        self.git("push", "origin", "main", cwd=other)
+        (self.repo / "compositions/service").write_text("local\n")
+        with self.assertRaises(SystemExit):
+            self.sync()
+        self.assertEqual(self.git("show", "HEAD:compositions/service", cwd=self.repo), "local")
+        self.assertEqual(self.git("--git-dir=" + str(self.remote), "show", "main:compositions/service"), "remote")
+        self.assertFalse((self.repo / ".git/rebase-merge").exists())
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_timezone.py

@@ -0,0 +1,72 @@
+import importlib.util
+from pathlib import Path
+import tempfile
+from types import SimpleNamespace
+import unittest
+from unittest.mock import Mock
+
+ROOT = Path(__file__).resolve().parents[1]
+spec = importlib.util.spec_from_file_location('timezone_action', ROOT / 'actions/configure/timezone.py')
+action = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(action)
+
+
+class TestTimezone(unittest.TestCase):
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.tmp.cleanup)
+        self.source = Path(self.tmp.name) / 'timezone'
+        self.zone = 'UTC'
+        self.cron_refreshes = []
+        def read(*args, **kwargs):
+            return SimpleNamespace(stdout=self.zone+'\n')
+        def apply(args):
+            self.zone = args[-1]
+        self.system = Mock()
+        self.system.run.side_effect = read
+        self.system.sudo.side_effect = apply
+        self.system.service_is_active.return_value = True
+        def restart(name):
+            self.cron_refreshes.append(name)
+            return True
+        self.system.restart.side_effect = restart
+        self.ctx = SimpleNamespace(paths=SimpleNamespace(configs=self.tmp.name), sys=self.system, log=Mock())
+
+    def test_timezone_transition_refreshes_cron_once_and_repeat_is_noop(self):
+        self.source.write_text('Europe/Moscow\n')
+        action.run(self.ctx, SimpleNamespace(dry_run=False))
+        self.assertEqual(self.zone, 'Europe/Moscow')
+        self.assertEqual(self.cron_refreshes, ['cron'])
+        action.run(self.ctx, SimpleNamespace(dry_run=False))
+        self.assertEqual(self.cron_refreshes, ['cron'])
+
+    def test_preview_and_absent_config_preserve_host(self):
+        action.run(self.ctx, SimpleNamespace(dry_run=False))
+        self.source.write_text('Europe/Moscow\n')
+        action.run(self.ctx, SimpleNamespace(dry_run=True))
+        self.assertEqual(self.zone, 'UTC')
+        self.assertEqual(self.cron_refreshes, [])
+        self.system.sudo.assert_not_called()
+
+    def test_failed_cron_restart_is_reported_as_failure(self):
+        self.source.write_text('Europe/Moscow\n')
+        self.system.restart.side_effect = None
+        self.system.restart.return_value = False
+        with self.assertRaises(SystemExit):
+            action.run(self.ctx, SimpleNamespace(dry_run=False))
+        self.ctx.log.success.assert_not_called()
+        self.assertEqual(self.zone, 'Europe/Moscow')
+
+    def test_inactive_cron_is_not_started(self):
+        self.source.write_text('Europe/Moscow\n')
+        self.system.service_is_active.return_value = False
+        action.run(self.ctx, SimpleNamespace(dry_run=False))
+        self.assertEqual(self.zone, 'Europe/Moscow')
+        self.assertEqual(self.cron_refreshes, [])
+
+    def test_invalid_zone_does_not_mutate_host(self):
+        self.source.write_text('../not-a-zone\n')
+        with self.assertRaises(SystemExit):
+            action.run(self.ctx, SimpleNamespace(dry_run=False))
+        self.system.sudo.assert_not_called()
+        self.assertEqual(self.cron_refreshes, [])

tests/test_tooling.py

@@ -0,0 +1,74 @@
+"""Exercise verification entrypoints in disposable source trees."""
+import os
+from pathlib import Path
+import shutil
+import subprocess
+import sys
+import tempfile
+import unittest
+
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+class ToolingTests(unittest.TestCase):
+    def test_lint_fallback_recurses_and_propagates_each_root_failure(self):
+        with tempfile.TemporaryDirectory() as directory:
+            root = Path(directory)
+            shutil.copyfile(ROOT / 'Makefile', root / 'Makefile')
+            binaries = root / 'bin'
+            binaries.mkdir()
+            make = shutil.which('make')
+            self.assertIsNotNone(make, 'make is required for the declared verification workflow')
+            (binaries / 'make').symlink_to(make)
+            (binaries / 'python3').symlink_to(sys.executable)
+            environment = dict(os.environ, PATH=str(binaries))
+            sources = []
+            for name in ('engine', 'actions', 'tests', 'ci/scripts'):
+                source = root / name / 'nested' / 'case.py'
+                source.parent.mkdir(parents=True)
+                source.write_text('value = 1\n')
+                sources.append(source)
+            for source in sources:
+                with self.subTest(root=source.parent.parent):
+                    invalid = source.with_name('invalid.py')
+                    invalid.write_text('def broken(:\n')
+                    result = subprocess.run([str(binaries / 'make'), 'lint'], cwd=root,
+                                            env=environment, capture_output=True, text=True,
+                                            timeout=30)
+                    self.assertNotEqual(result.returncode, 0, result.stdout + result.stderr)
+                    self.assertNotIn('verified python bytecode compilation', result.stdout)
+                    invalid.unlink()
+            result = subprocess.run([str(binaries / 'make'), 'lint'], cwd=root,
+                                    env=environment, capture_output=True, text=True, timeout=30)
+            self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+            self.assertIn('verified python bytecode compilation', result.stdout)
+
+    def test_suite_runner_discovers_both_suites_and_propagates_failures(self):
+        with tempfile.TemporaryDirectory() as directory:
+            root = Path(directory)
+            suites = (root / 'tests', root / 'ci/scripts/init-node/tests')
+            for suite in suites:
+                suite.mkdir(parents=True)
+            shutil.copyfile(ROOT / 'tests/run.sh', suites[0] / 'run.sh')
+            binaries = root / 'bin'
+            binaries.mkdir()
+            (binaries / 'python3').symlink_to(sys.executable)
+            environment = dict(os.environ, PATH=str(binaries) + os.pathsep + os.environ['PATH'])
+            for failure in (None, 0, 1):
+                with self.subTest(failing_suite=failure):
+                    for index, suite in enumerate(suites):
+                        (suite / 'test_case.py').write_text(
+                            'import unittest\nclass Case(unittest.TestCase):\n'
+                            '    def test_suite_' + str(index) + '(self):\n'
+                            '        self.assertTrue(' + repr(index != failure) + ')\n')
+                    result = subprocess.run(['/bin/bash', 'tests/run.sh'], cwd=root,
+                                            env=environment, capture_output=True, text=True,
+                                            timeout=30)
+                    if failure is None:
+                        self.assertEqual(result.returncode, 0, result.stderr)
+                        self.assertIn('test_suite_0', result.stderr)
+                        self.assertIn('test_suite_1', result.stderr)
+                    else:
+                        self.assertNotEqual(result.returncode, 0, result.stderr)
+                        self.assertIn('FAIL: test_suite_' + str(failure), result.stderr)

tests/test_tpl.py

@@ -1,22 +1,31 @@
 # tests/test_tpl.py
-import os, tempfile, unittest
-from engine.lib.tpl import Tpl
+import os
+import shutil
+import tempfile
+import unittest
+
 from engine.lib.log import Log
+from engine.lib.tpl import Tpl
+
 
 class TestTpl(unittest.TestCase):
     def setUp(self):
         self.cfg = tempfile.mkdtemp()
         self.tpl = Tpl(self.cfg, Log())
 
+    def tearDown(self):
+        shutil.rmtree(self.cfg, ignore_errors=True)
+
     def _w(self, rel, body):
         p = os.path.join(self.cfg, rel)
         os.makedirs(os.path.dirname(p), exist_ok=True)
-        open(p, "w").write(body)
+        with open(p, "w", encoding="utf-8") as f:
+            f.write(body)
 
     def test_render_substitutes_known_vars(self):
         self._w("cron/crontab", "root=${APEX_TIER1} node=$APEX_NODE\n")
-        out = self.tpl.render("cron/crontab", {"APEX_TIER1": "/mnt/t1", "APEX_NODE": "daedalus"})
-        self.assertEqual(out, "root=/mnt/t1 node=daedalus\n")
+        out = self.tpl.render("cron/crontab", {"APEX_TIER1": "/mnt/t1", "APEX_NODE": "node1"})
+        self.assertEqual(out, "root=/mnt/t1 node=node1\n")
 
     def test_render_leaves_shell_command_substitution(self):
         self._w("c", "x $(cat /f) ${APEX_NODE}\n")
@@ -29,5 +38,6 @@ class TestTpl(unittest.TestCase):
         out = self.tpl.render_dir("fail2ban", {"X": "1", "Y": "2"})
         self.assertEqual(out, {"jail.local": "a=1\n", "sub/extra.conf": "b=2\n"})
 
+
 if __name__ == "__main__":
     unittest.main()

tests/test_ufw.py

@@ -0,0 +1,27 @@
+import unittest
+import os
+import shutil
+from engine.lib.ufw import resolve_ufw_docker_bin
+
+class TestUfwDockerResolution(unittest.TestCase):
+    def test_prefers_path_or_usr_local_bin_when_present(self):
+        bin_path = resolve_ufw_docker_bin()
+        # Should return a valid string path
+        self.assertIsInstance(bin_path, str)
+        self.assertTrue(len(bin_path) > 0)
+
+    def test_fallback_path(self):
+        # If not in PATH, should fall back to user local bin
+        original_which = shutil.which
+        original_isfile = os.path.isfile
+        try:
+            shutil.which = lambda cmd: None
+            os.path.isfile = lambda p: False if p == "/usr/local/bin/ufw-docker" else original_isfile(p)
+            expected = os.path.expanduser("~/.local/bin/ufw-docker")
+            self.assertEqual(resolve_ufw_docker_bin(), expected)
+        finally:
+            shutil.which = original_which
+            os.path.isfile = original_isfile
+
+if __name__ == "__main__":
+    unittest.main()

vendor/VENDORED.md

@@ -1,13 +0,0 @@
-# Vendored third-party libraries
-
-Runtime is **git + python3 stdlib only**. This directory is currently **empty**.
-
-If a pure-python library ever becomes truly necessary:
-1. It must be **pure python** (no C extensions — there is no build step).
-2. Check it into `vendor/<pkg>/`. The `apex` launcher adds `vendor/` to `sys.path`.
-3. Add one record row below: `name, version, source URL, git SHA, sha256`.
-4. Never `pip install` at runtime. Updates are a manual re-vendor + record bump.
-
-| name | version | source | git sha | sha256 |
-|------|---------|--------|---------|--------|
-| _(none)_ | | | | |