Recover interrupted Docker policy changes and validate fleet bootstrap pins

.agents/skills/update-compositions/SKILL.md

@@ -13,7 +13,7 @@ This skill guides the safe, structured update of container images, compose defin
 2. **Hold critical services**: Always ask the user which services or compositions must NOT be touched before discovering or updating images.
 3. **Classify update scope**: Separate updates into **Minor/Patch** (non-breaking), **Major** (breaking / schema migration risks), and **Unstable SemVer / CalVer** (requiring changelog scrutiny).
 4. **Gate major and breaking updates**: Never apply a Major version bump, SemVer `0.x` minor change, or CalVer jump without migration research and explicit user approval.
-5. **Preserve APEX engine context**: Always orchestrate compose actions through `./commons/apex compose` (or with derived `APEX_*` variables and layered env files) rather than bare `docker compose`.
+5. **Preserve APEX engine context**: Always orchestrate compose actions through `./commons/apex composition/run` (or with derived `APEX_*` variables and layered env files) rather than bare `docker compose`.
 6. **Safeguard in-tree patches**: Do not bump images for containers with host bind mounts that patch application source code without explicit review.
 7. **Verify fleet observability post-rollout**: An update is not complete until containers reach `Up` and `healthy`, logs are clean, Prometheus scrape targets report `UP`, blackbox probes pass, and remote write streams are established.
 
@@ -145,7 +145,7 @@ For each node:
 1. **Pre-upgrade backup**:
    - Trigger a backup of stateful volumes:
      ```bash
-     ssh <node> "apex backup/run"
+     ssh <node> "apex tier/backup"
      ```
    - For database services being updated, execute an in-container logical export to host storage.
 2. **Update image tags in compose definitions**:
@@ -153,12 +153,12 @@ For each node:
 3. **Pull and recreate services via APEX CLI**:
    - Always run through the node's APEX CLI to preserve derived `APEX_*` variables and layered env files:
      ```bash
-     ssh <node> "cd /srv/<node_fqdn> && ./commons/apex compose pull"
-     ssh <node> "cd /srv/<node_fqdn> && ./commons/apex compose up -d"
+     ssh <node> "cd /srv/<node_fqdn> && ./commons/apex composition/run pull"
+     ssh <node> "cd /srv/<node_fqdn> && ./commons/apex composition/run up -d"
      ```
    - *Note on `--force-recreate`*: When updating containers using floating tags, recovering from host reboots, or handling services with `tmpfs` state (e.g. mail spools), pass `--force-recreate` to ensure the container entrypoint runs a clean initialization:
      ```bash
-     ssh <node> "cd /srv/<node_fqdn> && ./commons/apex compose up -d --force-recreate"
+     ssh <node> "cd /srv/<node_fqdn> && ./commons/apex composition/run up -d --force-recreate"
      ```
 4. **Completion check**: Compose commands finish with exit code 0.
 
@@ -195,7 +195,7 @@ When the update scope includes host operating system packages:
      2. Reboot node: `ssh <node> "sudo reboot"`
      3. Wait for SSH availability: poll connectivity with timeout.
      4. After boot, verify systemd state (`systemctl is-system-running`) and check that Docker containers auto-started.
-     5. If services with `tmpfs` mounts fail startup checks (e.g. Postfix spool directories in mail containers), run `./commons/apex compose up -d --force-recreate`.
+     5. If services with `tmpfs` mounts fail startup checks (e.g. Postfix spool directories in mail containers), run `./commons/apex composition/run up -d --force-recreate`.
 6. **Completion check**: Host packages are up to date and node is online.
 
 ---
@@ -229,6 +229,6 @@ Immediately after recreation and host reboots on each node:
    - If a container enters a crash loop (`Restarting`, `Exit 1`) or breaks ingress:
      1. Capture failure logs.
      2. Revert the image tag in the compose file to the prior tag.
-     3. Re-run `./commons/apex compose up -d --force-recreate`.
+     3. Re-run `./commons/apex composition/run up -d --force-recreate`.
      4. Notify user immediately with logs and root cause.
 6. **Completion check**: All containers are confirmed `Up` and `healthy`, external ingress functions normally, and telemetry metrics/targets are green.

.github/workflows/ci-packer-main-manual-build-golden-image.yml

@@ -1,116 +0,0 @@
-name: ci/packer - main/manual-build-golden-image
-
-on:
-  workflow_dispatch:
-    inputs:
-      upload_s3:
-        description: "Upload built image to specified static S3 Object Storage"
-        type: boolean
-        default: true
-
-concurrency:
-  group: ci-golden-image
-  cancel-in-progress: false
-
-permissions:
-  contents: read
-
-jobs:
-  validate-scripts:
-    name: "Validate: Provisioner Scripts"
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@v4
-
-      - name: Validate Shell Script Syntax
-        run: |
-          for script in ci/packer/scripts/*.sh engine/provisioning/seed.sh; do
-            bash -n "$script"
-          done
-          bash tests/run.sh
-
-      - name: Setup Packer
-        uses: hashicorp/setup-packer@main
-        with:
-          version: "latest"
-
-      - name: Validate Packer Template
-        run: |
-          packer init ci/packer/apex-debian-13.pkr.hcl
-          packer validate ci/packer/apex-debian-13.pkr.hcl
-
-  build-image:
-    name: "Build: Golden Image"
-    needs: validate-scripts
-    runs-on: ubuntu-24.04
-    outputs:
-      image_name: ${{ steps.vars.outputs.image_name }}
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@v4
-
-      - name: Compute Image Metadata
-        id: vars
-        run: |
-          SHORT_SHA="$(echo ${{ github.sha }} | cut -c1-7)"
-          IMAGE_NAME="com-ermnvldmr-apex-debian-13-amd64-${SHORT_SHA}.qcow2"
-          echo "image_name=${IMAGE_NAME}" >> "$GITHUB_OUTPUT"
-
-      - name: Install QEMU & Build Tools
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y qemu-system-x86 qemu-utils xorriso
-          sudo chmod 666 /dev/kvm 2>/dev/null || true
-
-      - name: Setup Packer
-        uses: hashicorp/setup-packer@main
-        with:
-          version: "latest"
-
-      - name: Build Debian 13 Image with Packer
-        run: |
-          packer init ci/packer/apex-debian-13.pkr.hcl
-          packer build -var "vm_name=${{ steps.vars.outputs.image_name }}" -var "provision_revision=${{ github.sha }}" ci/packer/apex-debian-13.pkr.hcl
-
-      - name: Upload Image Artifact
-        uses: actions/upload-artifact@v4
-        with:
-          name: golden-image
-          path: output-qemu/${{ steps.vars.outputs.image_name }}
-          retention-days: 2
-
-  upload-s3:
-    name: "Upload: S3 Storage"
-    needs: build-image
-    if: inputs.upload_s3 == true
-    runs-on: ubuntu-latest
-    steps:
-      - name: Download Image Artifact
-        uses: actions/download-artifact@v4
-        with:
-          name: golden-image
-          path: output-qemu
-
-      - name: Upload to Static S3 (/dev/images/)
-        env:
-          AWS_ACCESS_KEY_ID: ${{ secrets.STATIC_AWS_ACCESS_KEY_ID || secrets.COM_ERMNVLDMR_STATIC_AWS_ACCESS_KEY_ID }}
-          AWS_SECRET_ACCESS_KEY: ${{ secrets.STATIC_AWS_SECRET_ACCESS_KEY || secrets.COM_ERMNVLDMR_STATIC_AWS_SECRET_ACCESS_KEY }}
-          AWS_DEFAULT_REGION: ${{ secrets.STATIC_AWS_REGION || secrets.COM_ERMNVLDMR_STATIC_S3_REGION || 'us-east-1' }}
-          S3_ENDPOINT: ${{ secrets.STATIC_S3_ENDPOINT_URL || secrets.COM_ERMNVLDMR_STATIC_S3_ENDPOINT_URL }}
-          S3_BUCKET: ${{ secrets.STATIC_S3_BUCKET || secrets.COM_ERMNVLDMR_STATIC_S3_BUCKET }}
-          IMAGE_NAME: ${{ needs.build-image.outputs.image_name }}
-        run: |
-          if [ -n "${S3_BUCKET}" ] && [ -n "${AWS_ACCESS_KEY_ID}" ]; then
-            ENDPOINT_FLAG=""
-            if [ -n "${S3_ENDPOINT}" ]; then
-              ENDPOINT_FLAG="--endpoint-url=${S3_ENDPOINT}"
-            fi
-            echo "Cleaning up older builds in dev/images/com-ermnvldmr-apex-debian-13-amd64-*.qcow2 to maintain quota..."
-            aws ${ENDPOINT_FLAG} s3 rm "s3://${S3_BUCKET}/dev/images/" --recursive --exclude "*" --include "com-ermnvldmr-apex-debian-13-amd64-*.qcow2" || true
-            echo "Uploading ${IMAGE_NAME}..."
-            aws ${ENDPOINT_FLAG} s3 cp "output-qemu/${IMAGE_NAME}" "s3://${S3_BUCKET}/dev/images/${IMAGE_NAME}"
-            echo "Golden image uploaded successfully to s3://${S3_BUCKET}/dev/images/${IMAGE_NAME}"
-          else
-            echo "S3 credentials not configured; skipping upload. Image artifact preserved in GitHub Actions run."
-          fi

.gitignore

@@ -9,9 +9,12 @@
 __pycache__/
 *.pyc
 .pytest_cache/
+.venv/
 
 # Runtime / secrets (never in commons)
 *.log
 *.env
 .env
 *.bak
+._*
+.DS_Store

Makefile

@@ -1,13 +1,14 @@
 .PHONY: test lint fmt check
 
 test:
-	python3 -m unittest discover -s tests
+	bash tests/run.sh
 
 lint:
 	@if command -v ruff >/dev/null 2>&1; then \
-		ruff check .; \
+		ruff check . scripts/bootstrap-node; \
 	else \
-		python3 -m py_compile engine/**/*.py actions/**/*.py tests/*.py; \
+		python3 -m compileall -q actions tests scripts && \
+		python3 -m py_compile scripts/bootstrap-node && \
 		echo "ruff not installed in PATH; verified python bytecode compilation."; \
 	fi
 

README.md

@@ -1,33 +1,57 @@
 <p align="center">
-  <img alt="apex.ermnvldmr.com logo" src="./assets/logo.png" height="128" />
+  <img alt="apex.ermnvldmr.com logo" src="./etc/assets/logo.png" height="128" />
 </p>
 
 # [apex.ermnvldmr.com](https://apex.ermnvldmr.com)
 
-The boring, standard-library fleet operations framework powering self-hosted infrastructure across independent nodes.
+Local infrastructure operations for independently owned nodes, with a Python standard-library core.
 
 ## Quick Start
 
 - **Live Documentation**: <https://docs.ermnvldmr.com/en/apex/> (also available in [Russian](https://docs.ermnvldmr.com/ru/apex/))
 - **Getting Started Guide**: <https://docs.ermnvldmr.com/en/apex/getting-started/>
-- **Report an Issue**: <https://github.com/deytenit/apex.ermnvldmr.com/issues>
+- **Report an Issue**: <https://git.ermnvldmr.com/deytenit/apex.ermnvldmr.com/issues>
 - **License**: [BSD-2-Clause](#copyright--license)
 
+From an APEX checkout on your workstation, connect to the installed Debian or Proxmox VE host as root or as a user with passwordless sudo. The workstation needs Python 3.10 or newer and OpenSSH. The script installs Python 3 through APT if the host lacks it. Verify its SSH host key through a trusted channel and record it in your workstation’s SSH known hosts before connecting.
+
 ```sh
-# Clone your node's repository (which pins apex as commons/)
-git clone --recurse-submodules git@github.com:you/selfhost.example.com.git
-cd selfhost.example.com && ./init.sh && source ~/.bashrc
+scripts/bootstrap-node \
+  --target root@node.example.com \
+  --repo ssh://git@git.example.com:22222/infrastructure/selfhost.example.com.git \
+  --operator-key ~/.ssh/operator.pub \
+  --deploy-key ~/.ssh/selfhost-deploy \
+  --known-hosts ~/.ssh/git-known-hosts
+```
 
-# Configure node identity
-cat > node.env <<'EOF'
-APEX_NODE_FQDN=selfhost.example.com
-APEX_NODE_HOST=selfhost.example.com
-APEX_SUBNET=198.18.16.0/24
-EOF
+The operator public key grants SSH access to `adam`, who receives passwordless sudo. Use a separate, passphrase-free SSH deploy key for this node. Register its public half as a deploy key on the Git server before running the script. The script installs the private half on the node. The key must have read access to the node repository and any private SSH submodules it needs on that Git server. Grant write access to the node repository only if you use `apex repository/sync` to push changes. Supply verified Git server host keys in OpenSSH known_hosts format, including the port for nonstandard listeners (for example, `[git.example.com]:22222`).
 
-# Configure host baseline and bring up compositions
-apex configure        # firewall, crowdsec, cron, systemd, routing
-apex compose up       # bring node services online
+The script clones the repository under `/srv` as `adam`. Use `--node-name NAME` to choose the checkout directory under `/srv`. Before changing the SSH policy, log in with the matching operator private key on the original SSH port. Use the same host or SSH alias as the bootstrap target.
+
+```sh
+ssh -o StrictHostKeyChecking=yes -o IdentitiesOnly=yes -i ~/.ssh/operator adam@node.example.com
+```
+
+In that session, enter the checkout. If the node repository includes `init.sh`, review and run it to install the shell shortcuts. Otherwise call `./commons/apex` directly.
+
+```sh
+cd /srv/selfhost.example.com
+if [ -f ./init.sh ]; then ./init.sh && source ~/.bashrc; fi
+./commons/apex --help
+```
+
+On existing checkouts that use the APEX Git hooks, run `git config --local core.hooksPath commons/scripts/githooks` from the node repository after updating the submodule.
+
+Configure the node in its checkout. For example, on a stock Debian Docker host:
+
+```sh
+# Stock Debian Docker host, after configuring the node repository
+apex config/packages
+apex config/base
+apex config/docker
+apex config/ufw-docker
+apex config/ufw       # apply the node firewall rules
+apex composition/run up       # bring node services online
 ```
 
 ## Essential Information
@@ -37,18 +61,16 @@ You should understand the following invariants before exploring the codebase:
 ### Core Principles
 
 - **No Control Plane**: No central orchestrator, agent daemon, or cluster controller. Commands act locally on the node you run them on.
-- **Zero Build Step & Stdlib-Only**: Pure Python 3 standard library and Git. No `pip` dependencies, Node, Go, or code-generation steps.
+- **Core Runtime**: Python 3.10 or newer and Git, with no `pip` dependencies or build step. Standalone host provisioning and workstation checks have separate requirements below.
 - **Node-Owned Repositories**: Each machine is an independent Git repository pinning `apex` as a `commons/` submodule. It declares its own identity, compositions, and local overrides.
 - **Overlay Architecture**: Node repositories can extend or override any shared action by placing an executable in `proprietaries/actions/`.
 
 ### Repository Structure
 
-- **[`actions/`](./actions)**: Built-in engine actions (`compose.py`, `configure/`, `sync/`, `tiers/`, `backup/`, `utils/`).
+- **[`actions/`](./actions)**: Built-in engine actions (`composition/`, `config/`, `repository/`, `tier/`, `util/`).
 - **[`compositions/`](./compositions)**: The shared core Compose stack (`traefik`, `socket-proxy`, `alloy`, `adguard`, `xray`, `restic`).
-- **[`engine/`](./engine)**: Framework engine — action discovery, overlay resolution, context injection, and logging.
-- **[`etc/`](./etc)**: Static service configurations, daemon templates, and helper scripts (Alloy configs, Happ generator).
-- **[`ci/`](./ci)**: Host image build recipes (Packer for Debian 13) and cloud-init bootstrap templates.
-- **[`assets/`](./assets)**: Repository branding assets (`logo.png`).
+- **[`scripts/`](./scripts)**: Workstation bootstrap, the Python engine, host provisioning helpers, and Git hooks.
+- **[`etc/`](./etc)**: Shared configuration and static assets, including Alloy templates and the project logo.
 
 ---
 
@@ -60,37 +82,85 @@ Running and maintaining services on an existing host.
 
 - **Requirements**: Debian-based host with Docker, Docker Compose v2, Git, and Python 3.
 - **Workflow**:
-  - Update services: `apex compose pull && apex compose up`
-  - Run backups: `apex backup/run`
-  - Sync repository state: `apex sync/repository`
-- **Replication**: For multi-node high availability patterns and contracts, consult the [Composition Replication Documentation](https://docs.ermnvldmr.com/en/apex/concepts/replication/).
+  - Update services: `apex composition/run pull && apex composition/run up`
+  - Run backups: `apex tier/backup`
+  - Sync repository state: `apex repository/sync`
 
-### Infrastructure Administrator
+Target a composition explicitly when operating on one application:
 
-Bootstrapping clean Debian 13 hosts on bare metal or cloud providers without cloud-init support.
+```sh
+apex composition/run -p apex restart traefik
+apex composition/run -p app up -d
+apex composition/run --dry-run -p apex logs --tail 20 xray
+```
 
-- **Initialization Tool**: A standalone workstation bootstrap tool targets clean Debian 13 amd64 installations using the same baseline as Packer.
-- **Template**: Copy and fill out [`ci/cloud-init/user-data.template.yaml`](./ci/cloud-init/user-data.template.yaml) with target identity, operator keys, and node repository.
-- **Run Enrollment**:
-  ```sh
-  python3 ci/scripts/init-node.py root@server --user-data /private/path/location.user-data.yaml
-  ```
-- **Security Handover**: Safely switches SSH to key-only authentication on port 2222 (`adam@server`) protected by an automated rollback watchdog, activates UFW firewall rules, Docker iptables integration, and CrowdSec.
-- **Resuming**: Post-enrollment commands connect via `adam@server --port 2222`.
+The existing `apex compose` spelling is an alias for `apex composition/run`.
+Wrapper options precede the action; remaining arguments go to Docker Compose.
+The positional forms `apex composition/run restart apex traefik` and `apex composition/run up -d apex`
+remain supported. Use `-p` when service names or option values could be mistaken
+for composition names. Without a target, APEX visits core first, then other
+compositions alphabetically; `down` reverses that order.
+
+Replace `app` with a composition defined in your repository. Use targeted `up -d`
+after changing a container definition; `restart` retains the existing definition.
+
+For direct Docker Compose use in the core composition:
+
+```sh
+apex config/env --sync
+cd compositions/apex
+docker compose config --quiet
+docker compose restart traefik
+```
+
+Sync preserves existing `.env` credentials and UID/GID entries, including tier
+symlinks. It updates a managed block containing `apex.env` settings (including
+profiles) followed by synthesized node/path values, and writes with mode `0600`.
+Resync after changing `apex.env`, node identity, or tier paths. Shell environment
+values still take precedence in Docker Compose. Do not edit the generated block.
+
+`eval "$(apex config/env)"` exports synthesized APEX values to the current shell;
+`apex config/env --dotenv` prints dotenv syntax, and `apex config/env --write generated.env`
+writes those values to a separate file. These three modes do not merge the
+composition's secrets or `apex.env`; use `--sync` for the complete core bridge.
+Choose one output mode per invocation.
+
+### Preparing an existing host
+
+Run each setup action explicitly from the node checkout. Use `apex --help` to list the shared and node-owned actions. New actions follow `<singular-entity>/<action>`. Use `apex <entity>/<action> --help` for the arguments.
+
+On an installed Proxmox VE host, `config/packages` installs the administrator tools and `config/base` applies APT and journald policy. The PVE host keeps its own storage, networking, firewall, and management listeners. Run workload configuration inside the guest that owns the service.
+
+Before moving SSH to port 2222, allow inbound TCP/2222 from the operator’s address through the provider firewall and the node’s firewall (UFW on stock Debian or the PVE firewall). Keep the original administrator session open throughout the transition.
+
+1. Run `./commons/apex config/hardening begin` from the node checkout in the original SSH session.
+2. On your workstation, run `ssh -o ControlPath=none -o BatchMode=yes -p 2222 adam@node.example.com` using the operator key. In the new session, run `./commons/apex config/hardening finalize` from the checkout.
+3. Run the same SSH command again from your workstation. From the checkout in this second fresh session, run `./commons/apex config/hardening commit` to cancel automatic rollback.
+
+On stock Debian the final policy allows only `adam` on port 2222 using a public key. PVE retains ports 22 and 2222, requires public keys globally, and keeps root key access with `PermitRootLogin prohibit-password` and no `AllowUsers` restriction.
+
+If a login fails, run `./commons/apex config/hardening rollback` from the checkout in the original session or let the timer restore the original SSH configuration five minutes after `begin`. Use `./commons/apex config/hardening status` to inspect the transition. [Action reference](https://docs.ermnvldmr.com/en/apex/reference/actions/#confighardening)
 
 ### Developer & Contributor
 
 Adding shared actions or improving the engine.
 
 - **Conventions**: Maintain strict Python standard library compliance. No external runtime packages.
-- **Testing**: Run test suite via `python3 -m unittest discover -s tests`.
-- **Linting**: Keep code formatted and clean according to PEP 8 standards.
+- **Public documentation**: Describe supported interfaces using synthetic examples. Keep deployment inventories, internal plans, operational runbooks, and host-specific configuration in their owning private repositories.
+- **Setup**:
+  ```sh
+  python3 -m venv .venv
+  . .venv/bin/activate
+  python3 -m pip install ruff==0.17.2
+  ```
+- **Verification**: `make check` runs lint and the core and host provisioning suites. `make test` (or `bash tests/run.sh`) runs both suites. Use `PYTHONPATH=scripts` when running individual test modules. Environment integration tests use `docker compose` or standalone `docker-compose` with `config`; no daemon is needed.
+- **Linting**: Ruff checks correctness and security rules. Existing import-order, line-length, compact-statement, and annotation-style debt is explicitly deferred in `pyproject.toml` to avoid unrelated formatting changes. Without Ruff, `make lint` recursively checks Python compilation and fails on syntax errors; that fallback is not equivalent to Ruff. `make fmt` formats files and is a separate, deliberate operation.
 
 ---
 
 ## Communication & Support
 
-- **Issues**: <https://github.com/deytenit/apex.ermnvldmr.com/issues>
+- **Issues**: <https://git.ermnvldmr.com/deytenit/apex.ermnvldmr.com/issues>
 - **Email**: <personal@ermnvldmr.com>
 
 ## Copyright & License

actions/composition/run.py

@@ -1,41 +1,40 @@
-# actions/compose.py
+# actions/composition/run.py
 """Orchestrate docker compose across the node.
 
 Supports targeting all compositions in sequence (default: apex core first, services next;
-reverse on down) or targeting a specific composition (e.g. `apex compose restart apex`
-or `apex compose up -d sso`). Exports derived APEX_* vars and layers --env-file (.env secrets + apex.env).
+reverse on down) or targeting one with `apex composition/run -p apex restart traefik`.
+Wrapper options precede the action. Exports derived APEX_* vars and layers env files.
 """
 import os
-from engine.descriptor import Arg, Flag, Meta, Opt, Rest
+import shlex
+
+from engine.descriptor import Flag, Meta, Opt, Rest
 
 METADATA = Meta(
     summary="Orchestrate docker compose across the node or on a specific composition.",
     args=[
         Opt("-p", "Target composition name or 'all'.", default=None),
         Flag("--dry-run", "Print commands only."),
-        Arg("action", "up | down | restart | ps | logs | ..."),
-        Rest("extra", "Extra args (target composition name, service names, or docker compose flags)."),
+        Rest("command", "ACTION [COMPOSE_ARGS...]; wrapper options must precede ACTION."),
     ],
 )
 
 
-def _clean_xray_sockets(ctx):
-    """Unlink stale host-side Xray sockets in /dev/shm to prevent REALITY panic crash loops."""
-    shm = os.path.join(ctx.paths.stier3, "apex", "xray", "shm")
-    if os.path.isdir(shm):
-        for f in os.listdir(shm):
-            if f.endswith(".sock"):
-                try:
-                    os.unlink(os.path.join(shm, f))
-                    ctx.log.info(f"Cleaned stale Xray socket: {f}")
-                except OSError as e:
-                    ctx.log.warn(f"Failed to unlink {f}: {e}")
-
-
 def run(ctx, args):
-    log, comp, action = ctx.log, ctx.paths.compositions, args.action
-    extra = [a for a in (args.extra or []) if a != "--dry-run"]
-    dry_run = args.dry_run or len(extra) != len(args.extra or [])
+    log, comp = ctx.log, ctx.paths.compositions
+    command = list(args.command)
+    if command and command[0] == "--":
+        command.pop(0)
+    if not command:
+        log.error("ACTION is required (up, down, restart, logs, ...).")
+        raise SystemExit(64)
+    action, extra = command[0], command[1:]
+    dry_run = args.dry_run
+    before_separator = extra[:extra.index("--")] if "--" in extra else extra
+    if any(arg in ("-p", "--dry-run") or arg.startswith("-p=")
+           or arg.startswith("--dry-run=") for arg in before_separator):
+        log.error("Wrapper options -p and --dry-run must precede ACTION; use -- for passthrough.")
+        raise SystemExit(64)
     env = dict(os.environ)
     env.update({k: v for k, v in ctx.vars().items() if k.startswith("APEX_")})
 
@@ -49,31 +48,25 @@ def run(ctx, args):
         ]
 
     target = getattr(args, "p", None)
-    if not target and extra:
-        first = extra[0]
-        if first == "all":
-            target = "all"
-            extra = extra[1:]
-        elif first in available:
-            target = first
-            extra = extra[1:]
-        elif not first.startswith("-"):
-            target = first
-            extra = extra[1:]
-        elif extra[-1] in available and len(extra) > 1:
-            target = extra[-1]
-            extra = extra[:-1]
+    if target is None and extra:
+        if extra[0] in ["all", *available]:
+            target, extra = extra[0], extra[1:]
+        elif action == "up" and len(extra) >= 2 and extra[0] == "-d":
+            # Preserve the documented `up -d apex` spelling without scanning flag values.
+            if extra[1] in ["all", *available]:
+                if len(extra) != 2:
+                    log.error("Use -p COMPOSITION before up when adding service arguments.")
+                    raise SystemExit(64)
+                target, extra = extra[1], ["-d"]
 
-    if not target:
+    if target is None:
         target = "all"
 
     if target != "all" and target not in available:
-        log.error(f"Unknown composition '{target}'. Available: {', '.join(sorted(available)) if available else 'none'}")
+        names = ", ".join(sorted(available)) if available else "none"
+        log.error(f"Unknown composition '{target}'. Available: {names}")
         raise SystemExit(1)
 
-    if action in ("up", "restart", "start") and target in ("all", "apex") and not dry_run:
-        _clean_xray_sockets(ctx)
-
     def sequence():
         if target != "all":
             return [os.path.join(comp, target)]
@@ -89,13 +82,13 @@ def run(ctx, args):
             if os.path.isfile(os.path.join(dirpath, ef)):
                 base += ["--env-file", ef]
         if action == "up":
-            up_flags = [] if "-d" in extra else ["-d"]
+            up_flags = [] if any(flag in extra for flag in ("-d", "--detach")) else ["-d"]
             cmd = base + ["up"] + up_flags + extra
         else:
             cmd = base + [action] + extra
 
         if dry_run:
-            log.info(f"[DRY-RUN] (cd {dirpath} && {' '.join(cmd)})")
+            log.info(f"[DRY-RUN] (cd {shlex.quote(dirpath)} && {shlex.join(cmd)})")
         else:
             log.info(f"{action} {name}...")
             ctx.sys.run(cmd, cwd=dirpath, env=env, check=True)

actions/config/alloy.py

@@ -0,0 +1,77 @@
+"""Deploy the opt-in native Alloy service using shared host/guest configs."""
+import json
+from pathlib import Path
+import re
+import tempfile
+
+from engine.descriptor import Flag, Meta
+
+METADATA = Meta(summary="Configure native Alloy from configs/alloy/config.json.",
+                args=[Flag("--dry-run", "Validate the deployment plan without changes.")])
+
+
+def deployment(ctx):
+    settings = Path(ctx.paths.configs) / "alloy/config.json"
+    if not settings.exists():
+        return None
+    values = json.loads(settings.read_text())
+    role = values.get("role")
+    if role not in ("host", "guest"):
+        raise ValueError("Alloy role must be host or guest")
+    env_file = values.get("environment_file", "")
+    if not re.fullmatch(r"/[A-Za-z0-9_./-]+", env_file):
+        raise ValueError("Alloy environment_file must be an absolute path without whitespace")
+    source = Path(ctx.paths.commons) / "etc/alloy" / (
+        "native-host.alloy" if role == "host" else "native.alloy")
+    config = source.read_text()
+    extra = Path(ctx.paths.configs) / "alloy/extra.alloy"
+    if extra.is_file():
+        config += "\n" + extra.read_text()
+    unit = ("[Service]\nEnvironmentFile=" + env_file + "\n"
+            'ExecStart=\nExecStart=/usr/bin/alloy run --storage.path=/var/lib/alloy --server.http.listen-addr=127.0.0.1:12346 /etc/alloy/config.alloy\n'
+            + ("User=root\nGroup=root\n" if role == "guest" else "") +
+            "SupplementaryGroups=systemd-journal" + (" docker" if role == "guest" else "") + "\n")
+    return config, unit, env_file
+
+
+def run(ctx, args):
+    plan = deployment(ctx)
+    if plan is None:
+        ctx.log.info("Native Alloy is not configured for this node; skipping.")
+        return
+    config, unit, env_file = plan
+    if args.dry_run:
+        ctx.log.info("Native Alloy deployment plan validated; no changes made.")
+        return
+    ctx.sys.sudo(["test", "-f", env_file])
+    # Validate with the actual service environment without printing credentials.
+    validator = '''import os, stat, subprocess, sys
+from pathlib import Path
+# Environment files deliberately use plain KEY=value, with no shell evaluation.
+secret = Path(sys.argv[1])
+info = secret.stat()
+if info.st_uid != 0 or stat.S_IMODE(info.st_mode) & 0o077:
+    raise RuntimeError('Alloy environment file must be root-owned with mode 0600 or stricter')
+for line in secret.read_text().splitlines():
+    if not line.strip() or line.lstrip().startswith('#'): continue
+    key, value = line.split('=', 1)
+    os.environ[key] = value
+subprocess.run(['alloy', 'validate', sys.argv[2]], check=True)
+'''
+    with tempfile.TemporaryDirectory(prefix="apex-alloy-") as directory:
+        config_path = Path(directory) / "config.alloy"
+        unit_path = Path(directory) / "apex.conf"
+        config_path.write_text(config)
+        unit_path.write_text(unit)
+        ctx.sys.sudo(["python3", "-c", validator, env_file, str(config_path)])
+        ctx.sys.sudo(["install", "-d", "-m", "0755", "/etc/alloy",
+                      "/etc/systemd/system/alloy.service.d"])
+        ctx.sys.sudo(["install", "-m", "0644", str(config_path), "/etc/alloy/config.alloy"])
+        ctx.sys.sudo(["install", "-m", "0644", str(unit_path),
+                      "/etc/systemd/system/alloy.service.d/apex.conf"])
+    ctx.sys.sudo(["systemctl", "unmask", "alloy.service"])
+    ctx.sys.daemon_reload()
+    ctx.sys.sudo(["systemctl", "enable", "alloy.service"])
+    ctx.sys.sudo(["systemctl", "restart", "alloy.service"])
+    ctx.sys.sudo(["systemctl", "is-active", "--quiet", "alloy.service"])
+    ctx.log.success("Configured native Alloy.")

actions/config/base.py

@@ -0,0 +1,40 @@
+"""Apply the common APT archive and journald retention policy."""
+from pathlib import Path
+from engine.descriptor import Meta
+
+METADATA = Meta(summary="Configure APT archive cleanup and journald retention.")
+
+
+def _configure(ctx, path, legacy, content, restart=False):
+    for target in (path, legacy):
+        if target.is_symlink():
+            raise RuntimeError(f"Refusing redirected configuration: {target}")
+    previous = path.read_text() if path.exists() else None
+    backup = ctx.host.backup(str(legacy), sudo=True) if legacy.exists() else None
+    if previous == content and not backup:
+        return
+    if previous != content:
+        ctx.host.write_file(str(path), content, sudo=True)
+    if backup:
+        ctx.sys.sudo(["rm", "--", str(legacy)])
+    if restart:
+        try:
+            ctx.sys.sudo(["systemctl", "restart", "systemd-journald"])
+        except Exception:
+            if backup:
+                ctx.sys.sudo(["cp", "--preserve=all", "--", backup, str(legacy)])
+            if previous is None:
+                ctx.sys.sudo(["rm", "--", str(path)])
+            elif previous != content:
+                ctx.host.write_file(str(path), previous, backup=False, sudo=True)
+            raise
+
+
+def run(ctx, args):
+    if not Path("/etc/debian_version").is_file():
+        raise RuntimeError("Debian required.")
+    apt = Path("/etc/apt/apt.conf.d")
+    journal = Path("/etc/systemd/journald.conf.d")
+    _configure(ctx, apt / "99-apex", apt / "01clean", 'APT::Keep-Downloaded-Packages "0";\n')
+    _configure(ctx, journal / "99-apex.conf", journal / "maxuse.conf",
+               "[Journal]\nSystemMaxUse=200M\n", restart=True)

actions/config/cron.py

@@ -1,5 +1,5 @@
-# actions/configure/cron.py
-"""Port of configure/cron — install crontab from configs/cron/crontab."""
+# actions/config/cron.py
+"""Port of config/cron — install crontab from configs/cron/crontab."""
 import os
 from engine.descriptor import Meta, Flag
 

actions/config/crowdsec.py

@@ -1,5 +1,5 @@
-# actions/configure/crowdsec.py
-"""Port of configure/crowdsec — install, deploy /etc/crowdsec, collections, bouncer."""
+# actions/config/crowdsec.py
+"""Port of config/crowdsec — install, deploy /etc/crowdsec, collections, bouncer."""
 import os
 from engine.descriptor import Meta, Flag
 

actions/config/docker.py

@@ -0,0 +1,12 @@
+"""Install a local Docker Engine, Compose and Buildx on Debian 13."""
+from pathlib import Path
+
+from engine.descriptor import Meta
+
+METADATA = Meta(summary="Verify a working Docker toolchain or install it on stock Debian 13.")
+
+
+def run(ctx, args):
+    provisioning = Path(ctx.paths.commons) / "scripts"
+    ctx.sys.sudo(["python3", "-E", "-s", "-m", "provisioning.docker"],
+                 cwd=str(provisioning))

actions/config/env.py

@@ -0,0 +1,82 @@
+"""Export synthesized APEX values or bridge a composition's environment for Compose."""
+import os
+import shlex
+
+from engine.descriptor import Flag, Meta, Opt
+
+METADATA = Meta(
+    summary="Export or sync synthesized APEX_* variables for shell or docker compose.",
+    args=[
+        Flag("--export", "Format as shell export statements (default)."),
+        Flag("--dotenv", "Print quoted generated APEX_* dotenv values only."),
+        Opt("--write", "Write a standalone generated dotenv file (0600).", default=None),
+        Flag("--sync", "Preserve user .env text and refresh the core Compose managed block."),
+    ],
+)
+
+_START = "# BEGIN APEX MANAGED ENV"
+_END = "# END APEX MANAGED ENV"
+
+
+def _dotenv_value(value):
+    """Quote a literal using Compose dotenv escapes, including interpolation dollars."""
+    value = str(value)
+    for old, new in (("\\", "\\\\"), ('"', '\\"'), ("$", "$$"),
+                     ("\r", "\\r"), ("\n", "\\n"), ("\t", "\\t")):
+        value = value.replace(old, new)
+    return f'"{value}"'
+
+
+def _read_optional(path):
+    try:
+        with open(path, encoding="utf-8", newline="") as source:
+            return source.read()
+    except FileNotFoundError:
+        return ""
+
+
+def _sync_body(existing, public, generated):
+    if any(marker in public or marker in generated for marker in (_START, _END)):
+        raise ValueError("Managed environment marker collides with source content.")
+    lines = existing.splitlines(keepends=True)
+    markers = [(index, line.rstrip("\r\n")) for index, line in enumerate(lines)
+               if _START in line or _END in line]
+    if markers:
+        if (len(markers) != 2 or markers[0][1] != _START or markers[1][1] != _END):
+            raise ValueError("Malformed or duplicate managed environment markers; .env unchanged.")
+        start, end = markers[0][0], markers[1][0]
+        existing = "".join(lines[:start] + lines[end + 1:])
+    # Keep user bytes in order, but place managed values last to preserve precedence.
+    separator = "" if not existing or existing.endswith("\n") else "\n"
+    public_separator = "" if not public or public.endswith("\n") else "\n"
+    return (existing + separator + _START + "\n" + public + public_separator
+            + generated + _END + "\n")
+
+
+def run(ctx, args):
+    modes = (args.export, args.dotenv, args.write is not None, args.sync)
+    if sum(bool(mode) for mode in modes) > 1:
+        ctx.log.error("Choose only one of --export, --dotenv, --write, or --sync.")
+        raise SystemExit(64)
+    variables = {k: v for k, v in ctx.vars().items() if k.startswith("APEX_")}
+    generated = "".join(f"{key}={_dotenv_value(value)}\n"
+                        for key, value in sorted(variables.items()))
+    target = args.write
+    if args.sync:
+        core = os.path.join(ctx.paths.compositions, "apex")
+        target = os.path.join(core, ".env")
+        try:
+            generated = _sync_body(_read_optional(target),
+                                   _read_optional(os.path.join(core, "apex.env")), generated)
+        except ValueError as error:
+            ctx.log.error(str(error))
+            raise SystemExit(65) from error
+    if target is not None:
+        ctx.host.write_file(target, generated, mode="0600")
+        ctx.log.success(f"Wrote {len(variables)} synthesized variables to {target}.")
+    elif args.dotenv:
+        print(generated, end="")
+    else:
+        for key, value in sorted(variables.items()):
+            print(f"export {key}={shlex.quote(str(value))}")
+    return 0

actions/config/hardening.py

@@ -0,0 +1,189 @@
+"""Hand SSH access over to fresh, verified adam sessions on port 2222."""
+import ipaddress
+import json
+import os
+import platform
+import pwd
+import re
+import time
+import uuid
+from pathlib import Path
+from engine.descriptor import Arg, Meta, Opt
+
+METADATA = Meta(summary="Manage the protected SSH handover to port 2222.", args=[
+    Arg("operation", "begin (default), finalize, commit, rollback, or status", required=False),
+    Opt("--original-port", "Current SSH port when beginning from a console"),
+    Opt("--client-address", "SSH client IP when beginning from a console"),
+])
+STATE = "/var/lib/apex-init/access/manual-state.json"
+BUNDLES = Path("/usr/local/lib/apex-provisioner")
+
+
+def _ancestors():
+    ancestors = {}
+    pid = os.getppid()
+    ticks = os.sysconf("SC_CLK_TCK")
+    # proc starttime includes suspend time, while journald monotonic timestamps do not.
+    suspend_us = (time.clock_gettime_ns(time.CLOCK_BOOTTIME) - time.monotonic_ns()) // 1000
+    while pid > 1:
+        # comm can contain spaces and parentheses; the remaining fields start at state.
+        fields = Path(f"/proc/{pid}/stat").read_text().rsplit(")", 1)[1].split()
+        ancestors[pid] = int(fields[19]) * 1_000_000 // ticks - max(0, suspend_us)
+        pid = int(fields[1])
+    return ancestors
+
+
+def _socket(ctx, ancestors):
+    sockets = ctx.sys.sudo(["ss", "-Hntp", "state", "established"], capture=True).stdout
+    connections = set()
+    for line in sockets.splitlines():
+        fields = line.split(None, 4)
+        if len(fields) == 5 and any(int(pid) in ancestors for pid in
+                                    re.findall(r'"sshd(?:-session)?",pid=(\d+),', fields[4])):
+            connections.add((fields[2], fields[3]))
+    if not connections:
+        return None
+    if len(connections) != 1:
+        raise RuntimeError("Cannot identify a unique established SSH socket for this process.")
+    local, peer = connections.pop()
+    peer, peer_port = peer.rsplit(":", 1)
+    local, local_port = local.rsplit(":", 1)
+    def address(value):
+        ip = ipaddress.ip_address(value.strip("[]"))
+        return str(getattr(ip, "ipv4_mapped", None) or ip)
+    peer, local = address(peer), address(local)
+    return local, int(local_port), peer, int(peer_port)
+
+
+def _connection(ctx, *, after=None):
+    if pwd.getpwuid(os.getuid()).pw_name != "adam":
+        raise RuntimeError("Run this step inside a fresh SSH public-key session as adam.")
+    ancestors = _ancestors()
+    endpoint = _socket(ctx, ancestors)
+    if not endpoint:
+        raise RuntimeError("Cannot identify the SSH connection for this process.")
+    local, local_port, peer, peer_port = endpoint
+    if after is not None and local_port != 2222:
+        raise RuntimeError("Open a fresh adam SSH connection on port 2222.")
+    record = rf"^Accepted publickey for adam from {re.escape(peer)} port {int(peer_port)} ssh2:"
+    result = ctx.sys.sudo(["journalctl", "-b", "_UID=0", "_EXE=/usr/sbin/sshd",
+                           "_EXE=/usr/lib/openssh/sshd-session", "_EXE=/usr/lib/openssh/sshd-auth",
+                           "--no-pager", "-o", "json", "--grep", record], capture=True, check=False)
+    if not result.returncode:
+        for line in result.stdout.splitlines():
+            entry = json.loads(line)
+            pid = int(entry["_PID"])
+            if (pid in ancestors and int(entry["__MONOTONIC_TIMESTAMP"]) >= ancestors[pid]
+                    and (after is None or ancestors[pid] > after)
+                    and re.match(record, entry["MESSAGE"])):
+                ctx.sys.run(["sudo", "-n", "true"])
+                return f"user=adam,host={peer},addr={peer},laddr={local},lport={int(local_port)}"
+    raise RuntimeError("Cannot verify a fresh public-key login for this SSH process in the root sshd journal.")
+
+
+def _operator_access(ctx):
+    account = pwd.getpwnam("adam")
+    if account.pw_uid < 1000 or account.pw_dir != "/home/adam":
+        raise RuntimeError("Prepare the adam account from your workstation with scripts/bootstrap-node first.")
+    if pwd.getpwuid(os.getuid()).pw_name == "adam":
+        return _connection(ctx)
+    loginuid = int(Path("/proc/self/loginuid").read_text())
+    if loginuid == account.pw_uid or (loginuid == 4294967295 and os.getuid() != 0) or loginuid < 0:
+        raise RuntimeError("Use an independent administrator login or an SSH public-key session as adam.")
+    if os.getuid() not in (0, loginuid):
+        raise RuntimeError("Run from the independent administrator's own login.")
+    ctx.sys.sudo(["true"])
+    home = Path(account.pw_dir)
+    authorized = home / ".ssh" / "authorized_keys"
+    # Inspect as adam so private .ssh directories remain usable from another administrator.
+    command = ["runuser", "-u", "adam", "--"]
+    for path in (home, authorized.parent, authorized):
+        metadata = ctx.sys.sudo([*command, "env", "LC_ALL=C", "stat", "-c", "%u %a %F", "--", str(path)],
+                                capture=True).stdout.strip().split(" ", 2)
+        owner, mode, kind = metadata
+        expected = "regular file" if path == authorized else "directory"
+        if int(owner) not in (0, account.pw_uid) or int(mode, 8) & 0o022 or kind != expected:
+            raise RuntimeError(f"Unsafe operator SSH path: {path}")
+    keys = ctx.sys.sudo([*command, "cat", "--", str(authorized)], capture=True).stdout
+    for key in keys.splitlines():
+        if key.startswith(("ssh-", "ecdsa-", "sk-")) and ctx.sys.run(
+            ["ssh-keygen", "-lf", "-"], input=key + "\n", capture=True, check=False,
+        ).returncode == 0:
+            break
+    else:
+        raise RuntimeError("Install a valid unrestricted operator public key before applying SSH hardening.")
+    ctx.sys.sudo([*command, "sudo", "-n", "true"])
+    connection = "user=adam,host=localhost,addr=127.0.0.1"
+    paths = _effective(ctx.sys, connection).get("authorizedkeysfile", "").split()
+    if not {".ssh/authorized_keys", "%h/.ssh/authorized_keys", str(authorized)}.intersection(paths):
+        raise RuntimeError("sshd does not use the operator's verified authorized_keys file.")
+    return connection
+
+
+def _effective(system, connection):
+    result = system.sudo(["/usr/sbin/sshd", "-T", "-C", connection], capture=True)
+    return dict(line.split(" ", 1) for line in result.stdout.splitlines() if " " in line)
+
+
+def _state(system):
+    if system.sudo(["test", "-f", STATE], check=False).returncode:
+        return {}
+    return json.loads(system.sudo(["cat", STATE], capture=True).stdout)
+
+
+def _original_endpoint(ctx, args):
+    endpoint = _socket(ctx, _ancestors())
+    port = getattr(args, "original_port", None)
+    address = getattr(args, "client_address", None)
+    if endpoint:
+        if port or address:
+            raise ValueError("SSH sessions detect the peer address and port automatically.")
+        return endpoint[1], endpoint[2]
+    if not port or not address or not 1 <= int(port) <= 65535:
+        raise ValueError("From a console, supply --original-port and --client-address for the SSH client.")
+    return int(port), str(ipaddress.ip_address(address))
+
+
+def run(ctx, args):
+    operation = getattr(args, "operation", None) or "begin"
+    if operation not in ("begin", "finalize", "commit", "rollback", "status"):
+        raise ValueError("Choose begin, finalize, commit, rollback, or status.")
+    if operation == "begin":
+        _operator_access(ctx)
+    state = _state(ctx.sys)
+    phase = state.get("status", "absent")
+    if operation == "status" and phase == "absent":
+        print("absent")
+        return
+    if operation == "begin" and phase in ("transition", "final", "recovery_pending"):
+        raise RuntimeError(f"Handover is {phase}; inspect status, continue from a fresh SSH login, or roll back.")
+    if operation in ("finalize", "commit"):
+        required = "transition" if operation == "finalize" else "final"
+        if phase != required:
+            raise RuntimeError(f"{operation} requires phase {required}; current phase is {phase}.")
+        _connection(ctx, after=state["authenticated_after_us"])
+    profile = state.get("profile") if operation != "begin" else None
+    if not profile:
+        if Path("/usr/bin/pveversion").is_file():
+            profile = "proxmox"
+        elif platform.freedesktop_os_release().get("ID") == "debian":
+            profile = "debian"
+        else:
+            raise RuntimeError("SSH hardening supports Debian and Proxmox VE.")
+    options = ["--profile", profile]
+    if operation == "begin":
+        port, address = _original_endpoint(ctx, args)
+        options.extend(["--original-port", str(port), "--client-address", address, "--token", uuid.uuid4().hex])
+        installer = Path(__file__).resolve().parents[2] / "scripts/provisioning/manual_bundle.py"
+        bundle = ctx.sys.sudo(["/usr/bin/python3", "-I", "-B", str(installer)], capture=True).stdout.strip()
+    else:
+        if not state:
+            raise RuntimeError("No SSH handover has started.")
+        bundle = state["bundle"]
+        options.extend(["--token", state["token"]])
+    if not re.fullmatch(re.escape(str(BUNDLES)) + "/[0-9a-f]{64}", bundle):
+        raise RuntimeError("Invalid installed rollback bundle location.")
+    command = ["/usr/bin/python3", "-I", "-c",
+               "import sys; sys.path.insert(0, sys.argv.pop(1)); from provisioning.access import main; sys.exit(main())",
+               bundle, "tick" if operation == "status" else operation, *options]
+    ctx.sys.sudo(command)

actions/config/packages.py

@@ -0,0 +1,50 @@
+"""Install common Debian administrator tools without replacing host packages."""
+import re
+from pathlib import Path
+from engine.descriptor import Meta
+
+METADATA = Meta(summary="Install Debian administrator packages and enable rsyslog.")
+BASE_PACKAGES = ("curl", "wget", "git", "rsyslog", "ufw")
+
+
+ADMIN_PACKAGES = (
+    "tcpdump", "socat", "rsync", "bind9-dnsutils", "iproute2", "net-tools",
+    "traceroute", "ca-certificates", "jq", "sed", "gawk", "tar", "gzip",
+    "bzip2", "xz-utils", "unzip", "htop", "iotop", "strace", "lsof",
+    "sysstat", "parted", "fdisk", "pciutils", "usbutils", "smartmontools",
+    "tmux", "vim", "less", "tree", "bash-completion", "python3",
+    "python3-minimal", "python3-yaml", "sudo", "cron", "iptables", "ipset",
+    "debian-archive-keyring", "kmod",
+)
+
+def _install_missing(s, packages):
+    missing = []
+    for package in packages:
+        result = s.run(["dpkg-query", "-W", "-f=${Status}", package],
+                       check=False, capture=True)
+        if result.returncode or result.stdout.split()[1:] != ["ok", "installed"]:
+            missing.append(package)
+    if not missing:
+        return
+    apt = ["env", "LC_ALL=C", "DEBIAN_FRONTEND=noninteractive", "apt-get",
+           "-o", "DPkg::Lock::Timeout=60", "-o", "Acquire::Retries=2",
+           "-o", "Acquire::http::Timeout=30", "-o", "Acquire::https::Timeout=30"]
+    s.sudo([*apt, "update", "--error-on=any"])
+    command = [*apt, "install", "-y", "--no-install-recommends", "--no-upgrade",
+               "--no-remove", *missing]
+    plan = s.sudo([*command, "--simulate"], capture=True)
+    if any(line.startswith("Remv ") or re.match(r"^Inst \S+ \[", line)
+           for line in plan.stdout.splitlines()):
+        raise RuntimeError("Missing packages require upgrades/removals; reconcile packages first.")
+    s.sudo(command)
+
+
+def run(ctx, args):
+    if not Path("/etc/debian_version").is_file():
+        raise RuntimeError("Debian required.")
+    packages = (*BASE_PACKAGES, *ADMIN_PACKAGES)
+    if Path("/usr/bin/pveversion").exists():
+        packages = tuple(package for package in packages if package != "ufw")
+    _install_missing(ctx.sys, packages)
+    if not ctx.sys.start_and_enable("rsyslog"):
+        raise RuntimeError("rsyslog failed to start.")

actions/config/routing.py

@@ -1,5 +1,5 @@
-# actions/configure/routing.py
-"""Port of configure/routing — idempotent node routing (ip_forward, fwmark rule, table 100)."""
+# actions/config/routing.py
+"""Port of config/routing — idempotent node routing (ip_forward, fwmark rule, table 100)."""
 from engine.descriptor import Meta
 
 METADATA = Meta(summary="Idempotent node-level routing rules and routes.")

actions/config/systemd.py

@@ -1,5 +1,5 @@
-# actions/configure/systemd.py
-"""Port of configure/systemd — deploy + enable units from configs/systemd."""
+# actions/config/systemd.py
+"""Port of config/systemd — deploy + enable units from configs/systemd."""
 import os
 from engine.descriptor import Meta, Flag
 

actions/config/timezone.py

@@ -18,7 +18,7 @@ def run(ctx, args):
         ZoneInfo(zone)
     except (ValueError, ZoneInfoNotFoundError):
         ctx.log.error("configs/timezone must name an installed IANA timezone.")
-        raise SystemExit(1)
+        raise SystemExit(1) from None
     current = ctx.sys.run(["timedatectl", "show", "--property=Timezone", "--value"], capture=True).stdout.strip()
     if current == zone:
         ctx.log.info(f"Timezone already set to {zone}.")

actions/config/ufw-docker.py

@@ -0,0 +1,65 @@
+"""Install the pinned ufw-docker integration on Docker hosts."""
+import hashlib
+import os
+from pathlib import Path
+import stat
+import urllib.request
+from engine.descriptor import Meta
+
+METADATA = Meta(summary="Install the pinned Docker integration for UFW.")
+UFW_DOCKER_REVISION = "020a8699f95592561f254d8d4ad1bb40d401dfc7"
+UFW_DOCKER_SHA256 = "643e56b080567c567b4aa28650196849b2a2da5dd0473fd3e5216b0886035ab0"
+UFW_DOCKER_URL = (
+    "https://raw.githubusercontent.com/chaifeng/ufw-docker/"
+    + UFW_DOCKER_REVISION + "/ufw-docker"
+)
+UFW_DOCKER_PATH = Path("/usr/local/bin/ufw-docker")
+def _check_helper_path(path):
+    for parent in reversed(path.parents):
+        info = parent.lstat()
+        if (not stat.S_ISDIR(info.st_mode) or info.st_uid != 0 or
+                info.st_mode & 0o022):
+            raise RuntimeError(f"Unsafe ufw-docker ancestor: {parent}")
+    try:
+        info = path.lstat()
+    except FileNotFoundError:
+        return False
+    if (not stat.S_ISREG(info.st_mode) or info.st_uid != 0 or info.st_gid != 0 or
+            info.st_mode & 0o022 or info.st_nlink != 1):
+        raise RuntimeError(f"Unsafe ufw-docker target: {path}")
+    return True
+
+
+def _install_helper(ctx):
+    path = UFW_DOCKER_PATH
+    exists = _check_helper_path(path)
+    payload = path.read_bytes() if exists else None
+    if payload is None or hashlib.sha256(payload).hexdigest() != UFW_DOCKER_SHA256:
+        ctx.log.info(f"Downloading pinned ufw-docker helper to {path}...")
+        # The immutable HTTPS URL is authenticated by the digest below.
+        with urllib.request.urlopen(UFW_DOCKER_URL, timeout=30) as response:  # noqa: S310
+            payload = response.read(1024 * 1024 + 1)
+        if len(payload) > 1024 * 1024 or hashlib.sha256(payload).hexdigest() != UFW_DOCKER_SHA256:
+            raise RuntimeError("ufw-docker checksum mismatch; refusing installation.")
+        _check_helper_path(path)
+        ctx.host.write_file(str(path), payload.decode("utf-8"), mode="0755",
+                            backup=False, sudo=True)
+    elif stat.S_IMODE(path.stat().st_mode) != 0o755:
+        ctx.host.write_file(str(path), payload.decode("utf-8"), mode="0755",
+                            backup=False, sudo=True)
+    if (not _check_helper_path(path) or
+            hashlib.sha256(path.read_bytes()).hexdigest() != UFW_DOCKER_SHA256 or
+            stat.S_IMODE(path.stat().st_mode) != 0o755):
+        raise RuntimeError("ufw-docker verification failed; refusing execution.")
+    ctx.log.info("Ensuring ufw-docker routing rules are installed in UFW...")
+    ctx.sys.sudo([str(path), "install"])
+
+
+def run(ctx, args):
+    _check_helper_path(UFW_DOCKER_PATH)
+    if os.path.exists("/usr/bin/pveversion"):
+        raise RuntimeError("Use the Proxmox firewall on a PVE host.")
+    for binary in ("ufw", "docker"):
+        if not ctx.sys.ok(["which", binary]):
+            raise RuntimeError(f"Install {binary} before configuring ufw-docker.")
+    _install_helper(ctx)

actions/config/ufw.py

@@ -1,5 +1,5 @@
-# actions/configure/ufw.py
-"""Port of configure/ufw — apply docker + host ufw rules from configs/ufw."""
+# actions/config/ufw.py
+"""Port of config/ufw — apply docker + host ufw rules from configs/ufw."""
 import os
 from engine.descriptor import Meta, Flag
 

actions/configure.py

@@ -1,34 +0,0 @@
-# actions/configure.py
-"""Umbrella: run every configure/* action in a stable order (SP2 §3.1)."""
-import os
-from engine.descriptor import Meta, Flag, build_parser
-from engine import overlay
-
-METADATA = Meta(summary="Run every configure/* action in stable order.",
-                args=[Flag("--dry-run", "Pass --dry-run to sub-actions that support it.")])
-ORDER = ["configure/base", "configure/timezone", "configure/ufw", "configure/crowdsec",
-         "configure/cron", "configure/systemd", "configure/routing"]
-
-def run(ctx, args):
-    commons_actions = os.path.join(ctx.paths.commons, "actions")
-    prop = os.path.join(ctx.paths.proprietaries, "actions")
-    prop = prop if os.path.isdir(prop) else None
-    names = [i.name for i in overlay.discover(commons_actions, prop) if i.name.startswith("configure/")]
-    ordered = [n for n in ORDER if n in names] + sorted(n for n in names if n not in ORDER)
-    rc = 0
-    for name in ordered:
-        r = overlay.resolve(name, commons_actions, prop)
-        mod = overlay.load_module(r.path, "apex_umbrella_" + name.replace("/", "_"))
-        if getattr(mod, "DISABLED", None):
-            ctx.log.warn(f"Skipping disabled {name}"); continue
-        meta = getattr(mod, "METADATA")
-        has_dry = any(getattr(a, "name", None) == "--dry-run" for a in meta.args)
-        argv = ["--dry-run"] if (args.dry_run and has_dry) else []
-        sub_args = build_parser(meta, f"apex {name}").parse_args(argv)
-        ctx.log.info(f"=== {name} ===")
-        try:
-            mod.run(ctx, sub_args)
-        except SystemExit as e:
-            if e.code:
-                ctx.log.error(f"{name} exited {e.code}"); rc = rc or e.code
-    return rc

actions/configure/base.py

@@ -1,89 +0,0 @@
-# actions/configure/base.py
-"""Port of configure/base — base packages + ufw + ufw-docker helper."""
-import os
-import shutil
-import tempfile
-import urllib.request
-from engine.descriptor import Meta
-
-METADATA = Meta(summary="Configure base system packages (curl, wget, git, rsyslog, ufw, ufw-docker).")
-UFW_DOCKER_URL = "https://github.com/chaifeng/ufw-docker/raw/master/ufw-docker"
-
-
-def run(ctx, args):
-    log, s = ctx.log, ctx.sys
-    log.info("Starting base system initialization...")
-    if not os.path.isfile("/etc/debian_version"):
-        log.error("Requires a Debian-based system (Debian/Ubuntu).")
-        raise SystemExit(1)
-    needed = [p for p in ["curl", "wget", "git", "rsyslog"] if not shutil.which(p)]
-    if needed:
-        log.info("Updating package lists...")
-        s.sudo(["apt-get", "update", "-y"])
-        log.info(f"Installing missing essential packages: {', '.join(needed)}...")
-        s.sudo(["apt-get", "install", "-y", "-o", "DPkg::Lock::Timeout=60", *needed])
-    else:
-        log.info("Essential packages already installed.")
-    s.ensure_running("rsyslog")
-
-    # Configure APT to discard downloaded archives
-    apt_clean_path = "/etc/apt/apt.conf.d/99-apex"
-    apt_clean_content = 'APT::Keep-Downloaded-Packages "0";\n'
-    if os.path.exists("/etc/apt/apt.conf.d/01clean"):
-        s.sudo(["rm", "-f", "/etc/apt/apt.conf.d/01clean"])
-    current_apt = None
-    if os.path.exists(apt_clean_path):
-        try:
-            with open(apt_clean_path, encoding="utf-8") as f:
-                current_apt = f.read()
-        except OSError:
-            current_apt = None
-    if current_apt != apt_clean_content:
-        log.info("Configuring APT to discard downloaded archives...")
-        s.sudo(["tee", apt_clean_path], input=apt_clean_content)
-
-    # Limit systemd-journald max disk usage
-    journal_d = "/etc/systemd/journald.conf.d"
-    journal_conf = os.path.join(journal_d, "99-apex.conf")
-    journal_content = "[Journal]\nSystemMaxUse=200M\n"
-    if os.path.exists(os.path.join(journal_d, "maxuse.conf")):
-        s.sudo(["rm", "-f", os.path.join(journal_d, "maxuse.conf")])
-    current_journal = None
-    if os.path.exists(journal_conf):
-        try:
-            with open(journal_conf, encoding="utf-8") as f:
-                current_journal = f.read()
-        except OSError:
-            current_journal = None
-    if current_journal != journal_content:
-        log.info("Configuring systemd-journald SystemMaxUse=200M...")
-        s.sudo(["mkdir", "-p", journal_d])
-        s.sudo(["tee", journal_conf], input=journal_content)
-        s.sudo(["systemctl", "restart", "systemd-journald"])
-
-    if not shutil.which("ufw"):
-        log.info("Installing UFW...")
-        s.sudo(["apt-get", "install", "-y", "-o", "DPkg::Lock::Timeout=60", "ufw"])
-    else:
-        log.info("UFW already installed.")
-
-    target_bin = "/usr/local/bin/ufw-docker"
-    if not (os.path.isfile(target_bin) and os.access(target_bin, os.X_OK)):
-        log.info("Downloading ufw-docker helper to /usr/local/bin/ufw-docker...")
-        with tempfile.NamedTemporaryFile(delete=False) as tf:
-            tmp_name = tf.name
-        try:
-            urllib.request.urlretrieve(UFW_DOCKER_URL, tmp_name)
-            s.sudo(["cp", tmp_name, target_bin])
-            s.sudo(["chown", "root:root", target_bin])
-            s.sudo(["chmod", "0755", target_bin])
-            log.success(f"ufw-docker installed to {target_bin}.")
-        finally:
-            if os.path.exists(tmp_name):
-                os.unlink(tmp_name)
-    else:
-        log.info(f"ufw-docker already exists at {target_bin}.")
-
-    log.info("Ensuring ufw-docker routing rules are installed in UFW...")
-    s.sudo([target_bin, "install"])
-    log.success("Base system initialization completed.")

actions/env.py

@@ -1,54 +0,0 @@
-# actions/env.py
-"""Export or synchronize synthesized APEX_* environment variables.
-
-Outputs shell export statements (for `eval $(apex env)`) or writes a .env file
-for direct compatibility with bare `docker compose` commands.
-"""
-import os
-import shlex
-from engine.descriptor import Flag, Meta, Opt
-
-METADATA = Meta(
-    summary="Export or sync synthesized APEX_* variables for shell or docker compose.",
-    args=[
-        Flag("--export", "Format as shell export statements (default)."),
-        Flag("--dotenv", "Format as raw KEY=VAL lines without 'export'."),
-        Opt("--write", "Write variables to the specified file path.", default=None),
-        Flag("--sync", "Sync variables into compositions/apex/.env for bare docker compose."),
-    ],
-)
-
-
-def run(ctx, args):
-    log = ctx.log
-    vars_dict = {k: v for k, v in ctx.vars().items() if k.startswith("APEX_")}
-
-    target_file = None
-    if args.sync:
-        target_file = os.path.join(ctx.paths.compositions, "apex", ".env")
-    elif args.write:
-        target_file = args.write
-
-    if target_file:
-        content = [
-            "# Generated by apex env. Do not edit directly.",
-            "# Contains synthesized node and path scalars for docker compose interpolation.",
-        ]
-        for k in sorted(vars_dict.keys()):
-            content.append(f"{k}={vars_dict[k]}")
-        content.append("")
-        body = "\n".join(content)
-
-        os.makedirs(os.path.dirname(os.path.abspath(target_file)), exist_ok=True)
-        ctx.host.write_file(target_file, body, mode="0600")
-        log.success(f"Synced {len(vars_dict)} variables to {target_file}.")
-        return 0
-
-    if args.dotenv:
-        for k in sorted(vars_dict.keys()):
-            print(f"{k}={vars_dict[k]}")
-    else:
-        for k in sorted(vars_dict.keys()):
-            print(f"export {k}={shlex.quote(str(vars_dict[k]))}")
-
-    return 0

actions/repository/chown.py

@@ -0,0 +1,62 @@
+"""Give adam ownership of the active node's tracked checkout and Git metadata."""
+import inspect
+import json
+from pathlib import Path
+import re
+from engine.descriptor import Meta
+
+METADATA = Meta(summary="Set ownership of tracked files and Git metadata under /srv to adam.")
+
+
+def _repair(root, tracked, mounts, uid, gid):
+    import os
+    import stat
+
+    directories = {""}
+    for item in tracked:
+        directory = os.path.dirname(item)
+        while directory and directory not in directories:
+            directories.add(directory)
+            directory = os.path.dirname(directory)
+    metadata = os.path.join(root, ".git")
+
+    def fail(error):
+        raise error
+
+    for parent, dirs, files, fd in os.fwalk(root, onerror=fail, follow_symlinks=False):
+        os.fchown(fd, uid, gid)
+        for name in dirs.copy() + files:
+            path = os.path.join(parent, name)
+            relative = os.path.relpath(path, root)
+            if path in mounts or not (relative in tracked or relative in directories or
+                                      path == metadata or path.startswith(metadata + "/")):
+                if name in dirs:
+                    dirs.remove(name)
+                continue
+            info = os.stat(name, dir_fd=fd, follow_symlinks=False)
+            if stat.S_ISDIR(info.st_mode):
+                continue
+            if (stat.S_ISREG(info.st_mode) or stat.S_ISLNK(info.st_mode)) and info.st_nlink == 1:
+                os.chown(name, uid, gid, dir_fd=fd, follow_symlinks=False)
+
+
+def run(ctx, args):
+    root = Path(ctx.paths.repo_root)
+    if (root.parent != Path("/srv") or root != root.resolve(strict=True) or
+            not (root / "node.env").is_file() or (root / ".git").is_symlink() or
+            not (root / ".git").is_dir()):
+        raise RuntimeError("Expected the current node checkout directly under /srv.")
+    git = ["git", "-c", f"safe.directory={root}", "-c", f"safe.directory={root}/commons",
+               "-C", str(root)]
+    entries = ctx.sys.run([*git, "ls-files", "--cached", "--recurse-submodules", "-z"],
+                          capture=True).stdout
+    gitfiles = ctx.sys.run([*git, "submodule", "foreach", "--quiet", "--recursive",
+                            'printf "%s\\0" "$displaypath/.git"'], capture=True).stdout
+    mounts = [re.sub(r"\\([0-7]{3})", lambda match: chr(int(match[1], 8)), line.split()[4])
+              for line in Path("/proc/self/mountinfo").read_text().splitlines()]
+    script = (inspect.getsource(_repair) + "\nimport json, pwd, sys\n"
+              "root, entries, mounts = json.load(sys.stdin)\n"
+              "adam = pwd.getpwnam('adam')\n"
+              "_repair(root, set(entries), set(mounts), adam.pw_uid, adam.pw_gid)")
+    ctx.sys.sudo(["python3", "-I", "-c", script],
+                 input=json.dumps([str(root), (entries + gitfiles).split("\0"), mounts]))

actions/repository/sync.py

@@ -0,0 +1,59 @@
+"""Commit node configuration and synchronize main directly with its remote."""
+import os
+from datetime import datetime
+from engine.descriptor import Meta, Arg
+from engine.identity import read_env
+
+METADATA = Meta(summary="Commit node changes, rebase onto origin/main, and push main.",
+                args=[Arg("telegram_bot_url", "Optional notification URL", required=False)])
+TITLE = "Repository Sync"
+
+
+def run(ctx, args):
+    log, node = ctx.log, ctx.node.name
+    url = args.telegram_bot_url
+    repo, commons = ctx.paths.repo_root, ctx.paths.commons
+    ne = read_env(os.path.join(repo, "node.env"))
+    author = ne.get("APEX_GIT_AUTHOR_NAME", "apex [bot]")
+    email = ne.get("APEX_GIT_AUTHOR_EMAIL", "apex@localhost")
+    env = dict(os.environ, GIT_AUTHOR_NAME=author, GIT_AUTHOR_EMAIL=email,
+               GIT_COMMITTER_NAME=author, GIT_COMMITTER_EMAIL=email)
+
+    def git(*a, check=True, capture=False):
+        return ctx.sys.run(["git", "-C", repo, *a], check=check, capture=capture, env=env)
+
+    try:
+        branch = git("symbolic-ref", "--quiet", "--short", "HEAD", capture=True).stdout.strip()
+        if branch != "main":
+            raise RuntimeError("Repository sync requires main; switch branches explicitly first.")
+        cp = ctx.sys.run(["git", "-C", commons, "status", "--porcelain"], capture=True)
+        if (cp.stdout or "").strip():
+            raise RuntimeError(f"Commons submodule at {commons} has uncommitted changes.")
+        if not os.path.isdir(ctx.paths.compositions):
+            raise RuntimeError(f"Directory '{ctx.paths.compositions}' does not exist.")
+        expected = git("rev-parse", "HEAD:commons", capture=True).stdout.strip()
+        actual = ctx.sys.run(["git", "-C", commons, "rev-parse", "HEAD"], capture=True).stdout.strip()
+        if actual != expected:
+            raise RuntimeError("Commons checkout differs from the committed pin; update the submodule before retrying.")
+        git("add", "-A")
+        if git("diff", "--cached", "--quiet", check=False).returncode != 0:
+            ts = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+            git("commit", "-m", f"[{node}] Auto-commit: {ts}")
+        else:
+            log.info("No changes to commit.")
+        git("fetch", "origin", "main")
+        if git("rebase", "origin/main", check=False).returncode != 0:
+            git("rebase", "--abort", check=False)
+            raise RuntimeError("Rebase conflict; local commits retained for manual resolution.")
+        git("submodule", "update", "--init", "--recursive")
+        git("push", "origin", "HEAD:refs/heads/main")
+        log.success("Synchronized main.")
+        if url and not ctx.notify.success(TITLE, url, "Sync to main completed."):
+            raise SystemExit(1)
+    except SystemExit:
+        raise
+    except Exception as e:
+        log.error(f"{type(e).__name__}: {e}")
+        if url:
+            ctx.notify.error(TITLE, url, f"Sync failed: {e}")
+        raise SystemExit(1) from e

actions/sync/repository.py

@@ -1,64 +0,0 @@
-# actions/sync/repository.py
-"""Port of sync/repository — capture-up commit + rebase onto origin/main + force-push."""
-import os
-from datetime import datetime
-from engine.descriptor import Meta, Arg
-from engine.identity import read_env
-
-METADATA = Meta(summary="Capture-up: commit node changes, rebase onto origin/main, force-push sync/<node>.",
-                args=[Arg("telegram_bot_url", "Telegram Bot URL for notifications")])
-TITLE = "Annual Remote Sync"
-
-def run(ctx, args):
-    log, node, url = ctx.log, ctx.node.name, args.telegram_bot_url
-    repo, commons = ctx.paths.repo_root, ctx.paths.commons
-    ne = read_env(os.path.join(repo, "node.env"))       # per-node bot identity (generic defaults)
-    author = ne.get("APEX_GIT_AUTHOR_NAME", "apex [bot]")
-    email = ne.get("APEX_GIT_AUTHOR_EMAIL", "apex@localhost")
-    env = dict(os.environ,
-               GIT_AUTHOR_NAME=author, GIT_AUTHOR_EMAIL=email,
-               GIT_COMMITTER_NAME=author, GIT_COMMITTER_EMAIL=email)
-
-    def git(*a, check=True):
-        return ctx.sys.run(["git", "-C", repo, *a], check=check, env=env)
-
-    try:
-        cp = ctx.sys.run(["git", "-C", commons, "status", "--porcelain"], capture=True)
-        if (cp.stdout or "").strip():
-            log.error(f"Commons submodule at {commons} has uncommitted changes. Aborting.")
-            raise SystemExit(1)
-        if not os.path.isdir(ctx.paths.compositions):
-            # Without this, `git add -A` would stage a deleted node tree and an
-            # unattended cron run would commit + force-push the deletion.
-            log.error(f"Directory '{ctx.paths.compositions}' does not exist. Cannot sync.")
-            raise SystemExit(1)
-        branch = f"sync/{node}"
-        ts = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
-        if ctx.sys.ok(["git", "-C", repo, "rev-parse", "--verify", branch]):
-            git("checkout", branch)
-        else:
-            git("checkout", "-b", branch)
-        git("add", "-A")
-        git("submodule", "update", "--init", "--recursive")
-        git("add", "commons")
-        if git("diff", "--cached", "--quiet", check=False).returncode != 0:
-            git("commit", "-m", f"[{node}] Auto-commit: {ts}")
-        else:
-            log.info("No changes to commit.")
-        git("fetch", "origin", "main")
-        if git("rebase", "origin/main", check=False).returncode != 0:
-            log.error("Rebase conflict — aborting rebase; manual intervention required.")
-            git("rebase", "--abort", check=False)
-            ctx.notify.error(TITLE, url, "Rebase conflict. Manual intervention required.")
-            raise SystemExit(1)
-        git("submodule", "update", "--init", "--recursive")
-        git("push", "--force-with-lease", "origin", branch)
-        log.success(f"Pushed {branch}.")
-        if not ctx.notify.success(TITLE, url, f"Sync to '{branch}' completed."):
-            raise SystemExit(1)
-    except SystemExit:
-        raise
-    except Exception as e:
-        log.error(f"{type(e).__name__}: {e}")
-        ctx.notify.error(TITLE, url, f"Sync failed: {e}")
-        raise SystemExit(1)

actions/tier/backup.py

@@ -1,5 +1,5 @@
-# actions/backup/run.py
-"""Label-driven restic backup via resticontainer (github.com/deytenit/resticontainer).
+# actions/tier/backup.py
+"""Label-driven restic backup via resticontainer (git.ermnvldmr.com/deytenit/resticontainer).
 Runs the resticontainer core service one-shot: init-if-needed -> backup (discovers
 `restic.*`-labelled services, runs their pre-hooks/stops, snapshots the union of the
 resolved host paths in a single restic run) -> forget --prune -> Telegram. Every
@@ -7,12 +7,44 @@ non-`backup` subcommand passes straight through to restic, so init/snapshots/for
 behave exactly as before."""
 import json
 import os
+from pathlib import Path
+import time
 from engine.descriptor import Meta, Arg
 
 METADATA = Meta(summary="Run the resticontainer label-driven backup (init-if-needed, backup, forget --prune).",
                 args=[Arg("telegram_bot_url", "Telegram Bot URL for notifications")])
 TITLE = "Restic Host Backup"
 
+def write_backup_status(path, state, timestamp):
+    path = Path(path)
+    previous = json.loads(path.read_text()) if path.exists() else {}
+    status = {"state": state, "updated_at": timestamp,
+              "last_success": timestamp if state == "success" else previous.get("last_success", 0)}
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_name(path.name + f".{os.getpid()}.tmp")
+    temporary.write_text(json.dumps(status) + "\n")
+    temporary.chmod(0o644)
+    temporary.replace(path)
+
+def restore_containers(ctx, container_ids):
+    """Recover previously running backup targets, retrying namespace dependents."""
+    pending = list(container_ids)
+    if not pending:
+        return
+    for _ in range(len(pending) + 1):
+        cp = ctx.sys.run(["docker", "inspect", *pending], capture=True)
+        stopped = [c["Id"] for c in json.loads(cp.stdout) if not c["State"]["Running"]]
+        if not stopped:
+            return
+        progress = False
+        for container_id in stopped:
+            cp = ctx.sys.run(["docker", "start", container_id], check=False, capture=True)
+            progress = progress or cp.returncode == 0
+        if not progress:
+            raise RuntimeError("Backup left containers stopped: " + ", ".join(stopped))
+        pending = stopped
+    raise RuntimeError("Backup containers did not stay running: " + ", ".join(pending))
+
 def run(ctx, args):
     log, node, url = ctx.log, ctx.node.name, args.telegram_bot_url
     core = ctx.paths.core
@@ -42,6 +74,8 @@ def run(ctx, args):
         except (ValueError, TypeError):
             return None
 
+    status_path = Path("/var/lib/apex/backup-status.json")
+    write_backup_status(status_path, "running", time.time())
     try:
         log.info("Checking restic repository availability...")
         if restic("snapshots", check=False).returncode == 0:
@@ -50,8 +84,16 @@ def run(ctx, args):
             ctx.notify.error(TITLE, url, "restic init failed"); raise SystemExit(1)
         before = host_snapshot_ids()
         log.info("Backing up `restic.*`-labelled services (resticontainer discovers paths)...")
-        if restic("backup", "--host", host, "--tag", tag, "--compression", "max",
-                  check=False).returncode != 0:
+        targets = ctx.sys.run(["docker", "ps", "-q", "--no-trunc",
+                               "--filter", "label=restic.enable=true",
+                               "--filter", "label=restic.backup.stop=true"], capture=True).stdout.split()
+        try:
+            result = restic("backup", "--host", host, "--tag", tag, "--compression", "max",
+                            check=False)
+        finally:
+            # Resticontainer may start a network-namespace dependent before its owner.
+            restore_containers(ctx, targets)
+        if result.returncode != 0:
             ctx.notify.error(TITLE, url, "restic backup failed"); raise SystemExit(1)
         # Guard: the label-driven backup is a silent no-op if no service carries the
         # `restic.enable` label (resticontainer finds zero paths and skips restic).
@@ -70,9 +112,12 @@ def run(ctx, args):
         log.success("Restic backup completed.")
         if not ctx.notify.success(TITLE, url, f"Restic backup + retention succeeded for {node}"):
             raise SystemExit(1)   # bash exited 1 when the success telegram failed to send
+        write_backup_status(status_path, "success", time.time())
     except SystemExit:
+        write_backup_status(status_path, "failed", time.time())
         raise
     except Exception as e:
+        write_backup_status(status_path, "failed", time.time())
         log.error(f"{type(e).__name__}: {e}")
         ctx.notify.error(TITLE, url, f"backup failed: {e}")
-        raise SystemExit(1)
+        raise SystemExit(1) from e

actions/tier/chown.py

@@ -1,5 +1,5 @@
-# actions/tiers/chown.py
-"""Port of tiers/chown — ownership: {tier}/{proj} -> uid:gid; {tier}/shared/{proj} ->
+# actions/tier/chown.py
+"""Port of tier/chown — ownership: {tier}/{proj} -> uid:gid; {tier}/shared/{proj} ->
 uid:shared_gid (+sticky); base/node-infra -> root:root. Reads APEX_UID/GID from each .env."""
 import os
 from engine.descriptor import Meta, Arg
@@ -31,7 +31,7 @@ def run(ctx, args):
 
     cp = s.run(["getent", "group", SHARED_GROUP], check=False, capture=True)
     if cp.returncode != 0:
-        log.error(f"Shared group '{SHARED_GROUP}' not found. Run tiers/useradd first."); raise SystemExit(1)
+        log.error(f"Shared group '{SHARED_GROUP}' not found. Run tier/useradd first."); raise SystemExit(1)
 
     log.info("Ensuring base structures owned by root...")
     for tp in tiers:
@@ -62,7 +62,7 @@ def run(ctx, args):
         gid = _env_val(env_path, "APEX_GID")
         sgid = _env_val(env_path, "APEX_SHARED_GID")
         if not (uid and gid and sgid):
-            log.warn(f"Incomplete APEX_UID/GID/SHARED_GID for {proj}. Run tiers/useradd first. Skipping.")
+            log.warn(f"Incomplete APEX_UID/GID/SHARED_GID for {proj}. Run tier/useradd first. Skipping.")
             continue
         log.info(f"Applying permissions for {proj} (UID={uid} GID={gid} SHARED_GID={sgid})")
         for tp in tiers:

actions/tier/link.py

@@ -1,5 +1,5 @@
-# actions/tiers/link.py
-"""Port of tiers/link — @tier symlink structure for the node + each composition project."""
+# actions/tier/link.py
+"""Port of tier/link — @tier symlink structure for the node + each composition project."""
 import os
 from engine.descriptor import Meta, Arg
 

actions/tier/useradd.py

@@ -1,5 +1,5 @@
-# actions/tiers/useradd.py
-"""Port of tiers/useradd — one noroot system user per composition project; inject the
+# actions/tier/useradd.py
+"""Port of tier/useradd — one noroot system user per composition project; inject the
 APEX_UID/APEX_GID/APEX_SHARED_GID block into each project .env."""
 import os, re
 from engine.descriptor import Meta
@@ -36,7 +36,7 @@ def run(ctx, args):
             continue
         # opt-out: a project marked .apex-root-owned runs as root and keeps its tier
         # data root-owned (e.g. seafile-mc, whose entrypoint refuses non-root-owned
-        # data). Skip the noroot user + .env UID injection; tiers/chown forces root.
+        # data). Skip the noroot user + .env UID injection; tier/chown forces root.
         if os.path.isfile(os.path.join(full, ".apex-root-owned")):
             log.info(f"{proj}: .apex-root-owned — runs as root; skipping noroot user")
             continue

actions/util/check-updates.py

@@ -1,4 +1,4 @@
-# actions/sync/packages.py
+# actions/util/check-updates.py
 """Check pending APT and running-image updates without pulling images."""
 import json
 import os
@@ -220,7 +220,7 @@ def run(ctx, args):
     except Exception as e:   # bash ERR trap parity: unhandled failures page via Telegram
         log.error(f"{type(e).__name__}: {e}")
         ctx.notify.error(TITLE, url, f"Critical error: {e}")
-        raise SystemExit(1)
+        raise SystemExit(1) from e
 
 def _check(ctx, args, log, node, url, s):
     if not os.path.isfile("/etc/debian_version"):

actions/util/extract-traefik-certs.py

@@ -1,5 +1,5 @@
-# actions/utils/extract-traefik-certs.py
-"""Port of utils/extract-traefik-certs — dump traefik v3 acme.json to pem via docker."""
+# actions/util/extract-traefik-certs.py
+"""Port of util/extract-traefik-certs — dump traefik v3 acme.json to pem via docker."""
 import os
 from engine.descriptor import Meta, Arg
 
@@ -33,4 +33,4 @@ def run(ctx, args):
         ctx.notify.success(TITLE, url, f"Certificates extracted for {node}")
     except Exception as e:
         log.error(str(e)); ctx.notify.error(TITLE, url, f"cert extraction failed: {e}")
-        raise SystemExit(1)
+        raise SystemExit(1) from e

actions/util/lint-docker-compose.py

@@ -1,7 +1,7 @@
-# actions/utils/lint-docker-compose.py
-"""Port of utils/lint-docker-compose (dclint via docker) + stdlib anchor-IP validation:
+# actions/util/lint-docker-compose.py
+"""Run dclint through Docker and validate anchor IPs:
 every ipv4_address in compositions/*/docker-compose.yml must be unique and inside the
-node's /24 (SP3 §6.2). No new deps — ipaddress + re are stdlib."""
+node's /24."""
 import ipaddress, os, re, shutil
 from engine.descriptor import Meta, Flag
 
@@ -12,8 +12,6 @@ def run(ctx, args):
     log, s = ctx.log, ctx.sys
     repo = ctx.paths.repo_root
     rc_ip = _validate_anchor_ips(ctx)
-    if not shutil.which("docker"):
-        log.error("docker is not installed or not in PATH."); return rc_ip or 1
 
     files_string, targets = "", ["."]
     if args.hook:
@@ -25,13 +23,16 @@ def run(ctx, args):
             log.info("No staged docker-compose files to lint."); return rc_ip
         targets = [f"/workdir/{f}" for f in files_string.splitlines()]
 
+    if not shutil.which("docker"):
+        log.error("docker is not installed or not in PATH."); return rc_ip or 1
+
     log.info("Running dclint --fix (via docker)...")
     env_args = []
     for k in ("HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy", "NO_PROXY", "no_proxy"):
         if k in os.environ:
             env_args.extend(["-e", f"{k}={os.environ[k]}"])
     # "-e commons": never --fix inside the pinned submodule — a dirtied commons makes
-    # sync/repository abort the nightly capture-up.
+    # repository/sync abort the nightly capture-up.
     r = s.run(["docker", "run", "--rm", "--network", "host", *env_args, "-v", f"{repo}:/workdir", "-w", "/workdir", "node:lts-alpine",
                "npx", "--yes", "dclint", "-e", "@tier1", "-e", "@tier2", "-e", "@tier3", "-e", ".env",
                "-e", "commons", "--disable-rule", "no-build-and-image", "-r", "--fix", *targets], check=False)

apex

@@ -1,10 +1,10 @@
 #!/usr/bin/env python3
-"""apex launcher — stdlib only. Puts commons/ (+ vendor/ + proprietaries/) on sys.path."""
+"""apex launcher — stdlib only. Loads the shared engine and optional checkout libraries."""
 import os
 import sys
 
 _here = os.path.dirname(os.path.realpath(__file__))     # commons dir (or standalone repo root)
-sys.path.insert(0, _here)                               # make `engine` importable
+sys.path.insert(0, os.path.join(_here, "scripts"))
 _vendor = os.path.join(_here, "vendor")
 if os.path.isdir(_vendor):
     sys.path.insert(0, _vendor)

ci/cloud-init/user-data.template.yaml

@@ -1,33 +0,0 @@
-#cloud-config
-# Use this same file for cloud-init or: python3 ci/scripts/init-node.py root@HOST --user-data FILE
-# Replace the hostname, repository and public key. Keep filled files outside Git.
-# SSH repositories additionally require approved deferred id_ed25519 (0600)
-# and known_hosts (0644) writes under /home/adam/.ssh, owned by adam:adam.
-# Supply trusted Git host keys; do not infer trust from ssh-keyscan alone.
-preserve_hostname: false
-hostname: node1.example.com
-users:
-  - name: adam
-    sudo: ALL=(ALL) NOPASSWD:ALL
-    groups: [sudo, docker]
-    shell: /bin/bash
-    ssh_authorized_keys:
-      - "ssh-ed25519 REPLACE_WITH_OPERATOR_PUBLIC_KEY operator@workstation"
-write_files:
-  - path: /etc/apex/bootstrap.env
-    owner: root:root
-    permissions: '0600'
-    content: |
-      APEX_REPO_NAME="node1.example.com"
-      APEX_REPO_URL="https://github.com/you/node1.example.com.git"
-      APEX_TIER1_MODE="directory"
-      APEX_TIER1_DEVICE=""
-      APEX_TIER2_MODE="directory"
-      APEX_TIER2_DEVICE=""
-      APEX_TIER3_MODE="directory"
-      APEX_TIER3_DEVICE=""
-# For a verified empty disk, explicitly select MODE=create and DEVICE=/dev/disk/by-id/...
-# For an existing single-device pool, select MODE=import, DEVICE and POOL_GUID.
-# Occupied devices are never erased; missing devices never fall back to directories.
-runcmd:
-  - /usr/local/bin/apex-bootstrap

ci/packer/apex-debian-13.pkr.hcl

@@ -1,94 +0,0 @@
-packer {
-  required_plugins {
-    qemu = {
-      version = ">= 1.1.0"
-      source  = "github.com/hashicorp/qemu"
-    }
-  }
-}
-
-variable "debian_cloud_image_url" {
-  type    = string
-  default = "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-amd64-daily.qcow2"
-}
-
-variable "debian_cloud_image_checksum" {
-  type    = string
-  default = "none"
-}
-
-variable "vm_name" {
-  type    = string
-  default = "com-ermnvldmr-apex-debian-13-amd64.qcow2"
-}
-
-variable "accelerator" {
-  type    = string
-  default = "kvm"
-}
-
-variable "provision_revision" {
-  type    = string
-  default = "working-tree"
-}
-
-source "qemu" "apex_debian" {
-  iso_url          = var.debian_cloud_image_url
-  iso_checksum     = var.debian_cloud_image_checksum
-  disk_image       = true
-  disk_size        = "10G"
-  format           = "qcow2"
-  output_directory = "output-qemu"
-  shutdown_command = "sudo -n shutdown -P now"
-  accelerator      = var.accelerator
-  cd_files         = ["ci/packer/cidata/user-data", "ci/packer/cidata/meta-data"]
-  cd_label         = "cidata"
-  ssh_username     = "debian"
-  ssh_password     = "packer"
-  ssh_timeout      = "10m"
-  vm_name          = var.vm_name
-  net_device       = "virtio-net"
-  disk_interface   = "virtio"
-  headless         = true
-
-  qemuargs = [
-    ["-m", "2048M"],
-    ["-smp", "2"]
-  ]
-}
-
-build {
-  sources = ["source.qemu.apex_debian"]
-
-  provisioner "file" {
-    source      = "engine"
-    destination = "/tmp/"
-  }
-
-  provisioner "shell" {
-    execute_command = "sudo -n sh -c '{{ .Vars }} {{ .Path }}'"
-    environment_vars = ["APEX_PROVISION_REVISION=${var.provision_revision}"]
-    inline = [
-      "bash /tmp/engine/provisioning/seed.sh",
-      "cd /tmp && python3 -m engine.provisioning.image install"
-    ]
-  }
-
-  provisioner "shell" {
-    execute_command = "sudo -n sh -c '{{ .Vars }} {{ .Path }}'"
-    scripts = [
-      "ci/packer/scripts/01-base-system.sh",
-      "ci/packer/scripts/02-storage-docker.sh",
-      "ci/packer/scripts/03-bootstrap-helper.sh",
-      "ci/packer/scripts/04-cleanup.sh"
-    ]
-  }
-
-  post-processor "shell-local" {
-    inline = [
-      "echo 'Compressing and optimizing qcow2 image with 2M cluster size...'",
-      "qemu-img convert -f qcow2 -O qcow2 -o cluster_size=2M -c output-qemu/${var.vm_name} output-qemu/com-ermnvldmr-apex-debian-13-optimized.qcow2",
-      "mv output-qemu/com-ermnvldmr-apex-debian-13-optimized.qcow2 output-qemu/${var.vm_name}"
-    ]
-  }
-}

ci/packer/cidata/meta-data

@@ -1,2 +0,0 @@
-instance-id: apex-golden-image-build
-local-hostname: apex-builder

ci/packer/cidata/user-data

@@ -1,14 +0,0 @@
-#cloud-config
-ssh_pwauth: true
-disable_root: false
-chpasswd:
-  list: |
-    debian:packer
-    root:packer
-  expire: false
-users:
-  - default
-  - name: debian
-    sudo: ALL=(ALL) NOPASSWD:ALL
-    shell: /bin/bash
-    lock_passwd: false

ci/packer/scripts/01-base-system.sh

@@ -1,60 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-export DEBIAN_FRONTEND=noninteractive
-apt-get -o DPkg::Lock::Timeout=60 update
-apt-get -o DPkg::Lock::Timeout=60 install -y --no-upgrade --no-install-recommends qemu-guest-agent cloud-init
-
-echo "=== [3/6] Configuring VirtIO Kernel Modules ==="
-cat << 'EOF_VIRTIO' > /etc/modules-load.d/virtio.conf
-virtio
-virtio_net
-virtio_blk
-virtio_pci
-virtio_scsi
-virtiofs
-EOF_VIRTIO
-
-echo "=== [4/6] Configuring GRUB Serial Console & Predictable Interfaces ==="
-sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,115200n8 net.ifnames=0 biosdevname=0"/' /etc/default/grub
-sed -i 's/^#GRUB_TERMINAL=.*/GRUB_TERMINAL="console serial"/' /etc/default/grub
-if ! grep -q "GRUB_SERIAL_COMMAND" /etc/default/grub; then
-  echo 'GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"' >> /etc/default/grub
-fi
-update-grub
-
-mkdir -p /etc/systemd/system/serial-getty@ttyS0.service.d
-systemctl enable serial-getty@ttyS0.service || true
-systemctl enable qemu-guest-agent 2>/dev/null || true
-
-echo "=== [5/6] Configuring Cloud-Init Datasources & Service Enablement ==="
-mkdir -p /etc/cloud /etc/cloud/cloud.cfg.d
-cat << 'EOF_DS' > /etc/cloud/ds-identify.cfg
-policy: enabled
-EOF_DS
-
-cat << 'EOF_CLOUD_CFG' > /etc/cloud/cloud.cfg.d/99-cloud-init.cfg
-datasource_list: [ NoCloud, ConfigDrive, OpenStack, Ec2, None ]
-disable_vm_template: false
-EOF_CLOUD_CFG
-
-# Cloud-init services are static and enabled by generator via ds-identify
-systemctl enable cloud-init-local.service cloud-init.service cloud-config.service cloud-final.service 2>/dev/null || true
-
-echo "=== [6/6] Configuring Predictable Network DHCP ==="
-mkdir -p /etc/systemd/network
-cat << 'EOF_NET' > /etc/systemd/network/10-eth.network
-[Match]
-Name=eth* en*
-
-[Network]
-DHCP=yes
-EOF_NET
-systemctl enable systemd-networkd systemd-resolved || true
-
-mkdir -p /etc/network/interfaces.d
-cat << 'EOF_IF' > /etc/network/interfaces.d/eth0
-auto eth0
-iface eth0 inet dhcp
-EOF_IF
-
-echo "=== Base System Configuration Complete ==="

ci/packer/scripts/02-storage-docker.sh

@@ -1,3 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-/usr/local/bin/apex-provision-image baseline

ci/packer/scripts/03-bootstrap-helper.sh

@@ -1,3 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-/usr/local/bin/apex-provision-image install-runner

ci/packer/scripts/04-cleanup.sh

@@ -1,6 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-/usr/local/bin/apex-provision-image sanitize
-apt-get clean
-cloud-init clean --logs --seed
-sync

ci/scripts/init-node.py

@@ -1,64 +0,0 @@
-#!/usr/bin/env python3
-"""Initialize a plain Debian target server via SSH enrollment.
-
-Standalone tool for hosts on providers that do not support cloud-init.
-Kept separate from the core local-only apex engine.
-"""
-import argparse
-import os
-import sys
-
-_here = os.path.dirname(os.path.abspath(__file__))
-_repo_root = os.path.dirname(os.path.dirname(_here))
-_prov_dir = os.path.join(_here, "init-node")
-sys.path.insert(0, _prov_dir)
-
-from provisioning.transport import initialize
-
-
-def main(argv=None):
-    parser = argparse.ArgumentParser(
-        description="Initialize a plain Debian server through protected SSH enrollment.",
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
-    )
-    parser.add_argument(
-        "target",
-        help="Bootstrap USER@HOST (root or an existing administrator with passwordless sudo)",
-    )
-    parser.add_argument(
-        "--user-data",
-        dest="user_data",
-        help="Path to APEX cloud-config file (e.g., ci/cloud-init/user-data.template.yaml)",
-    )
-    parser.add_argument(
-        "--port",
-        dest="port",
-        help="Bootstrap SSH port; otherwise uses standard SSH configuration",
-    )
-    parser.add_argument(
-        "--identity-file",
-        dest="identity_file",
-        help="SSH private identity file; otherwise uses SSH configuration/agent",
-    )
-
-    args = parser.parse_args(argv)
-
-    print(f"[init-node] Starting enrollment for {args.target}...")
-    try:
-        initialize(
-            _repo_root,
-            args.target,
-            args.user_data,
-            args.port,
-            args.identity_file,
-            progress=lambda msg: print(f"[init-node] {msg}"),
-        )
-        print(f"[init-node] Successfully initialized {args.target}.")
-        return 0
-    except Exception as e:
-        sys.stderr.write(f"[init-node] Error: {e}\n")
-        return 1
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

ci/scripts/init-node/provisioning/__init__.py

@@ -1,3 +0,0 @@
-"""Shared initialization primitives; importing this package performs no host changes."""
-
-ENROLLMENT_INTERFACE = 1

ci/scripts/init-node/provisioning/baseline.py

@@ -1,487 +0,0 @@
-"""Shared Debian baseline. Callers own the exclusive initialization lock."""
-from __future__ import annotations
-
-from contextlib import contextmanager
-import fcntl
-from functools import wraps
-import hashlib
-import json
-import os
-from pathlib import Path
-import re
-import shutil
-import stat
-import subprocess
-import tempfile
-import urllib.request
-import uuid
-
-
-class BaselineError(RuntimeError):
-    """A required baseline postcondition could not be established."""
-
-
-def expected_errors(operation):
-    @wraps(operation)
-    def guarded(*args, **kwargs):
-        try:
-            return operation(*args, **kwargs)
-        except (OSError, ValueError) as exc:
-            raise BaselineError("Baseline filesystem or saved state is invalid; inspect the target before retrying.") from exc
-    return guarded
-
-
-BASE_PACKAGES = ("curl", "wget", "git", "rsyslog", "ufw")
-ADMIN_PACKAGES = (
-    "tcpdump", "socat", "rsync", "bind9-dnsutils", "iproute2", "net-tools",
-    "traceroute", "ca-certificates", "jq", "sed", "gawk", "tar", "gzip",
-    "bzip2", "xz-utils", "unzip", "htop", "iotop", "strace", "lsof",
-    "sysstat", "parted", "fdisk", "pciutils", "usbutils", "smartmontools",
-    "tmux", "vim", "less", "tree", "bash-completion", "python3",
-    "python3-minimal", "python3-yaml", "sudo", "cron", "iptables", "ipset",
-    "debian-archive-keyring", "kmod",
-)
-DOCKER_PACKAGES = ("docker-ce", "docker-ce-cli", "containerd.io",
-                   "docker-buildx-plugin", "docker-compose-plugin")
-SECURITY_PACKAGES = ("crowdsec", "crowdsec-firewall-bouncer-iptables")
-DOCKER_CONFLICTS = ("docker.io", "docker-doc", "docker-compose", "docker-buildx",
-                    "podman-docker", "containerd", "runc")
-DOCKER_CONFIG = {"data-root": "/srv/docker", "log-driver": "json-file",
-                 "log-opts": {"max-size": "20m", "max-file": "3"}}
-HELPER_REVISION = "020a8699f95592561f254d8d4ad1bb40d401dfc7"
-HELPER_URL = f"https://raw.githubusercontent.com/chaifeng/ufw-docker/{HELPER_REVISION}/ufw-docker"
-HELPER_SHA256 = "643e56b080567c567b4aa28650196849b2a2da5dd0473fd3e5216b0886035ab0"
-POLICY = Path("/usr/sbin/policy-rc.d")
-POLICY_STATE = Path("/usr/sbin/.apex-policy-rc.d")
-SECURITY_STATE = Path("/var/lib/apex/baseline-security.json")
-SECURITY_SERVICES = {"ufw": "ufw", "crowdsec": "crowdsec",
-                     "crowdsec-firewall-bouncer-iptables": "crowdsec-firewall-bouncer"}
-SECURITY_CONDITION = "[Unit]\nConditionPathExists=/dev/null/apex-security-start-allowed\n"
-APT = ["apt-get", "-o", "DPkg::Lock::Timeout=60", "-o",
-       "Acquire::Retries=2", "-o", "Acquire::http::Timeout=30", "-o",
-       "Acquire::https::Timeout=30"]
-
-
-def run(args, *, check=True, timeout=300):
-    try:
-        result = subprocess.run(args, text=True, capture_output=True, timeout=timeout,
-                                env={**os.environ, "LC_ALL": "C", "DEBIAN_FRONTEND": "noninteractive"})
-    except (OSError, subprocess.TimeoutExpired) as exc:
-        raise BaselineError(f"Cannot complete {args[0]}: {exc}") from exc
-    if check and result.returncode:
-        raise BaselineError(f"{args[0]} failed with exit status {result.returncode}; inspect target system logs.")
-    return result
-
-
-def preflight(*, strict=True):
-    if os.geteuid() != 0:
-        raise BaselineError("Baseline requires root.")
-    if not strict:
-        if not Path("/etc/debian_version").is_file():
-            raise BaselineError("Base policy requires a Debian-based system.")
-        return
-    if not Path("/etc/os-release").is_file():
-        raise BaselineError("Supported target is Debian 13 amd64 only.")
-    release = dict(line.split("=", 1) for line in Path("/etc/os-release").read_text().splitlines()
-                   if "=" in line and not line.startswith("#"))
-    if (release.get("ID", "").strip('"') != "debian" or
-            release.get("VERSION_ID", "").strip('"') != "13" or
-            run(["dpkg", "--print-architecture"]).stdout.strip() != "amd64"):
-        raise BaselineError("Supported target is Debian 13 amd64 only.")
-
-
-def package_versions():
-    result = run(["dpkg-query", "-W", "-f=${binary:Package}\t${Status}\t${Version}\n"])
-    installed = {}
-    for line in result.stdout.splitlines():
-        fields = line.split("\t")
-        if len(fields) == 3 and fields[1].split()[1:] == ["ok", "installed"]:
-            installed[fields[0].removesuffix(":amd64")] = fields[2]
-    return installed
-
-
-def write_managed(path, content, mode=0o644):
-    path = Path(path)
-    for parent in (path, *path.parents):
-        if parent.is_symlink():
-            raise BaselineError(f"Refusing managed symlink: {parent}")
-    payload = content.encode() if isinstance(content, str) else content
-    if path.exists() and path.read_bytes() == payload:
-        if path.stat().st_mode & 0o777 != mode:
-            path.chmod(mode)
-        return False
-    path.parent.mkdir(parents=True, exist_ok=True)
-    fd, temporary = tempfile.mkstemp(prefix=".apex-", dir=path.parent)
-    try:
-        with os.fdopen(fd, "wb") as stream:
-            stream.write(payload)
-            stream.flush()
-            os.fsync(stream.fileno())
-        os.chmod(temporary, mode)
-        os.replace(temporary, path)
-    finally:
-        if os.path.exists(temporary):
-            os.unlink(temporary)
-    return True
-
-
-def restore_service_policy():
-    if not POLICY_STATE.exists():
-        return
-    backup = POLICY_STATE / "original"
-    if (POLICY_STATE.is_symlink() or POLICY_STATE.stat().st_uid != os.geteuid()):
-        raise BaselineError("Refusing untrusted service policy recovery state.")
-    if (POLICY_STATE / "ready").exists() and POLICY.exists():
-        original_matches = (backup.exists() and not POLICY.is_symlink() and
-                            POLICY.read_bytes() == backup.read_bytes())
-        symlink_matches = (backup.is_symlink() and POLICY.is_symlink() and
-                           os.readlink(backup) == os.readlink(POLICY))
-        guard_matches = not POLICY.is_symlink() and POLICY.read_bytes() == b"#!/bin/sh\nexit 101\n"
-        if not (original_matches or symlink_matches or guard_matches):
-            raise BaselineError("Service policy changed outside APEX; recovery requires explicit reconciliation.")
-    if backup.exists() or backup.is_symlink():
-        # Copy first: an interrupted restoration must retain its recovery copy.
-        temporary = POLICY_STATE / "restoring"
-        if temporary.exists() or temporary.is_symlink():
-            temporary.unlink()
-        shutil.copy2(backup, temporary, follow_symlinks=False)
-        os.replace(temporary, POLICY)
-    elif (POLICY_STATE / "absent").exists():
-        POLICY.unlink(missing_ok=True)
-    elif (POLICY_STATE / "ready").exists():
-        raise BaselineError("Service policy recovery is incomplete; inspect /usr/sbin/.apex-policy-rc.d.")
-    shutil.rmtree(POLICY_STATE)
-
-
-@contextmanager
-def service_start_guard():
-    lock_path = Path("/run/lock/apex-package-policy.lock")
-    lock_path.parent.mkdir(parents=True, exist_ok=True)
-    with lock_path.open("a") as lock:
-        try:
-            fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
-        except BlockingIOError:
-            raise BaselineError("Another APEX package operation is running.") from None
-        with _service_start_guard():
-            yield
-
-
-@contextmanager
-def _service_start_guard():
-    restore_service_policy()
-    POLICY_STATE.mkdir(mode=0o700)
-    try:
-        if POLICY.exists() or POLICY.is_symlink():
-            shutil.copy2(POLICY, POLICY_STATE / "original", follow_symlinks=False)
-        else:
-            (POLICY_STATE / "absent").touch()
-        (POLICY_STATE / "ready").touch()
-        # Replace the path itself, preserving any original symlink in the backup.
-        temporary = POLICY_STATE / "guard"
-        temporary.write_text("#!/bin/sh\nexit 101\n")
-        temporary.chmod(0o755)
-        os.replace(temporary, POLICY)
-        yield
-    finally:
-        restore_service_policy()
-
-
-@contextmanager
-def security_service_guard(packages):
-    """Guard new security units without occupying package-owned unit paths."""
-    record = json.loads(SECURITY_STATE.read_text()) if SECURITY_STATE.exists() else {"version": 2, "services": {}}
-    if not isinstance(record, dict) or record.get("version") != 2:
-        raise BaselineError("Legacy security service guard state requires explicit reconciliation.")
-    managed = record.get("services")
-    if not isinstance(managed, dict) or any(SECURITY_SERVICES.get(key) != value for key, value in managed.items()):
-        raise BaselineError("Invalid baseline security service recovery state.")
-    installed = package_versions()
-    for package, service in SECURITY_SERVICES.items():
-        if package in packages and package not in installed and package not in managed:
-            if run(["systemctl", "is-enabled", service], check=False).stdout.strip() == "masked":
-                continue
-            managed[package] = service
-            write_managed(SECURITY_STATE, json.dumps({"version": 2, "services": managed}), 0o600)
-    for service in managed.values():
-        dropin = Path(f"/etc/systemd/system/{service}.service.d/99-apex-bootstrap-guard.conf")
-        if dropin.exists() and dropin.read_text() != SECURITY_CONDITION:
-            raise BaselineError("Security service guard changed outside APEX; reconcile it explicitly.")
-        # /dev/null cannot have children: this condition stays false after reboot.
-        write_managed(dropin, SECURITY_CONDITION)
-    if managed:
-        run(["systemctl", "daemon-reload"])
-    try:
-        yield
-    finally:
-        installed = package_versions()
-        for package, service in managed.items():
-            unit_exists = any(Path(f"{directory}/{service}.service").exists() for directory in
-                              ("/etc/systemd/system", "/usr/lib/systemd/system", "/lib/systemd/system"))
-            run(["systemctl", "disable", service], check=package in installed or unit_exists)
-            dropin = Path(f"/etc/systemd/system/{service}.service.d/99-apex-bootstrap-guard.conf")
-            if dropin.exists():
-                if dropin.is_symlink() or dropin.read_text() != SECURITY_CONDITION:
-                    raise BaselineError("Security service guard changed outside APEX; reconcile it explicitly.")
-                dropin.unlink()
-        if managed:
-            run(["systemctl", "daemon-reload"])
-        SECURITY_STATE.unlink(missing_ok=True)
-
-
-def install_missing(packages):
-    installed = package_versions()
-    missing = [package for package in packages if package not in installed]
-    if not missing:
-        return []
-    run([*APT, "update", "--error-on=any"])
-    command = [*APT, "install", "-y", "--no-install-recommends", "--no-upgrade", "--no-remove", *missing]
-    plan = run([*command, "--simulate"]).stdout
-    if any(line.startswith("Remv ") or re.match(r"^Inst \S+ \[", line)
-           for line in plan.splitlines()):
-        raise BaselineError("Missing packages require an upgrade/removal; reconcile packages explicitly before retrying.")
-    # Package writes must settle even after coordinator loss; stages gate later work.
-    run(command, timeout=None)
-    remaining = set(missing) - package_versions().keys()
-    if remaining:
-        raise BaselineError("Required packages not installed: " + ", ".join(sorted(remaining)))
-    return missing
-
-
-def base_policy():
-    return {"packages": list(BASE_PACKAGES), "files": {
-        "/etc/apt/apt.conf.d/99-apex": 'APT::Keep-Downloaded-Packages "0";\n',
-        "/etc/systemd/journald.conf.d/99-apex.conf": "[Journal]\nSystemMaxUse=200M\n",
-    }, "helper": {"url": HELPER_URL, "sha256": HELPER_SHA256}}
-
-
-def configure_common():
-    policy = base_policy()
-    for legacy in ("/etc/apt/apt.conf.d/01clean", "/etc/systemd/journald.conf.d/maxuse.conf"):
-        path = Path(legacy)
-        if path.exists() or path.is_symlink():
-            path.unlink()
-    for path, content in policy["files"].items():
-        changed = write_managed(path, content)
-        if changed and path.endswith("99-apex.conf"):
-            run(["systemctl", "restart", "systemd-journald"])
-    target = Path("/usr/local/bin/ufw-docker")
-    if (not target.is_file() or hashlib.sha256(target.read_bytes()).hexdigest() != HELPER_SHA256):
-        try:
-            with urllib.request.urlopen(HELPER_URL, timeout=30) as response:
-                data = response.read()
-        except OSError as exc:
-            raise BaselineError("Unable to download pinned ufw-docker helper.") from exc
-        if hashlib.sha256(data).hexdigest() != HELPER_SHA256:
-            raise BaselineError("Pinned ufw-docker checksum mismatch.")
-        write_managed(target, data, 0o755)
-    elif target.is_symlink():
-        raise BaselineError("Refusing a symlinked ufw-docker helper.")
-    else:
-        target.chmod(0o755)
-
-
-def docker_preflight(installed):
-    conflicts = sorted(set(DOCKER_CONFLICTS) & installed.keys())
-    if conflicts:
-        raise BaselineError("Conflicting Docker packages require explicit reconciliation: " + ", ".join(conflicts))
-    path = Path("/etc/docker/daemon.json")
-    try:
-        config = json.loads(path.read_text()) if path.exists() else {}
-    except (OSError, ValueError) as exc:
-        raise BaselineError("Cannot validate existing Docker daemon configuration.") from exc
-    if not isinstance(config, dict):
-        raise BaselineError("Docker daemon configuration must be an object.")
-    for name, value in DOCKER_CONFIG.items():
-        if name in config and config[name] != value:
-            raise BaselineError(f"Conflicting Docker {name}; reconcile explicitly before initialization.")
-    old_data = Path("/var/lib/docker")
-    if old_data.exists() and any(old_data.iterdir()):
-        raise BaselineError("Existing /var/lib/docker data must be reconciled explicitly; no data migration is automatic.")
-    data = Path("/srv/docker")
-    if data.is_symlink() or data.parent.is_symlink():
-        raise BaselineError("Refusing a redirected Docker data-root.")
-    if data.exists() and any(data.iterdir()) and config.get("data-root") != "/srv/docker":
-        raise BaselineError("Unmanaged /srv/docker data requires explicit reconciliation.")
-    if "docker-ce" in installed:
-        active = run(["systemctl", "is-active", "--quiet", "docker"], check=False).returncode == 0
-        if active:
-            root = run(["docker", "info", "--format", "{{.DockerRootDir}}"] ).stdout.strip()
-            if root != "/srv/docker" or any(config.get(key) != value for key, value in DOCKER_CONFIG.items()):
-                raise BaselineError("Running Docker has incompatible data-root/log policy; reconcile it before retrying.")
-    return {**config, **DOCKER_CONFIG}
-
-
-def configure_sources():
-    write_managed("/etc/apt/sources.list.d/apex-zfs.sources",
-                  "Types: deb\nURIs: https://deb.debian.org/debian\nSuites: trixie\n"
-                  "Components: contrib\nArchitectures: amd64\n"
-                  "Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg\n")
-    for name, key_url, uri, suite, component in (
-        ("docker", "https://download.docker.com/linux/debian/gpg",
-         "https://download.docker.com/linux/debian", "trixie", "stable"),
-        ("crowdsec", "https://packagecloud.io/crowdsec/crowdsec/gpgkey",
-         "https://packagecloud.io/crowdsec/crowdsec/any", "any", "main"),
-    ):
-        key = Path(f"/etc/apt/keyrings/apex-{name}.asc")
-        if not key.exists():
-            try:
-                with urllib.request.urlopen(key_url, timeout=30) as response:
-                    data = response.read()
-            except OSError as exc:
-                raise BaselineError(f"Cannot download {name} repository signing key.") from exc
-            if not data.startswith(b"-----BEGIN PGP PUBLIC KEY BLOCK-----"):
-                raise BaselineError(f"Invalid {name} repository signing key.")
-            write_managed(key, data)
-        write_managed(f"/etc/apt/sources.list.d/apex-{name}.sources",
-                      f"Types: deb\nURIs: {uri}\nSuites: {suite}\nComponents: {component}\n"
-                      f"Architectures: amd64\nSigned-By: {key}\n")
-
-
-def ensure_service(name):
-    if run(["systemctl", "is-enabled", "--quiet", name], check=False).returncode:
-        run(["systemctl", "enable", name])
-    if run(["systemctl", "is-active", "--quiet", name], check=False).returncode:
-        run(["systemctl", "start", name])
-
-
-def repair_bouncer_credentials():
-    """Repair package placeholders without rewriting administrator YAML settings."""
-    config = Path("/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml")
-    identity = config.with_suffix(".yaml.id")
-    for path in (config, identity):
-        if any(parent.is_symlink() for parent in (path, *path.parents)):
-            raise BaselineError("Refusing redirected CrowdSec bouncer credential paths.")
-        if path.exists():
-            metadata = path.stat()
-            if (not stat.S_ISREG(metadata.st_mode) or metadata.st_nlink != 1 or
-                    metadata.st_uid != os.geteuid()):
-                raise BaselineError("Refusing untrusted CrowdSec bouncer credential files.")
-    if not config.exists():
-        raise BaselineError("CrowdSec bouncer configuration is missing; reconcile the package installation.")
-    content = config.read_text()
-    entries = list(re.finditer(r"^(?:api_key|'api_key'|\"api_key\"):[^\r\n]*", content, re.MULTILINE))
-    if len(entries) != 1:
-        raise BaselineError("CrowdSec bouncer requires one explicit api_key; reconcile its configuration.")
-    entry = entries[0]
-    for following in content[entry.end():].splitlines():
-        if not following.strip() or following.lstrip().startswith("#"):
-            continue
-        if following[:1].isspace():
-            raise BaselineError("Multiline CrowdSec bouncer api_key requires explicit reconciliation.")
-        break
-    scalar = re.fullmatch(r"(?:api_key|'api_key'|\"api_key\"):[ \t]*(?P<value>\"[^\"]*\"|'[^']*'|[^ \t#'\"]*)(?:[ \t]+(?:#.*)?)?", entry.group())
-    if scalar is None:
-        raise BaselineError("Unsupported CrowdSec bouncer api_key; reconcile its configuration.")
-    value = scalar.group("value")
-    uninitialized = value in ("null", "Null", "NULL", "~")
-    if value[:1] in ("'", '"'):
-        value = value[1:-1]
-    if not uninitialized and value not in ("", "<API_KEY>", "$API_KEY", "${API_KEY}"):
-        return
-    metadata = config.stat()
-    registration = "cs-firewall-bouncer-apex-" + uuid.uuid4().hex
-    key = run(["cscli", "-oraw", "bouncers", "add", registration]).stdout.strip()
-    if not re.fullmatch(r"[A-Za-z0-9_+/=-]{24,256}", key):
-        raise BaselineError("CrowdSec bouncer registration did not return a usable API key.")
-    start, end = scalar.span("value")
-    separator = " " if entry.group()[start - 1] == ":" else ""
-    updated = content[:entry.start() + start] + separator + key + content[entry.start() + end:]
-
-    def publish(path, payload, mode, owner):
-        fd, temporary = tempfile.mkstemp(prefix=".apex-", dir=path.parent)
-        try:
-            with os.fdopen(fd, "w") as stream:
-                os.fchown(stream.fileno(), *owner)
-                os.fchmod(stream.fileno(), mode)
-                stream.write(payload)
-                stream.flush()
-                os.fsync(stream.fileno())
-            os.replace(temporary, path)
-            directory = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY)
-            try:
-                os.fsync(directory)
-            finally:
-                os.close(directory)
-        finally:
-            if os.path.exists(temporary):
-                os.unlink(temporary)
-
-    # Publish the key last: interruption can orphan a registration, but retries
-    # replace the placeholder and its package bookkeeping together in this order.
-    owner = (metadata.st_uid, metadata.st_gid)
-    publish(identity, registration + "\n", 0o600, owner)
-    publish(config, updated, (stat.S_IMODE(metadata.st_mode) & 0o640) | 0o600, owner)
-    if config.read_text() != updated or identity.read_text().strip() != registration:
-        raise BaselineError("CrowdSec bouncer credential publication did not converge.")
-
-
-@expected_errors
-def apply_base_policy():
-    """The bounded configure/base policy: no Docker, storage or enrollment."""
-    preflight(strict=False)
-    with service_start_guard(), security_service_guard(BASE_PACKAGES):
-        install_missing(BASE_PACKAGES)
-    configure_common()
-    ensure_service("rsyslog")
-    run(["/usr/local/bin/ufw-docker", "install"])
-    return {"packages": {key: value for key, value in package_versions().items() if key in BASE_PACKAGES}}
-
-
-@expected_errors
-def apply_baseline():
-    """Converge baseline and return verified package versions and capabilities."""
-    preflight()
-    installed = package_versions()
-    docker_config = docker_preflight(installed)
-    kernel = run(["uname", "-r"]).stdout.strip()
-    if not re.fullmatch(r"[A-Za-z0-9.+_-]+", kernel):
-        raise BaselineError("Unsupported running kernel release.")
-    packages = (*BASE_PACKAGES, *ADMIN_PACKAGES, *DOCKER_PACKAGES,
-                *SECURITY_PACKAGES, "linux-headers-amd64", f"linux-headers-{kernel}",
-                "zfsutils-linux", "zfs-dkms")
-    with service_start_guard(), security_service_guard(packages):
-        install_missing((*BASE_PACKAGES, *ADMIN_PACKAGES))
-        configure_sources()
-        install_missing(("crowdsec",))
-        install_missing(packages)
-        repair_bouncer_credentials()
-    configure_common()
-    write_managed("/etc/docker/daemon.json", json.dumps(docker_config, indent=2) + "\n")
-    Path("/srv/docker").mkdir(parents=True, exist_ok=True)
-    ensure_service("containerd")
-    ensure_service("docker")
-    ensure_service("rsyslog")
-    ensure_service("cron")
-    root = run(["docker", "info", "--format", "{{.DockerRootDir}}"] ).stdout.strip()
-    if root != "/srv/docker":
-        raise BaselineError("Docker did not activate the required /srv/docker data-root.")
-    compose = run(["docker", "compose", "version", "--short"]).stdout.strip()
-    if not compose:
-        raise BaselineError("Docker Compose capability is unavailable.")
-    module = run(["modprobe", "zfs"], check=False)
-    if module.returncode:
-        raise BaselineError(f"ZFS unavailable for running kernel {kernel}; install matching headers/module or reboot into the supported kernel, then retry. No automatic reboot was performed.")
-    zfs = run(["zfs", "version"]).stdout.strip()
-    versions = package_versions()
-    if set(packages) - versions.keys():
-        raise BaselineError("Baseline package verification failed.")
-    return {"packages": {name: versions[name] for name in packages},
-            "capabilities": {"docker_data_root": root, "compose": compose, "zfs": zfs,
-                             "kernel": kernel, "ufw_docker_sha256": HELPER_SHA256},
-            "sources": {"docker": "docker.com/debian trixie stable",
-                        "crowdsec": "packagecloud.io/crowdsec/crowdsec/any any main",
-                        "zfs": "Debian trixie contrib"}}
-
-
-if __name__ == "__main__":
-    import argparse
-    from .state import ProvisionError, exclusive_lock
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument("--base-policy", action="store_true")
-    args = parser.parse_args()
-    try:
-        preflight(strict=not args.base_policy)
-        with exclusive_lock("/var/lib/apex-init/init.lock"):
-            print(json.dumps(apply_base_policy() if args.base_policy else apply_baseline(), sort_keys=True))
-    except (BaselineError, OSError, ProvisionError) as exc:
-        parser.exit(1, f"Baseline failed: {exc}\n")

ci/scripts/init-node/provisioning/contract.py

@@ -1,236 +0,0 @@
-"""Pure validation for the supported APEX enrollment contract."""
-from __future__ import annotations
-
-import re
-from dataclasses import dataclass, field
-from pathlib import PurePosixPath
-from typing import Any, Dict, Mapping, Optional, Tuple
-from urllib.parse import urlsplit
-
-
-class ContractError(ValueError):
-    """A safe diagnostic that never includes a supplied value."""
-
-
-@dataclass(frozen=True)
-class Repository:
-    name: str
-    url: str
-    ref: Optional[str]
-
-
-@dataclass(frozen=True)
-class Tier:
-    number: int
-    mode: str
-    device: Optional[str]
-    pool_guid: Optional[str]
-
-
-@dataclass(frozen=True)
-class ManagedFile:
-    path: str
-    owner: str
-    mode: int
-    content: str = field(repr=False)
-
-
-@dataclass(frozen=True)
-class Enrollment:
-    hostname: str
-    repository: Repository
-    tiers: Tuple[Tier, ...]
-    authorized_keys: Tuple[str, ...] = field(repr=False)
-    files: Tuple[ManagedFile, ...] = field(repr=False)
-
-    def identity(self) -> Dict[str, Any]:
-        return {
-            'hostname': self.hostname,
-            'repository': {
-                'name': self.repository.name,
-                'url': self.repository.url,
-                'ref': self.repository.ref,
-            },
-            'tiers': [
-                {'number': t.number, 'mode': t.mode,
-                 'device': t.device, 'pool_guid': t.pool_guid}
-                for t in self.tiers
-            ],
-        }
-
-
-_ENV_KEYS = {'APEX_REPO_NAME', 'APEX_REPO_URL', 'APEX_REPO_REF'} | {
-    f'APEX_TIER{number}_{suffix}'
-    for number in (1, 2, 3)
-    for suffix in ('DEVICE', 'MODE', 'POOL_GUID')
-}
-_ASSIGNMENT = re.compile(
-    r'''\s*([A-Z][A-Z0-9_]*)\s*=\s*(?:"([^"\\]*)"|'([^'\\]*)'|([^\s'"\\()#]*))\s*'''
-)
-_FILE_POLICY = {
-    '/etc/apex/bootstrap.env': ('root:root', 0o600, False),
-    '/home/adam/.ssh/id_ed25519': ('adam:adam', 0o600, True),
-    '/home/adam/.ssh/known_hosts': ('adam:adam', 0o644, True),
-}
-
-
-def _require(condition: bool, diagnostic: str) -> None:
-    if not condition:
-        raise ContractError(diagnostic)
-
-
-def _mapping(value: Any, required: set, optional: set, diagnostic: str) -> Mapping:
-    _require(isinstance(value, Mapping), diagnostic)
-    _require(set(value) >= required and set(value) <= required | optional, diagnostic)
-    return value
-
-
-def _plain(value: Any, diagnostic: str, allow_empty: bool = False) -> str:
-    _require(type(value) is str, diagnostic)
-    _require(allow_empty or bool(value), diagnostic)
-    _require(not any(ord(c) < 32 or ord(c) == 127 for c in value), diagnostic)
-    return value
-
-
-def parse_bootstrap_env(text: str) -> Dict[str, str]:
-    _require(type(text) is str, 'bootstrap.env must contain literal assignments')
-    result = {}
-    for line in text.splitlines():
-        if not line.strip() or line.lstrip().startswith('#'):
-            continue
-        match = _ASSIGNMENT.fullmatch(line)
-        _require(match is not None, 'bootstrap.env must contain literal assignments')
-        key = match.group(1)
-        _require(key in _ENV_KEYS, 'bootstrap.env contains an unsupported assignment')
-        _require(key not in result, 'bootstrap.env contains a duplicate assignment')
-        value = next(part for part in match.groups()[1:] if part is not None)
-        _require(not any(c in value for c in '$`;|&<>'),
-                 'bootstrap.env contains unsupported shell syntax')
-        result[key] = _plain(value, 'bootstrap.env contains an invalid literal', True)
-    return result
-
-
-def _repository(values: Mapping[str, str]) -> Repository:
-    name = values.get('APEX_REPO_NAME', '')
-    _require(bool(re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9._-]*', name))
-             and name not in ('.', '..'), 'repository name must be one safe directory name')
-    url = values.get('APEX_REPO_URL', '')
-    _require(bool(url) and not any(c.isspace() for c in url), 'repository URL is invalid')
-    if '://' in url:
-        try:
-            parsed = urlsplit(url)
-            port = parsed.port
-        except ValueError:
-            raise ContractError('repository URL is invalid') from None
-        _require(parsed.scheme in ('https', 'ssh') and bool(parsed.hostname)
-                 and parsed.password is None and not parsed.query and not parsed.fragment
-                 and bool(parsed.path.strip('/')) and (port is None or port > 0),
-                 'repository URL is invalid or contains credentials')
-        _require(parsed.scheme != 'https' or parsed.username is None,
-                 'HTTPS repository URL must not contain credentials')
-    else:
-        _require(bool(re.fullmatch(
-            r'[A-Za-z0-9._-]+@[A-Za-z0-9.-]+:[A-Za-z0-9_./-]+', url)),
-            'repository URL must use HTTPS or SSH')
-        remote_path = url.split(':', 1)[1]
-        _require(not remote_path.startswith('-') and remote_path not in ('.', '..'),
-                 'repository URL is invalid')
-    ref = values.get('APEX_REPO_REF') or None
-    if ref is not None:
-        _require(not ref.startswith('-') and not any(c.isspace() for c in ref),
-                 'repository ref is invalid')
-    return Repository(name, url, ref)
-
-
-def _tiers(values: Mapping[str, str]) -> Tuple[Tier, ...]:
-    result = []
-    devices = set()
-    guids = set()
-    for number in (1, 2, 3):
-        prefix = f'APEX_TIER{number}_'
-        device = values.get(prefix + 'DEVICE') or None
-        mode = values.get(prefix + 'MODE')
-        guid = values.get(prefix + 'POOL_GUID') or None
-        _require(mode is not None or device is None,
-                 'a mapped tier device requires an explicit mode')
-        mode = mode or 'directory'
-        _require(mode in ('directory', 'create', 'import'), 'tier mode is invalid')
-        if mode == 'directory':
-            _require(device is None and guid is None,
-                     'directory tier must not select a device or pool')
-        else:
-            _require(device is not None and device.startswith('/dev/')
-                     and not any(c.isspace() for c in device)
-                     and '..' not in PurePosixPath(device).parts
-                     and str(PurePosixPath(device)) == device,
-                     'block-backed tier requires a normalized device path')
-            _require(device not in devices, 'a device is selected by multiple tiers')
-            devices.add(device)
-            if mode == 'create':
-                _require(guid is None, 'create tier must not name an existing pool')
-            else:
-                _require(guid is not None and bool(re.fullmatch(r'[1-9][0-9]{0,19}', guid))
-                         and int(guid) < 2**64, 'import tier requires a valid pool GUID')
-                _require(guid not in guids, 'a pool is selected by multiple tiers')
-                guids.add(guid)
-        result.append(Tier(number, mode, device, guid))
-    return tuple(result)
-
-
-def normalize_user_data(document: Any) -> Enrollment:
-    doc = _mapping(document,
-                   {'preserve_hostname', 'hostname', 'users', 'write_files', 'runcmd'},
-                   set(), 'unsupported cloud-config structure')
-    _require(doc['preserve_hostname'] is False, 'preserve_hostname must be false')
-    hostname = _plain(doc['hostname'], 'hostname is invalid')
-    labels = hostname.split('.')
-    _require(len(hostname) <= 253 and len(labels) >= 2
-             and all(re.fullmatch(r'[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?', x)
-                     for x in labels), 'hostname must be a valid FQDN')
-    users = doc['users']
-    _require(type(users) is list and len(users) == 1, 'exactly one administrator is required')
-    user = _mapping(users[0],
-                    {'name', 'sudo', 'groups', 'shell', 'ssh_authorized_keys'},
-                    set(), 'unsupported administrator configuration')
-    _require(user['name'] == 'adam' and user['shell'] == '/bin/bash',
-             'administrator must be adam with Bash')
-    _require(type(user['sudo']) is str and ''.join(user['sudo'].split()) in
-             ('ALL=(ALL)NOPASSWD:ALL', 'ALL=(ALL:ALL)NOPASSWD:ALL'),
-             'administrator requires passwordless sudo')
-    groups = user['groups']
-    _require(type(groups) is list and len(groups) == 2
-             and all(type(g) is str for g in groups) and set(groups) == {'sudo', 'docker'},
-             'administrator requires sudo and docker groups')
-    keys = user['ssh_authorized_keys']
-    _require(type(keys) is list and len(keys) > 0, 'administrator requires public SSH keys')
-    authorized_keys = tuple(_plain(k, 'administrator SSH key is invalid') for k in keys)
-    _require(all(k.strip() for k in authorized_keys), 'administrator SSH key is empty')
-    commands = doc['runcmd']
-    _require(commands in (['/usr/local/bin/apex-bootstrap'],
-                          [['/usr/local/bin/apex-bootstrap']]),
-             'only the standard APEX bootstrap command is supported')
-    writes = doc['write_files']
-    _require(type(writes) is list, 'write_files must be a list')
-    files = []
-    seen = set()
-    for write in writes:
-        item = _mapping(write, {'path', 'owner', 'permissions', 'content'}, {'defer'},
-                        'unsupported write_files entry')
-        path = _plain(item['path'], 'managed file path is invalid')
-        _require(path in _FILE_POLICY and path not in seen,
-                 'managed file path is unsupported or repeated')
-        seen.add(path)
-        owner, mode, deferred = _FILE_POLICY[path]
-        permission = item['permissions']
-        _require(type(permission) is str and bool(re.fullmatch(r'0?[0-7]{3}', permission))
-                 and int(permission, 8) == mode and item['owner'] == owner,
-                 'managed file ownership or permissions are invalid')
-        _require(item.get('defer', False) is deferred, 'managed file defer setting is invalid')
-        content = item['content']
-        _require(type(content) is str and bool(content.strip()) and '\x00' not in content,
-                 'managed file content is invalid')
-        files.append(ManagedFile(path, owner, mode, content))
-    bootstrap = next((f for f in files if f.path == '/etc/apex/bootstrap.env'), None)
-    _require(bootstrap is not None, 'bootstrap.env is required')
-    values = parse_bootstrap_env(bootstrap.content)
-    return Enrollment(hostname, _repository(values), _tiers(values), authorized_keys, tuple(files))

ci/scripts/init-node/provisioning/image.py

@@ -1,124 +0,0 @@
-"""Image-only installation and sanitization; never called by remote enrollment."""
-from __future__ import annotations
-
-import argparse
-import io
-import json
-import os
-from pathlib import Path
-import shutil
-import tarfile
-import uuid
-
-from . import ENROLLMENT_INTERFACE
-from .state import ProvisionError, atomic_write, exclusive_lock, safe_path
-from .target import preflight, run
-
-
-def install():
-    from .transport import build_bundle
-    source = Path(__file__).resolve().parents[2]
-    data, digest, _ = build_bundle(source)
-    destination = safe_path(Path('/usr/local/lib/apex-provisioner') / digest)
-    destination.mkdir(mode=0o755, parents=True, exist_ok=False)
-    with tarfile.open(fileobj=io.BytesIO(data)) as archive:
-        archive.extractall(destination, filter='data')
-    revision = os.environ.get('APEX_PROVISION_REVISION', 'unversioned-modified')
-    metadata = {'bundle_digest': digest, 'revision': revision, 'path': str(destination),
-                'baseline_revision': revision, 'enrollment_interface': ENROLLMENT_INTERFACE}
-    atomic_write('/etc/apex/provisioner.json', json.dumps(metadata) + '\n')
-    for executable, operation in (('apex-provision-image', ''), ('apex-bootstrap', 'cloud-enroll ')):
-        content = ('#!/bin/sh\nset -eu\ncd ' + str(destination) + '\n'
-                   'exec /usr/bin/python3 -m engine.provisioning.image ' + operation + '"$@"\n')
-        atomic_write('/usr/local/bin/' + executable, content, 0o755)
-    return metadata
-
-
-def cloud_enroll():
-    from .access import Transition
-    from .target import enroll
-    from .userdata import decode_user_data
-    metadata = json.loads(Path('/etc/apex/provisioner.json').read_text())
-    if metadata.get('enrollment_interface') != ENROLLMENT_INTERFACE:
-        raise ProvisionError('image enrollment interface is missing or unsupported; rebuild the image')
-    instance = Path('/var/lib/cloud/instance').resolve(strict=True)
-    try:
-        instance.relative_to('/var/lib/cloud/instances')
-    except ValueError:
-        raise RuntimeError('cloud-init instance path is outside its managed directory') from None
-    path = safe_path(instance / 'user-data.txt')
-    if path.stat().st_uid != 0 or path.stat().st_size > 1024 * 1024:
-        raise RuntimeError('cloud-init user-data is not a supported root-owned file')
-    spec = decode_user_data(path.read_bytes())
-    result = enroll(spec, metadata['revision'], metadata['bundle_digest'])
-    token = uuid.uuid4().hex
-    transition = Transition()
-    try:
-        current = transition.execute('begin', token, 22, '127.0.0.1')
-        run(['runuser', '-u', 'adam', '--', 'sudo', '-n', 'true'])
-        if current['status'] not in ('existing', 'committed'):
-            transition.execute('finalize', token)
-        if current['status'] != 'committed':
-            transition.execute('commit', token)
-    except Exception:
-        transition.execute('rollback', token)
-        raise
-    result['external_access'] = 'not-verified-cloud-init'
-    atomic_write('/var/lib/apex-init/enrollment.json', json.dumps(result, sort_keys=True) + '\n')
-    return result
-
-
-def sanitize_files(root=Path('/')):
-    root = Path(root)
-    for home in ('root', 'home/debian'):
-        for name in ('.ssh/authorized_keys', '.ssh/authorized_keys2', '.bash_history'):
-            path = root / home / name
-            if path.exists() or path.is_symlink():
-                path.unlink()
-    for path in (root / 'etc/ssh').glob('ssh_host_*'):
-        if path.is_file() or path.is_symlink():
-            path.unlink()
-    atomic_write(root / 'etc/machine-id', '', 0o444)
-    dbus = root / 'var/lib/dbus/machine-id'
-    if dbus.exists() or dbus.is_symlink():
-        dbus.unlink()
-    dbus.parent.mkdir(parents=True, exist_ok=True)
-    dbus.symlink_to('/etc/machine-id')
-    for name in ('var/lib/apex-init', 'tmp/engine', 'var/lib/cloud'):
-        path = root / name
-        if path.is_symlink():
-            path.unlink()
-        elif path.exists():
-            shutil.rmtree(path)
-    autologin = root / 'etc/systemd/system/serial-getty@ttyS0.service.d/autologin.conf'
-    autologin.unlink(missing_ok=True)
-    atomic_write(root / 'etc/systemd/system/ssh.service.d/10-generate-host-keys.conf',
-                 '[Service]\nExecStartPre=\nExecStartPre=/usr/bin/ssh-keygen -A\n'
-                 'ExecStartPre=/usr/sbin/sshd -t\n', 0o644)
-
-
-def main(argv=None):
-    parser = argparse.ArgumentParser()
-    parser.add_argument('operation', choices=('install', 'baseline', 'install-runner', 'sanitize', 'cloud-enroll'))
-    args = parser.parse_args(argv)
-    preflight()
-    if args.operation == 'install':
-        print(json.dumps(install()))
-    elif args.operation == 'baseline':
-        from .baseline import apply_baseline
-        with exclusive_lock('/var/lib/apex-init/init.lock'):
-            result = apply_baseline()
-            atomic_write('/etc/apex/image-baseline.json', json.dumps(result, sort_keys=True) + '\n')
-    elif args.operation == 'install-runner':
-        run(['test', '-x', '/usr/local/bin/apex-bootstrap'])
-    elif args.operation == 'sanitize':
-        run(['usermod', '--password', '!', 'root'])
-        run(['usermod', '--password', '!', 'debian'])
-        sanitize_files()
-    else:
-        print(json.dumps(cloud_enroll(), sort_keys=True))
-    return 0
-
-
-if __name__ == '__main__':
-    raise SystemExit(main())

ci/scripts/init-node/provisioning/repository.py

@@ -1,121 +0,0 @@
-"""Pinned, non-destructive checkout enrollment as the node administrator."""
-from __future__ import annotations
-
-import os
-from pathlib import Path
-import pwd
-import subprocess
-
-from engine.identity import read_env
-from . import ENROLLMENT_INTERFACE
-from .state import ProvisionError, safe_path
-
-
-def as_adam(args, cwd=None):
-    environment = dict(os.environ, GIT_TERMINAL_PROMPT='0',
-                       GIT_SSH_COMMAND='ssh -o BatchMode=yes -o StrictHostKeyChecking=yes',
-                       LC_ALL='C')
-    result = subprocess.run(['runuser', '-u', 'adam', '--', *args], cwd=cwd,
-                            env=environment, text=True, capture_output=True, timeout=300)
-    if result.returncode:
-        # Git/hook stderr may contain credential files or remote URL credentials.
-        raise ProvisionError('repository operation failed; check administrator Git access and the pinned checkout')
-    return result.stdout.strip()
-
-
-def verify_checkout_setup(destination, account):
-    if as_adam(['git', 'config', '--get', 'core.hooksPath'], destination) != 'commons/githooks':
-        raise ProvisionError('checkout hook configuration is missing or incompatible')
-    home = safe_path(account.pw_dir)
-    bashrc = safe_path(home / '.bashrc')
-    entry = 'export PATH="' + str(destination / 'commons') + ':$PATH"'
-    if not bashrc.is_file() or bashrc.read_text().splitlines().count(entry) != 1:
-        raise ProvisionError('checkout PATH setup is missing or duplicated; reconcile init.sh before retrying')
-    vim = home / '.vimrc'
-    if not vim.is_symlink() or vim.resolve() != (destination / 'commons/etc/.vimrc').resolve():
-        raise ProvisionError('checkout editor link is missing or incompatible')
-
-
-def reconcile_repository(spec, state, save, srv=Path('/srv')):
-    repository = spec.repository
-    destination = safe_path(Path(srv) / repository.name)
-    staging = safe_path(Path(srv) / ('.apex-clone-' + repository.name))
-    expected = {'name': repository.name, 'url': repository.url, 'ref': repository.ref}
-    account = pwd.getpwnam('adam')
-    record = state.setdefault('repository', {})
-    if record and record.get('request') != expected:
-        raise ProvisionError('repository enrollment conflicts with its recorded request')
-    if not record:
-        if destination.exists() or staging.exists():
-            raise ProvisionError('unrecorded repository path exists; explicit adoption is required')
-        record.update(request=expected, clone_intent=True)
-        save()
-    if not destination.exists():
-        if staging.exists():
-            # An interrupted clone may be incomplete: never erase or guess its state.
-            raise ProvisionError('an interrupted clone remains; inspect the staging checkout before retrying')
-        staging.mkdir(mode=0o755)
-        os.chown(staging, account.pw_uid, account.pw_gid)
-        as_adam(['git', 'clone', '--no-checkout', '--', repository.url, str(staging)])
-        os.rename(staging, destination)
-    safe_path(destination / '.git')
-    origin = as_adam(['git', 'remote', 'get-url', 'origin'], destination)
-    if origin != repository.url:
-        raise ProvisionError('existing checkout has a different origin; it was preserved')
-    if 'commit' not in record:
-        reference = repository.ref or 'HEAD'
-        if repository.ref:
-            as_adam(['git', 'fetch', '--no-tags', 'origin', reference], destination)
-            reference = 'FETCH_HEAD'
-        commit = as_adam(['git', 'rev-parse', '--verify', reference + '^{commit}'], destination)
-        record['commit'] = commit
-        record['checkout_intent'] = True
-        save()
-    commit = record['commit']
-    if record.get('checkout_intent'):
-        # --no-checkout index intentionally looks dirty before the first checkout.
-        if any(p.name != '.git' for p in destination.iterdir()):
-            head = as_adam(['git', 'rev-parse', 'HEAD'], destination)
-            dirty = as_adam(['git', 'status', '--porcelain'], destination)
-            if head != commit or dirty:
-                raise ProvisionError('interrupted checkout has unexpected contents; it was preserved')
-        else:
-            as_adam(['git', 'checkout', '--detach', commit], destination)
-        record.pop('checkout_intent')
-        save()
-    if as_adam(['git', 'rev-parse', 'HEAD'], destination) != commit:
-        raise ProvisionError('checkout revision changed; initialization will not reset it')
-    if as_adam(['git', 'status', '--porcelain'], destination):
-        raise ProvisionError('checkout contains local changes; initialization will not reset it')
-    node_env = safe_path(destination / 'node.env')
-    node_identity = read_env(str(node_env))
-    if node_identity.get('APEX_NODE_FQDN') != spec.hostname:
-        raise ProvisionError('repository node.env does not declare the requested hostname')
-    if node_identity.get('APEX_ENROLLMENT_INTERFACE') != str(ENROLLMENT_INTERFACE):
-        raise ProvisionError('node checkout enrollment interface is missing or unsupported; update its init.sh and node.env together')
-    as_adam(['git', 'submodule', 'update', '--init', '--recursive'], destination)
-    commons = safe_path(destination / 'commons')
-    if not commons.is_dir():
-        raise ProvisionError('node checkout does not contain its pinned commons submodule')
-    commons_commit = as_adam(['git', 'rev-parse', 'HEAD'], commons)
-    gitlink = as_adam(['git', 'ls-tree', 'HEAD', '--', 'commons'], destination).split()
-    if gitlink != ['160000', 'commit', commons_commit, 'commons']:
-        raise ProvisionError('commons is not the exact pinned Git submodule')
-    if Path(as_adam(['git', 'rev-parse', '--show-toplevel'], commons)).resolve() != commons.resolve():
-        raise ProvisionError('commons does not have its own Git worktree')
-    if record.get('commons_commit', commons_commit) != commons_commit:
-        raise ProvisionError('pinned commons revision changed unexpectedly')
-    hook = safe_path(destination / 'init.sh')
-    if not hook.is_file():
-        raise ProvisionError('node repository is missing its checkout initializer')
-    if not record.get('hook_complete'):
-        as_adam(['bash', './init.sh'], destination)
-        verify_checkout_setup(destination, account)
-        record['hook_complete'] = True
-    else:
-        verify_checkout_setup(destination, account)
-    if as_adam(['git', 'status', '--porcelain'], destination):
-        raise ProvisionError('checkout initializer changed tracked files; inspect before resuming')
-    record['commons_commit'] = commons_commit
-    save()
-    return {'path': str(destination), 'commit': commit, 'commons_commit': commons_commit}

ci/scripts/init-node/provisioning/security.py

@@ -1,591 +0,0 @@
-"""Security activation owned by the protected access transaction."""
-from __future__ import annotations
-
-import base64
-import hashlib
-import ipaddress
-import json
-import os
-import re
-import shlex
-import tempfile
-import uuid
-
-from .access import atomic
-from .baseline import HELPER_SHA256
-
-
-SERVICES = ('ufw', 'crowdsec', 'crowdsec-firewall-bouncer')
-FILES = ('/etc/default/ufw', '/etc/ufw/ufw.conf', '/etc/ufw/before.rules',
-         '/etc/ufw/before6.rules', '/etc/ufw/after.rules', '/etc/ufw/after6.rules',
-         '/etc/ufw/user.rules', '/etc/ufw/user6.rules')
-HELPER = '/usr/local/bin/ufw-docker'
-BLOCK = re.compile(r'^# BEGIN UFW AND DOCKER\n.*?^# END UFW AND DOCKER\n?', re.M | re.S)
-
-
-def _helper_lines(ipv6):
-    prefix = 'ufw6' if ipv6 else 'ufw'
-    subnets = ('fd00::/8',) if ipv6 else ('10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16')
-    return ['# BEGIN UFW AND DOCKER', '*filter', ':' + prefix + '-user-forward - [0:0]',
-            ':' + prefix + '-docker-logging-deny - [0:0]', ':DOCKER-USER - [0:0]',
-            '-A DOCKER-USER -j ' + prefix + '-user-forward',
-            '-A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j RETURN',
-            '-A DOCKER-USER -m conntrack --ctstate INVALID -j DROP',
-            '-A DOCKER-USER -i docker0 -o docker0 -j ACCEPT',
-            *['-A DOCKER-USER -j RETURN -s ' + subnet for subnet in subnets],
-            *['-A DOCKER-USER -j ' + prefix + '-docker-logging-deny -m conntrack --ctstate NEW -d ' + subnet
-              for subnet in subnets],
-            '-A DOCKER-USER -j RETURN',
-            '-A ' + prefix + '-docker-logging-deny -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW DOCKER BLOCK] "',
-            '-A ' + prefix + '-docker-logging-deny -j DROP', 'COMMIT', '# END UFW AND DOCKER']
-
-
-def _runtime_rule(line):
-    tokens = shlex.split(line)
-    if len(tokens) % 2:
-        raise SecurityError('Unsupported runtime firewall rule')
-    pairs = []
-    for key, value in zip(tokens[::2], tokens[1::2]):
-        if key in ('-s', '-d'):
-            value = str(ipaddress.ip_network(value, strict=False))
-        elif key == '--ctstate':
-            value = ','.join(sorted(value.split(',')))
-        pairs.append((key, value))
-    return sorted(pairs)
-
-
-class SecurityError(RuntimeError):
-    pass
-
-
-class Security:
-    def __init__(self, host):
-        self.host = host
-
-    def preflight(self):
-        services = {name: self._service(name) for name in SERVICES}
-        for name, state in services.items():
-            if (state.get('LoadState') != 'loaded' or state.get('UnitFileState') not in ('enabled', 'disabled')
-                    or state.get('ActiveState') not in ('active', 'inactive')
-                    or state.get('DropInPaths') or not self._supported_unit(name, state.get('FragmentPath'))):
-                raise SecurityError('Unsupported or masked security service: ' + name)
-        if self._digest(self._path(HELPER).read_bytes()) != HELPER_SHA256:
-            raise SecurityError('The ufw-docker helper does not match the pinned baseline')
-        files = self._files()
-        active = self._active()
-        pristine = False
-        if not active:
-            output = self.host.command(['dpkg-query', '-W', '-f=${Conffiles}', 'ufw']).stdout
-            conffiles = dict(re.findall(r'^\s*(/\S+)\s+([a-f0-9]{32})(?:\s|$)', output, re.M))
-            pristine = (hashlib.md5(base64.b64decode(files['/etc/default/ufw']['data'])).hexdigest()
-                        == conffiles.get('/etc/default/ufw'))
-            manifest = dict((name, digest) for digest, name in re.findall(
-                r'^([a-f0-9]{32})\s+(\S+)\s*$', self._text('/var/lib/dpkg/info/ufw.md5sums'), re.M))
-            # Debian's postinst copies/ucf-installs these files; they are not dpkg conffiles.
-            for name in FILES[1:]:
-                filename = name.rsplit('/', 1)[1]
-                template = '/usr/share/ufw/' + ('' if filename == 'ufw.conf' else 'iptables/') + filename
-                data = self._path(template).read_bytes()
-                pristine = pristine and (hashlib.md5(data).hexdigest() == manifest.get(template.lstrip('/'))
-                                         and data == base64.b64decode(files[name]['data']))
-            if not pristine:
-                raise SecurityError('Inactive UFW is not provably pristine; reconcile its existing policy first')
-        texts = [self._text('/etc/ufw/' + name) for name in ('after.rules', 'after6.rules')]
-        present = any('UFW AND DOCKER' in text for text in texts)
-        if present:
-            self._check_helper_files()
-            runtime = None
-        else:
-            runtime = {executable: self._helper_runtime(executable) for executable in ('iptables', 'ip6tables')}
-            for chains in runtime.values():
-                self._validate_helper_before(chains)
-        return {'version': 1, 'pristine': pristine, 'active': active, 'services': services,
-                'files': files, 'helper_present': present, 'helper_runtime_before': runtime,
-                'ipv6': bool(re.search(r'^IPV6\s*=\s*[\"\x27]?yes', self._text('/etc/default/ufw'), re.M))}
-
-    def _supported_unit(self, name, path):
-        if path in ('/usr/lib/systemd/system/' + name + '.service', '/lib/systemd/system/' + name + '.service'):
-            return True
-        if name != 'crowdsec-firewall-bouncer' or path != '/etc/systemd/system/crowdsec-firewall-bouncer.service':
-            return False
-        output = self.host.command(['dpkg-query', '-W', '-f=${Conffiles}',
-                                    'crowdsec-firewall-bouncer-iptables']).stdout
-        conffiles = dict(re.findall(r'^\s*(/\S+)\s+([a-f0-9]{32})(?:\s|$)', output, re.M))
-        return hashlib.md5(self._path(path).read_bytes()).hexdigest() == conffiles.get(path)
-
-    def _path(self, name):
-        path = self.host.path(name)
-        for parent in (path, *path.parents):
-            if parent.is_symlink():
-                raise SecurityError('Refusing redirected security path: ' + name)
-        if not path.is_file():
-            raise SecurityError('Required security file is missing: ' + name)
-        return path
-
-    def _text(self, name):
-        return self._path(name).read_text()
-
-    @staticmethod
-    def _digest(data):
-        return hashlib.sha256(data).hexdigest()
-
-    def _files(self):
-        return {name: self._file(name) for name in FILES}
-
-    def _file(self, name):
-        path = self._path(name)
-        data, metadata = path.read_bytes(), path.stat()
-        result = {'data': base64.b64encode(data).decode(), 'digest': self._digest(data),
-                  'mode': metadata.st_mode & 0o777, 'uid': metadata.st_uid, 'gid': metadata.st_gid}
-        result['fingerprint'] = self._digest(json.dumps(result, sort_keys=True).encode())
-        return result
-
-    def _service(self, name):
-        output = self.host.command(['systemctl', 'show', name, '-p',
-                                   'LoadState,ActiveState,UnitFileState,FragmentPath,DropInPaths']).stdout
-        return dict(line.split('=', 1) for line in output.splitlines() if '=' in line)
-
-    def _active(self):
-        output = self.host.command(['ufw', 'status']).stdout
-        if 'Status: inactive' in output:
-            return False
-        if 'Status: active' in output:
-            return True
-        raise SecurityError('Cannot determine UFW runtime state')
-
-    def snapshot(self, plan, original_port, client_address):
-        if not isinstance(original_port, int) or not 1 <= original_port <= 65535:
-            raise SecurityError('Invalid bootstrap port')
-        try:
-            address = str(ipaddress.ip_address(client_address))
-        except ValueError:
-            raise SecurityError('Invalid bootstrap peer address') from None
-        if plan != self.preflight():
-            raise SecurityError('Security configuration changed after preflight')
-        if ipaddress.ip_address(address).version == 6 and not plan['ipv6']:
-            raise SecurityError('IPv6 bootstrap access requires enabled UFW IPv6 support')
-        if plan['services']['crowdsec']['ActiveState'] == 'active':
-            self._check_decisions(address)
-        return {'version': 1, 'plan': plan, 'original_port': original_port,
-                'client_address': address, 'token': uuid.uuid4().hex, 'rules': [],
-                'expected': {name: value['fingerprint'] for name, value in plan['files'].items()},
-                'service_changes': [], 'inflight': None, 'phase': 'prepared',
-                'access_recovered': False, 'cleanup': 'not-started', 'helper_blocks': {},
-                'helper_runtime': {'introduced': False, 'restored': []}}
-
-    def activate(self, state, persist):
-        if state['phase'] != 'prepared' or state['inflight']:
-            raise SecurityError('Security activation requires a fresh prepared snapshot')
-        self._assert_files(state)
-        state['phase'] = 'activating'
-        persist(state)
-        self._add_rule(state, state['original_port'], state['client_address'], True, persist)
-        self._add_rule(state, 2222, '0.0.0.0/0', False, persist)
-        if state['plan']['ipv6']:
-            self._add_rule(state, 2222, '::/0', False, persist)
-        if state['plan']['pristine']:
-            for policy, direction in (('deny', 'incoming'), ('allow', 'outgoing'), ('allow', 'routed')):
-                self._mutate(state, ['ufw', 'default', policy, direction], ['/etc/default/ufw'], persist)
-        self._mutate(state, ['ufw', '--force', 'enable'],
-                     ['/etc/ufw/ufw.conf', '/etc/ufw/user.rules', '/etc/ufw/user6.rules'], persist)
-        if not state['plan']['helper_present']:
-            if any(self._helper_runtime(executable) != before for executable, before in
-                   state['plan']['helper_runtime_before'].items()):
-                raise SecurityError('Docker firewall rules changed before helper installation')
-            state['helper_runtime']['introduced'] = True
-            persist(state)
-            names = ['/etc/ufw/after.rules', '/etc/ufw/after6.rules']
-            self._mutate(state, [HELPER, 'install'], names, persist)
-            for name in names:
-                blocks = BLOCK.findall(self._text(name))
-                if len(blocks) != 1:
-                    raise SecurityError('ufw-docker did not install a verifiable integration block')
-                state['helper_blocks'][name] = blocks[0]
-            persist(state)
-        self._check_helper_files()
-        self._mutate(state, ['ufw', 'reload'], [], persist)
-        self._enable(state, 'ufw', persist)
-        self.host.command(['crowdsec', '-t', '-c', '/etc/crowdsec/config.yaml'])
-        self._enable(state, 'crowdsec', persist)
-        self.host.command(['cscli', 'lapi', 'status'])
-        self._check_decisions(state['client_address'])
-        self._enable(state, 'crowdsec-firewall-bouncer', persist)
-        state['phase'] = 'active'
-        persist(state)
-        self.verify(state)
-
-    def finalize(self, state, persist):
-        self.verify(state)
-        for rule in state['rules']:
-            if rule['temporary'] and rule['owned']:
-                self._delete_rule(state, rule, persist)
-        state['phase'] = 'final'
-        persist(state)
-        self.verify(state)
-
-    def verify(self, state):
-        if state['phase'] not in ('active', 'final') or state['inflight']:
-            raise SecurityError('Security activation is incomplete')
-        self._assert_files(state)
-        if not self._active():
-            raise SecurityError('UFW is not active')
-        for name in SERVICES:
-            service = self._service(name)
-            if service.get('ActiveState') != 'active' or service.get('UnitFileState') != 'enabled':
-                raise SecurityError('Security service is not active and enabled: ' + name)
-        self.host.command(['cscli', 'lapi', 'status'])
-        self._check_decisions(state['client_address'])
-        self._check_helper_files()
-        self._check_helper_runtime(state['plan']['ipv6'])
-        for rule in state['rules']:
-            if not (rule['temporary'] and state['phase'] == 'final') and self._rule_comment(rule) is None:
-                raise SecurityError('Required SSH firewall allowance is missing')
-
-    def _check_helper_files(self):
-        for ipv6, name in ((False, 'after.rules'), (True, 'after6.rules')):
-            text = self._text('/etc/ufw/' + name)
-            blocks = BLOCK.findall(text)
-            if (len(blocks) != 1 or text.count('UFW AND DOCKER') != 2
-                    or [line.strip() for line in blocks[0].splitlines() if line.strip()] != _helper_lines(ipv6)):
-                raise SecurityError('Existing ufw-docker integration is custom or incomplete')
-
-    def _check_helper_runtime(self, ipv6):
-        for version in (False, True) if ipv6 else (False,):
-            executable = 'ip6tables' if version else 'iptables'
-            prefix = 'ufw6' if version else 'ufw'
-            for chain in ('DOCKER-USER', prefix + '-docker-logging-deny'):
-                result = self.host.command([executable, '-S', chain], False)
-                expected = ['-N ' + chain] + [line for line in _helper_lines(version)
-                                              if line.startswith('-A ' + chain + ' ')]
-                try:
-                    matches = (not result.returncode and [_runtime_rule(line) for line in result.stdout.splitlines()]
-                               == [_runtime_rule(line) for line in expected])
-                except ValueError:
-                    matches = False
-                if not matches:
-                    raise SecurityError('ufw-docker runtime chain is missing or changed: ' + executable + ' ' + chain)
-            if self.host.command([executable, '-C', 'FORWARD', '-j', 'DOCKER-USER'], False).returncode:
-                raise SecurityError('Docker forwarding hook is missing: ' + executable)
-
-    def _helper_runtime(self, executable):
-        logging = ('ufw6' if executable == 'ip6tables' else 'ufw') + '-docker-logging-deny'
-        chains = {'DOCKER-USER': None, logging: None}
-        for line in self.host.command([executable, '-S']).stdout.splitlines():
-            tokens = shlex.split(line)
-            if len(tokens) < 2 or tokens[1] not in chains:
-                continue
-            if tokens[0] == '-N' and len(tokens) == 2 and chains[tokens[1]] is None:
-                chains[tokens[1]] = [line]
-            elif tokens[0] == '-A' and chains[tokens[1]] is not None:
-                chains[tokens[1]].append(line)
-            else:
-                raise SecurityError('Unsupported Docker firewall chain declaration')
-        return chains
-
-    @staticmethod
-    def _validate_helper_before(chains):
-        if (not isinstance(chains, dict) or len(chains) != 2 or 'DOCKER-USER' not in chains
-                or chains['DOCKER-USER'] not in (None, ['-N DOCKER-USER'],
-                                                ['-N DOCKER-USER', '-A DOCKER-USER -j RETURN'])
-                or any(value is not None for name, value in chains.items() if name != 'DOCKER-USER')):
-            raise SecurityError('Unmanaged Docker firewall rules require reconciliation')
-
-    def _restore_helper_runtime(self, state, persist):
-        if state['plan']['helper_present']:
-            return
-        before = state['plan'].get('helper_runtime_before')
-        progress = state.get('helper_runtime')
-        if (not isinstance(before, dict) or set(before) != {'iptables', 'ip6tables'}
-                or not isinstance(progress, dict) or not isinstance(progress.get('introduced'), bool)
-                or not isinstance(progress.get('restored'), list)):
-            raise SecurityError('Missing Docker firewall recovery snapshot; reconcile explicitly')
-        for executable, original in before.items():
-            self._validate_helper_before(original)
-            current = self._helper_runtime(executable)
-            if set(original) != set(current):
-                raise SecurityError('Unsupported Docker firewall recovery snapshot')
-            if current != original:
-                ipv6 = executable == 'ip6tables'
-                expected = {name: ['-N ' + name] + [line for line in _helper_lines(ipv6)
-                            if line.startswith('-A ' + name + ' ')] for name in current}
-                try:
-                    owned = (progress['introduced'] and (not ipv6 or state['plan']['ipv6'])
-                             and all(current[name] is not None and
-                                     [_runtime_rule(line) for line in current[name]] ==
-                                     [_runtime_rule(line) for line in expected[name]] for name in current))
-                except (ValueError, SecurityError):
-                    owned = False
-                if not owned:
-                    raise SecurityError('Docker firewall runtime changed; preserve it for reconciliation')
-                logging = ('ufw6' if ipv6 else 'ufw') + '-docker-logging-deny'
-                commands = ['*filter', '-F DOCKER-USER', *(original['DOCKER-USER'] or [])[1:],
-                            '-F ' + logging, '-X ' + logging]
-                if original['DOCKER-USER'] is None:
-                    commands.append('-X DOCKER-USER')
-                commands.append('COMMIT')
-                progress['restoring'] = executable
-                persist(state)
-                with tempfile.NamedTemporaryFile(mode='w', prefix='apex-firewall-', suffix='.rules') as stream:
-                    stream.write('\n'.join(commands) + '\n')
-                    stream.flush()
-                    if self._helper_runtime(executable) != current:
-                        raise SecurityError('Docker firewall runtime changed immediately before restoration')
-                    self.host.command([executable + '-restore', '--wait', '10', '--noflush', stream.name])
-                if self._helper_runtime(executable) != original:
-                    raise SecurityError('Docker firewall restoration did not converge')
-            if executable not in progress['restored']:
-                progress['restored'].append(executable)
-            progress.pop('restoring', None)
-            persist(state)
-
-    def restore(self, state, persist):
-        state['cleanup'] = 'pending'
-        state['access_recovered'] = False
-        persist(state)
-        errors = []
-        for change in reversed(state['service_changes']):
-            name, field = change['service'], change['field']
-            if name == 'ufw' and field == 'ActiveState' and state['plan']['active']:
-                continue
-            try:
-                current = self._service(name)
-                if any(current.get(key) != change['metadata'].get(key) for key in change['metadata']):
-                    raise SecurityError('Service unit changed during activation')
-                owned_start = (field == 'ActiveState' and change['before'] == 'inactive'
-                               and change['after'] == 'active' and current[field] in
-                               ('active', 'inactive', 'activating', 'deactivating', 'failed', 'reloading'))
-                # Stop even when inactive: systemd may still have a queued start job.
-                if current[field] == change['after'] or owned_start:
-                    actions = {'active': 'start', 'inactive': 'stop', 'enabled': 'enable', 'disabled': 'disable'}
-                    change['status'] = 'restoring'
-                    persist(state)
-                    self.host.command(['systemctl', actions[change['before']], name])
-                    stopped = self._service(name)
-                    if any(stopped.get(key) != change['metadata'].get(key) for key in change['metadata']):
-                        raise SecurityError('Service unit changed during restoration')
-                    if owned_start and stopped[field] == 'failed':
-                        self.host.command(['systemctl', 'reset-failed', name])
-                elif current[field] != change['before']:
-                    raise SecurityError('Service state changed outside this transition')
-                if self._service(name)[field] != change['before']:
-                    raise SecurityError('Service state restoration did not converge')
-                change['status'] = 'restored'
-                persist(state)
-            except Exception:
-                errors.append('service:' + name + ':' + field)
-        ambiguous = []
-        for name in FILES:
-            try:
-                if self._file(name)['fingerprint'] not in (state['plan']['files'][name]['fingerprint'], state['expected'][name]):
-                    ambiguous.append(name)
-            except Exception:
-                ambiguous.append(name)
-        pending = bool(ambiguous or state['inflight'] or errors)
-        original_active = state['plan']['active']
-        # Recover runtime access even when an unrelated edit prevents file cleanup.
-        try:
-            if not original_active:
-                try:
-                    self._path('/etc/ufw/ufw.conf')
-                except SecurityError:
-                    # A service stop removes runtime rules without following the replaced enable file.
-                    self.host.command(['systemctl', 'stop', 'ufw'])
-                else:
-                    self.host.command(['ufw', 'disable'])
-                state['access_recovered'] = not self._active()
-            elif pending:
-                self._retain_bootstrap(state, persist)
-                if not self._active():
-                    self.host.command(['ufw', '--force', 'enable'])
-                state['access_recovered'] = self._active()
-        except Exception:
-            errors.append('bootstrap-firewall')
-        for name, previous in state['plan']['files'].items():
-            try:
-                path = self._path(name)
-                content = path.read_bytes()
-                fingerprint = self._file(name)['fingerprint']
-                if fingerprint == previous['fingerprint']:
-                    continue
-                if fingerprint == state['expected'][name] and not (original_active and pending and name.endswith(('user.rules', 'user6.rules'))):
-                    atomic(path, base64.b64decode(previous['data']), previous['mode'])
-                    os.chown(path, previous['uid'], previous['gid'])
-                    continue
-                block = state['helper_blocks'].get(name)
-                if block and content.decode().count(block) == 1:
-                    metadata = path.stat()
-                    atomic(path, content.replace(block.encode(), b'', 1), metadata.st_mode & 0o777)
-                    os.chown(path, metadata.st_uid, metadata.st_gid)
-                else:
-                    errors.append('file:' + name)
-            except Exception:
-                errors.append('file:' + name)
-        # Rule removal is ownership-checked independently of whole-file snapshots.
-        for rule in state['rules']:
-            if rule['owned'] and not (original_active and pending and rule['temporary']):
-                try:
-                    if self._rule_comment(rule) == rule['comment']:
-                        self.host.command(self._rule_command(rule, delete=True))
-                    elif self._rule_comment(rule) is not None:
-                        errors.append('rule-ownership')
-                except Exception:
-                    errors.append('rule-cleanup')
-        try:
-            self._restore_helper_runtime(state, persist)
-        except Exception:
-            errors.append('docker-firewall-runtime')
-        try:
-            if original_active:
-                if errors or pending:
-                    self._retain_bootstrap(state, persist)
-                else:
-                    self.host.command(['ufw', 'reload'])
-                state['access_recovered'] = self._active()
-            else:
-                # Restoring the former ufw.conf must not reactivate the runtime.
-                state['access_recovered'] = not self._active()
-        except Exception:
-            errors.append('bootstrap-firewall')
-        if not state['access_recovered']:
-            errors.append('bootstrap-firewall')
-        if not state['plan']['helper_present']:
-            try:
-                if any(self._helper_runtime(executable) != before for executable, before in
-                       state['plan']['helper_runtime_before'].items()):
-                    raise SecurityError('Docker firewall restoration changed before cleanup completed')
-            except Exception:
-                errors.append('docker-firewall-runtime')
-        for change in state['service_changes']:
-            name, field = change['service'], change['field']
-            if name == 'ufw' and field == 'ActiveState' and original_active:
-                continue
-            try:
-                current = self._service(name)
-                if (current[field] != change['before'] or any(current.get(key) != value
-                                                            for key, value in change['metadata'].items())):
-                    raise SecurityError('Service restoration changed before cleanup completed')
-            except Exception:
-                errors.append('service:' + name + ':' + field)
-        if errors or pending:
-            state['diagnostics'] = sorted(set(errors + ['file:' + name for name in ambiguous]
-                                              + (['interrupted-mutation'] if state['inflight'] else [])))
-            persist(state)
-            raise SecurityError('Security cleanup pending; bootstrap firewall recovery=' + str(state['access_recovered'])
-                                + '; inspect the root-owned recovery record')
-        state.update(cleanup='complete', phase='restored', inflight=None, diagnostics=[])
-        persist(state)
-
-    def _assert_files(self, state):
-        if any(value['fingerprint'] != state['expected'][name] for name, value in self._files().items()):
-            raise SecurityError('Security files changed outside this transition; recovery remains required')
-
-    def _mutate(self, state, args, changed, persist):
-        self._assert_files(state)
-        state['inflight'] = {'command': args[:2], 'files': changed}
-        persist(state)
-        self.host.command(args)
-        current = self._files()
-        for name in changed:
-            state['expected'][name] = current[name]['fingerprint']
-        if any(value['fingerprint'] != state['expected'][name] for name, value in current.items()):
-            raise SecurityError('Concurrent security file change during activation')
-        state['inflight'] = None
-        persist(state)
-
-    def _enable(self, state, name, persist):
-        for field, before, after, operation in (
-                ('UnitFileState', 'disabled', 'enabled', 'enable'),
-                ('ActiveState', 'inactive', 'active', 'start')):
-            current = self._service(name)
-            if current[field] == after:
-                continue
-            if current[field] != before:
-                raise SecurityError('Security service changed before activation: ' + name)
-            metadata = {key: current.get(key) for key in ('LoadState', 'FragmentPath', 'DropInPaths')}
-            if any(value != state['plan']['services'][name].get(key) for key, value in metadata.items()):
-                raise SecurityError('Security service unit changed before activation: ' + name)
-            change = {'service': name, 'field': field, 'before': before, 'after': after,
-                      'metadata': metadata, 'status': 'pending'}
-            state['service_changes'].append(change)
-            persist(state)
-            self.host.command(['systemctl', operation, name])
-            if self._service(name)[field] != after:
-                raise SecurityError('Security service did not activate: ' + name)
-            change['status'] = 'applied'
-            persist(state)
-
-    def _check_decisions(self, peer):
-        self.host.command(['cscli', 'lapi', 'status'])
-        try:
-            alerts = json.loads(self.host.command(
-                ['cscli', 'decisions', 'list', '--all', '--limit', '0', '--no-simu', '-o', 'json']).stdout)
-            if not isinstance(alerts, list):
-                raise ValueError()
-            address = ipaddress.ip_address(peer)
-            for alert in alerts:
-                if not isinstance(alert, dict) or not isinstance(alert.get('decisions'), list):
-                    raise ValueError()
-                for decision in alert['decisions']:
-                    if (not isinstance(decision, dict) or any(not isinstance(decision.get(key), str)
-                            or not decision[key] for key in ('scope', 'value', 'type'))
-                            or decision.get('simulated', False) is not False):
-                        raise ValueError()
-                    if decision['scope'].lower() not in ('ip', 'range'):
-                        continue
-                    network = ipaddress.ip_network(decision['value'], strict=False)
-                    if address in network:
-                        raise SecurityError('An existing CrowdSec decision covers the bootstrap peer')
-        except (ValueError, TypeError, KeyError):
-            raise SecurityError('Cannot verify CrowdSec decisions safely') from None
-
-    @staticmethod
-    def _rule_file(rule):
-        return '/etc/ufw/user6.rules' if ipaddress.ip_network(rule['source'], strict=False).version == 6 else '/etc/ufw/user.rules'
-
-    def _rule_comment(self, rule):
-        source = ipaddress.ip_network(rule['source'], strict=False)
-        any_address = '::/0' if source.version == 6 else '0.0.0.0/0'
-        for line in self._text(self._rule_file(rule)).splitlines():
-            if not line.startswith('### tuple ### '):
-                continue
-            parts = line[len('### tuple ### '):].split()
-            if len(parts) < 7 or parts[:5] != ['allow', 'tcp', str(rule['port']), any_address, 'any'] or parts[6] != 'in':
-                continue
-            try:
-                if ipaddress.ip_network(parts[5], strict=False) != source:
-                    continue
-                comment = next((p[8:] for p in parts[7:] if p.startswith('comment=')), '')
-                return bytes.fromhex(comment).decode() if comment else ''
-            except (ValueError, UnicodeError):
-                raise SecurityError('Cannot prove ownership of an existing SSH firewall rule') from None
-        return None
-
-    @staticmethod
-    def _rule_command(rule, delete=False):
-        command = ['ufw', '--force', 'delete'] if delete else (['ufw', 'prepend'] if rule['temporary'] else ['ufw'])
-        return command + ['allow', 'proto', 'tcp', 'from', rule['source'], 'to', 'any', 'port', str(rule['port']),
-                          'comment', rule['comment']]
-
-    def _add_rule(self, state, port, source, temporary, persist):
-        rule = {'port': port, 'source': source, 'temporary': temporary,
-                'comment': 'apex-init-' + state['token'] + ('-bootstrap' if temporary else '-final')}
-        rule['owned'] = self._rule_comment(rule) is None
-        state['rules'].append(rule)
-        persist(state)
-        if rule['owned']:
-            self._mutate(state, self._rule_command(rule), ['/etc/ufw/user.rules', '/etc/ufw/user6.rules'], persist)
-
-    def _delete_rule(self, state, rule, persist):
-        if self._rule_comment(rule) != rule['comment']:
-            raise SecurityError('Owned SSH rule changed; preserve it for explicit reconciliation')
-        self._mutate(state, self._rule_command(rule, delete=True), ['/etc/ufw/user.rules', '/etc/ufw/user6.rules'], persist)
-
-    def _retain_bootstrap(self, state, persist):
-        rule = next((rule for rule in state['rules'] if rule['temporary']), None)
-        if rule is None:
-            raise SecurityError('No recorded bootstrap allowance exists')
-        if self._rule_comment(rule) is None:
-            rule['owned'] = True
-            state['bootstrap_retained'] = True
-            persist(state)
-            self.host.command(self._rule_command(rule))

ci/scripts/init-node/provisioning/seed.sh

@@ -1,87 +0,0 @@
-#!/usr/bin/env bash
-# Only the parser runtime may be installed before semantic input validation.
-set -euo pipefail
-export LC_ALL=C DEBIAN_FRONTEND=noninteractive
-[[ $(id -u) == 0 ]] || { echo 'APEX seed requires root.' >&2; exit 1; }
-[[ -r /etc/os-release ]] || { echo 'APEX seed requires Debian 13 amd64.' >&2; exit 1; }
-. /etc/os-release
-[[ ${ID:-} == debian && ${VERSION_ID:-} == 13 && $(dpkg --print-architecture) == amd64 ]] || {
-    echo 'APEX seed requires Debian 13 amd64.' >&2; exit 1;
-}
-
-policy=/usr/sbin/policy-rc.d
-state=/usr/sbin/.apex-policy-rc.d
-for path in /var /var/lib /var/lib/apex-init; do
-    [[ ! -L $path ]] || { echo 'Refusing redirected initialization state.' >&2; exit 1; }
-done
-mkdir -p -m 0700 /var/lib/apex-init
-[[ $(stat -c %u /var/lib/apex-init) == 0 ]] || {
-    echo 'Initialization state must be root-owned.' >&2; exit 1;
-}
-(( (8#$(stat -c %a /var/lib/apex-init) & 022) == 0 )) || {
-    echo 'Initialization state must not be writable by other users.' >&2; exit 1;
-}
-[[ ! -L /var/lib/apex-init/init.lock ]] || { echo 'Refusing redirected initialization lock.' >&2; exit 1; }
-exec 8>/var/lib/apex-init/init.lock
-flock -n 8 || { echo 'Another APEX initialization is running.' >&2; exit 1; }
-mkdir -p /run/lock
-exec 9>/run/lock/apex-package-policy.lock
-flock -n 9 || { echo 'Another APEX package operation is running.' >&2; exit 1; }
-restore_policy() {
-    [[ -d $state ]] || return 0
-    [[ ! -L $state && $(stat -c %u "$state") == 0 ]] || {
-        echo 'Refusing untrusted service policy recovery state.' >&2; return 1;
-    }
-    if [[ -e $state/ready && ( -e $policy || -L $policy ) ]]; then
-        if ! cmp -s "$policy" <(printf '#!/bin/sh\nexit 101\n') &&
-           ! cmp -s "$policy" "$state/original"; then
-            echo 'Service policy changed outside APEX; reconcile it explicitly.' >&2
-            return 1
-        fi
-    fi
-    if [[ -e $state/original || -L $state/original ]]; then
-        rm -f -- "$state/restoring"
-        cp -a -- "$state/original" "$state/restoring"
-        mv -Tf -- "$state/restoring" "$policy"
-    elif [[ -e $state/absent ]]; then
-        rm -f -- "$policy"
-    elif [[ -e $state/ready ]]; then
-        echo 'APEX service policy recovery is incomplete; inspect /usr/sbin/.apex-policy-rc.d.' >&2
-        return 1
-    fi
-    rm -rf -- "$state"
-}
-
-# Release the shared host lock on exit; the target runner reacquires it later.
-restore_policy
-missing=()
-for package in python3 python3-yaml; do
-    status=$(dpkg-query -W -f='${Status}' "$package" 2>/dev/null || true)
-    [[ $status == *' ok installed' ]] || missing+=("$package")
-done
-if (( ${#missing[@]} )); then
-    mkdir -m 0700 -- "$state"
-    trap restore_policy EXIT
-    trap 'exit 130' INT
-    trap 'exit 143' TERM
-    if [[ -e $policy || -L $policy ]]; then
-        cp -a -- "$policy" "$state/original"
-    else
-        touch "$state/absent"
-    fi
-    touch "$state/ready"
-    printf '#!/bin/sh\nexit 101\n' > "$state/guard"
-    chmod 0755 "$state/guard"
-    mv -Tf -- "$state/guard" "$policy"
-    apt=(apt-get -o DPkg::Lock::Timeout=60 -o Acquire::Retries=2
-         -o Acquire::http::Timeout=30 -o Acquire::https::Timeout=30)
-    timeout 300 "${apt[@]}" update --error-on=any
-    install=("${apt[@]}" install -y --no-install-recommends --no-upgrade --no-remove "${missing[@]}")
-    plan=$(timeout 300 "${install[@]}" --simulate)
-    if grep -Eq '^Remv |^Inst [^ ]+ \[' <<< "$plan"; then
-        echo 'Parser installation requires a package upgrade/removal; reconcile it explicitly.' >&2
-        exit 1
-    fi
-    "${install[@]}"
-fi
-python3 -c 'import yaml'

ci/scripts/init-node/provisioning/state.py

@@ -1,128 +0,0 @@
-"""Durable enrollment records and writes that refuse redirected paths."""
-from __future__ import annotations
-
-import contextlib
-import fcntl
-import json
-import os
-from pathlib import Path
-import stat
-import tempfile
-import time
-
-from . import ENROLLMENT_INTERFACE
-
-
-class ProvisionError(RuntimeError):
-    pass
-
-
-def safe_path(path):
-    path = Path(path).absolute()
-    for part in (*reversed(path.parents), path):
-        try:
-            info = part.lstat()
-        except FileNotFoundError:
-            continue
-        if stat.S_ISLNK(info.st_mode):
-            raise ProvisionError('managed path contains a symbolic link')
-    return path
-
-
-def atomic_write(path, content, mode=0o600, uid=None, gid=None):
-    path = safe_path(path)
-    path.parent.mkdir(parents=True, exist_ok=True)
-    if path.exists() and (not path.is_file() or path.stat().st_nlink != 1):
-        raise ProvisionError('managed file is not an ordinary unlinked file')
-    data = content.encode() if isinstance(content, str) else content
-    fd, temporary = tempfile.mkstemp(prefix='.apex-', dir=path.parent)
-    try:
-        with os.fdopen(fd, 'wb') as stream:
-            os.fchmod(stream.fileno(), mode)
-            if uid is not None:
-                os.fchown(stream.fileno(), uid, gid)
-            stream.write(data)
-            stream.flush()
-            os.fsync(stream.fileno())
-        os.replace(temporary, path)
-        directory = os.open(path.parent, os.O_RDONLY)
-        try:
-            os.fsync(directory)
-        finally:
-            os.close(directory)
-    finally:
-        if os.path.exists(temporary):
-            os.unlink(temporary)
-
-
-@contextlib.contextmanager
-def exclusive_lock(path):
-    path = safe_path(path)
-    path.parent.mkdir(parents=True, exist_ok=True)
-    fd = os.open(path, os.O_CREAT | os.O_RDWR | os.O_NOFOLLOW, 0o600)
-    try:
-        try:
-            fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
-        except BlockingIOError:
-            raise ProvisionError('another initialization is running; inspect status before retrying') from None
-        yield
-    finally:
-        os.close(fd)
-
-
-def check_heartbeat(path, now=None):
-    if path is None:
-        return
-    try:
-        age = (time.time() if now is None else now) - safe_path(path).stat().st_mtime
-    except FileNotFoundError:
-        raise ProvisionError('coordinator heartbeat is missing; stopping before the next stage') from None
-    if not 0 <= age < 90:
-        raise ProvisionError('coordinator heartbeat expired; stopping before the next stage')
-
-
-class Journal:
-    def __init__(self, path, identity, revision, bundle_digest):
-        self.path = safe_path(path)
-        if self.path.exists():
-            try:
-                self.data = json.loads(self.path.read_text())
-            except (ValueError, OSError):
-                raise ProvisionError('invalid enrollment journal; reconcile it before retrying') from None
-            if self.data.get('enrollment_interface') != ENROLLMENT_INTERFACE:
-                raise ProvisionError('recorded enrollment interface is missing or unsupported')
-            if self.data.get('identity') != identity:
-                raise ProvisionError('enrollment identity conflicts with the recorded target')
-            if self.data.get('bundle_digest') != bundle_digest:
-                raise ProvisionError('provisioner bundle changed; resume using the recorded bundle')
-        else:
-            self.data = dict(identity=identity, revision=revision, bundle_digest=bundle_digest,
-                             baseline_revision=revision, enrollment_interface=ENROLLMENT_INTERFACE,
-                             status='running', stages={})
-
-    def save(self):
-        atomic_write(self.path, json.dumps(self.data, sort_keys=True, indent=2) + '\n')
-
-    def stage(self, name, operation, heartbeat=None):
-        try:
-            check_heartbeat(heartbeat)
-        except ProvisionError:
-            self.data['status'] = 'stopped'
-            self.data['failed_stage'] = name
-            self.save()
-            raise
-        self.data['status'] = 'running'
-        self.data['stages'][name] = {'status': 'running'}
-        self.save()
-        try:
-            result = operation()
-        except Exception:
-            self.data['status'] = 'failed'
-            self.data['failed_stage'] = name
-            self.data['stages'][name] = {'status': 'failed'}
-            self.save()
-            raise
-        self.data['stages'][name] = {'status': 'done', 'result': result}
-        self.data.pop('failed_stage', None)
-        self.save()
-        return result

ci/scripts/init-node/provisioning/storage.py

@@ -1,509 +0,0 @@
-"""Conservative target-side tier and optional swap reconciliation.
-
-The caller holds the initialization lock and supplies a zero-argument atomic
-journal writer. Ambiguous device or pool ownership always requires remediation.
-"""
-from __future__ import annotations
-
-import hashlib
-import json
-import os
-from pathlib import Path
-import re
-import stat
-import subprocess
-import tempfile
-
-from .contract import Enrollment
-
-
-SRV = Path('/srv')
-SWAPFILE = Path('/swapfile')
-FSTAB = Path('/etc/fstab')
-MEMINFO = Path('/proc/meminfo')
-SWAPS = Path('/proc/swaps')
-_OWNER = 'org.apex:enrollment'
-_GIB = 1024 ** 3
-
-
-class StorageError(RuntimeError):
-    """Storage could not be positively identified or safely reconciled."""
-
-
-def _require(condition, message):
-    if not condition:
-        raise StorageError(message)
-
-
-def _run(argv, *, full_vdev_paths=False):
-    environment = {**os.environ, 'LC_ALL': 'C'}
-    if full_vdev_paths:
-        environment.update(ZPOOL_VDEV_NAME_PATH='1', ZPOOL_VDEV_NAME_FOLLOW_LINKS='1')
-        environment.pop('ZPOOL_VDEV_NAME_GUID', None)
-    try:
-        result = subprocess.run(argv, check=False, capture_output=True, text=True,
-                                timeout=300, env=environment)
-    except (OSError, subprocess.TimeoutExpired):
-        raise StorageError('required storage tool is unavailable or timed out: ' + argv[0]) from None
-    _require(result.returncode == 0,
-             'storage inspection or operation failed: ' + ' '.join(argv[:2]))
-    return result.stdout
-
-
-def _json(argv, key):
-    try:
-        value = json.loads(_run(argv))[key]
-        _require(isinstance(value, list), 'storage inspection returned an invalid list')
-        return value
-    except (ValueError, KeyError, TypeError):
-        raise StorageError('storage inspection returned unsupported JSON') from None
-
-
-def _flatten(items):
-    for item in items:
-        yield item
-        yield from _flatten(item.get('children', []))
-
-
-def _mounts():
-    return list(_flatten(_json(['findmnt', '--json', '--output', 'TARGET,SOURCE,FSTYPE,MAJ:MIN'],
-                               'filesystems')))
-
-
-def _device_number(path):
-    try:
-        info = os.stat(path)
-    except OSError:
-        raise StorageError('selected device is missing or is not a block device') from None
-    _require(stat.S_ISBLK(info.st_mode), 'selected device is not a block device')
-    return f'{os.major(info.st_rdev)}:{os.minor(info.st_rdev)}'
-
-
-def _inventory():
-    roots = _json(['lsblk', '--json', '--bytes', '--paths', '--output',
-                   'NAME,MAJ:MIN,TYPE,PKNAME,MOUNTPOINTS,FSTYPE,WWN,SERIAL,SIZE,PARTUUID'], 'blockdevices')
-    nodes, edges = {}, {}
-
-    def visit(items, parent=None):
-        for item in items:
-            number = item['maj:min']
-            nodes[number] = item
-            edges.setdefault(number, set())
-            if parent:
-                edges[number].add(parent)
-                edges[parent].add(number)
-            visit(item.get('children', []), number)
-
-    try:
-        visit(roots)
-        by_name = {item['name']: number for number, item in nodes.items()}
-        for number, item in nodes.items():
-            parent = by_name.get(item.get('pkname'))
-            if parent:
-                edges[number].add(parent)
-                edges[parent].add(number)
-    except (KeyError, TypeError):
-        raise StorageError('block-device topology could not be established') from None
-    return nodes, edges
-
-
-def _connected(number, edges):
-    found, pending = set(), [number]
-    while pending:
-        item = pending.pop()
-        if item not in found:
-            found.add(item)
-            pending.extend(edges.get(item, ()))
-    return found
-
-
-def _pools():
-    result = {}
-    for line in _run(['zpool', 'list', '-H', '-o', 'name,guid,health']).splitlines():
-        fields = line.split('\t')
-        _require(len(fields) == 3 and fields[1].isdigit(), 'active pool identity is ambiguous')
-        result[fields[0]] = {'guid': fields[1], 'health': fields[2]}
-    return result
-
-
-def _members(text, name, imported=True):
-    """Accept only a plain single-device ONLINE pool, never infer topology."""
-    marker = re.search(r'^\s*config:\s*$', text, re.M)
-    _require(marker is not None, 'pool membership is unavailable')
-    lines = text[marker.end():].splitlines()
-    rows = []
-    for line in lines:
-        fields = line.split()
-        if not fields or fields[0] == 'NAME':
-            continue
-        if fields[0] == 'errors:':
-            _require(line.strip() == 'errors: No known data errors', 'pool reports data errors')
-            break
-        _require(len(fields) >= 2 and fields[1] == 'ONLINE',
-                 'pool is not a healthy single-device topology')
-        if imported:
-            _require(len(fields) == 5 and fields[2:] == ['0', '0', '0'],
-                     'pool reports device errors')
-        rows.append(fields[0])
-    _require(len(rows) == 2 and rows[0] == name,
-             'pool must contain exactly one identifiable device')
-    leaf = rows[1]
-    if not imported and re.fullmatch(r'[A-Za-z0-9_.-]+', leaf):
-        leaf = '/dev/' + leaf
-    _require(leaf.startswith('/dev/'), 'pool device path cannot be positively resolved')
-    return leaf
-
-
-def _pool_device(name):
-    return _members(_run(['zpool', 'status', '-P', name], full_vdev_paths=True), name)
-
-
-def _inspect_device(device, mounts, allowed_pool=None, allowed_path=None):
-    number = _device_number(device)
-    nodes, edges = _inventory()
-    _require(number in nodes, 'selected device is absent from the block topology')
-    node = nodes[number]
-    hardware = node.get('wwn') or node.get('serial')
-    partition = None
-    if node.get('type') == 'part':
-        parents = [item for item in nodes.values() if item['name'] == node.get('pkname')]
-        _require(len(parents) == 1 and parents[0]['type'] == 'disk' and node.get('partuuid'),
-                 'selected partition needs a stable partition UUID and identifiable parent disk')
-        hardware = parents[0].get('wwn') or parents[0].get('serial')
-        partition = node['partuuid']
-    _require(isinstance(hardware, str) and bool(hardware.strip()),
-             'selected device needs a stable WWN or hardware serial')
-    _require(node.get('type') in ('disk', 'part') and int(node.get('size') or 0) > 0,
-             'selected device type or capacity is unsupported')
-    related = _connected(number, edges)
-    protected = set()
-    for other, data in nodes.items():
-        for mount in data.get('mountpoints') or []:
-            if mount and mount != allowed_path:
-                protected.add(other)
-    for mount in mounts:
-        if mount['target'] != allowed_path:
-            protected.add(mount.get('maj:min'))
-    active = _pools()
-    for pool in active:
-        leaf = _device_number(_pool_device(pool))
-        if pool != allowed_pool or any(
-                m['source'].split('/')[0] == pool and m['target'] != allowed_path for m in mounts):
-            protected.add(leaf)
-    _require(not (related & protected),
-             'selected device intersects mounted, root, boot, swap, or active pool storage')
-    identity = {'hardware': hardware, 'size': int(node['size']), 'type': node['type']}
-    if partition:
-        identity['partition'] = partition
-    return {'number': number, 'identity': identity, 'related': related, 'node': node, 'pools': active}
-
-
-def _safe_path(path):
-    for parent in reversed((path, *path.parents)):
-        try:
-            info = parent.lstat()
-        except FileNotFoundError:
-            _require(parent == path, 'storage parent directory is missing')
-            continue
-        _require(stat.S_ISDIR(info.st_mode), 'storage path contains a symlink or non-directory')
-
-
-def _directory(path, mounts, pool=None):
-    _safe_path(path)
-    exact = [m for m in mounts if m['target'] == str(path)]
-    _require(not exact or (len(exact) == 1 and pool is not None and
-                          exact[0]['fstype'] == 'zfs' and exact[0]['source'] == pool),
-             'tier path is occupied by an unexpected mount')
-    _require(not any(m['target'].startswith(str(path) + '/') for m in mounts),
-             'tier path contains an unexpected nested mount')
-    created = not path.exists()
-    if created:
-        path.mkdir(mode=0o755)
-        path.chmod(0o755)
-    return created
-
-
-def _directory_identity(path):
-    _safe_path(path)
-    try:
-        info = path.lstat()
-    except FileNotFoundError:
-        raise StorageError('managed directory identity is missing') from None
-    return {'device': info.st_dev, 'inode': info.st_ino}
-
-
-def _verify_ownership_identity(record, path):
-    ownership = record.get('ownership')
-    if ownership:
-        _require(ownership['identity'] == _directory_identity(path),
-                 'managed directory identity changed before ownership handoff')
-    return ownership
-
-
-def _record_pending_ownership(record, path):
-    if not _verify_ownership_identity(record, path):
-        record['ownership'] = {'status': 'pending', 'identity': _directory_identity(path)}
-
-
-def _verify_pool(name, guid, device_info, path, owner, require_owner=True):
-    pools = _pools()
-    _require(name in pools and pools[name] == {'guid': guid, 'health': 'ONLINE'},
-             'pool name, GUID, or health does not match enrollment')
-    _require(_member_matches(_pool_device(name), device_info),
-             'pool device membership does not match the selected device')
-    rows = [line.split('\t') for line in _run(
-        ['zfs', 'list', '-H', '-r', '-o', 'name,mountpoint,type,mounted', name]).splitlines()]
-    _require(len(rows) == 1 and len(rows[0]) == 4 and rows[0][:3] == [name, str(path), 'filesystem'],
-             'pool dataset or mountpoint requires explicit reconciliation')
-    if require_owner:
-        actual = _run(['zfs', 'get', '-H', '-o', 'value', _OWNER, name]).strip()
-        _require(actual == owner, 'pool lacks matching enrollment ownership')
-    return rows[0][3] == 'yes'
-
-
-def _member_matches(member, info):
-    number = _device_number(member)
-    return number == info['number'] or (info['node']['type'] == 'disk' and number in info['related'])
-
-
-def _import_candidate(device, guid):
-    text = _run(['zpool', 'import', '-d', device], full_vdev_paths=True)
-    names = re.findall(r'^\s*pool:\s*(\S+)\s*$', text, re.M)
-    guids = re.findall(r'^\s*id:\s*(\d+)\s*$', text, re.M)
-    states = re.findall(r'^\s*state:\s*(\S+)\s*$', text, re.M)
-    _require(len(names) == 1 and guids == [guid] and states == ['ONLINE'],
-             'import discovery does not match the declared healthy pool GUID')
-    actions = re.findall(r'^\s*action:\s*(.+)$', text, re.M)
-    _require(not re.search(r'^\s*status:', text, re.M) and
-             all(action == 'The pool can be imported using its name or numeric identifier.' for action in actions),
-             'pool import reports ownership or recovery conditions; reconcile manually')
-    return names[0], _members(text, names[0], imported=False)
-
-
-def _block_tier(tier, path, journal, owner, save):
-    record = journal.get(str(tier.number))
-    created = False
-    pool = record.get('pool') if record else None
-    info = _inspect_device(tier.device, _mounts(), pool, str(path) if pool else None)
-    if record:
-        _require(record['device'] == info['identity'], 'selected device identity changed since enrollment')
-        _require(record['status'] in ('intent', 'complete'), 'tier journal status is unsupported')
-    if pool and pool in info['pools']:
-        guid = record.get('guid') or info['pools'][pool]['guid']
-        _verify_pool(pool, guid, info, path, owner)
-    else:
-        _require(not record or record['status'] == 'intent',
-                 'previously enrolled pool is not active; explicit reconciliation is required')
-        if tier.mode == 'create':
-            pool = f'tier-{tier.number}'
-            _require(pool not in info['pools'], 'existing pool name does not prove enrollment ownership')
-            _require(info['node']['type'] == 'disk', 'pool creation requires an unused whole disk')
-            _require(len(info['related']) == 1, 'selected disk has partitions or dependent devices')
-            signatures = _json(['wipefs', '--json', '--output', 'TYPE,UUID,LABEL', tier.device], 'signatures')
-            _require(not signatures and not info['node'].get('fstype'), 'selected device contains existing signatures')
-        else:
-            pool, member = _import_candidate(tier.device, tier.pool_guid)
-            _require(_member_matches(member, info), 'import membership differs from the selected device')
-            _require(pool not in info['pools'], 'pool is already active without matching enrollment ownership')
-        created = _directory(path, _mounts())
-        _require(not path.exists() or not any(path.iterdir()), 'pool mountpoint contains existing data')
-        record = {'status': 'intent', 'device': info['identity'], 'pool': pool,
-                  'ownership_intent': tier.mode == 'create'}
-        journal[str(tier.number)] = record
-        save()
-        fresh = _inspect_device(tier.device, _mounts())
-        _require(fresh['identity'] == info['identity'] and fresh['number'] == info['number'],
-                 'selected device identity changed before pool operation')
-        _directory(path, _mounts())
-        _require(not any(path.iterdir()), 'pool mountpoint acquired existing data before operation')
-        if tier.mode == 'create':
-            _require(not _json(['wipefs', '--json', '--output', 'TYPE,UUID,LABEL', tier.device], 'signatures'),
-                     'selected device acquired signatures before creation')
-            _run(['zpool', 'create', '-o', 'ashift=12', '-O', f'mountpoint={path}',
-                  '-O', f'{_OWNER}={owner}', pool, tier.device])
-        else:
-            _require(_import_candidate(tier.device, tier.pool_guid) == (pool, member),
-                     'pool identity changed before import')
-            _run(['zpool', 'import', '-N', '-d', tier.device, tier.pool_guid])
-        pools = _pools()
-        _require(pool in pools, 'pool operation did not establish an active pool')
-        guid = pools[pool]['guid']
-        _require(tier.mode != 'import' or guid == tier.pool_guid, 'imported pool GUID differs from intent')
-        refreshed = _inspect_device(tier.device, _mounts(), pool, str(path))
-        _require(refreshed['identity'] == info['identity'], 'selected device changed during pool operation')
-        info = refreshed
-        _verify_pool(pool, guid, info, path, owner, require_owner=tier.mode == 'create')
-        if tier.mode == 'import':
-            previous_owner = _run(['zfs', 'get', '-H', '-o', 'value', _OWNER, pool]).strip()
-            _require(previous_owner in ('-', owner), 'imported pool belongs to a different enrollment')
-            _run(['zfs', 'set', f'{_OWNER}={owner}', pool])
-    created = _directory(path, _mounts(), pool) or created
-    if not _verify_pool(pool, guid, info, path, owner):
-        _require(not any(path.iterdir()), 'refusing to hide existing files beneath a pool mount')
-        _run(['zfs', 'mount', pool])
-    _require(any(m['target'] == str(path) and m['source'] == pool and m['fstype'] == 'zfs'
-                 for m in _mounts()), 'pool is not mounted at its enrolled path')
-    record.update(status='complete', guid=guid)
-    if tier.mode == 'create' and record.get('ownership_intent'):
-        _record_pending_ownership(record, path)
-    save()
-    return {'path': str(path), 'mode': tier.mode, 'created': created and tier.mode == 'create',
-            'pool': pool, 'guid': guid}
-
-
-def _persist_swap():
-    info = FSTAB.lstat()
-    _require(stat.S_ISREG(info.st_mode) and info.st_uid == 0, 'fstab must be a root-owned regular file')
-    original = FSTAB.read_text()
-    lines = [line.split() for line in original.splitlines() if line.strip() and not line.lstrip().startswith('#')]
-    matches = [line for line in lines if line[0] == str(SWAPFILE)]
-    if matches:
-        _require(len(matches) == 1 and len(matches[0]) >= 4 and matches[0][2] == 'swap'
-                 and 'noauto' not in matches[0][3].split(','),
-                 'existing swap fstab entry is incompatible')
-        return
-    fd, temporary = tempfile.mkstemp(prefix='.apex-swap-', dir=FSTAB.parent)
-    try:
-        with os.fdopen(fd, 'w') as output:
-            os.fchmod(output.fileno(), stat.S_IMODE(info.st_mode))
-            os.fchown(output.fileno(), info.st_uid, info.st_gid)
-            output.write(original + ('' if original.endswith('\n') else '\n') +
-                         f'{SWAPFILE} none swap sw 0 0\n')
-            output.flush()
-            os.fsync(output.fileno())
-        current = FSTAB.lstat()
-        _require(all(getattr(current, key) == getattr(info, key) for key in
-                     ('st_dev', 'st_ino', 'st_size', 'st_mtime_ns', 'st_ctime_ns')),
-                 'fstab changed during swap setup')
-        os.replace(temporary, FSTAB)
-    finally:
-        if os.path.exists(temporary):
-            os.unlink(temporary)
-
-
-def _active_swaps():
-    lines = SWAPS.read_bytes().splitlines()
-    _require(bool(lines) and lines[0].split() == [b'Filename', b'Type', b'Size', b'Used', b'Priority'],
-             'kernel swap inventory header is invalid')
-    paths = []
-    for line in lines[1:]:
-        fields = line.split()
-        _require(len(fields) == 5 and fields[0].startswith(b'/') and fields[1] in (b'file', b'partition')
-                 and fields[2].isdigit() and fields[3].isdigit()
-                 and re.fullmatch(rb'-?[0-9]+', fields[4]) is not None,
-                 'kernel swap inventory entry is invalid')
-        _require(re.search(rb'\\(?![0-3][0-7]{2})', fields[0]) is None,
-                 'kernel swap inventory path escaping is invalid')
-        decoded = re.sub(rb'\\([0-3][0-7]{2})', lambda match: bytes([int(match.group(1), 8)]), fields[0])
-        paths.append(os.fsdecode(decoded))
-    return paths
-
-
-def _swap(journal, save):
-    try:
-        active = _active_swaps()
-        if active:
-            return {'status': 'skipped', 'reason': 'existing active swap preserved'}
-        if os.path.lexists(SWAPFILE):
-            return {'status': 'skipped', 'reason': 'existing swapfile preserved; no overwrite or resize'}
-        mounts = _mounts()
-        covering = [m for m in mounts if str(SWAPFILE).startswith(m['target'].rstrip('/') + '/')]
-        _require(bool(covering), 'swap filesystem could not be identified')
-        filesystem = max(covering, key=lambda m: len(m['target']))['fstype']
-        _require(filesystem in ('ext4', 'xfs'), 'optional swap requires a supported ext4 or XFS filesystem')
-        match = re.search(r'^MemTotal:\s+(\d+)\s+kB$', MEMINFO.read_text(), re.M)
-        _require(match is not None, 'RAM size could not be established')
-        size = (4 if int(match.group(1)) * 1024 > 4 * _GIB else 2) * _GIB
-        space = os.statvfs(SWAPFILE.parent)
-        _require(space.f_bavail * space.f_frsize >= size + _GIB,
-                 'insufficient free space for optional swap and reserve')
-        _require(hasattr(os, 'posix_fallocate'), 'safe swap allocation is unavailable')
-        journal['swap'] = {'status': 'intent', 'size': size}
-        save()
-        fd = os.open(SWAPFILE, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600)
-        try:
-            os.posix_fallocate(fd, 0, size)
-            os.fsync(fd)
-        finally:
-            os.close(fd)
-        _run(['mkswap', str(SWAPFILE)])
-        _run(['swapon', str(SWAPFILE)])
-        active = _active_swaps()
-        _require(str(SWAPFILE) in active,
-                 'optional swap activation did not become observable')
-        _persist_swap()
-        return {'status': 'complete', 'size': size}
-    except (StorageError, OSError) as error:
-        return {'status': 'skipped', 'reason': str(error) if isinstance(error, StorageError)
-                else 'optional swap allocation or persistence failed; existing resources preserved'}
-
-
-def _enrollment_identity(spec):
-    return hashlib.sha256(json.dumps(spec.identity(), sort_keys=True,
-                                     separators=(',', ':')).encode()).hexdigest()
-
-
-def acknowledge_storage_ownership(spec: Enrollment, state: dict, paths, save) -> None:
-    """Persist the caller's successful, nonrecursive ownership handoff for paths."""
-    journal = state.get('storage', {})
-    _require(journal.get('identity') == _enrollment_identity(spec),
-             'storage enrollment identity conflicts with ownership acknowledgement')
-    allowed = {str(SRV / f'tier-{tier.number}.{spec.repository.name}'): tier for tier in spec.tiers}
-    acknowledged = []
-    for path in paths:
-        _require(path in allowed, 'ownership acknowledgement path is outside enrollment')
-        tier = allowed[path]
-        record = journal.get('tiers', {}).get(str(tier.number), {})
-        ownership = _verify_ownership_identity(record, Path(path))
-        _require(tier.mode != 'import' and ownership is not None,
-                 'ownership acknowledgement requires a directory created by enrollment')
-        _require(ownership['status'] in ('pending', 'applied'), 'ownership journal status is unsupported')
-        acknowledged.append(ownership)
-    for ownership in acknowledged:
-        ownership['status'] = 'applied'
-    save()
-
-
-def reconcile_storage(spec: Enrollment, state: dict, save) -> dict:
-    """Return ownership_pending paths until the caller acknowledges their chown."""
-    identity = _enrollment_identity(spec)
-    journal = state.setdefault('storage', {'identity': identity, 'tiers': {}})
-    _require(journal.get('identity') == identity, 'storage enrollment identity conflicts with journal')
-    _safe_path(SRV)
-    _require(SRV.exists(), 'storage parent directory is missing')
-    selected = set()
-    for tier in spec.tiers:
-        if tier.device:
-            number = _device_number(tier.device)
-            _require(number not in selected, 'multiple tiers select aliases of the same device')
-            selected.add(number)
-    result = {'tiers': [], 'ownership_pending': []}
-    for tier in spec.tiers:
-        path = SRV / f'tier-{tier.number}.{spec.repository.name}'
-        if tier.mode == 'directory':
-            record = journal['tiers'].get(str(tier.number), {})
-            _verify_ownership_identity(record, path)
-            _safe_path(path)
-            _require(not (record.get('ownership_intent') and not record.get('ownership') and path.exists()),
-                     'directory creation was interrupted before identity was recorded; reconcile manually')
-            if not path.exists():
-                record = {'status': 'intent', 'path': str(path), 'ownership_intent': True}
-                journal['tiers'][str(tier.number)] = record
-                save()
-            created = _directory(path, _mounts())
-            if created:
-                _record_pending_ownership(record, path)
-            entry = {'path': str(path), 'mode': tier.mode, 'created': created}
-            record.update(status='complete', path=str(path))
-            journal['tiers'][str(tier.number)] = record
-            save()
-        else:
-            entry = _block_tier(tier, path, journal['tiers'], identity, save)
-        ownership = journal['tiers'][str(tier.number)].get('ownership')
-        if ownership and ownership['status'] == 'pending':
-            result['ownership_pending'].append(str(path))
-        result['tiers'].append(entry)
-    result['swap'] = _swap(journal, save)
-    journal['swap'] = result['swap']
-    save()
-    return result

ci/scripts/init-node/provisioning/target.py

@@ -1,170 +0,0 @@
-"""Shared root runner for remote and cloud-init enrollment."""
-from __future__ import annotations
-
-import argparse
-import json
-import os
-from pathlib import Path
-import platform
-import pwd
-import subprocess
-import sys
-import tempfile
-
-from .state import Journal, ProvisionError, atomic_write, exclusive_lock, safe_path
-from .userdata import decode_user_data
-
-STATE = Path('/var/lib/apex-init')
-
-
-def run(args, **kwargs):
-    result = subprocess.run(args, text=True, capture_output=True, timeout=300, **kwargs)
-    if result.returncode:
-        raise ProvisionError('required target operation failed: ' + args[0])
-    return result.stdout.strip()
-
-
-def preflight():
-    if os.geteuid() != 0:
-        raise ProvisionError('target enrollment requires root privileges')
-    release = {}
-    for line in Path('/etc/os-release').read_text().splitlines():
-        key, separator, value = line.partition('=')
-        if separator:
-            release[key] = value.strip('"')
-    if release.get('ID') != 'debian' or release.get('VERSION_ID') != '13' or platform.machine() != 'x86_64':
-        raise ProvisionError('initialization supports Debian 13 amd64 only')
-
-
-def validate_keys(spec):
-    for key in spec.authorized_keys:
-        with tempfile.NamedTemporaryFile(mode='w') as stream:
-            stream.write(key + '\n')
-            stream.flush()
-            try:
-                run(['ssh-keygen', '-l', '-f', stream.name])
-            except ProvisionError:
-                raise ProvisionError('administrator public key failed cryptographic validation') from None
-
-
-def administrator(spec):
-    try:
-        account = pwd.getpwnam('adam')
-    except KeyError:
-        run(['useradd', '--create-home', '--shell', '/bin/bash', '--groups', 'sudo,docker', 'adam'])
-        account = pwd.getpwnam('adam')
-    if account.pw_dir != '/home/adam' or account.pw_uid == 0:
-        raise ProvisionError('existing administrator has incompatible home or UID')
-    safe_path(account.pw_dir)
-    if account.pw_shell != '/bin/bash':
-        run(['usermod', '--shell', '/bin/bash', 'adam'])
-    run(['usermod', '--append', '--groups', 'sudo,docker', 'adam'])
-    directory = safe_path(Path(account.pw_dir) / '.ssh')
-    directory.mkdir(mode=0o700, exist_ok=True)
-    os.chown(directory, account.pw_uid, account.pw_gid)
-    os.chmod(directory, 0o700)
-    authorized = safe_path(directory / 'authorized_keys')
-    existing = authorized.read_text().splitlines() if authorized.exists() else []
-    keys = existing + [key for key in spec.authorized_keys if key not in existing]
-    atomic_write(authorized, '\n'.join(keys) + '\n', uid=account.pw_uid, gid=account.pw_gid)
-    policy = 'adam ALL=(ALL:ALL) NOPASSWD: ALL\n'
-    with tempfile.NamedTemporaryFile(mode='w') as stream:
-        stream.write(policy)
-        stream.flush()
-        run(['visudo', '-cf', stream.name])
-    atomic_write('/etc/sudoers.d/99-apex-adam', policy, 0o440)
-    run(['visudo', '-c'])
-    run(['runuser', '-u', 'adam', '--', 'sudo', '-n', 'true'])
-    for item in spec.files:
-        uid, gid = (0, 0) if item.owner == 'root:root' else (account.pw_uid, account.pw_gid)
-        atomic_write(item.path, item.content, item.mode, uid=uid, gid=gid)
-    if run(['hostname']) != spec.hostname:
-        run(['hostnamectl', 'set-hostname', spec.hostname])
-    groups = run(['id', '-nG', 'adam']).split()
-    if not {'sudo', 'docker'} <= set(groups):
-        raise ProvisionError('administrator groups did not converge')
-    return {'administrator': 'adam', 'hostname': spec.hostname, 'sudo': 'verified'}
-
-
-def enroll(spec, revision, bundle_digest, heartbeat=None):
-    from .baseline import apply_baseline
-    from .storage import acknowledge_storage_ownership, reconcile_storage
-    from .repository import reconcile_repository
-
-    preflight()
-    validate_keys(spec)
-    safe_path(STATE).mkdir(mode=0o700, exist_ok=True)
-    if STATE.stat().st_uid != 0:
-        raise ProvisionError('enrollment state directory must be root-owned')
-    os.chmod(STATE, 0o700)
-    with exclusive_lock(STATE / 'init.lock'):
-        journal = Journal(STATE / 'enrollment.json', spec.identity(), revision, bundle_digest)
-        journal.save()
-        journal.stage('baseline', apply_baseline, heartbeat)
-        journal.stage('administrator', lambda: administrator(spec), heartbeat)
-        result = journal.stage('storage', lambda: reconcile_storage(spec, journal.data, journal.save), heartbeat)
-        account = pwd.getpwnam('adam')
-        for path in result['ownership_pending']:
-            os.chown(safe_path(path), account.pw_uid, account.pw_gid)
-            acknowledge_storage_ownership(spec, journal.data, [path], journal.save)
-        journal.stage('repository', lambda: reconcile_repository(spec, journal.data, journal.save), heartbeat)
-        journal.data['status'] = 'enrolled'
-        journal.data['external_access'] = 'pending'
-        journal.save()
-        return journal.data
-
-
-def main(argv=None):
-    parser = argparse.ArgumentParser()
-    parser.add_argument('operation', choices=('validate', 'enroll', 'status', 'complete'))
-    parser.add_argument('--user-data')
-    parser.add_argument('--revision', default='unknown')
-    parser.add_argument('--bundle-digest', default='unknown')
-    parser.add_argument('--heartbeat')
-    args = parser.parse_args(argv)
-    try:
-        preflight()
-        if args.operation == 'status':
-            path = safe_path(STATE / 'enrollment.json')
-            print(path.read_text() if path.exists() else json.dumps({'status': 'not-started'}))
-            return 0
-        if args.operation == 'complete':
-            with exclusive_lock(STATE / 'init.lock'):
-                access = json.loads(safe_path(STATE / 'access/state.json').read_text())
-                path = safe_path(STATE / 'enrollment.json')
-                record = json.loads(path.read_text())
-                if access.get('status') != 'committed' or record.get('status') not in ('enrolled', 'complete'):
-                    raise ProvisionError('enrollment and access must be verified before completion')
-                record['external_access'] = 'verified'
-                record['status'] = 'complete'
-                atomic_write(path, json.dumps(record, sort_keys=True) + '\n')
-                print(json.dumps(record, sort_keys=True))
-            return 0
-        if not args.user_data:
-            raise ProvisionError('--user-data is required')
-        path = safe_path(args.user_data)
-        if path.stat().st_uid != 0 or path.stat().st_mode & 0o077:
-            raise ProvisionError('user-data must be root-owned and private')
-        if not path.is_file() or path.stat().st_size > 1024 * 1024:
-            raise ProvisionError('user-data must be a regular file no larger than 1 MiB')
-        spec = decode_user_data(path.read_bytes())
-        validate_keys(spec)
-        if args.operation == 'validate':
-            print(json.dumps({'status': 'valid', 'hostname': spec.hostname}))
-        else:
-            print(json.dumps(enroll(spec, args.revision, args.bundle_digest, args.heartbeat), sort_keys=True))
-        return 0
-    except Exception as error:
-        from .contract import ContractError
-        from .baseline import BaselineError
-        from .storage import StorageError
-        if isinstance(error, (ProvisionError, ContractError, BaselineError, StorageError)):
-            print(str(error), file=sys.stderr)
-        else:
-            # Unexpected dependency failures must not render raw credential input.
-            print('initialization failed (' + type(error).__name__ + '); inspect stage status', file=sys.stderr)
-        return 1
-
-
-if __name__ == '__main__':
-    raise SystemExit(main())

ci/scripts/init-node/provisioning/transport.py

@@ -1,385 +0,0 @@
-"""Workstation SSH transport. User-data stays opaque until target validation."""
-from __future__ import annotations
-
-import hashlib
-import io
-import ipaddress
-import json
-import os
-from pathlib import Path
-import re
-import shlex
-import shutil
-import subprocess
-import tarfile
-import tempfile
-import time
-import uuid
-
-
-class TransportError(RuntimeError):
-    pass
-
-
-class HandoverUncertain(TransportError):
-    pass
-
-
-# This runs before Python is guaranteed to exist on the Debian target. All
-# ancestors must be trusted; a root-owned sticky temporary directory is safe
-# only because root owns each managed child and exclusive mktemp creates it.
-REMOTE_PATH_GUARD = r'''
-set -eu
-umask 077
-apex_guard_dir() {
-    apex_checked=$1
-    while :; do
-        test ! -L "$apex_checked" && test -d "$apex_checked" || exit 91
-        test "$(stat -c %u -- "$apex_checked")" = 0 || exit 91
-        apex_mode=$(stat -c %a -- "$apex_checked")
-        test "$((0$apex_mode & 0022))" = 0 || test "$((0$apex_mode & 01000))" != 0 || exit 91
-        test "$apex_checked" != / || break
-        apex_checked=${apex_checked%/*}
-        test -n "$apex_checked" || apex_checked=/
-    done
-}
-apex_ensure_dir() {
-    apex_new_dir=$1
-    if test -e "$apex_new_dir" || test -L "$apex_new_dir"; then
-        apex_guard_dir "$apex_new_dir"
-        apex_mode=$(stat -c %a -- "$apex_new_dir")
-        test "$((0$apex_mode & 0022))" = 0 || exit 91
-    else
-        apex_guard_dir "${apex_new_dir%/*}"
-        mkdir -m "$2" -- "$apex_new_dir"
-        apex_guard_dir "$apex_new_dir"
-    fi
-}
-apex_guard_file() {
-    apex_file=$1
-    apex_guard_dir "${apex_file%/*}"
-    apex_mode=$(stat -c %a -- "${apex_file%/*}")
-    test "$((0$apex_mode & 0022))" = 0 || exit 91
-    test ! -L "$apex_file" || exit 91
-    if test -e "$apex_file"; then
-        test -f "$apex_file" || exit 91
-        test "$(stat -c %u -- "$apex_file")" = 0 || exit 91
-        test "$(stat -c %h -- "$apex_file")" = 1 || exit 91
-        apex_mode=$(stat -c %a -- "$apex_file")
-        test "$((0$apex_mode & 0022))" = 0 || exit 91
-    fi
-}
-apex_atomic_input() {
-    apex_destination=$1
-    apex_guard_file "$apex_destination"
-    apex_temporary=$(mktemp "${apex_destination%/*}/.apex-write.XXXXXXXXXX")
-    trap 'rm -f -- "$apex_temporary"' EXIT HUP INT TERM
-    cat > "$apex_temporary"
-    chmod 0600 "$apex_temporary"
-    apex_guard_file "$apex_destination"
-    mv -T -- "$apex_temporary" "$apex_destination"
-    trap - EXIT HUP INT TERM
-}
-'''
-
-
-def remote_atomic_input(path):
-    return REMOTE_PATH_GUARD + '\napex_atomic_input ' + shlex.quote(path)
-
-
-def remote_bundle_install(stage, installed, digest):
-    return REMOTE_PATH_GUARD + '\n' + (
-        'apex_guard_dir ' + shlex.quote(stage) + '; apex_guard_file ' + shlex.quote(stage + '/bundle.tar') + '; '
-        'test "$(sha256sum ' + shlex.quote(stage + '/bundle.tar') + ' | cut -d " " -f 1)" = ' + digest + '; '
-        'apex_ensure_dir /usr/local/lib/apex-provisioner 0755; '
-        'if test -e ' + installed + ' || test -L ' + installed + '; then '
-        'apex_guard_dir ' + installed + '; '
-        'apex_bad=$(find ' + installed + ' -xdev \\( -type l -o ! -user root -o -perm /022 -o \\( ! -type d -a ! -type f \\) \\) -print -quit); test -z "$apex_bad"; '
-        'else apex_install_tmp=$(mktemp -d /usr/local/lib/apex-provisioner/.install.XXXXXXXXXX); '
-        'trap \'rm -rf -- "$apex_install_tmp"\' EXIT HUP INT TERM; '
-        'tar --no-same-owner --same-permissions -xf ' + stage + '/bundle.tar -C "$apex_install_tmp"; '
-        'mv -T -- "$apex_install_tmp" ' + installed + '; trap - EXIT HUP INT TERM; fi; '
-        'tar --compare -f ' + stage + '/bundle.tar -C ' + installed)
-
-
-def local_file(value, name):
-    if not value or any(ord(c) < 32 for c in str(value)):
-        raise TransportError(f'{name} must name a readable regular file')
-    try:
-        path = Path(value).expanduser().resolve(strict=True)
-    except (OSError, ValueError):
-        raise TransportError(f'{name} must name a readable regular file') from None
-    if not path.is_file() or not os.access(path, os.R_OK):
-        raise TransportError(f'{name} must name a readable regular file')
-    return path
-
-
-def destination(value, port=None):
-    match = re.fullmatch(r'([A-Za-z_][A-Za-z0-9_.-]{0,31})@([A-Za-z0-9][A-Za-z0-9_.:-]*|\[[0-9a-fA-F:]+\])', value or '')
-    if not match or '..' in match[2]:
-        raise TransportError('Target must be USER@HOST without SSH options; use root or an existing administrator with sudo -n')
-    hostname = match[2].strip('[]')
-    if ':' in hostname:
-        try:
-            ipaddress.IPv6Address(hostname)
-        except ValueError:
-            raise TransportError('IPv6 targets must contain a valid address; pass ports separately') from None
-    if port is not None and (not re.fullmatch(r'[0-9]{1,5}', str(port)) or not 1 <= int(port) <= 65535):
-        raise TransportError('Port must be an integer from 1 through 65535')
-    return match[1], hostname, int(port) if port is not None else None
-
-
-def build_bundle(commons):
-    root = Path(commons)
-    files = sorted((root / 'engine').rglob('*.py'))
-    seed = root / 'engine/provisioning/seed.sh'
-    if not seed.is_file():
-        raise TransportError('Provisioning seed is missing')
-    files.append(seed)
-    output = io.BytesIO()
-    with tarfile.open(fileobj=output, mode='w') as archive:
-        for path in files:
-            if path.is_symlink() or '__pycache__' in path.parts:
-                raise TransportError('Provisioning bundle contains an unsafe source path')
-            data = path.read_bytes()
-            info = tarfile.TarInfo(path.relative_to(root).as_posix())
-            info.size, info.mode, info.mtime = len(data), (0o755 if path == seed else 0o644), 0
-            info.uid = info.gid = 0
-            archive.addfile(info, io.BytesIO(data))
-    data = output.getvalue()
-    revision = 'unversioned-modified'
-    if shutil.which('git'):
-        result = subprocess.run(['git', '-C', str(root), 'rev-parse', 'HEAD'], capture_output=True, text=True, timeout=30)
-        if result.returncode == 0 and re.fullmatch(r'[a-f0-9]{40,64}\n?', result.stdout):
-            revision = result.stdout.strip()
-            dirty = subprocess.run(['git', '-C', str(root), 'status', '--porcelain', '--', 'engine'], capture_output=True, text=True, timeout=30)
-            if dirty.returncode or dirty.stdout.strip():
-                revision += '-modified'
-    return data, hashlib.sha256(data).hexdigest(), revision
-
-
-class SSH:
-    def __init__(self, target, port=None, identity_file=None, runner=subprocess.run):
-        self.user, self.host, self.port = destination(target, port)
-        self.identity = local_file(identity_file, 'Identity file') if identity_file else None
-        self.runner = runner
-        self.resolved_host = None
-        self.resolved_port = None
-        self.trust_file = None
-
-    def arguments(self, fresh=False):
-        args = ['ssh', '-T', '-o', 'StrictHostKeyChecking=yes', '-o', 'ConnectTimeout=15',
-                '-o', 'ServerAliveInterval=10', '-o', 'ServerAliveCountMax=3']
-        if self.identity:
-            args += ['-i', str(self.identity)]
-        args += ['-l', 'adam' if fresh else self.user]
-        if fresh:
-            if not self.trust_file or not self.resolved_host:
-                raise TransportError('Fresh access requires the verified bootstrap host identity')
-            args += ['-p', '2222', '-o', 'ControlMaster=no', '-o', 'ControlPath=none',
-                     '-o', 'ControlPersist=no', '-o', 'BatchMode=yes',
-                     '-o', 'PreferredAuthentications=publickey', '-o', 'PasswordAuthentication=no',
-                     '-o', 'KbdInteractiveAuthentication=no', '-o', 'Hostname=' + self.resolved_host,
-                     '-o', 'HostKeyAlias=apex-init-verified-target',
-                     '-o', 'UserKnownHostsFile=' + str(self.trust_file),
-                     '-o', 'GlobalKnownHostsFile=/dev/null']
-        elif self.port:
-            args += ['-p', str(self.port)]
-        return args
-
-    def resolve(self):
-        result = self.runner(self.arguments() + ['-G', '--', self.host], capture_output=True, text=True, timeout=30)
-        if result.returncode:
-            raise TransportError('Unable to resolve SSH connection configuration')
-        values = dict(line.split(' ', 1) for line in result.stdout.splitlines() if ' ' in line)
-        hostname = values.get('hostname', '')
-        port = values.get('port', '')
-        destination(self.user + '@' + hostname, port)
-        self.resolved_host, self.resolved_port = hostname, int(port)
-
-    def run(self, script, data=None, fresh=False, privileged=True, check=True, timeout=300):
-        command = ['sh', '-c', script]
-        if privileged and (fresh or self.user != 'root'):
-            command = ['sudo', '-n', '--', *command]
-        result = self.runner(self.arguments(fresh) + ['--', self.host, shlex.join(command)],
-                             input=data, capture_output=True, timeout=timeout)
-        if check and result.returncode:
-            # SSH and child diagnostics can contain user-data or credential-bearing URLs.
-            raise TransportError('Remote operation failed; inspect the root-owned target status for a redacted diagnosis')
-        return result
-
-    def fresh_login(self):
-        self.run('test "$(id -u)" = 0', fresh=True)
-
-
-def protected_handover(ssh, invoke, token, port, client_address):
-    result = invoke('begin', token, port, client_address, False)
-    try:
-        ssh.fresh_login()
-        if result['status'] == 'committed':
-            return
-        if result['status'] != 'existing':
-            invoke('finalize', token, None, None, True)
-            ssh.fresh_login()
-        try:
-            invoke('commit', token, None, None, True)
-        except Exception:
-            try:
-                state = invoke('status', token, None, None, True)
-                if state.get('token') != token or state.get('status') not in ('committed', 'existing', 'transition', 'final', 'rolled_back', 'recovery_pending'):
-                    raise TransportError('Unexpected access transition state')
-            except Exception:
-                raise HandoverUncertain('Access commit outcome is unknown; verify the administrator and bootstrap endpoints before resuming') from None
-            if state['status'] != 'committed':
-                raise
-    except HandoverUncertain:
-        raise
-    except Exception:
-        try:
-            invoke('rollback', token, None, None, True)
-        except Exception:
-            try:
-                invoke('rollback', token, None, None, False)
-            except Exception:
-                pass  # Persistent timer is authoritative if both connections are gone.
-        raise
-
-
-def initialize(commons, target, user_data, port=None, identity_file=None, progress=print):
-    if not shutil.which('ssh'):
-        raise TransportError('OpenSSH ssh is required on the workstation')
-    source = local_file(user_data, 'User-data')
-    if source.stat().st_size > 1024 * 1024:
-        raise TransportError('User-data must be no larger than 1 MiB')
-    ssh = SSH(target, port, identity_file)
-    ssh.resolve()
-    bundle, digest, revision = build_bundle(commons)
-    run_id = uuid.uuid4().hex
-    installed = '/usr/local/lib/apex-provisioner/' + digest
-    stage = None
-    launched = False
-    finished = False
-    access_committed = False
-    access_uncertain = False
-    cleanup_policy = '/etc/tmpfiles.d/apex-init-' + run_id + '.conf'
-    with tempfile.TemporaryDirectory(prefix='apex-init-trust-') as local:
-        try:
-            preflight = ssh.run('set -eu; test "$(id -u)" = 0; . /etc/os-release; '
-                                'test "$ID" = debian; test "$VERSION_ID" = 13; '
-                                'test "$(dpkg --print-architecture)" = amd64; '
-                                'command -v systemd-run >/dev/null; '
-                                'printf "%s\\n" "${SSH_CONNECTION:-}"; '
-                                'cat /etc/ssh/ssh_host_*_key.pub')
-            lines = preflight.stdout.decode().splitlines()
-            connection = lines[0].split() if lines else []
-            if len(connection) != 4:
-                # sudo commonly clears SSH_CONNECTION; read it before elevation instead.
-                raw = ssh.run('printf "%s\\n" "$SSH_CONNECTION"', privileged=False).stdout.decode().split()
-                connection = raw
-            if len(connection) != 4:
-                raise TransportError('Unable to inspect the established SSH connection')
-            keys = [line for line in lines[1:] if re.fullmatch(r'(ssh-[\w-]+|ecdsa-[\w-]+) [A-Za-z0-9+/=]+(?: .*?)?', line)]
-            if not keys:
-                raise TransportError('Unable to obtain public host keys through the verified connection')
-            trust = Path(local) / 'known_hosts'
-            trust.write_text(''.join('apex-init-verified-target,[apex-init-verified-target]:2222 ' + ' '.join(key.split()[:2]) + '\n' for key in keys))
-            trust.chmod(0o600)
-            ssh.trust_file = trust
-            existing = ssh.run(REMOTE_PATH_GUARD + '\napex_ensure_dir /var/lib/apex-init 0700; '
-                               'apex_guard_file /var/lib/apex-init/remote.json; '
-                               'if test -f /var/lib/apex-init/remote.json; then cat /var/lib/apex-init/remote.json; fi').stdout
-            if existing:
-                prior = json.loads(existing)
-                previous_unit = prior.get('unit', '')
-                if not re.fullmatch(r'apex-init-[a-f0-9]{32}', previous_unit):
-                    raise TransportError('Existing remote coordinator state requires manual inspection')
-                active = ssh.run(shlex.join(['systemctl', 'is-active', '--quiet', previous_unit]), check=False)
-                if active.returncode == 0:
-                    raise TransportError('An enrollment unit is still running: ' + previous_unit + '; wait for it to settle before resuming')
-            stage = ssh.run(REMOTE_PATH_GUARD + '\napex_guard_dir /var/tmp; mktemp -d /var/tmp/apex-init.XXXXXXXXXX').stdout.decode().strip()
-            if not re.fullmatch(r'/var/tmp/apex-init\.[A-Za-z0-9]{10}', stage):
-                raise TransportError('Target returned an invalid staging path')
-            # Hard reboot can bypass the shell trap; boot-time tmpfiles removes opaque inputs.
-            cleanup = ('r ' + stage + '/user-data - - - -\n' +
-                       'r ' + stage + '/heartbeat - - - -\n').encode()
-            ssh.run(REMOTE_PATH_GUARD + '\napex_ensure_dir /etc/tmpfiles.d 0755; apex_atomic_input ' + cleanup_policy, cleanup)
-            ssh.run(remote_atomic_input(stage + '/bundle.tar'), bundle)
-            ssh.run(remote_atomic_input(stage + '/user-data'), source.read_bytes())
-            install = remote_bundle_install(stage, installed, digest)
-            progress('Installing the versioned target runtime and validating enrollment input.')
-            ssh.run(install)
-            base = ['python3', '-m', 'engine.provisioning.target']
-            options = ['--user-data', stage + '/user-data', '--revision', revision, '--bundle-digest', digest]
-            # State remains inspectable after disconnect; user-data is removed by the durable unit.
-            enrollment = shlex.join(base + ['enroll'] + options + ['--heartbeat', stage + '/heartbeat'])
-            validation = shlex.join(base + ['validate'] + options)
-            access_preflight = shlex.join(['python3', '-m', 'engine.provisioning.access', 'preflight'])
-            seed = shlex.join(['bash', installed + '/engine/provisioning/seed.sh'])
-            wrapper = ('umask 077; cd ' + installed + '; '
-                       'trap ' + shlex.quote('rm -f ' + stage + '/user-data ' + stage + '/heartbeat ' + cleanup_policy) + ' EXIT; '
-                       '{ ' + seed + ' && ' + validation + ' && ' + access_preflight + ' && ' + enrollment + '; } >' + stage + '/runner.log 2>&1; rc=$?; '
-                       'printf "%s\\n" "$rc" >' + stage + '/result.new; mv ' + stage + '/result.new ' + stage + '/result; exit "$rc"')
-            unit = 'apex-init-' + run_id
-            record = json.dumps({'unit': unit, 'staging': stage, 'revision': revision, 'bundle_digest': digest}).encode()
-            ssh.run(remote_atomic_input('/var/lib/apex-init/remote.json'), record)
-            ssh.run('touch ' + stage + '/heartbeat; ' + shlex.join(['systemd-run', '--quiet', '--unit', unit,
-                    '--property=Type=exec', '--property=KillMode=process', '/bin/sh', '-c', wrapper]))
-            launched = True
-            progress('Enrollment is running under ' + unit + '; target staging: ' + stage)
-            while True:
-                status = ssh.run('if test -f ' + stage + '/result; then cat ' + stage + '/result; '
-                                 'elif systemctl is-active --quiet ' + unit + '; then touch ' + stage + '/heartbeat; printf running; '
-                                 'else printf stopped; fi').stdout.decode().strip()
-                if status != 'running':
-                    finished = True
-                    if status != '0':
-                        raise TransportError('Target provisioning failed; inspect root-owned ' + stage +
-                                             '/runner.log and /var/lib/apex-init/enrollment.json')
-                    break
-                time.sleep(5)
-            progress('Enrollment verified. Checking protected administrator SSH handover.')
-
-            def access(operation, token, original_port, address, fresh):
-                if operation == 'status':
-                    result = ssh.run(REMOTE_PATH_GUARD + '\napex_guard_file /var/lib/apex-init/access/state.json; '
-                                     'cat /var/lib/apex-init/access/state.json', fresh=True)
-                    return json.loads(result.stdout)
-                args = ['python3', '-m', 'engine.provisioning.access', operation, '--token', token]
-                if original_port:
-                    args += ['--original-port', str(original_port), '--client-address', address]
-                result = ssh.run('cd ' + installed + ' && ' + shlex.join(args), fresh=fresh)
-                return json.loads(result.stdout)
-
-            protected_handover(ssh, access, run_id, int(connection[3]), connection[0])
-            access_committed = True
-            result = ssh.run('cd ' + installed + ' && ' + shlex.join(base + ['complete']), fresh=True)
-            completed = json.loads(result.stdout)
-            progress('Initialization and fresh administrator access verified; provisioner ' + revision + ', bundle ' + digest)
-            repository = completed.get('stages', {}).get('repository', {}).get('result', {})
-            if repository.get('commit'):
-                progress('Repository ' + repository['commit'] + '; commons ' + repository.get('commons_commit', 'unknown'))
-            return completed
-        except HandoverUncertain:
-            access_uncertain = True
-            raise
-        finally:
-            if stage and (not launched or finished):
-                for fresh in (True, False):
-                    try:
-                        ssh.run('rm -f ' + stage + '/user-data ' + stage + '/heartbeat ' + stage + '/bundle.tar ' + cleanup_policy, fresh=fresh)
-                        break
-                    except Exception:
-                        pass
-            if launched and not finished:
-                progress('Coordinator stopped. The current target stage may finish; later stages stop after heartbeat expiry.')
-            if access_uncertain:
-                progress('Access commit outcome is unknown; verify which endpoint is active before resuming. Rollback may already be disabled.')
-                endpoints = [('Administrator candidate', 'adam', 2222), ('Bootstrap candidate', ssh.user, ssh.resolved_port)]
-            else:
-                label = 'Safe resume' + ('' if access_committed else ' after any pending rollback')
-                endpoints = [(label, 'adam' if access_committed else ssh.user, 2222 if access_committed else ssh.resolved_port)]
-            for label, resume_user, resume_port in endpoints:
-                resume = './apex init ' + resume_user + '@' + ssh.host + ' --port ' + str(resume_port) + ' --user-data ' + shlex.quote(str(source))
-                if ssh.identity:
-                    resume += ' --identity-file ' + shlex.quote(str(ssh.identity))
-                progress(label + ': ' + resume)

ci/scripts/init-node/provisioning/userdata.py

@@ -1,29 +0,0 @@
-"""Target-only decoding; importing the CLI never requires a YAML dependency."""
-from .contract import ContractError, normalize_user_data
-
-
-def decode_user_data(content):
-    import yaml
-
-    class Loader(yaml.SafeLoader):
-        def compose_node(self, parent, index):
-            if self.check_event(yaml.AliasEvent):
-                raise ContractError('YAML aliases are not supported')
-            return super().compose_node(parent, index)
-
-        def construct_mapping(self, node, deep=False):
-            result = {}
-            for key_node, value_node in node.value:
-                key = self.construct_object(key_node, deep=deep)
-                if not isinstance(key, str) or key in result:
-                    raise ContractError('YAML mapping keys must be unique strings')
-                result[key] = self.construct_object(value_node, deep=deep)
-            return result
-
-    if len(content) > 1024 * 1024:
-        raise ContractError('user-data exceeds the supported size')
-    try:
-        document = yaml.load(content, Loader=Loader)
-    except (yaml.YAMLError, UnicodeError, RecursionError):
-        raise ContractError('invalid or unsupported YAML user-data') from None
-    return normalize_user_data(document)

ci/scripts/init-node/tests/provision_fakes.py

@@ -1,38 +0,0 @@
-from engine.provisioning.security import SecurityError
-
-
-class SecurityBoundary:
-    """Isolate firewall/service effects while exercising the durable coordinator."""
-    def __init__(self):
-        self.active = False
-        self.fail_activation = False
-        self.fail_restore = False
-
-    def preflight(self):
-        return {}
-
-    def snapshot(self, plan, port, address):
-        return {'phase': 'prepared', 'was_active': self.active}
-
-    def activate(self, state, persist):
-        self.active = True
-        state['phase'] = 'active'
-        persist(state)
-        if self.fail_activation:
-            raise SecurityError('activation failed')
-
-    def finalize(self, state, persist):
-        self.verify(state)
-        state['phase'] = 'final'
-        persist(state)
-
-    def verify(self, state):
-        if not self.active:
-            raise SecurityError('security stopped')
-
-    def restore(self, state, persist):
-        self.active = state['was_active']
-        state['cleanup'] = 'pending' if self.fail_restore else 'complete'
-        persist(state)
-        if self.fail_restore:
-            raise SecurityError('cleanup pending')

ci/scripts/init-node/tests/test_provision_baseline.py

@@ -1,511 +0,0 @@
-"""Exercise baseline decisions with a temporary host filesystem and fake commands."""
-import hashlib
-import importlib.util
-import io
-import json
-from pathlib import Path
-import subprocess
-import tempfile
-import unittest
-from unittest.mock import patch
-
-from engine.provisioning import baseline
-
-
-class Host:
-    def __init__(self, root):
-        self.root = root
-        self.commands = []
-        self.installed = {}
-        self.partial = {}
-        self.active = set()
-        self.enabled = set()
-        self.masked = set()
-        self.fail_install = False
-        self.plan = ''
-        self.zfs_available = True
-        self.docker_root = '/srv/docker'
-        self.bouncer_key = 'testOnlyGeneratedCredential1234567890'
-        self.registration_failure = False
-        self.registrations = {}
-        self.path('/etc/os-release').parent.mkdir(parents=True)
-        self.path('/etc/os-release').write_text('ID=debian\nVERSION_ID="13"\n')
-        self.path('/etc/debian_version').touch()
-        self.path('/usr/sbin').mkdir(parents=True)
-
-    def path(self, path):
-        path = Path(path)
-        if str(path).startswith(str(self.root)):
-            return path
-        return self.root / str(path).lstrip('/')
-
-    def command(self, args, **kwargs):
-        self.commands.append(args)
-        code, output = 0, ''
-        if args[:2] == ['dpkg', '--print-architecture']:
-            output = 'amd64\n'
-        elif args[0] == 'dpkg-query':
-            output = ''.join(f'{name}\tinstall ok installed\t{version}\n' for name, version in self.installed.items())
-            output += ''.join(f'{name}\t{status}\t1.0\n' for name, status in self.partial.items())
-        elif args == ['uname', '-r']:
-            output = '6.12.0-amd64\n'
-        elif args[0] == 'apt-get' and 'install' in args:
-            if '--simulate' in args:
-                output = self.plan
-            elif self.fail_install:
-                code = 100
-            else:
-                start = args.index('--no-remove') + 1
-                # Reproduce dpkg configuring the bouncer before CrowdSec in one batch.
-                if 'crowdsec-firewall-bouncer-iptables' in args[start:]:
-                    config = self.path('/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml')
-                    config.parent.mkdir(parents=True, exist_ok=True)
-                    if not config.exists():
-                        key = 'packageGeneratedCredential123456789' if 'crowdsec' in self.installed else '<API_KEY>'
-                        config.write_text(f'api_url: http://127.0.0.1:8080/\napi_key: {key}\nmode: iptables\n')
-                        config.chmod(0o640)
-                for name in args[start:]:
-                    self.installed[name] = '1.0'
-                    self.partial.pop(name, None)
-        elif args[:4] == ['cscli', '-oraw', 'bouncers', 'add']:
-            if self.registration_failure:
-                code, output = 1, 'sensitive diagnostic'
-            else:
-                output = self.bouncer_key + '\n'
-                self.registrations[args[4]] = self.bouncer_key
-        elif args[0] == 'systemctl':
-            service = args[-1]
-            operation = args[1]
-            if operation == 'is-active':
-                code = 0 if service in self.active else 3
-            elif operation == 'is-enabled':
-                code = 0 if service in self.enabled else 1
-                output = 'masked\n' if service in self.masked else ''
-            elif operation == 'start':
-                conditions = []
-                for dropin in self.path(f'/etc/systemd/system/{service}.service.d').glob('*.conf'):
-                    conditions.extend(line.partition('=')[2] for line in dropin.read_text().splitlines()
-                                      if line.startswith('ConditionPathExists='))
-                if service not in self.masked and all(self.path(path).exists() for path in conditions):
-                    self.active.add(service)
-            elif operation == 'enable':
-                self.enabled.add(service)
-            elif operation == 'disable':
-                self.enabled.discard(service)
-            elif operation == 'mask':
-                self.masked.add(service)
-                unit = self.path(f'/etc/systemd/system/{service}.service')
-                unit.parent.mkdir(parents=True, exist_ok=True)
-                unit.symlink_to('/dev/null')
-            elif operation == 'unmask':
-                self.masked.discard(service)
-                unit = self.path(f'/etc/systemd/system/{service}.service')
-                if unit.is_symlink():
-                    unit.unlink()
-        elif args[:2] == ['docker', 'info']:
-            output = self.docker_root + '\n'
-        elif args[:3] == ['docker', 'compose', 'version']:
-            output = '2.39.2\n'
-        elif args == ['modprobe', 'zfs']:
-            code = 0 if self.zfs_available else 1
-        elif args == ['zfs', 'version']:
-            output = 'zfs-2.3.2\nzfs-kmod-2.3.2\n'
-        return subprocess.CompletedProcess(args, code, output, 'installation failed' if code else '')
-
-
-class BaselineTests(unittest.TestCase):
-    def setUp(self):
-        self.temp = tempfile.TemporaryDirectory()
-        self.addCleanup(self.temp.cleanup)
-        self.host = Host(Path(self.temp.name).resolve())
-        self.helper = b'#!/bin/bash\necho pinned test helper\n'
-        patches = [
-            patch.object(baseline, 'Path', self.host.path),
-            patch.object(baseline, 'POLICY', self.host.path('/usr/sbin/policy-rc.d')),
-            patch.object(baseline, 'POLICY_STATE', self.host.path('/usr/sbin/.apex-policy-rc.d')),
-            patch.object(baseline, 'SECURITY_STATE', self.host.path('/var/lib/apex/baseline-security.json')),
-            patch.object(baseline.os, 'geteuid', return_value=Path(self.temp.name).stat().st_uid),
-            patch.object(baseline.subprocess, 'run', side_effect=self.host.command),
-            patch.object(baseline.urllib.request, 'urlopen', side_effect=self.download),
-            patch.object(baseline, 'HELPER_SHA256', hashlib.sha256(self.helper).hexdigest()),
-        ]
-        for item in patches:
-            item.start()
-            self.addCleanup(item.stop)
-
-    def download(self, url, **kwargs):
-        if url == baseline.HELPER_URL:
-            return io.BytesIO(self.helper)
-        return io.BytesIO(b'-----BEGIN PGP PUBLIC KEY BLOCK-----\nfixture\n')
-
-    def apply(self):
-        # The filesystem remains owned by the test uid; only root preflight is isolated.
-        with patch.object(baseline, 'preflight'):
-            return baseline.apply_baseline()
-
-    def test_package_database_drives_installation_and_repeat_does_not_upgrade(self):
-        self.host.installed['curl'] = '7.0'
-        self.host.partial['rsyslog'] = 'deinstall ok config-files'
-        self.assertEqual(baseline.install_missing(['curl', 'rsyslog']), ['rsyslog'])
-        self.assertEqual(self.host.installed['curl'], '7.0')
-        self.assertEqual(self.host.installed['rsyslog'], '1.0')
-        count = len(self.host.commands)
-        self.assertEqual(baseline.install_missing(['curl', 'rsyslog']), [])
-        self.assertFalse(any(command[0] == 'apt-get' for command in self.host.commands[count:]))
-
-    def test_dependency_upgrade_is_rejected_before_install(self):
-        self.host.plan = 'Inst libc6 [2.0] (3.0 Debian:13)\n'
-        with self.assertRaisesRegex(baseline.BaselineError, 'upgrade/removal'):
-            baseline.install_missing(['git'])
-        self.assertNotIn('git', self.host.installed)
-
-    def test_held_installed_packages_are_preserved(self):
-        self.host.partial['git'] = 'hold ok installed'
-        self.assertEqual(baseline.install_missing(['git']), [])
-        self.assertFalse(any(command[0] == 'apt-get' for command in self.host.commands))
-
-    def test_conflicting_docker_config_is_rejected_without_host_mutation(self):
-        path = self.host.path('/etc/docker/daemon.json')
-        path.parent.mkdir(parents=True)
-        path.write_text('{"data-root": "/data/docker"}')
-        before = sorted(str(path.relative_to(self.host.root)) for path in self.host.root.rglob('*'))
-        with self.assertRaisesRegex(baseline.BaselineError, 'Conflicting Docker'):
-            self.apply()
-        self.assertEqual(before, sorted(str(path.relative_to(self.host.root)) for path in self.host.root.rglob('*')))
-        self.assertFalse(any(command[0] in ('apt-get', 'systemctl') for command in self.host.commands))
-
-    def test_existing_docker_data_is_preserved(self):
-        data = self.host.path('/var/lib/docker/volumes/user-data')
-        data.parent.mkdir(parents=True)
-        data.write_text('valuable')
-        with self.assertRaisesRegex(baseline.BaselineError, 'Existing /var/lib/docker'):
-            self.apply()
-        self.assertEqual(data.read_text(), 'valuable')
-
-    def test_conflicting_distribution_docker_package_is_never_removed(self):
-        self.host.installed['docker.io'] = '26.1.5'
-        with self.assertRaisesRegex(baseline.BaselineError, 'Conflicting Docker packages'):
-            self.apply()
-        self.assertEqual(self.host.installed['docker.io'], '26.1.5')
-        self.assertFalse(any(command[0] == 'apt-get' for command in self.host.commands))
-
-    def test_unmanaged_destination_data_is_not_adopted(self):
-        data = self.host.path('/srv/docker/private-data')
-        data.parent.mkdir(parents=True)
-        data.write_text('preserve me')
-        with self.assertRaisesRegex(baseline.BaselineError, 'Unmanaged /srv/docker'):
-            self.apply()
-        self.assertEqual(data.read_text(), 'preserve me')
-
-    def test_package_failure_restores_existing_service_policy(self):
-        policy = self.host.path('/usr/sbin/policy-rc.d')
-        policy.write_text('#!/bin/sh\nexit 42\n')
-        policy.chmod(0o751)
-        self.host.fail_install = True
-        with self.assertRaises(baseline.BaselineError):
-            self.apply()
-        self.assertEqual(policy.read_text(), '#!/bin/sh\nexit 42\n')
-        self.assertEqual(policy.stat().st_mode & 0o777, 0o751)
-        self.assertFalse(self.host.path('/usr/sbin/.apex-policy-rc.d').exists())
-        self.assertFalse(self.host.masked)
-
-    def test_service_policy_symlink_is_restored_after_failure(self):
-        policy = self.host.path('/usr/sbin/policy-rc.d')
-        original = self.host.path('/usr/sbin/provider-policy')
-        original.write_text('provider policy')
-        policy.symlink_to(original)
-        with self.assertRaisesRegex(RuntimeError, 'boom'):
-            with baseline.service_start_guard():
-                self.assertFalse(policy.is_symlink())
-                raise RuntimeError('boom')
-        self.assertTrue(policy.is_symlink())
-        self.assertEqual(original.read_text(), 'provider policy')
-
-    def test_interrupted_guard_is_recovered_before_next_attempt(self):
-        state = self.host.path('/usr/sbin/.apex-policy-rc.d')
-        state.mkdir()
-        (state / 'original').write_text('original policy')
-        (state / 'ready').touch()
-        self.host.path('/usr/sbin/policy-rc.d').write_text('#!/bin/sh\nexit 101\n')
-        with baseline.service_start_guard():
-            pass
-        self.assertEqual(self.host.path('/usr/sbin/policy-rc.d').read_text(), 'original policy')
-
-    def test_external_policy_change_is_not_overwritten(self):
-        policy = self.host.path('/usr/sbin/policy-rc.d')
-        with self.assertRaisesRegex(baseline.BaselineError, 'changed outside APEX'):
-            with baseline.service_start_guard():
-                policy.write_text('external policy')
-        self.assertEqual(policy.read_text(), 'external policy')
-
-    def test_baseline_reports_capabilities_and_repeats_without_restarts(self):
-        result = self.apply()
-        self.assertEqual(result['capabilities']['docker_data_root'], '/srv/docker')
-        self.assertEqual(result['capabilities']['compose'], '2.39.2')
-        self.assertIn('zfs-kmod-2.3.2', result['capabilities']['zfs'])
-        self.assertEqual(result['packages']['zfs-dkms'], '1.0')
-        self.assertEqual(self.host.path('/etc/apt/apt.conf.d/99-apex').read_text(), 'APT::Keep-Downloaded-Packages "0";\n')
-        self.assertFalse(self.host.path('/usr/sbin/policy-rc.d').exists())
-        self.assertFalse(self.host.masked)
-        self.assertNotIn('ufw', self.host.active)
-        self.assertNotIn('crowdsec-firewall-bouncer', self.host.enabled)
-        self.assertFalse(any(command[0] in ('ufw', '/usr/local/bin/ufw-docker') for command in self.host.commands))
-        count = len(self.host.commands)
-        again = self.apply()
-        self.assertEqual(again, result)
-        repeated = self.host.commands[count:]
-        self.assertFalse(any(command[0] == 'apt-get' or command[:2] in
-                             (['systemctl', 'restart'], ['systemctl', 'start']) for command in repeated))
-
-    def bouncer_config(self, key='<API_KEY>'):
-        self.host.installed.update({'crowdsec': '1.8.1', 'crowdsec-firewall-bouncer-iptables': '0.0.36'})
-        config = self.host.path('/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml')
-        config.parent.mkdir(parents=True, exist_ok=True)
-        config.write_text(f'# custom settings\napi_url: http://127.0.0.1:9090/\napi_key: {key} # keep comment\nmode: iptables\nblacklists_ipv4: [custom]\n')
-        config.chmod(0o640)
-        return config
-
-    def test_fresh_crowdsec_is_configured_before_bouncer_install(self):
-        self.apply()
-        installs = [args for args in self.host.commands if args[0] == 'apt-get' and 'install' in args and '--simulate' not in args]
-        crowdsec = next(index for index, args in enumerate(installs) if 'crowdsec' in args)
-        bouncer = next(index for index, args in enumerate(installs) if 'crowdsec-firewall-bouncer-iptables' in args)
-        self.assertLess(crowdsec, bouncer)
-        self.assertFalse(self.host.registrations)
-        self.assertFalse(self.host.active & {'crowdsec', 'crowdsec-firewall-bouncer'})
-
-    def test_broken_installed_bouncer_is_repaired_and_repeat_preserves_registration(self):
-        config = self.bouncer_config()
-        before = config.read_text()
-        owner = (config.stat().st_uid, config.stat().st_gid)
-        self.apply()
-        self.assertEqual(config.read_text(), before.replace('<API_KEY>', self.host.bouncer_key))
-        registration = config.with_suffix('.yaml.id').read_text().strip()
-        self.assertEqual(self.host.registrations[registration], self.host.bouncer_key)
-        self.assertEqual((config.stat().st_uid, config.stat().st_gid), owner)
-        self.assertEqual(config.stat().st_mode & 0o777, 0o640)
-        count = len(self.host.registrations)
-        self.apply()
-        self.assertEqual(len(self.host.registrations), count)
-        self.assertFalse(self.host.active & {'crowdsec', 'crowdsec-firewall-bouncer'})
-
-    def test_existing_bouncer_key_and_custom_settings_are_preserved(self):
-        config = self.bouncer_config('existing-custom-key')
-        before = config.read_bytes()
-        self.apply()
-        self.assertEqual(config.read_bytes(), before)
-        self.assertFalse(self.host.registrations)
-
-    def test_registration_failure_leaves_placeholder_without_exposing_output(self):
-        config = self.bouncer_config()
-        before = config.read_bytes()
-        self.host.registration_failure = True
-        with self.assertRaises(baseline.BaselineError) as failure:
-            self.apply()
-        self.assertEqual(config.read_bytes(), before)
-        self.assertFalse(config.with_suffix('.yaml.id').exists())
-        self.assertNotIn('sensitive diagnostic', str(failure.exception))
-
-    def test_invalid_registered_key_is_not_published(self):
-        for key in ('', '<API_KEY>', 'not a key\nsensitive diagnostic'):
-            with self.subTest(key=key):
-                config = self.bouncer_config()
-                before = config.read_bytes()
-                self.host.bouncer_key = key
-                with self.assertRaises(baseline.BaselineError):
-                    self.apply()
-                self.assertEqual(config.read_bytes(), before)
-                self.assertFalse(config.with_suffix('.yaml.id').exists())
-
-    def test_bouncer_config_symlink_is_rejected_before_registration(self):
-        config = self.bouncer_config()
-        destination = config.with_suffix('.provider')
-        config.rename(destination)
-        config.symlink_to(destination)
-        with self.assertRaises(baseline.BaselineError):
-            self.apply()
-        self.assertIn('<API_KEY>', destination.read_text())
-        self.assertFalse(self.host.registrations)
-
-    def test_quoted_null_is_an_existing_key_not_an_uninitialized_yaml_value(self):
-        config = self.bouncer_config('"null"')
-        before = config.read_bytes()
-        self.apply()
-        self.assertEqual(config.read_bytes(), before)
-        self.assertFalse(self.host.registrations)
-
-    def test_package_placeholder_variants_are_repaired(self):
-        for placeholder in ('"<API_KEY>"', "'$API_KEY'", '${API_KEY}', 'null', '~', '', '""'):
-            with self.subTest(placeholder=placeholder):
-                config = self.bouncer_config(placeholder)
-                self.apply()
-                self.assertIn('api_key: ' + self.host.bouncer_key, config.read_text())
-
-    def test_ambiguous_or_missing_api_key_is_not_mutated(self):
-        for content in ('api_key: <API_KEY>\n"api_key": existing-key\n', 'api_url: http://localhost:8080/\n',
-                        'api_key: "<API_KEY>x\n'):
-            with self.subTest(content=content):
-                config = self.bouncer_config()
-                config.write_text(content)
-                with self.assertRaises(baseline.BaselineError):
-                    self.apply()
-                self.assertEqual(config.read_text(), content)
-                self.assertFalse(self.host.registrations)
-
-    def test_multiline_custom_credentials_are_preserved_without_registration(self):
-        for credential in ('\n  existing-custom-credential', ' # comment\n\n  existing-custom-credential',
-                           ' <API_KEY>\n  existing-custom-credential', '\n  # comment\n  existing-custom-credential'):
-            with self.subTest(credential=credential):
-                config = self.bouncer_config()
-                config.write_text('api_key:' + credential + '\napi_url: http://localhost:9090/\n')
-                identity = config.with_suffix('.yaml.id')
-                identity.write_text('existing-registration\n')
-                before = config.read_bytes()
-                with self.assertRaises(baseline.BaselineError):
-                    self.apply()
-                self.assertEqual(config.read_bytes(), before)
-                self.assertEqual(identity.read_text(), 'existing-registration\n')
-                self.assertFalse(self.host.registrations)
-
-    @unittest.skipUnless(importlib.util.find_spec('yaml'), 'target-only python3-yaml is unavailable')
-    def test_empty_credential_replacement_parses_as_the_generated_key(self):
-        import yaml
-        for entry in ('api_key:', 'api_key: # comment', 'api_key:    ',
-                      'api_key:\n  # next-line comment'):
-            with self.subTest(entry=entry):
-                config = self.bouncer_config()
-                config.write_text(entry + '\napi_url: http://localhost:9090/\n')
-                self.assertIsNone(yaml.safe_load(config.read_text())['api_key'])
-                self.apply()
-                parsed = yaml.safe_load(config.read_text())
-                self.assertEqual(parsed['api_key'], self.host.bouncer_key)
-                self.assertEqual(parsed['api_url'], 'http://localhost:9090/')
-                if '# ' in entry:
-                    self.assertIn(entry[entry.index('# '):], config.read_text())
-
-    def test_redirected_bookkeeping_is_rejected_before_registration(self):
-        config = self.bouncer_config()
-        destination = config.with_suffix('.provider')
-        destination.write_text('provider-data')
-        config.with_suffix('.yaml.id').symlink_to(destination)
-        with self.assertRaises(baseline.BaselineError):
-            self.apply()
-        self.assertEqual(destination.read_text(), 'provider-data')
-        self.assertFalse(self.host.registrations)
-
-    def test_interrupted_credential_publication_can_retry_without_reusing_wrong_identity(self):
-        config = self.bouncer_config()
-        replace = baseline.os.replace
-        def interrupt_config(source, destination):
-            if destination == config:
-                raise OSError('interrupted publication')
-            return replace(source, destination)
-        with patch.object(baseline.os, 'replace', side_effect=interrupt_config):
-            with self.assertRaises(baseline.BaselineError):
-                self.apply()
-        self.assertIn('<API_KEY>', config.read_text())
-        orphan = config.with_suffix('.yaml.id').read_text().strip()
-        self.assertIn(orphan, self.host.registrations)
-        self.apply()
-        current = config.with_suffix('.yaml.id').read_text().strip()
-        self.assertNotEqual(orphan, current)
-        self.assertIn(self.host.registrations[current], config.read_text())
-        self.assertFalse(list(config.parent.glob('.apex-*')))
-
-    def test_zfs_missing_for_running_kernel_is_an_explicit_incomplete_result(self):
-        self.host.zfs_available = False
-        with self.assertRaisesRegex(baseline.BaselineError, 'reboot.*retry'):
-            self.apply()
-
-    def test_narrow_base_policy_preserves_scope_and_shared_configuration(self):
-        with patch.object(baseline, 'preflight'):
-            result = baseline.apply_base_policy()
-        self.assertEqual(set(result['packages']), {'curl', 'wget', 'git', 'rsyslog', 'ufw'})
-        self.assertFalse(self.host.path('/etc/docker/daemon.json').exists())
-        self.assertEqual(self.host.path('/etc/systemd/journald.conf.d/99-apex.conf').read_text(),
-                         '[Journal]\nSystemMaxUse=200M\n')
-        self.assertIn(['/usr/local/bin/ufw-docker', 'install'], self.host.commands)
-
-    def test_unverified_download_is_not_installed_or_executed(self):
-        with patch.object(baseline.urllib.request, 'urlopen', return_value=io.BytesIO(b'tampered')):
-            with self.assertRaisesRegex(baseline.BaselineError, 'checksum mismatch'):
-                baseline.configure_common()
-        self.assertFalse(self.host.path('/usr/local/bin/ufw-docker').exists())
-        self.assertFalse(any(command[0] == '/usr/local/bin/ufw-docker' for command in self.host.commands))
-
-    def test_recovery_does_not_remove_preexisting_service_masks(self):
-        self.host.masked.add('ufw')
-        with baseline.security_service_guard(['ufw']):
-            self.host.installed['ufw'] = '0.36'
-        self.assertIn('ufw', self.host.masked)
-
-    def test_package_owned_etc_service_file_can_be_unpacked_under_guard(self):
-        unit = self.host.path('/etc/systemd/system/crowdsec-firewall-bouncer.service')
-        with baseline.security_service_guard(['crowdsec-firewall-bouncer-iptables']):
-            self.assertFalse(unit.is_symlink(), 'dpkg must be able to install its own conffile')
-            unit.parent.mkdir(parents=True, exist_ok=True)
-            unit.write_text('[Service]\nExecStart=/usr/bin/crowdsec-firewall-bouncer\n')
-            self.host.installed['crowdsec-firewall-bouncer-iptables'] = '0.0.36'
-            self.host.enabled.add('crowdsec-firewall-bouncer')
-            baseline.run(['systemctl', 'start', 'crowdsec-firewall-bouncer'])
-            self.assertNotIn('crowdsec-firewall-bouncer', self.host.active)
-        self.assertTrue(unit.is_file())
-        self.assertIn('ExecStart=/usr/bin/crowdsec-firewall-bouncer', unit.read_text())
-        self.assertNotIn('crowdsec-firewall-bouncer', self.host.enabled)
-        baseline.run(['systemctl', 'start', 'crowdsec-firewall-bouncer'])
-        self.assertIn('crowdsec-firewall-bouncer', self.host.active)
-
-    def test_disable_failure_retains_guard_and_retry_recovers_it(self):
-        original_command = self.host.command
-        def fail_disable(args, **kwargs):
-            if args == ['systemctl', 'disable', 'crowdsec-firewall-bouncer']:
-                return subprocess.CompletedProcess(args, 1, '', '')
-            return original_command(args, **kwargs)
-        with patch.object(baseline.subprocess, 'run', side_effect=fail_disable):
-            with self.assertRaises(baseline.BaselineError):
-                with baseline.security_service_guard(['crowdsec-firewall-bouncer-iptables']):
-                    self.host.installed['crowdsec-firewall-bouncer-iptables'] = '0.0.36'
-                    self.host.enabled.add('crowdsec-firewall-bouncer')
-        baseline.run(['systemctl', 'start', 'crowdsec-firewall-bouncer'])
-        self.assertNotIn('crowdsec-firewall-bouncer', self.host.active)
-        with baseline.security_service_guard(['crowdsec-firewall-bouncer-iptables']):
-            pass
-        self.assertNotIn('crowdsec-firewall-bouncer', self.host.enabled)
-        baseline.run(['systemctl', 'start', 'crowdsec-firewall-bouncer'])
-        self.assertIn('crowdsec-firewall-bouncer', self.host.active)
-
-    def test_generic_package_failure_does_not_suggest_zfs_reboot(self):
-        original_command = self.host.command
-        def fail_security_packages(args, **kwargs):
-            if args[0] == 'apt-get' and 'crowdsec-firewall-bouncer-iptables' in args and '--simulate' not in args:
-                return subprocess.CompletedProcess(args, 100, '', 'secret source diagnostic')
-            return original_command(args, **kwargs)
-        with patch.object(baseline.subprocess, 'run', side_effect=fail_security_packages):
-            with self.assertRaises(baseline.BaselineError) as failure:
-                self.apply()
-        self.assertNotIn('reboot', str(failure.exception))
-        self.assertNotIn('secret source diagnostic', str(failure.exception))
-
-    def test_unsupported_os_fails_before_package_commands(self):
-        self.host.path('/etc/os-release').write_text('ID=ubuntu\nVERSION_ID=24.04\n')
-        with patch.object(baseline.os, 'geteuid', return_value=0):
-            with self.assertRaisesRegex(baseline.BaselineError, 'Debian 13 amd64'):
-                baseline.apply_baseline()
-        self.assertEqual(self.host.commands, [])
-
-    def test_non_root_fails_before_any_command(self):
-        with patch.object(baseline.os, 'geteuid', return_value=1000):
-            with self.assertRaisesRegex(baseline.BaselineError, 'root'):
-                baseline.apply_baseline()
-        self.assertEqual(self.host.commands, [])
-
-    def test_unsupported_architecture_fails_before_apt_mutation(self):
-        with patch.object(baseline.os, 'geteuid', return_value=0), patch.object(
-                baseline.subprocess, 'run', return_value=subprocess.CompletedProcess([], 0, 'arm64\n', '')) as command:
-            with self.assertRaisesRegex(baseline.BaselineError, 'Debian 13 amd64'):
-                baseline.apply_baseline()
-        self.assertEqual([call.args[0] for call in command.call_args_list], [['dpkg', '--print-architecture']])
-
-
-if __name__ == '__main__':
-    unittest.main()

ci/scripts/init-node/tests/test_provision_contract.py

@@ -1,230 +0,0 @@
-import copy
-import unittest
-
-from engine.provisioning.contract import ContractError, normalize_user_data, parse_bootstrap_env
-
-
-def example():
-    return {
-        'preserve_hostname': False,
-        'hostname': 'metis.a1.example.com',
-        'users': [{
-            'name': 'adam',
-            'sudo': 'ALL=(ALL) NOPASSWD:ALL',
-            'groups': ['sudo', 'docker'],
-            'shell': '/bin/bash',
-            'ssh_authorized_keys': ['ssh-ed25519 PUBLIC_TEST_KEY operator'],
-        }],
-        'write_files': [{
-            'path': '/etc/apex/bootstrap.env',
-            'owner': 'root:root',
-            'permissions': '0600',
-            'content': 'APEX_REPO_NAME="metis.example.com"\n'
-                       'APEX_REPO_URL="https://example.com/team/metis.git"\n',
-        }],
-        'runcmd': ['/usr/local/bin/apex-bootstrap'],
-    }
-
-
-def add_env(doc, text):
-    doc['write_files'][0]['content'] += text
-    return doc
-
-
-class TestBootstrapAssignments(unittest.TestCase):
-    def test_literal_quotes_comments_and_empty_values(self):
-        values = parse_bootstrap_env(
-            '# enrollment\nAPEX_REPO_NAME=metis.example.com\n'
-            "APEX_REPO_REF='release-1'\nAPEX_TIER1_DEVICE=\n")
-        self.assertEqual(values['APEX_REPO_REF'], 'release-1')
-        self.assertEqual(values['APEX_TIER1_DEVICE'], '')
-
-    def test_rejects_shell_forms_without_repeating_input(self):
-        payloads = [
-            'APEX_REPO_NAME=$(touch /tmp/SECRET_SENTINEL)',
-            'APEX_REPO_NAME="${SECRET_SENTINEL}"',
-            'APEX_REPO_NAME=`SECRET_SENTINEL`',
-            'APEX_REPO_NAME=x;SECRET_SENTINEL',
-            'export APEX_REPO_NAME=SECRET_SENTINEL',
-        ]
-        for payload in payloads:
-            with self.subTest(payload=payload):
-                with self.assertRaises(ContractError) as caught:
-                    parse_bootstrap_env(payload)
-                self.assertNotIn('SECRET_SENTINEL', str(caught.exception))
-
-    def test_rejects_duplicate_and_unknown_assignments(self):
-        for text in ('APEX_REPO_NAME=a\nAPEX_REPO_NAME=b\n', 'OTHER_SECRET=value\n'):
-            with self.subTest(text=text):
-                with self.assertRaises(ContractError):
-                    parse_bootstrap_env(text)
-
-    def test_rejects_unquoted_parentheses_and_inline_comments(self):
-        for value in ('(main)', 'main(', 'main)', ' #comment', 'main #comment'):
-            with self.subTest(value=value):
-                with self.assertRaises(ContractError):
-                    parse_bootstrap_env('APEX_REPO_REF=' + value)
-
-    def test_parentheses_and_hashes_remain_quoted_literals(self):
-        for quote in ('"', "'"):
-            for value in ('(main)', '#comment', 'main #comment'):
-                with self.subTest(quote=quote, value=value):
-                    values = parse_bootstrap_env('APEX_REPO_REF=' + quote + value + quote)
-                    self.assertEqual(values['APEX_REPO_REF'], value)
-
-
-class TestUserDataContract(unittest.TestCase):
-    def test_public_repository_and_directory_defaults(self):
-        spec = normalize_user_data(example())
-        self.assertEqual(spec.hostname, 'metis.a1.example.com')
-        self.assertEqual(spec.repository.name, 'metis.example.com')
-        self.assertIsNone(spec.repository.ref)
-        self.assertEqual([tier.mode for tier in spec.tiers], ['directory'] * 3)
-        self.assertEqual(spec.tiers[0].number, 1)
-
-    def test_ref_and_explicit_storage_intent(self):
-        doc = add_env(example(),
-            'APEX_REPO_REF="refs/tags/v1"\n'
-            'APEX_TIER1_MODE=create\nAPEX_TIER1_DEVICE=/dev/disk/by-id/disk-a\n'
-            'APEX_TIER2_MODE=import\nAPEX_TIER2_DEVICE=/dev/disk/by-id/disk-b\n'
-            'APEX_TIER2_POOL_GUID=1234567\n')
-        spec = normalize_user_data(doc)
-        self.assertEqual(spec.repository.ref, 'refs/tags/v1')
-        self.assertEqual(spec.tiers[0].mode, 'create')
-        self.assertEqual(spec.tiers[1].pool_guid, '1234567')
-
-    def test_legacy_device_requires_explicit_mode(self):
-        with self.assertRaisesRegex(ContractError, 'explicit mode'):
-            normalize_user_data(add_env(example(), 'APEX_TIER1_DEVICE=/dev/vdb\n'))
-
-    def test_storage_conflicts_are_rejected(self):
-        cases = [
-            'APEX_TIER1_MODE=create\n',
-            'APEX_TIER1_MODE=import\nAPEX_TIER1_DEVICE=/dev/vdb\n',
-            'APEX_TIER1_MODE=directory\nAPEX_TIER1_DEVICE=/dev/vdb\n',
-            'APEX_TIER1_MODE=create\nAPEX_TIER1_DEVICE=/dev/vdb\nAPEX_TIER1_POOL_GUID=1\n',
-            'APEX_TIER1_MODE=erase\n',
-            'APEX_TIER1_MODE=create\nAPEX_TIER1_DEVICE=/dev/vdb\n'
-            'APEX_TIER2_MODE=create\nAPEX_TIER2_DEVICE=/dev/vdb\n',
-        ]
-        for assignment in cases:
-            with self.subTest(assignment=assignment):
-                with self.assertRaises(ContractError):
-                    normalize_user_data(add_env(example(), assignment))
-
-    def test_arbitrary_cloud_init_operations_are_rejected(self):
-        for field in ('packages', 'bootcmd', 'disk_setup'):
-            doc = example()
-            doc[field] = []
-            with self.subTest(field=field):
-                with self.assertRaises(ContractError):
-                    normalize_user_data(doc)
-        doc = example()
-        doc['runcmd'] = ['touch /tmp/unapproved']
-        with self.assertRaises(ContractError):
-            normalize_user_data(doc)
-
-    def test_standard_list_form_of_bootstrap_command(self):
-        doc = example()
-        doc['runcmd'] = [['/usr/local/bin/apex-bootstrap']]
-        self.assertEqual(normalize_user_data(doc).hostname, doc['hostname'])
-
-    def test_invalid_administrator_and_missing_keys(self):
-        changes = [
-            ('name', 'root'), ('shell', '/bin/sh'), ('groups', ['sudo']),
-            ('sudo', 'ALL=(ALL) ALL'), ('ssh_authorized_keys', []),
-        ]
-        for key, value in changes:
-            doc = example()
-            doc['users'][0][key] = value
-            with self.subTest(key=key):
-                with self.assertRaises(ContractError):
-                    normalize_user_data(doc)
-
-    def test_unapproved_or_duplicate_file_writes(self):
-        for mutate in ('path', 'owner', 'permissions', 'append', 'duplicate'):
-            doc = example()
-            item = doc['write_files'][0]
-            if mutate == 'path':
-                item['path'] = '/etc/shadow'
-            elif mutate == 'owner':
-                item['owner'] = 'adam:adam'
-            elif mutate == 'permissions':
-                item['permissions'] = '0644'
-            elif mutate == 'append':
-                item['append'] = True
-            else:
-                doc['write_files'].append(copy.deepcopy(item))
-            with self.subTest(mutate=mutate):
-                with self.assertRaises(ContractError):
-                    normalize_user_data(doc)
-
-    def test_rejects_wrong_types_and_missing_required_fields(self):
-        cases = [None, [], {'hostname': 'x.example.com'}]
-        for value in cases:
-            with self.subTest(value=value):
-                with self.assertRaises(ContractError):
-                    normalize_user_data(value)
-        doc = example()
-        doc['preserve_hostname'] = 'false'
-        with self.assertRaises(ContractError):
-            normalize_user_data(doc)
-
-    def test_repository_paths_and_embedded_passwords(self):
-        for name, url in [
-            ('../escape', 'https://example.com/repo.git'),
-            ('metis', 'https://user:SECRET_SENTINEL@example.com/repo.git'),
-            ('metis', 'file:///etc/shadow'),
-            ('metis', '-uSECRET_SENTINEL'),
-        ]:
-            doc = example()
-            doc['write_files'][0]['content'] = (
-                f'APEX_REPO_NAME="{name}"\nAPEX_REPO_URL="{url}"\n')
-            with self.subTest(name=name, url=url):
-                with self.assertRaises(ContractError) as caught:
-                    normalize_user_data(doc)
-                self.assertNotIn('SECRET_SENTINEL', str(caught.exception))
-
-    def test_runtime_secrets_are_excluded_from_repr_and_identity(self):
-        doc = example()
-        for path, content, permission in [
-            ('/home/adam/.ssh/id_ed25519', 'SECRET_SENTINEL', '0600'),
-            ('/home/adam/.ssh/known_hosts', 'HOST_TRUST_SENTINEL', '0644'),
-        ]:
-            doc['write_files'].append({
-                'path': path, 'owner': 'adam:adam', 'permissions': permission,
-                'defer': True, 'content': content,
-            })
-        spec = normalize_user_data(doc)
-        text = repr(spec) + repr(spec.identity())
-        self.assertNotIn('SECRET_SENTINEL', text)
-        self.assertNotIn('PUBLIC_TEST_KEY', text)
-        self.assertNotIn('HOST_TRUST_SENTINEL', text)
-        self.assertEqual(spec.files[1].content, 'SECRET_SENTINEL')
-
-    def test_administrator_files_require_defer(self):
-        doc = example()
-        doc['write_files'].append({
-            'path': '/home/adam/.ssh/known_hosts', 'owner': 'adam:adam',
-            'permissions': '0644', 'content': 'example.com ssh-ed25519 VALUE',
-        })
-        with self.assertRaises(ContractError):
-            normalize_user_data(doc)
-
-    def test_contract_does_not_mutate_caller_input(self):
-        doc = example()
-        before = copy.deepcopy(doc)
-        normalize_user_data(doc)
-        self.assertEqual(doc, before)
-
-    def test_identity_changes_for_target_but_not_credentials(self):
-        doc = example()
-        before = normalize_user_data(doc).identity()
-        doc['users'][0]['ssh_authorized_keys'] = ['ssh-ed25519 ANOTHER_PUBLIC_KEY operator']
-        self.assertEqual(normalize_user_data(doc).identity(), before)
-        doc['hostname'] = 'other.a1.example.com'
-        self.assertNotEqual(normalize_user_data(doc).identity(), before)
-
-
-if __name__ == '__main__':
-    unittest.main()

ci/scripts/init-node/tests/test_provision_image.py

@@ -1,37 +0,0 @@
-from pathlib import Path
-import tempfile
-import unittest
-from unittest.mock import patch
-
-from engine.provisioning.image import cloud_enroll, sanitize_files
-from engine.provisioning.state import ProvisionError
-
-
-class ImageTests(unittest.TestCase):
-    def test_incompatible_image_stops_before_any_enrollment(self):
-        for metadata in ('{}', '{"enrollment_interface":999}'):
-            with self.subTest(metadata=metadata), patch.object(Path, 'read_text', return_value=metadata), \
-                    patch('engine.provisioning.target.enroll') as enroll:
-                with self.assertRaisesRegex(ProvisionError, 'image enrollment interface'):
-                    cloud_enroll()
-                enroll.assert_not_called()
-
-    def test_sanitization_removes_cloned_identity_and_enrollment_but_preserves_runner(self):
-        with tempfile.TemporaryDirectory() as temporary:
-            root = Path(temporary).resolve()
-            for relative in ('etc/ssh/ssh_host_ed25519_key', 'etc/ssh/ssh_host_ed25519_key.pub',
-                             'root/.ssh/authorized_keys', 'home/debian/.ssh/authorized_keys',
-                             'etc/machine-id', 'var/lib/dbus/machine-id',
-                             'var/lib/apex-init/enrollment.json', 'var/lib/cloud/instance/user-data.txt',
-                             'usr/local/lib/apex-provisioner/digest/engine/provisioning/target.py'):
-                path = root / relative
-                path.parent.mkdir(parents=True, exist_ok=True)
-                path.write_text('build-specific')
-            sanitize_files(root)
-            self.assertEqual(list((root / 'etc/ssh').glob('ssh_host_*')), [])
-            self.assertEqual((root / 'etc/machine-id').read_text(), '')
-            self.assertFalse((root / 'root/.ssh/authorized_keys').exists())
-            self.assertFalse((root / 'var/lib/apex-init').exists())
-            self.assertFalse((root / 'var/lib/cloud').exists())
-            self.assertTrue((root / 'usr/local/lib/apex-provisioner/digest/engine/provisioning/target.py').is_file())
-            self.assertTrue((root / 'etc/systemd/system/ssh.service.d/10-generate-host-keys.conf').is_file())

ci/scripts/init-node/tests/test_provision_repository.py

@@ -1,132 +0,0 @@
-import os
-from pathlib import Path
-from types import SimpleNamespace
-import subprocess
-import tempfile
-import unittest
-from unittest.mock import patch
-
-from engine.provisioning.contract import Enrollment, Repository
-from engine.provisioning.repository import reconcile_repository
-from engine.provisioning.state import ProvisionError
-
-
-class RepositoryTests(unittest.TestCase):
-    def setUp(self):
-        self.temp = tempfile.TemporaryDirectory()
-        self.addCleanup(self.temp.cleanup)
-        self.root = Path(self.temp.name).resolve()
-        self.upstream = self.root / 'upstream'
-        self.commons = self.root / 'commons'
-        self.srv = self.root / 'srv'
-        self.srv.mkdir()
-        self.home = self.root / 'adam'
-        self.home.mkdir()
-        for path in (self.upstream, self.commons):
-            path.mkdir()
-            self.git(['init', '-q'], path)
-            self.git(['config', 'user.name', 'Test'], path)
-            self.git(['config', 'user.email', 'test@example.invalid'], path)
-        (self.commons / 'marker').write_text('commons')
-        (self.commons / 'etc').mkdir()
-        (self.commons / 'etc/.vimrc').write_text('set number\n')
-        self.git(['add', '.'], self.commons)
-        self.git(['commit', '-qm', 'initial'], self.commons)
-        (self.upstream / 'node.env').write_text('APEX_NODE_FQDN=metis.example.com\nAPEX_ENROLLMENT_INTERFACE=1\n')
-        (self.upstream / 'init.sh').write_text(
-            '#!/bin/sh\nset -eu\ngit config core.hooksPath commons/githooks\n'
-            'printf \'export PATH="%s/commons:$PATH"\\n\' "$PWD" > "$HOME/.bashrc"\n'
-            'ln -sf "$PWD/commons/etc/.vimrc" "$HOME/.vimrc"\n')
-        self.git(['-c', 'protocol.file.allow=always', 'submodule', 'add', str(self.commons), 'commons'], self.upstream)
-        self.git(['add', '.'], self.upstream)
-        self.git(['commit', '-qm', 'initial'], self.upstream)
-        self.spec = Enrollment('metis.example.com', Repository('metis', str(self.upstream), None), (), (), ())
-        self.state = {}
-        self.saved = []
-        self.account = patch('engine.provisioning.repository.pwd.getpwnam',
-                             return_value=SimpleNamespace(pw_uid=os.getuid(), pw_gid=os.getgid(), pw_dir=str(self.home)))
-        self.account.start()
-        self.addCleanup(self.account.stop)
-        self.runner = patch('engine.provisioning.repository.as_adam', side_effect=self.local_run)
-        self.runner.start()
-        self.addCleanup(self.runner.stop)
-
-    @staticmethod
-    def git(args, path):
-        return subprocess.run(['git', *args], cwd=path, text=True, capture_output=True, check=True).stdout.strip()
-
-    def local_run(self, args, cwd=None):
-        if args[0] == 'git':
-            args = ['git', '-c', 'protocol.file.allow=always', *args[1:]]
-        return subprocess.run(args, cwd=cwd, text=True, capture_output=True, check=True,
-                              env={**os.environ, 'HOME': str(self.home)}).stdout.strip()
-
-    def enroll(self):
-        return reconcile_repository(self.spec, self.state, lambda: self.saved.append(True), self.srv)
-
-    def test_repeat_does_not_follow_new_upstream_commit(self):
-        first = self.enroll()
-        (self.upstream / 'new').write_text('upstream advanced')
-        self.git(['add', '.'], self.upstream)
-        self.git(['commit', '-qm', 'advance'], self.upstream)
-        second = self.enroll()
-        self.assertEqual(first['commit'], second['commit'])
-        self.assertFalse((self.srv / 'metis' / 'new').exists())
-        self.assertTrue(self.state['repository']['hook_complete'])
-
-    def test_local_changes_are_preserved_and_stop_initialization(self):
-        self.enroll()
-        path = self.srv / 'metis' / 'node.env'
-        path.write_text('local changes')
-        with self.assertRaisesRegex(ProvisionError, 'local changes'):
-            self.enroll()
-        self.assertEqual(path.read_text(), 'local changes')
-
-    def test_unrecorded_checkout_is_not_adopted(self):
-        (self.srv / 'metis').mkdir()
-        with self.assertRaisesRegex(ProvisionError, 'adoption'):
-            self.enroll()
-
-    def test_noop_hook_cannot_claim_success(self):
-        (self.upstream / 'init.sh').write_text('true\n')
-        self.git(['add', '.'], self.upstream)
-        self.git(['commit', '-qm', 'noop'], self.upstream)
-        with self.assertRaises((ProvisionError, subprocess.CalledProcessError)):
-            self.enroll()
-        self.assertFalse(self.state['repository'].get('hook_complete', False))
-
-    def test_missing_or_unsupported_interface_stops_before_hook(self):
-        for version in ('', 'APEX_ENROLLMENT_INTERFACE=999\n'):
-            with self.subTest(version=version):
-                (self.upstream / 'node.env').write_text('APEX_NODE_FQDN=metis.example.com\n' + version)
-                self.git(['add', '.'], self.upstream)
-                self.git(['commit', '-qm', 'unsupported interface'], self.upstream)
-                self.spec = Enrollment('metis.example.com', Repository('missing-' + str(len(version)), str(self.upstream), None), (), (), ())
-                self.state = {}
-                with self.assertRaisesRegex(ProvisionError, 'enrollment interface'):
-                    self.enroll()
-                self.assertFalse((self.home / '.bashrc').exists())
-                self.assertNotIn('hook_complete', self.state['repository'])
-
-    def test_retry_checks_hook_state_instead_of_completion_marker(self):
-        self.enroll()
-        self.git(['config', 'core.hooksPath', 'wrong-path'], self.srv / 'metis')
-        with self.assertRaisesRegex(ProvisionError, 'hook configuration'):
-            self.enroll()
-
-    def test_ordinary_commons_directory_is_not_a_pinned_submodule(self):
-        self.git(['rm', '-f', 'commons'], self.upstream)
-        (self.upstream / 'commons').mkdir()
-        (self.upstream / 'commons/ordinary').write_text('no submodule')
-        self.git(['add', '.'], self.upstream)
-        self.git(['commit', '-qm', 'ordinary directory'], self.upstream)
-        with self.assertRaisesRegex(ProvisionError, 'pinned Git submodule'):
-            self.enroll()
-
-    def test_mismatched_node_hostname_stops_before_hook(self):
-        (self.upstream / 'node.env').write_text('APEX_NODE_FQDN=wrong.example.com\n')
-        self.git(['add', '.'], self.upstream)
-        self.git(['commit', '-qm', 'wrong'], self.upstream)
-        with self.assertRaisesRegex(ProvisionError, 'hostname'):
-            self.enroll()
-        self.assertNotIn('hook_complete', self.state['repository'])

ci/scripts/init-node/tests/test_provision_security.py

@@ -1,878 +0,0 @@
-import base64
-import copy
-import hashlib
-import json
-import ipaddress
-import re
-from pathlib import Path
-import subprocess
-import tempfile
-import unittest
-from unittest.mock import patch
-
-from engine.provisioning.access import Host
-from engine.provisioning import security
-
-
-# Pinned helper default output, kept independent of the adapter's validation code.
-HELPER_V4 = """# BEGIN UFW AND DOCKER
-*filter
-:ufw-user-forward - [0:0]
-:ufw-docker-logging-deny - [0:0]
-:DOCKER-USER - [0:0]
--A DOCKER-USER -j ufw-user-forward
--A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j RETURN
--A DOCKER-USER -m conntrack --ctstate INVALID -j DROP
--A DOCKER-USER -i docker0 -o docker0 -j ACCEPT
--A DOCKER-USER -j RETURN -s 10.0.0.0/8
--A DOCKER-USER -j RETURN -s 172.16.0.0/12
--A DOCKER-USER -j RETURN -s 192.168.0.0/16
--A DOCKER-USER -j ufw-docker-logging-deny -m conntrack --ctstate NEW -d 10.0.0.0/8
--A DOCKER-USER -j ufw-docker-logging-deny -m conntrack --ctstate NEW -d 172.16.0.0/12
--A DOCKER-USER -j ufw-docker-logging-deny -m conntrack --ctstate NEW -d 192.168.0.0/16
--A DOCKER-USER -j RETURN
--A ufw-docker-logging-deny -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW DOCKER BLOCK] "
--A ufw-docker-logging-deny -j DROP
-COMMIT
-# END UFW AND DOCKER
-"""
-HELPER_V6 = """# BEGIN UFW AND DOCKER
-*filter
-:ufw6-user-forward - [0:0]
-:ufw6-docker-logging-deny - [0:0]
-:DOCKER-USER - [0:0]
--A DOCKER-USER -j ufw6-user-forward
--A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j RETURN
--A DOCKER-USER -m conntrack --ctstate INVALID -j DROP
--A DOCKER-USER -i docker0 -o docker0 -j ACCEPT
--A DOCKER-USER -j RETURN -s fd00::/8
--A DOCKER-USER -j ufw6-docker-logging-deny -m conntrack --ctstate NEW -d fd00::/8
--A DOCKER-USER -j RETURN
--A ufw6-docker-logging-deny -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW DOCKER BLOCK] "
--A ufw6-docker-logging-deny -j DROP
-COMMIT
-# END UFW AND DOCKER
-"""
-
-
-def native_alert(value, scope='Ip', origin='cscli', identifier=1):
-    return {'id': identifier, 'created_at': '2026-10-10T12:00:00Z', 'events_count': 1,
-            'events': [], 'message': 'manual decision', 'scenario': 'manual', 'simulated': False,
-            'source': {'scope': scope, 'value': value},
-            'decisions': [{'id': identifier, 'duration': '4h', 'origin': origin, 'scenario': 'manual',
-                           'scope': scope, 'simulated': False, 'type': 'ban', 'value': value}]}
-
-
-class SecurityHost(Host):
-    """Keep filesystem effects real; substitute Debian commands and services."""
-    def __init__(self, root):
-        super().__init__(root)
-        self.active = False
-        self.services = {name: {'LoadState': 'loaded', 'ActiveState': 'inactive',
-                               'UnitFileState': 'disabled', 'DropInPaths': '',
-                               'FragmentPath': '/usr/lib/systemd/system/' + name + '.service'}
-                         for name in ('ufw', 'crowdsec', 'crowdsec-firewall-bouncer')}
-        self.commands = []
-        self.decisions = []
-        self.runtime_overrides = {}
-        self.runtime_chains = {name: {'DOCKER-USER': ['-N DOCKER-USER'],
-                                      'ADMIN-SENTINEL': ['-N ADMIN-SENTINEL', '-A ADMIN-SENTINEL -j RETURN']}
-                               for name in ('iptables', 'ip6tables')}
-        self.forward_hook = True
-        self.fail = None
-        self.after_command = None
-        self.pending_starts = set()
-        contents = {
-            '/etc/default/ufw': 'IPV6=yes\nDEFAULT_INPUT_POLICY="ACCEPT"\nDEFAULT_OUTPUT_POLICY="ACCEPT"\nDEFAULT_FORWARD_POLICY="DROP"\n',
-            '/etc/ufw/ufw.conf': 'ENABLED=no\n',
-            '/etc/ufw/user.rules': '*filter\n-A ufw-after-logging-forward -j LOG --log-prefix "[UFW BLOCK] " -m limit --limit 3/min --limit-burst 10\nCOMMIT\n',
-            '/etc/ufw/user6.rules': '*filter\n-A ufw6-after-logging-forward -j LOG --log-prefix "[UFW BLOCK] " -m limit --limit 3/min --limit-burst 10\nCOMMIT\n',
-            '/etc/ufw/before.rules': '*filter\nCOMMIT\n',
-            '/etc/ufw/before6.rules': '*filter\nCOMMIT\n',
-            '/etc/ufw/after.rules': '*filter\nCOMMIT\n',
-            '/etc/ufw/after6.rules': '*filter\nCOMMIT\n',
-            '/usr/local/bin/ufw-docker': '#!/bin/sh\n',
-        }
-        for name, content in contents.items():
-            path = self.path(name)
-            path.parent.mkdir(parents=True, exist_ok=True)
-            path.write_text(content)
-        self.conffiles = {'/etc/default/ufw': hashlib.md5(contents['/etc/default/ufw'].encode()).hexdigest()}
-        self.bouncer_conffiles = {}
-        checksums = []
-        for name, content in contents.items():
-            if not name.startswith('/etc/ufw/'):
-                continue
-            filename = name.rsplit('/', 1)[1]
-            template = '/usr/share/ufw/' + ('' if filename == 'ufw.conf' else 'iptables/') + filename
-            path = self.path(template)
-            path.parent.mkdir(parents=True, exist_ok=True)
-            path.write_text(content)
-            checksums.append(hashlib.md5(content.encode()).hexdigest() + '  ' + template.lstrip('/'))
-        manifest = self.path('/var/lib/dpkg/info/ufw.md5sums')
-        manifest.parent.mkdir(parents=True, exist_ok=True)
-        manifest.write_text('\n'.join(checksums) + '\n')
-
-    def load_helper_runtime(self):
-        for executable, filename, fixture in (('iptables', 'after.rules', HELPER_V4),
-                                              ('ip6tables', 'after6.rules', HELPER_V6)):
-            if executable == 'ip6tables' and 'IPV6=yes' not in self.path('/etc/default/ufw').read_text():
-                continue
-            if '# BEGIN UFW AND DOCKER' not in self.path('/etc/ufw/' + filename).read_text():
-                continue
-            logging_chain = 'ufw6-docker-logging-deny' if executable == 'ip6tables' else 'ufw-docker-logging-deny'
-            for chain in ('DOCKER-USER', logging_chain):
-                self.runtime_chains[executable][chain] = ['-N ' + chain] + [
-                    line for line in fixture.splitlines() if line.startswith('-A ' + chain + ' ')]
-
-    def assert_restore_scope(self, args):
-        if '--noflush' not in args:
-            raise AssertionError('whole-table restore is forbidden')
-        if Path(args[-1]).stat().st_mode & 0o777 != 0o600:
-            raise AssertionError('restore input must be private')
-
-
-    def command(self, args, check=True):
-        self.commands.append(args)
-        output, code = '', 0
-        if args[:2] == ['systemctl', 'show']:
-            output = '\n'.join(k + '=' + v for k, v in self.services[args[2]].items())
-        elif args[:2] == ['ufw', 'status']:
-            output = 'Status: ' + ('active' if self.active else 'inactive')
-        elif args[0] == 'dpkg-query':
-            conffiles = self.bouncer_conffiles if args[-1] == 'crowdsec-firewall-bouncer-iptables' else self.conffiles
-            output = '\n'.join(' ' + name + ' ' + digest for name, digest in conffiles.items())
-        elif args[0] in ('iptables', 'ip6tables'):
-            if args[1] == '-C':
-                code = 0 if self.forward_hook else 1
-            elif args[1] == '-S':
-                chains = self.runtime_chains[args[0]]
-                if len(args) == 3:
-                    chain = args[2]
-                    code = 0 if chain in chains else 1
-                    output = self.runtime_overrides.get((args[0], chain), '\n'.join(chains.get(chain, [])) + '\n')
-                else:
-                    output = '-P FORWARD ACCEPT\n' + ('-A FORWARD -j DOCKER-USER\n' if self.forward_hook else '')
-                    output += ''.join(self.runtime_overrides.get((args[0], chain), '\n'.join(lines) + '\n')
-                                      for chain, lines in chains.items())
-            else:
-                raise AssertionError(args)
-        elif args[0] in ('iptables-restore', 'ip6tables-restore'):
-            self.assert_restore_scope(args)
-            executable = args[0].removesuffix('-restore')
-            chains = copy.deepcopy(self.runtime_chains[executable])
-            for line in Path(args[-1]).read_text().splitlines():
-                if line in ('*filter', 'COMMIT'):
-                    continue
-                operation, chain, *rest = line.split()
-                if operation == '-F':
-                    chains[chain] = ['-N ' + chain]
-                elif operation == '-A':
-                    chains[chain].append(line)
-                elif operation == '-X':
-                    del chains[chain]
-                else:
-                    raise AssertionError(line)
-            self.runtime_chains[executable] = chains
-        elif args[0] == '/usr/local/bin/ufw-docker':
-            if args[1] == 'check':
-                code = 0 if '# BEGIN UFW AND DOCKER' in self.path('/etc/ufw/after.rules').read_text() else 1
-            elif args[1] == 'install':
-                if not self.active:
-                    raise RuntimeError('UFW is disabled')
-                for name in ('after.rules', 'after6.rules'):
-                    path = self.path('/etc/ufw/' + name)
-                    path.write_text(path.read_text() + (HELPER_V6 if name == 'after6.rules' else HELPER_V4))
-        elif args[0] == 'ufw':
-            if 'allow' in args and 'proto' in args:
-                if args[1:3] == ['insert', '1'] and not any(
-                        '### tuple ###' in self.path('/etc/ufw/' + name).read_text()
-                        for name in ('user.rules', 'user6.rules')):
-                    if check:
-                        raise RuntimeError("ERROR: Invalid position '1'")
-                    return subprocess.CompletedProcess(args, 1, '', "ERROR: Invalid position '1'")
-                source, port = args[args.index('from') + 1], args[args.index('port') + 1]
-                v6 = ipaddress.ip_network(source, strict=False).version == 6
-                path = self.path('/etc/ufw/user6.rules' if v6 else '/etc/ufw/user.rules')
-                line = '### tuple ### allow tcp ' + port + (' ::/0' if v6 else ' 0.0.0.0/0') + ' any ' + source + ' in comment=' + args[-1].encode().hex() + '\n'
-                path.write_text(path.read_text().replace(line, '') if 'delete' in args else path.read_text() + line)
-            elif args[1] == 'default':
-                key = {'incoming': 'INPUT', 'outgoing': 'OUTPUT', 'routed': 'FORWARD'}[args[3]]
-                path = self.path('/etc/default/ufw')
-                path.write_text(re.sub('DEFAULT_' + key + '_POLICY="[A-Z]+"', 'DEFAULT_' + key + '_POLICY="' + ('DROP' if args[2] == 'deny' else 'ACCEPT') + '"', path.read_text()))
-            elif args[-1] in ('enable', 'disable'):
-                self.active = args[-1] == 'enable'
-                self.path('/etc/ufw/ufw.conf').write_text('ENABLED=' + ('yes' if self.active else 'no') + '\n')
-                if self.active and 'DEFAULT_FORWARD_POLICY="ACCEPT"' in self.path('/etc/default/ufw').read_text():
-                    for name, prefix in (('user.rules', 'ufw'), ('user6.rules', 'ufw6')):
-                        path = self.path('/etc/ufw/' + name)
-                        path.write_text(''.join(line for line in path.read_text().splitlines(keepends=True)
-                                                if not line.startswith('-A ' + prefix + '-after-logging-forward -j LOG ')))
-            elif args[-1] != 'reload':
-                raise AssertionError(args)
-            if self.active and args[-1] in ('enable', 'reload'):
-                self.load_helper_runtime()
-        elif args[0] == 'systemctl':
-            field, value = {'enable': ('UnitFileState', 'enabled'), 'disable': ('UnitFileState', 'disabled'),
-                            'start': ('ActiveState', 'active'), 'stop': ('ActiveState', 'inactive'),
-                            'reset-failed': ('ActiveState', 'inactive')}[args[1]]
-            if args[1] == 'reset-failed' and self.services[args[2]]['ActiveState'] != 'failed':
-                raise AssertionError('reset-failed must only follow an observed failure')
-            self.services[args[2]][field] = value
-            if args[1] == 'stop':
-                self.pending_starts.discard(args[2])
-            if args[1:3] == ['stop', 'ufw']:
-                self.active = False
-        elif args[0] == 'crowdsec':
-            pass
-        elif args[:2] == ['cscli', 'lapi']:
-            code = 0 if self.services['crowdsec']['ActiveState'] == 'active' else 1
-        elif args[:2] == ['cscli', 'decisions']:
-            alerts = self.decisions
-            if isinstance(alerts, list) and all(isinstance(a, dict) and isinstance(a.get('decisions'), list) for a in alerts):
-                if '--all' not in args:
-                    alerts = [a for a in alerts if all(d['origin'] != 'CAPI' for d in a['decisions'])]
-                limit = int(args[args.index('--limit') + 1]) if '--limit' in args else 100
-                if limit:
-                    alerts = alerts[:limit]
-            output = json.dumps(alerts)
-        else:
-            raise AssertionError('Unexpected command: ' + repr(args))
-        result = subprocess.CompletedProcess(args, code, output, '')
-        if self.after_command:
-            self.after_command(args)
-        if check and code:
-            raise RuntimeError('simulated command failure')
-        return result
-
-
-class SecurityTests(unittest.TestCase):
-    def setUp(self):
-        self.temp = tempfile.TemporaryDirectory()
-        self.addCleanup(self.temp.cleanup)
-        self.host = SecurityHost(Path(self.temp.name))
-        self.adapter = security.Security(self.host)
-        self.addCleanup(patch.stopall)
-        patch.object(security, 'HELPER_SHA256', hashlib.sha256(b'#!/bin/sh\n').hexdigest(), create=True).start()
-
-    def test_masked_security_service_refuses_before_mutation(self):
-        self.host.services['crowdsec']['UnitFileState'] = 'masked'
-        before = self.host.path('/etc/default/ufw').read_bytes()
-        with self.assertRaises(security.SecurityError):
-            self.adapter.preflight()
-        self.assertEqual(self.host.path('/etc/default/ufw').read_bytes(), before)
-        self.assertFalse(self.host.active)
-
-    def test_inactive_custom_policy_is_not_mistaken_for_pristine(self):
-        path = self.host.path('/etc/default/ufw')
-        path.write_text(path.read_text() + 'MANAGED_BY_ADMIN=yes\n')
-        with self.assertRaises(security.SecurityError):
-            self.adapter.preflight()
-        self.assertFalse(self.host.active)
-
-    def test_packaged_bouncer_unit_in_etc_can_activate_and_restore(self):
-        name = '/etc/systemd/system/crowdsec-firewall-bouncer.service'
-        path = self.host.path(name)
-        path.parent.mkdir(parents=True)
-        path.write_text('[Service]\nExecStart=/usr/bin/crowdsec-firewall-bouncer\n')
-        self.host.bouncer_conffiles[name] = hashlib.md5(path.read_bytes()).hexdigest()
-        self.host.services['crowdsec-firewall-bouncer']['FragmentPath'] = name
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        self.assertEqual(self.host.services['crowdsec-firewall-bouncer']['ActiveState'], 'active')
-        self.adapter.restore(state, self.persist)
-        self.assertEqual(self.host.services['crowdsec-firewall-bouncer']['ActiveState'], 'inactive')
-        self.assertEqual(state['cleanup'], 'complete')
-
-    def test_modified_or_unmanaged_bouncer_unit_in_etc_refuses_before_activation(self):
-        name = '/etc/systemd/system/crowdsec-firewall-bouncer.service'
-        path = self.host.path(name)
-        path.parent.mkdir(parents=True)
-        path.write_text('[Service]\nExecStart=/usr/bin/crowdsec-firewall-bouncer\n')
-        self.host.services['crowdsec-firewall-bouncer']['FragmentPath'] = name
-        for conffiles in ({}, {name: '0' * 32}):
-            with self.subTest(conffiles=conffiles):
-                self.host.bouncer_conffiles = conffiles
-                with self.assertRaises(security.SecurityError):
-                    self.adapter.preflight()
-                self.assertFalse(self.host.active)
-
-    def test_native_generated_ufw_files_are_verified_against_package_templates(self):
-        self.assertEqual(set(self.host.conffiles), {'/etc/default/ufw'})
-        self.assertTrue(self.adapter.preflight()['pristine'])
-
-    def test_modified_generated_file_or_package_template_is_not_pristine(self):
-        for name in ('/etc/ufw/user.rules', '/usr/share/ufw/iptables/after.rules'):
-            with self.subTest(path=name):
-                path = self.host.path(name)
-                before = path.read_bytes()
-                path.write_bytes(before + b'# modified\n')
-                with self.assertRaises(security.SecurityError):
-                    self.adapter.preflight()
-                path.write_bytes(before)
-
-    def test_active_custom_policy_and_ipv6_rules_are_preserved_in_plan(self):
-        self.host.active = True
-        self.host.path('/etc/default/ufw').write_text('IPV6=yes\nDEFAULT_FORWARD_POLICY="DROP"\n')
-        self.host.path('/etc/ufw/user6.rules').write_text('# existing administrator sentinel\n')
-        plan = self.adapter.preflight()
-        self.assertFalse(plan['pristine'])
-        self.assertTrue(plan['ipv6'])
-        self.assertIn('sentinel', base64.b64decode(plan['files']['/etc/ufw/user6.rules']['data']).decode())
-
-    def test_incomplete_helper_is_rejected(self):
-        self.host.active = True
-        self.host.path('/etc/ufw/after.rules').write_text('# BEGIN UFW AND DOCKER\ncustom\n# END UFW AND DOCKER\n')
-        with self.assertRaises(security.SecurityError):
-            self.adapter.preflight()
-
-    def prepared(self):
-        self.saved = []
-        state = self.adapter.snapshot(self.adapter.preflight(), 22, '192.0.2.10')
-        self.saved.append(copy.deepcopy(state))
-        return state
-
-    def persist(self, state):
-        self.saved.append(copy.deepcopy(state))
-
-    def test_pristine_activation_establishes_security_without_losing_bootstrap(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        self.assertTrue(self.host.active)
-        self.assertIn('DEFAULT_INPUT_POLICY="DROP"', self.host.path('/etc/default/ufw').read_text())
-        self.assertIn('DEFAULT_FORWARD_POLICY="ACCEPT"', self.host.path('/etc/default/ufw').read_text())
-        rules = self.host.path('/etc/ufw/user.rules').read_text()
-        self.assertIn('192.0.2.10', rules)
-        self.assertIn('2222', rules)
-        for service in self.host.services.values():
-            self.assertEqual(service['ActiveState'], 'active')
-            self.assertEqual(service['UnitFileState'], 'enabled')
-        self.adapter.verify(state)
-
-    def test_bootstrap_allowance_exists_when_pristine_firewall_first_activates(self):
-        enabled = []
-        def inspect_enable(args):
-            if args == ['ufw', '--force', 'enable']:
-                rules = self.host.path('/etc/ufw/user.rules').read_text()
-                self.assertIn('192.0.2.10', rules)
-                self.assertIn('2222', rules)
-                self.assertIn('2222', self.host.path('/etc/ufw/user6.rules').read_text())
-                enabled.append(True)
-        self.host.after_command = inspect_enable
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        self.assertEqual(enabled, [True])
-
-    def test_active_policy_survives_activation_and_rollback(self):
-        self.host.active = True
-        self.host.services['ufw'].update(ActiveState='active', UnitFileState='enabled')
-        self.host.path('/etc/ufw/user6.rules').write_text('# IPv6 administrator sentinel\n')
-        before = {name: self.host.path(name).read_bytes() for name in security.FILES}
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        self.assertEqual(self.host.path('/etc/default/ufw').read_bytes(), before['/etc/default/ufw'])
-        self.adapter.restore(state, self.persist)
-        self.assertTrue(self.host.active)
-        for name, content in before.items():
-            self.assertEqual(self.host.path(name).read_bytes(), content)
-
-    def test_peer_range_ban_refuses_bouncer_and_can_restore_access(self):
-        self.host.decisions = [native_alert('192.0.2.0/24', 'Range')]
-        state = self.prepared()
-        with self.assertRaises(security.SecurityError):
-            self.adapter.activate(state, self.persist)
-        self.assertEqual(self.host.services['crowdsec-firewall-bouncer']['ActiveState'], 'inactive')
-        self.adapter.restore(state, self.persist)
-        self.assertFalse(self.host.active)
-        self.assertEqual(self.host.services['crowdsec']['ActiveState'], 'inactive')
-
-    def test_concurrent_edit_is_preserved_and_inactive_access_recovers(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        path = self.host.path('/etc/default/ufw')
-        path.write_text(path.read_text() + 'ADMIN_SENTINEL=yes\n')
-        with self.assertRaises(security.SecurityError):
-            self.adapter.restore(state, self.persist)
-        self.assertFalse(self.host.active)
-        self.assertIn('ADMIN_SENTINEL=yes', path.read_text())
-        self.assertTrue(state['access_recovered'])
-        self.assertEqual(state['cleanup'], 'pending')
-        self.assertEqual(self.host.services['crowdsec-firewall-bouncer']['ActiveState'], 'inactive')
-
-    def test_finalization_removes_only_temporary_bootstrap_allowance(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        self.adapter.finalize(state, self.persist)
-        rules = self.host.path('/etc/ufw/user.rules').read_text()
-        self.assertNotIn('192.0.2.10', rules)
-        self.assertIn('2222', rules)
-        self.adapter.verify(state)
-
-    def test_clean_rollback_restores_inactive_service_states(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        self.adapter.restore(state, self.persist)
-        for service in self.host.services.values():
-            self.assertEqual(service['ActiveState'], 'inactive')
-            self.assertEqual(service['UnitFileState'], 'disabled')
-        self.assertEqual(state['cleanup'], 'complete')
-
-    def test_rollback_restores_live_helper_chains_and_allows_next_preflight(self):
-        for original in (['-N DOCKER-USER'], ['-N DOCKER-USER', '-A DOCKER-USER -j RETURN']):
-            with self.subTest(original=original):
-                for executable in ('iptables', 'ip6tables'):
-                    self.host.runtime_chains[executable]['DOCKER-USER'] = original[:]
-                before = copy.deepcopy(self.host.runtime_chains)
-                state = self.prepared()
-                self.adapter.activate(state, self.persist)
-                self.adapter.restore(state, self.persist)
-                self.assertEqual(self.host.runtime_chains, before)
-                self.assertTrue(self.host.forward_hook)
-                self.assertTrue(self.adapter.preflight()['pristine'])
-
-    def test_preexisting_helper_specific_chain_is_rejected_before_mutation(self):
-        self.host.runtime_chains['ip6tables']['ufw6-docker-logging-deny'] = ['-N ufw6-docker-logging-deny']
-        before = copy.deepcopy(self.host.runtime_chains)
-        with self.assertRaises(security.SecurityError):
-            self.adapter.preflight()
-        self.assertEqual(self.host.runtime_chains, before)
-        self.assertFalse(self.host.active)
-
-    def test_docker_runtime_must_match_preflight_at_capture(self):
-        plan = self.adapter.preflight()
-        self.host.runtime_chains['iptables']['DOCKER-USER'].append('-A DOCKER-USER -j RETURN')
-        with self.assertRaises(security.SecurityError):
-            self.adapter.snapshot(plan, 22, '192.0.2.10')
-        self.assertFalse(self.host.active)
-
-    def test_changed_helper_runtime_is_preserved_while_access_recovers(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        self.host.runtime_chains['iptables']['DOCKER-USER'].append('-A DOCKER-USER -j ACCEPT')
-        before = copy.deepcopy(self.host.runtime_chains)
-        with self.assertRaises(security.SecurityError):
-            self.adapter.restore(state, self.persist)
-        self.assertEqual(self.host.runtime_chains, before)
-        self.assertTrue(state['access_recovered'])
-        self.assertEqual(state['cleanup'], 'pending')
-        self.assertIn('docker-firewall-runtime', state['diagnostics'])
-
-    def test_interrupted_helper_runtime_cleanup_resumes_each_family(self):
-        state = self.prepared()
-        before = copy.deepcopy(self.host.runtime_chains)
-        self.adapter.activate(state, self.persist)
-        def interrupt(args):
-            if args[0] == 'iptables-restore':
-                raise KeyboardInterrupt()
-        self.host.after_command = interrupt
-        with self.assertRaises(KeyboardInterrupt):
-            self.adapter.restore(state, self.persist)
-        self.host.after_command = None
-        self.assertEqual(self.host.runtime_chains['iptables'], before['iptables'])
-        self.assertNotEqual(self.host.runtime_chains['ip6tables'], before['ip6tables'])
-        resumed = copy.deepcopy(self.saved[-1])
-        self.adapter.restore(resumed, self.persist)
-        self.assertEqual(self.host.runtime_chains, before)
-        self.assertEqual(resumed['cleanup'], 'complete')
-        self.assertEqual(sum(args[0] == 'iptables-restore' for args in self.host.commands), 1)
-
-    def test_successful_restore_command_requires_runtime_postcondition(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        native = self.host.command
-        def no_change(args, check=True):
-            if args[0].endswith('tables-restore'):
-                return subprocess.CompletedProcess(args, 0, '', '')
-            return native(args, check)
-        with patch.object(self.host, 'command', side_effect=no_change):
-            with self.assertRaises(security.SecurityError):
-                self.adapter.restore(state, self.persist)
-        self.assertTrue(state['access_recovered'])
-        self.assertEqual(state['cleanup'], 'pending')
-
-    def test_runtime_drift_after_cleanup_persistence_is_not_overwritten(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        changed = []
-        def change_during_persist(value):
-            self.persist(value)
-            if value['helper_runtime'].get('restoring') == 'iptables' and not changed:
-                self.host.runtime_chains['iptables']['DOCKER-USER'].append('-A DOCKER-USER -j ACCEPT')
-                changed.append(True)
-        with self.assertRaises(security.SecurityError):
-            self.adapter.restore(state, change_during_persist)
-        self.assertEqual(self.host.runtime_chains['iptables']['DOCKER-USER'][-1], '-A DOCKER-USER -j ACCEPT')
-        self.assertTrue(state['access_recovered'])
-        self.assertFalse(any(args[0].endswith('tables-restore') for args in self.host.commands))
-
-    def test_runtime_drift_during_final_reload_prevents_false_complete_cleanup(self):
-        self.host.active = True
-        self.host.services['ufw'].update(ActiveState='active', UnitFileState='enabled')
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        def change_after_reload(args):
-            if args == ['ufw', 'reload']:
-                self.host.runtime_chains['iptables']['DOCKER-USER'].append('-A DOCKER-USER -j ACCEPT')
-        self.host.after_command = change_after_reload
-        with self.assertRaises(security.SecurityError):
-            self.adapter.restore(state, self.persist)
-        self.assertEqual(self.host.runtime_chains['iptables']['DOCKER-USER'][-1], '-A DOCKER-USER -j ACCEPT')
-        self.assertEqual(state['cleanup'], 'pending')
-        self.assertTrue(state['access_recovered'])
-
-    def test_legacy_runtime_snapshot_requires_reconciliation_but_recovers_access(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        del state['plan']['helper_runtime_before']
-        del state['helper_runtime']
-        before = copy.deepcopy(self.host.runtime_chains)
-        with self.assertRaises(security.SecurityError):
-            self.adapter.restore(state, self.persist)
-        self.assertEqual(self.host.runtime_chains, before)
-        self.assertTrue(state['access_recovered'])
-        self.assertEqual(state['cleanup'], 'pending')
-
-    def test_disabled_ipv6_runtime_is_preserved_during_helper_cleanup(self):
-        path = self.host.path('/etc/default/ufw')
-        path.write_text(path.read_text().replace('IPV6=yes', 'IPV6=no'))
-        self.host.conffiles['/etc/default/ufw'] = hashlib.md5(path.read_bytes()).hexdigest()
-        before = copy.deepcopy(self.host.runtime_chains)
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        self.adapter.restore(state, self.persist)
-        self.assertEqual(self.host.runtime_chains, before)
-        self.assertFalse(any(args[0] == 'ip6tables-restore' for args in self.host.commands))
-
-    def test_rollback_does_not_restart_preexisting_service_stopped_by_administrator(self):
-        self.host.services['crowdsec'].update(ActiveState='active', UnitFileState='enabled')
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        self.host.services['crowdsec']['ActiveState'] = 'inactive'
-        self.adapter.restore(state, self.persist)
-        self.assertEqual(self.host.services['crowdsec']['ActiveState'], 'inactive')
-        self.assertEqual(self.host.services['crowdsec']['UnitFileState'], 'enabled')
-        self.assertNotIn(['systemctl', 'start', 'crowdsec'], self.host.commands)
-
-    def test_owned_enable_does_not_claim_preexisting_activity_field(self):
-        self.host.services['crowdsec']['ActiveState'] = 'active'
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        self.host.services['crowdsec']['ActiveState'] = 'inactive'
-        self.adapter.restore(state, self.persist)
-        self.assertEqual(self.host.services['crowdsec']['ActiveState'], 'inactive')
-        self.assertEqual(self.host.services['crowdsec']['UnitFileState'], 'disabled')
-
-    def test_rollback_stops_owned_services_that_are_restarting_or_failed(self):
-        for activity in ('activating', 'deactivating', 'failed', 'reloading'):
-            with self.subTest(activity=activity):
-                self.host = SecurityHost(Path(self.temp.name) / activity)
-                self.adapter = security.Security(self.host)
-                state = self.prepared()
-                self.adapter.activate(state, self.persist)
-                self.host.services['crowdsec-firewall-bouncer']['ActiveState'] = activity
-                self.adapter.restore(state, self.persist)
-                self.assertEqual(self.host.services['crowdsec-firewall-bouncer']['ActiveState'], 'inactive')
-                self.assertEqual(state['cleanup'], 'complete')
-
-    def test_owned_inactive_service_stop_cancels_queued_start(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        self.host.services['crowdsec-firewall-bouncer']['ActiveState'] = 'inactive'
-        self.host.pending_starts.add('crowdsec-firewall-bouncer')
-        self.adapter.restore(state, self.persist)
-        self.assertFalse(self.host.pending_starts)
-        self.assertEqual(state['cleanup'], 'complete')
-
-    def test_failed_owned_stop_is_reset_and_verified_inactive(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        def failed_stop(args):
-            if args == ['systemctl', 'stop', 'crowdsec']:
-                self.host.services['crowdsec']['ActiveState'] = 'failed'
-        self.host.after_command = failed_stop
-        self.adapter.restore(state, self.persist)
-        self.assertEqual(self.host.services['crowdsec']['ActiveState'], 'inactive')
-        self.assertIn(['systemctl', 'reset-failed', 'crowdsec'], self.host.commands)
-        self.assertNotIn(['systemctl', 'reset-failed', 'crowdsec-firewall-bouncer'], self.host.commands)
-        self.assertEqual(state['cleanup'], 'complete')
-
-    def test_previously_restored_owned_service_is_rechecked_and_stopped_again(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        self.adapter.restore(state, self.persist)
-        self.host.services['crowdsec-firewall-bouncer']['ActiveState'] = 'activating'
-        self.adapter.restore(state, self.persist)
-        self.assertEqual(self.host.services['crowdsec-firewall-bouncer']['ActiveState'], 'inactive')
-        self.assertEqual(state['cleanup'], 'complete')
-
-    def test_late_owned_service_start_prevents_false_complete_cleanup(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        def late_start(args):
-            if args == ['ufw', 'disable']:
-                self.host.services['crowdsec-firewall-bouncer']['ActiveState'] = 'activating'
-        self.host.after_command = late_start
-        with self.assertRaises(security.SecurityError):
-            self.adapter.restore(state, self.persist)
-        self.assertEqual(state['cleanup'], 'pending')
-        self.assertTrue(state['access_recovered'])
-        self.host.after_command = None
-        self.adapter.restore(state, self.persist)
-        self.assertEqual(self.host.services['crowdsec-firewall-bouncer']['ActiveState'], 'inactive')
-        self.assertEqual(state['cleanup'], 'complete')
-
-    def test_failed_preexisting_service_is_not_stopped_or_reset(self):
-        self.host.services['crowdsec'].update(ActiveState='active', UnitFileState='enabled')
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        self.host.services['crowdsec']['ActiveState'] = 'failed'
-        self.adapter.restore(state, self.persist)
-        self.assertEqual(self.host.services['crowdsec']['ActiveState'], 'failed')
-        self.assertNotIn(['systemctl', 'stop', 'crowdsec'], self.host.commands)
-        self.assertNotIn(['systemctl', 'reset-failed', 'crowdsec'], self.host.commands)
-
-    def test_failed_reset_does_not_claim_service_recovery(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        def remains_failed(args):
-            if args in (['systemctl', 'stop', 'crowdsec'], ['systemctl', 'reset-failed', 'crowdsec']):
-                self.host.services['crowdsec']['ActiveState'] = 'failed'
-        self.host.after_command = remains_failed
-        with self.assertRaises(security.SecurityError):
-            self.adapter.restore(state, self.persist)
-        self.assertEqual(state['cleanup'], 'pending')
-        self.assertTrue(state['access_recovered'])
-
-    def test_changed_previously_restored_service_unit_is_preserved(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        self.adapter.restore(state, self.persist)
-        self.host.services['crowdsec-firewall-bouncer'].update(
-            ActiveState='activating', DropInPaths='/etc/systemd/system/bouncer.service.d/custom.conf')
-        count = len(self.host.commands)
-        with self.assertRaises(security.SecurityError):
-            self.adapter.restore(state, self.persist)
-        self.assertEqual(self.host.services['crowdsec-firewall-bouncer']['ActiveState'], 'activating')
-        self.assertNotIn(['systemctl', 'stop', 'crowdsec-firewall-bouncer'], self.host.commands[count:])
-        self.assertTrue(state['access_recovered'])
-
-    def test_changed_service_unit_is_preserved_and_reported_pending(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        self.host.services['crowdsec']['ActiveState'] = 'failed'
-        self.host.services['crowdsec']['DropInPaths'] = '/etc/systemd/system/crowdsec.service.d/administrator.conf'
-        with self.assertRaises(security.SecurityError):
-            self.adapter.restore(state, self.persist)
-        self.assertEqual(self.host.services['crowdsec']['ActiveState'], 'failed')
-        self.assertTrue(state['access_recovered'])
-        self.assertIn('service:crowdsec:ActiveState', state['diagnostics'])
-
-    def test_redirected_file_does_not_prevent_runtime_access_recovery(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        target = self.host.path('/etc/ufw/after.rules')
-        sentinel = self.host.path('/administrator-owned')
-        sentinel.write_text('do not touch\n')
-        target.unlink()
-        target.symlink_to(sentinel)
-        with self.assertRaises(security.SecurityError):
-            self.adapter.restore(state, self.persist)
-        self.assertFalse(self.host.active)
-        self.assertTrue(state['access_recovered'])
-        self.assertEqual(sentinel.read_text(), 'do not touch\n')
-
-    def test_metadata_change_is_not_silently_overwritten(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        path = self.host.path('/etc/default/ufw')
-        path.chmod(0o600)
-        with self.assertRaises(security.SecurityError):
-            self.adapter.verify(state)
-        with self.assertRaises(security.SecurityError):
-            self.adapter.restore(state, self.persist)
-        self.assertEqual(path.stat().st_mode & 0o777, 0o600)
-        self.assertFalse(self.host.active)
-
-    def test_process_death_after_native_write_preserves_unknown_bytes(self):
-        state = self.prepared()
-        def die(args):
-            if args[:2] == ['ufw', 'default']:
-                raise KeyboardInterrupt()
-        self.host.after_command = die
-        with self.assertRaises(KeyboardInterrupt):
-            self.adapter.activate(state, self.persist)
-        self.host.after_command = None
-        recovered = copy.deepcopy(self.saved[-1])
-        with self.assertRaises(security.SecurityError):
-            security.Security(self.host).restore(recovered, self.persist)
-        self.assertFalse(self.host.active)
-        self.assertTrue(recovered['access_recovered'])
-        self.assertEqual(recovered['cleanup'], 'pending')
-        self.assertIn('interrupted-mutation', recovered['diagnostics'])
-
-    def test_active_firewall_keeps_source_limited_bootstrap_on_ambiguous_recovery(self):
-        self.host.active = True
-        self.host.services['ufw'].update(ActiveState='active', UnitFileState='enabled')
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        self.adapter.finalize(state, self.persist)
-        path = self.host.path('/etc/ufw/user6.rules')
-        path.write_text(path.read_text() + '# concurrent IPv6 sentinel\n')
-        before_restore = len(self.host.commands)
-        with self.assertRaises(security.SecurityError):
-            self.adapter.restore(state, self.persist)
-        self.assertTrue(self.host.active)
-        self.assertTrue(state['access_recovered'])
-        self.assertIn('192.0.2.10', self.host.path('/etc/ufw/user.rules').read_text())
-        self.assertIn('# concurrent IPv6 sentinel', path.read_text())
-        self.assertNotIn(['ufw', 'disable'], self.host.commands)
-        self.assertNotIn(['ufw', 'reload'], self.host.commands[before_restore:])
-
-    def test_preexisting_ssh_rule_is_never_relabelled_or_deleted(self):
-        self.host.active = True
-        self.host.services['ufw'].update(ActiveState='active', UnitFileState='enabled')
-        path = self.host.path('/etc/ufw/user.rules')
-        sentinel = '### tuple ### allow tcp 2222 0.0.0.0/0 any 0.0.0.0/0 in comment=' + b'administrator'.hex() + '\n'
-        path.write_text(path.read_text() + sentinel)
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        self.adapter.finalize(state, self.persist)
-        self.assertIn(sentinel, path.read_text())
-        self.adapter.restore(state, self.persist)
-        self.assertIn(sentinel, path.read_text())
-
-    def test_retry_verification_detects_stopped_bouncer(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        self.host.services['crowdsec-firewall-bouncer']['ActiveState'] = 'inactive'
-        with self.assertRaises(security.SecurityError):
-            self.adapter.verify(state)
-
-    def test_redirected_ufw_enable_file_is_not_written_during_recovery(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        path = self.host.path('/etc/ufw/ufw.conf')
-        sentinel = self.host.path('/administrator-config')
-        sentinel.write_text('do not change\n')
-        path.unlink()
-        path.symlink_to(sentinel)
-        with self.assertRaises(security.SecurityError):
-            self.adapter.restore(state, self.persist)
-        self.assertEqual(sentinel.read_text(), 'do not change\n')
-        self.assertFalse(self.host.active)
-        self.assertTrue(state['access_recovered'])
-
-    def test_compatible_existing_helper_is_preserved(self):
-        self.host.active = True
-        for name in ('after.rules', 'after6.rules'):
-            path = self.host.path('/etc/ufw/' + name)
-            path.write_text(path.read_text() + (HELPER_V6 if name == 'after6.rules' else HELPER_V4))
-        self.host.load_helper_runtime()
-        before = copy.deepcopy(self.host.runtime_chains)
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        self.assertNotIn([security.HELPER, 'install'], self.host.commands)
-        self.adapter.restore(state, self.persist)
-        self.assertEqual(self.host.runtime_chains, before)
-        self.assertFalse(any(args[0].endswith('tables-restore') for args in self.host.commands))
-
-    def test_helper_check_success_does_not_accept_permissive_marker_block(self):
-        self.host.active = True
-        for name in ('after.rules', 'after6.rules'):
-            self.host.path('/etc/ufw/' + name).write_text(
-                '# BEGIN UFW AND DOCKER\n*filter\n-A DOCKER-USER -j ACCEPT\nCOMMIT\n# END UFW AND DOCKER\n')
-        self.assertEqual(self.host.command([security.HELPER, 'check']).returncode, 0)
-        with self.assertRaises(security.SecurityError):
-            self.adapter.preflight()
-
-    def test_valid_helper_files_require_live_rules_and_forward_hook(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        for executable, chain, contents in (
-                ('iptables', 'DOCKER-USER', '-N DOCKER-USER\n-A DOCKER-USER -j RETURN\n'),
-                ('ip6tables', 'DOCKER-USER', '-N DOCKER-USER\n-A DOCKER-USER -j ACCEPT\n'),
-                ('iptables', 'DOCKER-USER', '-N DOCKER-USER\n-A DOCKER-USER -j ACCEPT\n' + '\n'.join(
-                    line for line in HELPER_V4.splitlines() if line.startswith('-A DOCKER-USER ')) + '\n'),
-                ('iptables', 'ufw-docker-logging-deny', '-N ufw-docker-logging-deny\n')):
-            with self.subTest(executable=executable, chain=chain):
-                self.host.runtime_overrides[(executable, chain)] = contents
-                with self.assertRaises(security.SecurityError):
-                    self.adapter.verify(state)
-                self.host.runtime_overrides.clear()
-        self.host.forward_hook = False
-        with self.assertRaises(security.SecurityError):
-            self.adapter.verify(state)
-
-    def test_native_runtime_option_order_is_accepted_without_reordering_rules(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        output = self.host.command(['iptables', '-S', 'DOCKER-USER']).stdout
-        output = output.replace('RELATED,ESTABLISHED', 'ESTABLISHED,RELATED').replace(
-            '-j RETURN -s 10.0.0.0/8', '-s 10.0.0.0/8 -j RETURN')
-        self.host.runtime_overrides[('iptables', 'DOCKER-USER')] = output
-        self.adapter.verify(state)
-        lines = output.splitlines()
-        lines[1], lines[-1] = lines[-1], lines[1]
-        self.host.runtime_overrides[('iptables', 'DOCKER-USER')] = '\n'.join(lines) + '\n'
-        with self.assertRaises(security.SecurityError):
-            self.adapter.verify(state)
-
-    def test_existing_live_ban_is_rejected_before_firewall_changes(self):
-        self.host.services['crowdsec'].update(ActiveState='active', UnitFileState='enabled')
-        self.host.decisions = [native_alert('192.0.2.10')]
-        with self.assertRaises(security.SecurityError):
-            self.prepared()
-        self.assertFalse(self.host.active)
-        self.assertNotIn('2222', self.host.path('/etc/ufw/user.rules').read_text())
-
-    def test_ipv6_peer_requires_ipv6_firewall_support(self):
-        path = self.host.path('/etc/default/ufw')
-        path.write_text(path.read_text().replace('IPV6=yes', 'IPV6=no'))
-        self.host.conffiles['/etc/default/ufw'] = hashlib.md5(path.read_bytes()).hexdigest()
-        with self.assertRaises(security.SecurityError):
-            self.adapter.snapshot(self.adapter.preflight(), 22, '2001:db8::1')
-
-    def test_decision_check_covers_alerts_after_default_limit_and_central_api(self):
-        self.host.services['crowdsec'].update(ActiveState='active', UnitFileState='enabled')
-        harmless = [native_alert('198.51.100.' + str(i), identifier=i) for i in range(1, 101)]
-        for origin in ('cscli', 'CAPI'):
-            with self.subTest(origin=origin):
-                self.host.decisions = harmless + [native_alert('192.0.2.0/24', 'Range', origin, 101)]
-                with self.assertRaises(security.SecurityError):
-                    self.prepared()
-        self.assertFalse(self.host.active)
-
-    def test_non_native_decision_json_is_rejected_instead_of_skipped(self):
-        self.host.services['crowdsec'].update(ActiveState='active', UnitFileState='enabled')
-        for malformed in ([{'scope': 'Ip', 'value': '192.0.2.10'}], [{}], {'decisions': []}, None):
-            with self.subTest(value=malformed):
-                self.host.decisions = malformed
-                with self.assertRaises(security.SecurityError):
-                    self.prepared()
-
-    def test_optional_simulated_field_is_not_required_for_native_decisions(self):
-        self.host.services['crowdsec'].update(ActiveState='active', UnitFileState='enabled')
-        alert = native_alert('198.51.100.3')
-        del alert['decisions'][0]['simulated']
-        self.host.decisions = [alert]
-        self.prepared()
-        alert['decisions'][0]['value'] = '192.0.2.0/24'
-        alert['decisions'][0]['scope'] = 'Range'
-        with self.assertRaises(security.SecurityError):
-            self.prepared()
-
-    def test_surgical_helper_cleanup_preserves_concurrent_metadata_and_text(self):
-        state = self.prepared()
-        self.adapter.activate(state, self.persist)
-        path = self.host.path('/etc/ufw/after.rules')
-        path.write_text(path.read_text() + '# administrator sentinel\n')
-        path.chmod(0o600)
-        with self.assertRaises(security.SecurityError):
-            self.adapter.restore(state, self.persist)
-        self.assertIn('# administrator sentinel', path.read_text())
-        self.assertNotIn('# BEGIN UFW AND DOCKER', path.read_text())
-        self.assertEqual(path.stat().st_mode & 0o777, 0o600)
-
-
-if __name__ == '__main__':
-    unittest.main()

ci/scripts/init-node/tests/test_provision_security_access.py

@@ -1,105 +0,0 @@
-import tempfile
-import unittest
-from pathlib import Path
-
-from engine.provisioning.access import AccessError, Transition
-from engine.provisioning.security import SecurityError
-from test_provision_access import FakeHost, TOKEN
-
-
-from provision_fakes import SecurityBoundary
-
-
-class SecurityAccessTests(unittest.TestCase):
-    def setUp(self):
-        self.temp = tempfile.TemporaryDirectory()
-        self.addCleanup(self.temp.cleanup)
-        self.host = FakeHost(Path(self.temp.name))
-        self.security = SecurityBoundary()
-        self.clock = 1000
-        self.transition = Transition(self.host, now=lambda: self.clock)
-        self.transition.security = self.security
-
-    def begin(self, port=22):
-        return self.transition.execute('begin', TOKEN, port, '192.0.2.10')
-
-    def test_activation_failure_restores_original_access_and_security(self):
-        self.security.fail_activation = True
-        with self.assertRaises((AccessError, SecurityError)):
-            self.begin()
-        self.assertEqual(self.host.ports, [22])
-        self.assertFalse(self.security.active)
-        self.assertFalse(self.host.armed)
-        self.assertEqual(self.transition.load()['status'], 'rolled_back')
-
-    def test_security_cleanup_failure_does_not_prevent_ssh_recovery(self):
-        self.begin()
-        self.transition.execute('finalize', TOKEN)
-        self.security.fail_restore = True
-        self.clock += 301
-        with self.assertRaises(AccessError):
-            self.transition.execute('tick')
-        self.assertEqual(self.host.ports, [22])
-        self.assertTrue(self.host.armed)
-        self.assertEqual(self.transition.load()['status'], 'recovery_pending')
-        self.security.fail_restore = False
-        self.transition.execute('tick')
-        self.assertEqual(self.transition.load()['status'], 'rolled_back')
-        self.assertFalse(self.host.armed)
-
-    def test_committed_retry_cannot_accept_stopped_security(self):
-        self.begin()
-        self.transition.execute('finalize', TOKEN)
-        self.transition.execute('commit', TOKEN)
-        self.security.active = False
-        with self.assertRaises((AccessError, SecurityError)):
-            self.begin(2222)
-        self.assertEqual(self.host.ports, [2222])
-
-    def test_existing_final_ssh_still_activates_security_with_watchdog(self):
-        self.host.existing_final = True
-        self.host.ports = [2222]
-        self.assertEqual(self.begin(2222)['status'], 'transition')
-        self.assertTrue(self.security.active)
-        self.assertTrue(self.host.armed)
-        self.assertEqual(self.host.ports, [2222])
-        self.transition.execute('finalize', TOKEN)
-        self.transition.execute('commit', TOKEN)
-        self.assertFalse(self.host.path('/etc/systemd/system/ssh.service.d/apex-init.conf').exists())
-        self.assertFalse(self.host.armed)
-
-    def test_legacy_committed_access_gains_security_without_reopening_root(self):
-        self.begin()
-        self.transition.execute('finalize', TOKEN)
-        self.transition.execute('commit', TOKEN)
-        state = self.transition.load()
-        state.pop('security')
-        self.transition.save(state)
-        self.security.active = False
-        self.assertEqual(self.begin(2222)['status'], 'transition')
-        self.assertEqual(self.host.ports, [2222])
-        self.assertTrue(self.security.active)
-        self.assertTrue(self.host.armed)
-
-    def test_failed_ssh_restore_still_restores_security_and_retries(self):
-        self.begin()
-        restore = self.host.restore
-        self.host.restore = lambda state: (_ for _ in ()).throw(AccessError('restart failed'))
-        self.clock += 301
-        with self.assertRaises(AccessError):
-            self.transition.execute('tick')
-        self.assertFalse(self.security.active)
-        self.assertTrue(self.host.armed)
-        self.host.restore = restore
-        self.transition.execute('tick')
-        self.assertEqual(self.host.ports, [22])
-        self.assertEqual(self.transition.load()['status'], 'rolled_back')
-
-    def test_security_failure_at_commit_keeps_rollback_armed(self):
-        self.begin()
-        self.transition.execute('finalize', TOKEN)
-        self.security.active = False
-        with self.assertRaises(SecurityError):
-            self.transition.execute('commit', TOKEN)
-        self.assertTrue(self.host.armed)
-        self.assertEqual(self.transition.load()['status'], 'final')

ci/scripts/init-node/tests/test_provision_state.py

@@ -1,80 +0,0 @@
-import json
-import os
-from pathlib import Path
-import tempfile
-import unittest
-
-from engine.provisioning.state import (Journal, ProvisionError, atomic_write,
-                                      check_heartbeat, exclusive_lock)
-
-
-class StateTests(unittest.TestCase):
-    def setUp(self):
-        self.temp = tempfile.TemporaryDirectory()
-        self.addCleanup(self.temp.cleanup)
-        self.root = Path(self.temp.name).resolve()
-
-    def test_atomic_private_file_and_symlink_rejection(self):
-        path = self.root / 'state' / 'record'
-        atomic_write(path, 'first')
-        atomic_write(path, 'second')
-        self.assertEqual(path.read_text(), 'second')
-        self.assertEqual(path.stat().st_mode & 0o777, 0o600)
-        link = self.root / 'redirect'
-        link.symlink_to(path.parent)
-        with self.assertRaises(ProvisionError):
-            atomic_write(link / 'record', 'corruption')
-        self.assertEqual(path.read_text(), 'second')
-
-    def test_lock_rejects_competing_initializer_and_releases(self):
-        path = self.root / 'lock'
-        with exclusive_lock(path):
-            with self.assertRaises(ProvisionError):
-                with exclusive_lock(path):
-                    self.fail('competing lock acquired')
-        with exclusive_lock(path):
-            pass
-
-    def test_failed_stage_is_retryable_and_success_is_reverified(self):
-        path = self.root / 'journal'
-        record = Journal(path, {'hostname': 'metis'}, 'rev', 'digest')
-        with self.assertRaises(RuntimeError):
-            record.stage('package', lambda: (_ for _ in ()).throw(RuntimeError('SECRET')))
-        self.assertNotIn('SECRET', path.read_text())
-        self.assertEqual(json.loads(path.read_text())['failed_stage'], 'package')
-        record = Journal(path, {'hostname': 'metis'}, 'rev', 'digest')
-        results = iter(({'version': 'one'}, {'version': 'two'}))
-        record.stage('package', lambda: next(results))
-        record.stage('package', lambda: next(results))
-        self.assertEqual(record.data['stages']['package']['result']['version'], 'two')
-
-    def test_identity_or_bundle_change_cannot_retarget_partial_run(self):
-        path = self.root / 'journal'
-        Journal(path, {'hostname': 'metis'}, 'rev', 'digest').save()
-        for identity, digest in (({'hostname': 'other'}, 'digest'), ({'hostname': 'metis'}, 'other')):
-            with self.assertRaises(ProvisionError):
-                Journal(path, identity, 'rev', digest)
-
-    def test_unsupported_interface_cannot_resume_and_revisions_survive_retry(self):
-        path = self.root / 'journal'
-        Journal(path, {'hostname': 'metis'}, 'source-rev', 'digest').save()
-        resumed = Journal(path, {'hostname': 'metis'}, 'source-rev', 'digest')
-        self.assertEqual(resumed.data['baseline_revision'], 'source-rev')
-        for version in (None, 999):
-            resumed.data['enrollment_interface'] = version
-            resumed.save()
-            with self.assertRaisesRegex(ProvisionError, 'enrollment interface'):
-                Journal(path, {'hostname': 'metis'}, 'source-rev', 'digest')
-
-    def test_expired_heartbeat_prevents_next_operation(self):
-        beat = self.root / 'heartbeat'
-        beat.touch()
-        os.utime(beat, (100, 100))
-        check_heartbeat(beat, now=110)
-        with self.assertRaises(ProvisionError):
-            check_heartbeat(beat, now=200)
-        record = Journal(self.root / 'journal', {}, 'rev', 'digest')
-        called = []
-        with self.assertRaises(ProvisionError):
-            record.stage('next', lambda: called.append(True), beat)
-        self.assertEqual(called, [])

ci/scripts/init-node/tests/test_provision_storage.py

@@ -1,471 +0,0 @@
-import copy
-import json
-import os
-from pathlib import Path
-import subprocess
-import tempfile
-from types import SimpleNamespace
-import unittest
-from unittest.mock import patch
-
-from engine.provisioning.contract import Enrollment, Repository, Tier
-from engine.provisioning import storage
-from engine.provisioning.storage import StorageError, _persist_swap, reconcile_storage
-
-
-def enrollment(*tiers):
-    return Enrollment('node.example.com', Repository('node', 'https://example.com/node', None),
-                      tuple(tiers) or (Tier(1, 'directory', None, None),), (), ())
-
-
-class TestStorage(unittest.TestCase):
-    def setUp(self):
-        self.temp = tempfile.TemporaryDirectory()
-        self.addCleanup(self.temp.cleanup)
-        self.root = Path(self.temp.name).resolve()
-        self.srv = self.root / 'srv'
-        self.srv.mkdir()
-        self.swap = self.root / 'swapfile'
-        self.fstab = self.root / 'fstab'
-        self.fstab.write_text('# existing entries\n')
-        self.meminfo = self.root / 'meminfo'
-        self.meminfo.write_text('MemTotal:       8388608 kB\n')
-        self.swaps = self.root / 'swaps'
-        self.calls = []
-        self.state = {}
-        self.saved = []
-        self.mounts = [{'target': '/', 'source': '/dev/root', 'fstype': 'ext4', 'maj:min': '8:0'}]
-        self.devices = [self.device('/dev/root', '8:0', 'ROOT', ['/']),
-                        self.device('/dev/spare', '8:16', 'SPARE', [])]
-        self.signatures = []
-        self.pool = None
-        self.pool_member = '/dev/spare'
-        self.pool_mountpoint = self.srv / 'tier-1.node'
-        self.pool_owner = None
-        self.import_listing = ''
-        self.import_full_path_listing = None
-        self.set_active_swaps(['/dev/existing-swap'])
-        self.activate_swap = True
-        self.create_hook = None
-        for name, value in [('SRV', self.srv), ('SWAPFILE', self.swap),
-                            ('FSTAB', self.fstab), ('MEMINFO', self.meminfo), ('SWAPS', self.swaps)]:
-            patcher = patch('engine.provisioning.storage.' + name, value, create=True)
-            patcher.start()
-            self.addCleanup(patcher.stop)
-        patcher = patch('engine.provisioning.storage.subprocess.run', side_effect=self.run_command)
-        patcher.start()
-        self.addCleanup(patcher.stop)
-        patcher = patch('engine.provisioning.storage._device_number', side_effect=self.device_number)
-        patcher.start()
-        self.addCleanup(patcher.stop)
-
-    @staticmethod
-    def device(name, number, serial, mounts):
-        return {'name': name, 'maj:min': number, 'type': 'disk', 'pkname': None,
-                'mountpoints': mounts, 'fstype': None, 'wwn': None,
-                'serial': serial, 'size': 1024 ** 4}
-
-    def device_number(self, path):
-        aliases = {'/dev/alias': '/dev/spare'}
-        path = aliases.get(str(path), str(path))
-        for item in self.devices:
-            if item['name'] == path:
-                return item['maj:min']
-        raise StorageError('selected device is missing or is not a block device')
-
-    def save(self):
-        self.saved.append(copy.deepcopy(self.state))
-
-    def set_active_swaps(self, paths):
-        rows = ['Filename\tType\tSize\tUsed\tPriority']
-        for path in paths:
-            escaped = path.replace('\\', r'\134').replace(' ', r'\040').replace('\t', r'\011').replace('\n', r'\012')
-            rows.append(f'{escaped}\tfile\t2097148\t0\t-2')
-        self.swaps.write_text('\n'.join(rows) + '\n')
-
-    def run_command(self, argv, **kwargs):
-        self.calls.append(argv)
-        output = ''
-        if argv[0] == 'findmnt':
-            output = json.dumps({'filesystems': self.mounts})
-        elif argv[0] == 'lsblk':
-            output = json.dumps({'blockdevices': self.devices})
-        elif argv[0] == 'wipefs':
-            output = json.dumps({'signatures': self.signatures})
-        elif argv[:2] == ['zpool', 'list']:
-            if self.pool:
-                output = f"tier-1\t{self.pool['guid']}\tONLINE\n"
-        elif argv[:2] == ['zpool', 'status']:
-            output = ('  pool: tier-1\n state: ONLINE\nconfig:\n\n'
-                      '\tNAME STATE READ WRITE CKSUM\n'
-                      '\ttier-1 ONLINE 0 0 0\n'
-                      f'\t  {self.pool_member} ONLINE 0 0 0\n\nerrors: No known data errors\n')
-        elif argv[:2] == ['zpool', 'create']:
-            self.assertTrue(self.saved)
-            self.assertEqual(self.saved[-1]['storage']['tiers']['1']['status'], 'intent')
-            if self.create_hook:
-                self.create_hook()
-            self.pool = {'guid': '12345'}
-        elif argv[:2] == ['zpool', 'import']:
-            if '-N' in argv:
-                self.pool = {'guid': argv[-1]}
-            else:
-                output = self.import_listing
-                if self.import_full_path_listing is not None and kwargs['env'].get('ZPOOL_VDEV_NAME_PATH') == '1' \
-                        and kwargs['env'].get('ZPOOL_VDEV_NAME_FOLLOW_LINKS') == '1' \
-                        and 'ZPOOL_VDEV_NAME_GUID' not in kwargs['env']:
-                    output = self.import_full_path_listing
-        elif argv[:2] == ['zfs', 'list']:
-            mounted = 'yes' if any(m['source'] == 'tier-1' for m in self.mounts) else 'no'
-            output = f'tier-1\t{self.pool_mountpoint}\tfilesystem\t{mounted}\n'
-        elif argv[:2] == ['zfs', 'get']:
-            output = (self.pool_owner or self.state['storage']['identity']) + '\n'
-        elif argv[:2] == ['zfs', 'set']:
-            self.pool_owner = argv[2].split('=', 1)[1]
-        elif argv[:2] == ['zfs', 'mount']:
-            self.mounts.append({'target': str(self.srv / 'tier-1.node'), 'source': 'tier-1',
-                                'fstype': 'zfs', 'maj:min': '0:77'})
-        elif argv[0] == 'mkswap':
-            self.assertTrue(self.swap.exists())
-            self.assertEqual(self.saved[-1]['storage']['swap']['status'], 'intent')
-        elif argv == ['swapon', str(self.swap)]:
-            if self.activate_swap:
-                self.set_active_swaps([str(self.swap)])
-        else:
-            self.fail('Unexpected command: ' + repr(argv))
-        return subprocess.CompletedProcess(argv, 0, output, '')
-
-    def test_directory_preserves_contents_and_only_reports_new_directory(self):
-        path = self.srv / 'tier-1.node'
-        path.mkdir()
-        (path / 'keep').write_text('existing data')
-        result = reconcile_storage(enrollment(Tier(1, 'directory', None, None),
-                                              Tier(2, 'directory', None, None)), self.state, self.save)
-        self.assertEqual((path / 'keep').read_text(), 'existing data')
-        self.assertFalse(result['tiers'][0]['created'])
-        self.assertTrue(result['tiers'][1]['created'])
-        self.assertEqual(result['swap']['status'], 'skipped')
-
-    def test_new_directory_mode_is_exact_under_restrictive_umask(self):
-        self.srv.chmod(0o710)
-        previous_umask = os.umask(0o077)
-        try:
-            reconcile_storage(enrollment(), self.state, self.save)
-        finally:
-            os.umask(previous_umask)
-        self.assertEqual((self.srv / 'tier-1.node').stat().st_mode & 0o777, 0o755)
-        self.assertEqual(self.srv.stat().st_mode & 0o777, 0o710)
-
-    def test_preexisting_directory_keeps_restrictive_mode(self):
-        path = self.srv / 'tier-1.node'
-        path.mkdir()
-        path.chmod(0o700)
-        result = reconcile_storage(enrollment(), self.state, self.save)
-        self.assertFalse(result['tiers'][0]['created'])
-        self.assertEqual(path.stat().st_mode & 0o777, 0o700)
-
-    def test_directory_symlink_and_unexpected_mount_are_rejected(self):
-        path = self.srv / 'tier-1.node'
-        path.symlink_to(self.root)
-        with self.assertRaises(StorageError):
-            reconcile_storage(enrollment(), self.state, self.save)
-        path.unlink()
-        path.mkdir()
-        self.mounts.append({'target': str(path), 'source': '/dev/other', 'fstype': 'ext4', 'maj:min': '8:32'})
-        with self.assertRaises(StorageError):
-            reconcile_storage(enrollment(), self.state, self.save)
-
-    def test_pending_directory_ownership_survives_later_tier_failure_until_ack(self):
-        spec = enrollment(Tier(1, 'directory', None, None), Tier(2, 'directory', None, None))
-        first = self.srv / 'tier-1.node'
-        second = self.srv / 'tier-2.node'
-        second.symlink_to(self.root)
-        with self.assertRaises(StorageError):
-            reconcile_storage(spec, self.state, self.save)
-        self.assertTrue(first.is_dir())
-        self.state = copy.deepcopy(self.saved[-1])
-        second.unlink()
-        second.mkdir()
-        result = reconcile_storage(spec, self.state, self.save)
-        self.assertEqual(result['ownership_pending'], [str(first)])
-        self.assertEqual(reconcile_storage(spec, self.state, self.save)['ownership_pending'], [str(first)])
-        storage.acknowledge_storage_ownership(spec, self.state, [str(first)], self.save)
-        self.state = copy.deepcopy(self.saved[-1])
-        self.assertEqual(reconcile_storage(spec, self.state, self.save)['ownership_pending'], [])
-
-    def test_replaced_pending_directory_is_never_adopted_or_acknowledged(self):
-        spec = enrollment()
-        path = self.srv / 'tier-1.node'
-        reconcile_storage(spec, self.state, self.save)
-        path.rename(self.srv / 'original')
-        path.mkdir()
-        with self.assertRaisesRegex(StorageError, 'identity'):
-            reconcile_storage(spec, self.state, self.save)
-        with self.assertRaisesRegex(StorageError, 'identity'):
-            storage.acknowledge_storage_ownership(spec, self.state, [str(path)], self.save)
-
-    def test_preexisting_directory_cannot_be_acknowledged(self):
-        path = self.srv / 'tier-1.node'
-        path.mkdir()
-        spec = enrollment()
-        result = reconcile_storage(spec, self.state, self.save)
-        self.assertEqual(result['ownership_pending'], [])
-        with self.assertRaises(StorageError):
-            storage.acknowledge_storage_ownership(spec, self.state, [str(path)], self.save)
-
-    def test_ownership_acknowledgement_rejects_retargeted_enrollment(self):
-        spec = enrollment()
-        result = reconcile_storage(spec, self.state, self.save)
-        changed = Enrollment('other.example.com', spec.repository, spec.tiers, (), ())
-        with self.assertRaisesRegex(StorageError, 'identity'):
-            storage.acknowledge_storage_ownership(changed, self.state, result['ownership_pending'], self.save)
-        self.assertEqual(reconcile_storage(spec, self.state, self.save)['ownership_pending'],
-                         result['ownership_pending'])
-
-    def test_missing_root_and_occupied_device_never_create_pool(self):
-        for device in ('/dev/missing', '/dev/root', '/dev/spare'):
-            with self.subTest(device=device):
-                self.signatures = [{'type': 'ext4'}] if device == '/dev/spare' else []
-                with self.assertRaises(StorageError):
-                    reconcile_storage(enrollment(Tier(1, 'create', device, None)), {}, lambda: None)
-        self.assertFalse(any(c[:2] == ['zpool', 'create'] for c in self.calls))
-
-    def test_aliases_selected_by_two_tiers_are_rejected_before_creation(self):
-        with self.assertRaises(StorageError):
-            reconcile_storage(enrollment(Tier(1, 'create', '/dev/spare', None),
-                                          Tier(2, 'create', '/dev/alias', None)), self.state, self.save)
-        self.assertFalse(any(c[:2] == ['zpool', 'create'] for c in self.calls))
-
-    def test_create_records_guid_and_retry_verifies_without_recreation(self):
-        spec = enrollment(Tier(1, 'create', '/dev/spare', None))
-        reconcile_storage(spec, self.state, self.save)
-        self.assertEqual(self.state['storage']['tiers']['1']['guid'], '12345')
-        reconcile_storage(spec, self.state, self.save)
-        self.assertEqual(sum(c[:2] == ['zpool', 'create'] for c in self.calls), 1)
-        self.devices[1]['serial'] = 'REPLACED'
-        with self.assertRaises(StorageError):
-            reconcile_storage(spec, self.state, self.save)
-
-    def test_existing_pool_name_without_journal_is_not_ownership(self):
-        self.pool = {'guid': '12345'}
-        with self.assertRaises(StorageError):
-            reconcile_storage(enrollment(Tier(1, 'create', '/dev/spare', None)), self.state, self.save)
-
-    def test_import_requires_exact_guid_and_single_device(self):
-        self.import_listing = ('   pool: tier-1\n     id: 54321\n  state: ONLINE\n'
-                               ' config:\n\n\ttier-1 ONLINE\n\t  /dev/spare ONLINE\n')
-        with self.assertRaises(StorageError):
-            reconcile_storage(enrollment(Tier(1, 'import', '/dev/spare', '12345')), self.state, self.save)
-        self.assertFalse(any(c[:2] == ['zpool', 'import'] and '-N' in c for c in self.calls))
-
-    def test_unknown_swapfile_is_preserved(self):
-        self.set_active_swaps([])
-        self.swap.write_text('do not overwrite')
-        result = reconcile_storage(enrollment(), self.state, self.save)
-        self.assertEqual(self.swap.read_text(), 'do not overwrite')
-        self.assertEqual(result['swap']['status'], 'skipped')
-        self.assertIn('existing', result['swap']['reason'])
-
-    def test_root_disk_descendant_is_protected(self):
-        child = self.device('/dev/root1', '8:1', 'ROOT', ['/boot'])
-        child.update(type='part', pkname='/dev/root', partuuid='root-partition')
-        self.devices.append(child)
-        for device in ('/dev/root', '/dev/root1'):
-            with self.subTest(device=device):
-                with self.assertRaisesRegex(StorageError, 'intersects'):
-                    reconcile_storage(enrollment(Tier(1, 'create', device, None)), self.state, self.save)
-                self.state.clear()
-        self.assertFalse(any(c[:2] == ['zpool', 'create'] for c in self.calls))
-
-    def test_missing_hardware_identity_stops_before_creation(self):
-        self.devices[1]['serial'] = None
-        with self.assertRaisesRegex(StorageError, 'stable'):
-            reconcile_storage(enrollment(Tier(1, 'create', '/dev/spare', None)), self.state, self.save)
-        self.assertFalse(any(c[:2] == ['zpool', 'create'] for c in self.calls))
-
-    def test_identity_rechecked_after_intent_is_saved(self):
-        def changed_device():
-            self.save()
-            self.devices[1]['serial'] = 'REPLACED'
-        with self.assertRaisesRegex(StorageError, 'changed before'):
-            reconcile_storage(enrollment(Tier(1, 'create', '/dev/spare', None)), self.state, changed_device)
-        self.assertFalse(any(c[:2] == ['zpool', 'create'] for c in self.calls))
-
-    def test_created_whole_disk_pool_can_use_a_new_partition(self):
-        def partition_created():
-            child = self.device('/dev/spare1', '8:17', 'SPARE', [])
-            child.update(type='part', pkname='/dev/spare')
-            self.devices.append(child)
-            self.pool_member = '/dev/spare1'
-        self.create_hook = partition_created
-        result = reconcile_storage(enrollment(Tier(1, 'create', '/dev/spare', None)), self.state, self.save)
-        self.assertEqual(result['tiers'][0]['guid'], '12345')
-
-    def importable(self, members='\t  spare ONLINE\n'):
-        self.import_listing = ('   pool: tier-1\n     id: 12345\n  state: ONLINE\n'
-                               ' action: The pool can be imported using its name or numeric identifier.\n'
-                               ' config:\n\n\ttier-1 ONLINE\n' + members)
-
-    def test_imports_declared_guid_without_force_or_mountpoint_rewrite(self):
-        self.importable()
-        result = reconcile_storage(enrollment(Tier(1, 'import', '/dev/spare', '12345')), self.state, self.save)
-        self.assertEqual(result['tiers'][0]['guid'], '12345')
-        self.assertFalse(result['tiers'][0]['created'])
-        self.assertEqual(result['ownership_pending'], [])
-        with self.assertRaises(StorageError):
-            storage.acknowledge_storage_ownership(
-                enrollment(Tier(1, 'import', '/dev/spare', '12345')), self.state,
-                [str(self.pool_mountpoint)], self.save)
-        self.assertIn(['zpool', 'import', '-N', '-d', '/dev/spare', '12345'], self.calls)
-        self.assertFalse(any('-f' in command for command in self.calls))
-        self.assertFalse(any('mountpoint=' in arg for command in self.calls for arg in command))
-
-    def test_multidevice_import_is_rejected(self):
-        self.importable('\t  spare ONLINE\n\t  other ONLINE\n')
-        with self.assertRaisesRegex(StorageError, 'exactly one'):
-            reconcile_storage(enrollment(Tier(1, 'import', '/dev/spare', '12345')), self.state, self.save)
-        self.assertFalse(any(c[:2] == ['zpool', 'import'] and '-N' in c for c in self.calls))
-
-    def partitioned_import_fixture(self):
-        for suffix, number in (('1', '8:17'), ('9', '8:25')):
-            child = self.device('/dev/spare' + suffix, number, None, [])
-            child.update(type='part', pkname='/dev/spare', partuuid='partition-' + suffix)
-            self.devices.append(child)
-        self.pool_member = '/dev/spare1'
-        self.importable()
-        self.import_full_path_listing = self.import_listing.replace('spare ONLINE', '/dev/spare1 ONLINE')
-
-    def test_import_discovery_preserves_partition_path_instead_of_whole_disk_display(self):
-        self.partitioned_import_fixture()
-        with patch.dict(os.environ, {'ZPOOL_VDEV_NAME_GUID': '1'}):
-            result = reconcile_storage(enrollment(Tier(1, 'import', '/dev/spare1', '12345')),
-                                       self.state, self.save)
-        self.assertEqual(result['tiers'][0]['guid'], '12345')
-        self.assertIn(['zpool', 'import', '-N', '-d', '/dev/spare1', '12345'], self.calls)
-
-    def test_import_discovery_never_accepts_a_sibling_partition(self):
-        self.partitioned_import_fixture()
-        with self.assertRaisesRegex(StorageError, 'membership differs'):
-            reconcile_storage(enrollment(Tier(1, 'import', '/dev/spare9', '12345')), self.state, self.save)
-        self.assertFalse(any(c[:2] == ['zpool', 'import'] and '-N' in c for c in self.calls))
-
-    def test_import_discovery_parent_only_output_is_not_partition_proof(self):
-        self.partitioned_import_fixture()
-        self.import_full_path_listing = self.import_listing.replace('spare ONLINE', '/dev/spare ONLINE')
-        with self.assertRaisesRegex(StorageError, 'membership differs'):
-            reconcile_storage(enrollment(Tier(1, 'import', '/dev/spare1', '12345')), self.state, self.save)
-        self.assertFalse(any(c[:2] == ['zpool', 'import'] and '-N' in c for c in self.calls))
-
-    def test_import_does_not_mount_or_reassign_unrelated_mountpoint(self):
-        self.importable()
-        self.pool_mountpoint = '/important-data'
-        with self.assertRaisesRegex(StorageError, 'mountpoint'):
-            reconcile_storage(enrollment(Tier(1, 'import', '/dev/spare', '12345')), self.state, self.save)
-        self.assertFalse(any(c[:2] in (['zfs', 'mount'], ['zfs', 'set']) for c in self.calls))
-
-    def test_swap_skip_reports_low_space_and_unsupported_filesystems(self):
-        self.set_active_swaps([])
-        with patch('engine.provisioning.storage.os.statvfs', return_value=SimpleNamespace(f_bavail=1, f_frsize=4096)):
-            result = reconcile_storage(enrollment(), self.state, self.save)
-        self.assertIn('insufficient', result['swap']['reason'])
-        self.mounts[0]['fstype'] = 'btrfs'
-        result = reconcile_storage(enrollment(), self.state, self.save)
-        self.assertIn('supported', result['swap']['reason'])
-        self.assertFalse(self.swap.exists())
-
-    def test_swap_sizes_allocation_and_records_verified_result(self):
-        self.set_active_swaps([])
-        with patch('engine.provisioning.storage.os.statvfs', return_value=SimpleNamespace(f_bavail=16, f_frsize=1024 ** 3)), \
-                patch('engine.provisioning.storage.os.posix_fallocate', create=True) as allocate, \
-                patch('engine.provisioning.storage._persist_swap') as persist:
-            result = reconcile_storage(enrollment(), self.state, self.save)
-        self.assertEqual(allocate.call_args.args[1:], (0, 4 * 1024 ** 3))
-        self.assertEqual(result['swap'], {'status': 'complete', 'size': 4 * 1024 ** 3})
-        persist.assert_called_once()
-
-    def test_existing_active_swap_preserved_without_cli_inspection(self):
-        before = self.swaps.read_text()
-        result = reconcile_storage(enrollment(), self.state, self.save)
-        self.assertEqual(result['swap'], {'status': 'skipped', 'reason': 'existing active swap preserved'})
-        self.assertEqual(self.swaps.read_text(), before)
-        self.assertFalse(any(command[0] in ('swapon', 'mkswap') for command in self.calls))
-        self.assertFalse(self.swap.exists())
-
-    def test_proc_swaps_empty_and_escaped_paths(self):
-        self.set_active_swaps([])
-        self.assertEqual(storage._active_swaps(), [])
-        self.swaps.write_text('Filename\tType\tSize\tUsed\tPriority\n'
-                              '/swap\\040file\\011name\\012with\\134slash\tfile\t2097148\t0\t-2\n')
-        self.assertEqual(storage._active_swaps(), ['/swap file\tname\nwith\\slash'])
-
-    def test_unobserved_swap_activation_is_not_persisted_or_reported_complete(self):
-        self.set_active_swaps([])
-        self.activate_swap = False
-        with patch('engine.provisioning.storage.os.statvfs', return_value=SimpleNamespace(f_bavail=16, f_frsize=1024 ** 3)), \
-                patch('engine.provisioning.storage.os.posix_fallocate', create=True), \
-                patch('engine.provisioning.storage._persist_swap') as persist:
-            result = reconcile_storage(enrollment(), self.state, self.save)
-        self.assertEqual(result['swap']['status'], 'skipped')
-        self.assertIn('activation did not become observable', result['swap']['reason'])
-        persist.assert_not_called()
-
-    def test_malformed_proc_swaps_stops_optional_allocation(self):
-        for text in ('', 'wrong header\n', 'Filename Type Size Used Priority\n/swapfile file broken 0 -2\n'):
-            with self.subTest(text=text):
-                self.swaps.write_text(text)
-                result = reconcile_storage(enrollment(), self.state, self.save)
-                self.assertEqual(result['swap']['status'], 'skipped')
-                self.assertIn('swap inventory', result['swap']['reason'])
-                self.assertFalse(self.swap.exists())
-
-    def test_small_memory_swap_is_two_gib(self):
-        self.set_active_swaps([])
-        self.meminfo.write_text('MemTotal:       4194304 kB\n')
-        with patch('engine.provisioning.storage.os.statvfs', return_value=SimpleNamespace(f_bavail=16, f_frsize=1024 ** 3)), \
-                patch('engine.provisioning.storage.os.posix_fallocate', create=True) as allocate, \
-                patch('engine.provisioning.storage._persist_swap'):
-            result = reconcile_storage(enrollment(), self.state, self.save)
-        self.assertEqual(allocate.call_args.args[1:], (0, 2 * 1024 ** 3))
-        self.assertEqual(result['swap']['size'], 2 * 1024 ** 3)
-
-    def test_fstab_preserves_existing_content_and_does_not_duplicate_swap(self):
-        actual_lstat = Path.lstat
-
-        def root_owned(path, *args, **kwargs):
-            info = actual_lstat(path, *args, **kwargs)
-            if path == self.fstab:
-                fields = list(info)
-                fields[4] = 0
-                return os.stat_result(fields)
-            return info
-
-        with patch('engine.provisioning.storage.Path.lstat', root_owned), \
-                patch('engine.provisioning.storage.os.fchown'):
-            _persist_swap()
-            _persist_swap()
-        self.assertEqual(self.fstab.read_text(), '# existing entries\n' +
-                         f'{self.swap} none swap sw 0 0\n')
-
-    def test_pool_guid_change_on_retry_is_rejected(self):
-        spec = enrollment(Tier(1, 'create', '/dev/spare', None))
-        reconcile_storage(spec, self.state, self.save)
-        self.pool['guid'] = '67890'
-        with self.assertRaisesRegex(StorageError, 'GUID'):
-            reconcile_storage(spec, self.state, self.save)
-
-    def test_interrupted_create_recovers_only_matching_pool_ownership(self):
-        spec = enrollment(Tier(1, 'create', '/dev/spare', None))
-        reconcile_storage(spec, self.state, self.save)
-        record = self.state['storage']['tiers']['1']
-        record['status'] = 'intent'
-        del record['guid']
-        reconcile_storage(spec, self.state, self.save)
-        self.assertEqual(record['guid'], '12345')
-        self.assertEqual(sum(c[:2] == ['zpool', 'create'] for c in self.calls), 1)
-        self.pool_owner = 'other-enrollment'
-        with self.assertRaisesRegex(StorageError, 'ownership'):
-            reconcile_storage(spec, self.state, self.save)
-
-
-if __name__ == '__main__':
-    unittest.main()

ci/scripts/init-node/tests/test_provision_target.py

@@ -1,87 +0,0 @@
-import contextlib
-import io
-import json
-import os
-from pathlib import Path
-from types import SimpleNamespace
-import tempfile
-import unittest
-from unittest.mock import patch
-
-from engine.provisioning import target
-from engine.provisioning.contract import normalize_user_data
-from engine.provisioning.state import ProvisionError, atomic_write, safe_path
-from tests.test_provision_contract import example
-
-
-class TargetTests(unittest.TestCase):
-    def setUp(self):
-        self.temp = tempfile.TemporaryDirectory()
-        self.addCleanup(self.temp.cleanup)
-        self.root = Path(self.temp.name).resolve()
-
-    def test_invalid_public_key_rejected_before_host_enrollment(self):
-        with self.assertRaisesRegex(ProvisionError, 'cryptographic validation'):
-            target.validate_keys(normalize_user_data(example()))
-
-    def test_administrator_preserves_existing_keys_and_uses_narrow_writes(self):
-        home = self.root / 'home/adam'
-        (home / '.ssh').mkdir(parents=True)
-        (home / '.ssh/authorized_keys').write_text('existing-key\n')
-        account = SimpleNamespace(pw_dir='/home/adam', pw_uid=os.getuid(), pw_gid=os.getgid(), pw_shell='/bin/bash')
-        commands = []
-
-        def command(args, **kwargs):
-            commands.append(args)
-            if args == ['hostname']:
-                return 'provider.example.com'
-            if args == ['id', '-nG', 'adam']:
-                return 'adam sudo docker'
-            return ''
-
-        def mapped(path):
-            path = Path(path)
-            return safe_path(path if path.is_relative_to(self.root) else self.root / str(path).lstrip('/'))
-
-        def write(path, content, mode=0o600, **kwargs):
-            # Keep real file writes; privileged ownership is the isolated boundary.
-            kwargs.pop('uid', None)
-            kwargs.pop('gid', None)
-            atomic_write(mapped(path), content, mode, **kwargs)
-
-        with patch.object(target.pwd, 'getpwnam', return_value=account), \
-                patch.object(target, 'safe_path', side_effect=mapped), \
-                patch.object(target, 'atomic_write', side_effect=write), \
-                patch.object(target, 'run', side_effect=command):
-            spec = normalize_user_data(example())
-            target.administrator(spec)
-            target.administrator(spec)
-        content = (home / '.ssh/authorized_keys').read_text().splitlines()
-        self.assertEqual(content, ['existing-key', 'ssh-ed25519 PUBLIC_TEST_KEY operator'])
-        self.assertIn(['hostnamectl', 'set-hostname', spec.hostname], commands)
-        self.assertEqual((self.root / 'etc/sudoers.d/99-apex-adam').stat().st_mode & 0o777, 0o440)
-        self.assertFalse((self.root / 'etc/machine-id').exists())
-        self.assertFalse((self.root / 'etc/network').exists())
-
-    def test_completion_requires_committed_access_and_preserves_revisions(self):
-        state = self.root / 'state'
-        (state / 'access').mkdir(parents=True)
-        record = {'status': 'enrolled', 'repository': {'commit': 'nodecommit', 'commons_commit': 'commonscommit'}}
-        atomic_write(state / 'enrollment.json', json.dumps(record))
-        atomic_write(state / 'access/state.json', json.dumps({'status': 'final'}))
-        with patch.object(target, 'STATE', state), patch.object(target, 'preflight'), \
-                contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()):
-            self.assertEqual(target.main(['complete']), 1)
-            self.assertEqual(json.loads((state / 'enrollment.json').read_text())['status'], 'enrolled')
-            atomic_write(state / 'access/state.json', json.dumps({'status': 'committed'}))
-            self.assertEqual(target.main(['complete']), 0)
-        actual = json.loads((state / 'enrollment.json').read_text())
-        self.assertEqual(actual['external_access'], 'verified')
-        self.assertEqual(actual['repository'], record['repository'])
-
-    def test_preflight_rejects_wrong_os_before_any_enrollment(self):
-        with patch.object(target.os, 'geteuid', return_value=0), \
-                patch.object(target.platform, 'machine', return_value='x86_64'), \
-                patch.object(Path, 'read_text', return_value='ID=ubuntu\nVERSION_ID="24.04"\n'):
-            with self.assertRaisesRegex(ProvisionError, 'Debian 13'):
-                target.preflight()

ci/scripts/init-node/tests/test_provision_transport.py

@@ -1,517 +0,0 @@
-import io
-import json
-import os
-from pathlib import Path
-import shlex
-import subprocess
-import sys
-import tarfile
-import tempfile
-import unittest
-from unittest.mock import patch
-
-from engine.provisioning.transport import (
-    SSH, TransportError, HandoverUncertain, build_bundle, destination, local_file, protected_handover,
-    initialize,
-    REMOTE_PATH_GUARD, remote_atomic_input,
-)
-
-
-class TransportTests(unittest.TestCase):
-    def test_durable_runner_checks_access_after_seed_before_enrollment(self):
-        for compatible in (False, True):
-            with self.subTest(compatible=compatible), tempfile.TemporaryDirectory() as directory:
-                root = Path(directory)
-                source = root / 'user-data'
-                source.write_bytes(b'opaque input')
-                stage = root / 'stage'
-                stage.mkdir()
-                installed = root / 'bundle'
-                seed = installed / 'engine/provisioning/seed.sh'
-                seed.parent.mkdir(parents=True)
-                seed.write_text('printf "seed\\n" >> "$APEX_TEST_EVENTS"\n')
-                binaries = root / 'bin'
-                binaries.mkdir()
-                python = binaries / 'python3'
-                python.write_text('#!' + sys.executable + '\n' + '''
-import os, sys
-module, operation = sys.argv[2:4]
-event = 'access-' + operation if module.endswith('.access') else operation
-with open(os.environ['APEX_TEST_EVENTS'], 'a') as stream:
-    stream.write(event + '\\n')
-if event == 'access-preflight' and os.environ['APEX_TEST_COMPATIBLE'] == 'False':
-    sys.exit(42)
-if event == 'enroll':
-    sys.exit(43)
-''')
-                python.chmod(0o755)
-                events = root / 'events'
-                environment = dict(os.environ, PATH=str(binaries) + os.pathsep + os.environ['PATH'],
-                                   APEX_TEST_EVENTS=str(events), APEX_TEST_COMPATIBLE=str(compatible))
-                class Remote:
-                    user, host, identity = 'root', 'example.com', None
-                    resolved_port = 22
-                    def __init__(self, *args):
-                        pass
-                    def resolve(self):
-                        pass
-                    def run(self, script, data=None, **kwargs):
-                        value = b''
-                        if 'ssh_host_' in script:
-                            value = b'192.0.2.1 50000 192.0.2.2 22\nssh-ed25519 AAAA fixture\n'
-                        elif 'mktemp -d /var/tmp/apex-init.' in script:
-                            value = b'/var/tmp/apex-init.0123456789\n'
-                        elif 'systemd-run --quiet' in script:
-                            arguments = shlex.split(script)
-                            wrapper = arguments[-1].replace('/usr/local/lib/apex-provisioner/' + 'a' * 64, str(installed))
-                            wrapper = wrapper.replace('/var/tmp/apex-init.0123456789', str(stage))
-                            wrapper = wrapper.replace('/etc/tmpfiles.d/', str(root) + '/')
-                            subprocess.run(['/bin/sh', '-c', wrapper], env=environment,
-                                           capture_output=True, timeout=30)
-                        elif script.startswith('if test -f /var/tmp'):
-                            value = (stage / 'result').read_bytes()
-                        return subprocess.CompletedProcess([], 0, value, b'')
-                with patch('engine.provisioning.transport.SSH', Remote), \
-                        patch('engine.provisioning.transport.build_bundle', return_value=(b'bundle', 'a' * 64, 'b' * 40)), \
-                        patch('engine.provisioning.transport.shutil.which', return_value='/usr/bin/ssh'):
-                    with self.assertRaises(TransportError):
-                        initialize(root, 'root@example.com', source, progress=lambda message: None)
-                expected = ['seed', 'validate', 'access-preflight']
-                if compatible:
-                    expected.append('enroll')
-                self.assertEqual(events.read_text().splitlines(), expected)
-                self.assertEqual((stage / 'result').read_text().strip(), '43' if compatible else '42')
-
-    def test_rejects_options_shell_fragments_and_invalid_ports(self):
-        for value in ['root@-oProxyCommand=id', 'root@host;id', '-oProxyCommand=id@host',
-                      'bob;id@host', 'bob name@host', 'bob\n@host', 'bob/other@host',
-                      'bob$(id)@host', '@host', '9bob@host', 'a' * 33 + '@host',
-                      'root@host\nfoo', 'host', 'debian@host -oProxyCommand=id']:
-            with self.subTest(value=value), self.assertRaises(TransportError):
-                destination(value)
-        for port in ['0', '65536', '-1', '22;id', '2.5', '']:
-            with self.subTest(port=port), self.assertRaises(TransportError):
-                destination('root@example.com', port)
-        self.assertEqual(destination('adam@node', 2222), ('adam', 'node', 2222))
-
-    def test_existing_bootstrap_accounts_are_accepted(self):
-        for user in ['root', 'adam', 'debian', 'bob', 'cloud-admin', '_admin', 'Admin.1']:
-            with self.subTest(user=user):
-                self.assertEqual(destination(user + '@example.com'), (user, 'example.com', None))
-        self.assertEqual(destination('debian@[2001:db8::1]', 22), ('debian', '2001:db8::1', 22))
-
-    def test_nonroot_bootstrap_uses_noninteractive_sudo_and_preserves_stdin(self):
-        calls = []
-        def runner(args, **kwargs):
-            calls.append((args, kwargs))
-            return subprocess.CompletedProcess(args, 0, b'root verified', b'')
-        ssh = SSH('debian@host', runner=runner)
-        self.assertEqual(ssh.run('id -u', b'opaque input').stdout, b'root verified')
-        args, kwargs = calls[0]
-        self.assertEqual(args[args.index('-l') + 1], 'debian')
-        self.assertEqual(shlex.split(args[-1]), ['sudo', '-n', '--', 'sh', '-c', 'id -u'])
-        self.assertEqual(kwargs['input'], b'opaque input')
-        ssh.run('printf %s "$SSH_CONNECTION"', privileged=False)
-        self.assertEqual(shlex.split(calls[1][0][-1])[:2], ['sh', '-c'])
-
-    def test_sudo_cleared_connection_environment_uses_unprivileged_fallback(self):
-        class PreflightComplete(Exception):
-            pass
-        with tempfile.TemporaryDirectory() as directory:
-            root = Path(directory)
-            source = root / 'user-data'
-            source.write_bytes(b'opaque input')
-            release = root / 'os-release'
-            release.write_text('ID=debian\nVERSION_ID=13\n')
-            systemd_run = root / 'systemd-run'
-            systemd_run.write_text('#!/bin/sh\nexit 0\n')
-            systemd_run.chmod(0o755)
-            environment = dict(os.environ, PATH=str(root) + os.pathsep + os.environ['PATH'])
-            environment.pop('SSH_CONNECTION', None)
-            fallbacks = []
-            def runner(args, **kwargs):
-                if '-G' in args:
-                    return subprocess.CompletedProcess(args, 0, 'hostname 192.0.2.2\nport 2222\n', '')
-                command = shlex.split(args[-1])
-                script = command[-1]
-                if 'ssh_host_' in script:
-                    self.assertEqual(command[:3], ['sudo', '-n', '--'])
-                    tools = ('id() { printf 0; }; dpkg() { printf amd64; }; '
-                             "cat() { printf 'ssh-ed25519 AAAA fixture\\n'; }; ")
-                    script = script.replace('/etc/os-release', shlex.quote(str(release)))
-                    result = subprocess.run(['sh', '-c', tools + script], capture_output=True, env=environment)
-                    self.assertEqual(result.returncode, 0, result.stderr)
-                    return result
-                if 'SSH_CONNECTION' in script:
-                    self.assertEqual(command[:2], ['sh', '-c'])
-                    fallbacks.append(script)
-                    return subprocess.run(command, capture_output=True,
-                                          env=dict(environment, SSH_CONNECTION='192.0.2.1 50000 192.0.2.2 2222'))
-                self.assertIn('/var/lib/apex-init/remote.json', script)
-                self.assertIn('ssh-ed25519 AAAA', transport.trust_file.read_text())
-                raise PreflightComplete
-            transport = SSH('adam@example.com', runner=runner)
-            with patch('engine.provisioning.transport.SSH', return_value=transport), \
-                    patch('engine.provisioning.transport.build_bundle', return_value=(b'bundle', 'a' * 64, 'b' * 40)), \
-                    patch('engine.provisioning.transport.shutil.which', return_value='/usr/bin/ssh'):
-                with self.assertRaises(PreflightComplete):
-                    initialize(root, 'adam@example.com', source, progress=lambda message: None)
-            self.assertEqual(len(fallbacks), 1)
-
-    def test_remote_script_quoting_transports_literal_values(self):
-        received = []
-        def runner(args, **kwargs):
-            command = shlex.split(args[-1])
-            self.assertEqual(command[:2], ['sh', '-c'])
-            result = subprocess.run(command, input=kwargs.get('input'), capture_output=True)
-            received.append(result.stdout)
-            return result
-        value = "spaces '$HOME' `echo unsafe`; $(echo unsafe)"
-        SSH('root@host', runner=runner).run('printf %s ' + shlex.quote(value))
-        self.assertEqual(received, [value.encode()])
-
-    def test_secret_bytes_go_over_stdin_and_diagnostics_are_redacted(self):
-        secret = b'private key material'
-        def runner(args, **kwargs):
-            self.assertEqual(kwargs['input'], secret)
-            self.assertNotIn(secret.decode(), repr(args))
-            return subprocess.CompletedProcess(args, 1, b'', secret)
-        with self.assertRaises(TransportError) as caught:
-            SSH('root@host', runner=runner).run('cat >/dev/null', secret)
-        self.assertNotIn(secret.decode(), str(caught.exception))
-
-    def test_fresh_login_disables_mux_password_and_pins_verified_identity(self):
-        calls = []
-        def runner(args, **kwargs):
-            calls.append(args)
-            return subprocess.CompletedProcess(args, 0, b'', b'')
-        ssh = SSH('root@original-alias', runner=runner)
-        ssh.resolved_host = '192.0.2.42'
-        ssh.trust_file = Path('/tmp/trusted')
-        ssh.fresh_login()
-        args = calls[0]
-        self.assertIn('-T', args)
-        for option in ('ControlMaster=no', 'ControlPath=none', 'BatchMode=yes',
-                       'PreferredAuthentications=publickey', 'StrictHostKeyChecking=yes',
-                       'Hostname=192.0.2.42', 'HostKeyAlias=apex-init-verified-target'):
-            self.assertIn(option, args)
-        self.assertEqual(args[args.index('-l') + 1], 'adam')
-        self.assertEqual(args[args.index('-p') + 1], '2222')
-
-    def test_failed_first_fresh_login_rolls_back_without_finalizing(self):
-        calls = []
-        class FailedSSH:
-            def fresh_login(self):
-                raise TransportError('key rejected')
-        def invoke(operation, *args):
-            calls.append(operation)
-            return {'status': 'transition'}
-        with self.assertRaises(TransportError):
-            protected_handover(FailedSSH(), invoke, 'a' * 32, 22, '192.0.2.1')
-        self.assertEqual(calls, ['begin', 'rollback'])
-
-    def test_failed_second_fresh_login_never_commits(self):
-        calls = []
-        class SecondFailure:
-            count = 0
-            def fresh_login(self):
-                self.count += 1
-                if self.count == 2:
-                    raise TransportError('final policy rejected login')
-        def invoke(operation, *args):
-            calls.append(operation)
-            return {'status': 'transition'}
-        with self.assertRaises(TransportError):
-            protected_handover(SecondFailure(), invoke, 'a' * 32, 22, '192.0.2.1')
-        self.assertEqual(calls, ['begin', 'finalize', 'rollback'])
-
-    def test_two_successful_fresh_logins_precede_commit(self):
-        calls = []
-        class VerifiedSSH:
-            def fresh_login(self):
-                calls.append('fresh')
-        def invoke(operation, *args):
-            calls.append(operation)
-            return {'status': 'transition'}
-        protected_handover(VerifiedSSH(), invoke, 'a' * 32, 22, '192.0.2.1')
-        self.assertEqual(calls, ['begin', 'fresh', 'finalize', 'fresh', 'commit'])
-
-    def test_lost_commit_reply_is_reconciled_without_rolling_back_committed_access(self):
-        from engine.provisioning.access import Transition
-        from tests.test_provision_access import FakeHost
-        with tempfile.TemporaryDirectory() as directory:
-            host = FakeHost(Path(directory))
-            transition = Transition(host, now=lambda: 1000, security=host.security)
-            calls = []
-            class VerifiedSSH:
-                def fresh_login(self):
-                    pass
-            def invoke(operation, token, port, address, fresh):
-                calls.append(operation)
-                if operation == 'status':
-                    self.assertTrue(fresh)
-                    return transition.load()
-                result = transition.execute(operation, token, port, address)
-                if operation == 'commit':
-                    raise TransportError('SSH reply lost')
-                return result
-            protected_handover(VerifiedSSH(), invoke, 'a' * 32, 22, '192.0.2.1')
-            self.assertEqual(transition.load()['status'], 'committed')
-            self.assertEqual(host.ports, [2222])
-            self.assertFalse(host.armed)
-            self.assertEqual(host.restored, 0)
-            self.assertEqual(calls, ['begin', 'finalize', 'commit', 'status'])
-
-    def test_known_uncommitted_lost_reply_still_rolls_back(self):
-        calls = []
-        class VerifiedSSH:
-            def fresh_login(self):
-                pass
-        def invoke(operation, *args):
-            calls.append(operation)
-            if operation == 'commit':
-                raise TransportError('SSH failed before commit')
-            return {'status': 'final' if operation == 'status' else 'transition', 'token': 'a' * 32}
-        with self.assertRaises(TransportError):
-            protected_handover(VerifiedSSH(), invoke, 'a' * 32, 22, '192.0.2.1')
-        self.assertEqual(calls, ['begin', 'finalize', 'commit', 'status', 'rollback'])
-
-    def test_lost_commit_reply_reports_verified_or_uncertain_resume_endpoints(self):
-        from engine.provisioning.access import Transition
-        from tests.test_provision_access import FakeHost
-        for unreachable in (False, True):
-            with self.subTest(unreachable=unreachable), tempfile.TemporaryDirectory() as directory:
-                root = Path(directory)
-                source = root / 'user-data'
-                source.write_bytes(b'opaque input')
-                host = FakeHost(root)
-                transition = Transition(host, now=lambda: 1000, security=host.security)
-                messages, commands = [], []
-                def runner(args, **kwargs):
-                    if '-G' in args:
-                        return subprocess.CompletedProcess(args, 0, 'hostname 192.0.2.2\nport 22\n', '')
-                    command = shlex.split(args[-1])
-                    script = command[-1]
-                    commands.append(script)
-                    value = b''
-                    if 'ssh_host_' in script:
-                        value = b'192.0.2.1 50000 192.0.2.2 22\nssh-ed25519 AAAA fixture\n'
-                    elif 'mktemp -d /var/tmp/apex-init.' in script:
-                        value = b'/var/tmp/apex-init.0123456789\n'
-                    elif script.startswith('if test -f /var/tmp'):
-                        value = b'0'
-                    elif script.startswith('cd ') and 'engine.provisioning.access' in script:
-                        access_args = shlex.split(script)[3:]
-                        operation = access_args[3]
-                        token = access_args[access_args.index('--token') + 1]
-                        port = int(access_args[access_args.index('--original-port') + 1]) if '--original-port' in access_args else None
-                        address = access_args[access_args.index('--client-address') + 1] if port else None
-                        value = json.dumps(transition.execute(operation, token, port, address)).encode()
-                        if operation == 'commit':
-                            return subprocess.CompletedProcess(args, 255, b'', b'connection lost')
-                    elif 'cat /var/lib/apex-init/access/state.json' in script:
-                        self.assertEqual(args[args.index('-l') + 1], 'adam')
-                        self.assertEqual(args[args.index('-p') + 1], '2222')
-                        self.assertEqual(command[:3], ['sudo', '-n', '--'])
-                        if unreachable:
-                            return subprocess.CompletedProcess(args, 255, b'', b'connection lost')
-                        value = json.dumps(transition.load()).encode()
-                    elif 'engine.provisioning.target complete' in script:
-                        value = b'{"status": "complete"}'
-                    return subprocess.CompletedProcess(args, 0, value, b'')
-                def transport(*args):
-                    return SSH(*args, runner=runner)
-                with patch('engine.provisioning.transport.SSH', side_effect=transport), \
-                        patch('engine.provisioning.transport.build_bundle', return_value=(b'bundle', 'a' * 64, 'b' * 40)), \
-                        patch('engine.provisioning.transport.shutil.which', return_value='/usr/bin/ssh'):
-                    if unreachable:
-                        with self.assertRaises(HandoverUncertain):
-                            initialize(root, 'debian@example.com', source, progress=messages.append)
-                        self.assertTrue(any('outcome is unknown' in message for message in messages))
-                        self.assertIn('adam@example.com --port 2222', messages[-2])
-                        self.assertIn('debian@example.com --port 22', messages[-1])
-                        self.assertFalse(any('Safe resume' in message or 'pending rollback' in message for message in messages))
-                    else:
-                        self.assertEqual(initialize(root, 'debian@example.com', source, progress=messages.append), {'status': 'complete'})
-                        self.assertIn('Safe resume: ./apex init adam@example.com --port 2222', messages[-1])
-                self.assertEqual(transition.load()['status'], 'committed')
-                self.assertFalse(host.armed)
-                self.assertFalse(any('access rollback' in script for script in commands))
-
-    def test_bundle_excludes_repository_and_environment_secrets(self):
-        with tempfile.TemporaryDirectory() as directory:
-            root = Path(directory)
-            (root / 'engine/provisioning').mkdir(parents=True)
-            (root / 'engine/__init__.py').write_text('')
-            (root / 'engine/provisioning/seed.sh').write_text('#!/bin/sh\n')
-            (root / '.env').write_text('private secret')
-            (root / 'engine/private.env').write_text('private secret')
-            data, digest, revision = build_bundle(root)
-            with tarfile.open(fileobj=io.BytesIO(data)) as archive:
-                self.assertEqual(set(archive.getnames()), {'engine/__init__.py', 'engine/provisioning/seed.sh'})
-            self.assertNotIn(b'private secret', data)
-            self.assertEqual(build_bundle(root)[1], digest)
-
-    def test_missing_input_is_a_safe_validation_error(self):
-        with self.assertRaises(TransportError):
-            local_file('/nonexistent/apex-user-data', 'User-data')
-
-    def test_init_from_node_checkout_does_not_resolve_local_identity(self):
-        from engine import cli
-        root = Path(__file__).resolve().parents[1]
-        with patch.object(cli, '_resolve_layout', return_value=(str(root), str(root), None, True)), \
-                patch.object(cli, 'resolve', side_effect=AssertionError('local identity used')):
-            with self.assertRaises(SystemExit) as caught:
-                cli.main(['init', '--help'])
-        self.assertEqual(caught.exception.code, 0)
-
-    def test_dead_enrollment_unit_stops_polling_and_removes_input(self):
-        with tempfile.TemporaryDirectory() as directory:
-            root = Path(directory)
-            source = root / 'user-data'
-            source.write_bytes(b'opaque input')
-            calls = []
-            class Remote:
-                user, host, identity = 'root', 'example.com', None
-                def __init__(self, *args):
-                    self.resolved_port = 22
-                def resolve(self):
-                    pass
-                def run(self, script, data=None, **kwargs):
-                    calls.append(script)
-                    if 'ssh_host_' in script:
-                        value = b'192.0.2.1 50000 192.0.2.2 22\nssh-ed25519 AAAA fixture\n'
-                    elif 'mktemp -d /var/tmp/apex-init.' in script:
-                        value = b'/var/tmp/apex-init.0123456789\n'
-                    elif script.startswith('if test -f /var/tmp'):
-                        value = b'stopped'
-                    else:
-                        value = b''
-                    return subprocess.CompletedProcess([], 0, value, b'')
-            messages = []
-            with patch('engine.provisioning.transport.SSH', Remote), \
-                    patch('engine.provisioning.transport.build_bundle', return_value=(b'bundle', 'a' * 64, 'b' * 40)), \
-                    patch('engine.provisioning.transport.shutil.which', return_value='/usr/bin/ssh'):
-                with self.assertRaises(TransportError) as caught:
-                    initialize(root, 'root@example.com', source, progress=messages.append)
-            self.assertIn('/runner.log', str(caught.exception))
-            self.assertTrue(any(script.startswith('rm -f ') and '/user-data ' in script for script in calls))
-            self.assertIn('root@example.com --port 22', messages[-1])
-
-    def test_completion_failure_keeps_administrator_resume_endpoint(self):
-        with tempfile.TemporaryDirectory() as directory:
-            root = Path(directory)
-            source = root / 'user-data'
-            source.write_bytes(b'opaque input')
-            class Remote:
-                user, host, identity = 'root', 'example.com', None
-                def __init__(self, *args):
-                    self.resolved_port = 22
-                def resolve(self):
-                    pass
-                def run(self, script, data=None, **kwargs):
-                    if 'ssh_host_' in script:
-                        value = b'192.0.2.1 50000 192.0.2.2 22\nssh-ed25519 AAAA fixture\n'
-                    elif 'mktemp -d /var/tmp/apex-init.' in script:
-                        value = b'/var/tmp/apex-init.0123456789\n'
-                    elif script.startswith('if test -f /var/tmp'):
-                        value = b'0'
-                    elif 'engine.provisioning.target complete' in script:
-                        raise TransportError('record failed')
-                    else:
-                        value = b''
-                    return subprocess.CompletedProcess([], 0, value, b'')
-            messages = []
-            with patch('engine.provisioning.transport.SSH', Remote), \
-                    patch('engine.provisioning.transport.build_bundle', return_value=(b'bundle', 'a' * 64, 'b' * 40)), \
-                    patch('engine.provisioning.transport.protected_handover'), \
-                    patch('engine.provisioning.transport.shutil.which', return_value='/usr/bin/ssh'):
-                with self.assertRaises(TransportError):
-                    initialize(root, 'root@example.com', source, progress=messages.append)
-            self.assertIn('adam@example.com --port 2222', messages[-1])
-
-
-class TargetPathTests(unittest.TestCase):
-    """Run the real shell guards with only Debian stat/mv translated on macOS."""
-    def setUp(self):
-        temporary = tempfile.TemporaryDirectory()
-        self.addCleanup(temporary.cleanup)
-        self.root = Path(temporary.name).resolve()
-        self.bin = self.root / 'bin'
-        self.bin.mkdir()
-        (self.bin / 'stat').write_text('#!' + sys.executable + '\n' + '''
-import os, stat, sys
-path = sys.argv[-1]
-info = os.lstat(path)
-field = sys.argv[sys.argv.index('-c') + 1]
-values = {'%a': format(stat.S_IMODE(info.st_mode), 'o'), '%h': str(info.st_nlink),
-          '%u': '1000' if path == os.environ.get('TEST_NONROOT_PATH') else '0'}
-print(values[field])
-''')
-        (self.bin / 'mv').write_text('#!' + sys.executable + '\nimport os, sys\nos.replace(sys.argv[-2], sys.argv[-1])\n')
-        for command in self.bin.iterdir():
-            command.chmod(0o755)
-        self.environment = dict(os.environ, PATH=str(self.bin) + os.pathsep + os.environ['PATH'])
-
-    def write(self, path, data=b'new metadata'):
-        return subprocess.run(['sh', '-c', remote_atomic_input(str(path))], input=data,
-                              capture_output=True, env=self.environment)
-
-    def test_atomic_write_ignores_predictable_old_temporary_symlink(self):
-        destination = self.root / 'remote.json'
-        victim = self.root / 'victim'
-        victim.write_bytes(b'preserve me')
-        destination.with_name('remote.json.new').symlink_to(victim)
-        result = self.write(destination)
-        self.assertEqual(result.returncode, 0, result.stderr)
-        self.assertEqual(destination.read_bytes(), b'new metadata')
-        self.assertEqual(victim.read_bytes(), b'preserve me')
-        self.assertEqual(destination.stat().st_mode & 0o777, 0o600)
-        self.assertFalse(list(self.root.glob('.apex-write.*')))
-
-    def test_destination_symlink_is_rejected_without_touching_victim(self):
-        victim = self.root / 'victim'
-        victim.write_bytes(b'preserve me')
-        destination = self.root / 'remote.json'
-        destination.symlink_to(victim)
-        result = self.write(destination)
-        self.assertNotEqual(result.returncode, 0)
-        self.assertEqual(victim.read_bytes(), b'preserve me')
-
-    def test_redirected_parent_is_rejected_before_creating_files(self):
-        victim = self.root / 'victim'
-        victim.mkdir()
-        redirected = self.root / 'apex-init'
-        redirected.symlink_to(victim, target_is_directory=True)
-        result = self.write(redirected / 'remote.json')
-        self.assertNotEqual(result.returncode, 0)
-        self.assertEqual(list(victim.iterdir()), [])
-
-    def test_unowned_parent_is_rejected_before_creating_files(self):
-        state = self.root / 'apex-init'
-        state.mkdir()
-        self.environment['TEST_NONROOT_PATH'] = str(state)
-        result = self.write(state / 'remote.json')
-        self.assertNotEqual(result.returncode, 0)
-        self.assertEqual(list(state.iterdir()), [])
-
-    def test_writable_ancestor_is_rejected_before_creating_files(self):
-        state = self.root / 'apex-init'
-        state.mkdir(mode=0o777)
-        state.chmod(0o777)
-        result = self.write(state / 'remote.json')
-        self.assertNotEqual(result.returncode, 0)
-        self.assertEqual(list(state.iterdir()), [])
-
-    def test_hardlinked_destination_is_rejected_without_touching_victim(self):
-        victim = self.root / 'victim'
-        victim.write_bytes(b'preserve me')
-        destination = self.root / 'remote.json'
-        os.link(victim, destination)
-        result = self.write(destination)
-        self.assertNotEqual(result.returncode, 0)
-        self.assertEqual(victim.read_bytes(), b'preserve me')
-
-
-if __name__ == '__main__':
-    unittest.main()

ci/scripts/init-node/tests/test_provision_userdata.py

@@ -1,30 +0,0 @@
-import importlib.util
-import unittest
-
-from engine.provisioning.contract import ContractError
-from engine.provisioning.userdata import decode_user_data
-
-
-@unittest.skipUnless(importlib.util.find_spec('yaml'), 'target-only python3-yaml is unavailable')
-class UserDataTests(unittest.TestCase):
-    def test_duplicate_mapping_keys_rejected_before_normalization(self):
-        with self.assertRaisesRegex(ContractError, 'unique'):
-            decode_user_data(b'hostname: one.example.com\nhostname: two.example.com\n')
-
-    def test_alias_and_unsafe_yaml_do_not_execute_or_echo_content(self):
-        for content in (b'a: &a [*a]', b'!!python/object/apply:os.system [SECRET_SENTINEL]',
-                        b'a: [SECRET_SENTINEL'):
-            with self.subTest(content=content):
-                with self.assertRaises(ContractError) as error:
-                    decode_user_data(content)
-                self.assertNotIn('SECRET_SENTINEL', str(error.exception))
-
-    def test_normal_supported_document(self):
-        import yaml
-        from tests.test_provision_contract import example
-        spec = decode_user_data(yaml.safe_dump(example()).encode())
-        self.assertEqual(spec.hostname, 'metis.a1.example.com')
-
-    def test_yaml_boolean_mapping_key_rejected(self):
-        with self.assertRaisesRegex(ContractError, 'unique strings'):
-            decode_user_data(b'true: value\n')

compositions/apex/docker-compose.yml

@@ -1,5 +1,5 @@
 # commons/compositions/apex/docker-compose.yml
-# The shared apex CORE composition. Consumed by each node via include: (SP3).
+# The shared apex CORE composition. Available to node compositions through include:.
 # FULLY ${env}-parameterized; NO './' host paths (relative paths in an included file
 # resolve against THIS dir — always use absolute ${APEX_*} vars). The engine exports
 # APEX_TIER*/APEX_TIER*_SHARED/APEX_TIER_ROOT*/APEX_SUBNET/APEX_NODE_FQDN at invoke.
@@ -18,7 +18,7 @@ services:
     network_mode: "service:traefik"
     restart: unless-stopped
     labels:
-      # backup (resticontainer, core/all-nodes): adguard config only (work data regenerable)
+      # backup (resticontainer): adguard config only (work data regenerable)
       - "restic.enable=true"
       - "restic.backup.paths=/opt/adguardhome/conf"
     cap_add:
@@ -30,12 +30,8 @@ services:
       - no-new-privileges:true
 
   # --- optional: observability agent (profile-gated) ---
-  # Grafana Alloy: one agent per node. Collects host metrics (node_exporter-
-  # equivalent) + container metrics (cadvisor) and PUSHES them via remote_write;
-  # collects systemd-journal + docker logs and PUSHES them to Loki. All targets
-  # are OUTBOUND to a central Prometheus + Loki (hosted on whichever node runs them), so
-  # Alloy needs egress (direct, not enclave). Replaces the old store-only `loki`
-  # service (which had no feeder). Config is a committed, env-parameterized asset.
+  # Collect host/container metrics and logs for configured remote endpoints.
+  # Alloy requires outbound access to those endpoints through the direct network.
   alloy:
     image: grafana/alloy:v1.20.0
     container_name: apex-alloy
@@ -86,8 +82,7 @@ services:
   # container mounts to host paths (via the /srv mount), snapshots the union
   # in a single restic run, then restarts + runs post-hooks. Every other subcommand
   # (init/snapshots/forget/…) passes straight through to restic. Needs the docker
-  # socket (list/inspect/exec/stop/start) and /srv read-only (the fleet's data root)
-  # — a privileged manual one-shot, same trust surface as the old restic core.
+  # socket (list/inspect/exec/stop/start) and read-only access to /srv.
   resticontainer:
     image: ghcr.io/deytenit/resticontainer:0.19.1@sha256:85457ee665b1c6c8438ac3494c286d267169ee023857da3a605a6fe16dab3d82
     container_name: apex-resticontainer
@@ -112,9 +107,8 @@ services:
       # for any other S3-compatible store (Yandex, Backblaze B2, MinIO, …).
       AWS_REGION: "${APEX_RESTIC_AWS_REGION:-us-east-1}"
       AWS_ENDPOINT_URL: "${APEX_RESTIC_AWS_ENDPOINT:-}"
-      # max compression is fleet policy (backup/run also forces --compression max on
-      # the CLI; needs repo format v2, which a fresh restic init creates). Per-node
-      # override via APEX_RESTIC_COMPRESSION only if ever needed.
+      # Compression requires repository format v2. The default is max;
+      # override it with APEX_RESTIC_COMPRESSION for direct container use.
       RESTIC_COMPRESSION: "${APEX_RESTIC_COMPRESSION:-max}"
       RESTIC_CACHE_DIR: /var/restic-cache
     network_mode: "host"
@@ -195,7 +189,7 @@ services:
       - "traefik.http.middlewares.apex-adguard-strip-prefix.stripprefix.prefixes=/adguard"
       - "traefik.http.routers.apex-adguard.service=apex-adguard"
       - "traefik.http.services.apex-adguard.loadbalancer.server.port=3000"
-      # backup (resticontainer, core/all-nodes): dynamic routing rules only
+      # backup (resticontainer): dynamic routing rules only
       # (letsencrypt certs are regenerable via ACME/Cloudflare DNS-01)
       - "restic.enable=true"
       - "restic.backup.paths=/etc/traefik/dynamic"
@@ -218,7 +212,7 @@ services:
     command: [ "-c", "/etc/xray/config.jsonc" ]
     restart: unless-stopped
     labels:
-      # backup (resticontainer, core/all-nodes): xray config only (logs + shm are ephemeral)
+      # backup (resticontainer): xray config only (logs + shm are ephemeral)
       - "restic.enable=true"
       - "restic.backup.paths=/etc/xray/config.jsonc"
     user: "0:0"

etc/alloy/config.alloy

@@ -1,13 +1,6 @@
-// ============================================================================
-// APEX fleet observability agent — per-node Grafana Alloy config (shared).
-// Committed, env-parameterized: every node runs this SAME file; per-node values
-// come from sys.env(...) (APEX_NODE_HOST, push URLs, obs basic-auth creds).
-// Verified against grafana.com/docs/alloy/latest (Alloy v1.17.1).
-// ============================================================================
-// Model: PUSH. Metrics -> remote_write to a central Prometheus; logs -> loki.write
-// to a central Loki. Push endpoints + basic-auth are per-node env
-// (APEX_PROM_PUSH_URL, APEX_LOKI_PUSH_URL, APEX_OBS_BASIC_AUTH_USER/PASS).
-// ============================================================================
+// Optional shared Grafana Alloy configuration.
+// Metrics and logs use the endpoints and credentials supplied through
+// APEX_PROM_PUSH_URL, APEX_LOKI_PUSH_URL, and APEX_OBS_BASIC_AUTH_USER/PASS.
 
 // ----------------------------------------------------------------------------
 // METRICS
@@ -81,10 +74,8 @@ prometheus.scrape "containers" {
   scrape_interval = "30s"
 }
 
-// CrowdSec agent metrics. CrowdSec runs on the host (not a container) and exposes Prometheus
-// metrics on :6060; nodes that run it bind that endpoint on 0.0.0.0 (the host firewall keeps it
-// off the public interface) so Alloy can reach it via the docker gateway. Nodes that don't run
-// CrowdSec simply report this target down.
+// The CrowdSec scrape requires a reachable host metrics endpoint on port 6060.
+// If no endpoint is configured, this target reports down.
 prometheus.scrape "crowdsec" {
   targets = [{
     __address__ = "host.docker.internal:6060",
@@ -96,7 +87,7 @@ prometheus.scrape "crowdsec" {
   scrape_interval = "30s"
 }
 
-// Push metrics to the central Prometheus (needs --web.enable-remote-write-receiver).
+// The configured Prometheus-compatible endpoint must accept remote writes.
 prometheus.remote_write "default" {
   endpoint {
     url = sys.env("APEX_PROM_PUSH_URL")

etc/alloy/native-host.alloy

@@ -0,0 +1,102 @@
+// Shared native Grafana Alloy configuration for the physical host.
+// Metrics and logs use the endpoints and credentials supplied through
+// APEX_PROM_PUSH_URL, APEX_LOKI_PUSH_URL, and APEX_OBS_BASIC_AUTH_USER/PASS.
+
+// ----------------------------------------------------------------------------
+// METRICS
+// ----------------------------------------------------------------------------
+
+// Native filesystem views; physical capacity is collected on the Proxmox host.
+prometheus.exporter.unix "node" {
+  enable_collectors = ["textfile"]
+  textfile {
+    directory = "/var/lib/alloy/textfile"
+  }
+  procfs_path = "/proc"
+  sysfs_path  = "/sys"
+  rootfs_path = "/"
+}
+
+// Container metrics (cadvisor-equivalent) via the docker socket.
+
+
+// Stamp a static `node` label onto each exporter's targets.
+discovery.relabel "node_metrics" {
+  targets = prometheus.exporter.unix.node.targets
+  rule {
+    action       = "replace"
+    target_label = "node"
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+  // Stamp a stable `instance` (the node's public host) instead of the default,
+  // which is the Alloy container id/address.
+  rule {
+    action       = "replace"
+    target_label = "instance"
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+}
+
+
+prometheus.scrape "node" {
+  targets         = discovery.relabel.node_metrics.output
+  forward_to      = [prometheus.remote_write.default.receiver]
+  scrape_interval = "30s"
+}
+
+
+// The configured Prometheus-compatible endpoint must accept remote writes.
+prometheus.remote_write "default" {
+  endpoint {
+    url = sys.env("APEX_PROM_PUSH_URL")
+    basic_auth {
+      username = sys.env("APEX_OBS_BASIC_AUTH_USER")
+      password = sys.env("APEX_OBS_BASIC_AUTH_PASS")
+    }
+    queue_config {
+      capacity             = 5000
+      max_shards           = 10
+      min_shards           = 1
+      max_samples_per_send = 1000
+      batch_send_deadline  = "5s"
+      min_backoff          = "100ms"
+      max_backoff          = "10s"
+    }
+  }
+  wal {
+    truncate_frequency = "30m"
+    min_keepalive_time = "5m"
+    max_keepalive_time = "30m"
+  }
+}
+
+// ----------------------------------------------------------------------------
+// LOGS
+// ----------------------------------------------------------------------------
+
+// systemd journal (host services, ufw, sshd, crowdsec, etc.).
+loki.source.journal "systemd" {
+  path    = "/var/log/journal"
+  max_age = "12h"
+  labels  = {
+    node = sys.env("APEX_NODE_HOST"),
+    job  = "systemd-journal",
+  }
+  forward_to = [loki.write.default.receiver]
+}
+
+// All docker containers' stdout/stderr.
+
+
+
+
+// Push logs to the central Loki.
+loki.write "default" {
+  endpoint {
+    url = sys.env("APEX_LOKI_PUSH_URL")
+    basic_auth {
+      username = sys.env("APEX_OBS_BASIC_AUTH_USER")
+      password = sys.env("APEX_OBS_BASIC_AUTH_PASS")
+    }
+  }
+}

etc/alloy/native.alloy

@@ -0,0 +1,128 @@
+// Shared native Grafana Alloy configuration for Docker guests.
+// Metrics and logs use the endpoints and credentials supplied through
+// APEX_PROM_PUSH_URL, APEX_LOKI_PUSH_URL, and APEX_OBS_BASIC_AUTH_USER/PASS.
+
+// ----------------------------------------------------------------------------
+// METRICS
+// ----------------------------------------------------------------------------
+
+// Native filesystem views; physical capacity is collected on the Proxmox host.
+
+
+// Container metrics (cadvisor-equivalent) via the docker socket.
+prometheus.exporter.cadvisor "containers" {
+  docker_host                  = "unix:///var/run/docker.sock"
+  store_container_labels       = false
+  allowlisted_container_labels = [
+    "com.docker.compose.project",
+    "com.docker.compose.service",
+  ]
+  docker_only                  = true
+  enabled_metrics              = [
+    "cpu",
+    "cpuLoad",
+    "diskIO",
+    "memory",
+    "network",
+    "oom_event",
+  ]
+}
+
+// Stamp a static `node` label onto each exporter's targets.
+
+discovery.relabel "container_metrics" {
+  targets = prometheus.exporter.cadvisor.containers.targets
+  rule {
+    action       = "replace"
+    target_label = "node"
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+  rule {
+    action       = "replace"
+    target_label = "instance"
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+}
+
+
+prometheus.scrape "containers" {
+  targets         = discovery.relabel.container_metrics.output
+  forward_to      = [prometheus.remote_write.default.receiver]
+  scrape_interval = "30s"
+}
+
+// The configured Prometheus-compatible endpoint must accept remote writes.
+prometheus.remote_write "default" {
+  endpoint {
+    url = sys.env("APEX_PROM_PUSH_URL")
+    basic_auth {
+      username = sys.env("APEX_OBS_BASIC_AUTH_USER")
+      password = sys.env("APEX_OBS_BASIC_AUTH_PASS")
+    }
+    queue_config {
+      capacity             = 5000
+      max_shards           = 10
+      min_shards           = 1
+      max_samples_per_send = 1000
+      batch_send_deadline  = "5s"
+      min_backoff          = "100ms"
+      max_backoff          = "10s"
+    }
+  }
+  wal {
+    truncate_frequency = "30m"
+    min_keepalive_time = "5m"
+    max_keepalive_time = "30m"
+  }
+}
+
+// ----------------------------------------------------------------------------
+// LOGS
+// ----------------------------------------------------------------------------
+
+// systemd journal (host services, ufw, sshd, crowdsec, etc.).
+loki.source.journal "systemd" {
+  path    = "/var/log/journal"
+  max_age = "12h"
+  labels  = {
+    node = sys.env("APEX_NODE_HOST"),
+    job  = "systemd-journal",
+  }
+  forward_to = [loki.write.default.receiver]
+}
+
+// All docker containers' stdout/stderr.
+discovery.docker "containers" {
+  host = "unix:///var/run/docker.sock"
+}
+discovery.relabel "container_logs" {
+  targets = discovery.docker.containers.targets
+  rule {
+    action        = "replace"
+    source_labels = ["__meta_docker_container_name"]
+    regex         = "/?(.*)"
+    target_label  = "container"
+  }
+  rule {
+    action       = "replace"
+    target_label = "node"
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+}
+loki.source.docker "containers" {
+  host          = "unix:///var/run/docker.sock"
+  targets       = discovery.relabel.container_logs.output
+  relabel_rules = discovery.relabel.container_logs.rules
+  forward_to    = [loki.write.default.receiver]
+}
+
+// Push logs to the central Loki.
+loki.write "default" {
+  endpoint {
+    url = sys.env("APEX_LOKI_PUSH_URL")
+    basic_auth {
+      username = sys.env("APEX_OBS_BASIC_AUTH_USER")
+      password = sys.env("APEX_OBS_BASIC_AUTH_PASS")
+    }
+  }
+}

pyproject.toml

@@ -6,13 +6,18 @@ readme = "README.md"
 license = { text = "BSD-2-Clause" }
 requires-python = ">=3.10"
 
+[project.optional-dependencies]
+dev = ["ruff==0.17.2"]
+
 [tool.ruff]
 line-length = 100
 target-version = "py310"
 
 [tool.ruff.lint]
 select = ["E", "F", "W", "I", "B", "UP", "S"]
-ignore = ["S603", "S607"]  # Subprocess invocations are mediated via ctx.sys
+# Existing formatting and typing modernization are outside correctness checks.
+# Subprocess arguments are constructed explicitly; no implicit shell execution.
+ignore = ["E401", "E501", "E702", "E741", "I001", "UP006", "UP035", "UP045", "S603", "S607"]
 
 [tool.mypy]
 python_version = "3.10"

scripts/bootstrap-node

@@ -0,0 +1,95 @@
+#!/usr/bin/env python3
+"""Bootstrap a fleet node through a trusted OpenSSH connection."""
+
+import argparse
+import json
+import re
+import shlex
+import subprocess
+import sys
+from pathlib import Path
+from urllib.parse import urlsplit
+
+
+def repository(url):
+    if any(ch.isspace() or ord(ch) < 32 for ch in url):
+        raise ValueError("invalid repository URL")
+    if url.startswith("ssh://"):
+        parts = urlsplit(url)
+        if parts.password is not None or parts.query or parts.fragment:
+            raise ValueError("expected an SSH repository URL without password, query or fragment")
+        host, port, path = parts.hostname, parts.port, parts.path
+    else:
+        match = re.fullmatch(r"(?:[\w.-]+@)?(\[[\da-fA-F:]+\]|[\w.-]+):(.+)", url)
+        if not match:
+            raise ValueError("expected git@host:path or ssh://user@host[:port]/path")
+        host, path = match.groups()
+        host, port = host.strip("[]"), None
+    if not host or not re.fullmatch(r"[a-zA-Z0-9:][a-zA-Z0-9.:-]*", host) or not path:
+        raise ValueError("invalid Git host or repository path")
+    return f"[{host}]:{port}" if port not in (None, 22) else host, Path(path).name.removesuffix(".git")
+
+
+def keygen(args, data=None):
+    return subprocess.run(["ssh-keygen", *args], input=data or b"", capture_output=True)
+
+
+def bootstrap(args):
+    endpoint, basename = repository(args.repo)
+    name = args.node_name if args.node_name is not None else basename
+    if not re.fullmatch(r"[a-zA-Z0-9][a-zA-Z0-9._-]*", name):
+        raise ValueError("invalid node name")
+    operator = args.operator_key.read_text().strip()
+    fields = operator.split()
+    if (len(operator.splitlines()) != 1 or len(fields) < 2
+            or keygen(["-lf", "-"], (" ".join(fields[:2]) + "\n").encode()).returncode):
+        raise ValueError("operator key must contain one plain SSH public key")
+    derived = keygen(["-y", "-P", "", "-f", str(args.deploy_key.resolve())])
+    if derived.returncode or derived.stdout.split()[0].startswith(b"sk-"):
+        raise ValueError("deploy key must be a standalone private key with no passphrase and owner-only permissions")
+    if derived.stdout.split()[:2] == [v.encode() for v in fields[:2]]:
+        raise ValueError("operator and deploy keys must be different")
+    matches = keygen(["-F", endpoint, "-f", str(args.known_hosts.resolve())])
+    pins = [line for line in matches.stdout.decode().splitlines() if line and not line.startswith("#")]
+    if matches.returncode or not pins or not any(
+        not pin.startswith("@revoked ") and keygen(["-lf", "-"], (pin + "\n").encode()).returncode == 0
+        for pin in pins
+    ):
+        raise ValueError(f"no usable known_hosts pin for {endpoint}")
+    payload = json.dumps(dict(operator_key=operator, deploy_key=args.deploy_key.read_text(),
+                              known_hosts="\n".join(pins) + "\n", repo_url=args.repo, node_name=name))
+    helper = Path(__file__).resolve().with_name("bootstrap-node-remote.py").read_text()
+    # Package management must never consume the credential payload on standard input.
+    remote = """set -e
+if [ "$(id -u)" = 0 ]; then set --; else set -- sudo -n --; fi
+if ! command -v python3 >/dev/null; then
+    test -f /etc/debian_version
+    "$@" apt-get -o DPkg::Lock::Timeout=60 update </dev/null
+    "$@" env DEBIAN_FRONTEND=noninteractive apt-get -o DPkg::Lock::Timeout=60 install -y python3 </dev/null
+fi
+exec "$@" python3 -I -c """ + shlex.quote(helper)
+    result = subprocess.run(["ssh", "-T", "-a", "-S", "none", "-o", "StrictHostKeyChecking=yes",
+                           "-o", "ClearAllForwardings=yes", "--", args.target, remote],
+                          input=payload.encode())
+    if result.returncode == 0:
+        print(f"Checkout ready for adam at /srv/{name}.")
+    return result.returncode
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--target", required=True, help="trusted SSH target with root or noninteractive sudo")
+    parser.add_argument("--repo", required=True, help="SSH URL of the node repository")
+    parser.add_argument("--operator-key", required=True, type=Path, help="operator public key")
+    parser.add_argument("--deploy-key", required=True, type=Path, help="repository deploy private key")
+    parser.add_argument("--known-hosts", required=True, type=Path, help="trusted Git server host keys")
+    parser.add_argument("--node-name", help="defaults to repository basename without .git")
+    args = parser.parse_args()
+    try:
+        return bootstrap(args)
+    except (OSError, ValueError) as exc:
+        parser.exit(1, f"bootstrap-node: {exc}\n")
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/bootstrap-node-remote.py

@@ -0,0 +1,275 @@
+"""Root-side Debian/PVE bootstrap, staged and invoked by bootstrap-node."""
+import fcntl
+import json
+import os
+from pathlib import Path
+import pwd
+import re
+import stat
+import subprocess
+import sys
+import tempfile
+from urllib.parse import urlsplit
+
+
+HOME = Path("/home/adam")
+
+
+def checked(command, *, data=None, capture=False, **kwargs):
+    # A Git hook or remote can emit credentials. Report the operation, never its output.
+    result = subprocess.run(command, input=data, text=True, stderr=subprocess.DEVNULL,
+                            stdout=subprocess.PIPE if capture else subprocess.DEVNULL, **kwargs)
+    if result.returncode:
+        raise RuntimeError(f"{Path(command[0]).name} failed (exit {result.returncode})")
+    return result.stdout
+
+
+def repository_endpoint(url):
+    if not isinstance(url, str) or any(c.isspace() or ord(c) < 32 for c in url):
+        raise ValueError("Invalid SSH repository URL")
+    if url.startswith("ssh://"):
+        parsed = urlsplit(url)
+        if (parsed.password is not None or not parsed.path.strip("/") or parsed.fragment
+                or parsed.query or parsed.port == 0):
+            raise ValueError("Invalid SSH repository URL")
+        host, port = parsed.hostname, parsed.port
+    else:
+        match = re.fullmatch(r"(?:[\w.-]+@)?(\[[a-fA-F0-9:]+\]|[\w.-]+):(.+)", url, re.ASCII)
+        host, port = (match[1].strip("[]"), None) if match else (None, None)
+    if not host or not re.fullmatch(r"[a-zA-Z0-9:][a-zA-Z0-9.:-]*", host):
+        raise ValueError("An SSH repository host is required")
+    return host, f"[{host}]:{port}" if port not in (None, 22) else host
+
+
+def existing(path):
+    try:
+        info = path.lstat()
+    except FileNotFoundError:
+        return b""
+    if not stat.S_ISREG(info.st_mode) or info.st_nlink != 1 or info.st_uid != os.geteuid():
+        raise ValueError(f"Unsafe file: {path.name}")
+    return path.read_bytes()
+
+
+def install(path, content, mode):
+    existing(path)
+    fd, tmp = tempfile.mkstemp(dir=path.parent)
+    try:
+        with os.fdopen(fd, "wb") as stream:
+            stream.write(content)
+            os.fchmod(stream.fileno(), mode)
+        os.replace(tmp, path)
+    finally:
+        Path(tmp).unlink(missing_ok=True)
+
+
+def validate_credentials(spec, endpoint):
+    key = spec["operator_key"].strip()
+    if len(key.splitlines()) != 1 or not key.startswith(("ssh-", "ecdsa-", "sk-")):
+        raise ValueError("Expected one operator public key")
+    checked(["ssh-keygen", "-lf", "-"], data=key + "\n")
+    with tempfile.TemporaryDirectory() as temp:
+        private = Path(temp) / "deploy"
+        install(private, (spec["deploy_key"].rstrip() + "\n").encode(), 0o600)
+        public = checked(["ssh-keygen", "-y", "-P", "", "-f", str(private)], capture=True)
+        if public.startswith("sk-") or public.split()[:2] == key.split()[:2]:
+            raise ValueError("Use a separate unattended deploy key")
+        pins = Path(temp) / "known_hosts"
+        pins.write_text(spec["known_hosts"])
+        matches = checked(["ssh-keygen", "-F", endpoint, "-f", str(pins)], capture=True)
+        trusted = [line for line in matches.splitlines() if not line.startswith(("#", "@revoked "))]
+        if not trusted:
+            raise ValueError("No trusted host key for the repository")
+        for pin in trusted:
+            checked(["ssh-keygen", "-lf", "-"], data=pin + "\n")
+
+
+def configure_ssh(home, spec, host):
+    directory = home / ".ssh"
+    directory.mkdir(mode=0o700, exist_ok=True)
+    info = directory.lstat()
+    if not stat.S_ISDIR(info.st_mode) or info.st_uid != os.geteuid():
+        raise ValueError("Unsafe SSH directory")
+    directory.chmod(0o700)
+    key = spec["operator_key"].strip()
+    authorized = directory / "authorized_keys"
+    content = existing(authorized)
+    identity = key.split()[:2]
+    present = False
+    for line in content.decode(errors="replace").splitlines():
+        fields = line.split()
+        if fields[:2] == identity:
+            present = True
+        elif fields and not line.lstrip().startswith("#") and identity[1] in fields:
+            raise ValueError("Existing operator key has SSH authorization restrictions; inspect it")
+    if not present:
+        content = content + (b"\n" if content and not content.endswith(b"\n") else b"")
+        content += key.encode() + b"\n"
+    # Git also evaluates includeIf for the separate gitdirs of recursive submodules.
+    config = home / ".gitconfig"
+    old = existing(config)
+    gitdir = f"/srv/{spec['node_name']}/.git"
+    prefix = directory / ("apex-" + spec["node_name"])
+    include = (f'[includeIf "gitdir:{gitdir}"]\n\tpath = {prefix}.gitconf\n'
+               f'[includeIf "gitdir:{gitdir}/"]\n\tpath = {prefix}.gitconf\n').encode()
+    for suffix in ("key", "hosts", "sshconf", "gitconf"):
+        existing(Path(f"{prefix}.{suffix}"))
+    write_repository_credentials(prefix, spec, host)
+    install(Path(f"{prefix}.gitconf"),
+            f"[core]\n\tsshCommand = ssh -F {prefix}.sshconf\n".encode(), 0o600)
+    if include not in old:
+        install(config, old + b"\n" + include, 0o600)
+    install(authorized, content, 0o600)
+
+
+def write_repository_credentials(prefix, spec, host):
+    policy = (f"Host {host}\n    IdentitiesOnly yes\n"
+              f"    IdentityFile {prefix}.key\n    UserKnownHostsFile {prefix}.hosts\n"
+              "    GlobalKnownHostsFile /dev/null\nHost *\n    StrictHostKeyChecking yes\n"
+              "    BatchMode yes\n")
+    for suffix, content in (("key", spec["deploy_key"]), ("hosts", spec["known_hosts"]),
+                            ("sshconf", policy)):
+        install(Path(f"{prefix}.{suffix}"), (content.rstrip() + "\n").encode(), 0o600)
+
+
+def verify_and_clone(spec, host, account):
+    srv = Path("/srv")
+    if not srv.exists():
+        srv.mkdir()
+        srv.chmod(0o755)
+    if srv.resolve() != srv or srv.stat().st_uid != 0 or srv.stat().st_mode & 0o022:
+        raise RuntimeError("/srv must be a root-controlled directory")
+    with tempfile.TemporaryDirectory(prefix=".apex-credentials-", dir=srv) as temp:
+        stage = Path(temp)
+        prefix = stage / "git"
+        write_repository_credentials(prefix, spec, host)
+        for path in stage.iterdir():
+            os.chown(path, account.pw_uid, account.pw_gid)
+        stage.chmod(0o711)
+
+        def git(*args):
+            return checked(["runuser", "-u", "adam", "--", "env", "GIT_TERMINAL_PROMPT=0",
+                            f"GIT_SSH_COMMAND=ssh -F {prefix}.sshconf", "git", *args],
+                           capture=True, cwd="/")
+
+        try:
+            git("ls-remote", "--exit-code", "--", spec["repo_url"], "HEAD")
+        except RuntimeError:
+            raise RuntimeError("Git access check failed; check the repository URL, deploy-key "
+                               "grants and pinned host key") from None
+        checkout(srv, spec["node_name"], spec["repo_url"], account, git)
+
+
+def prepare_packages():
+    if not Path("/etc/debian_version").is_file():
+        raise RuntimeError("Debian or Proxmox VE is required")
+    missing = []
+    for package in ("git", "sudo", "openssh-client"):
+        result = subprocess.run(["dpkg-query", "-W", "-f=${Status}", package], text=True,
+                                capture_output=True)
+        if result.returncode or result.stdout.split()[1:] != ["ok", "installed"]:
+            missing.append(package)
+    if missing:
+        checked(["apt-get", "update"])
+        checked(["apt-get", "install", "-y", "--no-remove", "--no-install-recommends", *missing])
+
+
+def prepare_account():
+    try:
+        account = pwd.getpwnam("adam")
+    except KeyError:
+        if os.path.lexists(HOME):
+            raise RuntimeError("Existing /home/adam requires inspection") from None
+        checked(["useradd", "--create-home", "--user-group", "--shell", "/bin/bash", "adam"])
+        account = pwd.getpwnam("adam")
+    if (not 1000 <= account.pw_uid < 65534 or account.pw_gid == 0
+            or account.pw_dir != str(HOME) or HOME.resolve() != HOME
+            or HOME.stat().st_uid != account.pw_uid or HOME.stat().st_mode & 0o022
+            or not HOME.is_dir()):
+        raise RuntimeError("Existing adam account or home requires inspection")
+    groups = "sudo"
+    if not Path("/usr/bin/pveversion").exists():
+        checked(["groupadd", "--system", "--force", "docker"])
+        groups += ",docker"
+    checked(["usermod", "--shell", "/bin/bash", "--append", "--groups", groups, "adam"])
+    return account
+
+
+def checkout(srv, node, url, account, git):
+    destination = srv / node
+    if os.path.lexists(destination):
+        info = destination.lstat()
+        if not stat.S_ISDIR(info.st_mode) or info.st_uid != account.pw_uid:
+            raise RuntimeError("Existing checkout path requires inspection")
+
+        def query(*args):
+            return git("-C", str(destination), *args).strip()
+
+        if query("rev-parse", "--show-toplevel") != str(destination) or query(
+                "config", "--get", "remote.origin.url") != url:
+            raise RuntimeError("Existing checkout does not match the requested repository")
+        for line in query("submodule", "status", "--recursive").splitlines():
+            if line.startswith(("-", "+", "U")):
+                raise RuntimeError(f"Existing submodule does not match its recorded commit: {line}")
+        return
+    with tempfile.TemporaryDirectory(prefix=".apex-bootstrap-", dir=srv) as staging:
+        stage = Path(staging)
+        stage.chmod(0o711)
+        clone = stage / "checkout"
+        clone.mkdir()
+        os.chown(clone, account.pw_uid, account.pw_gid)
+        try:
+            git("clone", "--recurse-submodules", "--", url, str(clone))
+        except RuntimeError:
+            raise RuntimeError("Recursive clone failed; check access to pinned submodules "
+                               "and available disk space") from None
+        clone.rename(destination)
+
+
+def bootstrap(spec):
+    node = spec["node_name"]
+    if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._-]{0,252}", node):
+        raise ValueError("Invalid node checkout name")
+    host, endpoint = repository_endpoint(spec["repo_url"])
+    if not all(isinstance(spec[k], str) and spec[k].strip()
+               for k in ("known_hosts", "deploy_key", "operator_key")):
+        raise ValueError("Missing SSH credentials")
+    if os.geteuid() != 0:
+        raise RuntimeError("Root login is required")
+    os.umask(0o077)
+    os.environ.clear()
+    os.environ.update(PATH="/usr/sbin:/usr/bin:/sbin:/bin", LC_ALL="C",
+                      DEBIAN_FRONTEND="noninteractive", HOME="/root")
+    # Serialize account, credential and checkout changes from competing bootstraps.
+    with open("/run/apex-bootstrap.lock", "a") as lock:
+        fcntl.flock(lock, fcntl.LOCK_EX)
+        prepare_packages()
+        validate_credentials(spec, endpoint)
+        account = prepare_account()
+        verify_and_clone(spec, host, account)
+        child = os.fork()
+        if child == 0:
+            try:
+                os.initgroups("adam", account.pw_gid)
+                os.setgid(account.pw_gid)
+                os.setuid(account.pw_uid)
+                os.chdir(HOME)
+                configure_ssh(HOME, spec, host)
+            except Exception as error:
+                print(f"SSH setup failed: {error}", file=sys.stderr, flush=True)
+                os._exit(1)
+            os._exit(0)
+        if os.waitpid(child, 0)[1]:
+            raise RuntimeError("Unable to install SSH credentials")
+        policy = "adam ALL=(ALL:ALL) NOPASSWD: ALL\n"
+        checked(["visudo", "-cf", "-"], data=policy)
+        install(Path("/etc/sudoers.d/99-apex-adam"), policy.encode(), 0o440)
+        checked(["visudo", "-c"])
+        checked(["runuser", "-u", "adam", "--", "sudo", "-n", "true"])
+
+
+if __name__ == "__main__":
+    try:
+        bootstrap(json.load(sys.stdin))
+    except (ValueError, KeyError, RuntimeError, OSError) as error:
+        sys.exit(f"Bootstrap failed: {error}")

scripts/engine/cli.py

@@ -1,9 +1,10 @@
-# engine/cli.py
+# scripts/engine/cli.py
 """apex CLI: list / dispatch / help. Acts on the local checkout (no <node> arg)."""
 from __future__ import annotations
 import os
 import socket
 import sys
+from pathlib import Path
 
 from engine import overlay
 from engine.identity import resolve
@@ -12,7 +13,7 @@ from engine.descriptor import build_parser, Meta
 
 
 def _resolve_layout():
-    commons_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    commons_dir = str(Path(__file__).resolve().parents[2])
     parent = os.path.dirname(commons_dir)
     cwd = os.getcwd()
 
@@ -28,7 +29,7 @@ def _resolve_layout():
 
 
 def _print_list(commons_actions, prop_actions):
-    print("Usage: apex <group>/<action> [args...]\n\nAvailable actions:")
+    print("Usage: apex <entity>/<action> [args...]\n\nAvailable actions:")
     tag = {"commons": "", "local": "  (local)", "override": "  (local override)"}
     for i in overlay.discover(commons_actions, prop_actions):
         mark = f"  [DISABLED: {i.disabled}]" if i.disabled else ""
@@ -44,6 +45,8 @@ def main(argv) -> int:
         return 0
 
     name, rest = argv[0], argv[1:]
+    if name == "compose":
+        name = "composition/run"
     r = overlay.resolve(name, commons_actions, prop_actions)
     if r is None:
         sys.stderr.write(f"[global] [core] [ERROR] Unknown action: {name}\n")

scripts/engine/context.py

@@ -1,5 +1,5 @@
-# engine/context.py
-"""Context (ctx) — the single seam between an action and the world (SP2 §2.1)."""
+# scripts/engine/context.py
+"""Context and filesystem paths available to actions."""
 from __future__ import annotations
 import os
 

scripts/engine/descriptor.py

@@ -1,4 +1,4 @@
-# engine/descriptor.py
+# scripts/engine/descriptor.py
 """Declarative action descriptors — successor to the bash action_metadata() heredoc.
 
 Meta drives BOTH the help text and argparse parsing. Exit code 64 (EX_USAGE) on a

scripts/engine/identity.py

@@ -1,4 +1,4 @@
-# engine/identity.py
+# scripts/engine/identity.py
 """Node identity — fully explicit, from node.env. The framework assumes no domain
 and no clustering: a node declares its real FQDN and its public host name, and the
 short `name` is just the first label. The OS hostname is only a fallback for the FQDN.

scripts/engine/lib/crowdsec.py

@@ -1,5 +1,5 @@
-# engine/lib/crowdsec.py
-"""ctx.crowdsec — install + config deploy + collections (port of configure/crowdsec)."""
+# scripts/engine/lib/crowdsec.py
+"""ctx.crowdsec — install + config deploy + collections (port of config/crowdsec)."""
 from __future__ import annotations
 import os
 import shutil

scripts/engine/lib/host.py

@@ -1,4 +1,4 @@
-# engine/lib/host.py
+# scripts/engine/lib/host.py
 """ctx.host — idempotent host mutation: backup, write_file, inject_block, install_crontab.
 
 Ports root_template_crontab (install + rollback) and the ufw block-injection logic
@@ -6,10 +6,10 @@ Ports root_template_crontab (install + rollback) and the ufw block-injection log
 """
 from __future__ import annotations
 
+import inspect
+import json
 import os
-import shutil
 import subprocess
-import tempfile
 import time
 from typing import List, Optional, Sequence, Tuple
 
@@ -55,29 +55,88 @@ def apply_block(text: str, content: str, start: str, end: str,
     return "\n".join(result) + "\n", inserted
 
 
+def _atomic_write(path, content, mode):
+    """Self-contained so sudo executes the same operation without importing user code."""
+    import os
+    import stat
+    import tempfile
+
+    target = os.path.realpath(path)
+    parent = os.path.dirname(target)
+    try:
+        previous = os.stat(target)
+    except FileNotFoundError:
+        previous = None
+    if previous is not None and not stat.S_ISREG(previous.st_mode):
+        raise ValueError(f"Not a regular file: {path}")
+    target_mode = (int(mode, 8) if mode is not None else
+                   stat.S_IMODE(previous.st_mode) if previous is not None else 0o644)
+    os.makedirs(parent, exist_ok=True)
+    fd, temporary = tempfile.mkstemp(prefix=".apex-", dir=parent)
+    try:
+        with os.fdopen(fd, "w", encoding="utf-8", newline="") as stream:
+            stream.write(content)
+            stream.flush()
+            if previous is not None:
+                current = os.fstat(stream.fileno())
+                if (current.st_uid, current.st_gid) != (previous.st_uid, previous.st_gid):
+                    os.fchown(stream.fileno(), previous.st_uid, previous.st_gid)
+            os.fchmod(stream.fileno(), target_mode)
+            os.fsync(stream.fileno())
+        os.replace(temporary, target)
+    finally:
+        if os.path.exists(temporary):
+            os.unlink(temporary)
+
+
+def _atomic_backup(path, destination):
+    import os
+    import shutil
+    import tempfile
+
+    destination = os.path.abspath(destination)
+    fd, temporary = tempfile.mkstemp(prefix=".apex-", dir=os.path.dirname(destination))
+    os.close(fd)
+    try:
+        shutil.copyfile(path, temporary)
+        previous, current = os.stat(path), os.stat(temporary)
+        if (current.st_uid, current.st_gid) != (previous.st_uid, previous.st_gid):
+            os.chown(temporary, previous.st_uid, previous.st_gid)
+        shutil.copystat(path, temporary)
+        os.replace(temporary, destination)
+    finally:
+        if os.path.exists(temporary):
+            os.unlink(temporary)
+
+
 class Host:
     def __init__(self, log, sys_):
         self.log = log
         self.sys = sys_
 
+    def _file_operation(self, operation, args, sudo):
+        if sudo:
+            script = (inspect.getsource(operation) + "\nimport json, sys\n" +
+                      f"{operation.__name__}(*json.load(sys.stdin))\n")
+            self.sys.sudo(["python3", "-I", "-c", script], input=json.dumps(args))
+        else:
+            operation(*args)
+
     # --- read/write with sudo awareness ---
     def _read(self, path: str, sudo: bool = False) -> str:
         if sudo:
-            cp = self.sys.run(["sudo", "cat", path], check=True, capture=True)
+            cp = self.sys.sudo(["cat", path], check=True, capture=True)
             return cp.stdout or ""
         with open(path, encoding="utf-8") as f:
             return f.read()
 
     def backup(self, path: str, sudo: bool = False) -> Optional[str]:
-        exists = (self.sys.ok(["sudo", "test", "-f", path]) if sudo
+        exists = (self.sys.sudo(["test", "-f", path], check=False).returncode == 0 if sudo
                   else os.path.isfile(path))
         if not exists:
             return None
         bkp = f"{path}.{int(time.time())}.bak"
-        if sudo:
-            self.sys.sudo(["cp", "-a", path, bkp])
-        else:
-            shutil.copy2(path, bkp)
+        self._file_operation(_atomic_backup, [path, bkp], sudo)
         self.log.info(f"Backed up {path} -> {bkp}")
         return bkp
 
@@ -88,29 +147,14 @@ class Host:
             return
         if backup:
             self.backup(path, sudo=sudo)
-        fd, tmp_name = tempfile.mkstemp(prefix="apex-")
-        try:
-            with os.fdopen(fd, "w", encoding="utf-8") as tmp:
-                tmp.write(content)
-            target_mode = mode or "0644"
-            if sudo:
-                self.sys.sudo(["mkdir", "-p", os.path.dirname(path)])
-                self.sys.sudo(["cp", tmp_name, path])
-                self.sys.sudo(["chmod", target_mode, path])
-            else:
-                os.makedirs(os.path.dirname(path), exist_ok=True)
-                shutil.copyfile(tmp_name, path)
-                os.chmod(path, int(target_mode, 8))
-        finally:
-            if os.path.exists(tmp_name):
-                os.unlink(tmp_name)
+        self._file_operation(_atomic_write, [path, content, mode], sudo)
         self.log.success(f"Wrote {path}")
 
     def inject_block(self, target: str, content: str, start: str, end: str,
                      anchor: Optional[str] = None,
                      old_pairs: Sequence[Tuple[str, str]] = (),
                      dry_run: bool = False, sudo: bool = True) -> bool:
-        exists = (self.sys.ok(["sudo", "test", "-f", target]) if sudo
+        exists = (self.sys.sudo(["test", "-f", target], check=False).returncode == 0 if sudo
                   else os.path.isfile(target))
         if not exists:
             self.log.error(f"Target file does not exist: {target}")

scripts/engine/lib/log.py

@@ -1,4 +1,4 @@
-# engine/lib/log.py
+# scripts/engine/lib/log.py
 """ctx.log — [node] [action] [level] message (port of root_log_*)."""
 from __future__ import annotations
 import sys

scripts/engine/lib/notify.py

@@ -1,4 +1,4 @@
-# engine/lib/notify.py
+# scripts/engine/lib/notify.py
 """ctx.notify — Telegram notifications.
 
 The alert `instance` is the node's public host (APEX_NODE_HOST), passed in from the
@@ -73,7 +73,7 @@ class Notify:
         data = urllib.parse.urlencode({"parse_mode": "HTML", "text": text}).encode()
         self.log.info(f"Sending Telegram notification ({level}): {title}")
         try:
-            with urllib.request.urlopen(urllib.request.Request(bot_url, data=data),
+            with urllib.request.urlopen(urllib.request.Request(bot_url, data=data),  # noqa: S310 -- HTTPS validated above.
                                         timeout=NOTIFY_TIMEOUT) as r:
                 r.read()
             return True

scripts/engine/lib/sys.py

@@ -1,6 +1,7 @@
-# engine/lib/sys.py
+# scripts/engine/lib/sys.py
 """ctx.sys — command execution + systemd service management (port of root_sys.sh)."""
 from __future__ import annotations
+import os
 import shutil
 import subprocess
 import time
@@ -21,6 +22,8 @@ class Sys:
                               stderr=subprocess.PIPE if capture else None)
 
     def sudo(self, cmd: Sequence[str], **kw) -> subprocess.CompletedProcess:
+        if os.geteuid() == 0 and shutil.which("sudo") is None:
+            return self.run(cmd, **kw)
         return self.run(["sudo", *cmd], **kw)
 
     def ok(self, cmd: Sequence[str], **kw) -> bool:

scripts/engine/lib/systemd.py

@@ -1,4 +1,4 @@
-# engine/lib/systemd.py
+# scripts/engine/lib/systemd.py
 """ctx.systemd — render + install unit files, daemon-reload, enable+start."""
 from __future__ import annotations
 import os

scripts/engine/lib/tpl.py

@@ -1,9 +1,8 @@
-# engine/lib/tpl.py
-"""ctx.tpl — stdlib string.Template rendering (envsubst successor, SP2 decision #3).
+# scripts/engine/lib/tpl.py
+"""ctx.tpl — stdlib string.Template rendering.
 
-safe_substitute replaces known $VAR/${VAR} and leaves unknown ones literal (deliberate
-delta: envsubst emptied them — literal survivors surface typos instead of silently
-blanking). Shell $(...) command-substitution survives untouched.
+safe_substitute replaces known $VAR/${VAR} and leaves unknown ones literal.
+Shell $(...) command-substitution survives untouched.
 Loops/conditionals live in the python action, never in the template file.
 """
 from __future__ import annotations

scripts/engine/lib/ufw.py

@@ -1,9 +1,8 @@
-# engine/lib/ufw.py
-"""ctx.ufw — ufw + ufw-docker rule application with self-healing marker migration.
+# scripts/engine/lib/ufw.py
+"""Apply ufw and ufw-docker rules.
 
-Injects the APEX marker block into before/after.rules, deleting BOTH the old ROOT
-and the new APEX blocks first (so re-running after cutover migrates the marker in
-place — SP2 §6). Docker rules go through ufw-docker; host rules through plain ufw.
+Replace existing supported marker blocks before inserting current rules.
+Docker rules go through ufw-docker; host rules through plain ufw.
 """
 from __future__ import annotations
 
@@ -68,10 +67,10 @@ class Ufw:
             self.log.warn("Neither docker nor host ufw rules directory found; nothing to apply.")
             return
         if not self.sys.ok(["sudo", "ufw", "--version"]):
-            self.log.error("ufw not installed/working via sudo. Run configure/base first.")
+            self.log.error("ufw not installed/working via sudo. Run config/packages first.")
             raise SystemExit(1)
         if not (os.path.isfile(UFW_DOCKER_BIN) and os.access(UFW_DOCKER_BIN, os.X_OK)):
-            self.log.error(f"ufw-docker not found at {UFW_DOCKER_BIN}. Run configure/base first.")
+            self.log.error(f"ufw-docker not found at {UFW_DOCKER_BIN}. Run config/ufw-docker first.")
             raise SystemExit(1)
 
         if config_dir:

scripts/engine/overlay.py

@@ -1,4 +1,4 @@
-# engine/overlay.py
+# scripts/engine/overlay.py
 """Resolve action names across commons + proprietaries.
 
 proprietaries first (override), commons second. Listing uses static ast scanning
@@ -9,10 +9,12 @@ from __future__ import annotations
 import ast
 import importlib.util
 import os
+import re
 from dataclasses import dataclass
 from typing import List, Optional
 
 ACTION_EXT = ".py"
+ACTION_NAME = re.compile(r"[A-Za-z0-9_-]+/[A-Za-z0-9_-]+")
 
 
 @dataclass
@@ -40,11 +42,15 @@ def _rel_names(root: Optional[str]):
                 continue
             full = os.path.join(dirpath, fn)
             rel = os.path.relpath(full, root)[: -len(ACTION_EXT)]
-            out[rel.replace(os.sep, "/")] = full
+            name = rel.replace(os.sep, "/")
+            if ACTION_NAME.fullmatch(name):
+                out[name] = full
     return out
 
 
 def resolve(name: str, commons_root: str, prop_root: Optional[str]) -> Optional[Resolved]:
+    if not ACTION_NAME.fullmatch(name):
+        return None
     rel = name + ACTION_EXT
     commons_path = os.path.join(commons_root, rel)
     c = os.path.isfile(commons_path)

scripts/githooks/pre-commit

@@ -6,4 +6,4 @@ else
     APEX_CMD="$(git rev-parse --show-toplevel)/commons/apex"
     [[ -x "$APEX_CMD" ]] || APEX_CMD="$(git rev-parse --show-toplevel)/apex"  # standalone commons repo
 fi
-"$APEX_CMD" utils/lint-docker-compose --hook || { echo "[pre-commit] Aborting: apex utils/lint-docker-compose failed."; exit 1; }
+"$APEX_CMD" util/lint-docker-compose --hook || { echo "[pre-commit] Aborting: apex util/lint-docker-compose failed."; exit 1; }

scripts/provisioning/__init__.py

@@ -0,0 +1 @@
+"""Host-side Docker installation and SSH handover."""

scripts/provisioning/access.py

@@ -54,8 +54,21 @@ def include_arguments(value):
     return arguments
 
 
+MANUAL_PROFILES = ('debian', 'proxmox')
+OPERATOR_MATCH = """Match User adam
+AuthenticationMethods publickey
+PubkeyAuthentication yes
+PasswordAuthentication no
+KbdInteractiveAuthentication no
+PermitEmptyPasswords no
+X11Forwarding no""".splitlines()
+
+
 class Host:
-    def __init__(self, root=Path('/'), runner=subprocess.run):
+    def __init__(self, root=Path('/'), runner=subprocess.run, profile=None):
+        if profile is not None and profile not in MANUAL_PROFILES:
+            raise AccessError('Unknown SSH access profile')
+        self.profile = profile
         self.root = Path(root).resolve()
         self.runner = runner
 
@@ -136,7 +149,16 @@ class Host:
             remaining -= len(content)
             if remaining < 0:
                 raise AccessError('SSH configuration exceeds supported inspection limits; manual reconciliation required')
-            for line in content.decode().splitlines():
+            lines = content.decode().splitlines()
+            if self.profile and path == self.path('/etc/ssh/sshd_config'):
+                # The legacy action appended this exact terminal block. Any additional
+                # directives would change its meaning and require manual inspection.
+                marker = '# APEX operator SSH policy'
+                if marker in lines:
+                    position = lines.index(marker)
+                    if [item.strip() for item in lines[position + 1:] if item.strip()] == OPERATOR_MATCH:
+                        lines = lines[:position]
+            for line in lines:
                 line = line.lstrip()
                 if not line or line.startswith('#'):
                     continue
@@ -236,6 +258,10 @@ class Host:
     def verify_active(self, ports, final, address):
         args, pid = self.live_arguments()
         self.verify_policy([arg for arg in args if arg != '-D'], ports, final, address)
+        if self.listener_ports(pid) != set(ports):
+            raise AccessError('Active SSH listening ports differ from the declared policy')
+
+    def listener_ports(self, pid):
         output = self.command(['ss', '-H', '-ltnp']).stdout
         actual_ports = set()
         for line in output.splitlines():
@@ -247,19 +273,41 @@ class Host:
             if len(fields) < 5 or not fields[3].rsplit(':', 1)[-1].isdecimal():
                 raise AccessError('Unable to verify the active SSH listening sockets')
             actual_ports.add(int(fields[3].rsplit(':', 1)[-1]))
-        if actual_ports != set(ports):
-            raise AccessError('Active SSH listening ports differ from the declared policy')
+        return actual_ports
+
+    @property
+    def final_ports(self):
+        return [22, 2222] if self.profile == 'proxmox' else [2222]
+
+    def authentication_options(self):
+        proxmox = self.profile == 'proxmox'
+        options = {'PermitRootLogin': 'prohibit-password' if proxmox else 'no',
+                   'PasswordAuthentication': 'no', 'KbdInteractiveAuthentication': 'no',
+                   'PubkeyAuthentication': 'yes', 'AuthenticationMethods': 'publickey'}
+        if not proxmox:
+            options['AllowUsers'] = 'adam'
+        options.update(X11Forwarding='no', PermitEmptyPasswords='no')
+        return options
 
     def policy(self, ports, final, address):
         args = ['/usr/sbin/sshd']
         for port in sorted(set(ports)):
             args += ['-p', str(port)]
         if final:
-            for value in ('PermitRootLogin=no', 'PasswordAuthentication=no',
-                          'KbdInteractiveAuthentication=no', 'PubkeyAuthentication=yes',
-                          'AuthenticationMethods=publickey', 'AllowUsers=adam',
-                          'X11Forwarding=no', 'PermitEmptyPasswords=no'):
-                args += ['-o', value]
+            for key, value in self.authentication_options().items():
+                args += ['-o', f'{key}={value}']
+        elif self.profile:
+            # Keep existing managed authentication options when adding the probe port.
+            current = iter(self.service_arguments()[1:])
+            for flag in current:
+                if flag == '-D':
+                    continue
+                value = next(current, '')
+                if flag == '-p' and value.isdecimal():
+                    continue
+                if flag != '-o' or not re.fullmatch(r'[A-Za-z]+=[A-Za-z0-9,-]+', value):
+                    raise AccessError('Cannot safely extend the existing SSH invocation')
+                args += [flag, value]
         self.verify_policy(args, ports, final, address)
         return args
 
@@ -277,19 +325,15 @@ class Host:
                 values.setdefault(key, []).append(value)
             if set(values.get('port', [])) != {str(p) for p in ports}:
                 raise AccessError('Effective SSH ports do not match the handover policy')
-            if final and any(values.get(k) != [v] for k, v in {
-                    'permitrootlogin': 'no', 'passwordauthentication': 'no',
-                    'kbdinteractiveauthentication': 'no', 'pubkeyauthentication': 'yes',
-                    'authenticationmethods': 'publickey', 'allowusers': 'adam',
-                    'x11forwarding': 'no', 'permitemptypasswords': 'no'}.items()):
-                raise AccessError('Effective SSH authentication policy is inconsistent')
-
-    def current_final(self, address):
-        try:
-            self.verify_active([2222], True, address)
-            return True
-        except AccessError:
-            return False
+            if final:
+                expected = {key.lower(): [value] for key, value in self.authentication_options().items()}
+                if self.profile == 'proxmox':
+                    if values.get('permitrootlogin') == ['without-password']:
+                        values['permitrootlogin'] = ['prohibit-password']
+                    if any(key in values for key in ('allowusers', 'denyusers', 'allowgroups', 'denygroups')):
+                        raise AccessError('SSH account restrictions conflict with Proxmox root-key access')
+                if any(values.get(key) != value for key, value in expected.items()):
+                    raise AccessError('Effective SSH authentication policy is inconsistent')
 
     def install_policy(self, ports, final, address):
         args = self.policy(ports, final, address)
@@ -303,28 +347,13 @@ class Host:
         self.command(['systemctl', 'restart', 'ssh.service'])
         self.verify_active(ports, final, address)
 
-    def firewall_needed(self):
-        result = self.command(['ufw', 'status'], False)
-        if result.returncode or 'Status: active' not in result.stdout:
-            return False
-        if re.search(r'^2222/tcp(?:\s|\(v6\)).*ALLOW', result.stdout, re.MULTILINE):
-            return False
-        return True
-
-    def open_firewall(self):
-        # Existing allows remain owned by their administrator. Only our commented rule is removed.
-        self.command(['ufw', 'allow', '2222/tcp', 'comment', 'apex-init-access'])
-
-    def close_firewall(self):
-        self.command(['ufw', '--force', 'delete', 'allow', '2222/tcp', 'comment', 'apex-init-access'])
-
     def arm(self):
-        bundle = Path(__file__).resolve().parents[2]
+        bundle = Path(__file__).resolve().parents[1]
         if not re.fullmatch(r'/usr/local/lib/apex-provisioner/[a-f0-9]{64}', str(bundle)):
             raise AccessError('Access runner must execute from the installed root-owned bundle')
         service = ('[Unit]\nDescription=APEX pending access rollback\nAfter=local-fs.target\n'
                    '[Service]\nType=oneshot\nWorkingDirectory=' + str(bundle) + '\n'
-                   'ExecStart=/usr/bin/python3 -m engine.provisioning.access tick\n')
+                   'ExecStart=/usr/bin/python3 -m provisioning.access tick\n')
         timer = ('[Unit]\nDescription=APEX access rollback deadline and boot recovery\n'
                  '[Timer]\nOnBootSec=5s\nOnUnitActiveSec=10s\nAccuracySec=1s\n'
                  '[Install]\nWantedBy=timers.target\n')
@@ -345,8 +374,6 @@ class Host:
         self.command(['/usr/sbin/sshd', '-t'])
         self.command(['systemctl', 'daemon-reload'])
         self.command(['systemctl', 'restart', 'ssh.service'])
-        if state.get('firewall_added'):
-            self.close_firewall()
 
 
 def atomic(path, data, mode=0o600):
@@ -374,22 +401,24 @@ def atomic(path, data, mode=0o600):
 
 
 class Transition:
-    def __init__(self, host=None, now=time.monotonic, security=None):
-        from .security import Security
+    def __init__(self, host=None, now=time.monotonic, profile=None):
+        if profile is not None and profile not in MANUAL_PROFILES:
+            raise AccessError('Unknown SSH access profile')
+        self.profile = profile
         self.host = host or Host()
-        self.security = security if security is not None else Security(self.host)
+        self.host.profile = profile
         self.now = now
         self.directory = self.host.path('/var/lib/apex-init/access')
 
     def load(self):
-        path = self.directory / 'state.json'
+        path = self.directory / 'manual-state.json'
         return json.loads(path.read_text()) if path.exists() else None
 
     def save(self, state):
-        atomic(self.directory / 'state.json', json.dumps(state, sort_keys=True).encode())
+        atomic(self.directory / 'manual-state.json', json.dumps(state, sort_keys=True).encode())
 
     def execute(self, operation, token=None, original_port=None, client_address=None):
-        # Rollback must never wait for a stalled enrollment stage holding init.lock.
+        # The watchdog must recover SSH even when another APEX action holds init.lock.
         if operation in ('tick', 'rollback'):
             return self._execute(operation, token, original_port, client_address)
         from .state import ProvisionError, exclusive_lock
@@ -413,6 +442,22 @@ class Transition:
         with os.fdopen(descriptor, 'w') as lock:
             fcntl.flock(lock, fcntl.LOCK_EX)
             state = self.load()
+            active = ('transition', 'final', 'recovery_pending')
+            # A historical bundle owns its pending rollback and the shared timer.
+            legacy_path = self.directory / 'state.json'
+            if legacy_path.exists():
+                legacy = json.loads(legacy_path.read_text())
+                if legacy['status'] in active or legacy.get('disarm_pending'):
+                    raise AccessError('Another access workflow is pending; wait for rollback before retrying')
+            if state and self.profile and state.get('profile') != self.profile and operation != 'tick':
+                if operation != 'begin' or state['status'] in active:
+                    raise AccessError('Access transition belongs to a different platform')
+                state = None
+            if state and state.get('disarm_pending') and state['status'] in ('committed', 'rolled_back'):
+                self._disarm(state)
+            if (state and token and token == state.get('token')
+                    and (operation, state['status']) in (('commit', 'committed'), ('rollback', 'rolled_back'))):
+                return {'status': state['status']}
             if state and state['status'] == 'recovery_pending':
                 self._rollback(state)
                 if operation != 'tick':
@@ -437,43 +482,9 @@ class Transition:
                     raise AccessError('Invalid SSH client address') from None
                 self.host.preflight()
                 path = self.host.path('/etc/systemd/system/ssh.service.d/apex-init.conf')
-                preserve_policy = False
-                if state and state['status'] == 'committed':
-                    if state['fingerprint'] != self.host.fingerprint():
-                        raise AccessError('Committed access configuration changed; reconcile before retrying')
-                    self.host.verify_active([2222], True, client_address)
-                    if state.get('security'):
-                        self.security.verify(state['security'])
-                        return {'status': 'committed'}
-                    preserve_policy = True
-                if original_port == 2222 and not path.exists() and self.host.current_final(client_address):
-                    preserve_policy = True
-                security_state = self.security.snapshot(self.security.preflight(), original_port, client_address)
-                self.host.arm()
-                state = {'status': 'transition', 'token': token, 'deadline': self.now() + 300,
-                         'boot_id': self.host.boot_id(), 'client_address': client_address,
-                         'original_port': original_port, 'firewall_added': False,
-                         'preserve_policy': preserve_policy, 'security': security_state,
-                         'previous': base64.b64encode(path.read_bytes()).decode() if path.exists() else None}
-                self.save(state)
-                try:
-                    self.security.activate(state['security'], lambda value: self._save_security(state, value))
-                    if not preserve_policy:
-                        self.host.install_policy([original_port, 2222], False, client_address)
-                    state['fingerprint'] = self.host.fingerprint()
-                    self.save(state)
-                except Exception:
-                    self._rollback(state)
-                    raise
-                return {'status': 'transition'}
-            if not state or (not pending and state['status'] != 'existing') or token != state['token']:
+                return self._begin_manual(token, original_port, client_address, path)
+            if not pending or token != state['token']:
                 raise AccessError('No matching pending access transition')
-            if state['status'] == 'existing':
-                if operation == 'rollback':
-                    state['status'] = 'rolled_back'
-                    self.save(state)
-                    return {'status': 'rolled_back'}
-                raise AccessError('Legacy access verification lacks security activation; retry initialization')
             if operation == 'rollback':
                 self._rollback(state)
                 return {'status': 'rolled_back'}
@@ -484,8 +495,10 @@ class Transition:
                     raise AccessError('Access transition is not ready for final policy')
                 try:
                     if not state.get('preserve_policy'):
-                        self.host.install_policy([2222], True, state['client_address'])
-                    self.security.finalize(state['security'], lambda value: self._save_security(state, value))
+                        state['ssh_changed'] = True
+                        self.save(state)
+                        self.host.install_policy(self.host.final_ports, True, state['client_address'])
+                    state['authenticated_after_us'] = time.monotonic_ns() // 1000
                     state['status'] = 'final'
                     state['fingerprint'] = self.host.fingerprint()
                     self.save(state)
@@ -494,26 +507,48 @@ class Transition:
                     raise
                 return {'status': 'final'}
             if operation == 'commit' and state['status'] == 'final':
-                self.host.verify_active([2222], True, state['client_address'])
-                self.security.verify(state['security'])
+                self.host.verify_active(self.host.final_ports, True, state['client_address'])
                 state['status'] = 'committed'
+                state['disarm_pending'] = True
                 self.save(state)  # Timer observes this before it can restore anything.
-                self.host.disarm()
+                self._disarm(state)
                 return {'status': 'committed'}
             raise AccessError('Invalid access transition operation')
 
-    def _save_security(self, state, value):
-        state['security'] = value
+    def _begin_manual(self, token, original_port, address, path):
+        _, pid = self.host.live_arguments()
+        ports = self.host.listener_ports(pid)
+        if original_port not in ports:
+            raise AccessError('Original SSH port is not active on the managed daemon')
+        preserve_policy = False
+        if ports == set(self.host.final_ports):
+            try:
+                self.host.verify_active(self.host.final_ports, True, address)
+                preserve_policy = True
+            except AccessError:
+                pass
+        previous = base64.b64encode(path.read_bytes()).decode() if path.exists() else None
+        state = {'status': 'transition', 'profile': self.profile, 'token': token,
+                 'deadline': self.now() + 300, 'boot_id': self.host.boot_id(),
+                 'bundle': str(Path(__file__).resolve().parents[1]),
+                 'client_address': address, 'original_port': original_port, 'previous': previous,
+                 'preserve_policy': preserve_policy, 'ssh_changed': 2222 not in ports}
+        self.host.arm()
         self.save(state)
+        try:
+            if 2222 not in ports:
+                self.host.install_policy(sorted(ports | {2222}), False, address)
+            state['fingerprint'] = self.host.fingerprint()
+            state['authenticated_after_us'] = time.monotonic_ns() // 1000
+            self.save(state)
+        except Exception:
+            self._rollback(state)
+            raise
+        return {'status': 'transition'}
 
     def _rollback(self, state):
         errors = []
-        if state.get('security'):
-            try:
-                self.security.restore(state['security'], lambda value: self._save_security(state, value))
-            except Exception:
-                errors.append('security')
-        if not state.get('ssh_restored'):
+        if not state.get('ssh_restored') and state.get('ssh_changed', True):
             try:
                 self.host.restore(state)
                 state['ssh_restored'] = True
@@ -527,14 +562,20 @@ class Transition:
                               '; watchdog remains armed; inspect the root-owned access record')
         state['status'] = 'rolled_back'
         state.pop('recovery_errors', None)
+        state['disarm_pending'] = True
         self.save(state)
+        self._disarm(state)
+
+    def _disarm(self, state):
         self.host.disarm()
+        state.pop('disarm_pending', None)
+        self.save(state)
 
 
 def main(argv=None):
-    from .security import SecurityError
     parser = argparse.ArgumentParser()
-    parser.add_argument('operation', choices=['preflight', 'begin', 'finalize', 'commit', 'rollback', 'tick'])
+    parser.add_argument('operation', choices=['begin', 'finalize', 'commit', 'rollback', 'tick'])
+    parser.add_argument('--profile', choices=MANUAL_PROFILES)
     parser.add_argument('--token')
     parser.add_argument('--original-port', type=int)
     parser.add_argument('--client-address')
@@ -542,14 +583,12 @@ def main(argv=None):
     if os.geteuid() != 0:
         parser.error('Access handover requires root')
     try:
-        if args.operation == 'preflight':
-            Host().bootstrap_preflight()
-            result = {'status': 'prerequisites-verified'}
-        else:
-            result = Transition().execute(args.operation, args.token, args.original_port, args.client_address)
+        if args.operation != 'tick' and not args.profile:
+            parser.error('A host profile is required')
+        result = Transition(profile=args.profile).execute(args.operation, args.token, args.original_port, args.client_address)
         print(json.dumps(result))
         return 0
-    except (AccessError, SecurityError) as exc:
+    except AccessError as exc:
         print(str(exc), file=sys.stderr)
         return 1
 

scripts/provisioning/docker.py

@@ -0,0 +1,312 @@
+"""Verify or install a local Docker Engine and CLI plugins."""
+from __future__ import annotations
+
+from contextlib import contextmanager
+import fcntl
+from functools import wraps
+import hashlib
+import json
+import os
+from pathlib import Path
+import re
+import shutil
+import subprocess
+import tempfile
+import urllib.request
+
+
+class DockerError(RuntimeError):
+    """A required Docker postcondition could not be established."""
+
+
+def expected_errors(operation):
+    @wraps(operation)
+    def guarded(*args, **kwargs):
+        try:
+            return operation(*args, **kwargs)
+        except (OSError, ValueError) as exc:
+            raise DockerError("Docker filesystem or saved state is invalid; inspect the target before retrying.") from exc
+    return guarded
+
+
+DOCKER_PACKAGES = ("docker-ce", "docker-ce-cli", "containerd.io",
+                   "docker-buildx-plugin", "docker-compose-plugin")
+DOCKER_CONFLICTS = ("docker.io", "docker-doc", "docker-compose", "docker-buildx",
+                    "podman-docker", "containerd", "runc")
+DOCKER_CONFIG = {"data-root": "/srv/docker", "log-driver": "json-file",
+                 "log-opts": {"max-size": "20m", "max-file": "3"}}
+POLICY = Path("/usr/sbin/policy-rc.d")
+POLICY_STATE = Path("/usr/sbin/.apex-policy-rc.d")
+APT = ["apt-get", "-o", "DPkg::Lock::Timeout=60", "-o",
+       "Acquire::Retries=2", "-o", "Acquire::http::Timeout=30", "-o",
+       "Acquire::https::Timeout=30"]
+
+
+def run(args, *, check=True, timeout=300):
+    try:
+        result = subprocess.run(args, text=True, capture_output=True, timeout=timeout,
+                                env={**os.environ, "LC_ALL": "C", "DEBIAN_FRONTEND": "noninteractive"})
+    except (OSError, subprocess.TimeoutExpired) as exc:
+        raise DockerError(f"Cannot complete {args[0]}: {exc}") from exc
+    if check and result.returncode:
+        raise DockerError(f"{args[0]} failed with exit status {result.returncode}; inspect target system logs.")
+    return result
+
+
+def package_versions():
+    architecture = run(["dpkg", "--print-architecture"]).stdout.strip()
+    result = run(["dpkg-query", "-W", "-f=${binary:Package}\t${Status}\t${Version}\n"])
+    installed = {}
+    for line in result.stdout.splitlines():
+        fields = line.split("\t")
+        if len(fields) == 3 and fields[1].split()[1:] == ["ok", "installed"]:
+            installed[fields[0].removesuffix(":" + architecture)] = fields[2]
+    return installed
+
+
+def write_managed(path, content, mode=0o644):
+    path = Path(path)
+    for parent in (path, *path.parents):
+        if parent.is_symlink():
+            raise DockerError(f"Refusing managed symlink: {parent}")
+    payload = content.encode() if isinstance(content, str) else content
+    if path.exists() and path.read_bytes() == payload:
+        if path.stat().st_mode & 0o777 != mode:
+            path.chmod(mode)
+        return False
+    path.parent.mkdir(parents=True, exist_ok=True)
+    fd, temporary = tempfile.mkstemp(prefix=".apex-", dir=path.parent)
+    try:
+        with os.fdopen(fd, "wb") as stream:
+            stream.write(payload)
+            stream.flush()
+            os.fsync(stream.fileno())
+        os.chmod(temporary, mode)
+        os.replace(temporary, path)
+    finally:
+        if os.path.exists(temporary):
+            os.unlink(temporary)
+    return True
+
+
+def restore_service_policy():
+    if not POLICY_STATE.exists():
+        return
+    backup = POLICY_STATE / "original"
+    if (POLICY_STATE.is_symlink() or POLICY_STATE.stat().st_uid != os.geteuid()):
+        raise DockerError("Refusing untrusted service policy recovery state.")
+    ready = POLICY_STATE / "ready"
+    if not ready.exists():
+        # The live policy is untouched until the backup is complete.
+        shutil.rmtree(POLICY_STATE)
+        return
+    if POLICY.exists() or POLICY.is_symlink():
+        original_matches = (backup.is_file() and not backup.is_symlink() and not POLICY.is_symlink() and
+                            POLICY.read_bytes() == backup.read_bytes())
+        symlink_matches = (backup.is_symlink() and POLICY.is_symlink() and
+                           os.readlink(backup) == os.readlink(POLICY))
+        guard_matches = not POLICY.is_symlink() and POLICY.read_bytes() == b"#!/bin/sh\nexit 101\n"
+        if not (original_matches or symlink_matches or guard_matches):
+            raise DockerError("Service policy changed outside APEX; recovery requires explicit reconciliation.")
+    if backup.exists() or backup.is_symlink():
+        # Copy first: an interrupted restoration must retain its recovery copy.
+        temporary = POLICY_STATE / "restoring"
+        if temporary.exists() or temporary.is_symlink():
+            temporary.unlink()
+        shutil.copy2(backup, temporary, follow_symlinks=False)
+        os.replace(temporary, POLICY)
+    elif (POLICY_STATE / "absent").exists():
+        POLICY.unlink(missing_ok=True)
+    else:
+        raise DockerError("Service policy recovery is incomplete; inspect /usr/sbin/.apex-policy-rc.d.")
+    ready.unlink()
+    shutil.rmtree(POLICY_STATE)
+
+
+@contextmanager
+def package_policy_lock():
+    lock_path = Path("/run/lock/apex-package-policy.lock")
+    lock_path.parent.mkdir(parents=True, exist_ok=True)
+    with lock_path.open("a") as lock:
+        try:
+            fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
+        except BlockingIOError:
+            raise DockerError("Another APEX package operation is running.") from None
+        restore_service_policy()
+        yield
+
+
+@contextmanager
+def service_start_guard():
+    """Temporarily inhibit service starts while the caller holds the package lock."""
+    POLICY_STATE.mkdir(mode=0o700)
+    try:
+        if POLICY.exists() or POLICY.is_symlink():
+            shutil.copy2(POLICY, POLICY_STATE / "original", follow_symlinks=False)
+        else:
+            (POLICY_STATE / "absent").touch()
+        (POLICY_STATE / "ready").touch()
+        # Replace the path itself, preserving any original symlink in the backup.
+        temporary = POLICY_STATE / "guard"
+        temporary.write_text("#!/bin/sh\nexit 101\n")
+        temporary.chmod(0o755)
+        os.replace(temporary, POLICY)
+        yield
+    finally:
+        restore_service_policy()
+
+
+def install_missing(packages):
+    installed = package_versions()
+    missing = [package for package in packages if package not in installed]
+    if not missing:
+        return []
+    run([*APT, "update", "--error-on=any"])
+    command = [*APT, "install", "-y", "--no-install-recommends", "--no-upgrade", "--no-remove", *missing]
+    plan = run([*command, "--simulate"]).stdout
+    if any(line.startswith("Remv ") or re.match(r"^Inst \S+ \[", line)
+           for line in plan.splitlines()):
+        raise DockerError("Missing packages require an upgrade/removal; reconcile packages explicitly before retrying.")
+    # Let dpkg finish before restoring the service-start policy.
+    run(command, timeout=None)
+    remaining = set(missing) - package_versions().keys()
+    if remaining:
+        raise DockerError("Required packages not installed: " + ", ".join(sorted(remaining)))
+    return missing
+
+
+def docker_preflight(installed):
+    conflicts = sorted(set(DOCKER_CONFLICTS) & installed.keys())
+    if conflicts:
+        raise DockerError("Conflicting Docker packages require explicit reconciliation: " + ", ".join(conflicts))
+    path = Path("/etc/docker/daemon.json")
+    try:
+        config = json.loads(path.read_text()) if path.exists() else {}
+    except (OSError, ValueError) as exc:
+        raise DockerError("Cannot validate existing Docker daemon configuration.") from exc
+    if not isinstance(config, dict):
+        raise DockerError("Docker daemon configuration must be an object.")
+    for name, value in DOCKER_CONFIG.items():
+        if name in config and config[name] != value:
+            raise DockerError(f"Conflicting Docker {name}; reconcile explicitly before initialization.")
+    old_data = Path("/var/lib/docker")
+    if old_data.exists() and any(old_data.iterdir()):
+        raise DockerError("Existing /var/lib/docker data must be reconciled explicitly; no data migration is automatic.")
+    data = Path("/srv/docker")
+    if data.is_symlink() or data.parent.is_symlink():
+        raise DockerError("Refusing a redirected Docker data-root.")
+    if data.exists() and any(data.iterdir()) and config.get("data-root") != "/srv/docker":
+        raise DockerError("Unmanaged /srv/docker data requires explicit reconciliation.")
+    if "docker-ce" in installed:
+        active = run(["systemctl", "is-active", "--quiet", "docker"], check=False).returncode == 0
+        if active:
+            root = run(["docker", "info", "--format", "{{.DockerRootDir}}"] ).stdout.strip()
+            if root != "/srv/docker" or any(config.get(key) != value for key, value in DOCKER_CONFIG.items()):
+                raise DockerError("Running Docker has incompatible data-root/log policy; reconcile it before retrying.")
+    return {**config, **DOCKER_CONFIG}
+
+
+DOCKER_KEY_SHA256 = "1500c1f56fa9e26b9b8f42452a553675796ade0807cdce11975eb98170b3a570"
+LOCAL_DOCKER = ["docker", "--host", "unix:///var/run/docker.sock"]
+
+
+def _docker_source(arch):
+    key = Path("/etc/apt/keyrings/apex-docker.asc")
+    source = Path("/etc/apt/sources.list.d/apex-docker.sources")
+    content = ("Types: deb\nURIs: https://download.docker.com/linux/debian\nSuites: trixie\n"
+               f"Components: stable\nArchitectures: {arch}\nSigned-By: {key}\n")
+    for path in (key, source):
+        if any(part.is_symlink() for part in (path, *path.parents)):
+            raise DockerError(f"Refusing redirected APT path: {path}")
+    if source.exists() and source.read_text() != content:
+        raise DockerError(f"Reconcile existing Docker repository configuration: {source}")
+    if key.exists():
+        data = key.read_bytes()
+    else:
+        url = "https://download.docker.com/linux/debian/gpg"
+        with urllib.request.urlopen(url, timeout=30) as response:  # noqa: S310
+            data = response.read(1024 * 1024)
+    if hashlib.sha256(data).hexdigest() != DOCKER_KEY_SHA256:
+        raise DockerError("Docker repository signing key differs from the pinned release key.")
+    write_managed(key, data)
+    write_managed(source, content)
+
+
+def _docker_capabilities(root, *, complete_plugins=False):
+    capabilities = {"docker_data_root": root}
+    plugins = {"docker-compose-plugin": ("compose", ["compose", "version", "--short"]),
+               "docker-buildx-plugin": ("buildx", ["buildx", "version"])}
+    missing = []
+    for package, (name, args) in plugins.items():
+        result = run([*LOCAL_DOCKER, *args], check=False)
+        if result.returncode or not result.stdout.strip():
+            missing.append(package)
+        else:
+            capabilities[name] = result.stdout.strip()
+    if missing:
+        if complete_plugins and "docker-ce" in package_versions():
+            with service_start_guard():
+                install_missing(missing)
+            return _docker_capabilities(root)
+        raise DockerError("Existing Docker needs functional plugins: " + ", ".join(missing))
+    if not root:
+        raise DockerError("Unable to verify the local Docker data-root.")
+    return capabilities
+
+
+@expected_errors
+def apply_docker_policy():
+    if Path("/usr/bin/pveversion").exists():
+        raise DockerError("Docker installation on a Proxmox host requires separate administration.")
+    release = dict(line.split("=", 1) for line in Path("/etc/os-release").read_text().splitlines()
+                   if "=" in line and not line.startswith("#"))
+    arch = run(["dpkg", "--print-architecture"]).stdout.strip()
+    if (release.get("ID", "").strip('"') != "debian" or
+            release.get("VERSION_ID", "").strip('"') != "13" or arch not in ("amd64", "arm64")):
+        raise DockerError("Docker installation supports stock Debian 13 amd64 or arm64.")
+    with package_policy_lock():
+        return _apply_docker_policy(arch)
+
+
+def _apply_docker_policy(arch):
+    info = [*LOCAL_DOCKER, "info", "--format", "{{.DockerRootDir}}"]
+    if shutil.which("docker"):
+        current = run(info, check=False)
+        if current.returncode == 0:
+            return _docker_capabilities(current.stdout.strip(), complete_plugins=True)
+    installed = package_versions()
+    source = Path("/etc/apt/sources.list.d/apex-docker.sources")
+    if (not source.exists() and
+            (shutil.which("docker") or set(DOCKER_PACKAGES).intersection(installed))):
+        raise DockerError("Existing Docker installation is inactive; reconcile it before retrying.")
+    config = docker_preflight(installed)
+    with service_start_guard():
+        install_missing(("ca-certificates",))
+        _docker_source(arch)
+        write_managed("/etc/docker/daemon.json", json.dumps(config, indent=2) + "\n")
+        install_missing(DOCKER_PACKAGES)
+    ensure_service("containerd")
+    ensure_service("docker")
+    root = run(info).stdout.strip()
+    if root != DOCKER_CONFIG["data-root"]:
+        raise DockerError("Docker started with an unexpected data-root; inspect its configuration.")
+    return _docker_capabilities(root)
+
+
+def ensure_service(name):
+    if run(["systemctl", "is-enabled", "--quiet", name], check=False).returncode:
+        run(["systemctl", "enable", name])
+    if run(["systemctl", "is-active", "--quiet", name], check=False).returncode:
+        run(["systemctl", "start", name])
+
+
+if __name__ == "__main__":
+    from .state import ProvisionError, exclusive_lock
+
+    try:
+        if os.geteuid() != 0 or not Path("/etc/debian_version").is_file():
+            raise DockerError("Docker setup requires root on Debian.")
+        with exclusive_lock("/var/lib/apex-init/init.lock"):
+            print(json.dumps(apply_docker_policy(), sort_keys=True))
+    except (DockerError, OSError, ProvisionError) as exc:
+        raise SystemExit(f"Docker setup failed: {exc}") from None

scripts/provisioning/manual_bundle.py

@@ -0,0 +1,78 @@
+"""Install an immutable watchdog bundle from the operator's checkout."""
+import errno
+import hashlib
+import os
+from pathlib import Path
+import stat
+import tempfile
+
+
+MODULES = ('__init__.py', 'access.py', 'state.py')
+
+
+def _trusted_directory(path):
+    info = path.lstat()
+    if not stat.S_ISDIR(info.st_mode) or info.st_uid != os.geteuid() or info.st_mode & 0o022:
+        raise RuntimeError(f'Untrusted provisioning directory: {path}')
+
+
+def _verify(destination, files):
+    _trusted_directory(destination)
+    package = destination / 'provisioning'
+    _trusted_directory(package)
+    # Python can create bytecode caches when the installed watchdog runs.
+    for path in destination.rglob('*'):
+        info = path.lstat()
+        if (info.st_uid != os.geteuid() or info.st_mode & 0o022
+                or not (stat.S_ISDIR(info.st_mode) or stat.S_ISREG(info.st_mode))):
+            raise RuntimeError(f'Untrusted installed provisioning file: {path}')
+    for name, data in files.items():
+        target = package / name
+        if not target.is_file() or target.read_bytes() != data:
+            raise RuntimeError('Installed provisioning bundle differs from the checkout')
+
+
+def install(root=Path('/')):
+    source = Path(__file__).resolve().parent
+    files = {}
+    digest = hashlib.sha256()
+    for name in MODULES:
+        path = source / name
+        if not stat.S_ISREG(path.lstat().st_mode):
+            raise RuntimeError(f'Unsafe provisioning source file: {path}')
+        data = files[name] = path.read_bytes()
+        digest.update(name.encode() + b'\0' + len(data).to_bytes(8, 'big') + data)
+
+    parent = root
+    _trusted_directory(parent)
+    for component in ('usr', 'local', 'lib', 'apex-provisioner'):
+        parent = parent / component
+        parent.mkdir(mode=0o755, exist_ok=True)
+        _trusted_directory(parent)
+    destination = parent / digest.hexdigest()
+    if destination.exists():
+        _verify(destination, files)
+        return destination
+
+    with tempfile.TemporaryDirectory(prefix='.install-', dir=parent) as temporary:
+        stage = Path(temporary) / 'bundle'
+        package = stage / 'provisioning'
+        stage.mkdir(mode=0o755)
+        package.mkdir(mode=0o755)
+        for name, data in files.items():
+            target = package / name
+            target.write_bytes(data)
+            target.chmod(0o644)
+        try:
+            stage.rename(destination)
+        except OSError as error:
+            if error.errno not in (errno.EEXIST, errno.ENOTEMPTY):
+                raise
+            _verify(destination, files)
+    return destination
+
+
+if __name__ == '__main__':
+    if os.geteuid() != 0:
+        raise SystemExit('Run bundle installation as root.')
+    print(install())

scripts/provisioning/state.py

@@ -0,0 +1,37 @@
+"""Serialize host configuration changes across provisioning commands."""
+import contextlib
+import fcntl
+import os
+from pathlib import Path
+import stat
+
+
+class ProvisionError(RuntimeError):
+    pass
+
+
+def safe_path(path):
+    path = Path(path).absolute()
+    for part in (*reversed(path.parents), path):
+        try:
+            info = part.lstat()
+        except FileNotFoundError:
+            continue
+        if stat.S_ISLNK(info.st_mode):
+            raise ProvisionError('managed path contains a symbolic link')
+    return path
+
+
+@contextlib.contextmanager
+def exclusive_lock(path):
+    path = safe_path(path)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    fd = os.open(path, os.O_CREAT | os.O_RDWR | os.O_NOFOLLOW, 0o600)
+    try:
+        try:
+            fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
+        except BlockingIOError:
+            raise ProvisionError('another initialization is running; inspect status before retrying') from None
+        yield
+    finally:
+        os.close(fd)

tests/provisioning/test_docker_host.py

@@ -0,0 +1,141 @@
+import fcntl
+import hashlib
+import io
+import json
+from pathlib import Path
+from subprocess import CompletedProcess
+import tempfile
+import unittest
+from unittest.mock import patch
+
+from provisioning import docker
+
+
+class ManualDockerTests(unittest.TestCase):
+    def setUp(self):
+        tmp = tempfile.TemporaryDirectory()
+        self.addCleanup(tmp.cleanup)
+        self.root = Path(tmp.name).resolve()
+        (self.root / "etc").mkdir()
+        (self.root / "etc/os-release").write_text("ID=debian\nVERSION_ID=13\n")
+        self.policy = self.root / "usr/sbin/policy-rc.d"
+        self.policy.parent.mkdir(parents=True)
+        self.state = self.policy.parent / ".apex-policy-rc.d"
+        self.commands = []
+        self.arch = "arm64"
+        self.running = False
+        self.data_root = "/custom/data"
+        self.missing_buildx = False
+        self.versions = {}
+        self.installations = []
+
+    def run_command(self, cmd, **kwargs):
+        self.commands.append(cmd)
+        if cmd == ["dpkg", "--print-architecture"]:
+            return CompletedProcess(cmd, 0, self.arch + "\n", "")
+        if cmd[-1] == "{{.DockerRootDir}}":
+            return CompletedProcess(cmd, 0 if self.running else 1, self.data_root + "\n", "")
+        if cmd[-2:] in (["version", "--short"], ["buildx", "version"]):
+            if self.missing_buildx and cmd[-2:] == ["buildx", "version"]:
+                if ["docker-buildx-plugin"] not in self.installations:
+                    return CompletedProcess(cmd, 1, "", "plugin missing")
+            return CompletedProcess(cmd, 0, "installed\n", "")
+        raise AssertionError(cmd)
+
+    def apply(self):
+        def target(path):
+            path = Path(path)
+            return path if self.root in path.parents else self.root / str(path).lstrip("/")
+
+        with (patch.object(docker, "Path", side_effect=target),
+              patch.object(docker, "POLICY", self.policy),
+              patch.object(docker, "POLICY_STATE", self.state),
+              patch.object(docker, "run", side_effect=self.run_command),
+              patch.object(docker, "package_versions", return_value=self.versions),
+              patch.object(docker.shutil, "which", return_value="/usr/bin/docker" if self.running else None),
+              patch.object(docker, "install_missing", side_effect=self.installations.append)):
+            return docker.apply_docker_policy()
+
+    def test_working_engine_keeps_local_data_root_without_writes(self):
+        self.running = True
+        self.versions = {"docker.io": "1"}
+        self.assertEqual(self.apply()["docker_data_root"], "/custom/data")
+        self.assertEqual(self.installations, [])
+        self.assertFalse((self.root / "etc/docker").exists())
+
+    def test_working_engine_restores_interrupted_policy(self):
+        self.running = True
+        self.state.mkdir()
+        (self.state / "original").write_text("provider policy")
+        (self.state / "ready").touch()
+        self.policy.write_text("#!/bin/sh\nexit 101\n")
+        self.assertEqual(self.apply()["docker_data_root"], "/custom/data")
+        self.assertEqual(self.policy.read_text(), "provider policy")
+        self.assertFalse(self.state.exists())
+
+    def test_working_engine_refuses_an_active_package_operation(self):
+        self.running = True
+        lock = self.root / "run/lock/apex-package-policy.lock"
+        lock.parent.mkdir(parents=True)
+        with lock.open("a") as other:
+            fcntl.flock(other, fcntl.LOCK_EX)
+            with self.assertRaisesRegex(docker.DockerError, "Another APEX"):
+                self.apply()
+
+    def test_pve_refused_before_packages_or_daemon(self):
+        binary = self.root / "usr/bin/pveversion"
+        binary.parent.mkdir(parents=True)
+        binary.touch()
+        with self.assertRaisesRegex(docker.DockerError, "Proxmox"):
+            self.apply()
+        self.assertEqual(self.commands, [])
+
+    def test_unsupported_architecture_refused(self):
+        self.arch = "i386"
+        with self.assertRaisesRegex(docker.DockerError, "amd64|arm64"):
+            self.apply()
+        self.assertEqual(self.installations, [])
+
+    def test_existing_runtime_packages_not_replaced(self):
+        self.versions = {"containerd": "1.7"}
+        with self.assertRaisesRegex(docker.DockerError, "containerd"):
+            self.apply()
+        self.assertEqual(self.installations, [])
+
+    def test_fresh_arm64_install_verifies_the_pinned_key_and_starts_local_engine(self):
+        key = b"test repository signing key"
+        self.data_root = "/srv/docker"
+
+        def start(service):
+            if service == "docker":
+                self.running = True
+
+        with (patch.object(docker.urllib.request, "urlopen", return_value=io.BytesIO(key)),
+              patch.object(docker, "DOCKER_KEY_SHA256", hashlib.sha256(key).hexdigest()),
+              patch.object(docker, "ensure_service", side_effect=start)):
+            self.assertEqual(self.apply()["docker_data_root"], "/srv/docker")
+        source = (self.root / "etc/apt/sources.list.d/apex-docker.sources").read_text()
+        self.assertIn("Architectures: arm64", source)
+        self.assertIn("Signed-By: ", source)
+        self.assertEqual((self.root / "etc/apt/keyrings/apex-docker.asc").read_bytes(), key)
+        self.assertEqual(json.loads((self.root / "etc/docker/daemon.json").read_text())["data-root"],
+                         "/srv/docker")
+        self.assertIn("docker-compose-plugin", self.installations[-1])
+        self.assertIn("docker-buildx-plugin", self.installations[-1])
+        self.assertNotIn("ufw", self.installations[-1])
+
+    def test_untrusted_signing_key_cannot_publish_apt_source_or_install_docker(self):
+        with patch.object(docker.urllib.request, "urlopen", return_value=io.BytesIO(b"wrong key")):
+            with self.assertRaisesRegex(docker.DockerError, "signing key"):
+                self.apply()
+        self.assertFalse((self.root / "etc/apt/sources.list.d/apex-docker.sources").exists())
+        self.assertFalse((self.root / "etc/docker/daemon.json").exists())
+        self.assertEqual(self.installations, [("ca-certificates",)])
+
+    def test_working_ce_engine_only_installs_the_missing_plugin(self):
+        self.running = True
+        self.missing_buildx = True
+        self.versions = {"docker-ce": "1"}
+        self.assertEqual(self.apply()["docker_data_root"], "/custom/data")
+        self.assertEqual(self.installations, [["docker-buildx-plugin"]])
+        self.assertFalse((self.root / "etc/docker").exists())

tests/provisioning/test_docker_packages.py

@@ -0,0 +1,146 @@
+from pathlib import Path
+import subprocess
+import tempfile
+import unittest
+from unittest.mock import patch
+
+from provisioning import docker
+
+
+class Host:
+    def __init__(self, root):
+        self.root = root
+        self.commands = []
+        self.installed = {}
+        self.partial = {}
+        self.plan = ''
+        self.path('/usr/sbin').mkdir(parents=True)
+
+    def path(self, path):
+        path = Path(path)
+        if str(path).startswith(str(self.root)):
+            return path
+        return self.root / str(path).lstrip('/')
+
+    def command(self, args, **kwargs):
+        self.commands.append(args)
+        code, output = 0, ''
+        if args[:2] == ['dpkg', '--print-architecture']:
+            output = 'amd64\n'
+        elif args[0] == 'dpkg-query':
+            output = ''.join(f'{name}\tinstall ok installed\t{version}\n' for name, version in self.installed.items())
+            output += ''.join(f'{name}\t{status}\t1.0\n' for name, status in self.partial.items())
+        elif args[0] == 'apt-get' and 'install' in args:
+            if '--simulate' in args:
+                output = self.plan
+            else:
+                for name in args[args.index('--no-remove') + 1:]:
+                    self.installed[name] = '1.0'
+                    self.partial.pop(name, None)
+        return subprocess.CompletedProcess(args, code, output, '')
+
+
+class DockerPackageTests(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.temp.cleanup)
+        self.host = Host(Path(self.temp.name).resolve())
+        for item in (
+            patch.object(docker, 'Path', self.host.path),
+            patch.object(docker, 'POLICY', self.host.path('/usr/sbin/policy-rc.d')),
+            patch.object(docker, 'POLICY_STATE', self.host.path('/usr/sbin/.apex-policy-rc.d')),
+            patch.object(docker.subprocess, 'run', side_effect=self.host.command),
+        ):
+            item.start()
+            self.addCleanup(item.stop)
+
+    def test_package_database_drives_installation_and_repeat_does_not_upgrade(self):
+        self.host.installed['curl'] = '7.0'
+        self.host.partial['rsyslog'] = 'deinstall ok config-files'
+        self.assertEqual(docker.install_missing(['curl', 'rsyslog']), ['rsyslog'])
+        self.assertEqual(self.host.installed['curl'], '7.0')
+        self.assertEqual(self.host.installed['rsyslog'], '1.0')
+        count = len(self.host.commands)
+        self.assertEqual(docker.install_missing(['curl', 'rsyslog']), [])
+        self.assertFalse(any(command[0] == 'apt-get' for command in self.host.commands[count:]))
+
+    def test_dependency_upgrade_is_rejected_before_install(self):
+        self.host.plan = 'Inst libc6 [2.0] (3.0 Debian:13)\n'
+        with self.assertRaisesRegex(docker.DockerError, 'upgrade/removal'):
+            docker.install_missing(['git'])
+        self.assertNotIn('git', self.host.installed)
+
+    def test_held_installed_packages_are_preserved(self):
+        self.host.partial['git'] = 'hold ok installed'
+        self.assertEqual(docker.install_missing(['git']), [])
+        self.assertFalse(any(command[0] == 'apt-get' for command in self.host.commands))
+
+    def test_conflicting_docker_config_is_rejected_without_host_mutation(self):
+        path = self.host.path('/etc/docker/daemon.json')
+        path.parent.mkdir(parents=True)
+        path.write_text('{"data-root": "/data/docker"}')
+        before = sorted(str(path.relative_to(self.host.root)) for path in self.host.root.rglob('*'))
+        with self.assertRaisesRegex(docker.DockerError, 'Conflicting Docker'):
+            docker.docker_preflight(self.host.installed)
+        self.assertEqual(before, sorted(str(path.relative_to(self.host.root)) for path in self.host.root.rglob('*')))
+        self.assertFalse(any(command[0] in ('apt-get', 'systemctl') for command in self.host.commands))
+
+    def test_existing_docker_data_is_preserved(self):
+        data = self.host.path('/var/lib/docker/volumes/user-data')
+        data.parent.mkdir(parents=True)
+        data.write_text('valuable')
+        with self.assertRaisesRegex(docker.DockerError, 'Existing /var/lib/docker'):
+            docker.docker_preflight(self.host.installed)
+        self.assertEqual(data.read_text(), 'valuable')
+
+    def test_conflicting_distribution_docker_package_is_never_removed(self):
+        self.host.installed['docker.io'] = '26.1.5'
+        with self.assertRaisesRegex(docker.DockerError, 'Conflicting Docker packages'):
+            docker.docker_preflight(self.host.installed)
+        self.assertEqual(self.host.installed['docker.io'], '26.1.5')
+        self.assertFalse(any(command[0] == 'apt-get' for command in self.host.commands))
+
+    def test_unmanaged_destination_data_is_not_adopted(self):
+        data = self.host.path('/srv/docker/private-data')
+        data.parent.mkdir(parents=True)
+        data.write_text('preserve me')
+        with self.assertRaisesRegex(docker.DockerError, 'Unmanaged /srv/docker'):
+            docker.docker_preflight(self.host.installed)
+        self.assertEqual(data.read_text(), 'preserve me')
+
+    def test_service_policy_symlink_is_restored_after_failure(self):
+        policy = self.host.path('/usr/sbin/policy-rc.d')
+        original = self.host.path('/usr/sbin/provider-policy')
+        original.write_text('provider policy')
+        policy.symlink_to(original)
+        with self.assertRaisesRegex(RuntimeError, 'boom'):
+            with docker.package_policy_lock(), docker.service_start_guard():
+                self.assertFalse(policy.is_symlink())
+                raise RuntimeError('boom')
+        self.assertTrue(policy.is_symlink())
+        self.assertEqual(original.read_text(), 'provider policy')
+
+    def test_unfinished_backup_is_discarded_without_replacing_live_policy(self):
+        docker.POLICY_STATE.mkdir()
+        (docker.POLICY_STATE / 'original').write_text('partial')
+        docker.POLICY.write_text('complete provider policy')
+        with docker.package_policy_lock(), docker.service_start_guard():
+            pass
+        self.assertEqual(docker.POLICY.read_text(), 'complete provider policy')
+
+    def test_interrupted_guard_is_recovered_before_next_attempt(self):
+        state = self.host.path('/usr/sbin/.apex-policy-rc.d')
+        state.mkdir()
+        (state / 'original').write_text('original policy')
+        (state / 'ready').touch()
+        self.host.path('/usr/sbin/policy-rc.d').write_text('#!/bin/sh\nexit 101\n')
+        with docker.package_policy_lock(), docker.service_start_guard():
+            pass
+        self.assertEqual(self.host.path('/usr/sbin/policy-rc.d').read_text(), 'original policy')
+
+    def test_external_policy_change_is_not_overwritten(self):
+        policy = self.host.path('/usr/sbin/policy-rc.d')
+        with self.assertRaisesRegex(docker.DockerError, 'changed outside APEX'):
+            with docker.package_policy_lock(), docker.service_start_guard():
+                policy.write_text('external policy')
+        self.assertEqual(policy.read_text(), 'external policy')

tests/provisioning/test_provision_access.py

@@ -7,9 +7,7 @@ import tempfile
 import unittest
 from unittest.mock import patch
 
-from engine.provisioning.access import AccessError, Host, Transition, atomic, main
-from provision_fakes import SecurityBoundary
-
+from provisioning.access import AccessError, Host, Transition, atomic
 
 TOKEN = 'a' * 32
 
@@ -18,15 +16,12 @@ class FakeHost(Host):
     """Replace system service effects; keep snapshots and transition files real."""
     def __init__(self, root):
         super().__init__(root)
-        self.security = SecurityBoundary()
         self.boot = 'first-boot'
         self.armed = False
         self.restored = 0
         self.ports = [22]
-        self.firewall = False
         self.bad_sudo = False
         self.fail_final = False
-        self.existing_final = False
         atomic(self.path('/etc/ssh/sshd_config'), b'Port 22\n')
 
     def boot_id(self):
@@ -42,19 +37,10 @@ class FakeHost(Host):
     def disarm(self):
         self.armed = False
 
-    def firewall_needed(self):
-        return True
-
-    def open_firewall(self):
-        self.firewall = True
-
     def policy(self, ports, final, address):
         if final and self.fail_final:
             raise AccessError('policy failed')
 
-    def current_final(self, address):
-        return self.existing_final
-
     def verify_active(self, ports, final, address):
         self.policy(ports, final, address)
         if self.ports != sorted(set(ports)):
@@ -69,7 +55,6 @@ class FakeHost(Host):
     def restore(self, state):
         self.restored += 1
         self.ports = [state['original_port']]
-        self.firewall = False
         path = self.path('/etc/systemd/system/ssh.service.d/apex-init.conf')
         if state['previous'] is None:
             path.unlink(missing_ok=True)
@@ -84,7 +69,7 @@ class AccessTests(unittest.TestCase):
         self.addCleanup(self.temp.cleanup)
         self.host = FakeHost(Path(self.temp.name))
         self.clock = 1000
-        self.transition = Transition(self.host, now=lambda: self.clock, security=self.host.security)
+        self.transition = self.manual('debian')
 
     def begin(self):
         return self.transition.execute('begin', TOKEN, 22, '192.0.2.10')
@@ -110,19 +95,10 @@ class AccessTests(unittest.TestCase):
         self.clock += 301
         self.transition.execute('tick')
         self.assertEqual(self.host.ports, [22])
-        self.assertFalse(self.host.firewall)
         self.assertEqual(self.transition.load()['status'], 'rolled_back')
 
-    def test_reboot_recovers_persisted_transition_with_fresh_runner(self):
-        self.begin()
-        self.host.boot = 'second-boot'
-        fresh = Transition(self.host, now=lambda: 0, security=self.host.security)
-        fresh.execute('tick')
-        self.assertEqual(self.host.ports, [22])
-        self.assertEqual(self.host.restored, 1)
-
     def test_watchdog_restores_access_while_enrollment_lock_is_held(self):
-        from engine.provisioning.state import exclusive_lock
+        from provisioning.state import exclusive_lock
         self.begin()
         self.clock += 301
         with exclusive_lock(self.host.path('/var/lib/apex-init/init.lock')):
@@ -163,34 +139,71 @@ class AccessTests(unittest.TestCase):
             self.transition.execute('commit', 'b' * 32)
         self.assertTrue(self.host.armed)
 
-    def test_committed_retry_does_not_reopen_root_listener(self):
-        self.begin()
-        self.transition.execute('finalize', TOKEN)
-        self.transition.execute('commit', TOKEN)
-        result = self.transition.execute('begin', 'b' * 32, 2222, '192.0.2.10')
-        self.assertEqual(result['status'], 'committed')
+    def manual(self, profile):
+        self.host.live_arguments = lambda: (['/usr/sbin/sshd', '-D'], '755')
+        self.host.listener_ports = lambda pid: set(self.host.ports)
+        return Transition(self.host, now=lambda: self.clock, profile=profile)
+
+    def test_manual_debian_keeps_all_old_ports_then_limits_to_operator_port(self):
+        self.host.ports = [22, 2200]
+        manual = self.manual('debian')
+        manual.execute('begin', TOKEN, 2200, '192.0.2.1')
+        self.assertEqual(self.host.ports, [22, 2200, 2222])
+        boundary = manual.load()['authenticated_after_us']
+        manual.execute('finalize', TOKEN)
         self.assertEqual(self.host.ports, [2222])
+        self.assertGreaterEqual(manual.load()['authenticated_after_us'], boundary)
+        manual.execute('commit', TOKEN)
+        self.assertEqual(manual.load()['status'], 'committed')
+
+    def test_manual_proxmox_installed_policy_never_restarts_and_preserves_image_state(self):
+        self.host.ports = [22, 2222]
+        legacy = self.host.path('/var/lib/apex-init/access/state.json')
+        manual = self.manual('proxmox')
+        with patch.object(self.host, 'install_policy', side_effect=AssertionError('restarted sshd')):
+            for round_number, status in enumerate(('committed', 'existing')):
+                record = {'status': status}
+                atomic(legacy, json.dumps(record).encode())
+                token = str(round_number) * 32
+                manual.execute('begin', token, 22, '192.0.2.1')
+                manual.execute('finalize', token)
+                manual.execute('commit', token)
+                self.assertEqual(json.loads(legacy.read_text()), record)
+        self.assertEqual(self.host.ports, [22, 2222])
+
+    def test_inflight_legacy_handover_retains_its_timer_and_state(self):
+        legacy = self.host.path('/var/lib/apex-init/access/state.json')
+        for record in ({'status': 'final'}, {'status': 'committed', 'disarm_pending': True}):
+            atomic(legacy, json.dumps(record).encode())
+            for operation in ('begin', 'tick'):
+                with self.assertRaisesRegex(AccessError, 'workflow is pending'):
+                    self.transition.execute(operation, TOKEN, 22, '192.0.2.1')
+            self.assertEqual(json.loads(legacy.read_text()), record)
+            self.assertIsNone(self.transition.load())
+
+    def test_plain_watchdog_restores_manual_override_on_reboot(self):
+        override = self.host.path('/etc/systemd/system/ssh.service.d/apex-init.conf')
+        atomic(override, b'original managed invocation')
+        manual = self.manual('proxmox')
+        manual.execute('begin', TOKEN, 22, '192.0.2.1')
+        manual.execute('finalize', TOKEN)
+        self.assertEqual(self.host.ports, [22, 2222])
+        self.host.boot = 'second-boot'
+        Transition(self.host, now=lambda: 0).execute('tick')
+        self.assertEqual(override.read_bytes(), b'original managed invocation')
+        self.assertEqual(manual.load()['status'], 'rolled_back')
         self.assertFalse(self.host.armed)
 
-    def test_committed_retry_rejects_live_listener_drift(self):
+    def test_failed_disarm_is_retried_after_durable_commit_without_rollback(self):
         self.begin()
         self.transition.execute('finalize', TOKEN)
-        self.transition.execute('commit', TOKEN)
-        self.host.ports = [22, 2222]
-        with self.assertRaises(AccessError):
-            self.transition.execute('begin', 'b' * 32, 2222, '192.0.2.10')
-
-    def test_preexisting_final_policy_is_preserved_during_security_activation(self):
-        self.host.existing_final = True
-        self.host.ports = [2222]
-        result = self.transition.execute('begin', TOKEN, 2222, '192.0.2.10')
-        self.assertEqual(result['status'], 'transition')
-        self.assertTrue(self.host.armed)
-        self.assertFalse(self.host.firewall)
-        self.transition.execute('finalize', TOKEN)
-        self.transition.execute('commit', TOKEN)
+        with patch.object(self.host, 'disarm', side_effect=AccessError('unavailable')), self.assertRaises(AccessError):
+            self.transition.execute('commit', TOKEN)
+        self.clock += 400
+        self.transition.execute('tick')
         self.assertEqual(self.transition.load()['status'], 'committed')
-        self.assertFalse(self.host.path('/etc/systemd/system/ssh.service.d/apex-init.conf').exists())
+        self.assertFalse(self.host.armed)
+        self.assertEqual(self.host.restored, 0)
 
     def test_reject_symlink_state_ancestor(self):
         outside = self.host.path('/outside')
@@ -288,28 +301,12 @@ class LivePolicyTests(unittest.TestCase):
                     self.host.verify_active([2222], True, '192.0.2.1')
                 self.effective = original
 
-    def test_bootstrap_prerequisites_do_not_require_adam_or_sudo_or_write_state(self):
-        self.arguments = self.running_arguments = ['/usr/sbin/sshd', '-D']
-        self.cmdline()
-        def unseeded(args, **kwargs):
-            if args[0] in ('visudo', 'runuser', 'sudo'):
-                raise AssertionError('administrator tools do not exist before baseline')
-            return self.runner(args, **kwargs)
-        self.host.runner = unseeded
-        before = sorted(self.root.rglob('*'))
-        with patch('engine.provisioning.access.Host', return_value=self.host), \
-                patch('engine.provisioning.access.os.geteuid', return_value=0):
-            self.assertEqual(main(['preflight']), 0)
-        self.assertEqual(sorted(self.root.rglob('*')), before)
-        self.assertIn(['/usr/sbin/sshd', '-t'], self.calls)
-        self.assertTrue(any('MainPID' in args for args in self.calls))
-
     def test_bootstrap_prerequisites_reject_socket_and_live_invocation_drift(self):
         self.arguments = self.running_arguments = ['/usr/sbin/sshd', '-D']
         self.cmdline()
         for failure in ('socket', 'invocation'):
             with self.subTest(failure=failure):
-                def incompatible(args, **kwargs):
+                def incompatible(args, failure=failure, **kwargs):
                     if failure == 'socket' and args == ['systemctl', 'is-enabled', '--quiet', 'ssh.socket']:
                         return subprocess.CompletedProcess(args, 0, '', '')
                     return self.runner(args, **kwargs)
@@ -445,6 +442,35 @@ class OpenSSHPolicyTests(unittest.TestCase):
                 self.assertIn('x11forwarding no', effective)
                 self.assertIn('permitemptypasswords no', effective)
 
+    def test_proxmox_effective_root_key_auth_and_both_ports_with_legacy_operator_block(self):
+        self.host.profile = 'proxmox'
+        with self.config.open('a') as stream:
+            stream.write('# APEX operator SSH policy\nMatch User adam\n'
+                         '  AuthenticationMethods publickey\n  PubkeyAuthentication yes\n'
+                         '  PasswordAuthentication no\n  KbdInteractiveAuthentication no\n'
+                         '  PermitEmptyPasswords no\n  X11Forwarding no\n')
+        final = self.host.policy([22, 2222], True, '192.0.2.1')
+        for user in ('root', 'adam'):
+            effective = self.effective(final, user)
+            self.assertEqual({item for item in effective if item.startswith('port ')}, {'port 22', 'port 2222'})
+            self.assertTrue({'permitrootlogin without-password', 'permitrootlogin prohibit-password'}.intersection(effective))
+            self.assertIn('authenticationmethods publickey', effective)
+            self.assertFalse(any(item.startswith('allowusers ') for item in effective))
+        with self.config.open('a') as stream:
+            stream.write('  PasswordAuthentication yes\n')
+        with self.assertRaisesRegex(AccessError, 'Match'):
+            self.host.policy([22, 2222], True, '192.0.2.1')
+
+    def test_adding_probe_port_preserves_authentication_from_the_existing_service(self):
+        self.host.profile = 'debian'
+        args = ['/usr/sbin/sshd', '-p', '2200', '-o', 'PermitRootLogin=no', '-o', 'PasswordAuthentication=no', '-D']
+        with patch.object(self.host, 'service_arguments', return_value=args):
+            transition = self.host.policy([2200, 2222], False, '192.0.2.1')
+        effective = self.effective(transition, 'root')
+        self.assertIn('permitrootlogin no', effective)
+        self.assertIn('passwordauthentication no', effective)
+        self.assertIn('port 2200', effective)
+
     def test_additive_inherited_allowusers_stops_finalization(self):
         with self.config.open('a') as stream:
             stream.write('AllowUsers provider\n')

tests/run.sh

@@ -1,5 +1,7 @@
 #!/usr/bin/env bash
-# Run the full engine test suite (stdlib unittest — no pytest).
+# Run CLI, bootstrap, and host provisioning checks with the project modules on the import path.
 set -euo pipefail
 cd "$(dirname "$0")/.."
-exec python3 -m unittest discover -s tests -p 'test_*.py' -v
+export PYTHONPATH="$PWD/scripts${PYTHONPATH:+:$PYTHONPATH}"
+python3 -m unittest discover -s tests -p 'test_*.py' -v
+python3 -m unittest discover -s tests/provisioning -p 'test_*.py' -v

tests/test_backup_run.py

@@ -0,0 +1,49 @@
+import json
+from pathlib import Path
+from types import SimpleNamespace
+import unittest
+import tempfile
+from unittest.mock import Mock
+
+from engine import overlay
+
+backup = overlay.load_module(str(Path(__file__).resolve().parents[1] / 'actions/tier/backup.py'),
+                             'test_backup_action')
+
+
+class BackupRecovery(unittest.TestCase):
+    def test_failed_attempt_preserves_last_success_and_records_new_failure(self):
+        with tempfile.TemporaryDirectory() as directory:
+            path = Path(directory) / 'status.json'
+            backup.write_backup_status(path, 'success', 100)
+            backup.write_backup_status(path, 'running', 200)
+            backup.write_backup_status(path, 'failed', 210)
+            status = json.loads(path.read_text())
+            self.assertEqual(status['last_success'], 100)
+            self.assertEqual(status['state'], 'failed')
+            self.assertEqual(status['updated_at'], 210)
+
+    def test_recovers_namespace_owner_before_dependent_without_starting_other_containers(self):
+        running = {'komf': False, 'komga': False, 'intentionally-stopped': False}
+        def docker(command, **kwargs):
+            if command[1] == 'inspect':
+                return SimpleNamespace(returncode=0, stdout=json.dumps([
+                    {'Id': name, 'State': {'Running': running[name]}} for name in command[2:]]))
+            name = command[2]
+            if name == 'komf' and not running['komga']:
+                return SimpleNamespace(returncode=1)
+            running[name] = True
+            return SimpleNamespace(returncode=0)
+        ctx = SimpleNamespace(sys=SimpleNamespace(run=docker), log=Mock())
+        backup.restore_containers(ctx, ['komf', 'komga'])
+        self.assertTrue(running['komf'])
+        self.assertTrue(running['komga'])
+        self.assertFalse(running['intentionally-stopped'])
+
+    def test_unrecoverable_container_prevents_success(self):
+        def docker(command, **kwargs):
+            return SimpleNamespace(returncode=0 if command[1] == 'inspect' else 1,
+                                   stdout='[{"Id":"broken","State":{"Running":false}}]')
+        ctx = SimpleNamespace(sys=SimpleNamespace(run=docker), log=Mock())
+        with self.assertRaisesRegex(RuntimeError, 'broken'):
+            backup.restore_containers(ctx, ['broken'])