Bootstrap private node checkouts and operator access from the workstation

Makefile

@@ -5,9 +5,10 @@ test:
 
 lint:
 	@if command -v ruff >/dev/null 2>&1; then \
-		ruff check .; \
+		ruff check . scripts/bootstrap-node; \
 	else \
-		python3 -m compileall -q engine actions tests ci/scripts && \
+		python3 -m compileall -q engine actions tests ci/scripts scripts && \
+		python3 -m py_compile scripts/bootstrap-node && \
 		echo "ruff not installed in PATH; verified python bytecode compilation."; \
 	fi
 

README.md

@@ -13,19 +13,36 @@ Local infrastructure operations for independently owned nodes, with a Python sta
 - **Report an Issue**: <https://git.ermnvldmr.com/deytenit/apex.ermnvldmr.com/issues>
 - **License**: [BSD-2-Clause](#copyright--license)
 
+From an APEX checkout on your workstation, connect to the installed Debian or Proxmox VE host as root or as a user with passwordless sudo. The workstation needs Python 3.10 or newer and OpenSSH. The script installs Python 3 through APT if the host lacks it. Verify its SSH host key through a trusted channel and record it in your workstation’s SSH known hosts before connecting.
+
 ```sh
-# Clone your node's repository (which pins apex as commons/)
-cd /srv
-git clone --recurse-submodules ssh://git@git.example.com:22222/infrastructure/selfhost.example.com.git
-cd selfhost.example.com && ./init.sh && source ~/.bashrc
+scripts/bootstrap-node \
+  --target root@node.example.com \
+  --repo ssh://git@git.example.com:22222/infrastructure/selfhost.example.com.git \
+  --operator-key ~/.ssh/operator.pub \
+  --deploy-key ~/.ssh/selfhost-deploy \
+  --known-hosts ~/.ssh/git-known-hosts
+```
 
-# Configure node identity
-cat > node.env <<'EOF'
-APEX_NODE_FQDN=selfhost.example.com
-APEX_NODE_HOST=selfhost.example.com
-APEX_SUBNET=198.18.0.0/24
-EOF
+The operator public key grants SSH access to `adam`, who receives passwordless sudo. Use a separate, passphrase-free SSH deploy key for this node. Register its public half as a deploy key on the Git server before running the script. The script installs the private half on the node. The key must have read access to the node repository and any private SSH submodules it needs on that Git server. Grant write access to the node repository only if you use `apex repository/sync` to push changes. Supply verified Git server host keys in OpenSSH known_hosts format, including the port for nonstandard listeners (for example, `[git.example.com]:22222`).
 
+The script clones the repository under `/srv` as `adam`. Use `--node-name NAME` to choose the checkout directory under `/srv`. Before changing the SSH policy, log in with the matching operator private key on the original SSH port. Use the same host or SSH alias as the bootstrap target.
+
+```sh
+ssh -o StrictHostKeyChecking=yes -o IdentitiesOnly=yes -i ~/.ssh/operator adam@node.example.com
+```
+
+In that session, enter the checkout. If the node repository includes `init.sh`, review and run it to install the shell shortcuts. Otherwise call `./commons/apex` directly.
+
+```sh
+cd /srv/selfhost.example.com
+if [ -f ./init.sh ]; then ./init.sh && source ~/.bashrc; fi
+./commons/apex --help
+```
+
+Configure the node in its checkout. For example, on a stock Debian Docker host:
+
+```sh
 # Stock Debian Docker host, after configuring the node repository
 apex config/packages
 apex config/base
@@ -114,15 +131,6 @@ Run each setup action explicitly from the node checkout. Use `apex --help` to li
 
 On an installed Proxmox VE host, `config/packages` installs the administrator tools and `config/base` applies APT and journald policy. The PVE host keeps its own storage, networking, firewall, and management listeners. Run workload configuration inside the guest that owns the service.
 
-To set up the operator on either platform, start with an account that can administer the host:
-
-```sh
-apex config/adam --public-key-file /path/to/operator.pub
-apex repository/chown
-```
-
-`config/adam` grants passwordless sudo. `repository/chown` gives adam ownership of the tracked checkout directly under `/srv` and its Git metadata.
-
 Before moving SSH to port 2222, allow inbound TCP/2222 from the operator’s address through the provider firewall and the node’s firewall (UFW on stock Debian or the PVE firewall). Keep the original administrator session open throughout the transition.
 
 1. Run `./commons/apex config/hardening begin` from the node checkout in the original SSH session.

actions/config/adam.py

@@ -1,54 +0,0 @@
-"""Provision the golden-image operator while preserving authorized keys."""
-import pwd
-import grp
-from pathlib import Path
-from engine.descriptor import Meta, Opt
-
-METADATA = Meta(summary="Configure the adam operator, sudo access and optional SSH public key.",
-                args=[Opt("--public-key-file", "Append one OpenSSH public key from this file.")])
-
-
-HOME = Path("/home/adam")
-
-
-def run(ctx, args):
-    system = ctx.sys
-    key = None
-    if args.public_key_file:
-        key = Path(args.public_key_file).read_text().strip()
-        if len(key.splitlines()) != 1 or not key.startswith(("ssh-", "ecdsa-", "sk-")):
-            raise ValueError("Supply one OpenSSH public key.")
-        system.run(["ssh-keygen", "-lf", "-"], input=key + "\n", capture=True)
-    groups = ["sudo"]
-    try:
-        grp.getgrnam("docker")
-    except KeyError:
-        pass
-    else:
-        groups.append("docker")
-    try:
-        account = pwd.getpwnam("adam")
-    except KeyError:
-        if HOME.exists() or HOME.is_symlink():
-            raise RuntimeError("Resolve existing /home/adam before creating the account.") from None
-        system.sudo(["useradd", "--create-home", "--shell", "/bin/bash", "--user-group", "adam"])
-        account = pwd.getpwnam("adam")
-    if account.pw_uid < 1000 or Path(account.pw_dir) != HOME or HOME.resolve() != HOME:
-        raise RuntimeError("Existing adam account has incompatible UID or home.")
-    if Path(account.pw_dir).stat().st_uid != account.pw_uid:
-        raise RuntimeError("The operator must own /home/adam.")
-    system.sudo(["usermod", "--shell", "/bin/bash", "--append", "--groups", ",".join(groups), "adam"])
-    if key:
-        system.sudo(["runuser", "-u", "adam", "--", "sh", "-ec",
-                     'umask 077; cd "$1"; mkdir -p .ssh; chmod 700 .ssh; '
-                     'touch .ssh/authorized_keys; chmod 600 .ssh/authorized_keys; '
-                     'IFS= read -r key; grep -Fxq -- "$key" .ssh/authorized_keys || '
-                     'printf "\\n%s\\n" "$key" >> .ssh/authorized_keys', "sh", str(HOME)], input=key + "\n")
-    content = "adam ALL=(ALL:ALL) NOPASSWD: ALL\n"
-    system.sudo(["visudo", "-cf", "-"], input=content)
-    path = Path("/etc/sudoers.d/99-apex-adam")
-    if path.is_symlink():
-        raise RuntimeError(f"Refusing redirected sudo policy: {path}")
-    ctx.host.write_file(str(path), content, mode="0440", backup=False, sudo=True)
-    system.sudo(["visudo", "-c"])
-    system.sudo(["runuser", "-u", "adam", "--", "sudo", "-n", "true"])

actions/config/hardening.py

@@ -84,7 +84,7 @@ def _connection(ctx, *, after=None):
 def _operator_access(ctx):
     account = pwd.getpwnam("adam")
     if account.pw_uid < 1000 or account.pw_dir != "/home/adam":
-        raise RuntimeError("Configure a regular adam operator account first.")
+        raise RuntimeError("Prepare the adam account from your workstation with scripts/bootstrap-node first.")
     if pwd.getpwuid(os.getuid()).pw_name == "adam":
         return _connection(ctx)
     loginuid = int(Path("/proc/self/loginuid").read_text())

scripts/bootstrap-node

@@ -0,0 +1,95 @@
+#!/usr/bin/env python3
+"""Bootstrap a fleet node through a trusted OpenSSH connection."""
+
+import argparse
+import json
+import re
+import shlex
+import subprocess
+import sys
+from pathlib import Path
+from urllib.parse import urlsplit
+
+
+def repository(url):
+    if any(ch.isspace() or ord(ch) < 32 for ch in url):
+        raise ValueError("invalid repository URL")
+    if url.startswith("ssh://"):
+        parts = urlsplit(url)
+        if parts.password is not None or parts.query or parts.fragment:
+            raise ValueError("expected an SSH repository URL without password, query or fragment")
+        host, port, path = parts.hostname, parts.port, parts.path
+    else:
+        match = re.fullmatch(r"(?:[\w.-]+@)?(\[[\da-fA-F:]+\]|[\w.-]+):(.+)", url)
+        if not match:
+            raise ValueError("expected git@host:path or ssh://user@host[:port]/path")
+        host, path = match.groups()
+        host, port = host.strip("[]"), None
+    if not host or not re.fullmatch(r"[a-zA-Z0-9:][a-zA-Z0-9.:-]*", host) or not path:
+        raise ValueError("invalid Git host or repository path")
+    return f"[{host}]:{port}" if port not in (None, 22) else host, Path(path).name.removesuffix(".git")
+
+
+def keygen(args, data=None):
+    return subprocess.run(["ssh-keygen", *args], input=data or b"", capture_output=True)
+
+
+def bootstrap(args):
+    endpoint, basename = repository(args.repo)
+    name = args.node_name if args.node_name is not None else basename
+    if not re.fullmatch(r"[a-zA-Z0-9][a-zA-Z0-9._-]*", name):
+        raise ValueError("invalid node name")
+    operator = args.operator_key.read_text().strip()
+    fields = operator.split()
+    if (len(operator.splitlines()) != 1 or len(fields) < 2
+            or keygen(["-lf", "-"], (" ".join(fields[:2]) + "\n").encode()).returncode):
+        raise ValueError("operator key must contain one plain SSH public key")
+    derived = keygen(["-y", "-P", "", "-f", str(args.deploy_key.resolve())])
+    if derived.returncode or derived.stdout.split()[0].startswith(b"sk-"):
+        raise ValueError("deploy key must be a standalone private key with no passphrase and owner-only permissions")
+    if derived.stdout.split()[:2] == [v.encode() for v in fields[:2]]:
+        raise ValueError("operator and deploy keys must be different")
+    matches = keygen(["-F", endpoint, "-f", str(args.known_hosts.resolve())])
+    pins = [line for line in matches.stdout.decode().splitlines() if line and not line.startswith("#")]
+    if matches.returncode or not pins or not any(
+        not pin.startswith("@revoked ") and keygen(["-lf", "-"], (pin + "\n").encode()).returncode == 0
+        for pin in pins
+    ):
+        raise ValueError(f"no usable known_hosts pin for {endpoint}")
+    payload = json.dumps(dict(operator_key=operator, deploy_key=args.deploy_key.read_text(),
+                              known_hosts="\n".join(pins) + "\n", repo_url=args.repo, node_name=name))
+    helper = Path(__file__).resolve().with_name("bootstrap-node-remote.py").read_text()
+    # Package management must never consume the credential payload on standard input.
+    remote = """set -e
+if [ "$(id -u)" = 0 ]; then set --; else set -- sudo -n --; fi
+if ! command -v python3 >/dev/null; then
+    test -f /etc/debian_version
+    "$@" apt-get -o DPkg::Lock::Timeout=60 update </dev/null
+    "$@" env DEBIAN_FRONTEND=noninteractive apt-get -o DPkg::Lock::Timeout=60 install -y python3 </dev/null
+fi
+exec "$@" python3 -I -c """ + shlex.quote(helper)
+    result = subprocess.run(["ssh", "-T", "-a", "-S", "none", "-o", "StrictHostKeyChecking=yes",
+                           "-o", "ClearAllForwardings=yes", "--", args.target, remote],
+                          input=payload.encode())
+    if result.returncode == 0:
+        print(f"Checkout ready for adam at /srv/{name}.")
+    return result.returncode
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--target", required=True, help="trusted SSH target with root or noninteractive sudo")
+    parser.add_argument("--repo", required=True, help="SSH URL of the node repository")
+    parser.add_argument("--operator-key", required=True, type=Path, help="operator public key")
+    parser.add_argument("--deploy-key", required=True, type=Path, help="repository deploy private key")
+    parser.add_argument("--known-hosts", required=True, type=Path, help="trusted Git server host keys")
+    parser.add_argument("--node-name", help="defaults to repository basename without .git")
+    args = parser.parse_args()
+    try:
+        return bootstrap(args)
+    except (OSError, ValueError) as exc:
+        parser.exit(1, f"bootstrap-node: {exc}\n")
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/bootstrap-node-remote.py

@@ -0,0 +1,274 @@
+"""Root-side Debian/PVE bootstrap, staged and invoked by bootstrap-node."""
+import fcntl
+import json
+import os
+from pathlib import Path
+import pwd
+import re
+import stat
+import subprocess
+import sys
+import tempfile
+from urllib.parse import urlsplit
+
+
+HOME = Path("/home/adam")
+
+
+def checked(command, *, data=None, capture=False, **kwargs):
+    # A Git hook or remote can emit credentials. Report the operation, never its output.
+    result = subprocess.run(command, input=data, text=True, stderr=subprocess.DEVNULL,
+                            stdout=subprocess.PIPE if capture else subprocess.DEVNULL, **kwargs)
+    if result.returncode:
+        raise RuntimeError(f"{Path(command[0]).name} failed (exit {result.returncode})")
+    return result.stdout
+
+
+def repository_endpoint(url):
+    if not isinstance(url, str) or any(c.isspace() or ord(c) < 32 for c in url):
+        raise ValueError("Invalid SSH repository URL")
+    if url.startswith("ssh://"):
+        parsed = urlsplit(url)
+        if (parsed.password is not None or not parsed.path.strip("/") or parsed.fragment
+                or parsed.query or parsed.port == 0):
+            raise ValueError("Invalid SSH repository URL")
+        host, port = parsed.hostname, parsed.port
+    else:
+        match = re.fullmatch(r"(?:[\w.-]+@)?(\[[a-fA-F0-9:]+\]|[\w.-]+):(.+)", url, re.ASCII)
+        host, port = (match[1].strip("[]"), None) if match else (None, None)
+    if not host or not re.fullmatch(r"[a-zA-Z0-9:][a-zA-Z0-9.:-]*", host):
+        raise ValueError("An SSH repository host is required")
+    return host, f"[{host}]:{port}" if port not in (None, 22) else host
+
+
+def existing(path):
+    try:
+        info = path.lstat()
+    except FileNotFoundError:
+        return b""
+    if not stat.S_ISREG(info.st_mode) or info.st_nlink != 1 or info.st_uid != os.geteuid():
+        raise ValueError(f"Unsafe file: {path.name}")
+    return path.read_bytes()
+
+
+def install(path, content, mode):
+    existing(path)
+    fd, tmp = tempfile.mkstemp(dir=path.parent)
+    try:
+        with os.fdopen(fd, "wb") as stream:
+            stream.write(content)
+            os.fchmod(stream.fileno(), mode)
+        os.replace(tmp, path)
+    finally:
+        Path(tmp).unlink(missing_ok=True)
+
+
+def validate_credentials(spec, endpoint):
+    key = spec["operator_key"].strip()
+    if len(key.splitlines()) != 1 or not key.startswith(("ssh-", "ecdsa-", "sk-")):
+        raise ValueError("Expected one operator public key")
+    checked(["ssh-keygen", "-lf", "-"], data=key + "\n")
+    with tempfile.TemporaryDirectory() as temp:
+        private = Path(temp) / "deploy"
+        install(private, (spec["deploy_key"].rstrip() + "\n").encode(), 0o600)
+        public = checked(["ssh-keygen", "-y", "-P", "", "-f", str(private)], capture=True)
+        if public.startswith("sk-") or public.split()[:2] == key.split()[:2]:
+            raise ValueError("Use a separate unattended deploy key")
+        pins = Path(temp) / "known_hosts"
+        pins.write_text(spec["known_hosts"])
+        matches = checked(["ssh-keygen", "-F", endpoint, "-f", str(pins)], capture=True)
+        trusted = [line for line in matches.splitlines() if not line.startswith(("#", "@revoked "))]
+        if not trusted:
+            raise ValueError("No trusted host key for the repository")
+        for pin in trusted:
+            checked(["ssh-keygen", "-lf", "-"], data=pin + "\n")
+
+
+def configure_ssh(home, spec, host):
+    directory = home / ".ssh"
+    directory.mkdir(mode=0o700, exist_ok=True)
+    info = directory.lstat()
+    if not stat.S_ISDIR(info.st_mode) or info.st_uid != os.geteuid():
+        raise ValueError("Unsafe SSH directory")
+    directory.chmod(0o700)
+    key = spec["operator_key"].strip()
+    authorized = directory / "authorized_keys"
+    content = existing(authorized)
+    identity = key.split()[:2]
+    present = False
+    for line in content.decode(errors="replace").splitlines():
+        fields = line.split()
+        if fields[:2] == identity:
+            present = True
+        elif fields and not line.lstrip().startswith("#") and identity[1] in fields:
+            raise ValueError("Existing operator key has SSH authorization restrictions; inspect it")
+    if not present:
+        content = content + (b"\n" if content and not content.endswith(b"\n") else b"")
+        content += key.encode() + b"\n"
+    # Git also evaluates includeIf for the separate gitdirs of recursive submodules.
+    config = home / ".gitconfig"
+    old = existing(config)
+    gitdir = f"/srv/{spec['node_name']}/.git"
+    prefix = directory / ("apex-" + spec["node_name"])
+    include = (f'[includeIf "gitdir:{gitdir}"]\n\tpath = {prefix}.gitconf\n'
+               f'[includeIf "gitdir:{gitdir}/"]\n\tpath = {prefix}.gitconf\n').encode()
+    for suffix in ("key", "hosts", "sshconf", "gitconf"):
+        existing(Path(f"{prefix}.{suffix}"))
+    write_repository_credentials(prefix, spec, host)
+    install(Path(f"{prefix}.gitconf"),
+            f"[core]\n\tsshCommand = ssh -F {prefix}.sshconf\n".encode(), 0o600)
+    if include not in old:
+        install(config, old + b"\n" + include, 0o600)
+    install(authorized, content, 0o600)
+
+
+def write_repository_credentials(prefix, spec, host):
+    policy = (f"Host {host}\n    IdentitiesOnly yes\n"
+              f"    IdentityFile {prefix}.key\n    UserKnownHostsFile {prefix}.hosts\n"
+              "    GlobalKnownHostsFile /dev/null\nHost *\n    StrictHostKeyChecking yes\n"
+              "    BatchMode yes\n")
+    for suffix, content in (("key", spec["deploy_key"]), ("hosts", spec["known_hosts"]),
+                            ("sshconf", policy)):
+        install(Path(f"{prefix}.{suffix}"), (content.rstrip() + "\n").encode(), 0o600)
+
+
+def verify_and_clone(spec, host, account):
+    srv = Path("/srv")
+    if not srv.exists():
+        srv.mkdir()
+        srv.chmod(0o755)
+    if srv.resolve() != srv or srv.stat().st_uid != 0 or srv.stat().st_mode & 0o022:
+        raise RuntimeError("/srv must be a root-controlled directory")
+    with tempfile.TemporaryDirectory(prefix=".apex-credentials-", dir=srv) as temp:
+        stage = Path(temp)
+        prefix = stage / "git"
+        write_repository_credentials(prefix, spec, host)
+        for path in stage.iterdir():
+            os.chown(path, account.pw_uid, account.pw_gid)
+        stage.chmod(0o711)
+
+        def git(*args):
+            return checked(["runuser", "-u", "adam", "--", "env", "GIT_TERMINAL_PROMPT=0",
+                            f"GIT_SSH_COMMAND=ssh -F {prefix}.sshconf", "git", *args],
+                           capture=True, cwd="/")
+
+        try:
+            git("ls-remote", "--exit-code", "--", spec["repo_url"], "HEAD")
+        except RuntimeError:
+            raise RuntimeError("Git access check failed; check the repository URL, deploy-key "
+                               "grants and pinned host key") from None
+        checkout(srv, spec["node_name"], spec["repo_url"], account, git)
+
+
+def prepare_packages():
+    if not Path("/etc/debian_version").is_file():
+        raise RuntimeError("Debian or Proxmox VE is required")
+    missing = []
+    for package in ("git", "sudo", "openssh-client"):
+        result = subprocess.run(["dpkg-query", "-W", "-f=${Status}", package], text=True,
+                                capture_output=True)
+        if result.returncode or result.stdout.split()[1:] != ["ok", "installed"]:
+            missing.append(package)
+    if missing:
+        checked(["apt-get", "update"])
+        checked(["apt-get", "install", "-y", "--no-remove", "--no-install-recommends", *missing])
+
+
+def prepare_account():
+    try:
+        account = pwd.getpwnam("adam")
+    except KeyError:
+        if os.path.lexists(HOME):
+            raise RuntimeError("Existing /home/adam requires inspection") from None
+        checked(["useradd", "--create-home", "--user-group", "--shell", "/bin/bash", "adam"])
+        account = pwd.getpwnam("adam")
+    if (not 1000 <= account.pw_uid < 65534 or account.pw_gid == 0
+            or account.pw_dir != str(HOME) or HOME.resolve() != HOME
+            or HOME.stat().st_uid != account.pw_uid or HOME.stat().st_mode & 0o022
+            or not HOME.is_dir()):
+        raise RuntimeError("Existing adam account or home requires inspection")
+    groups = "sudo"
+    if not Path("/usr/bin/pveversion").exists():
+        checked(["groupadd", "--system", "--force", "docker"])
+        groups += ",docker"
+    checked(["usermod", "--shell", "/bin/bash", "--append", "--groups", groups, "adam"])
+    return account
+
+
+def checkout(srv, node, url, account, git):
+    destination = srv / node
+    if os.path.lexists(destination):
+        info = destination.lstat()
+        if not stat.S_ISDIR(info.st_mode) or info.st_uid != account.pw_uid:
+            raise RuntimeError("Existing checkout path requires inspection")
+
+        def query(*args):
+            return git("-C", str(destination), *args).strip()
+
+        if query("rev-parse", "--show-toplevel") != str(destination) or query(
+                "config", "--get", "remote.origin.url") != url:
+            raise RuntimeError("Existing checkout does not match the requested repository")
+        if any(line.startswith(("-", "U")) for line in query("submodule", "status", "--recursive").splitlines()):
+            raise RuntimeError("Existing checkout has uninitialized or conflicting submodules")
+        return
+    with tempfile.TemporaryDirectory(prefix=".apex-bootstrap-", dir=srv) as staging:
+        stage = Path(staging)
+        stage.chmod(0o711)
+        clone = stage / "checkout"
+        clone.mkdir()
+        os.chown(clone, account.pw_uid, account.pw_gid)
+        try:
+            git("clone", "--recurse-submodules", "--", url, str(clone))
+        except RuntimeError:
+            raise RuntimeError("Recursive clone failed; check access to pinned submodules "
+                               "and available disk space") from None
+        clone.rename(destination)
+
+
+def bootstrap(spec):
+    node = spec["node_name"]
+    if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._-]{0,252}", node):
+        raise ValueError("Invalid node checkout name")
+    host, endpoint = repository_endpoint(spec["repo_url"])
+    if not all(isinstance(spec[k], str) and spec[k].strip()
+               for k in ("known_hosts", "deploy_key", "operator_key")):
+        raise ValueError("Missing SSH credentials")
+    if os.geteuid() != 0:
+        raise RuntimeError("Root login is required")
+    os.umask(0o077)
+    os.environ.clear()
+    os.environ.update(PATH="/usr/sbin:/usr/bin:/sbin:/bin", LC_ALL="C",
+                      DEBIAN_FRONTEND="noninteractive", HOME="/root")
+    # Serialize account, credential and checkout changes from competing bootstraps.
+    with open("/run/apex-bootstrap.lock", "a") as lock:
+        fcntl.flock(lock, fcntl.LOCK_EX)
+        prepare_packages()
+        validate_credentials(spec, endpoint)
+        account = prepare_account()
+        verify_and_clone(spec, host, account)
+        child = os.fork()
+        if child == 0:
+            try:
+                os.initgroups("adam", account.pw_gid)
+                os.setgid(account.pw_gid)
+                os.setuid(account.pw_uid)
+                os.chdir(HOME)
+                configure_ssh(HOME, spec, host)
+            except Exception as error:
+                print(f"SSH setup failed: {error}", file=sys.stderr, flush=True)
+                os._exit(1)
+            os._exit(0)
+        if os.waitpid(child, 0)[1]:
+            raise RuntimeError("Unable to install SSH credentials")
+        policy = "adam ALL=(ALL:ALL) NOPASSWD: ALL\n"
+        checked(["visudo", "-cf", "-"], data=policy)
+        install(Path("/etc/sudoers.d/99-apex-adam"), policy.encode(), 0o440)
+        checked(["visudo", "-c"])
+        checked(["runuser", "-u", "adam", "--", "sudo", "-n", "true"])
+
+
+if __name__ == "__main__":
+    try:
+        bootstrap(json.load(sys.stdin))
+    except (ValueError, KeyError, RuntimeError, OSError) as error:
+        sys.exit(f"Bootstrap failed: {error}")

tests/test_bootstrap_client.py

@@ -0,0 +1,61 @@
+import json
+import os
+import shutil
+import subprocess
+import tempfile
+import unittest
+from pathlib import Path
+from unittest import mock
+
+
+class BootstrapClientTest(unittest.TestCase):
+    def setUp(self):
+        temporary = tempfile.TemporaryDirectory()
+        self.addCleanup(temporary.cleanup)
+        self.root = Path(temporary.name)
+        source = Path(__file__).resolve().parents[1] / "scripts/bootstrap-node"
+        self.launcher = self.root / "bootstrap-node"
+        shutil.copy(source, self.launcher)
+        (self.root / "bootstrap-node-remote.py").write_text("pass\n")
+        for name in ("operator", "deploy"):
+            subprocess.run(["ssh-keygen", "-qt", "ed25519", "-N", "", "-f", str(self.root / name)],
+                           check=True)
+        self.public = (self.root / "operator.pub").read_text()
+        (self.root / "known").write_text("[git.example]:2222 " + self.public)
+        ssh = self.root / "ssh"
+        ssh.write_text('#!/bin/sh\nprintf "%s\\n" "$@" > args\ncat > payload\n')
+        ssh.chmod(0o700)
+
+    def launch(self, *extra):
+        with mock.patch.dict(os.environ, PATH=str(self.root) + os.pathsep + os.environ["PATH"]):
+            return subprocess.run(
+                [str(self.launcher), "--target", "admin@node", "--repo",
+                 "ssh://git@git.example:2222/fleet/example.git", "--operator-key", "operator.pub",
+                 "--deploy-key", "deploy", "--known-hosts", "known", *extra],
+                cwd=self.root, capture_output=True, text=True,
+            )
+
+    def test_custom_git_port_and_secrets_delivered_via_stdin(self):
+        run = self.launch()
+        self.assertEqual(run.returncode, 0, run.stderr)
+        payload = json.loads((self.root / "payload").read_text())
+        self.assertEqual(payload["node_name"], "example")
+        secret = (self.root / "deploy").read_text()
+        self.assertEqual(payload["deploy_key"], secret)
+        self.assertNotIn(secret.strip().splitlines()[1], (self.root / "args").read_text()
+                         + run.stdout + run.stderr)
+
+    def test_git_host_pin_for_default_port_rejected(self):
+        (self.root / "known").write_text("git.example " + self.public)
+        self.assertNotEqual(self.launch().returncode, 0)
+        self.assertFalse((self.root / "payload").exists())
+
+    def test_reused_operator_key_rejected(self):
+        self.assertNotEqual(self.launch("--deploy-key", "operator").returncode, 0)
+        self.assertFalse((self.root / "payload").exists())
+
+    def test_encrypted_deploy_key_rejected(self):
+        subprocess.run(["ssh-keygen", "-pq", "-P", "", "-N", "test", "-f",
+                        str(self.root / "deploy")], check=True, capture_output=True)
+        self.assertNotEqual(self.launch().returncode, 0)
+        self.assertFalse((self.root / "payload").exists())

tests/test_bootstrap_remote.py

@@ -0,0 +1,152 @@
+"""Remote bootstrap checks using disposable repositories, keys, and home directories."""
+import importlib.util
+import os
+from pathlib import Path
+import subprocess
+import tempfile
+import unittest
+from types import SimpleNamespace
+from unittest.mock import Mock, patch
+
+SCRIPT = Path(__file__).resolve().parents[1] / "scripts/bootstrap-node-remote.py"
+loader = importlib.util.spec_from_file_location("bootstrap_remote", SCRIPT)
+remote = importlib.util.module_from_spec(loader)
+loader.loader.exec_module(remote)
+
+
+def git(*args):
+    return subprocess.run(["git", "-c", "protocol.file.allow=always", *map(str, args)],
+                          text=True, capture_output=True, check=True).stdout
+
+
+class OperatorAccount(unittest.TestCase):
+    def test_new_debian_account_is_ready_for_sudo_and_later_docker_install(self):
+        with tempfile.TemporaryDirectory() as temp:
+            home = Path(temp).resolve() / "adam"
+            account = SimpleNamespace(pw_uid=1001, pw_gid=1001, pw_dir=str(home))
+            commands = []
+
+            def provision(command):
+                commands.append(command)
+                if command[0] == "useradd":
+                    home.mkdir(mode=0o700)
+
+            actual_stat = Path.stat
+
+            def stat(path, *args, **kwargs):
+                info = actual_stat(path, *args, **kwargs)
+                if path != home:
+                    return info
+                fields = list(info)
+                fields[4] = account.pw_uid
+                return os.stat_result(fields)
+
+            with (patch.object(remote, "HOME", home),
+                  patch.object(Path, "stat", stat),
+                  patch.object(Path, "exists", return_value=False),
+                  patch.object(remote.pwd, "getpwnam", side_effect=[KeyError, account]),
+                  patch.object(remote, "checked", side_effect=provision)):
+                self.assertEqual(remote.prepare_account(), account)
+            self.assertEqual(sum(command[0] == "useradd" for command in commands), 1)
+            self.assertIn("sudo,docker", commands[-1])
+
+    def test_existing_system_account_cannot_be_promoted(self):
+        command = Mock()
+        account = SimpleNamespace(pw_uid=998, pw_gid=998, pw_dir="/home/adam")
+        with (patch.object(remote.pwd, "getpwnam", return_value=account),
+              patch.object(remote, "checked", command)):
+            with self.assertRaisesRegex(RuntimeError, "account or home"):
+                remote.prepare_account()
+        command.assert_not_called()
+
+
+class Credentials(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.temp.cleanup)
+        self.root = Path(self.temp.name).resolve()
+        self.home = self.root / "adam"
+        self.home.mkdir()
+        for name in ("operator", "deploy"):
+            subprocess.run(["ssh-keygen", "-q", "-N", "", "-t", "ed25519", "-f",
+                            str(self.root / name)], check=True)
+        self.spec = dict(operator_key=(self.root / "operator.pub").read_text(),
+                         deploy_key=(self.root / "deploy").read_text(), node_name="web",
+                         known_hosts="[example.test]:2222 " + (self.root / "deploy.pub").read_text())
+        self.authorized = self.home / ".ssh/authorized_keys"
+
+    def test_credentials_validated_against_port_and_operator_identity(self):
+        remote.validate_credentials(self.spec, "[example.test]:2222")
+        with self.assertRaises(RuntimeError):
+            remote.validate_credentials(self.spec, "example.test")
+        self.spec["deploy_key"] = (self.root / "operator").read_text()
+        with self.assertRaisesRegex(ValueError, "separate"):
+            remote.validate_credentials(self.spec, "[example.test]:2222")
+
+    def test_repeat_and_second_checkout_preserve_operator_keys_and_git_identities(self):
+        self.authorized.parent.mkdir()
+        self.authorized.write_text((self.root / "deploy.pub").read_text())
+        remote.configure_ssh(self.home, self.spec, "example.test")
+        first_config = (self.home / ".gitconfig").read_bytes()
+        remote.configure_ssh(self.home, self.spec, "example.test")
+        self.assertEqual((self.home / ".gitconfig").read_bytes(), first_config)
+        self.assertEqual(self.authorized.read_text().splitlines(), [
+            (self.root / "deploy.pub").read_text().strip(), self.spec["operator_key"].strip()])
+        remote.configure_ssh(self.home, dict(self.spec, node_name="db", deploy_key="second"), "db.test")
+        self.assertEqual((self.authorized.parent / "apex-web.key").read_text(), self.spec["deploy_key"])
+        work = self.root / "checkout"
+        git("init", "-q", work)
+        config = self.home / ".gitconfig"
+        config.write_text(config.read_text().replace("/srv/web", str(work)))
+        env = dict(os.environ, HOME=str(self.home), GIT_CONFIG_NOSYSTEM="1")
+        self.assertIn("apex-web.sshconf", subprocess.check_output(
+            ["git", "-C", str(work), "config", "core.sshCommand"], text=True, env=env))
+        self.assertEqual(subprocess.run(["git", "config", "--get", "core.sshCommand"],
+                         cwd=self.root, env=env, capture_output=True).returncode, 1)
+
+    def test_restricted_key_and_redirected_authorization_are_preserved(self):
+        self.authorized.parent.mkdir()
+        restriction = 'from="192.0.2.1" ' + self.spec["operator_key"]
+        self.authorized.write_text(restriction)
+        with self.assertRaisesRegex(ValueError, "restrictions"):
+            remote.configure_ssh(self.home, self.spec, "example.test")
+        self.assertEqual(self.authorized.read_text(), restriction)
+        victim = self.root / "preserved"
+        self.authorized.rename(victim)
+        for link in (lambda: self.authorized.symlink_to(victim),
+                     lambda: os.link(victim, self.authorized)):
+            link()
+            with self.assertRaisesRegex(ValueError, "Unsafe file"):
+                remote.configure_ssh(self.home, self.spec, "example.test")
+            self.assertEqual(victim.read_text(), restriction)
+            self.authorized.unlink()
+
+
+class Checkout(unittest.TestCase):
+    def test_failed_clone_retries_recursively_and_preserves_local_work(self):
+        with tempfile.TemporaryDirectory() as temp:
+            root = Path(temp).resolve()
+            srv = root / "srv"
+            srv.mkdir()
+            source = root / "repo"
+            account = SimpleNamespace(pw_uid=os.getuid(), pw_gid=os.getgid())
+            with self.assertRaises(subprocess.CalledProcessError):
+                remote.checkout(srv, "node", str(source), account, git)
+            self.assertEqual(list(srv.iterdir()), [])
+            for repo in (source, root / "common"):
+                git("init", "-q", repo)
+                (repo / "sample").write_text("initial")
+                git("-C", repo, "add", "sample")
+                git("-C", repo, "-c", "user.name=T", "-c", "user.email=t@test.example",
+                    "commit", "-qm", "initial")
+            git("-C", source, "submodule", "add", str(root / "common"), "commons")
+            git("-C", source, "-c", "user.name=T", "-c", "user.email=t@test.example",
+                "commit", "-qm", "add submodule")
+            remote.checkout(srv, "node", str(source), account, git)
+            self.assertEqual((srv / "node/commons/sample").read_text(), "initial")
+            dirty = srv / "node/sample"
+            dirty.write_text("local edits")
+            remote.checkout(srv, "node", str(source), account, git)
+            with self.assertRaisesRegex(RuntimeError, "does not match"):
+                remote.checkout(srv, "node", "another-origin", account, git)
+            self.assertEqual(dirty.read_text(), "local edits")

tests/test_manual_host_setup.py

@@ -214,63 +214,3 @@ class HostSetup(unittest.TestCase):
             self.assertEqual({call.args[0] for call in ownership.call_args_list},
                              {"HEAD", "node", "other"})
             self.assertTrue(all(not call.kwargs["follow_symlinks"] for call in ownership.call_args_list))
-
-    def test_operator_provisioning_creates_account_then_preserves_keys_on_rerun(self):
-        adam = action("config/adam")
-        with tempfile.TemporaryDirectory() as temporary:
-            root = Path(temporary).resolve()
-            home = root / "adam"
-            account = SimpleNamespace(pw_dir=str(home), pw_uid=1001)
-            key = root / "operator"
-            subprocess.run(["ssh-keygen", "-q", "-t", "ed25519", "-N", "", "-f", str(key)], check=True)
-            public = key.with_suffix(".pub")
-            ctx = SimpleNamespace(sys=Mock(), host=Mock())
-            ctx.sys.run.side_effect = lambda cmd, **kw: subprocess.run(cmd, input=kw["input"], text=True,
-                                                                       capture_output=True, check=True)
-            def privileged(command, **kwargs):
-                if command[0] == "useradd":
-                    home.mkdir()
-                if command[:6] == ["runuser", "-u", "adam", "--", "sh", "-ec"]:
-                    subprocess.run(command[4:], text=True, input=kwargs["input"], check=True)
-            ctx.sys.sudo.side_effect = privileged
-            actual_stat = Path.stat
-            def metadata(path, *args, **kwargs):
-                stat = actual_stat(path, *args, **kwargs)
-                if path == home:
-                    fields = list(stat)
-                    fields[4] = account.pw_uid
-                    return os.stat_result(fields)
-                return stat
-            with (patch.object(adam, "HOME", home), patch.object(Path, "stat", metadata),
-                  patch.object(adam.grp, "getgrnam", side_effect=KeyError),
-                  patch.object(adam.pwd, "getpwnam", side_effect=[KeyError, account, account])):
-                args = SimpleNamespace(public_key_file=str(public))
-                adam.run(ctx, args)
-                authorized = home / ".ssh/authorized_keys"
-                with authorized.open("a") as stream:
-                    stream.write("existing operator entry\n")
-                adam.run(ctx, args)
-                self.assertEqual(authorized.read_text().splitlines().count(public.read_text().strip()), 1)
-                self.assertIn("existing operator entry", authorized.read_text())
-                commands = [call.args[0] for call in ctx.sys.sudo.call_args_list]
-                self.assertEqual(sum(cmd[0] == "useradd" for cmd in commands), 1)
-                self.assertIn(["runuser", "-u", "adam", "--", "sudo", "-n", "true"], commands)
-
-    def test_existing_system_account_cannot_be_promoted(self):
-        adam = action("config/adam")
-        ctx = SimpleNamespace(sys=Mock())
-        account = SimpleNamespace(pw_dir="/home/adam", pw_uid=999)
-        with patch.object(adam.pwd, "getpwnam", return_value=account):
-            with self.assertRaisesRegex(RuntimeError, "incompatible UID"):
-                adam.run(ctx, SimpleNamespace(public_key_file=None))
-        self.assertFalse(ctx.sys.mock_calls)
-
-    def test_invalid_operator_key_is_rejected_before_provisioning(self):
-        adam = action("config/adam")
-        with tempfile.NamedTemporaryFile(mode="w") as key:
-            key.write("private key fixture")
-            key.flush()
-            ctx = SimpleNamespace(sys=Mock())
-            with self.assertRaisesRegex(ValueError, "OpenSSH public key"):
-                adam.run(ctx, SimpleNamespace(public_key_file=key.name))
-            self.assertFalse(ctx.sys.mock_calls)

tests/test_tooling.py

@@ -24,7 +24,9 @@ class ToolingTests(unittest.TestCase):
             (binaries / 'python3').symlink_to(sys.executable)
             environment = dict(os.environ, PATH=str(binaries))
             sources = []
-            for name in ('engine', 'actions', 'tests', 'ci/scripts'):
+            (root / 'scripts').mkdir()
+            (root / 'scripts/bootstrap-node').write_text('print(1)\n')
+            for name in ('engine', 'actions', 'tests', 'ci/scripts', 'scripts'):
                 source = root / name / 'nested' / 'case.py'
                 source.parent.mkdir(parents=True)
                 source.write_text('value = 1\n')