refactor(provisioning): remove manual SSH bootstrap workflow
This commit is contained in:
deytenit committed
1 parent
f9091d116a
commit
7ad4473e93
9 files changed
+70
-1041
No files matched your search
@@ -47,7 +47,7 @@ You should understand the following invariants before exploring the codebase:
|
||||
- **[`compositions/`](./compositions)**: The shared core Compose stack (`traefik`, `socket-proxy`, `alloy`, `adguard`, `xray`, `restic`).
|
||||
- **[`engine/`](./engine)**: Framework engine — action discovery, overlay resolution, context injection, and logging.
|
||||
- **[`etc/`](./etc)**: Shared static configuration, including Alloy.
|
||||
- **[`ci/`](./ci)**: Host image build recipes, cloud-init templates, and standalone Debian enrollment tooling.
|
||||
- **[`ci/`](./ci)**: Host image build recipes, cloud-init templates, and image enrollment tooling.
|
||||
- **[`assets/`](./assets)**: Repository branding assets (`logo.png`).
|
||||
|
||||
---
|
||||
@@ -104,18 +104,19 @@ Choose one output mode per invocation.
|
||||
|
||||
### Infrastructure Administrator
|
||||
|
||||
Bootstrapping clean Debian 13 hosts on bare metal or cloud providers without cloud-init support.
|
||||
Build Debian 13 amd64 golden images with Packer, then supply node identity,
|
||||
operator keys, and the node repository through
|
||||
[`ci/cloud-init/user-data.template.yaml`](./ci/cloud-init/user-data.template.yaml)
|
||||
at first boot. Keep filled user-data and repository credentials outside Git.
|
||||
|
||||
- **Initialization Tool**: A standalone workstation bootstrap tool targets clean Debian 13 amd64 installations using the same baseline as Packer.
|
||||
- **Template**: Copy and fill out [`ci/cloud-init/user-data.template.yaml`](./ci/cloud-init/user-data.template.yaml) with target identity, operator keys, and node repository.
|
||||
- **Run Enrollment**:
|
||||
```sh
|
||||
python3 ci/scripts/init-node.py root@server --user-data /private/path/location.user-data.yaml
|
||||
```
|
||||
- **Prerequisites**: Run from this commons checkout with Python 3.10+ and OpenSSH. The target must be Debian 13 amd64, reachable using a trusted SSH host key as root or an administrator with passwordless `sudo`. The node checkout must declare `APEX_ENROLLMENT_INTERFACE=1` and have a matching enrollment-aware `init.sh`. Use the supported template subset, not arbitrary cloud-config directives. Keep filled user-data and repository credentials outside Git; supply trusted repository host keys for SSH clones.
|
||||
- **Bootstrap Dependencies**: The target seed installs distro `python3-yaml` for user-data validation; this is separate from the standard-library APEX core. ZFS setup needs matching running-kernel headers/modules. A mismatch stops with a diagnosis; enrollment does not reboot automatically.
|
||||
- **Security Handover**: Safely switches SSH to key-only authentication on port 2222 (`adam@server`) protected by an automated rollback watchdog, activates UFW firewall rules, Docker iptables integration, and CrowdSec.
|
||||
- **Resuming**: Follow the emitted standalone resume command. After committed access handover, use `python3 ci/scripts/init-node.py adam@server --port 2222 --user-data /private/path/location.user-data.yaml`; before handover, use the bootstrap endpoint after pending rollback. An uncertain handover requires checking which endpoint is active. Preserve any `--identity-file` option. Core `apex` has no remote `init` action.
|
||||
```sh
|
||||
packer init ci/packer/apex-debian-13.pkr.hcl
|
||||
packer validate ci/packer/apex-debian-13.pkr.hcl
|
||||
packer build ci/packer/apex-debian-13.pkr.hcl
|
||||
```
|
||||
|
||||
Cloud-init runs the image's bundled enrollment helper. After enrollment, run
|
||||
`apex configure` and `apex compose up` on the node.
|
||||
|
||||
### Developer & Contributor
|
||||
|
||||
@@ -129,9 +130,9 @@ Adding shared actions or improving the engine.
|
||||
. .venv/bin/activate
|
||||
python3 -m pip install ruff==0.17.2 'PyYAML>=6.0,<7'
|
||||
```
|
||||
- **Verification**: `make check` runs lint and both the core and standalone provisioning suites. `make test` (or `bash tests/run.sh`) runs both suites; discovery limited to `tests/` omits provisioning. PyYAML is needed for the provisioning tests, not core execution. Environment integration tests use `docker compose` or standalone `docker-compose` with `config`; no daemon is needed.
|
||||
- **Verification**: `make check` runs lint and both the core and image provisioning suites. `make test` (or `bash tests/run.sh`) runs both suites; discovery limited to `tests/` omits provisioning. PyYAML is needed for the provisioning tests, not core execution. Environment integration tests use `docker compose` or standalone `docker-compose` with `config`; no daemon is needed.
|
||||
- **Linting**: Ruff checks correctness and security rules. Existing import-order, line-length, compact-statement, and annotation-style debt is explicitly deferred in `pyproject.toml` to avoid unrelated formatting changes. Without Ruff, `make lint` recursively checks Python compilation and fails on syntax errors; that fallback is not equivalent to Ruff. `make fmt` formats files and is a separate, deliberate operation.
|
||||
- **Packer**: Run `packer init ci/packer/apex-debian-13.pkr.hcl` and `packer validate ci/packer/apex-debian-13.pkr.hcl` for template validation. A VM build or live enrollment is a separate operation; unit tests and template validation do not demonstrate a deployed host.
|
||||
- **Packer**: Run `packer init ci/packer/apex-debian-13.pkr.hcl` and `packer validate ci/packer/apex-debian-13.pkr.hcl` for template validation. A VM build or first-boot enrollment is a separate operation; unit tests and template validation do not demonstrate a deployed host.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#cloud-config
|
||||
# Use this same file for cloud-init or: python3 ci/scripts/init-node.py root@HOST --user-data FILE
|
||||
# Supply this file to cloud-init when booting the golden image.
|
||||
# Replace the hostname, repository and public key. Keep filled files outside Git.
|
||||
# SSH repositories additionally require approved deferred id_ed25519 (0600)
|
||||
# and known_hosts (0644) writes under /home/adam/.ssh, owned by adam:adam.
|
||||
|
||||
@@ -1,65 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Initialize a plain Debian target server via SSH enrollment.
|
||||
|
||||
Standalone tool for hosts on providers that do not support cloud-init.
|
||||
Kept separate from the core local-only apex engine.
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
|
||||
_here = os.path.dirname(os.path.abspath(__file__))
|
||||
_repo_root = os.path.dirname(os.path.dirname(_here))
|
||||
_prov_dir = os.path.join(_here, "init-node")
|
||||
sys.path.insert(0, _prov_dir)
|
||||
|
||||
# The standalone package lives beside this launcher, outside the engine.
|
||||
from provisioning.transport import initialize # noqa: E402
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Initialize a plain Debian server through protected SSH enrollment.",
|
||||
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
|
||||
)
|
||||
parser.add_argument(
|
||||
"target",
|
||||
help="Bootstrap USER@HOST (root or an existing administrator with passwordless sudo)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--user-data",
|
||||
dest="user_data",
|
||||
help="Path to APEX cloud-config file (e.g., ci/cloud-init/user-data.template.yaml)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--port",
|
||||
dest="port",
|
||||
help="Bootstrap SSH port; otherwise uses standard SSH configuration",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--identity-file",
|
||||
dest="identity_file",
|
||||
help="SSH private identity file; otherwise uses SSH configuration/agent",
|
||||
)
|
||||
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
print(f"[init-node] Starting enrollment for {args.target}...")
|
||||
try:
|
||||
initialize(
|
||||
_repo_root,
|
||||
args.target,
|
||||
args.user_data,
|
||||
args.port,
|
||||
args.identity_file,
|
||||
progress=lambda msg: print(f"[init-node] {msg}"),
|
||||
)
|
||||
print(f"[init-node] Successfully initialized {args.target}.")
|
||||
return 0
|
||||
except Exception as e:
|
||||
sys.stderr.write(f"[init-node] Error: {e}\n")
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,51 @@
|
||||
"""Deterministic provisioning payloads for golden-image builds."""
|
||||
import hashlib
|
||||
import io
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import tarfile
|
||||
|
||||
from .state import ProvisionError
|
||||
|
||||
|
||||
def _bundle_from_source(root):
|
||||
root = Path(root).resolve()
|
||||
package = root / 'provisioning'
|
||||
files = sorted(package.rglob('*.py'))
|
||||
seed = package / 'seed.sh'
|
||||
guard = package / 'profiles.sh'
|
||||
if not seed.is_file() or not (package / '__init__.py').is_file():
|
||||
raise ProvisionError('Provisioning package or seed is missing')
|
||||
files.append(seed)
|
||||
if guard.is_file():
|
||||
files.append(guard)
|
||||
output = io.BytesIO()
|
||||
with tarfile.open(fileobj=output, mode='w') as archive:
|
||||
for path in files:
|
||||
if (not path.is_file() or '__pycache__' in path.parts
|
||||
or any(part.is_symlink() for part in (path, *path.parents))):
|
||||
raise ProvisionError('Provisioning bundle contains an unsafe source path')
|
||||
data = path.read_bytes()
|
||||
info = tarfile.TarInfo(path.relative_to(root).as_posix())
|
||||
info.size, info.mode, info.mtime = len(data), (0o755 if path == seed else 0o644), 0
|
||||
info.uid = info.gid = 0
|
||||
archive.addfile(info, io.BytesIO(data))
|
||||
data = output.getvalue()
|
||||
return data, hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
def build_bundle(commons):
|
||||
root = Path(commons).resolve()
|
||||
data, digest = _bundle_from_source(root / 'ci/scripts/init-node')
|
||||
revision = 'unversioned-modified'
|
||||
if shutil.which('git'):
|
||||
result = subprocess.run(['git', '-C', str(root), 'rev-parse', 'HEAD'], capture_output=True, text=True, timeout=30)
|
||||
if result.returncode == 0 and re.fullmatch(r'[a-f0-9]{40,64}\n?', result.stdout):
|
||||
revision = result.stdout.strip()
|
||||
dirty = subprocess.run(['git', '-C', str(root), 'status', '--porcelain', '--', 'ci/scripts/init-node'], capture_output=True, text=True, timeout=30)
|
||||
if dirty.returncode or dirty.stdout.strip():
|
||||
revision += '-modified'
|
||||
return data, digest, revision
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Image-only installation and sanitization; never called by remote enrollment."""
|
||||
"""Image-only installation and sanitization; for golden-image builds."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
@@ -16,7 +16,7 @@ from .target import preflight, run
|
||||
|
||||
|
||||
def install():
|
||||
from .transport import _bundle_from_source
|
||||
from .bundle import _bundle_from_source
|
||||
source = Path(__file__).resolve().parents[1]
|
||||
data, digest = _bundle_from_source(source)
|
||||
destination = safe_path(Path('/usr/local/lib/apex-provisioner') / digest)
|
||||
|
||||
@@ -1,394 +0,0 @@
|
||||
"""Workstation SSH transport. User-data stays opaque until target validation."""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import ipaddress
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shlex
|
||||
import shutil
|
||||
import subprocess
|
||||
import tarfile
|
||||
import tempfile
|
||||
import time
|
||||
import uuid
|
||||
|
||||
|
||||
class TransportError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class HandoverUncertain(TransportError):
|
||||
pass
|
||||
|
||||
|
||||
# This runs before Python is guaranteed to exist on the Debian target. All
|
||||
# ancestors must be trusted; a root-owned sticky temporary directory is safe
|
||||
# only because root owns each managed child and exclusive mktemp creates it.
|
||||
REMOTE_PATH_GUARD = r'''
|
||||
set -eu
|
||||
umask 077
|
||||
apex_guard_dir() {
|
||||
apex_checked=$1
|
||||
while :; do
|
||||
test ! -L "$apex_checked" && test -d "$apex_checked" || exit 91
|
||||
test "$(stat -c %u -- "$apex_checked")" = 0 || exit 91
|
||||
apex_mode=$(stat -c %a -- "$apex_checked")
|
||||
test "$((0$apex_mode & 0022))" = 0 || test "$((0$apex_mode & 01000))" != 0 || exit 91
|
||||
test "$apex_checked" != / || break
|
||||
apex_checked=${apex_checked%/*}
|
||||
test -n "$apex_checked" || apex_checked=/
|
||||
done
|
||||
}
|
||||
apex_ensure_dir() {
|
||||
apex_new_dir=$1
|
||||
if test -e "$apex_new_dir" || test -L "$apex_new_dir"; then
|
||||
apex_guard_dir "$apex_new_dir"
|
||||
apex_mode=$(stat -c %a -- "$apex_new_dir")
|
||||
test "$((0$apex_mode & 0022))" = 0 || exit 91
|
||||
else
|
||||
apex_guard_dir "${apex_new_dir%/*}"
|
||||
mkdir -m "$2" -- "$apex_new_dir"
|
||||
apex_guard_dir "$apex_new_dir"
|
||||
fi
|
||||
}
|
||||
apex_guard_file() {
|
||||
apex_file=$1
|
||||
apex_guard_dir "${apex_file%/*}"
|
||||
apex_mode=$(stat -c %a -- "${apex_file%/*}")
|
||||
test "$((0$apex_mode & 0022))" = 0 || exit 91
|
||||
test ! -L "$apex_file" || exit 91
|
||||
if test -e "$apex_file"; then
|
||||
test -f "$apex_file" || exit 91
|
||||
test "$(stat -c %u -- "$apex_file")" = 0 || exit 91
|
||||
test "$(stat -c %h -- "$apex_file")" = 1 || exit 91
|
||||
apex_mode=$(stat -c %a -- "$apex_file")
|
||||
test "$((0$apex_mode & 0022))" = 0 || exit 91
|
||||
fi
|
||||
}
|
||||
apex_atomic_input() {
|
||||
apex_destination=$1
|
||||
apex_guard_file "$apex_destination"
|
||||
apex_temporary=$(mktemp "${apex_destination%/*}/.apex-write.XXXXXXXXXX")
|
||||
trap 'rm -f -- "$apex_temporary"' EXIT HUP INT TERM
|
||||
cat > "$apex_temporary"
|
||||
chmod 0600 "$apex_temporary"
|
||||
apex_guard_file "$apex_destination"
|
||||
mv -T -- "$apex_temporary" "$apex_destination"
|
||||
trap - EXIT HUP INT TERM
|
||||
}
|
||||
'''
|
||||
|
||||
|
||||
def remote_atomic_input(path):
|
||||
return REMOTE_PATH_GUARD + '\napex_atomic_input ' + shlex.quote(path)
|
||||
|
||||
|
||||
def remote_bundle_install(stage, installed, digest):
|
||||
return REMOTE_PATH_GUARD + '\n' + (
|
||||
'apex_guard_dir ' + shlex.quote(stage) + '; apex_guard_file ' + shlex.quote(stage + '/bundle.tar') + '; '
|
||||
'test "$(sha256sum ' + shlex.quote(stage + '/bundle.tar') + ' | cut -d " " -f 1)" = ' + digest + '; '
|
||||
'apex_ensure_dir /usr/local/lib/apex-provisioner 0755; '
|
||||
'if test -e ' + installed + ' || test -L ' + installed + '; then '
|
||||
'apex_guard_dir ' + installed + '; '
|
||||
'apex_bad=$(find ' + installed + ' -xdev \\( -type l -o ! -user root -o -perm /022 -o \\( ! -type d -a ! -type f \\) \\) -print -quit); test -z "$apex_bad"; '
|
||||
'else apex_install_tmp=$(mktemp -d /usr/local/lib/apex-provisioner/.install.XXXXXXXXXX); '
|
||||
'trap \'rm -rf -- "$apex_install_tmp"\' EXIT HUP INT TERM; '
|
||||
'tar --no-same-owner --same-permissions -xf ' + stage + '/bundle.tar -C "$apex_install_tmp"; '
|
||||
'mv -T -- "$apex_install_tmp" ' + installed + '; trap - EXIT HUP INT TERM; fi; '
|
||||
'tar --compare -f ' + stage + '/bundle.tar -C ' + installed)
|
||||
|
||||
|
||||
def local_file(value, name):
|
||||
if not value or any(ord(c) < 32 for c in str(value)):
|
||||
raise TransportError(f'{name} must name a readable regular file')
|
||||
try:
|
||||
path = Path(value).expanduser().resolve(strict=True)
|
||||
except (OSError, ValueError):
|
||||
raise TransportError(f'{name} must name a readable regular file') from None
|
||||
if not path.is_file() or not os.access(path, os.R_OK):
|
||||
raise TransportError(f'{name} must name a readable regular file')
|
||||
return path
|
||||
|
||||
|
||||
def destination(value, port=None):
|
||||
match = re.fullmatch(r'([A-Za-z_][A-Za-z0-9_.-]{0,31})@([A-Za-z0-9][A-Za-z0-9_.:-]*|\[[0-9a-fA-F:]+\])', value or '')
|
||||
if not match or '..' in match[2]:
|
||||
raise TransportError('Target must be USER@HOST without SSH options; use root or an existing administrator with sudo -n')
|
||||
hostname = match[2].strip('[]')
|
||||
if ':' in hostname:
|
||||
try:
|
||||
ipaddress.IPv6Address(hostname)
|
||||
except ValueError:
|
||||
raise TransportError('IPv6 targets must contain a valid address; pass ports separately') from None
|
||||
if port is not None and (not re.fullmatch(r'[0-9]{1,5}', str(port)) or not 1 <= int(port) <= 65535):
|
||||
raise TransportError('Port must be an integer from 1 through 65535')
|
||||
return match[1], hostname, int(port) if port is not None else None
|
||||
|
||||
|
||||
def _bundle_from_source(root):
|
||||
root = Path(root).resolve()
|
||||
package = root / 'provisioning'
|
||||
files = sorted(package.rglob('*.py'))
|
||||
seed = package / 'seed.sh'
|
||||
if not seed.is_file() or not (package / '__init__.py').is_file():
|
||||
raise TransportError('Provisioning package or seed is missing')
|
||||
files.append(seed)
|
||||
output = io.BytesIO()
|
||||
with tarfile.open(fileobj=output, mode='w') as archive:
|
||||
for path in files:
|
||||
if (not path.is_file() or '__pycache__' in path.parts
|
||||
or any(part.is_symlink() for part in (path, *path.parents))):
|
||||
raise TransportError('Provisioning bundle contains an unsafe source path')
|
||||
data = path.read_bytes()
|
||||
info = tarfile.TarInfo(path.relative_to(root).as_posix())
|
||||
info.size, info.mode, info.mtime = len(data), (0o755 if path == seed else 0o644), 0
|
||||
info.uid = info.gid = 0
|
||||
archive.addfile(info, io.BytesIO(data))
|
||||
data = output.getvalue()
|
||||
return data, hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
def build_bundle(commons):
|
||||
root = Path(commons).resolve()
|
||||
data, digest = _bundle_from_source(root / 'ci/scripts/init-node')
|
||||
revision = 'unversioned-modified'
|
||||
if shutil.which('git'):
|
||||
result = subprocess.run(['git', '-C', str(root), 'rev-parse', 'HEAD'], capture_output=True, text=True, timeout=30)
|
||||
if result.returncode == 0 and re.fullmatch(r'[a-f0-9]{40,64}\n?', result.stdout):
|
||||
revision = result.stdout.strip()
|
||||
dirty = subprocess.run(['git', '-C', str(root), 'status', '--porcelain', '--', 'ci/scripts/init-node', 'ci/scripts/init-node.py'], capture_output=True, text=True, timeout=30)
|
||||
if dirty.returncode or dirty.stdout.strip():
|
||||
revision += '-modified'
|
||||
return data, digest, revision
|
||||
|
||||
|
||||
class SSH:
|
||||
def __init__(self, target, port=None, identity_file=None, runner=subprocess.run):
|
||||
self.user, self.host, self.port = destination(target, port)
|
||||
self.identity = local_file(identity_file, 'Identity file') if identity_file else None
|
||||
self.runner = runner
|
||||
self.resolved_host = None
|
||||
self.resolved_port = None
|
||||
self.trust_file = None
|
||||
|
||||
def arguments(self, fresh=False):
|
||||
args = ['ssh', '-T', '-o', 'StrictHostKeyChecking=yes', '-o', 'ConnectTimeout=15',
|
||||
'-o', 'ServerAliveInterval=10', '-o', 'ServerAliveCountMax=3']
|
||||
if self.identity:
|
||||
args += ['-i', str(self.identity)]
|
||||
args += ['-l', 'adam' if fresh else self.user]
|
||||
if fresh:
|
||||
if not self.trust_file or not self.resolved_host:
|
||||
raise TransportError('Fresh access requires the verified bootstrap host identity')
|
||||
args += ['-p', '2222', '-o', 'ControlMaster=no', '-o', 'ControlPath=none',
|
||||
'-o', 'ControlPersist=no', '-o', 'BatchMode=yes',
|
||||
'-o', 'PreferredAuthentications=publickey', '-o', 'PasswordAuthentication=no',
|
||||
'-o', 'KbdInteractiveAuthentication=no', '-o', 'Hostname=' + self.resolved_host,
|
||||
'-o', 'HostKeyAlias=apex-init-verified-target',
|
||||
'-o', 'UserKnownHostsFile=' + str(self.trust_file),
|
||||
'-o', 'GlobalKnownHostsFile=/dev/null']
|
||||
elif self.port:
|
||||
args += ['-p', str(self.port)]
|
||||
return args
|
||||
|
||||
def resolve(self):
|
||||
result = self.runner(self.arguments() + ['-G', '--', self.host], capture_output=True, text=True, timeout=30)
|
||||
if result.returncode:
|
||||
raise TransportError('Unable to resolve SSH connection configuration')
|
||||
values = dict(line.split(' ', 1) for line in result.stdout.splitlines() if ' ' in line)
|
||||
hostname = values.get('hostname', '')
|
||||
port = values.get('port', '')
|
||||
destination(self.user + '@' + hostname, port)
|
||||
self.resolved_host, self.resolved_port = hostname, int(port)
|
||||
|
||||
def run(self, script, data=None, fresh=False, privileged=True, check=True, timeout=300):
|
||||
command = ['sh', '-c', script]
|
||||
if privileged and (fresh or self.user != 'root'):
|
||||
command = ['sudo', '-n', '--', *command]
|
||||
result = self.runner(self.arguments(fresh) + ['--', self.host, shlex.join(command)],
|
||||
input=data, capture_output=True, timeout=timeout)
|
||||
if check and result.returncode:
|
||||
# SSH and child diagnostics can contain user-data or credential-bearing URLs.
|
||||
raise TransportError('Remote operation failed; inspect the root-owned target status for a redacted diagnosis')
|
||||
return result
|
||||
|
||||
def fresh_login(self):
|
||||
self.run('test "$(id -u)" = 0', fresh=True)
|
||||
|
||||
|
||||
def protected_handover(ssh, invoke, token, port, client_address):
|
||||
result = invoke('begin', token, port, client_address, False)
|
||||
try:
|
||||
ssh.fresh_login()
|
||||
if result['status'] == 'committed':
|
||||
return
|
||||
if result['status'] != 'existing':
|
||||
invoke('finalize', token, None, None, True)
|
||||
ssh.fresh_login()
|
||||
try:
|
||||
invoke('commit', token, None, None, True)
|
||||
except Exception:
|
||||
try:
|
||||
state = invoke('status', token, None, None, True)
|
||||
if state.get('token') != token or state.get('status') not in ('committed', 'existing', 'transition', 'final', 'rolled_back', 'recovery_pending'):
|
||||
raise TransportError('Unexpected access transition state')
|
||||
except Exception:
|
||||
raise HandoverUncertain('Access commit outcome is unknown; verify the administrator and bootstrap endpoints before resuming') from None
|
||||
if state['status'] != 'committed':
|
||||
raise
|
||||
except HandoverUncertain:
|
||||
raise
|
||||
except Exception:
|
||||
try:
|
||||
invoke('rollback', token, None, None, True)
|
||||
except Exception:
|
||||
try:
|
||||
invoke('rollback', token, None, None, False)
|
||||
except Exception: # noqa: S110 — diagnostics may contain secrets.
|
||||
pass # Persistent timer is authoritative if both connections are gone.
|
||||
raise
|
||||
|
||||
|
||||
def initialize(commons, target, user_data, port=None, identity_file=None, progress=print):
|
||||
if not shutil.which('ssh'):
|
||||
raise TransportError('OpenSSH ssh is required on the workstation')
|
||||
source = local_file(user_data, 'User-data')
|
||||
if source.stat().st_size > 1024 * 1024:
|
||||
raise TransportError('User-data must be no larger than 1 MiB')
|
||||
ssh = SSH(target, port, identity_file)
|
||||
ssh.resolve()
|
||||
bundle, digest, revision = build_bundle(commons)
|
||||
run_id = uuid.uuid4().hex
|
||||
installed = '/usr/local/lib/apex-provisioner/' + digest
|
||||
stage = None
|
||||
launched = False
|
||||
finished = False
|
||||
access_committed = False
|
||||
access_uncertain = False
|
||||
cleanup_policy = '/etc/tmpfiles.d/apex-init-' + run_id + '.conf'
|
||||
with tempfile.TemporaryDirectory(prefix='apex-init-trust-') as local:
|
||||
try:
|
||||
preflight = ssh.run('set -eu; test "$(id -u)" = 0; . /etc/os-release; '
|
||||
'test "$ID" = debian; test "$VERSION_ID" = 13; '
|
||||
'test "$(dpkg --print-architecture)" = amd64; '
|
||||
'command -v systemd-run >/dev/null; '
|
||||
'printf "%s\\n" "${SSH_CONNECTION:-}"; '
|
||||
'cat /etc/ssh/ssh_host_*_key.pub')
|
||||
lines = preflight.stdout.decode().splitlines()
|
||||
connection = lines[0].split() if lines else []
|
||||
if len(connection) != 4:
|
||||
# sudo commonly clears SSH_CONNECTION; read it before elevation instead.
|
||||
raw = ssh.run('printf "%s\\n" "$SSH_CONNECTION"', privileged=False).stdout.decode().split()
|
||||
connection = raw
|
||||
if len(connection) != 4:
|
||||
raise TransportError('Unable to inspect the established SSH connection')
|
||||
keys = [line for line in lines[1:] if re.fullmatch(r'(ssh-[\w-]+|ecdsa-[\w-]+) [A-Za-z0-9+/=]+(?: .*?)?', line)]
|
||||
if not keys:
|
||||
raise TransportError('Unable to obtain public host keys through the verified connection')
|
||||
trust = Path(local) / 'known_hosts'
|
||||
trust.write_text(''.join('apex-init-verified-target,[apex-init-verified-target]:2222 ' + ' '.join(key.split()[:2]) + '\n' for key in keys))
|
||||
trust.chmod(0o600)
|
||||
ssh.trust_file = trust
|
||||
existing = ssh.run(REMOTE_PATH_GUARD + '\napex_ensure_dir /var/lib/apex-init 0700; '
|
||||
'apex_guard_file /var/lib/apex-init/remote.json; '
|
||||
'if test -f /var/lib/apex-init/remote.json; then cat /var/lib/apex-init/remote.json; fi').stdout
|
||||
if existing:
|
||||
prior = json.loads(existing)
|
||||
previous_unit = prior.get('unit', '')
|
||||
if not re.fullmatch(r'apex-init-[a-f0-9]{32}', previous_unit):
|
||||
raise TransportError('Existing remote coordinator state requires manual inspection')
|
||||
active = ssh.run(shlex.join(['systemctl', 'is-active', '--quiet', previous_unit]), check=False)
|
||||
if active.returncode == 0:
|
||||
raise TransportError('An enrollment unit is still running: ' + previous_unit + '; wait for it to settle before resuming')
|
||||
stage = ssh.run(REMOTE_PATH_GUARD + '\napex_guard_dir /var/tmp; mktemp -d /var/tmp/apex-init.XXXXXXXXXX').stdout.decode().strip()
|
||||
if not re.fullmatch(r'/var/tmp/apex-init\.[A-Za-z0-9]{10}', stage): # noqa: S108 — validates remote mktemp output.
|
||||
raise TransportError('Target returned an invalid staging path')
|
||||
# Hard reboot can bypass the shell trap; boot-time tmpfiles removes opaque inputs.
|
||||
cleanup = ('r ' + stage + '/user-data - - - -\n' +
|
||||
'r ' + stage + '/heartbeat - - - -\n').encode()
|
||||
ssh.run(REMOTE_PATH_GUARD + '\napex_ensure_dir /etc/tmpfiles.d 0755; apex_atomic_input ' + cleanup_policy, cleanup)
|
||||
ssh.run(remote_atomic_input(stage + '/bundle.tar'), bundle)
|
||||
ssh.run(remote_atomic_input(stage + '/user-data'), source.read_bytes())
|
||||
install = remote_bundle_install(stage, installed, digest)
|
||||
progress('Installing the versioned target runtime and validating enrollment input.')
|
||||
ssh.run(install)
|
||||
base = ['python3', '-m', 'provisioning.target']
|
||||
options = ['--user-data', stage + '/user-data', '--revision', revision, '--bundle-digest', digest]
|
||||
# State remains inspectable after disconnect; user-data is removed by the durable unit.
|
||||
enrollment = shlex.join(base + ['enroll'] + options + ['--heartbeat', stage + '/heartbeat'])
|
||||
validation = shlex.join(base + ['validate'] + options)
|
||||
access_preflight = shlex.join(['python3', '-m', 'provisioning.access', 'preflight'])
|
||||
seed = shlex.join(['bash', installed + '/provisioning/seed.sh'])
|
||||
wrapper = ('umask 077; cd ' + installed + '; '
|
||||
'trap ' + shlex.quote('rm -f ' + stage + '/user-data ' + stage + '/heartbeat ' + cleanup_policy) + ' EXIT; '
|
||||
'{ ' + seed + ' && ' + validation + ' && ' + access_preflight + ' && ' + enrollment + '; } >' + stage + '/runner.log 2>&1; rc=$?; '
|
||||
'printf "%s\\n" "$rc" >' + stage + '/result.new; mv ' + stage + '/result.new ' + stage + '/result; exit "$rc"')
|
||||
unit = 'apex-init-' + run_id
|
||||
record = json.dumps({'unit': unit, 'staging': stage, 'revision': revision, 'bundle_digest': digest}).encode()
|
||||
ssh.run(remote_atomic_input('/var/lib/apex-init/remote.json'), record)
|
||||
ssh.run('touch ' + stage + '/heartbeat; ' + shlex.join(['systemd-run', '--quiet', '--unit', unit,
|
||||
'--property=Type=exec', '--property=KillMode=process', '/bin/sh', '-c', wrapper]))
|
||||
launched = True
|
||||
progress('Enrollment is running under ' + unit + '; target staging: ' + stage)
|
||||
while True:
|
||||
status = ssh.run('if test -f ' + stage + '/result; then cat ' + stage + '/result; '
|
||||
'elif systemctl is-active --quiet ' + unit + '; then touch ' + stage + '/heartbeat; printf running; '
|
||||
'else printf stopped; fi').stdout.decode().strip()
|
||||
if status != 'running':
|
||||
finished = True
|
||||
if status != '0':
|
||||
raise TransportError('Target provisioning failed; inspect root-owned ' + stage +
|
||||
'/runner.log and /var/lib/apex-init/enrollment.json')
|
||||
break
|
||||
time.sleep(5)
|
||||
progress('Enrollment verified. Checking protected administrator SSH handover.')
|
||||
|
||||
def access(operation, token, original_port, address, fresh):
|
||||
if operation == 'status':
|
||||
result = ssh.run(REMOTE_PATH_GUARD + '\napex_guard_file /var/lib/apex-init/access/state.json; '
|
||||
'cat /var/lib/apex-init/access/state.json', fresh=True)
|
||||
return json.loads(result.stdout)
|
||||
args = ['python3', '-m', 'provisioning.access', operation, '--token', token]
|
||||
if original_port:
|
||||
args += ['--original-port', str(original_port), '--client-address', address]
|
||||
result = ssh.run('cd ' + installed + ' && ' + shlex.join(args), fresh=fresh)
|
||||
return json.loads(result.stdout)
|
||||
|
||||
protected_handover(ssh, access, run_id, int(connection[3]), connection[0])
|
||||
access_committed = True
|
||||
result = ssh.run('cd ' + installed + ' && ' + shlex.join(base + ['complete']), fresh=True)
|
||||
completed = json.loads(result.stdout)
|
||||
progress('Initialization and fresh administrator access verified; provisioner ' + revision + ', bundle ' + digest)
|
||||
repository = completed.get('stages', {}).get('repository', {}).get('result', {})
|
||||
if repository.get('commit'):
|
||||
progress('Repository ' + repository['commit'] + '; commons ' + repository.get('commons_commit', 'unknown'))
|
||||
return completed
|
||||
except HandoverUncertain:
|
||||
access_uncertain = True
|
||||
raise
|
||||
finally:
|
||||
if stage and (not launched or finished):
|
||||
for fresh in (True, False):
|
||||
try:
|
||||
ssh.run('rm -f ' + stage + '/user-data ' + stage + '/heartbeat ' + stage + '/bundle.tar ' + cleanup_policy, fresh=fresh)
|
||||
break
|
||||
except Exception: # noqa: S110 — cleanup tries both verified endpoints.
|
||||
pass
|
||||
if launched and not finished:
|
||||
progress('Coordinator stopped. The current target stage may finish; later stages stop after heartbeat expiry.')
|
||||
if access_uncertain:
|
||||
progress('Access commit outcome is unknown; verify which endpoint is active before resuming. Rollback may already be disabled.')
|
||||
endpoints = [('Administrator candidate', 'adam', 2222), ('Bootstrap candidate', ssh.user, ssh.resolved_port)]
|
||||
else:
|
||||
label = 'Safe resume' + ('' if access_committed else ' after any pending rollback')
|
||||
endpoints = [(label, 'adam' if access_committed else ssh.user, 2222 if access_committed else ssh.resolved_port)]
|
||||
for label, resume_user, resume_port in endpoints:
|
||||
resume = ['python3', 'ci/scripts/init-node.py', resume_user + '@' + ssh.host,
|
||||
'--port', str(resume_port), '--user-data', str(source)]
|
||||
if ssh.identity:
|
||||
resume += ['--identity-file', str(ssh.identity)]
|
||||
progress(label + ': ' + shlex.join(resume))
|
||||
@@ -11,7 +11,7 @@ from unittest.mock import patch
|
||||
|
||||
from provisioning import access, image
|
||||
from provisioning.image import cloud_enroll, sanitize_files
|
||||
from provisioning.transport import build_bundle
|
||||
from provisioning.bundle import build_bundle
|
||||
from provisioning.state import atomic_write, safe_path
|
||||
from provisioning.state import ProvisionError
|
||||
|
||||
|
||||
@@ -1,560 +0,0 @@
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from provisioning.transport import (
|
||||
SSH, TransportError, HandoverUncertain, build_bundle, destination, local_file, protected_handover,
|
||||
initialize,
|
||||
remote_atomic_input,
|
||||
)
|
||||
|
||||
|
||||
class TransportTests(unittest.TestCase):
|
||||
def test_durable_runner_checks_access_after_seed_before_enrollment(self):
|
||||
for compatible in (False, True):
|
||||
with self.subTest(compatible=compatible), tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
source = root / 'user-data'
|
||||
source.write_bytes(b'opaque input')
|
||||
stage = root / 'stage'
|
||||
stage.mkdir()
|
||||
installed = root / 'bundle'
|
||||
seed = installed / 'provisioning/seed.sh'
|
||||
seed.parent.mkdir(parents=True)
|
||||
seed.write_text('printf "seed\\n" >> "$APEX_TEST_EVENTS"\n')
|
||||
binaries = root / 'bin'
|
||||
binaries.mkdir()
|
||||
python = binaries / 'python3'
|
||||
python.write_text('#!' + sys.executable + '\n' + '''
|
||||
import os, sys
|
||||
module, operation = sys.argv[2:4]
|
||||
event = 'access-' + operation if module.endswith('.access') else operation
|
||||
with open(os.environ['APEX_TEST_EVENTS'], 'a') as stream:
|
||||
stream.write(event + '\\n')
|
||||
if event == 'access-preflight' and os.environ['APEX_TEST_COMPATIBLE'] == 'False':
|
||||
sys.exit(42)
|
||||
if event == 'enroll':
|
||||
sys.exit(43)
|
||||
''')
|
||||
python.chmod(0o755)
|
||||
events = root / 'events'
|
||||
environment = dict(os.environ, PATH=str(binaries) + os.pathsep + os.environ['PATH'],
|
||||
APEX_TEST_EVENTS=str(events), APEX_TEST_COMPATIBLE=str(compatible))
|
||||
class Remote:
|
||||
user, host, identity = 'root', 'example.com', None
|
||||
resolved_port = 22
|
||||
def __init__(self, *args):
|
||||
pass
|
||||
def resolve(self):
|
||||
pass
|
||||
def run(self, script, data=None, installed=installed, stage=stage,
|
||||
root=root, environment=environment, **kwargs):
|
||||
value = b''
|
||||
if 'ssh_host_' in script:
|
||||
value = b'192.0.2.1 50000 192.0.2.2 22\nssh-ed25519 AAAA fixture\n'
|
||||
elif 'mktemp -d /var/tmp/apex-init.' in script:
|
||||
value = b'/var/tmp/apex-init.0123456789\n'
|
||||
elif 'systemd-run --quiet' in script:
|
||||
arguments = shlex.split(script)
|
||||
wrapper = arguments[-1].replace('/usr/local/lib/apex-provisioner/' + 'a' * 64, str(installed))
|
||||
wrapper = wrapper.replace('/var/tmp/apex-init.0123456789', str(stage))
|
||||
wrapper = wrapper.replace('/etc/tmpfiles.d/', str(root) + '/')
|
||||
subprocess.run(['/bin/sh', '-c', wrapper], env=environment,
|
||||
capture_output=True, timeout=30)
|
||||
elif script.startswith('if test -f /var/tmp'):
|
||||
value = (stage / 'result').read_bytes()
|
||||
return subprocess.CompletedProcess([], 0, value, b'')
|
||||
with patch('provisioning.transport.SSH', Remote), \
|
||||
patch('provisioning.transport.build_bundle', return_value=(b'bundle', 'a' * 64, 'b' * 40)), \
|
||||
patch('provisioning.transport.shutil.which', return_value='/usr/bin/ssh'):
|
||||
with self.assertRaises(TransportError):
|
||||
initialize(root, 'root@example.com', source, progress=lambda message: None)
|
||||
expected = ['seed', 'validate', 'access-preflight']
|
||||
if compatible:
|
||||
expected.append('enroll')
|
||||
self.assertEqual(events.read_text().splitlines(), expected)
|
||||
self.assertEqual((stage / 'result').read_text().strip(), '43' if compatible else '42')
|
||||
|
||||
def test_rejects_options_shell_fragments_and_invalid_ports(self):
|
||||
for value in ['root@-oProxyCommand=id', 'root@host;id', '-oProxyCommand=id@host',
|
||||
'bob;id@host', 'bob name@host', 'bob\n@host', 'bob/other@host',
|
||||
'bob$(id)@host', '@host', '9bob@host', 'a' * 33 + '@host',
|
||||
'root@host\nfoo', 'host', 'debian@host -oProxyCommand=id']:
|
||||
with self.subTest(value=value), self.assertRaises(TransportError):
|
||||
destination(value)
|
||||
for port in ['0', '65536', '-1', '22;id', '2.5', '']:
|
||||
with self.subTest(port=port), self.assertRaises(TransportError):
|
||||
destination('root@example.com', port)
|
||||
self.assertEqual(destination('adam@node', 2222), ('adam', 'node', 2222))
|
||||
|
||||
def test_existing_bootstrap_accounts_are_accepted(self):
|
||||
for user in ['root', 'adam', 'debian', 'bob', 'cloud-admin', '_admin', 'Admin.1']:
|
||||
with self.subTest(user=user):
|
||||
self.assertEqual(destination(user + '@example.com'), (user, 'example.com', None))
|
||||
self.assertEqual(destination('debian@[2001:db8::1]', 22), ('debian', '2001:db8::1', 22))
|
||||
|
||||
def test_nonroot_bootstrap_uses_noninteractive_sudo_and_preserves_stdin(self):
|
||||
calls = []
|
||||
def runner(args, **kwargs):
|
||||
calls.append((args, kwargs))
|
||||
return subprocess.CompletedProcess(args, 0, b'root verified', b'')
|
||||
ssh = SSH('debian@host', runner=runner)
|
||||
self.assertEqual(ssh.run('id -u', b'opaque input').stdout, b'root verified')
|
||||
args, kwargs = calls[0]
|
||||
self.assertEqual(args[args.index('-l') + 1], 'debian')
|
||||
self.assertEqual(shlex.split(args[-1]), ['sudo', '-n', '--', 'sh', '-c', 'id -u'])
|
||||
self.assertEqual(kwargs['input'], b'opaque input')
|
||||
ssh.run('printf %s "$SSH_CONNECTION"', privileged=False)
|
||||
self.assertEqual(shlex.split(calls[1][0][-1])[:2], ['sh', '-c'])
|
||||
|
||||
def test_sudo_cleared_connection_environment_uses_unprivileged_fallback(self):
|
||||
class PreflightComplete(Exception):
|
||||
pass
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
source = root / 'user-data'
|
||||
source.write_bytes(b'opaque input')
|
||||
release = root / 'os-release'
|
||||
release.write_text('ID=debian\nVERSION_ID=13\n')
|
||||
systemd_run = root / 'systemd-run'
|
||||
systemd_run.write_text('#!/bin/sh\nexit 0\n')
|
||||
systemd_run.chmod(0o755)
|
||||
environment = dict(os.environ, PATH=str(root) + os.pathsep + os.environ['PATH'])
|
||||
environment.pop('SSH_CONNECTION', None)
|
||||
fallbacks = []
|
||||
def runner(args, **kwargs):
|
||||
if '-G' in args:
|
||||
return subprocess.CompletedProcess(args, 0, 'hostname 192.0.2.2\nport 2222\n', '')
|
||||
command = shlex.split(args[-1])
|
||||
script = command[-1]
|
||||
if 'ssh_host_' in script:
|
||||
self.assertEqual(command[:3], ['sudo', '-n', '--'])
|
||||
tools = ('id() { printf 0; }; dpkg() { printf amd64; }; '
|
||||
"cat() { printf 'ssh-ed25519 AAAA fixture\\n'; }; ")
|
||||
script = script.replace('/etc/os-release', shlex.quote(str(release)))
|
||||
result = subprocess.run(['sh', '-c', tools + script], capture_output=True, env=environment)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
return result
|
||||
if 'SSH_CONNECTION' in script:
|
||||
self.assertEqual(command[:2], ['sh', '-c'])
|
||||
fallbacks.append(script)
|
||||
return subprocess.run(command, capture_output=True,
|
||||
env=dict(environment, SSH_CONNECTION='192.0.2.1 50000 192.0.2.2 2222'))
|
||||
self.assertIn('/var/lib/apex-init/remote.json', script)
|
||||
self.assertIn('ssh-ed25519 AAAA', transport.trust_file.read_text())
|
||||
raise PreflightComplete
|
||||
transport = SSH('adam@example.com', runner=runner)
|
||||
with patch('provisioning.transport.SSH', return_value=transport), \
|
||||
patch('provisioning.transport.build_bundle', return_value=(b'bundle', 'a' * 64, 'b' * 40)), \
|
||||
patch('provisioning.transport.shutil.which', return_value='/usr/bin/ssh'):
|
||||
with self.assertRaises(PreflightComplete):
|
||||
initialize(root, 'adam@example.com', source, progress=lambda message: None)
|
||||
self.assertEqual(len(fallbacks), 1)
|
||||
|
||||
def test_remote_script_quoting_transports_literal_values(self):
|
||||
received = []
|
||||
def runner(args, **kwargs):
|
||||
command = shlex.split(args[-1])
|
||||
self.assertEqual(command[:2], ['sh', '-c'])
|
||||
result = subprocess.run(command, input=kwargs.get('input'), capture_output=True)
|
||||
received.append(result.stdout)
|
||||
return result
|
||||
value = "spaces '$HOME' `echo unsafe`; $(echo unsafe)"
|
||||
SSH('root@host', runner=runner).run('printf %s ' + shlex.quote(value))
|
||||
self.assertEqual(received, [value.encode()])
|
||||
|
||||
def test_secret_bytes_go_over_stdin_and_diagnostics_are_redacted(self):
|
||||
secret = b'private key material'
|
||||
def runner(args, **kwargs):
|
||||
self.assertEqual(kwargs['input'], secret)
|
||||
self.assertNotIn(secret.decode(), repr(args))
|
||||
return subprocess.CompletedProcess(args, 1, b'', secret)
|
||||
with self.assertRaises(TransportError) as caught:
|
||||
SSH('root@host', runner=runner).run('cat >/dev/null', secret)
|
||||
self.assertNotIn(secret.decode(), str(caught.exception))
|
||||
|
||||
def test_fresh_login_disables_mux_password_and_pins_verified_identity(self):
|
||||
calls = []
|
||||
def runner(args, **kwargs):
|
||||
calls.append(args)
|
||||
return subprocess.CompletedProcess(args, 0, b'', b'')
|
||||
ssh = SSH('root@original-alias', runner=runner)
|
||||
ssh.resolved_host = '192.0.2.42'
|
||||
ssh.trust_file = Path('/tmp/trusted')
|
||||
ssh.fresh_login()
|
||||
args = calls[0]
|
||||
self.assertIn('-T', args)
|
||||
for option in ('ControlMaster=no', 'ControlPath=none', 'BatchMode=yes',
|
||||
'PreferredAuthentications=publickey', 'StrictHostKeyChecking=yes',
|
||||
'Hostname=192.0.2.42', 'HostKeyAlias=apex-init-verified-target'):
|
||||
self.assertIn(option, args)
|
||||
self.assertEqual(args[args.index('-l') + 1], 'adam')
|
||||
self.assertEqual(args[args.index('-p') + 1], '2222')
|
||||
|
||||
def test_failed_first_fresh_login_rolls_back_without_finalizing(self):
|
||||
calls = []
|
||||
class FailedSSH:
|
||||
def fresh_login(self):
|
||||
raise TransportError('key rejected')
|
||||
def invoke(operation, *args):
|
||||
calls.append(operation)
|
||||
return {'status': 'transition'}
|
||||
with self.assertRaises(TransportError):
|
||||
protected_handover(FailedSSH(), invoke, 'a' * 32, 22, '192.0.2.1')
|
||||
self.assertEqual(calls, ['begin', 'rollback'])
|
||||
|
||||
def test_failed_second_fresh_login_never_commits(self):
|
||||
calls = []
|
||||
class SecondFailure:
|
||||
count = 0
|
||||
def fresh_login(self):
|
||||
self.count += 1
|
||||
if self.count == 2:
|
||||
raise TransportError('final policy rejected login')
|
||||
def invoke(operation, *args):
|
||||
calls.append(operation)
|
||||
return {'status': 'transition'}
|
||||
with self.assertRaises(TransportError):
|
||||
protected_handover(SecondFailure(), invoke, 'a' * 32, 22, '192.0.2.1')
|
||||
self.assertEqual(calls, ['begin', 'finalize', 'rollback'])
|
||||
|
||||
def test_two_successful_fresh_logins_precede_commit(self):
|
||||
calls = []
|
||||
class VerifiedSSH:
|
||||
def fresh_login(self):
|
||||
calls.append('fresh')
|
||||
def invoke(operation, *args):
|
||||
calls.append(operation)
|
||||
return {'status': 'transition'}
|
||||
protected_handover(VerifiedSSH(), invoke, 'a' * 32, 22, '192.0.2.1')
|
||||
self.assertEqual(calls, ['begin', 'fresh', 'finalize', 'fresh', 'commit'])
|
||||
|
||||
def test_lost_commit_reply_is_reconciled_without_rolling_back_committed_access(self):
|
||||
from provisioning.access import Transition
|
||||
from test_provision_access import FakeHost
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
host = FakeHost(Path(directory))
|
||||
transition = Transition(host, now=lambda: 1000, security=host.security)
|
||||
calls = []
|
||||
class VerifiedSSH:
|
||||
def fresh_login(self):
|
||||
pass
|
||||
def invoke(operation, token, port, address, fresh):
|
||||
calls.append(operation)
|
||||
if operation == 'status':
|
||||
self.assertTrue(fresh)
|
||||
return transition.load()
|
||||
result = transition.execute(operation, token, port, address)
|
||||
if operation == 'commit':
|
||||
raise TransportError('SSH reply lost')
|
||||
return result
|
||||
protected_handover(VerifiedSSH(), invoke, 'a' * 32, 22, '192.0.2.1')
|
||||
self.assertEqual(transition.load()['status'], 'committed')
|
||||
self.assertEqual(host.ports, [2222])
|
||||
self.assertFalse(host.armed)
|
||||
self.assertEqual(host.restored, 0)
|
||||
self.assertEqual(calls, ['begin', 'finalize', 'commit', 'status'])
|
||||
|
||||
def test_known_uncommitted_lost_reply_still_rolls_back(self):
|
||||
calls = []
|
||||
class VerifiedSSH:
|
||||
def fresh_login(self):
|
||||
pass
|
||||
def invoke(operation, *args):
|
||||
calls.append(operation)
|
||||
if operation == 'commit':
|
||||
raise TransportError('SSH failed before commit')
|
||||
return {'status': 'final' if operation == 'status' else 'transition', 'token': 'a' * 32}
|
||||
with self.assertRaises(TransportError):
|
||||
protected_handover(VerifiedSSH(), invoke, 'a' * 32, 22, '192.0.2.1')
|
||||
self.assertEqual(calls, ['begin', 'finalize', 'commit', 'status', 'rollback'])
|
||||
|
||||
def test_lost_commit_reply_reports_verified_or_uncertain_resume_endpoints(self):
|
||||
from provisioning.access import Transition
|
||||
from test_provision_access import FakeHost
|
||||
for unreachable in (False, True):
|
||||
with self.subTest(unreachable=unreachable), tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
source = root / 'user data $literal'
|
||||
source.write_bytes(b'opaque input')
|
||||
identity = root / 'identity file'
|
||||
identity.write_bytes(b'synthetic private identity')
|
||||
host = FakeHost(root)
|
||||
transition = Transition(host, now=lambda: 1000, security=host.security)
|
||||
messages, commands = [], []
|
||||
def runner(args, commands=commands, transition=transition,
|
||||
unreachable=unreachable, **kwargs):
|
||||
if '-G' in args:
|
||||
return subprocess.CompletedProcess(args, 0, 'hostname 192.0.2.2\nport 22\n', '')
|
||||
command = shlex.split(args[-1])
|
||||
script = command[-1]
|
||||
commands.append(script)
|
||||
value = b''
|
||||
if 'ssh_host_' in script:
|
||||
value = b'192.0.2.1 50000 192.0.2.2 22\nssh-ed25519 AAAA fixture\n'
|
||||
elif 'mktemp -d /var/tmp/apex-init.' in script:
|
||||
value = b'/var/tmp/apex-init.0123456789\n'
|
||||
elif script.startswith('if test -f /var/tmp'):
|
||||
value = b'0'
|
||||
elif script.startswith('cd ') and 'provisioning.access' in script:
|
||||
access_args = shlex.split(script)[3:]
|
||||
operation = access_args[3]
|
||||
token = access_args[access_args.index('--token') + 1]
|
||||
port = int(access_args[access_args.index('--original-port') + 1]) if '--original-port' in access_args else None
|
||||
address = access_args[access_args.index('--client-address') + 1] if port else None
|
||||
value = json.dumps(transition.execute(operation, token, port, address)).encode()
|
||||
if operation == 'commit':
|
||||
return subprocess.CompletedProcess(args, 255, b'', b'connection lost')
|
||||
elif 'cat /var/lib/apex-init/access/state.json' in script:
|
||||
self.assertEqual(args[args.index('-l') + 1], 'adam')
|
||||
self.assertEqual(args[args.index('-p') + 1], '2222')
|
||||
self.assertEqual(command[:3], ['sudo', '-n', '--'])
|
||||
if unreachable:
|
||||
return subprocess.CompletedProcess(args, 255, b'', b'connection lost')
|
||||
value = json.dumps(transition.load()).encode()
|
||||
elif 'provisioning.target complete' in script:
|
||||
value = b'{"status": "complete"}'
|
||||
return subprocess.CompletedProcess(args, 0, value, b'')
|
||||
def transport(*args):
|
||||
return SSH(*args, runner=runner)
|
||||
with patch('provisioning.transport.SSH', side_effect=transport), \
|
||||
patch('provisioning.transport.build_bundle', return_value=(b'bundle', 'a' * 64, 'b' * 40)), \
|
||||
patch('provisioning.transport.shutil.which', return_value='/usr/bin/ssh'):
|
||||
if unreachable:
|
||||
with self.assertRaises(HandoverUncertain):
|
||||
initialize(root, 'debian@example.com', source, identity_file=identity, progress=messages.append)
|
||||
self.assertTrue(any('outcome is unknown' in message for message in messages))
|
||||
self.assertIn('adam@example.com --port 2222', messages[-2])
|
||||
self.assertIn('debian@example.com --port 22', messages[-1])
|
||||
self.assertFalse(any('Safe resume' in message or 'pending rollback' in message for message in messages))
|
||||
else:
|
||||
self.assertEqual(initialize(root, 'debian@example.com', source, identity_file=identity, progress=messages.append), {'status': 'complete'})
|
||||
self.assertIn('Safe resume: python3 ci/scripts/init-node.py adam@example.com --port 2222', messages[-1])
|
||||
resume = shlex.split(messages[-1].split(': ', 1)[1])
|
||||
self.assertEqual(resume[:2], ['python3', 'ci/scripts/init-node.py'])
|
||||
self.assertEqual(resume[resume.index('--user-data') + 1], str(source.resolve()))
|
||||
self.assertEqual(resume[resume.index('--identity-file') + 1], str(identity.resolve()))
|
||||
self.assertEqual(transition.load()['status'], 'committed')
|
||||
self.assertFalse(host.armed)
|
||||
self.assertFalse(any('access rollback' in script for script in commands))
|
||||
|
||||
def test_bundle_excludes_repository_and_environment_secrets(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
(root / 'ci/scripts/init-node/provisioning').mkdir(parents=True)
|
||||
(root / 'ci/scripts/init-node/provisioning/__init__.py').write_text('')
|
||||
(root / 'ci/scripts/init-node/provisioning/seed.sh').write_text('#!/bin/sh\n')
|
||||
(root / '.env').write_text('private secret')
|
||||
(root / 'ci/scripts/init-node/provisioning/private.env').write_text('private secret')
|
||||
data, digest, revision = build_bundle(root)
|
||||
with tarfile.open(fileobj=io.BytesIO(data)) as archive:
|
||||
self.assertEqual(set(archive.getnames()), {'provisioning/__init__.py', 'provisioning/seed.sh'})
|
||||
self.assertNotIn(b'private secret', data)
|
||||
self.assertEqual(build_bundle(root)[1], digest)
|
||||
|
||||
def test_real_archive_runs_without_engine_or_site_packages(self):
|
||||
commons = Path(__file__).resolve().parents[4]
|
||||
data, digest, revision = build_bundle(commons)
|
||||
self.assertEqual(build_bundle(commons)[1], digest)
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
with tarfile.open(fileobj=io.BytesIO(data)) as archive:
|
||||
names = archive.getnames()
|
||||
self.assertTrue(names)
|
||||
self.assertTrue(all(name.startswith('provisioning/') for name in names))
|
||||
self.assertIn('provisioning/__init__.py', names)
|
||||
self.assertIn('provisioning/seed.sh', names)
|
||||
archive.extractall(directory, filter='data')
|
||||
environment = dict(os.environ)
|
||||
environment.pop('PYTHONPATH', None)
|
||||
for module in ('provisioning.target', 'provisioning.access', 'provisioning.image'):
|
||||
with self.subTest(module=module):
|
||||
result = subprocess.run([sys.executable, '-S', '-m', module, '--help'],
|
||||
cwd=directory, env=environment, capture_output=True,
|
||||
text=True, timeout=30)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertIn('usage:', result.stdout)
|
||||
|
||||
def test_bundle_rejects_symlinked_source(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
package = root / 'ci/scripts/init-node/provisioning'
|
||||
package.mkdir(parents=True)
|
||||
(package / '__init__.py').write_text('')
|
||||
(package / 'seed.sh').write_text('#!/bin/sh\n')
|
||||
outside = root / 'outside.py'
|
||||
outside.write_text('private = True\n')
|
||||
(package / 'target.py').symlink_to(outside)
|
||||
with self.assertRaisesRegex(TransportError, 'unsafe source path'):
|
||||
build_bundle(root)
|
||||
|
||||
def test_missing_input_is_a_safe_validation_error(self):
|
||||
with self.assertRaises(TransportError):
|
||||
local_file('/nonexistent/apex-user-data', 'User-data')
|
||||
|
||||
def test_standalone_help_runs_without_engine_or_yaml_imports(self):
|
||||
launcher = Path(__file__).resolve().parents[2] / 'init-node.py'
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
result = subprocess.run([sys.executable, '-S', str(launcher), '--help'],
|
||||
cwd=directory, capture_output=True, text=True, timeout=30)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertIn('--user-data', result.stdout)
|
||||
self.assertIn('--identity-file', result.stdout)
|
||||
|
||||
def test_dead_enrollment_unit_stops_polling_and_removes_input(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
source = root / 'user-data'
|
||||
source.write_bytes(b'opaque input')
|
||||
calls = []
|
||||
class Remote:
|
||||
user, host, identity = 'root', 'example.com', None
|
||||
def __init__(self, *args):
|
||||
self.resolved_port = 22
|
||||
def resolve(self):
|
||||
pass
|
||||
def run(self, script, data=None, **kwargs):
|
||||
calls.append(script)
|
||||
if 'ssh_host_' in script:
|
||||
value = b'192.0.2.1 50000 192.0.2.2 22\nssh-ed25519 AAAA fixture\n'
|
||||
elif 'mktemp -d /var/tmp/apex-init.' in script:
|
||||
value = b'/var/tmp/apex-init.0123456789\n'
|
||||
elif script.startswith('if test -f /var/tmp'):
|
||||
value = b'stopped'
|
||||
else:
|
||||
value = b''
|
||||
return subprocess.CompletedProcess([], 0, value, b'')
|
||||
messages = []
|
||||
with patch('provisioning.transport.SSH', Remote), \
|
||||
patch('provisioning.transport.build_bundle', return_value=(b'bundle', 'a' * 64, 'b' * 40)), \
|
||||
patch('provisioning.transport.shutil.which', return_value='/usr/bin/ssh'):
|
||||
with self.assertRaises(TransportError) as caught:
|
||||
initialize(root, 'root@example.com', source, progress=messages.append)
|
||||
self.assertIn('/runner.log', str(caught.exception))
|
||||
self.assertTrue(any(script.startswith('rm -f ') and '/user-data ' in script for script in calls))
|
||||
self.assertIn('root@example.com --port 22', messages[-1])
|
||||
|
||||
def test_completion_failure_keeps_administrator_resume_endpoint(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
source = root / 'user-data'
|
||||
source.write_bytes(b'opaque input')
|
||||
class Remote:
|
||||
user, host, identity = 'root', 'example.com', None
|
||||
def __init__(self, *args):
|
||||
self.resolved_port = 22
|
||||
def resolve(self):
|
||||
pass
|
||||
def run(self, script, data=None, **kwargs):
|
||||
if 'ssh_host_' in script:
|
||||
value = b'192.0.2.1 50000 192.0.2.2 22\nssh-ed25519 AAAA fixture\n'
|
||||
elif 'mktemp -d /var/tmp/apex-init.' in script:
|
||||
value = b'/var/tmp/apex-init.0123456789\n'
|
||||
elif script.startswith('if test -f /var/tmp'):
|
||||
value = b'0'
|
||||
elif 'provisioning.target complete' in script:
|
||||
raise TransportError('record failed')
|
||||
else:
|
||||
value = b''
|
||||
return subprocess.CompletedProcess([], 0, value, b'')
|
||||
messages = []
|
||||
with patch('provisioning.transport.SSH', Remote), \
|
||||
patch('provisioning.transport.build_bundle', return_value=(b'bundle', 'a' * 64, 'b' * 40)), \
|
||||
patch('provisioning.transport.protected_handover'), \
|
||||
patch('provisioning.transport.shutil.which', return_value='/usr/bin/ssh'):
|
||||
with self.assertRaises(TransportError):
|
||||
initialize(root, 'root@example.com', source, progress=messages.append)
|
||||
self.assertIn('adam@example.com --port 2222', messages[-1])
|
||||
|
||||
|
||||
class TargetPathTests(unittest.TestCase):
|
||||
"""Run the real shell guards with only Debian stat/mv translated on macOS."""
|
||||
def setUp(self):
|
||||
temporary = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(temporary.cleanup)
|
||||
self.root = Path(temporary.name).resolve()
|
||||
self.bin = self.root / 'bin'
|
||||
self.bin.mkdir()
|
||||
(self.bin / 'stat').write_text('#!' + sys.executable + '\n' + '''
|
||||
import os, stat, sys
|
||||
path = sys.argv[-1]
|
||||
info = os.lstat(path)
|
||||
field = sys.argv[sys.argv.index('-c') + 1]
|
||||
values = {'%a': format(stat.S_IMODE(info.st_mode), 'o'), '%h': str(info.st_nlink),
|
||||
'%u': '1000' if path == os.environ.get('TEST_NONROOT_PATH') else '0'}
|
||||
print(values[field])
|
||||
''')
|
||||
(self.bin / 'mv').write_text('#!' + sys.executable + '\nimport os, sys\nos.replace(sys.argv[-2], sys.argv[-1])\n')
|
||||
for command in self.bin.iterdir():
|
||||
command.chmod(0o755)
|
||||
self.environment = dict(os.environ, PATH=str(self.bin) + os.pathsep + os.environ['PATH'])
|
||||
|
||||
def write(self, path, data=b'new metadata'):
|
||||
return subprocess.run(['sh', '-c', remote_atomic_input(str(path))], input=data,
|
||||
capture_output=True, env=self.environment)
|
||||
|
||||
def test_atomic_write_ignores_predictable_old_temporary_symlink(self):
|
||||
destination = self.root / 'remote.json'
|
||||
victim = self.root / 'victim'
|
||||
victim.write_bytes(b'preserve me')
|
||||
destination.with_name('remote.json.new').symlink_to(victim)
|
||||
result = self.write(destination)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertEqual(destination.read_bytes(), b'new metadata')
|
||||
self.assertEqual(victim.read_bytes(), b'preserve me')
|
||||
self.assertEqual(destination.stat().st_mode & 0o777, 0o600)
|
||||
self.assertFalse(list(self.root.glob('.apex-write.*')))
|
||||
|
||||
def test_destination_symlink_is_rejected_without_touching_victim(self):
|
||||
victim = self.root / 'victim'
|
||||
victim.write_bytes(b'preserve me')
|
||||
destination = self.root / 'remote.json'
|
||||
destination.symlink_to(victim)
|
||||
result = self.write(destination)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertEqual(victim.read_bytes(), b'preserve me')
|
||||
|
||||
def test_redirected_parent_is_rejected_before_creating_files(self):
|
||||
victim = self.root / 'victim'
|
||||
victim.mkdir()
|
||||
redirected = self.root / 'apex-init'
|
||||
redirected.symlink_to(victim, target_is_directory=True)
|
||||
result = self.write(redirected / 'remote.json')
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertEqual(list(victim.iterdir()), [])
|
||||
|
||||
def test_unowned_parent_is_rejected_before_creating_files(self):
|
||||
state = self.root / 'apex-init'
|
||||
state.mkdir()
|
||||
self.environment['TEST_NONROOT_PATH'] = str(state)
|
||||
result = self.write(state / 'remote.json')
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertEqual(list(state.iterdir()), [])
|
||||
|
||||
def test_writable_ancestor_is_rejected_before_creating_files(self):
|
||||
state = self.root / 'apex-init'
|
||||
state.mkdir(mode=0o777)
|
||||
state.chmod(0o777)
|
||||
result = self.write(state / 'remote.json')
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertEqual(list(state.iterdir()), [])
|
||||
|
||||
def test_hardlinked_destination_is_rejected_without_touching_victim(self):
|
||||
victim = self.root / 'victim'
|
||||
victim.write_bytes(b'preserve me')
|
||||
destination = self.root / 'remote.json'
|
||||
os.link(victim, destination)
|
||||
result = self.write(destination)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertEqual(victim.read_bytes(), b'preserve me')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -19,10 +19,6 @@ select = ["E", "F", "W", "I", "B", "UP", "S"]
|
||||
# Subprocess arguments are constructed explicitly; no implicit shell execution.
|
||||
ignore = ["E401", "E501", "E702", "E741", "I001", "UP006", "UP035", "UP045", "S603", "S607"]
|
||||
|
||||
[tool.ruff.lint.per-file-ignores]
|
||||
# These tests use synthetic paths; no fixed temporary path is written.
|
||||
"ci/scripts/init-node/tests/test_provision_transport.py" = ["S108"]
|
||||
|
||||
[tool.mypy]
|
||||
python_version = "3.10"
|
||||
warn_return_any = true
|
||||
|
||||
Reference in new issue
Block a user