Split host setup into explicit entity actions

This commit is contained in:
deytenit committed 2026-11-07 17:15:49 +03:00
1 parent 9d733a8f5d
commit e452c981aa
51 files changed
+1023 -275

No files matched your search

+7 -7
View File
@@ -13,7 +13,7 @@ This skill guides the safe, structured update of container images, compose defin
2. **Hold critical services**: Always ask the user which services or compositions must NOT be touched before discovering or updating images.
3. **Classify update scope**: Separate updates into **Minor/Patch** (non-breaking), **Major** (breaking / schema migration risks), and **Unstable SemVer / CalVer** (requiring changelog scrutiny).
4. **Gate major and breaking updates**: Never apply a Major version bump, SemVer `0.x` minor change, or CalVer jump without migration research and explicit user approval.
5. **Preserve APEX engine context**: Always orchestrate compose actions through `./commons/apex compose` (or with derived `APEX_*` variables and layered env files) rather than bare `docker compose`.
5. **Preserve APEX engine context**: Always orchestrate compose actions through `./commons/apex composition/run` (or with derived `APEX_*` variables and layered env files) rather than bare `docker compose`.
6. **Safeguard in-tree patches**: Do not bump images for containers with host bind mounts that patch application source code without explicit review.
7. **Verify fleet observability post-rollout**: An update is not complete until containers reach `Up` and `healthy`, logs are clean, Prometheus scrape targets report `UP`, blackbox probes pass, and remote write streams are established.
@@ -145,7 +145,7 @@ For each node:
1. **Pre-upgrade backup**:
- Trigger a backup of stateful volumes:
```bash
ssh <node> "apex backup/run"
ssh <node> "apex tier/backup"
```
- For database services being updated, execute an in-container logical export to host storage.
2. **Update image tags in compose definitions**:
@@ -153,12 +153,12 @@ For each node:
3. **Pull and recreate services via APEX CLI**:
- Always run through the node's APEX CLI to preserve derived `APEX_*` variables and layered env files:
```bash
ssh <node> "cd /srv/<node_fqdn> && ./commons/apex compose pull"
ssh <node> "cd /srv/<node_fqdn> && ./commons/apex compose up -d"
ssh <node> "cd /srv/<node_fqdn> && ./commons/apex composition/run pull"
ssh <node> "cd /srv/<node_fqdn> && ./commons/apex composition/run up -d"
```
- *Note on `--force-recreate`*: When updating containers using floating tags, recovering from host reboots, or handling services with `tmpfs` state (e.g. mail spools), pass `--force-recreate` to ensure the container entrypoint runs a clean initialization:
```bash
ssh <node> "cd /srv/<node_fqdn> && ./commons/apex compose up -d --force-recreate"
ssh <node> "cd /srv/<node_fqdn> && ./commons/apex composition/run up -d --force-recreate"
```
4. **Completion check**: Compose commands finish with exit code 0.
@@ -195,7 +195,7 @@ When the update scope includes host operating system packages:
2. Reboot node: `ssh <node> "sudo reboot"`
3. Wait for SSH availability: poll connectivity with timeout.
4. After boot, verify systemd state (`systemctl is-system-running`) and check that Docker containers auto-started.
5. If services with `tmpfs` mounts fail startup checks (e.g. Postfix spool directories in mail containers), run `./commons/apex compose up -d --force-recreate`.
5. If services with `tmpfs` mounts fail startup checks (e.g. Postfix spool directories in mail containers), run `./commons/apex composition/run up -d --force-recreate`.
6. **Completion check**: Host packages are up to date and node is online.
---
@@ -229,6 +229,6 @@ Immediately after recreation and host reboots on each node:
- If a container enters a crash loop (`Restarting`, `Exit 1`) or breaks ingress:
1. Capture failure logs.
2. Revert the image tag in the compose file to the prior tag.
3. Re-run `./commons/apex compose up -d --force-recreate`.
3. Re-run `./commons/apex composition/run up -d --force-recreate`.
4. Notify user immediately with logs and root cause.
6. **Completion check**: All containers are confirmed `Up` and `healthy`, external ingress functions normally, and telemetry metrics/targets are green.
+36 -16
View File
@@ -15,6 +15,7 @@ Local infrastructure operations for independently owned nodes, with a Python sta
```sh
# Clone your node's repository (which pins apex as commons/)
cd /srv
git clone --recurse-submodules ssh://git@git.example.com:22222/infrastructure/selfhost.example.com.git
cd selfhost.example.com && ./init.sh && source ~/.bashrc
@@ -25,9 +26,13 @@ APEX_NODE_HOST=selfhost.example.com
APEX_SUBNET=198.18.0.0/24
EOF
# Configure host baseline and bring up compositions
apex configure # firewall, crowdsec, cron, systemd, routing
apex compose up # bring node services online
# Stock Debian Docker host, after configuring the node repository
apex config/packages
apex config/base
apex config/docker
apex config/ufw-docker
apex config/ufw # apply the node firewall rules
apex composition/run up # bring node services online
```
## Essential Information
@@ -43,7 +48,7 @@ You should understand the following invariants before exploring the codebase:
### Repository Structure
- **[`actions/`](./actions)**: Built-in engine actions (`compose.py`, `configure/`, `sync/`, `tiers/`, `backup/`, `utils/`).
- **[`actions/`](./actions)**: Built-in engine actions (`composition/`, `config/`, `repository/`, `tier/`, `util/`).
- **[`compositions/`](./compositions)**: The shared core Compose stack (`traefik`, `socket-proxy`, `alloy`, `adguard`, `xray`, `restic`).
- **[`engine/`](./engine)**: Framework engine — action discovery, overlay resolution, context injection, and logging.
- **[`etc/`](./etc)**: Shared static configuration, including Alloy.
@@ -60,20 +65,21 @@ Running and maintaining services on an existing host.
- **Requirements**: Debian-based host with Docker, Docker Compose v2, Git, and Python 3.
- **Workflow**:
- Update services: `apex compose pull && apex compose up`
- Run backups: `apex backup/run`
- Sync repository state: `apex sync/repository`
- Update services: `apex composition/run pull && apex composition/run up`
- Run backups: `apex tier/backup`
- Sync repository state: `apex repository/sync`
Target a composition explicitly when operating on one application:
```sh
apex compose -p apex restart traefik
apex compose -p app up -d
apex compose --dry-run -p apex logs --tail 20 xray
apex composition/run -p apex restart traefik
apex composition/run -p app up -d
apex composition/run --dry-run -p apex logs --tail 20 xray
```
The existing `apex compose` spelling is an alias for `apex composition/run`.
Wrapper options precede the action; remaining arguments go to Docker Compose.
The positional forms `apex compose restart apex traefik` and `apex compose up -d apex`
The positional forms `apex composition/run restart apex traefik` and `apex composition/run up -d apex`
remain supported. Use `-p` when service names or option values could be mistaken
for composition names. Without a target, APEX visits core first, then other
compositions alphabetically; `down` reverses that order.
@@ -84,7 +90,7 @@ after changing a container definition; `restart` retains the existing definition
For direct Docker Compose use in the core composition:
```sh
apex env --sync
apex config/env --sync
cd compositions/apex
docker compose config --quiet
docker compose restart traefik
@@ -96,12 +102,27 @@ profiles) followed by synthesized node/path values, and writes with mode `0600`.
Resync after changing `apex.env`, node identity, or tier paths. Shell environment
values still take precedence in Docker Compose. Do not edit the generated block.
`eval "$(apex env)"` exports synthesized APEX values to the current shell;
`apex env --dotenv` prints dotenv syntax, and `apex env --write generated.env`
`eval "$(apex config/env)"` exports synthesized APEX values to the current shell;
`apex config/env --dotenv` prints dotenv syntax, and `apex config/env --write generated.env`
writes those values to a separate file. These three modes do not merge the
composition's secrets or `apex.env`; use `--sync` for the complete core bridge.
Choose one output mode per invocation.
### Preparing an existing host
Run each setup action explicitly from the node checkout. Use `apex --help` to list the shared and node-owned actions. New actions follow `<singular-entity>/<action>`. Use `apex <entity>/<action> --help` for the arguments.
On an installed Proxmox VE host, `config/packages` installs the administrator tools and `config/base` applies APT and journald policy. The PVE host keeps its own storage, networking, firewall, and management listeners. Run workload configuration inside the guest that owns the service.
To set up the operator on either platform, start with an account that can administer the host:
```sh
apex config/adam --public-key-file /path/to/operator.pub
apex repository/chown
```
`config/adam` grants passwordless sudo. `repository/chown` gives adam ownership of the tracked checkout directly under `/srv` and its Git metadata. Open a new SSH connection as adam with the operator key and run `./commons/apex config/hardening` from the checkout in that session. An independent administrator login can also run the action after the operator’s authorized key and sudo access are configured. It applies the SSH public-key policy for adam and preserves the platform's existing management ports and other logins.
### Infrastructure Administrator
Build Debian 13 amd64 golden images with Packer, then supply node identity,
@@ -115,8 +136,7 @@ packer validate ci/packer/apex-debian-13.pkr.hcl
packer build ci/packer/apex-debian-13.pkr.hcl
```
Cloud-init runs the image's bundled enrollment helper. After enrollment, run
`apex configure` and `apex compose up` on the node.
Cloud-init runs the image's bundled enrollment helper. After enrollment, apply the individual `config/*` actions required by the node, then run `apex composition/run up`.
### Developer & Contributor
@@ -1,8 +1,8 @@
# actions/compose.py
# actions/composition/run.py
"""Orchestrate docker compose across the node.
Supports targeting all compositions in sequence (default: apex core first, services next;
reverse on down) or targeting one with `apex compose -p apex restart traefik`.
reverse on down) or targeting one with `apex composition/run -p apex restart traefik`.
Wrapper options precede the action. Exports derived APEX_* vars and layers env files.
"""
import os
+54
View File
@@ -0,0 +1,54 @@
"""Provision the golden-image operator while preserving authorized keys."""
import pwd
import grp
from pathlib import Path
from engine.descriptor import Meta, Opt
METADATA = Meta(summary="Configure the adam operator, sudo access and optional SSH public key.",
args=[Opt("--public-key-file", "Append one OpenSSH public key from this file.")])
HOME = Path("/home/adam")
def run(ctx, args):
system = ctx.sys
key = None
if args.public_key_file:
key = Path(args.public_key_file).read_text().strip()
if len(key.splitlines()) != 1 or not key.startswith(("ssh-", "ecdsa-", "sk-")):
raise ValueError("Supply one OpenSSH public key.")
system.run(["ssh-keygen", "-lf", "-"], input=key + "\n", capture=True)
groups = ["sudo"]
try:
grp.getgrnam("docker")
except KeyError:
pass
else:
groups.append("docker")
try:
account = pwd.getpwnam("adam")
except KeyError:
if HOME.exists() or HOME.is_symlink():
raise RuntimeError("Resolve existing /home/adam before creating the account.") from None
system.sudo(["useradd", "--create-home", "--shell", "/bin/bash", "--user-group", "adam"])
account = pwd.getpwnam("adam")
if account.pw_uid < 1000 or Path(account.pw_dir) != HOME or HOME.resolve() != HOME:
raise RuntimeError("Existing adam account has incompatible UID or home.")
if Path(account.pw_dir).stat().st_uid != account.pw_uid:
raise RuntimeError("The operator must own /home/adam.")
system.sudo(["usermod", "--shell", "/bin/bash", "--append", "--groups", ",".join(groups), "adam"])
if key:
system.sudo(["runuser", "-u", "adam", "--", "sh", "-ec",
'umask 077; cd "$1"; mkdir -p .ssh; chmod 700 .ssh; '
'touch .ssh/authorized_keys; chmod 600 .ssh/authorized_keys; '
'IFS= read -r key; grep -Fxq -- "$key" .ssh/authorized_keys || '
'printf "\\n%s\\n" "$key" >> .ssh/authorized_keys', "sh", str(HOME)], input=key + "\n")
content = "adam ALL=(ALL:ALL) NOPASSWD: ALL\n"
system.sudo(["visudo", "-cf", "-"], input=content)
path = Path("/etc/sudoers.d/99-apex-adam")
if path.is_symlink():
raise RuntimeError(f"Refusing redirected sudo policy: {path}")
ctx.host.write_file(str(path), content, mode="0440", backup=False, sudo=True)
system.sudo(["visudo", "-c"])
system.sudo(["runuser", "-u", "adam", "--", "sudo", "-n", "true"])
File renamed without changes.
+40
View File
@@ -0,0 +1,40 @@
"""Apply the common APT archive and journald retention policy."""
from pathlib import Path
from engine.descriptor import Meta
METADATA = Meta(summary="Configure APT archive cleanup and journald retention.")
def _configure(ctx, path, legacy, content, restart=False):
for target in (path, legacy):
if target.is_symlink():
raise RuntimeError(f"Refusing redirected configuration: {target}")
previous = path.read_text() if path.exists() else None
backup = ctx.host.backup(str(legacy), sudo=True) if legacy.exists() else None
if previous == content and not backup:
return
if previous != content:
ctx.host.write_file(str(path), content, sudo=True)
if backup:
ctx.sys.sudo(["rm", "--", str(legacy)])
if restart:
try:
ctx.sys.sudo(["systemctl", "restart", "systemd-journald"])
except Exception:
if backup:
ctx.sys.sudo(["cp", "--preserve=all", "--", backup, str(legacy)])
if previous is None:
ctx.sys.sudo(["rm", "--", str(path)])
elif previous != content:
ctx.host.write_file(str(path), previous, backup=False, sudo=True)
raise
def run(ctx, args):
if not Path("/etc/debian_version").is_file():
raise RuntimeError("Debian required.")
apt = Path("/etc/apt/apt.conf.d")
journal = Path("/etc/systemd/journald.conf.d")
_configure(ctx, apt / "99-apex", apt / "01clean", 'APT::Keep-Downloaded-Packages "0";\n')
_configure(ctx, journal / "99-apex.conf", journal / "maxuse.conf",
"[Journal]\nSystemMaxUse=200M\n", restart=True)
@@ -1,5 +1,5 @@
# actions/configure/cron.py
"""Port of configure/cron — install crontab from configs/cron/crontab."""
# actions/config/cron.py
"""Port of config/cron — install crontab from configs/cron/crontab."""
import os
from engine.descriptor import Meta, Flag
@@ -1,5 +1,5 @@
# actions/configure/crowdsec.py
"""Port of configure/crowdsec — install, deploy /etc/crowdsec, collections, bouncer."""
# actions/config/crowdsec.py
"""Port of config/crowdsec — install, deploy /etc/crowdsec, collections, bouncer."""
import os
from engine.descriptor import Meta, Flag
+12
View File
@@ -0,0 +1,12 @@
"""Install a local Docker Engine, Compose and Buildx on Debian 13."""
from pathlib import Path
from engine.descriptor import Meta
METADATA = Meta(summary="Verify a working Docker toolchain or install it on stock Debian 13.")
def run(ctx, args):
provisioning = Path(ctx.paths.commons) / "ci/scripts/init-node"
ctx.sys.sudo(["python3", "-E", "-s", "-m", "provisioning.baseline", "--docker-policy"],
cwd=str(provisioning))
File renamed without changes.
+143
View File
@@ -0,0 +1,143 @@
"""Apply the image's public-key SSH policy to adam after a verified key login."""
import ipaddress
import json
import os
import pwd
import re
from pathlib import Path
from engine.descriptor import Meta
METADATA = Meta(summary="Require SSH public-key authentication for adam from an independent administrator or verified adam key session.")
CONFIG = "/etc/ssh/sshd_config"
POLICY = """
# APEX operator SSH policy
Match User adam
AuthenticationMethods publickey
PubkeyAuthentication yes
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitEmptyPasswords no
X11Forwarding no
"""
def _ancestors():
ancestors = {}
pid = os.getppid()
ticks = os.sysconf("SC_CLK_TCK")
while pid > 1:
# comm can contain spaces and parentheses; the remaining fields start at state.
fields = Path(f"/proc/{pid}/stat").read_text().rsplit(")", 1)[1].split()
ancestors[pid] = int(fields[19]) * 1_000_000 // ticks
pid = int(fields[1])
return ancestors
def _connection(ctx):
if pwd.getpwuid(os.getuid()).pw_name != "adam":
raise RuntimeError("Run config/hardening inside an SSH public-key session as adam.")
ancestors = _ancestors()
sockets = ctx.sys.sudo(["ss", "-Hntp", "state", "established"], capture=True).stdout
connections = set()
for line in sockets.splitlines():
fields = line.split(None, 4)
if len(fields) == 5 and any(int(pid) in ancestors for pid in
re.findall(r'"sshd(?:-session)?",pid=(\d+),', fields[4])):
connections.add((fields[2], fields[3]))
if len(connections) != 1:
raise RuntimeError("Cannot identify a unique established SSH socket for this process.")
local, peer = connections.pop()
peer, peer_port = peer.rsplit(":", 1)
local, local_port = local.rsplit(":", 1)
def address(value):
ip = ipaddress.ip_address(value.strip("[]"))
return str(getattr(ip, "ipv4_mapped", None) or ip)
peer, local = address(peer), address(local)
record = rf"^Accepted publickey for adam from {re.escape(peer)} port {int(peer_port)} ssh2:"
result = ctx.sys.sudo(["journalctl", "-b", "_UID=0", "_EXE=/usr/sbin/sshd",
"_EXE=/usr/lib/openssh/sshd-session", "_EXE=/usr/lib/openssh/sshd-auth",
"--no-pager", "-o", "json", "--grep", record], capture=True, check=False)
if not result.returncode:
for line in result.stdout.splitlines():
entry = json.loads(line)
pid = int(entry["_PID"])
if (pid in ancestors and int(entry["__MONOTONIC_TIMESTAMP"]) >= ancestors[pid]
and re.match(record, entry["MESSAGE"])):
ctx.sys.run(["sudo", "-n", "true"])
return f"user=adam,host={peer},addr={peer},laddr={local},lport={int(local_port)}"
raise RuntimeError("Cannot verify public-key login for this SSH process in the root sshd journal.")
def _operator_access(ctx):
account = pwd.getpwnam("adam")
if account.pw_uid < 1000 or account.pw_dir != "/home/adam":
raise RuntimeError("Configure a regular adam operator account first.")
if pwd.getpwuid(os.getuid()).pw_name == "adam":
return _connection(ctx)
loginuid = int(Path("/proc/self/loginuid").read_text())
if loginuid in (account.pw_uid, 4294967295) or loginuid < 0:
raise RuntimeError("Use an independent administrator login or an SSH public-key session as adam.")
if os.getuid() not in (0, loginuid):
raise RuntimeError("Run from the independent administrator's own login.")
ctx.sys.sudo(["true"])
home = Path(account.pw_dir)
authorized = home / ".ssh" / "authorized_keys"
# Inspect as adam so private .ssh directories remain usable from another administrator.
command = ["runuser", "-u", "adam", "--"]
for path in (home, authorized.parent, authorized):
metadata = ctx.sys.sudo([*command, "env", "LC_ALL=C", "stat", "-c", "%u %a %F", "--", str(path)],
capture=True).stdout.strip().split(" ", 2)
owner, mode, kind = metadata
expected = "regular file" if path == authorized else "directory"
if int(owner) not in (0, account.pw_uid) or int(mode, 8) & 0o022 or kind != expected:
raise RuntimeError(f"Unsafe operator SSH path: {path}")
keys = ctx.sys.sudo([*command, "cat", "--", str(authorized)], capture=True).stdout
for key in keys.splitlines():
if key.startswith(("ssh-", "ecdsa-", "sk-")) and ctx.sys.run(
["ssh-keygen", "-lf", "-"], input=key + "\n", capture=True, check=False,
).returncode == 0:
break
else:
raise RuntimeError("Install a valid unrestricted operator public key before applying SSH hardening.")
ctx.sys.sudo([*command, "sudo", "-n", "true"])
connection = "user=adam,host=localhost,addr=127.0.0.1"
paths = _effective(ctx.sys, connection).get("authorizedkeysfile", "").split()
if not {".ssh/authorized_keys", "%h/.ssh/authorized_keys", str(authorized)}.intersection(paths):
raise RuntimeError("sshd does not use the operator's verified authorized_keys file.")
return connection
def _effective(system, connection):
result = system.sudo(["/usr/sbin/sshd", "-T", "-C", connection], capture=True)
return dict(line.split(" ", 1) for line in result.stdout.splitlines() if " " in line)
def run(ctx, args):
connection = _operator_access(ctx)
system = ctx.sys
system.sudo(["/usr/sbin/sshd", "-t"])
effective = _effective(system, connection)
if effective.get("authenticationmethods") not in ("any", "publickey"):
raise RuntimeError("Preserve the existing SSH authentication chain; reconcile it explicitly.")
required = {"authenticationmethods": "publickey", "pubkeyauthentication": "yes",
"passwordauthentication": "no", "kbdinteractiveauthentication": "no",
"permitemptypasswords": "no", "x11forwarding": "no"}
if all(effective.get(key) == value for key, value in required.items()):
return
if Path(CONFIG).is_symlink():
raise RuntimeError(f"Refusing redirected SSH configuration: {CONFIG}")
original = system.sudo(["cat", CONFIG], capture=True).stdout
if "# APEX operator SSH policy" in original:
raise RuntimeError("The managed SSH policy is ineffective; reconcile earlier directives.")
ctx.host.write_file(CONFIG, original + "\n" + POLICY, sudo=True)
try:
system.sudo(["/usr/sbin/sshd", "-t"])
effective = _effective(system, connection)
if any(effective.get(key) != value for key, value in required.items()):
raise RuntimeError("The effective SSH policy differs; inspect existing Match blocks.")
system.reload("ssh")
except BaseException:
ctx.host.write_file(CONFIG, original, backup=False, sudo=True)
system.sudo(["/usr/sbin/sshd", "-t"])
system.reload("ssh")
raise
+50
View File
@@ -0,0 +1,50 @@
"""Install common Debian administrator tools without replacing host packages."""
import re
from pathlib import Path
from engine.descriptor import Meta
METADATA = Meta(summary="Install Debian administrator packages and enable rsyslog.")
BASE_PACKAGES = ("curl", "wget", "git", "rsyslog", "ufw")
ADMIN_PACKAGES = (
"tcpdump", "socat", "rsync", "bind9-dnsutils", "iproute2", "net-tools",
"traceroute", "ca-certificates", "jq", "sed", "gawk", "tar", "gzip",
"bzip2", "xz-utils", "unzip", "htop", "iotop", "strace", "lsof",
"sysstat", "parted", "fdisk", "pciutils", "usbutils", "smartmontools",
"tmux", "vim", "less", "tree", "bash-completion", "python3",
"python3-minimal", "python3-yaml", "sudo", "cron", "iptables", "ipset",
"debian-archive-keyring", "kmod",
)
def _install_missing(s, packages):
missing = []
for package in packages:
result = s.run(["dpkg-query", "-W", "-f=${Status}", package],
check=False, capture=True)
if result.returncode or result.stdout.split()[1:] != ["ok", "installed"]:
missing.append(package)
if not missing:
return
apt = ["env", "LC_ALL=C", "DEBIAN_FRONTEND=noninteractive", "apt-get",
"-o", "DPkg::Lock::Timeout=60", "-o", "Acquire::Retries=2",
"-o", "Acquire::http::Timeout=30", "-o", "Acquire::https::Timeout=30"]
s.sudo([*apt, "update", "--error-on=any"])
command = [*apt, "install", "-y", "--no-install-recommends", "--no-upgrade",
"--no-remove", *missing]
plan = s.sudo([*command, "--simulate"], capture=True)
if any(line.startswith("Remv ") or re.match(r"^Inst \S+ \[", line)
for line in plan.stdout.splitlines()):
raise RuntimeError("Missing packages require upgrades/removals; reconcile packages first.")
s.sudo(command)
def run(ctx, args):
if not Path("/etc/debian_version").is_file():
raise RuntimeError("Debian required.")
packages = (*BASE_PACKAGES, *ADMIN_PACKAGES)
if Path("/usr/bin/pveversion").exists():
packages = tuple(package for package in packages if package != "ufw")
_install_missing(ctx.sys, packages)
if not ctx.sys.start_and_enable("rsyslog"):
raise RuntimeError("rsyslog failed to start.")
@@ -1,5 +1,5 @@
# actions/configure/routing.py
"""Port of configure/routing — idempotent node routing (ip_forward, fwmark rule, table 100)."""
# actions/config/routing.py
"""Port of config/routing — idempotent node routing (ip_forward, fwmark rule, table 100)."""
from engine.descriptor import Meta
METADATA = Meta(summary="Idempotent node-level routing rules and routes.")
@@ -1,5 +1,5 @@
# actions/configure/systemd.py
"""Port of configure/systemd — deploy + enable units from configs/systemd."""
# actions/config/systemd.py
"""Port of config/systemd — deploy + enable units from configs/systemd."""
import os
from engine.descriptor import Meta, Flag
File renamed without changes.
+65
View File
@@ -0,0 +1,65 @@
"""Install the pinned ufw-docker integration on Docker hosts."""
import hashlib
import os
from pathlib import Path
import stat
import urllib.request
from engine.descriptor import Meta
METADATA = Meta(summary="Install the pinned Docker integration for UFW.")
UFW_DOCKER_REVISION = "020a8699f95592561f254d8d4ad1bb40d401dfc7"
UFW_DOCKER_SHA256 = "643e56b080567c567b4aa28650196849b2a2da5dd0473fd3e5216b0886035ab0"
UFW_DOCKER_URL = (
"https://raw.githubusercontent.com/chaifeng/ufw-docker/"
+ UFW_DOCKER_REVISION + "/ufw-docker"
)
UFW_DOCKER_PATH = Path("/usr/local/bin/ufw-docker")
def _check_helper_path(path):
for parent in reversed(path.parents):
info = parent.lstat()
if (not stat.S_ISDIR(info.st_mode) or info.st_uid != 0 or
info.st_mode & 0o022):
raise RuntimeError(f"Unsafe ufw-docker ancestor: {parent}")
try:
info = path.lstat()
except FileNotFoundError:
return False
if (not stat.S_ISREG(info.st_mode) or info.st_uid != 0 or info.st_gid != 0 or
info.st_mode & 0o022 or info.st_nlink != 1):
raise RuntimeError(f"Unsafe ufw-docker target: {path}")
return True
def _install_helper(ctx):
path = UFW_DOCKER_PATH
exists = _check_helper_path(path)
payload = path.read_bytes() if exists else None
if payload is None or hashlib.sha256(payload).hexdigest() != UFW_DOCKER_SHA256:
ctx.log.info(f"Downloading pinned ufw-docker helper to {path}...")
# The immutable HTTPS URL is authenticated by the digest below.
with urllib.request.urlopen(UFW_DOCKER_URL, timeout=30) as response: # noqa: S310
payload = response.read(1024 * 1024 + 1)
if len(payload) > 1024 * 1024 or hashlib.sha256(payload).hexdigest() != UFW_DOCKER_SHA256:
raise RuntimeError("ufw-docker checksum mismatch; refusing installation.")
_check_helper_path(path)
ctx.host.write_file(str(path), payload.decode("utf-8"), mode="0755",
backup=False, sudo=True)
elif stat.S_IMODE(path.stat().st_mode) != 0o755:
ctx.host.write_file(str(path), payload.decode("utf-8"), mode="0755",
backup=False, sudo=True)
if (not _check_helper_path(path) or
hashlib.sha256(path.read_bytes()).hexdigest() != UFW_DOCKER_SHA256 or
stat.S_IMODE(path.stat().st_mode) != 0o755):
raise RuntimeError("ufw-docker verification failed; refusing execution.")
ctx.log.info("Ensuring ufw-docker routing rules are installed in UFW...")
ctx.sys.sudo([str(path), "install"])
def run(ctx, args):
_check_helper_path(UFW_DOCKER_PATH)
if os.path.exists("/usr/bin/pveversion"):
raise RuntimeError("Use the Proxmox firewall on a PVE host.")
for binary in ("ufw", "docker"):
if not ctx.sys.ok(["which", binary]):
raise RuntimeError(f"Install {binary} before configuring ufw-docker.")
_install_helper(ctx)
@@ -1,5 +1,5 @@
# actions/configure/ufw.py
"""Port of configure/ufw — apply docker + host ufw rules from configs/ufw."""
# actions/config/ufw.py
"""Port of config/ufw — apply docker + host ufw rules from configs/ufw."""
import os
from engine.descriptor import Meta, Flag
-34
View File
@@ -1,34 +0,0 @@
# actions/configure.py
"""Run every configure/* action in a stable order."""
import os
from engine.descriptor import Meta, Flag, build_parser
from engine import overlay
METADATA = Meta(summary="Run every configure/* action in stable order.",
args=[Flag("--dry-run", "Pass --dry-run to sub-actions that support it.")])
ORDER = ["configure/base", "configure/timezone", "configure/ufw", "configure/crowdsec",
"configure/cron", "configure/systemd", "configure/routing"]
def run(ctx, args):
commons_actions = os.path.join(ctx.paths.commons, "actions")
prop = os.path.join(ctx.paths.proprietaries, "actions")
prop = prop if os.path.isdir(prop) else None
names = [i.name for i in overlay.discover(commons_actions, prop) if i.name.startswith("configure/")]
ordered = [n for n in ORDER if n in names] + sorted(n for n in names if n not in ORDER)
rc = 0
for name in ordered:
r = overlay.resolve(name, commons_actions, prop)
mod = overlay.load_module(r.path, "apex_umbrella_" + name.replace("/", "_"))
if getattr(mod, "DISABLED", None):
ctx.log.warn(f"Skipping disabled {name}"); continue
meta = mod.METADATA
has_dry = any(getattr(a, "name", None) == "--dry-run" for a in meta.args)
argv = ["--dry-run"] if (args.dry_run and has_dry) else []
sub_args = build_parser(meta, f"apex {name}").parse_args(argv)
ctx.log.info(f"=== {name} ===")
try:
mod.run(ctx, sub_args)
except SystemExit as e:
if e.code:
ctx.log.error(f"{name} exited {e.code}"); rc = rc or e.code
return rc
-131
View File
@@ -1,131 +0,0 @@
# actions/configure/base.py
"""Port of configure/base — base packages + ufw + ufw-docker helper."""
import hashlib
import os
from pathlib import Path
import re
import stat
import urllib.request
from engine.descriptor import Meta
METADATA = Meta(summary="Configure base system packages (curl, wget, git, rsyslog, ufw, ufw-docker).")
UFW_DOCKER_REVISION = "020a8699f95592561f254d8d4ad1bb40d401dfc7"
UFW_DOCKER_SHA256 = "643e56b080567c567b4aa28650196849b2a2da5dd0473fd3e5216b0886035ab0"
UFW_DOCKER_URL = (
"https://raw.githubusercontent.com/chaifeng/ufw-docker/"
+ UFW_DOCKER_REVISION + "/ufw-docker"
)
UFW_DOCKER_PATH = Path("/usr/local/bin/ufw-docker")
BASE_PACKAGES = ("curl", "wget", "git", "rsyslog", "ufw")
def _install_missing(s, packages):
missing = []
for package in packages:
result = s.run(["dpkg-query", "-W", "-f=${Status}", package],
check=False, capture=True)
if result.returncode or result.stdout.split()[1:] != ["ok", "installed"]:
missing.append(package)
if not missing:
return
apt = ["env", "LC_ALL=C", "DEBIAN_FRONTEND=noninteractive", "apt-get",
"-o", "DPkg::Lock::Timeout=60", "-o", "Acquire::Retries=2",
"-o", "Acquire::http::Timeout=30", "-o", "Acquire::https::Timeout=30"]
s.sudo([*apt, "update", "--error-on=any"])
command = [*apt, "install", "-y", "--no-install-recommends", "--no-upgrade",
"--no-remove", *missing]
plan = s.sudo([*command, "--simulate"], capture=True)
if any(line.startswith("Remv ") or re.match(r"^Inst \S+ \[", line)
for line in plan.stdout.splitlines()):
raise RuntimeError("Missing packages require upgrades/removals; reconcile packages first.")
s.sudo(command)
def _check_helper_path(path):
for parent in reversed(path.parents):
info = parent.lstat()
if (not stat.S_ISDIR(info.st_mode) or info.st_uid != 0 or
info.st_mode & 0o022):
raise RuntimeError(f"Unsafe ufw-docker ancestor: {parent}")
try:
info = path.lstat()
except FileNotFoundError:
return False
if (not stat.S_ISREG(info.st_mode) or info.st_uid != 0 or info.st_gid != 0 or
info.st_mode & 0o022 or info.st_nlink != 1):
raise RuntimeError(f"Unsafe ufw-docker target: {path}")
return True
def _install_helper(ctx):
path = UFW_DOCKER_PATH
exists = _check_helper_path(path)
payload = path.read_bytes() if exists else None
if payload is None or hashlib.sha256(payload).hexdigest() != UFW_DOCKER_SHA256:
ctx.log.info(f"Downloading pinned ufw-docker helper to {path}...")
# The immutable HTTPS URL is authenticated by the digest below.
with urllib.request.urlopen(UFW_DOCKER_URL, timeout=30) as response: # noqa: S310
payload = response.read(1024 * 1024 + 1)
if len(payload) > 1024 * 1024 or hashlib.sha256(payload).hexdigest() != UFW_DOCKER_SHA256:
raise RuntimeError("ufw-docker checksum mismatch; refusing installation.")
_check_helper_path(path)
ctx.host.write_file(str(path), payload.decode("utf-8"), mode="0755",
backup=False, sudo=True)
elif stat.S_IMODE(path.stat().st_mode) != 0o755:
ctx.host.write_file(str(path), payload.decode("utf-8"), mode="0755",
backup=False, sudo=True)
if (not _check_helper_path(path) or
hashlib.sha256(path.read_bytes()).hexdigest() != UFW_DOCKER_SHA256 or
stat.S_IMODE(path.stat().st_mode) != 0o755):
raise RuntimeError("ufw-docker verification failed; refusing execution.")
ctx.log.info("Ensuring ufw-docker routing rules are installed in UFW...")
ctx.sys.sudo([str(path), "install"])
def run(ctx, args):
log, s = ctx.log, ctx.sys
log.info("Starting base system initialization...")
if not os.path.isfile("/etc/debian_version"):
log.error("Requires a Debian-based system (Debian/Ubuntu).")
raise SystemExit(1)
_check_helper_path(UFW_DOCKER_PATH)
_install_missing(s, BASE_PACKAGES)
s.ensure_running("rsyslog")
# Configure APT to discard downloaded archives
apt_clean_path = "/etc/apt/apt.conf.d/99-apex"
apt_clean_content = 'APT::Keep-Downloaded-Packages "0";\n'
if os.path.exists("/etc/apt/apt.conf.d/01clean"):
s.sudo(["rm", "-f", "/etc/apt/apt.conf.d/01clean"])
current_apt = None
if os.path.exists(apt_clean_path):
try:
with open(apt_clean_path, encoding="utf-8") as f:
current_apt = f.read()
except OSError:
current_apt = None
if current_apt != apt_clean_content:
log.info("Configuring APT to discard downloaded archives...")
s.sudo(["tee", apt_clean_path], input=apt_clean_content)
# Limit systemd-journald max disk usage
journal_d = "/etc/systemd/journald.conf.d"
journal_conf = os.path.join(journal_d, "99-apex.conf")
journal_content = "[Journal]\nSystemMaxUse=200M\n"
if os.path.exists(os.path.join(journal_d, "maxuse.conf")):
s.sudo(["rm", "-f", os.path.join(journal_d, "maxuse.conf")])
current_journal = None
if os.path.exists(journal_conf):
try:
with open(journal_conf, encoding="utf-8") as f:
current_journal = f.read()
except OSError:
current_journal = None
if current_journal != journal_content:
log.info("Configuring systemd-journald SystemMaxUse=200M...")
s.sudo(["mkdir", "-p", journal_d])
s.sudo(["tee", journal_conf], input=journal_content)
s.sudo(["systemctl", "restart", "systemd-journald"])
_install_helper(ctx)
log.success("Base system initialization completed.")
+62
View File
@@ -0,0 +1,62 @@
"""Give adam ownership of the active node's tracked checkout and Git metadata."""
import inspect
import json
from pathlib import Path
import re
from engine.descriptor import Meta
METADATA = Meta(summary="Set ownership of tracked files and Git metadata under /srv to adam.")
def _repair(root, tracked, mounts, uid, gid):
import os
import stat
directories = {""}
for item in tracked:
directory = os.path.dirname(item)
while directory and directory not in directories:
directories.add(directory)
directory = os.path.dirname(directory)
metadata = os.path.join(root, ".git")
def fail(error):
raise error
for parent, dirs, files, fd in os.fwalk(root, onerror=fail, follow_symlinks=False):
os.fchown(fd, uid, gid)
for name in dirs.copy() + files:
path = os.path.join(parent, name)
relative = os.path.relpath(path, root)
if path in mounts or not (relative in tracked or relative in directories or
path == metadata or path.startswith(metadata + "/")):
if name in dirs:
dirs.remove(name)
continue
info = os.stat(name, dir_fd=fd, follow_symlinks=False)
if stat.S_ISDIR(info.st_mode):
continue
if (stat.S_ISREG(info.st_mode) or stat.S_ISLNK(info.st_mode)) and info.st_nlink == 1:
os.chown(name, uid, gid, dir_fd=fd, follow_symlinks=False)
def run(ctx, args):
root = Path(ctx.paths.repo_root)
if (root.parent != Path("/srv") or root != root.resolve(strict=True) or
not (root / "node.env").is_file() or (root / ".git").is_symlink() or
not (root / ".git").is_dir()):
raise RuntimeError("Expected the current node checkout directly under /srv.")
git = ["git", "-c", f"safe.directory={root}", "-c", f"safe.directory={root}/commons",
"-C", str(root)]
entries = ctx.sys.run([*git, "ls-files", "--cached", "--recurse-submodules", "-z"],
capture=True).stdout
gitfiles = ctx.sys.run([*git, "submodule", "foreach", "--quiet", "--recursive",
'printf "%s\\0" "$displaypath/.git"'], capture=True).stdout
mounts = [re.sub(r"\\([0-7]{3})", lambda match: chr(int(match[1], 8)), line.split()[4])
for line in Path("/proc/self/mountinfo").read_text().splitlines()]
script = (inspect.getsource(_repair) + "\nimport json, pwd, sys\n"
"root, entries, mounts = json.load(sys.stdin)\n"
"adam = pwd.getpwnam('adam')\n"
"_repair(root, set(entries), set(mounts), adam.pw_uid, adam.pw_gid)")
ctx.sys.sudo(["python3", "-I", "-c", script],
input=json.dumps([str(root), (entries + gitfiles).split("\0"), mounts]))
File renamed without changes.
@@ -1,4 +1,4 @@
# actions/backup/run.py
# actions/tier/backup.py
"""Label-driven restic backup via resticontainer (git.ermnvldmr.com/deytenit/resticontainer).
Runs the resticontainer core service one-shot: init-if-needed -> backup (discovers
`restic.*`-labelled services, runs their pre-hooks/stops, snapshots the union of the
@@ -1,5 +1,5 @@
# actions/tiers/chown.py
"""Port of tiers/chown — ownership: {tier}/{proj} -> uid:gid; {tier}/shared/{proj} ->
# actions/tier/chown.py
"""Port of tier/chown — ownership: {tier}/{proj} -> uid:gid; {tier}/shared/{proj} ->
uid:shared_gid (+sticky); base/node-infra -> root:root. Reads APEX_UID/GID from each .env."""
import os
from engine.descriptor import Meta, Arg
@@ -31,7 +31,7 @@ def run(ctx, args):
cp = s.run(["getent", "group", SHARED_GROUP], check=False, capture=True)
if cp.returncode != 0:
log.error(f"Shared group '{SHARED_GROUP}' not found. Run tiers/useradd first."); raise SystemExit(1)
log.error(f"Shared group '{SHARED_GROUP}' not found. Run tier/useradd first."); raise SystemExit(1)
log.info("Ensuring base structures owned by root...")
for tp in tiers:
@@ -62,7 +62,7 @@ def run(ctx, args):
gid = _env_val(env_path, "APEX_GID")
sgid = _env_val(env_path, "APEX_SHARED_GID")
if not (uid and gid and sgid):
log.warn(f"Incomplete APEX_UID/GID/SHARED_GID for {proj}. Run tiers/useradd first. Skipping.")
log.warn(f"Incomplete APEX_UID/GID/SHARED_GID for {proj}. Run tier/useradd first. Skipping.")
continue
log.info(f"Applying permissions for {proj} (UID={uid} GID={gid} SHARED_GID={sgid})")
for tp in tiers:
@@ -1,5 +1,5 @@
# actions/tiers/link.py
"""Port of tiers/link — @tier symlink structure for the node + each composition project."""
# actions/tier/link.py
"""Port of tier/link — @tier symlink structure for the node + each composition project."""
import os
from engine.descriptor import Meta, Arg
@@ -1,5 +1,5 @@
# actions/tiers/useradd.py
"""Port of tiers/useradd — one noroot system user per composition project; inject the
# actions/tier/useradd.py
"""Port of tier/useradd — one noroot system user per composition project; inject the
APEX_UID/APEX_GID/APEX_SHARED_GID block into each project .env."""
import os, re
from engine.descriptor import Meta
@@ -36,7 +36,7 @@ def run(ctx, args):
continue
# opt-out: a project marked .apex-root-owned runs as root and keeps its tier
# data root-owned (e.g. seafile-mc, whose entrypoint refuses non-root-owned
# data). Skip the noroot user + .env UID injection; tiers/chown forces root.
# data). Skip the noroot user + .env UID injection; tier/chown forces root.
if os.path.isfile(os.path.join(full, ".apex-root-owned")):
log.info(f"{proj}: .apex-root-owned — runs as root; skipping noroot user")
continue
@@ -1,4 +1,4 @@
# actions/sync/packages.py
# actions/util/check-updates.py
"""Check pending APT and running-image updates without pulling images."""
import json
import os
@@ -1,5 +1,5 @@
# actions/utils/extract-traefik-certs.py
"""Port of utils/extract-traefik-certs — dump traefik v3 acme.json to pem via docker."""
# actions/util/extract-traefik-certs.py
"""Port of util/extract-traefik-certs — dump traefik v3 acme.json to pem via docker."""
import os
from engine.descriptor import Meta, Arg
@@ -1,4 +1,4 @@
# actions/utils/lint-docker-compose.py
# actions/util/lint-docker-compose.py
"""Run dclint through Docker and validate anchor IPs:
every ipv4_address in compositions/*/docker-compose.yml must be unique and inside the
node's /24."""
@@ -32,7 +32,7 @@ def run(ctx, args):
if k in os.environ:
env_args.extend(["-e", f"{k}={os.environ[k]}"])
# "-e commons": never --fix inside the pinned submodule — a dirtied commons makes
# sync/repository abort the nightly capture-up.
# repository/sync abort the nightly capture-up.
r = s.run(["docker", "run", "--rm", "--network", "host", *env_args, "-v", f"{repo}:/workdir", "-w", "/workdir", "node:lts-alpine",
"npx", "--yes", "dclint", "-e", "@tier1", "-e", "@tier2", "-e", "@tier3", "-e", ".env",
"-e", "commons", "--disable-rule", "no-build-and-image", "-r", "--fix", *targets], check=False)
+1 -1
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env bash
set -euo pipefail
# Service configuration and credentials are deployed later by apex configure/alloy.
# Service configuration and credentials are deployed later by apex config/alloy.
alloy_version=1.21.3
alloy_sha256=dbbd705de8ab37dec0cbd6361a3fc7e754d6b2854b57ce672a83e6366199ba73
[[ "$(dpkg --print-architecture)" == amd64 ]]
+96 -5
View File
@@ -91,12 +91,13 @@ def preflight(*, strict=True):
def package_versions():
architecture = run(["dpkg", "--print-architecture"]).stdout.strip()
result = run(["dpkg-query", "-W", "-f=${binary:Package}\t${Status}\t${Version}\n"])
installed = {}
for line in result.stdout.splitlines():
fields = line.split("\t")
if len(fields) == 3 and fields[1].split()[1:] == ["ok", "installed"]:
installed[fields[0].removesuffix(":amd64")] = fields[2]
installed[fields[0].removesuffix(":" + architecture)] = fields[2]
return installed
@@ -311,6 +312,88 @@ def docker_preflight(installed):
return {**config, **DOCKER_CONFIG}
DOCKER_KEY_SHA256 = "1500c1f56fa9e26b9b8f42452a553675796ade0807cdce11975eb98170b3a570"
LOCAL_DOCKER = ["docker", "--host", "unix:///var/run/docker.sock"]
def _docker_source(arch):
key = Path("/etc/apt/keyrings/apex-docker.asc")
source = Path("/etc/apt/sources.list.d/apex-docker.sources")
content = ("Types: deb\nURIs: https://download.docker.com/linux/debian\nSuites: trixie\n"
f"Components: stable\nArchitectures: {arch}\nSigned-By: {key}\n")
for path in (key, source):
if any(part.is_symlink() for part in (path, *path.parents)):
raise BaselineError(f"Refusing redirected APT path: {path}")
if source.exists() and source.read_text() != content:
raise BaselineError(f"Reconcile existing Docker repository configuration: {source}")
if key.exists():
data = key.read_bytes()
else:
url = "https://download.docker.com/linux/debian/gpg"
with urllib.request.urlopen(url, timeout=30) as response: # noqa: S310
data = response.read(1024 * 1024)
if hashlib.sha256(data).hexdigest() != DOCKER_KEY_SHA256:
raise BaselineError("Docker repository signing key differs from the pinned release key.")
write_managed(key, data)
write_managed(source, content)
def _docker_capabilities(root, *, complete_plugins=False):
capabilities = {"docker_data_root": root}
plugins = {"docker-compose-plugin": ("compose", ["compose", "version", "--short"]),
"docker-buildx-plugin": ("buildx", ["buildx", "version"])}
missing = []
for package, (name, args) in plugins.items():
result = run([*LOCAL_DOCKER, *args], check=False)
if result.returncode or not result.stdout.strip():
missing.append(package)
else:
capabilities[name] = result.stdout.strip()
if missing:
if complete_plugins and "docker-ce" in package_versions():
with service_start_guard():
install_missing(missing)
return _docker_capabilities(root)
raise BaselineError("Existing Docker needs functional plugins: " + ", ".join(missing))
if not root:
raise BaselineError("Unable to verify the local Docker data-root.")
return capabilities
@expected_errors
def apply_docker_policy():
if Path("/usr/bin/pveversion").exists():
raise BaselineError("Docker installation on a Proxmox host requires separate administration.")
release = dict(line.split("=", 1) for line in Path("/etc/os-release").read_text().splitlines()
if "=" in line and not line.startswith("#"))
arch = run(["dpkg", "--print-architecture"]).stdout.strip()
if (release.get("ID", "").strip('"') != "debian" or
release.get("VERSION_ID", "").strip('"') != "13" or arch not in ("amd64", "arm64")):
raise BaselineError("Docker installation supports stock Debian 13 amd64 or arm64.")
info = [*LOCAL_DOCKER, "info", "--format", "{{.DockerRootDir}}"]
if shutil.which("docker"):
current = run(info, check=False)
if current.returncode == 0:
return _docker_capabilities(current.stdout.strip(), complete_plugins=True)
installed = package_versions()
source = Path("/etc/apt/sources.list.d/apex-docker.sources")
if (not source.exists() and
(shutil.which("docker") or set(DOCKER_PACKAGES).intersection(installed))):
raise BaselineError("Existing Docker installation is inactive; reconcile it before retrying.")
config = docker_preflight(installed)
with service_start_guard():
install_missing(("ca-certificates",))
_docker_source(arch)
write_managed("/etc/docker/daemon.json", json.dumps(config, indent=2) + "\n")
install_missing(DOCKER_PACKAGES)
ensure_service("containerd")
ensure_service("docker")
root = run(info).stdout.strip()
if root != DOCKER_CONFIG["data-root"]:
raise BaselineError("Docker started with an unexpected data-root; inspect its configuration.")
return _docker_capabilities(root)
def configure_sources():
write_managed("/etc/apt/sources.list.d/apex-zfs.sources",
"Types: deb\nURIs: https://deb.debian.org/debian\nSuites: trixie\n"
@@ -418,7 +501,7 @@ def repair_bouncer_credentials():
@expected_errors
def apply_base_policy():
"""The bounded configure/base policy: no Docker, storage or enrollment."""
"""Apply the base package, logging, and firewall policy."""
preflight(strict=False)
with service_start_guard(), security_service_guard(BASE_PACKAGES):
install_missing(BASE_PACKAGES)
@@ -478,11 +561,19 @@ if __name__ == "__main__":
import argparse
from .state import ProvisionError, exclusive_lock
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--base-policy", action="store_true")
scope = parser.add_mutually_exclusive_group()
scope.add_argument("--base-policy", action="store_true")
scope.add_argument("--docker-policy", action="store_true")
args = parser.parse_args()
try:
preflight(strict=not args.base_policy)
preflight(strict=not (args.base_policy or args.docker_policy))
with exclusive_lock("/var/lib/apex-init/init.lock"):
print(json.dumps(apply_base_policy() if args.base_policy else apply_baseline(), sort_keys=True))
if args.docker_policy:
result = apply_docker_policy()
elif args.base_policy:
result = apply_base_policy()
else:
result = apply_baseline()
print(json.dumps(result, sort_keys=True))
except (BaselineError, OSError, ProvisionError) as exc:
parser.exit(1, f"Baseline failed: {exc}\n")
@@ -0,0 +1,120 @@
from contextlib import nullcontext
import hashlib
import io
import json
from pathlib import Path
from subprocess import CompletedProcess
import tempfile
import unittest
from unittest.mock import patch
from provisioning import baseline
class ManualDockerTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.root = Path(tmp.name).resolve()
(self.root / "etc").mkdir()
(self.root / "etc/os-release").write_text("ID=debian\nVERSION_ID=13\n")
self.commands = []
self.arch = "arm64"
self.running = False
self.data_root = "/custom/data"
self.missing_buildx = False
self.versions = {}
self.installations = []
def run_command(self, cmd, **kwargs):
self.commands.append(cmd)
if cmd == ["dpkg", "--print-architecture"]:
return CompletedProcess(cmd, 0, self.arch + "\n", "")
if cmd[-1] == "{{.DockerRootDir}}":
return CompletedProcess(cmd, 0 if self.running else 1, self.data_root + "\n", "")
if cmd[-2:] in (["version", "--short"], ["buildx", "version"]):
if self.missing_buildx and cmd[-2:] == ["buildx", "version"]:
if ["docker-buildx-plugin"] not in self.installations:
return CompletedProcess(cmd, 1, "", "plugin missing")
return CompletedProcess(cmd, 0, "installed\n", "")
raise AssertionError(cmd)
def apply(self):
def target(path):
path = Path(path)
return path if self.root in path.parents else self.root / str(path).lstrip("/")
with (patch.object(baseline, "Path", side_effect=target),
patch.object(baseline, "run", side_effect=self.run_command),
patch.object(baseline, "package_versions", return_value=self.versions),
patch.object(baseline.shutil, "which", return_value="/usr/bin/docker" if self.running else None),
patch.object(baseline, "install_missing", side_effect=self.installations.append)):
return baseline.apply_docker_policy()
def test_working_engine_keeps_local_data_root_without_writes(self):
self.running = True
self.versions = {"docker.io": "1"}
self.assertEqual(self.apply()["docker_data_root"], "/custom/data")
self.assertEqual(self.installations, [])
self.assertFalse((self.root / "etc/docker").exists())
def test_pve_refused_before_packages_or_daemon(self):
binary = self.root / "usr/bin/pveversion"
binary.parent.mkdir(parents=True)
binary.touch()
with self.assertRaisesRegex(baseline.BaselineError, "Proxmox"):
self.apply()
self.assertEqual(self.commands, [])
def test_unsupported_architecture_refused(self):
self.arch = "i386"
with self.assertRaisesRegex(baseline.BaselineError, "amd64|arm64"):
self.apply()
self.assertEqual(self.installations, [])
def test_existing_runtime_packages_not_replaced(self):
self.versions = {"containerd": "1.7"}
with self.assertRaisesRegex(baseline.BaselineError, "containerd"):
self.apply()
self.assertEqual(self.installations, [])
def test_fresh_arm64_install_verifies_the_pinned_key_and_starts_local_engine(self):
key = b"test repository signing key"
self.data_root = "/srv/docker"
def start(service):
if service == "docker":
self.running = True
with (patch.object(baseline, "service_start_guard", return_value=nullcontext()),
patch.object(baseline.urllib.request, "urlopen", return_value=io.BytesIO(key)),
patch.object(baseline, "DOCKER_KEY_SHA256", hashlib.sha256(key).hexdigest()),
patch.object(baseline, "ensure_service", side_effect=start)):
self.assertEqual(self.apply()["docker_data_root"], "/srv/docker")
source = (self.root / "etc/apt/sources.list.d/apex-docker.sources").read_text()
self.assertIn("Architectures: arm64", source)
self.assertIn("Signed-By: ", source)
self.assertEqual((self.root / "etc/apt/keyrings/apex-docker.asc").read_bytes(), key)
self.assertEqual(json.loads((self.root / "etc/docker/daemon.json").read_text())["data-root"],
"/srv/docker")
self.assertIn("docker-compose-plugin", self.installations[-1])
self.assertIn("docker-buildx-plugin", self.installations[-1])
self.assertNotIn("ufw", self.installations[-1])
def test_untrusted_signing_key_cannot_publish_apt_source_or_install_docker(self):
with (patch.object(baseline, "service_start_guard", return_value=nullcontext()),
patch.object(baseline.urllib.request, "urlopen", return_value=io.BytesIO(b"wrong key"))):
with self.assertRaisesRegex(baseline.BaselineError, "signing key"):
self.apply()
self.assertFalse((self.root / "etc/apt/sources.list.d/apex-docker.sources").exists())
self.assertFalse((self.root / "etc/docker/daemon.json").exists())
self.assertEqual(self.installations, [("ca-certificates",)])
def test_working_ce_engine_only_installs_the_missing_plugin(self):
self.running = True
self.missing_buildx = True
self.versions = {"docker-ce": "1"}
with patch.object(baseline, "service_start_guard", return_value=nullcontext()):
self.assertEqual(self.apply()["docker_data_root"], "/custom/data")
self.assertEqual(self.installations, [["docker-buildx-plugin"]])
self.assertFalse((self.root / "etc/docker").exists())
+3 -1
View File
@@ -28,7 +28,7 @@ def _resolve_layout():
def _print_list(commons_actions, prop_actions):
print("Usage: apex <group>/<action> [args...]\n\nAvailable actions:")
print("Usage: apex <entity>/<action> [args...]\n\nAvailable actions:")
tag = {"commons": "", "local": " (local)", "override": " (local override)"}
for i in overlay.discover(commons_actions, prop_actions):
mark = f" [DISABLED: {i.disabled}]" if i.disabled else ""
@@ -44,6 +44,8 @@ def main(argv) -> int:
return 0
name, rest = argv[0], argv[1:]
if name == "compose":
name = "composition/run"
r = overlay.resolve(name, commons_actions, prop_actions)
if r is None:
sys.stderr.write(f"[global] [core] [ERROR] Unknown action: {name}\n")
+1 -1
View File
@@ -1,5 +1,5 @@
# engine/lib/crowdsec.py
"""ctx.crowdsec — install + config deploy + collections (port of configure/crowdsec)."""
"""ctx.crowdsec — install + config deploy + collections (port of config/crowdsec)."""
from __future__ import annotations
import os
import shutil
+3 -3
View File
@@ -125,13 +125,13 @@ class Host:
# --- read/write with sudo awareness ---
def _read(self, path: str, sudo: bool = False) -> str:
if sudo:
cp = self.sys.run(["sudo", "cat", path], check=True, capture=True)
cp = self.sys.sudo(["cat", path], check=True, capture=True)
return cp.stdout or ""
with open(path, encoding="utf-8") as f:
return f.read()
def backup(self, path: str, sudo: bool = False) -> Optional[str]:
exists = (self.sys.ok(["sudo", "test", "-f", path]) if sudo
exists = (self.sys.sudo(["test", "-f", path], check=False).returncode == 0 if sudo
else os.path.isfile(path))
if not exists:
return None
@@ -154,7 +154,7 @@ class Host:
anchor: Optional[str] = None,
old_pairs: Sequence[Tuple[str, str]] = (),
dry_run: bool = False, sudo: bool = True) -> bool:
exists = (self.sys.ok(["sudo", "test", "-f", target]) if sudo
exists = (self.sys.sudo(["test", "-f", target], check=False).returncode == 0 if sudo
else os.path.isfile(target))
if not exists:
self.log.error(f"Target file does not exist: {target}")
+3
View File
@@ -1,6 +1,7 @@
# engine/lib/sys.py
"""ctx.sys — command execution + systemd service management (port of root_sys.sh)."""
from __future__ import annotations
import os
import shutil
import subprocess
import time
@@ -21,6 +22,8 @@ class Sys:
stderr=subprocess.PIPE if capture else None)
def sudo(self, cmd: Sequence[str], **kw) -> subprocess.CompletedProcess:
if os.geteuid() == 0 and shutil.which("sudo") is None:
return self.run(cmd, **kw)
return self.run(["sudo", *cmd], **kw)
def ok(self, cmd: Sequence[str], **kw) -> bool:
+2 -2
View File
@@ -67,10 +67,10 @@ class Ufw:
self.log.warn("Neither docker nor host ufw rules directory found; nothing to apply.")
return
if not self.sys.ok(["sudo", "ufw", "--version"]):
self.log.error("ufw not installed/working via sudo. Run configure/base first.")
self.log.error("ufw not installed/working via sudo. Run config/packages first.")
raise SystemExit(1)
if not (os.path.isfile(UFW_DOCKER_BIN) and os.access(UFW_DOCKER_BIN, os.X_OK)):
self.log.error(f"ufw-docker not found at {UFW_DOCKER_BIN}. Run configure/base first.")
self.log.error(f"ufw-docker not found at {UFW_DOCKER_BIN}. Run config/ufw-docker first.")
raise SystemExit(1)
if config_dir:
+7 -1
View File
@@ -9,10 +9,12 @@ from __future__ import annotations
import ast
import importlib.util
import os
import re
from dataclasses import dataclass
from typing import List, Optional
ACTION_EXT = ".py"
ACTION_NAME = re.compile(r"[A-Za-z0-9_-]+/[A-Za-z0-9_-]+")
@dataclass
@@ -40,11 +42,15 @@ def _rel_names(root: Optional[str]):
continue
full = os.path.join(dirpath, fn)
rel = os.path.relpath(full, root)[: -len(ACTION_EXT)]
out[rel.replace(os.sep, "/")] = full
name = rel.replace(os.sep, "/")
if ACTION_NAME.fullmatch(name):
out[name] = full
return out
def resolve(name: str, commons_root: str, prop_root: Optional[str]) -> Optional[Resolved]:
if not ACTION_NAME.fullmatch(name):
return None
rel = name + ACTION_EXT
commons_path = os.path.join(commons_root, rel)
c = os.path.isfile(commons_path)
+1 -1
View File
@@ -6,4 +6,4 @@ else
APEX_CMD="$(git rev-parse --show-toplevel)/commons/apex"
[[ -x "$APEX_CMD" ]] || APEX_CMD="$(git rev-parse --show-toplevel)/apex" # standalone commons repo
fi
"$APEX_CMD" utils/lint-docker-compose --hook || { echo "[pre-commit] Aborting: apex utils/lint-docker-compose failed."; exit 1; }
"$APEX_CMD" util/lint-docker-compose --hook || { echo "[pre-commit] Aborting: apex util/lint-docker-compose failed."; exit 1; }
+1 -1
View File
@@ -7,7 +7,7 @@ from unittest.mock import Mock
from engine import overlay
backup = overlay.load_module(str(Path(__file__).resolve().parents[1] / 'actions/backup/run.py'),
backup = overlay.load_module(str(Path(__file__).resolve().parents[1] / 'actions/tier/backup.py'),
'test_backup_action')
+16
View File
@@ -13,6 +13,22 @@ class TestCli(unittest.TestCase):
self.assertEqual(r.returncode, 0)
self.assertIn("Available actions", r.stdout)
def test_compose_alias_uses_canonical_action(self):
canonical = run("composition/run", "--help")
self.assertEqual(canonical.returncode, 0)
alias = run("compose", "--help")
self.assertEqual(alias.returncode, 0)
self.assertEqual(alias.stdout, canonical.stdout)
def test_configuration_must_be_selected_explicitly(self):
listing = run("--help")
self.assertIn("config/hardening", listing.stdout)
self.assertIn("repository/chown", listing.stdout)
for command in ("configure", "config"):
result = run(command)
self.assertEqual(result.returncode, 1)
self.assertIn("Unknown action", result.stderr)
def test_unknown_action_exits_one(self):
r = run("no/such")
self.assertEqual(r.returncode, 1)
+2 -2
View File
@@ -6,7 +6,7 @@ import unittest
from pathlib import Path
from unittest.mock import MagicMock
from actions import compose
from actions.composition import run as compose
from engine.descriptor import build_parser
@@ -25,7 +25,7 @@ class TestCompose(unittest.TestCase):
self.ctx.vars.return_value = {'APEX_NODE_HOST': 'test-host', 'OTHER': 'ignored'}
def run_compose(self, *argv):
args = build_parser(compose.METADATA, 'apex compose').parse_args(argv)
args = build_parser(compose.METADATA, 'apex composition/run').parse_args(argv)
compose.run(self.ctx, args)
def visited(self):
+1 -1
View File
@@ -7,7 +7,7 @@ import unittest
from unittest.mock import Mock, patch
from engine.lib.sys import Sys
spec = importlib.util.spec_from_file_location('compose_lint', Path(__file__).resolve().parents[1] / 'actions/utils/lint-docker-compose.py')
spec = importlib.util.spec_from_file_location('compose_lint', Path(__file__).resolve().parents[1] / 'actions/util/lint-docker-compose.py')
lint = importlib.util.module_from_spec(spec)
spec.loader.exec_module(lint)
+1 -1
View File
@@ -9,7 +9,7 @@ from unittest.mock import Mock
from engine import overlay
ROOT = Path(__file__).resolve().parents[1]
alloy = overlay.load_module(str(ROOT / 'actions/configure/alloy.py'), 'test_alloy_action')
alloy = overlay.load_module(str(ROOT / 'actions/config/alloy.py'), 'test_alloy_action')
class NativeAlloy(unittest.TestCase):
+29 -23
View File
@@ -16,16 +16,22 @@ from engine import overlay
from engine.lib.host import Host
from engine.lib.sys import Sys
base = overlay.load_module(
str(Path(__file__).resolve().parents[1] / "actions/configure/base.py"),
packages = overlay.load_module(
str(Path(__file__).resolve().parents[1] / "actions/config/packages.py"),
"apex_test_configure_base",
)
helper = overlay.load_module(
str(Path(__file__).resolve().parents[1] / "actions/config/ufw-docker.py"),
"apex_test_ufw_docker",
)
class TestBasePackages(unittest.TestCase):
def setUp(self):
self.system = Mock()
self.installed = set(base.BASE_PACKAGES)
self.installed = set(packages.BASE_PACKAGES)
self.system.run.side_effect = self.query
self.system.sudo.return_value = subprocess.CompletedProcess([], 0, stdout="")
@@ -35,7 +41,7 @@ class TestBasePackages(unittest.TestCase):
return subprocess.CompletedProcess(command, 0, stdout=status)
def test_installed_rsyslog_and_ufw_do_not_require_executable_or_apt_update(self):
base._install_missing(self.system, base.BASE_PACKAGES)
packages._install_missing(self.system, packages.BASE_PACKAGES)
self.assertEqual({call.args[0][-1] for call in self.system.run.call_args_list},
{"curl", "wget", "git", "rsyslog", "ufw"})
self.system.sudo.assert_not_called()
@@ -45,7 +51,7 @@ class TestBasePackages(unittest.TestCase):
self.system.sudo.return_value = subprocess.CompletedProcess(
[], 0, stdout="Inst ufw (0.36.2 Debian:13 [all])\nConf ufw (0.36.2 Debian:13 [all])\n",
)
base._install_missing(self.system, base.BASE_PACKAGES)
packages._install_missing(self.system, packages.BASE_PACKAGES)
commands = [call.args[0] for call in self.system.sudo.call_args_list]
self.assertEqual(len(commands), 3)
self.assertIn("update", commands[0])
@@ -79,7 +85,7 @@ class TestBasePackages(unittest.TestCase):
return subprocess.CompletedProcess(command, 0, stdout=plan)
with patch("engine.lib.sys.subprocess.run", side_effect=filtered_sudo):
base._install_missing(Sys(Mock()), ["ufw"])
packages._install_missing(Sys(Mock()), ["ufw"])
self.assertEqual(len(operations), 3)
self.assertIn("update", operations[0])
self.assertIn("--simulate", operations[1])
@@ -94,13 +100,13 @@ class TestBasePackages(unittest.TestCase):
self.system.sudo.reset_mock()
self.system.sudo.return_value = subprocess.CompletedProcess([], 0, stdout=plan)
with self.assertRaisesRegex(RuntimeError, "upgrades/removals"):
base._install_missing(self.system, base.BASE_PACKAGES)
packages._install_missing(self.system, packages.BASE_PACKAGES)
self.assertEqual(self.system.sudo.call_count, 2)
def test_failed_package_query_is_not_treated_as_installed(self):
self.system.run.return_value = subprocess.CompletedProcess([], 1, stdout="")
self.system.run.side_effect = None
base._install_missing(self.system, ["rsyslog"])
packages._install_missing(self.system, ["rsyslog"])
self.assertEqual(self.system.sudo.call_args.args[0][-1], "rsyslog")
@@ -131,9 +137,9 @@ class TestPinnedHelper(unittest.TestCase):
# Model root-owned ancestry without sudo/chown; content, modes and links stay real.
self.addCleanup(patch.stopall)
patch.object(Path, "lstat", autospec=True, side_effect=root_lstat).start()
patch.object(base, "UFW_DOCKER_PATH", self.path).start()
patch.object(base, "UFW_DOCKER_SHA256", hashlib.sha256(self.payload).hexdigest()).start()
self.download = patch.object(base.urllib.request, "urlopen").start()
patch.object(helper, "UFW_DOCKER_PATH", self.path).start()
patch.object(helper, "UFW_DOCKER_SHA256", hashlib.sha256(self.payload).hexdigest()).start()
self.download = patch.object(helper.urllib.request, "urlopen").start()
self.download.return_value = io.BytesIO(self.payload)
def sudo(self, command, **kwargs):
@@ -150,27 +156,27 @@ class TestPinnedHelper(unittest.TestCase):
def test_verified_existing_helper_executes_without_download(self):
self.write_helper()
base._install_helper(self.ctx)
helper._install_helper(self.ctx)
self.download.assert_not_called()
self.assertEqual(self.system.sudo.call_args.args[0], [str(self.path), "install"])
self.assertEqual(self.system.sudo.call_count, 1)
def test_wrong_executable_content_is_replaced_only_with_verified_download(self):
self.write_helper(b"#!/bin/sh\necho wrong\n")
base._install_helper(self.ctx)
helper._install_helper(self.ctx)
self.assertEqual(self.path.read_bytes(), self.payload)
self.assertEqual(self.system.sudo.call_count, 2)
self.download.assert_called_once_with(base.UFW_DOCKER_URL, timeout=30)
self.download.assert_called_once_with(helper.UFW_DOCKER_URL, timeout=30)
def test_missing_helper_is_published_with_executable_mode(self):
base._install_helper(self.ctx)
helper._install_helper(self.ctx)
self.assertEqual(self.path.read_bytes(), self.payload)
self.assertEqual(stat.S_IMODE(self.path.stat().st_mode), 0o755)
self.assertEqual(list(self.path.parent.iterdir()), [self.path])
def test_verified_nonexecutable_content_is_reused_and_mode_repaired(self):
self.write_helper(mode=0o600)
base._install_helper(self.ctx)
helper._install_helper(self.ctx)
self.download.assert_not_called()
self.assertEqual(stat.S_IMODE(self.path.stat().st_mode), 0o755)
self.assertEqual(self.system.sudo.call_count, 2)
@@ -178,7 +184,7 @@ class TestPinnedHelper(unittest.TestCase):
def test_bad_checksum_never_publishes_or_executes(self):
self.download.return_value = io.BytesIO(b"not the pinned helper")
with self.assertRaisesRegex(RuntimeError, "checksum mismatch"):
base._install_helper(self.ctx)
helper._install_helper(self.ctx)
self.system.sudo.assert_not_called()
self.assertFalse(self.path.exists())
@@ -186,13 +192,13 @@ class TestPinnedHelper(unittest.TestCase):
self.write_helper(b"old unknown executable")
self.download.return_value = io.BytesIO(b"bad download")
with self.assertRaisesRegex(RuntimeError, "checksum mismatch"):
base._install_helper(self.ctx)
helper._install_helper(self.ctx)
self.system.sudo.assert_not_called()
self.assertEqual(self.path.read_bytes(), b"old unknown executable")
def assert_rejected_before_network_or_sudo(self):
with self.assertRaisesRegex(RuntimeError, "Unsafe ufw-docker"):
base._install_helper(self.ctx)
helper._install_helper(self.ctx)
self.download.assert_not_called()
self.system.sudo.assert_not_called()
@@ -219,9 +225,9 @@ class TestPinnedHelper(unittest.TestCase):
def test_unsafe_helper_preflight_precedes_package_network_or_mutation(self):
self.path.parent.chmod(0o777)
with patch.object(base.os.path, "isfile", return_value=True):
with patch.object(helper.os.path, "isfile", return_value=True):
with self.assertRaisesRegex(RuntimeError, "Unsafe ufw-docker"):
base.run(self.ctx, None)
helper.run(self.ctx, None)
self.download.assert_not_called()
self.system.run.assert_not_called()
self.system.sudo.assert_not_called()
@@ -260,7 +266,7 @@ class TestPinnedHelper(unittest.TestCase):
self.download.side_effect = download
with self.assertRaisesRegex(RuntimeError, "Unsafe ufw-docker"):
base._install_helper(self.ctx)
helper._install_helper(self.ctx)
self.system.sudo.assert_not_called()
def test_changed_content_after_publication_is_not_executed(self):
@@ -273,7 +279,7 @@ class TestPinnedHelper(unittest.TestCase):
self.system.sudo.side_effect = tamper
with self.assertRaisesRegex(RuntimeError, "verification failed"):
base._install_helper(self.ctx)
helper._install_helper(self.ctx)
self.assertEqual(self.system.sudo.call_count, 1)
+2 -2
View File
@@ -10,7 +10,7 @@ import unittest
from pathlib import Path
from unittest.mock import MagicMock
from actions import env
from actions.config import env
from engine.descriptor import build_parser
from engine.lib.host import Host
@@ -31,7 +31,7 @@ class TestEnv(unittest.TestCase):
self.ctx.host = Host(self.ctx.log, self.ctx.sys)
def run_env(self, *argv):
args = build_parser(env.METADATA, 'apex env').parse_args(argv)
args = build_parser(env.METADATA, 'apex config/env').parse_args(argv)
with contextlib.redirect_stdout(io.StringIO()) as out:
env.run(self.ctx, args)
return out.getvalue()
+12 -1
View File
@@ -10,6 +10,7 @@ import unittest
from unittest.mock import Mock, patch
from engine.lib.host import Host, apply_block
from engine.lib.sys import Sys
def _check_backup_chmod(chmod, expected_group):
@@ -122,6 +123,15 @@ class TestHostWrites(unittest.TestCase):
self.assertEqual(stat.S_IMODE(backups[0].stat().st_mode), 0o600)
self.assertEqual(path.read_text(), "new")
def test_root_can_perform_privileged_file_operations_before_sudo_is_installed(self):
path = self.root / "bootstrap"
path.write_text("first")
host = Host(self.log, Sys(self.log))
with patch("os.geteuid", return_value=0), patch("shutil.which", return_value=None):
host.write_file(str(path), "second", sudo=True)
self.assertEqual(host._read(str(path), sudo=True), "second")
self.assertEqual(next(self.root.glob("bootstrap.*.bak")).read_text(), "first")
def test_backup_sets_owner_before_exposing_group_readable_content(self):
path = self.root / "group-secret"
path.write_text("synthetic group-only secret")
@@ -137,6 +147,8 @@ class TestHostWrites(unittest.TestCase):
os.utime(path, (1600000000, 1600000000))
def local_sudo(command, **kwargs):
if command[:2] == ["test", "-f"]:
return subprocess.run(command, **kwargs)
self.assertEqual(command[:3], ["python3", "-I", "-c"])
probe = ("import os\n" + inspect.getsource(_check_backup_chmod) +
f"\nos.chmod = _check_backup_chmod(os.chmod, {group})\n")
@@ -145,7 +157,6 @@ class TestHostWrites(unittest.TestCase):
self.assertEqual(result.returncode, 0, result.stderr)
return result
self.host.sys.ok.return_value = True
self.host.sys.sudo.side_effect = local_sudo
for sudo in (False, True):
with self.subTest(sudo=sudo):
+202
View File
@@ -0,0 +1,202 @@
import json
import os
from pathlib import Path
import subprocess
import tempfile
from types import SimpleNamespace
import unittest
from unittest.mock import Mock, patch
from engine import overlay
def action(name):
return overlay.load_module(str(Path(__file__).resolve().parents[1] / f"actions/{name}.py"),
"test_manual_" + name.replace("/", "_"))
class HostSetup(unittest.TestCase):
def test_packages_preserve_pve_firewall(self):
packages = action("config/packages")
ctx = SimpleNamespace(sys=Mock())
with (patch.object(Path, "is_file", return_value=True),
patch.object(Path, "exists", return_value=True),
patch.object(packages, "_install_missing") as install):
packages.run(ctx, None)
requested = install.call_args.args[1]
self.assertIn("sudo", requested)
self.assertNotIn("ufw", requested)
def test_failed_effective_policy_restores_original_configuration(self):
hardening = action("config/hardening")
system, host = Mock(), Mock()
system.sudo.return_value.stdout = "Port 22\n"
with (patch.object(hardening, "_operator_access", return_value="verified"),
patch.object(hardening, "_effective", return_value={"authenticationmethods": "any"})):
with self.assertRaisesRegex(RuntimeError, "effective SSH"):
hardening.run(SimpleNamespace(sys=system, host=host), None)
writes = host.write_file.call_args_list
self.assertTrue(writes[0].args[1].startswith("Port 22\n"))
self.assertEqual(writes[-1].args[1], "Port 22\n")
system.reload.assert_called_once_with("ssh")
def test_current_endpoint_must_have_authenticated_with_a_key(self):
hardening = action("config/hardening")
ctx = SimpleNamespace(sys=Mock())
sockets = SimpleNamespace(stdout='0 0 192.0.2.9:22 192.0.2.8:3002 users:(("sshd",pid=60,fd=4))')
accepted = {"_PID": "60", "__MONOTONIC_TIMESTAMP": "100",
"MESSAGE": "Accepted publickey for adam from 192.0.2.8 port 3002 ssh2: ED25519 SHA256:example"}
with (patch.dict(os.environ, {"SSH_CONNECTION": "192.0.2.100 4321 192.0.2.9 1234",
"SSH_USER_AUTH": "/arbitrary/file"}),
patch.object(hardening, "_ancestors", return_value={60: 50}),
patch.object(hardening.pwd, "getpwuid", return_value=SimpleNamespace(pw_name="adam"))):
for changes in ({"_PID": "42"}, {"__MONOTONIC_TIMESTAMP": "49"},
{"MESSAGE": accepted["MESSAGE"].replace("3002", "4002")}):
with self.subTest(changes=changes):
ctx.sys.sudo.side_effect = [sockets, SimpleNamespace(
returncode=0, stdout=json.dumps({**accepted, **changes}))]
with self.assertRaisesRegex(RuntimeError, "Cannot verify"):
hardening._connection(ctx)
ctx.sys.sudo.side_effect = [sockets, SimpleNamespace(returncode=0, stdout=json.dumps(accepted))]
self.assertEqual(hardening._connection(ctx),
"user=adam,host=192.0.2.8,addr=192.0.2.8,laddr=192.0.2.9,lport=22")
def test_journald_restart_failure_restores_previous_config(self):
base = action("config/base")
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
target = root / "99-apex.conf"
legacy = root / "maxuse.conf"
target.write_text("previous\n")
legacy.write_text("legacy\n")
backup = root / "legacy-backup"
host = Mock()
host.backup.side_effect = lambda path, **kwargs: (backup.write_bytes(Path(path).read_bytes())
and str(backup))
host.write_file.side_effect = lambda path, content, **kwargs: Path(path).write_text(content)
def sudo(command):
if command[0] == "systemctl":
raise RuntimeError("restart failed")
if command[0] == "cp":
Path(command[-1]).write_bytes(Path(command[-2]).read_bytes())
else:
Path(command[-1]).unlink()
ctx = SimpleNamespace(sys=SimpleNamespace(sudo=sudo), host=host)
with self.assertRaisesRegex(RuntimeError, "restart failed"):
base._configure(ctx, target, legacy, "updated\n", restart=True)
self.assertEqual(target.read_text(), "previous\n")
self.assertEqual(legacy.read_text(), "legacy\n")
def test_unverified_session_makes_no_host_changes(self):
hardening = action("config/hardening")
ctx = SimpleNamespace(sys=Mock(), host=Mock())
with (patch.object(hardening.pwd, "getpwnam", return_value=SimpleNamespace(
pw_uid=1001, pw_dir="/home/adam")),
patch.object(hardening.pwd, "getpwuid", return_value=SimpleNamespace(pw_name="root")),
patch.object(Path, "read_text", return_value="1001")):
with self.assertRaisesRegex(RuntimeError, "independent administrator"):
hardening.run(ctx, None)
self.assertFalse(ctx.host.mock_calls)
self.assertFalse(ctx.sys.mock_calls)
def test_independent_administrator_can_harden_only_with_an_operator_key(self):
hardening = action("config/hardening")
account = SimpleNamespace(pw_uid=1001, pw_dir="/home/adam")
system = Mock()
def privileged(command, **kwargs):
output = ""
if "stat" in command:
output = "1001 600 regular file" if command[-1].endswith("authorized_keys") else "1001 700 directory"
elif "cat" in command:
output = "ssh-ed25519 fixture"
elif "-T" in command:
output = "authorizedkeysfile .ssh/authorized_keys"
return SimpleNamespace(stdout=output, returncode=0)
system.sudo.side_effect = privileged
ctx = SimpleNamespace(sys=system)
with (patch.object(hardening.pwd, "getpwnam", return_value=account),
patch.object(hardening.pwd, "getpwuid", return_value=SimpleNamespace(pw_name="root")),
patch.object(hardening.os, "getuid", return_value=0),
patch.object(Path, "read_text", return_value="0")):
system.run.return_value.returncode = 1
with self.assertRaisesRegex(RuntimeError, "valid unrestricted operator public key"):
hardening._operator_access(ctx)
system.run.return_value.returncode = 0
self.assertEqual(hardening._operator_access(ctx), "user=adam,host=localhost,addr=127.0.0.1")
def test_chown_limits_changes_to_checkout_and_skips_mounts_and_hardlinks(self):
chown = action("repository/chown")
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary).resolve()
for name in (".git", "configs", "volumes"):
(root / name).mkdir()
for name in ("configs/node", ".git/HEAD", "private", "volumes/data"):
(root / name).write_text("fixture")
os.link(root / "private", root / "configs/key")
(root / "configs/other").symlink_to(root / "volumes", target_is_directory=True)
with patch("os.chown") as ownership, patch("os.fchown"):
chown._repair(str(root), {"configs/node", "configs/key", "configs/other", "volumes/data"},
{str(root / "volumes")}, os.getuid(), os.getgid())
self.assertEqual({call.args[0] for call in ownership.call_args_list},
{"HEAD", "node", "other"})
self.assertTrue(all(not call.kwargs["follow_symlinks"] for call in ownership.call_args_list))
def test_operator_provisioning_creates_account_then_preserves_keys_on_rerun(self):
adam = action("config/adam")
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary).resolve()
home = root / "adam"
account = SimpleNamespace(pw_dir=str(home), pw_uid=1001)
key = root / "operator"
subprocess.run(["ssh-keygen", "-q", "-t", "ed25519", "-N", "", "-f", str(key)], check=True)
public = key.with_suffix(".pub")
ctx = SimpleNamespace(sys=Mock(), host=Mock())
ctx.sys.run.side_effect = lambda cmd, **kw: subprocess.run(cmd, input=kw["input"], text=True,
capture_output=True, check=True)
def privileged(command, **kwargs):
if command[0] == "useradd":
home.mkdir()
if command[:6] == ["runuser", "-u", "adam", "--", "sh", "-ec"]:
subprocess.run(command[4:], text=True, input=kwargs["input"], check=True)
ctx.sys.sudo.side_effect = privileged
actual_stat = Path.stat
def metadata(path, *args, **kwargs):
stat = actual_stat(path, *args, **kwargs)
if path == home:
fields = list(stat)
fields[4] = account.pw_uid
return os.stat_result(fields)
return stat
with (patch.object(adam, "HOME", home), patch.object(Path, "stat", metadata),
patch.object(adam.grp, "getgrnam", side_effect=KeyError),
patch.object(adam.pwd, "getpwnam", side_effect=[KeyError, account, account])):
args = SimpleNamespace(public_key_file=str(public))
adam.run(ctx, args)
authorized = home / ".ssh/authorized_keys"
with authorized.open("a") as stream:
stream.write("existing operator entry\n")
adam.run(ctx, args)
self.assertEqual(authorized.read_text().splitlines().count(public.read_text().strip()), 1)
self.assertIn("existing operator entry", authorized.read_text())
commands = [call.args[0] for call in ctx.sys.sudo.call_args_list]
self.assertEqual(sum(cmd[0] == "useradd" for cmd in commands), 1)
self.assertIn(["runuser", "-u", "adam", "--", "sudo", "-n", "true"], commands)
def test_existing_system_account_cannot_be_promoted(self):
adam = action("config/adam")
ctx = SimpleNamespace(sys=Mock())
account = SimpleNamespace(pw_dir="/home/adam", pw_uid=999)
with patch.object(adam.pwd, "getpwnam", return_value=account):
with self.assertRaisesRegex(RuntimeError, "incompatible UID"):
adam.run(ctx, SimpleNamespace(public_key_file=None))
self.assertFalse(ctx.sys.mock_calls)
def test_invalid_operator_key_is_rejected_before_provisioning(self):
adam = action("config/adam")
with tempfile.NamedTemporaryFile(mode="w") as key:
key.write("private key fixture")
key.flush()
ctx = SimpleNamespace(sys=Mock())
with self.assertRaisesRegex(ValueError, "OpenSSH public key"):
adam.run(ctx, SimpleNamespace(public_key_file=key.name))
self.assertFalse(ctx.sys.mock_calls)
+19 -11
View File
@@ -17,15 +17,15 @@ class TestOverlay(unittest.TestCase):
self.prop = tempfile.mkdtemp()
def test_commons_only(self):
_write(self.commons, "configure/ufw.py", ACT.format(s="commons ufw", r="C"))
r = overlay.resolve("configure/ufw", self.commons, self.prop)
_write(self.commons, "config/ufw.py", ACT.format(s="commons ufw", r="C"))
r = overlay.resolve("config/ufw", self.commons, self.prop)
self.assertIsNotNone(r); self.assertIsNone(r.shadowed)
self.assertTrue(r.path.startswith(self.commons))
def test_override_exposes_shadowed(self):
_write(self.commons, "configure/ufw.py", ACT.format(s="commons", r="C"))
_write(self.prop, "configure/ufw.py", ACT.format(s="local", r="L"))
r = overlay.resolve("configure/ufw", self.commons, self.prop)
_write(self.commons, "config/ufw.py", ACT.format(s="commons", r="C"))
_write(self.prop, "config/ufw.py", ACT.format(s="local", r="L"))
r = overlay.resolve("config/ufw", self.commons, self.prop)
self.assertTrue(r.path.startswith(self.prop)) # local wins
self.assertTrue(r.shadowed.startswith(self.commons))
@@ -33,17 +33,25 @@ class TestOverlay(unittest.TestCase):
self.assertIsNone(overlay.resolve("no/such", self.commons, self.prop))
def test_discover_marks_sources_and_disabled(self):
_write(self.commons, "configure/ufw.py", ACT.format(s="commons ufw", r="C"))
_write(self.commons, "sync/repository.py", ACT.format(s="sync repo", r="C"))
_write(self.prop, "configure/ufw.py", ACT.format(s="local ufw", r="L"))
_write(self.commons, "config/ufw.py", ACT.format(s="commons ufw", r="C"))
_write(self.commons, "repository/sync.py", ACT.format(s="sync repo", r="C"))
_write(self.prop, "config/ufw.py", ACT.format(s="local ufw", r="L"))
_write(self.prop, "sync/minecraft.py",
'from engine.descriptor import Meta\nMETADATA = Meta(summary="mc")\nDISABLED = "wip"\ndef run(ctx, args): pass\n')
infos = {i.name: i for i in overlay.discover(self.commons, self.prop)}
self.assertEqual(infos["configure/ufw"].source, "override")
self.assertEqual(infos["sync/repository"].source, "commons")
self.assertEqual(infos["config/ufw"].source, "override")
self.assertEqual(infos["repository/sync"].source, "commons")
self.assertEqual(infos["sync/minecraft"].source, "local")
self.assertEqual(infos["sync/minecraft"].disabled, "wip")
self.assertEqual(infos["configure/ufw"].summary, "local ufw")
self.assertEqual(infos["config/ufw"].summary, "local ufw")
def test_only_entity_action_paths_can_be_listed_and_dispatched(self):
for name in ("configure", "config/nested/tool", "config/tool"):
_write(self.prop, name + ".py", ACT.format(s="test", r="ok"))
names = [item.name for item in overlay.discover(self.commons, self.prop)]
self.assertEqual(names, ["config/tool"])
for name in ("configure", "config/nested/tool", "config/../configure"):
self.assertIsNone(overlay.resolve(name, self.commons, self.prop))
def test_load_module_runs(self):
p = _write(self.commons, "x/y.py", ACT.format(s="s", r="Z"))
+3 -1
View File
@@ -6,7 +6,9 @@ import unittest
from types import SimpleNamespace
from unittest.mock import Mock, patch
from actions.sync import packages
from importlib import import_module
packages = import_module("actions.util.check-updates")
def digest(char):
+1 -1
View File
@@ -5,7 +5,7 @@ import unittest
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import Mock
from actions.sync import repository
from actions.repository import sync as repository
from engine.lib.sys import Sys
+1 -1
View File
@@ -6,7 +6,7 @@ import unittest
from unittest.mock import Mock
ROOT = Path(__file__).resolve().parents[1]
spec = importlib.util.spec_from_file_location('timezone_action', ROOT / 'actions/configure/timezone.py')
spec = importlib.util.spec_from_file_location('timezone_action', ROOT / 'actions/config/timezone.py')
action = importlib.util.module_from_spec(spec)
spec.loader.exec_module(action)