chore: genericize deployment-specific defaults for public release

- core compose: parameterize TZ (default Etc/UTC), restic S3 region+endpoint
  (vendor-neutral default: AWS S3 us-east-1, no custom endpoint), and the
  xray-enclave allowlist (APEX_ENCLAVE_SUBNET, default 198.18.0.0/16).
  Cloudflare + resticontainer kept as-is.
- the reference fleet pins its real values (Europe/Moscow, ru-central1, Yandex
  Object Storage) in each node's apex.env, so deployed behavior is unchanged.
- drop PARITY.md (internal bash->python port audit; node-specific narrative).
- genericize node-name leftovers (icarus/daedalus, MSK) in tests + comments.

Tests: 26/26 pass. Verified: resticontainer resolves ru-central1/Yandex/Moscow on a node.
This commit is contained in:
deytenit committed 2026-07-11 18:00:13 +03:00
1 parent fc11217a90
commit 94082ad9be
6 files changed
+18 -91

No files matched your search

-76
View File
@@ -1,76 +0,0 @@
# Port parity map (bash -> python)
Behavioral parity is validated on a scratch host in Phase 3. This maps each ported
action to its bash source and the behaviors that must match.
| apex action | bash source (root commons) | must-match behaviors |
|---|---|---|
| configure/base | actions/configure/base | apt update+install curl/wget/git/rsyslog; rsyslog running; ufw + ufw-docker at ~/.local/bin |
| configure/ufw | actions/configure/ufw + root_template_ufw | host.rules via plain ufw; before/after.rules block inject (APEX markers, self-heal ROOT); docker rules via ufw-docker |
| configure/crowdsec | actions/configure/crowdsec | install if cscli absent; render configs->/etc/crowdsec; collections linux+traefik; firewall bouncer if bouncer file present |
| configure/cron | actions/configure/cron + root_template_crontab | render crontab; backup+install; rollback on empty/failure |
| configure/systemd | actions/configure/systemd | render each unit -> /etc/systemd/system; daemon-reload; enable+start .service |
| configure/routing | actions/configure/routing | ip_forward=1; fwmark 0x1->table100 rule; local default route table100 (idempotent) |
| configure | (new umbrella) | run base,ufw,crowdsec,cron,systemd,routing in order |
| sync/repository | actions/sync/repository | sync/<node> branch; add -A + commons submodule; commit; rebase origin/main (was trunk in the monorepo); force-with-lease; telegram |
| sync/packages | actions/sync/packages | apt --just-print upgrade Inst list; docker+skopeo digest compare; telegram info |
| backup/run | actions/sync/tiers (restic-backup) | init-if-needed; label-driven backup via resticontainer (discovers `restic.*`-labelled services, runs pre-hooks/stops, snapshots the resolved host paths) --host $APEX_NODE_HOST --tag biweekly --compression max; guard on a fresh snapshot; forget --prune 7/4/12; telegram |
| tiers/link | actions/tiers/link | node + per-project @tierN symlinks; shared/ links; .env symlink |
| tiers/useradd | actions/tiers/useradd | noroot-<proj> system users; noroot-shared group; APEX_UID/GID block in .env |
| tiers/chown | actions/tiers/chown | {tier}/{proj}=uid:gid; {tier}/shared/{proj}=uid:sgid+sticky; base=root:root |
| utils/extract-traefik-certs | actions/utils/extract-traefik-certs | traefik-certs-dumper v3; 600 perms; chown caller |
| utils/generate-happ-subscriptions | actions/utils/generate-happ-subscriptions | MOVED to daedalus proprietaries (node-specific tooling; commons keeps only shared stuff) |
| utils/lint-docker-compose | actions/utils/lint-docker-compose | dclint --fix via docker; --hook staged-only + restage; + anchor-IP check |
| compose | actions/compose | apex core first then services alpha; reverse on down; --dry-run; extra passthrough |
## Deliberate deltas (rename + fixes)
- root->apex everywhere (CLI, container names apex-*, telegram instance = APEX_NODE_HOST, ufw markers APEX.*, networks direct/enclave/socket).
- No <node> arg; identity from FQDN + node.env.
- backup superseded by the resticontainer migration (see below): the old whole-tier-root
restic snapshot is replaced by per-service, `restic.*`-label-driven backups.
- ufw markers self-heal (delete ROOT + APEX blocks before inject).
- envsubst -> string.Template.safe_substitute; wget/curl download -> urllib. Unset template
vars stay LITERAL (envsubst emptied them) — typos surface instead of silently blanking.
- crowdsec install auto-confirms (`apt-get install -y`; bash ran promptful `apt install`,
which blocked on a tty and aborted non-interactive runs).
- pre-commit staged-compose pathspec widened to `*docker-compose.yml` — the old
`*.docker-compose.yml`/root-anchored patterns never matched nested compose files.
- anchor-IP validation in utils/lint-docker-compose is a planned addition (SP3 §6.2).
- new opt-in `--dry-run` flags on configure/* have no bash counterpart; the default
(flag absent) path is the ported behavior.
- ufw/crowdsec deploys keep a timestamped `.bak` next to overwritten system files
(bash: none / `/tmp`) as a rollback net; tiers/useradd `.env` rewrites deliberately do
NOT (no secret-bearing copies).
- non-conventional tool exit codes normalize to 1 (bash sometimes propagated e.g.
apt-get's 100); 64/65/66 conventions preserved.
## Core-compose consolidation deltas (beyond the rename)
- icarus: traefik no longer publishes 53/tcp+udp (AdGuard DNS) from the shared file —
restored via an icarus-only override layered in the node repo's include (Phase 2).
- icarus: ACME propagation-tuning flags and DEBUG log level drop to the shared defaults;
traefik now runs noroot (APEX_UID) with file provider + HTTP/3 like daedalus.
- xray is profile-gated (`xray` in COMPOSE_PROFILES): daedalus + icarus opt in,
morpheus does not (it never ran xray — owner decision 2026-07-05).
- loki (profile `loki`, no node opts in yet) is exposed via traefik at
`https://<node-fqdn>/loki` behind basicauth (`APEX_LOKI_BASIC_AUTH` in the node .env).
- morpheus: dashboard auth changes basicauth (TRAEFIK_INTERNAL_API_AUTH) -> enclave IP
allowlist; the adguard router labels exist without a backend (502, was 404, on /adguard).
- restic: per-node selective tier2 mounts replaced by whole-tier-root RO mounts
(superset; snapshot paths preserved); RESTIC_COMPRESSION per-node via
`APEX_RESTIC_COMPRESSION` (default auto).
## resticontainer migration (SP4 realized)
- The `restic` core service is replaced by `resticontainer`
(`ghcr.io/deytenit/resticontainer`), a restic wrapper that reads per-service
`restic.*` compose labels: `restic.enable`, `restic.backup.paths` (comma-separated
container mount destinations — matched exactly, whole mount dir backed up),
`restic.backup.stop`, and `restic.hooks.pre-backup`/`post-backup` (run via docker exec).
- It mounts the docker socket (list/inspect/exec/stop/start) and `/srv:/srv:ro`
(`RESTIC_HOSTFS=/`); apex data + checkouts live under /srv and the `@tierN` mounts
are absolute symlinks into /srv, so mounting /srv at the same path lets restic
follow them and record the real host paths, and snapshots the union in one run.
- The whole-tier-root RO mounts and the explicit `/data/@tier1,2` backup args are gone;
a service is backed up only if it carries `restic.enable=true`. DB consistency is
per-service: a pre-backup hook dumps into a dedicated backed-up mount (e.g. postgres
`pg_dump … > /dumps/dump.sql`), never the live data dir. `backup/run` now guards on a
fresh snapshot landing for the host (a no-label repo would otherwise be a silent no-op).
+8 -5
View File
@@ -4,7 +4,7 @@
# resolve against THIS dir — always use absolute ${APEX_*} vars). The engine exports
# APEX_TIER*/APEX_TIER*_SHARED/APEX_TIER_ROOT*/APEX_SUBNET/APEX_NODE_FQDN at invoke.
x-environment: &shared-environment
TZ: "Europe/Moscow"
TZ: "${TZ:-Etc/UTC}" # override per node via TZ in the env; default UTC
services:
# --- edge (host network; profile-gated — a node opts in via COMPOSE_PROFILES) ---
@@ -61,8 +61,11 @@ services:
RESTIC_PASSWORD: "${APEX_RESTIC_PASSWORD?error}"
AWS_ACCESS_KEY_ID: "${APEX_RESTIC_AWS_ACCESS_KEY_ID?error}"
AWS_SECRET_ACCESS_KEY: "${APEX_RESTIC_AWS_SECRET_ACCESS_KEY?error}"
AWS_REGION: "ru-central1"
AWS_ENDPOINT_URL: "https://storage.yandexcloud.net/"
# S3 backend region + endpoint. Defaults suit AWS S3 (us-east-1, no custom
# endpoint); set APEX_RESTIC_AWS_REGION / APEX_RESTIC_AWS_ENDPOINT (node apex.env)
# for any other S3-compatible store (Yandex, Backblaze B2, MinIO, …).
AWS_REGION: "${APEX_RESTIC_AWS_REGION:-us-east-1}"
AWS_ENDPOINT_URL: "${APEX_RESTIC_AWS_ENDPOINT:-}"
# max compression is fleet policy (backup/run also forces --compression max on
# the CLI; needs repo format v2, which a fresh restic init creates). Per-node
# override via APEX_RESTIC_COMPRESSION only if ever needed.
@@ -139,7 +142,7 @@ services:
labels:
- "traefik.enable=true"
- "traefik.http.routers.apex-traefik.rule=Host(`${APEX_NODE_FQDN}`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
- "traefik.http.middlewares.xray-enclave.ipallowlist.sourcerange=198.18.0.0/16"
- "traefik.http.middlewares.xray-enclave.ipallowlist.sourcerange=${APEX_ENCLAVE_SUBNET:-198.18.0.0/16}"
- "traefik.http.routers.apex-traefik.middlewares=xray-enclave@docker"
- "traefik.http.routers.apex-traefik.service=api@internal"
# adguard route (backend only present when the adguard profile is active)
@@ -182,7 +185,7 @@ services:
# Grafana Alloy: one agent per node. Collects host metrics (node_exporter-
# equivalent) + container metrics (cadvisor) and PUSHES them via remote_write;
# collects systemd-journal + docker logs and PUSHES them to Loki. All targets
# are OUTBOUND to the central gfn stack (Prometheus + Loki live on icarus), so
# are OUTBOUND to a central Prometheus + Loki (hosted on whichever node runs them), so
# Alloy needs egress (direct, not enclave). Replaces the old store-only `loki`
# service (which had no feeder). Config is a committed, env-parameterized asset.
alloy:
+1 -1
View File
@@ -20,7 +20,7 @@ from typing import Dict, List, Tuple
@dataclass
class Node:
name: str # short label (first segment of the host), e.g. "icarus" — logs, branch, messages
name: str # short label (first segment of the host), e.g. "node1" — logs, branch, messages
subnet: str # the node's direct-network /24
fqdn: str # real FQDN — traefik Host() routing
host: str # public host name — backup --host, obs `instance`, notify instance
+4 -4
View File
@@ -10,12 +10,12 @@ class TestIdentity(unittest.TestCase):
return d
def test_explicit_host_and_fqdn(self):
repo = self._envfile("APEX_NODE_FQDN=icarus.a1.example.com\n"
"APEX_NODE_HOST=icarus.example.com\n"
"APEX_SUBNET=198.18.17.0/24\n")
repo = self._envfile("APEX_NODE_FQDN=node1.dc1.example.com\n"
"APEX_NODE_HOST=node1.example.com\n"
"APEX_SUBNET=10.0.0.0/24\n")
n, warns = resolve("build-laptop", repo) # hostname ignored when node.env has FQDN
self.assertEqual((n.name, n.host, n.fqdn, n.subnet),
("icarus", "icarus.example.com", "icarus.a1.example.com", "198.18.17.0/24"))
("node1", "node1.example.com", "node1.dc1.example.com", "10.0.0.0/24"))
self.assertEqual(warns, [])
def test_host_defaults_to_fqdn(self):
+3 -3
View File
@@ -4,9 +4,9 @@ from engine.lib.notify import build_text
class TestNotify(unittest.TestCase):
def test_instance_is_the_host(self):
t = build_text("Title", "msg", "icarus.example.com", "INFO", "2026-07-04 00:00:00 MSK")
self.assertIn("icarus.example.com", t)
self.assertIn("#instance_icarus_example_com", t) # dots -> underscores in the hashtag
t = build_text("Title", "msg", "node1.example.com", "INFO", "2026-07-04 00:00:00 UTC")
self.assertIn("node1.example.com", t)
self.assertIn("#instance_node1_example_com", t) # dots -> underscores in the hashtag
def test_html_escaped_and_truncated(self):
t = build_text("A & B", "<script>" + "x" * 300, "d", "ERROR", "T")
+2 -2
View File
@@ -15,8 +15,8 @@ class TestTpl(unittest.TestCase):
def test_render_substitutes_known_vars(self):
self._w("cron/crontab", "root=${APEX_TIER1} node=$APEX_NODE\n")
out = self.tpl.render("cron/crontab", {"APEX_TIER1": "/mnt/t1", "APEX_NODE": "daedalus"})
self.assertEqual(out, "root=/mnt/t1 node=daedalus\n")
out = self.tpl.render("cron/crontab", {"APEX_TIER1": "/mnt/t1", "APEX_NODE": "node1"})
self.assertEqual(out, "root=/mnt/t1 node=node1\n")
def test_render_leaves_shell_command_substitution(self):
self._w("c", "x $(cat /f) ${APEX_NODE}\n")