refactor: make engine domain-agnostic — identity from node.env, no hardcoded domain

Drop the hardcoded-domain FQDN regex and every hardcoded domain from commons.
Node identity is now explicit in node.env: APEX_NODE_FQDN (real FQDN,
traefik Host()), APEX_NODE_HOST (public host name — backup --host,
obs/notify instance), APEX_SUBNET. The short name (log prefix, sync/<node>
branch, messages) is derived as the first host label.

notify instance is just the host (no hardcoded prefix); the git bot identity
moves to APEX_GIT_AUTHOR_NAME/EMAIL (node.env) with generic defaults. Alloy
node+instance labels use APEX_NODE_HOST. Fallbacks keep a node working if it
omits a var (host<-fqdn<-hostname).

BREAKING CHANGE: removes APEX_NODE and APEX_CLUSTER env vars; node name is now derived from APEX_NODE_FQDN

PARITY.md

@@ -14,7 +14,7 @@ action to its bash source and the behaviors that must match.
 | configure | (new umbrella) | run base,ufw,crowdsec,cron,systemd,routing in order |
 | sync/repository | actions/sync/repository | sync/<node> branch; add -A + commons submodule; commit; rebase origin/main (was trunk in the monorepo); force-with-lease; telegram |
 | sync/packages | actions/sync/packages | apt --just-print upgrade Inst list; docker+skopeo digest compare; telegram info |
-| backup/run | actions/sync/tiers (restic-backup) | init-if-needed; backup /data/@tier1,2 --host --tag com-ermnvldmr-root-<node>; forget --prune 7/4/12; telegram |
+| backup/run | actions/sync/tiers (restic-backup) | init-if-needed; label-driven backup via resticontainer (discovers `restic.*`-labelled services, runs pre-hooks/stops, snapshots the resolved host paths) --host $APEX_NODE_HOST --tag biweekly --compression max; guard on a fresh snapshot; forget --prune 7/4/12; telegram |
 | tiers/link | actions/tiers/link | node + per-project @tierN symlinks; shared/ links; .env symlink |
 | tiers/useradd | actions/tiers/useradd | noroot-<proj> system users; noroot-shared group; APEX_UID/GID block in .env |
 | tiers/chown | actions/tiers/chown | {tier}/{proj}=uid:gid; {tier}/shared/{proj}=uid:sgid+sticky; base=root:root |
@@ -24,9 +24,10 @@ action to its bash source and the behaviors that must match.
 | compose | actions/compose | apex core first then services alpha; reverse on down; --dry-run; extra passthrough |
 
 ## Deliberate deltas (rename + fixes)
-- root->apex everywhere (CLI, container names apex-*, telegram instance com.ermnvldmr.apex.*, ufw markers APEX.*, networks direct/enclave/socket).
+- root->apex everywhere (CLI, container names apex-*, telegram instance = APEX_NODE_HOST, ufw markers APEX.*, networks direct/enclave/socket).
 - No <node> arg; identity from FQDN + node.env.
-- restic core service gains read-only /data/@tier1,2 mounts (makes existing backup command functional).
+- backup superseded by the resticontainer migration (see below): the old whole-tier-root
+  restic snapshot is replaced by per-service, `restic.*`-label-driven backups.
 - ufw markers self-heal (delete ROOT + APEX blocks before inject).
 - envsubst -> string.Template.safe_substitute; wget/curl download -> urllib. Unset template
   vars stay LITERAL (envsubst emptied them) — typos surface instead of silently blanking.
@@ -57,3 +58,19 @@ action to its bash source and the behaviors that must match.
 - restic: per-node selective tier2 mounts replaced by whole-tier-root RO mounts
   (superset; snapshot paths preserved); RESTIC_COMPRESSION per-node via
   `APEX_RESTIC_COMPRESSION` (default auto).
+
+## resticontainer migration (SP4 realized)
+- The `restic` core service is replaced by `resticontainer`
+  (`ghcr.io/deytenit/resticontainer`), a restic wrapper that reads per-service
+  `restic.*` compose labels: `restic.enable`, `restic.backup.paths` (comma-separated
+  container mount destinations — matched exactly, whole mount dir backed up),
+  `restic.backup.stop`, and `restic.hooks.pre-backup`/`post-backup` (run via docker exec).
+- It mounts the docker socket (list/inspect/exec/stop/start) and `/srv:/srv:ro`
+  (`RESTIC_HOSTFS=/`); apex data + checkouts live under /srv and the `@tierN` mounts
+  are absolute symlinks into /srv, so mounting /srv at the same path lets restic
+  follow them and record the real host paths, and snapshots the union in one run.
+- The whole-tier-root RO mounts and the explicit `/data/@tier1,2` backup args are gone;
+  a service is backed up only if it carries `restic.enable=true`. DB consistency is
+  per-service: a pre-backup hook dumps into a dedicated backed-up mount (e.g. postgres
+  `pg_dump … > /dumps/dump.sql`), never the live data dir. `backup/run` now guards on a
+  fresh snapshot landing for the host (a no-label repo would otherwise be a silent no-op).

README.md

@@ -1,7 +1,9 @@
-# apex.ermnvldmr.com
+# apex
 
-The **commons** of the apex fleet: a stock-python engine (`apex <group>/<action>`) and
+The **commons** of an apex fleet: a stock-python engine (`apex <group>/<action>`) and
 the shared core docker-compose, consumed by every node as the `commons/` git submodule.
+The framework assumes no particular domain — every node declares its own identity in
+`node.env`.
 
 ## Layout
 - `apex` — launcher (python3, stdlib only); node `init.sh` symlinks it onto `PATH`.
@@ -15,8 +17,8 @@ Node-specific tooling lives in each node repo's `proprietaries/` — commons hol
 what every node shares.
 
 ## Usage (on a node)
-`apex` acts on the **local checkout** — no `<node>` argument. Identity comes from the
+`apex` acts on the **local checkout** — no `<node>` argument. Identity comes from
-host FQDN (`<node>.a<x>.apex.ermnvldmr.com`) and `node.env`.
+`node.env` (`APEX_NODE_HOST`, `APEX_NODE_FQDN`, `APEX_SUBNET`).
 - `apex` / `apex --help` — list actions
 - `apex configure/ufw` — render + apply this node's ufw config
 - `apex configure` — run every `configure/*` in order

actions/backup/run.py

@@ -1,17 +1,23 @@
 # actions/backup/run.py
-"""Port of sync/tiers restic-backup (SP4: behavior-identical). Runs the restic core
+"""Label-driven restic backup via resticontainer (github.com/deytenit/resticontainer).
-service one-shot: init-if-needed -> backup tiers -> forget --prune -> Telegram."""
+Runs the resticontainer core service one-shot: init-if-needed -> backup (discovers
+`restic.*`-labelled services, runs their pre-hooks/stops, snapshots the union of the
+resolved host paths in a single restic run) -> forget --prune -> Telegram. Every
+non-`backup` subcommand passes straight through to restic, so init/snapshots/forget
+behave exactly as before."""
+import json
 import os
 from engine.descriptor import Meta, Arg
 
-METADATA = Meta(summary="Run restic backup of the backed-up tiers (init-if-needed, backup, forget --prune).",
+METADATA = Meta(summary="Run the resticontainer label-driven backup (init-if-needed, backup, forget --prune).",
                 args=[Arg("telegram_bot_url", "Telegram Bot URL for notifications")])
 TITLE = "Restic Host Backup"
 
 def run(ctx, args):
     log, node, url = ctx.log, ctx.node.name, args.telegram_bot_url
     core = ctx.paths.core
-    tag = f"com-ermnvldmr-root-{node}"          # KEPT unchanged for backup continuity (SP4/Decision D)
+    host = ctx.node.host                        # restic --host: the node's public host name (APEX_NODE_HOST)
+    tag = "biweekly"                            # restic --tag: snapshot cadence (this cron fires on the 5th & 20th)
     if not os.path.isfile(os.path.join(core, "docker-compose.yml")):
         log.error(f"Core compose not found in {core}"); raise SystemExit(1)
     env = dict(os.environ, COMPOSE_PROFILES="manual")
@@ -21,30 +27,51 @@ def run(ctx, args):
         if os.path.isfile(os.path.join(core, ef)):
             envfiles += ["--env-file", ef]
 
-    def restic(*cmd, check=True):
+    def restic(*cmd, check=True, capture=False):
         return ctx.sys.run(["docker", "compose", *envfiles, "--profile", "manual",
-                            "run", "--rm", "restic", *cmd], cwd=core, env=env, check=check)
+                            "run", "--rm", "-T", "resticontainer", *cmd],
+                           cwd=core, env=env, check=check, capture=capture)
+
+    def host_snapshot_ids():
+        """This host's snapshot IDs, or None if it can't be determined (never fail on this)."""
+        cp = restic("snapshots", "--host", host, "--json", check=False, capture=True)
+        if cp.returncode != 0:
+            return None
+        try:
+            return {s.get("id") for s in json.loads(cp.stdout or "[]")}
+        except (ValueError, TypeError):
+            return None
 
     try:
         log.info("Checking restic repository availability...")
         if restic("snapshots", check=False).returncode == 0:
             log.info("Restic repository reachable.")
         elif restic("init", check=False).returncode != 0:
-            ctx.notify.error(TITLE, url, "restic init failed", node); raise SystemExit(1)
+            ctx.notify.error(TITLE, url, "restic init failed"); raise SystemExit(1)
-        log.info("Backing up /data/@tier1 and /data/@tier2 ...")
+        before = host_snapshot_ids()
-        if restic("backup", "/data/@tier1", "/data/@tier2", "--host", node, "--tag", tag,
+        log.info("Backing up `restic.*`-labelled services (resticontainer discovers paths)...")
-                  "--compression", "auto", check=False).returncode != 0:
+        if restic("backup", "--host", host, "--tag", tag, "--compression", "max",
-            ctx.notify.error(TITLE, url, "restic backup failed", node); raise SystemExit(1)
+                  check=False).returncode != 0:
+            ctx.notify.error(TITLE, url, "restic backup failed"); raise SystemExit(1)
+        # Guard: the label-driven backup is a silent no-op if no service carries the
+        # `restic.enable` label (resticontainer finds zero paths and skips restic).
+        # Prove a fresh snapshot for this host actually landed before we prune/notify.
+        after = host_snapshot_ids()
+        if before is not None and after is not None and not (after - before):
+            log.error("No new snapshot produced — refusing to report success.")
+            ctx.notify.error(TITLE, url,
+                             "backup produced no new snapshot (no restic.* labels discovered?)")
+            raise SystemExit(1)
         log.info("Applying retention (forget --prune)...")
         if restic("forget", "--prune", "--keep-daily", "7", "--keep-weekly", "4",
                   "--keep-monthly", "12", check=False).returncode != 0:
-            ctx.notify.error(TITLE, url, "restic forget/prune failed", node); raise SystemExit(1)
+            ctx.notify.error(TITLE, url, "restic forget/prune failed"); raise SystemExit(1)
         log.success("Restic backup completed.")
-        if not ctx.notify.success(TITLE, url, f"Restic backup + retention succeeded for {node}", node):
+        if not ctx.notify.success(TITLE, url, f"Restic backup + retention succeeded for {node}"):
             raise SystemExit(1)   # bash exited 1 when the success telegram failed to send
     except SystemExit:
         raise
     except Exception as e:
         log.error(f"{type(e).__name__}: {e}")
-        ctx.notify.error(TITLE, url, f"backup failed: {e}", node)
+        ctx.notify.error(TITLE, url, f"backup failed: {e}")
         raise SystemExit(1)

actions/sync/packages.py

@@ -15,13 +15,13 @@ def run(ctx, args):
         raise
     except Exception as e:   # bash ERR trap parity: unhandled failures page via Telegram
         log.error(f"{type(e).__name__}: {e}")
-        ctx.notify.error(TITLE, url, f"Critical error: {e}", node)
+        ctx.notify.error(TITLE, url, f"Critical error: {e}")
         raise SystemExit(1)
 
 def _check(ctx, args, log, node, url, s):
     if not os.path.isfile("/etc/debian_version"):
         log.error("Requires a Debian-based system.")
-        ctx.notify.error(TITLE, url, "Requires a Debian-based system.", node)
+        ctx.notify.error(TITLE, url, "Requires a Debian-based system.")
         raise SystemExit(1)
 
     log.info("Updating package lists...")
@@ -66,6 +66,6 @@ def _check(ctx, args, log, node, url, s):
         images = (f"*Images ({len(outdated)}):* " + ", ".join(f"`{i}`" for i in outdated)) if outdated else "*Images:* _none_"
 
     # A failed send exits 1 (bash: telegram_info's return 1 tripped set -e).
-    if not ctx.notify.info(TITLE, url, f"*APT:*\n{packages}\n\n*Skopeo:*\n{images}", node):
+    if not ctx.notify.info(TITLE, url, f"*APT:*\n{packages}\n\n*Skopeo:*\n{images}"):
         raise SystemExit(1)
     log.success("Security update check completed.")

actions/sync/repository.py

@@ -3,6 +3,7 @@
 import os
 from datetime import datetime
 from engine.descriptor import Meta, Arg
+from engine.identity import read_env
 
 METADATA = Meta(summary="Capture-up: commit node changes, rebase onto origin/main, force-push sync/<node>.",
                 args=[Arg("telegram_bot_url", "Telegram Bot URL for notifications")])
@@ -11,9 +12,12 @@ TITLE = "Annual Remote Sync"
 def run(ctx, args):
     log, node, url = ctx.log, ctx.node.name, args.telegram_bot_url
     repo, commons = ctx.paths.repo_root, ctx.paths.commons
+    ne = read_env(os.path.join(repo, "node.env"))       # per-node bot identity (generic defaults)
+    author = ne.get("APEX_GIT_AUTHOR_NAME", "apex [bot]")
+    email = ne.get("APEX_GIT_AUTHOR_EMAIL", "apex@localhost")
     env = dict(os.environ,
-               GIT_AUTHOR_NAME="Adam Jensen [bot]", GIT_AUTHOR_EMAIL="adam@ermnvldmr.com",
+               GIT_AUTHOR_NAME=author, GIT_AUTHOR_EMAIL=email,
-               GIT_COMMITTER_NAME="Adam Jensen [bot]", GIT_COMMITTER_EMAIL="adam@ermnvldmr.com")
+               GIT_COMMITTER_NAME=author, GIT_COMMITTER_EMAIL=email)
 
     def git(*a, check=True):
         return ctx.sys.run(["git", "-C", repo, *a], check=check, env=env)
@@ -45,16 +49,16 @@ def run(ctx, args):
         if git("rebase", "origin/main", check=False).returncode != 0:
             log.error("Rebase conflict — aborting rebase; manual intervention required.")
             git("rebase", "--abort", check=False)
-            ctx.notify.error(TITLE, url, "Rebase conflict. Manual intervention required.", node)
+            ctx.notify.error(TITLE, url, "Rebase conflict. Manual intervention required.")
             raise SystemExit(1)
         git("submodule", "update", "--init", "--recursive")
         git("push", "--force-with-lease", "origin", branch)
         log.success(f"Pushed {branch}.")
-        if not ctx.notify.success(TITLE, url, f"Sync to '{branch}' completed.", node):
+        if not ctx.notify.success(TITLE, url, f"Sync to '{branch}' completed."):
             raise SystemExit(1)
     except SystemExit:
         raise
     except Exception as e:
         log.error(f"{type(e).__name__}: {e}")
-        ctx.notify.error(TITLE, url, f"Sync failed: {e}", node)
+        ctx.notify.error(TITLE, url, f"Sync failed: {e}")
         raise SystemExit(1)

actions/tiers/chown.py

@@ -46,6 +46,16 @@ def run(ctx, args):
         full = os.path.join(comp, proj)
         if not os.path.isdir(full) or proj.startswith(".") or proj.startswith("@"):
             continue
+        # opt-out: .apex-root-owned forces this project's tier data (dir + children,
+        # recursively) to root:root — for services that run as root and reject
+        # non-root-owned data (e.g. seafile-mc). Blocks noroot ownership entirely.
+        if os.path.isfile(os.path.join(full, ".apex-root-owned")):
+            log.info(f"Applying root ownership for {proj} (.apex-root-owned)")
+            for tp in tiers:
+                for d in (os.path.join(tp, proj), os.path.join(tp, "shared", proj)):
+                    if os.path.isdir(d):
+                        s.sudo(["chown", "-R", "root:root", d]); log.success(f"  Owned {d} (root)")
+            continue
         env_path = os.path.join(full, ".env")
         uid = _env_val(env_path, "APEX_UID")
         gid = _env_val(env_path, "APEX_GID")

actions/tiers/useradd.py

@@ -34,6 +34,12 @@ def run(ctx, args):
         full = os.path.join(comp, proj)
         if not os.path.isdir(full) or proj.startswith(".") or proj.startswith("@"):
             continue
+        # opt-out: a project marked .apex-root-owned runs as root and keeps its tier
+        # data root-owned (e.g. seafile-mc, whose entrypoint refuses non-root-owned
+        # data). Skip the noroot user + .env UID injection; tiers/chown forces root.
+        if os.path.isfile(os.path.join(full, ".apex-root-owned")):
+            log.info(f"{proj}: .apex-root-owned — runs as root; skipping noroot user")
+            continue
         base = f"noroot-{_sanitize(proj)}"
         cand, i = base, 1
         while cand in used:

actions/utils/extract-traefik-certs.py

@@ -29,7 +29,7 @@ def run(ctx, args):
         ctx.sys.sudo(["find", abs_dest, "-type", "f", "-name", "*.pem", "-exec", "chmod", "600", "{}", "+"])
         ctx.sys.sudo(["chown", "-R", f"{os.getuid()}:{os.getgid()}", abs_dest])
         log.success("Certificates extracted.")
-        ctx.notify.success(TITLE, url, f"Certificates extracted for {node}", node)
+        ctx.notify.success(TITLE, url, f"Certificates extracted for {node}")
     except Exception as e:
-        log.error(str(e)); ctx.notify.error(TITLE, url, f"cert extraction failed: {e}", node)
+        log.error(str(e)); ctx.notify.error(TITLE, url, f"cert extraction failed: {e}")
         raise SystemExit(1)

assets/alloy/config.alloy

@@ -0,0 +1,131 @@
+// ============================================================================
+// APEX fleet observability agent — per-node Grafana Alloy config (shared).
+// Committed, env-parameterized: every node runs this SAME file; per-node values
+// come from sys.env(...) (APEX_NODE_HOST, push URLs, obs basic-auth creds).
+// Verified against grafana.com/docs/alloy/latest (Alloy v1.17.1).
+// ============================================================================
+// Model: PUSH. Metrics -> remote_write to a central Prometheus; logs -> loki.write
+// to a central Loki. Push endpoints + basic-auth are per-node env
+// (APEX_PROM_PUSH_URL, APEX_LOKI_PUSH_URL, APEX_OBS_BASIC_AUTH_USER/PASS).
+// ============================================================================
+
+// ----------------------------------------------------------------------------
+// METRICS
+// ----------------------------------------------------------------------------
+
+// Host metrics (node_exporter-equivalent). Container mounts host /proc,/sys,/
+// at /host/proc,/host/sys,/rootfs so the exporter reads the HOST.
+prometheus.exporter.unix "node" {
+  procfs_path = "/host/proc"
+  sysfs_path  = "/host/sys"
+  rootfs_path = "/rootfs"
+}
+
+// Container metrics (cadvisor-equivalent) via the docker socket.
+prometheus.exporter.cadvisor "containers" {
+  docker_host            = "unix:///var/run/docker.sock"
+  store_container_labels = true
+}
+
+// Stamp a static `node` label onto each exporter's targets.
+discovery.relabel "node_metrics" {
+  targets = prometheus.exporter.unix.node.targets
+  rule {
+    action       = "replace"
+    target_label = "node"
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+  // Stamp a stable `instance` (the node's public host) instead of the default,
+  // which is the Alloy container id/address.
+  rule {
+    action       = "replace"
+    target_label = "instance"
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+}
+discovery.relabel "container_metrics" {
+  targets = prometheus.exporter.cadvisor.containers.targets
+  rule {
+    action       = "replace"
+    target_label = "node"
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+  rule {
+    action       = "replace"
+    target_label = "instance"
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+}
+
+prometheus.scrape "node" {
+  targets         = discovery.relabel.node_metrics.output
+  forward_to      = [prometheus.remote_write.default.receiver]
+  scrape_interval = "15s"
+}
+prometheus.scrape "containers" {
+  targets         = discovery.relabel.container_metrics.output
+  forward_to      = [prometheus.remote_write.default.receiver]
+  scrape_interval = "15s"
+}
+
+// Push metrics to the central Prometheus (needs --web.enable-remote-write-receiver).
+prometheus.remote_write "default" {
+  endpoint {
+    url = sys.env("APEX_PROM_PUSH_URL")
+    basic_auth {
+      username = sys.env("APEX_OBS_BASIC_AUTH_USER")
+      password = sys.env("APEX_OBS_BASIC_AUTH_PASS")
+    }
+  }
+}
+
+// ----------------------------------------------------------------------------
+// LOGS
+// ----------------------------------------------------------------------------
+
+// systemd journal (host services, ufw, sshd, crowdsec, etc.).
+loki.source.journal "systemd" {
+  path    = "/var/log/journal"
+  max_age = "12h"
+  labels  = {
+    node = sys.env("APEX_NODE_HOST"),
+    job  = "systemd-journal",
+  }
+  forward_to = [loki.write.default.receiver]
+}
+
+// All docker containers' stdout/stderr.
+discovery.docker "containers" {
+  host = "unix:///var/run/docker.sock"
+}
+discovery.relabel "container_logs" {
+  targets = discovery.docker.containers.targets
+  rule {
+    action        = "replace"
+    source_labels = ["__meta_docker_container_name"]
+    regex         = "/?(.*)"
+    target_label  = "container"
+  }
+  rule {
+    action       = "replace"
+    target_label = "node"
+    replacement  = sys.env("APEX_NODE_HOST")
+  }
+}
+loki.source.docker "containers" {
+  host          = "unix:///var/run/docker.sock"
+  targets       = discovery.relabel.container_logs.output
+  relabel_rules = discovery.relabel.container_logs.rules
+  forward_to    = [loki.write.default.receiver]
+}
+
+// Push logs to the central Loki.
+loki.write "default" {
+  endpoint {
+    url = sys.env("APEX_LOKI_PUSH_URL")
+    basic_auth {
+      username = sys.env("APEX_OBS_BASIC_AUTH_USER")
+      password = sys.env("APEX_OBS_BASIC_AUTH_PASS")
+    }
+  }
+}

compositions/apex/docker-compose.yml

@@ -23,34 +23,52 @@ services:
     command: ["-c", "/etc/xray/config.jsonc"]
     restart: unless-stopped
     user: "0:0"
+    labels:
+      # backup (resticontainer, core/all-nodes): xray config only (logs + shm are ephemeral)
+      - "restic.enable=true"
+      - "restic.backup.paths=/etc/xray/config.jsonc"
     security_opt:
       - no-new-privileges:true
 
   # --- backups (host network, manual profile, one-shot) ---
-  restic:
+  # resticontainer (github.com/deytenit/resticontainer): a restic wrapper that
-    image: restic/restic:0.18.1
+  # discovers per-service backup intent from `restic.*` compose labels
-    container_name: apex-restic
+  # (restic.enable / restic.backup.paths / restic.backup.stop / restic.hooks.*).
+  # On `backup` it queries the docker daemon, runs each service's pre-hook (e.g.
+  # pg_dump), stops any container flagged restic.backup.stop, resolves the labeled
+  # container mounts to host paths (via the /srv mount), snapshots the union
+  # in a single restic run, then restarts + runs post-hooks. Every other subcommand
+  # (init/snapshots/forget/…) passes straight through to restic. Needs the docker
+  # socket (list/inspect/exec/stop/start) and /srv read-only (the fleet's data root)
+  # — a privileged manual one-shot, same trust surface as the old restic core.
+  resticontainer:
+    image: ghcr.io/deytenit/resticontainer:0.19.1
+    container_name: apex-resticontainer
     profiles: [manual]
     volumes:
       - ${APEX_TIER3}/restic/cache:/var/restic-cache
-      # FIX (Task 1.11): mount the backed-up tier roots read-only so `restic backup
+      - /var/run/docker.sock:/var/run/docker.sock          # query + exec/stop/start
-      # /data/@tier1 /data/@tier2` actually snapshots data (was missing today).
+      # apex tier data + checkouts live under /srv, and the @tierN bind mounts are
-      - ${APEX_TIER_ROOT1}:/data/@tier1:ro
+      # ABSOLUTE symlinks into /srv (compositions/<p>/@tierN -> /srv/com-…-tierN/<p>).
-      - ${APEX_TIER_ROOT2}:/data/@tier2:ro
+      # Mount /srv at the same path with RESTIC_HOSTFS=/ so restic follows those
+      # symlinks and records the real host paths — a bare /:/hostfs cannot, because
+      # the absolute symlink target /srv/… is dangling inside the container.
+      - /srv:/srv:ro                                        # read the data (resolves @tierN symlinks)
     environment:
       <<: *shared-environment
+      RESTIC_HOSTFS: "/"                                    # mount sources are already absolute (/srv/…)
       RESTIC_REPOSITORY: "${APEX_RESTIC_REPOSITORY?error}"
       RESTIC_PASSWORD: "${APEX_RESTIC_PASSWORD?error}"
       AWS_ACCESS_KEY_ID: "${APEX_RESTIC_AWS_ACCESS_KEY_ID?error}"
       AWS_SECRET_ACCESS_KEY: "${APEX_RESTIC_AWS_SECRET_ACCESS_KEY?error}"
       AWS_REGION: "ru-central1"
       AWS_ENDPOINT_URL: "https://storage.yandexcloud.net/"
-      # per-node override (icarus historically ran "max" for manual invocations;
+      # max compression is fleet policy (backup/run also forces --compression max on
-      # scheduled backups always force --compression auto on the CLI)
+      # the CLI; needs repo format v2, which a fresh restic init creates). Per-node
-      RESTIC_COMPRESSION: "${APEX_RESTIC_COMPRESSION:-auto}"
+      # override via APEX_RESTIC_COMPRESSION only if ever needed.
+      RESTIC_COMPRESSION: "${APEX_RESTIC_COMPRESSION:-max}"
       RESTIC_CACHE_DIR: /var/restic-cache
     network_mode: "host"
-    entrypoint: ["restic"]
 
   # --- docker socket proxy ---
   socket-proxy:
@@ -130,6 +148,10 @@ services:
       - "traefik.http.middlewares.apex-adguard-strip-prefix.stripprefix.prefixes=/adguard"
       - "traefik.http.routers.apex-adguard.service=apex-adguard"
       - "traefik.http.services.apex-adguard.loadbalancer.server.port=3000"
+      # backup (resticontainer, core/all-nodes): dynamic routing rules only
+      # (letsencrypt certs are regenerable via ACME/Cloudflare DNS-01)
+      - "restic.enable=true"
+      - "restic.backup.paths=/etc/traefik/dynamic"
     security_opt:
       - no-new-privileges:true
 
@@ -143,6 +165,10 @@ services:
       - ${APEX_TIER1}/adguard/conf:/opt/adguardhome/conf
     environment:
       <<: *shared-environment
+    labels:
+      # backup (resticontainer, core/all-nodes): adguard config only (work data regenerable)
+      - "restic.enable=true"
+      - "restic.backup.paths=/opt/adguardhome/conf"
     network_mode: "service:traefik"
     restart: unless-stopped
     cap_add:
@@ -152,27 +178,48 @@ services:
     security_opt:
       - no-new-privileges:true
 
-  # --- optional: loki log aggregation (node-local, enclave) ---
+  # --- optional: observability agent (profile-gated) ---
-  loki:
+  # Grafana Alloy: one agent per node. Collects host metrics (node_exporter-
-    image: grafana/loki:3.1.1
+  # equivalent) + container metrics (cadvisor) and PUSHES them via remote_write;
-    container_name: apex-loki
+  # collects systemd-journal + docker logs and PUSHES them to Loki. All targets
-    profiles: [loki]
+  # are OUTBOUND to the central gfn stack (Prometheus + Loki live on icarus), so
-    command: ["-config.file=/etc/loki/config.yaml"]
+  # Alloy needs egress (direct, not enclave). Replaces the old store-only `loki`
+  # service (which had no feeder). Config is a committed, env-parameterized asset.
+  alloy:
+    image: grafana/alloy:v1.17.1
+    container_name: apex-alloy
+    profiles: [alloy]
+    user: "0:0"  # root: required for host metrics, journal, and docker.sock
     volumes:
-      - ${APEX_TIER1}/loki/config.yaml:/etc/loki/config.yaml:ro
+      - ${APEX_COMMONS}/assets/alloy/config.alloy:/etc/alloy/config.alloy:ro
-      - ${APEX_TIER2}/loki/data:/loki
+      - ${APEX_TIER2}/alloy/data:/var/lib/alloy
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /proc:/host/proc:ro
+      - /sys:/host/sys:ro
+      - /:/rootfs:ro
+      - /var/log/journal:/var/log/journal:ro
+      - /run/log/journal:/run/log/journal:ro
+      - /etc/machine-id:/etc/machine-id:ro
+    environment:
+      <<: *shared-environment
+      APEX_NODE_HOST: "${APEX_NODE_HOST?error}"        # node + instance metric/log labels
+      # push targets + machine auth (per-node scalars in apex.env / secret .env)
+      APEX_LOKI_PUSH_URL: "${APEX_LOKI_PUSH_URL?error}"
+      APEX_PROM_PUSH_URL: "${APEX_PROM_PUSH_URL?error}"
+      APEX_OBS_BASIC_AUTH_USER: "${APEX_OBS_BASIC_AUTH_USER?error}"
+      APEX_OBS_BASIC_AUTH_PASS: "${APEX_OBS_BASIC_AUTH_PASS?error}"
+    command:
+      - run
+      - --server.http.listen-addr=0.0.0.0:12345
+      - --storage.path=/var/lib/alloy
+      - /etc/alloy/config.alloy
     networks:
-      enclave: {}
+      direct: {}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
     restart: unless-stopped
-    user: "${APEX_UID?error}:${APEX_GID?error}"
+    security_opt:
-    labels:
+      - no-new-privileges:true
-      # /loki/* is loki's native API prefix — no strip needed. basicauth users come
-      # from the node's uncommitted .env (only needed when the loki profile is on).
-      - "traefik.enable=true"
-      - "traefik.http.routers.apex-loki.rule=Host(`${APEX_NODE_FQDN}`) && PathPrefix(`/loki`)"
-      - "traefik.http.routers.apex-loki.middlewares=apex-loki-auth"
-      - "traefik.http.middlewares.apex-loki-auth.basicauth.users=${APEX_LOKI_BASIC_AUTH}"
-      - "traefik.http.services.apex-loki.loadbalancer.server.port=3100"
 
 networks:
   direct:

engine/context.py

@@ -81,7 +81,7 @@ class Context:
         self.ufw = Ufw(self.log, self.sys, self.host)
         self.crowdsec = Crowdsec(self.log, self.sys, self.tpl, self.host)
         self.systemd = Systemd(self.log, self.sys, self.tpl, self.host)
-        self.notify = Notify(self.log, name)
+        self.notify = Notify(self.log, node.host if node else "global")
         self._shadowed = shadowed
         self._commons = None
 
@@ -98,10 +98,9 @@ class Context:
         v["APEX_REPO_ROOT"] = self.paths.repo_root
         if self.node:
             v.update({
-                "APEX_NODE": self.node.name,
+                "APEX_NODE_HOST": self.node.host,
-                "APEX_CLUSTER": self.node.cluster,
-                "APEX_SUBNET": self.node.subnet,
                 "APEX_NODE_FQDN": self.node.fqdn,
+                "APEX_SUBNET": self.node.subnet,
                 "APEX_TIER1": self.paths.tier1, "APEX_TIER2": self.paths.tier2,
                 "APEX_TIER3": self.paths.tier3,
                 "APEX_TIER1_SHARED": self.paths.stier1,

engine/identity.py

@@ -1,29 +1,29 @@
 # engine/identity.py
-"""Node identity from host FQDN + node.env (replaces bash root_require_node).
+"""Node identity — fully explicit, from node.env. The framework assumes no domain
+and no clustering: a node declares its real FQDN and its public host name, and the
+short `name` is just the first label. The OS hostname is only a fallback for the FQDN.
 
-Decision B: the FQDN <node>.a<x>.apex.ermnvldmr.com keeps the cluster; we parse
+node.env keys:
-node+cluster from it. node.env owns APEX_SUBNET and mirrors APEX_CLUSTER; hostname
+  APEX_NODE_FQDN  real FQDN (traefik Host() routing).            default: OS hostname
-wins on drift. Falls back to the repo directory name when the FQDN is not yet
+  APEX_NODE_HOST  public host name (backup --host, obs/notify    default: APEX_NODE_FQDN
-apex-shaped (mid-cutover).
+                  `instance`); often the FQDN minus a cluster
+                  segment, but that's the node's choice.
+  APEX_SUBNET     the node's /24 for the `direct` network.       required
 """
 from __future__ import annotations
 
 import os
-import re
 import sys
 from dataclasses import dataclass
 from typing import Dict, List, Tuple
 
-_FQDN = re.compile(r"^(?P<node>[^.]+)\.(?P<cluster>a\d+)\.apex\.ermnvldmr\.com$")
-_DIRNAME = re.compile(r"^(?P<node>[^.]+)\.apex\.ermnvldmr\.com")
-
 
 @dataclass
 class Node:
-    name: str
+    name: str    # short label (first segment of the host), e.g. "icarus" — logs, branch, messages
-    cluster: str
+    subnet: str  # the node's direct-network /24
-    subnet: str
+    fqdn: str    # real FQDN — traefik Host() routing
-    fqdn: str
+    host: str    # public host name — backup --host, obs `instance`, notify instance
 
 
 def read_env(path: str) -> Dict[str, str]:
@@ -41,36 +41,24 @@ def read_env(path: str) -> Dict[str, str]:
     return out
 
 
-def resolve(fqdn: str, repo_root: str) -> Tuple[Node, List[str]]:
+def resolve(hostname: str, repo_root: str) -> Tuple[Node, List[str]]:
     """Return (Node, warnings). Exits 66 (EX_NOINPUT) if APEX_SUBNET is missing."""
     warns: List[str] = []
     env = read_env(os.path.join(repo_root, "node.env"))
 
-    m = _FQDN.match(fqdn or "")
+    fqdn = env.get("APEX_NODE_FQDN", "") or (hostname or "").strip()
-    if m:
+    if not fqdn:
-        name, cluster = m.group("node"), m.group("cluster")
+        sys.stderr.write("identity: APEX_NODE_FQDN missing from node.env and no hostname\n")
-        env_cluster = env.get("APEX_CLUSTER")
-        if env_cluster and env_cluster != cluster:
-            warns.append(
-                f"cluster drift: hostname says {cluster!r}, node.env says "
-                f"{env_cluster!r}; hostname wins."
-            )
-    else:
-        # Fallback: derive node from repo dir name; cluster from node.env.
-        warns.append(
-            f"FQDN {fqdn!r} is not apex-shaped; falling back to repo dir + node.env."
-        )
-        base = os.path.basename(os.path.realpath(repo_root))
-        dm = _DIRNAME.match(base)
-        name = dm.group("node") if dm else base
-        cluster = env.get("APEX_CLUSTER", "")
-        if not cluster:
-            sys.stderr.write("identity: no cluster in FQDN or node.env\n")
         raise SystemExit(66)
+    if not env.get("APEX_NODE_FQDN"):
+        warns.append(f"APEX_NODE_FQDN not set in node.env; using hostname {fqdn!r}.")
+
+    host = env.get("APEX_NODE_HOST", "") or fqdn      # node's public name; defaults to the FQDN
+    name = host.split(".")[0]                          # short label for logs/branch/messages
 
     subnet = env.get("APEX_SUBNET", "")
     if not subnet:
         sys.stderr.write("identity: APEX_SUBNET missing from node.env\n")
         raise SystemExit(66)
 
-    return Node(name=name, cluster=cluster, subnet=subnet, fqdn=fqdn), warns
+    return Node(name=name, subnet=subnet, fqdn=fqdn, host=host), warns

engine/lib/notify.py

@@ -1,9 +1,8 @@
 # engine/lib/notify.py
-"""ctx.notify — Telegram notifications (port of root_core_telegram*).
+"""ctx.notify — Telegram notifications.
 
-instance renamed com.ermnvldmr.root.<node> -> com.ermnvldmr.apex.<node>. POSTs
+The alert `instance` is the node's public host (APEX_NODE_HOST), passed in from the
-application/x-www-form-urlencoded via urllib (stdlib) — functionally identical to
+context. POSTs application/x-www-form-urlencoded via urllib (stdlib).
-the bash curl multipart for Telegram sendMessage.
 """
 from __future__ import annotations
 import html
@@ -20,7 +19,7 @@ def build_text(title: str, message: str, node: str, level: str, ts: str) -> str:
         severity = "critical"
     elif level == "WARN":
         severity = "warning"
-    instance = f"com.ermnvldmr.apex.{node}"
+    instance = node
     msg = message if len(message) <= 200 else message[:200] + "..."
     msg = html.escape(msg)
     safe_title = html.escape(title)

tests/test_identity.py

@@ -3,40 +3,37 @@ import os, tempfile, unittest
 from engine.identity import resolve, Node
 
 class TestIdentity(unittest.TestCase):
-    def _envfile(self, text, dirname=None):
+    def _envfile(self, text):
         d = tempfile.mkdtemp()
-        if dirname:
-            d = os.path.join(d, dirname)
-            os.makedirs(d)
         with open(os.path.join(d, "node.env"), "w") as f:
             f.write(text)
         return d
 
-    def test_fqdn_is_source_of_truth(self):
+    def test_explicit_host_and_fqdn(self):
-        repo = self._envfile("APEX_CLUSTER=a1\nAPEX_SUBNET=198.18.11.0/24\n")
+        repo = self._envfile("APEX_NODE_FQDN=icarus.a1.example.com\n"
-        n, warns = resolve("daedalus.a1.apex.ermnvldmr.com", repo)
+                             "APEX_NODE_HOST=icarus.example.com\n"
-        self.assertEqual((n.name, n.cluster, n.subnet), ("daedalus", "a1", "198.18.11.0/24"))
+                             "APEX_SUBNET=198.18.17.0/24\n")
+        n, warns = resolve("build-laptop", repo)   # hostname ignored when node.env has FQDN
+        self.assertEqual((n.name, n.host, n.fqdn, n.subnet),
+                         ("icarus", "icarus.example.com", "icarus.a1.example.com", "198.18.17.0/24"))
         self.assertEqual(warns, [])
 
-    def test_cluster_drift_warns_hostname_wins(self):
+    def test_host_defaults_to_fqdn(self):
-        repo = self._envfile("APEX_CLUSTER=a9\nAPEX_SUBNET=198.18.11.0/24\n")
+        repo = self._envfile("APEX_NODE_FQDN=n1.example.com\nAPEX_SUBNET=10.0.0.0/24\n")
-        n, warns = resolve("daedalus.a1.apex.ermnvldmr.com", repo)
+        n, warns = resolve("ignored", repo)
-        self.assertEqual(n.cluster, "a1")           # hostname wins
+        self.assertEqual((n.name, n.host, n.fqdn), ("n1", "n1.example.com", "n1.example.com"))
-        self.assertTrue(any("drift" in w.lower() for w in warns))
+        self.assertEqual(warns, [])
 
-    def test_fallback_when_fqdn_not_apex_shaped(self):
+    def test_fqdn_falls_back_to_hostname(self):
-        # repo dir basename carries the node name in fallback (real host layout:
+        repo = self._envfile("APEX_SUBNET=10.0.0.0/24\n")
-        # $HOME/morpheus.apex.ermnvldmr.com)
+        n, warns = resolve("box.local", repo)
-        repo = self._envfile("APEX_CLUSTER=a2\nAPEX_SUBNET=198.18.13.0/24\n",
+        self.assertEqual((n.name, n.host, n.fqdn), ("box", "box.local", "box.local"))
-                             dirname="morpheus.apex.ermnvldmr.com")
+        self.assertTrue(any("APEX_NODE_FQDN" in w for w in warns))
-        n, warns = resolve("build-laptop", repo)
-        self.assertEqual((n.name, n.cluster, n.subnet), ("morpheus", "a2", "198.18.13.0/24"))
-        self.assertTrue(any("fqdn" in w.lower() for w in warns))
 
     def test_missing_subnet_is_error(self):
-        repo = self._envfile("APEX_CLUSTER=a1\n")
+        repo = self._envfile("APEX_NODE_FQDN=n1.example.com\n")
         with self.assertRaises(SystemExit):
-            resolve("daedalus.a1.apex.ermnvldmr.com", repo)
+            resolve("n1.example.com", repo)
 
 if __name__ == "__main__":
     unittest.main()

tests/test_notify.py

@@ -3,10 +3,10 @@ import unittest
 from engine.lib.notify import build_text
 
 class TestNotify(unittest.TestCase):
-    def test_instance_uses_apex_prefix(self):
+    def test_instance_is_the_host(self):
-        t = build_text("Title", "msg", "daedalus", "INFO", "2026-07-04 00:00:00 MSK")
+        t = build_text("Title", "msg", "icarus.example.com", "INFO", "2026-07-04 00:00:00 MSK")
-        self.assertIn("com.ermnvldmr.apex.daedalus", t)
+        self.assertIn("icarus.example.com", t)
-        self.assertIn("#instance_com_ermnvldmr_apex_daedalus", t)
+        self.assertIn("#instance_icarus_example_com", t)   # dots -> underscores in the hashtag
 
     def test_html_escaped_and_truncated(self):
         t = build_text("A & B", "<script>" + "x" * 300, "d", "ERROR", "T")