feat: migrate to resticontainer with label-driven per-service backup

Swap the restic core service for resticontainer
(ghcr.io/deytenit/resticontainer:0.19.1): per-service backup intent via
restic.* compose labels (paths / stop / pre+post hooks), docker socket +
/:/hostfs:ro to resolve container mounts to host paths and snapshot the
union in one restic run. backup/run drops explicit /data/tier1,2 args
(paths now come from labels), keeps init/forget/notify, and guards on a
fresh snapshot. Snapshots are stamped --host <node>.apex.ermnvldmr.com
--tag biweekly and written with --compression max.

PARITY.md

@@ -12,21 +12,22 @@ action to its bash source and the behaviors that must match.
 | configure/systemd | actions/configure/systemd | render each unit -> /etc/systemd/system; daemon-reload; enable+start .service |
 | configure/routing | actions/configure/routing | ip_forward=1; fwmark 0x1->table100 rule; local default route table100 (idempotent) |
 | configure | (new umbrella) | run base,ufw,crowdsec,cron,systemd,routing in order |
-| sync/repository | actions/sync/repository | sync/<node> branch; add -A + commons submodule; commit; rebase origin/trunk; force-with-lease; telegram |
+| sync/repository | actions/sync/repository | sync/<node> branch; add -A + commons submodule; commit; rebase origin/main (was trunk in the monorepo); force-with-lease; telegram |
 | sync/packages | actions/sync/packages | apt --just-print upgrade Inst list; docker+skopeo digest compare; telegram info |
-| backup/run | actions/sync/tiers (restic-backup) | init-if-needed; backup /data/@tier1,2 --host --tag com-ermnvldmr-root-<node>; forget --prune 7/4/12; telegram |
+| backup/run | actions/sync/tiers (restic-backup) | init-if-needed; label-driven backup via resticontainer (discovers `restic.*`-labelled services, runs pre-hooks/stops, snapshots the resolved host paths) --host <node>.apex.ermnvldmr.com --tag biweekly --compression max; guard on a fresh snapshot; forget --prune 7/4/12; telegram |
 | tiers/link | actions/tiers/link | node + per-project @tierN symlinks; shared/ links; .env symlink |
 | tiers/useradd | actions/tiers/useradd | noroot-<proj> system users; noroot-shared group; APEX_UID/GID block in .env |
 | tiers/chown | actions/tiers/chown | {tier}/{proj}=uid:gid; {tier}/shared/{proj}=uid:sgid+sticky; base=root:root |
 | utils/extract-traefik-certs | actions/utils/extract-traefik-certs | traefik-certs-dumper v3; 600 perms; chown caller |
-| utils/generate-happ-subscriptions | actions/utils/generate-happ-subscriptions | build image; run generator input->output |
+| utils/generate-happ-subscriptions | actions/utils/generate-happ-subscriptions | MOVED to daedalus proprietaries (node-specific tooling; commons keeps only shared stuff) |
 | utils/lint-docker-compose | actions/utils/lint-docker-compose | dclint --fix via docker; --hook staged-only + restage; + anchor-IP check |
 | compose | actions/compose | apex core first then services alpha; reverse on down; --dry-run; extra passthrough |
 
 ## Deliberate deltas (rename + fixes)
 - root->apex everywhere (CLI, container names apex-*, telegram instance com.ermnvldmr.apex.*, ufw markers APEX.*, networks direct/enclave/socket).
 - No <node> arg; identity from FQDN + node.env.
-- restic core service gains read-only /data/@tier1,2 mounts (makes existing backup command functional).
+- backup superseded by the resticontainer migration (see below): the old whole-tier-root
+  restic snapshot is replaced by per-service, `restic.*`-label-driven backups.
 - ufw markers self-heal (delete ROOT + APEX blocks before inject).
 - envsubst -> string.Template.safe_substitute; wget/curl download -> urllib. Unset template
   vars stay LITERAL (envsubst emptied them) — typos surface instead of silently blanking.
@@ -48,10 +49,27 @@ action to its bash source and the behaviors that must match.
   restored via an icarus-only override layered in the node repo's include (Phase 2).
 - icarus: ACME propagation-tuning flags and DEBUG log level drop to the shared defaults;
   traefik now runs noroot (APEX_UID) with file provider + HTTP/3 like daedalus.
-- morpheus: gains apex-xray (design: xray is a core service on every node) — a config
-  must exist at `${APEX_TIER1}/xray/data/config.jsonc` before cutover or it crash-loops.
+- xray is profile-gated (`xray` in COMPOSE_PROFILES): daedalus + icarus opt in,
+  morpheus does not (it never ran xray — owner decision 2026-07-05).
+- loki (profile `loki`, no node opts in yet) is exposed via traefik at
+  `https://<node-fqdn>/loki` behind basicauth (`APEX_LOKI_BASIC_AUTH` in the node .env).
 - morpheus: dashboard auth changes basicauth (TRAEFIK_INTERNAL_API_AUTH) -> enclave IP
   allowlist; the adguard router labels exist without a backend (502, was 404, on /adguard).
 - restic: per-node selective tier2 mounts replaced by whole-tier-root RO mounts
   (superset; snapshot paths preserved); RESTIC_COMPRESSION per-node via
   `APEX_RESTIC_COMPRESSION` (default auto).
+
+## resticontainer migration (SP4 realized)
+- The `restic` core service is replaced by `resticontainer`
+  (`ghcr.io/deytenit/resticontainer`), a restic wrapper that reads per-service
+  `restic.*` compose labels: `restic.enable`, `restic.backup.paths` (comma-separated
+  container mount destinations — matched exactly, whole mount dir backed up),
+  `restic.backup.stop`, and `restic.hooks.pre-backup`/`post-backup` (run via docker exec).
+- It mounts the docker socket (list/inspect/exec/stop/start) and the host root
+  `/:/hostfs:ro` (`RESTIC_HOSTFS=/hostfs`); it resolves each labelled mount to
+  `/hostfs` + the mount's host source and snapshots the union in one restic run.
+- The whole-tier-root RO mounts and the explicit `/data/@tier1,2` backup args are gone;
+  a service is backed up only if it carries `restic.enable=true`. DB consistency is
+  per-service: a pre-backup hook dumps into a dedicated backed-up mount (e.g. postgres
+  `pg_dump … > /dumps/dump.sql`), never the live data dir. `backup/run` now guards on a
+  fresh snapshot landing for the host (a no-label repo would otherwise be a silent no-op).

README.md

@@ -3,15 +3,19 @@
 The **commons** of the apex fleet: a stock-python engine (`apex <group>/<action>`) and
 the shared core docker-compose, consumed by every node as the `commons/` git submodule.
 
+Full documentation: [docs.ermnvldmr.com/en/apex](https://docs.ermnvldmr.com/en/apex/).
+
 ## Layout
 - `apex` — launcher (python3, stdlib only); node `init.sh` symlinks it onto `PATH`.
 - `engine/` — the runner: CLI, identity, overlay, context, and the `ctx.*` helper library.
 - `actions/` — grouped actions; the path IS the name (`configure/ufw`, `tiers/link`, …).
 - `compositions/apex/docker-compose.yml` — the shared, `${env}`-parameterized core compose.
 - `vendor/` — pinned pure-python third-party (empty; see `vendor/VENDORED.md`).
-- `source/` — independent containerized tools (happ subscription generator).
 - `tests/` — stdlib `unittest` suite (`tests/run.sh`).
 
+Node-specific tooling lives in each node repo's `proprietaries/` — commons holds only
+what every node shares.
+
 ## Usage (on a node)
 `apex` acts on the **local checkout** — no `<node>` argument. Identity comes from the
 host FQDN (`<node>.a<x>.apex.ermnvldmr.com`) and `node.env`.

actions/backup/run.py

@@ -1,17 +1,23 @@
 # actions/backup/run.py
-"""Port of sync/tiers restic-backup (SP4: behavior-identical). Runs the restic core
-service one-shot: init-if-needed -> backup tiers -> forget --prune -> Telegram."""
+"""Label-driven restic backup via resticontainer (github.com/deytenit/resticontainer).
+Runs the resticontainer core service one-shot: init-if-needed -> backup (discovers
+`restic.*`-labelled services, runs their pre-hooks/stops, snapshots the union of the
+resolved host paths in a single restic run) -> forget --prune -> Telegram. Every
+non-`backup` subcommand passes straight through to restic, so init/snapshots/forget
+behave exactly as before."""
+import json
 import os
 from engine.descriptor import Meta, Arg
 
-METADATA = Meta(summary="Run restic backup of the backed-up tiers (init-if-needed, backup, forget --prune).",
+METADATA = Meta(summary="Run the resticontainer label-driven backup (init-if-needed, backup, forget --prune).",
                 args=[Arg("telegram_bot_url", "Telegram Bot URL for notifications")])
 TITLE = "Restic Host Backup"
 
 def run(ctx, args):
     log, node, url = ctx.log, ctx.node.name, args.telegram_bot_url
     core = ctx.paths.core
-    tag = f"com-ermnvldmr-root-{node}"          # KEPT unchanged for backup continuity (SP4/Decision D)
+    host = f"{node}.apex.ermnvldmr.com"         # restic --host: the per-node public name (matches the bucket)
+    tag = "biweekly"                            # restic --tag: snapshot cadence (this cron fires on the 5th & 20th)
     if not os.path.isfile(os.path.join(core, "docker-compose.yml")):
         log.error(f"Core compose not found in {core}"); raise SystemExit(1)
     env = dict(os.environ, COMPOSE_PROFILES="manual")
@@ -21,9 +27,20 @@ def run(ctx, args):
         if os.path.isfile(os.path.join(core, ef)):
             envfiles += ["--env-file", ef]
 
-    def restic(*cmd, check=True):
+    def restic(*cmd, check=True, capture=False):
         return ctx.sys.run(["docker", "compose", *envfiles, "--profile", "manual",
-                            "run", "--rm", "restic", *cmd], cwd=core, env=env, check=check)
+                            "run", "--rm", "-T", "resticontainer", *cmd],
+                           cwd=core, env=env, check=check, capture=capture)
+
+    def host_snapshot_ids():
+        """This host's snapshot IDs, or None if it can't be determined (never fail on this)."""
+        cp = restic("snapshots", "--host", host, "--json", check=False, capture=True)
+        if cp.returncode != 0:
+            return None
+        try:
+            return {s.get("id") for s in json.loads(cp.stdout or "[]")}
+        except (ValueError, TypeError):
+            return None
 
     try:
         log.info("Checking restic repository availability...")
@@ -31,10 +48,20 @@ def run(ctx, args):
             log.info("Restic repository reachable.")
         elif restic("init", check=False).returncode != 0:
             ctx.notify.error(TITLE, url, "restic init failed", node); raise SystemExit(1)
-        log.info("Backing up /data/@tier1 and /data/@tier2 ...")
-        if restic("backup", "/data/@tier1", "/data/@tier2", "--host", node, "--tag", tag,
-                  "--compression", "auto", check=False).returncode != 0:
+        before = host_snapshot_ids()
+        log.info("Backing up `restic.*`-labelled services (resticontainer discovers paths)...")
+        if restic("backup", "--host", host, "--tag", tag, "--compression", "max",
+                  check=False).returncode != 0:
             ctx.notify.error(TITLE, url, "restic backup failed", node); raise SystemExit(1)
+        # Guard: the label-driven backup is a silent no-op if no service carries the
+        # `restic.enable` label (resticontainer finds zero paths and skips restic).
+        # Prove a fresh snapshot for this host actually landed before we prune/notify.
+        after = host_snapshot_ids()
+        if before is not None and after is not None and not (after - before):
+            log.error("No new snapshot produced — refusing to report success.")
+            ctx.notify.error(TITLE, url,
+                             "backup produced no new snapshot (no restic.* labels discovered?)", node)
+            raise SystemExit(1)
         log.info("Applying retention (forget --prune)...")
         if restic("forget", "--prune", "--keep-daily", "7", "--keep-weekly", "4",
                   "--keep-monthly", "12", check=False).returncode != 0:

actions/sync/repository.py

@@ -1,10 +1,10 @@
 # actions/sync/repository.py
-"""Port of sync/repository — capture-up commit + rebase onto origin/trunk + force-push."""
+"""Port of sync/repository — capture-up commit + rebase onto origin/main + force-push."""
 import os
 from datetime import datetime
 from engine.descriptor import Meta, Arg
 
-METADATA = Meta(summary="Capture-up: commit node changes, rebase onto origin/trunk, force-push sync/<node>.",
+METADATA = Meta(summary="Capture-up: commit node changes, rebase onto origin/main, force-push sync/<node>.",
                 args=[Arg("telegram_bot_url", "Telegram Bot URL for notifications")])
 TITLE = "Annual Remote Sync"
 
@@ -41,8 +41,8 @@ def run(ctx, args):
             git("commit", "-m", f"[{node}] Auto-commit: {ts}")
         else:
             log.info("No changes to commit.")
-        git("fetch", "origin", "trunk")
-        if git("rebase", "origin/trunk", check=False).returncode != 0:
+        git("fetch", "origin", "main")
+        if git("rebase", "origin/main", check=False).returncode != 0:
             log.error("Rebase conflict — aborting rebase; manual intervention required.")
             git("rebase", "--abort", check=False)
             ctx.notify.error(TITLE, url, "Rebase conflict. Manual intervention required.", node)

actions/tiers/chown.py

@@ -46,6 +46,16 @@ def run(ctx, args):
         full = os.path.join(comp, proj)
         if not os.path.isdir(full) or proj.startswith(".") or proj.startswith("@"):
             continue
+        # opt-out: .apex-root-owned forces this project's tier data (dir + children,
+        # recursively) to root:root — for services that run as root and reject
+        # non-root-owned data (e.g. seafile-mc). Blocks noroot ownership entirely.
+        if os.path.isfile(os.path.join(full, ".apex-root-owned")):
+            log.info(f"Applying root ownership for {proj} (.apex-root-owned)")
+            for tp in tiers:
+                for d in (os.path.join(tp, proj), os.path.join(tp, "shared", proj)):
+                    if os.path.isdir(d):
+                        s.sudo(["chown", "-R", "root:root", d]); log.success(f"  Owned {d} (root)")
+            continue
         env_path = os.path.join(full, ".env")
         uid = _env_val(env_path, "APEX_UID")
         gid = _env_val(env_path, "APEX_GID")

actions/tiers/useradd.py

@@ -34,6 +34,12 @@ def run(ctx, args):
         full = os.path.join(comp, proj)
         if not os.path.isdir(full) or proj.startswith(".") or proj.startswith("@"):
             continue
+        # opt-out: a project marked .apex-root-owned runs as root and keeps its tier
+        # data root-owned (e.g. seafile-mc, whose entrypoint refuses non-root-owned
+        # data). Skip the noroot user + .env UID injection; tiers/chown forces root.
+        if os.path.isfile(os.path.join(full, ".apex-root-owned")):
+            log.info(f"{proj}: .apex-root-owned — runs as root; skipping noroot user")
+            continue
         base = f"noroot-{_sanitize(proj)}"
         cand, i = base, 1
         while cand in used:

actions/utils/generate-happ-subscriptions.py

@@ -1,30 +0,0 @@
-# actions/utils/generate-happ-subscriptions.py
-"""Port of utils/generate-happ-subscriptions — build + run the happ generator image."""
-import os
-import shutil
-from engine.descriptor import Meta, Arg
-
-METADATA = Meta(summary="Generate .psub files from a JSON config via the happ generator (docker).",
-                args=[Arg("input_json", "input JSON config"), Arg("output_dir", "output directory")])
-IMAGE = "happ-subscription-generator"
-
-def run(ctx, args):
-    log = ctx.log
-    src_dir = os.path.join(ctx.paths.commons, "source", "happ-subscription-generator")
-    inp, out = os.path.realpath(args.input_json), os.path.realpath(args.output_dir)
-    # GNU realpath (bash) fails unless all but the last component exist — without this
-    # guard a mistyped path reaches `docker run -v`, which creates it root-owned.
-    for label, p in (("input_json", inp), ("output_dir", out)):
-        parent = os.path.dirname(p)
-        if not os.path.isdir(parent):
-            log.error(f"Cannot resolve {label}: no such directory: {parent}")
-            raise SystemExit(1)
-    if not shutil.which("docker"):
-        log.error("docker is not installed or not in PATH.")
-        raise SystemExit(1)
-    log.info(f"Building {IMAGE}...")
-    ctx.sys.run(["docker", "build", "-t", IMAGE, src_dir], check=True)
-    log.info(f"Running generator: {inp} -> {out}")
-    ctx.sys.run(["docker", "run", "--rm", "-v", f"{inp}:/app/input.json:ro",
-                 "-v", f"{out}:/app/output:rw", IMAGE, "/app/input.json", "/app/output"], check=True)
-    log.success("Subscription generation completed.")

assets/alloy/config.alloy

@@ -0,0 +1,120 @@
+// ============================================================================
+// APEX fleet observability agent — per-node Grafana Alloy config (shared).
+// Committed, env-parameterized: every node runs this SAME file; per-node values
+// come from sys.env(...) (APEX_NODE, push URLs, obs basic-auth creds).
+// Verified against grafana.com/docs/alloy/latest (Alloy v1.17.1).
+// ============================================================================
+// Model: PUSH. Metrics -> remote_write to the central Prometheus (on icarus);
+// logs -> loki.write to the central Loki (on icarus). daedalus/morpheus reach
+// them through the public gfn.ermnvldmr.com front (authelia-bypassed + basicauth
+// on /loki and /prometheus/api/v1/write); icarus resolves gfn locally.
+// ============================================================================
+
+// ----------------------------------------------------------------------------
+// METRICS
+// ----------------------------------------------------------------------------
+
+// Host metrics (node_exporter-equivalent). Container mounts host /proc,/sys,/
+// at /host/proc,/host/sys,/rootfs so the exporter reads the HOST.
+prometheus.exporter.unix "node" {
+  procfs_path = "/host/proc"
+  sysfs_path  = "/host/sys"
+  rootfs_path = "/rootfs"
+}
+
+// Container metrics (cadvisor-equivalent) via the docker socket.
+prometheus.exporter.cadvisor "containers" {
+  docker_host            = "unix:///var/run/docker.sock"
+  store_container_labels = true
+}
+
+// Stamp a static `node` label onto each exporter's targets.
+discovery.relabel "node_metrics" {
+  targets = prometheus.exporter.unix.node.targets
+  rule {
+    action       = "replace"
+    target_label = "node"
+    replacement  = sys.env("APEX_NODE")
+  }
+}
+discovery.relabel "container_metrics" {
+  targets = prometheus.exporter.cadvisor.containers.targets
+  rule {
+    action       = "replace"
+    target_label = "node"
+    replacement  = sys.env("APEX_NODE")
+  }
+}
+
+prometheus.scrape "node" {
+  targets         = discovery.relabel.node_metrics.output
+  forward_to      = [prometheus.remote_write.default.receiver]
+  scrape_interval = "15s"
+}
+prometheus.scrape "containers" {
+  targets         = discovery.relabel.container_metrics.output
+  forward_to      = [prometheus.remote_write.default.receiver]
+  scrape_interval = "15s"
+}
+
+// Push metrics to the central Prometheus (needs --web.enable-remote-write-receiver).
+prometheus.remote_write "default" {
+  endpoint {
+    url = sys.env("APEX_PROM_PUSH_URL")
+    basic_auth {
+      username = sys.env("APEX_OBS_BASIC_AUTH_USER")
+      password = sys.env("APEX_OBS_BASIC_AUTH_PASS")
+    }
+  }
+}
+
+// ----------------------------------------------------------------------------
+// LOGS
+// ----------------------------------------------------------------------------
+
+// systemd journal (host services, ufw, sshd, crowdsec, etc.).
+loki.source.journal "systemd" {
+  path    = "/var/log/journal"
+  max_age = "12h"
+  labels  = {
+    node = sys.env("APEX_NODE"),
+    job  = "systemd-journal",
+  }
+  forward_to = [loki.write.default.receiver]
+}
+
+// All docker containers' stdout/stderr.
+discovery.docker "containers" {
+  host = "unix:///var/run/docker.sock"
+}
+discovery.relabel "container_logs" {
+  targets = discovery.docker.containers.targets
+  rule {
+    action        = "replace"
+    source_labels = ["__meta_docker_container_name"]
+    regex         = "/?(.*)"
+    target_label  = "container"
+  }
+  rule {
+    action       = "replace"
+    target_label = "node"
+    replacement  = sys.env("APEX_NODE")
+  }
+}
+loki.source.docker "containers" {
+  host          = "unix:///var/run/docker.sock"
+  targets       = discovery.relabel.container_logs.output
+  relabel_rules = discovery.relabel.container_logs.rules
+  forward_to    = [loki.write.default.receiver]
+}
+
+// Push logs to the central Loki.
+loki.write "default" {
+  endpoint {
+    url = sys.env("APEX_LOKI_PUSH_URL")
+    basic_auth {
+      username = sys.env("APEX_OBS_BASIC_AUTH_USER")
+      password = sys.env("APEX_OBS_BASIC_AUTH_PASS")
+    }
+  }
+}

compositions/apex/docker-compose.yml

@@ -7,10 +7,11 @@ x-environment: &shared-environment
   TZ: "Europe/Moscow"
 
 services:
-  # --- edge (host network) ---
+  # --- edge (host network; profile-gated — a node opts in via COMPOSE_PROFILES) ---
   xray:
     image: ghcr.io/xtls/xray-core:26.3.27
     container_name: apex-xray
+    profiles: [xray]
     volumes:
       - ${APEX_TIER1}/xray/data/config.jsonc:/etc/xray/config.jsonc:ro
       - ${APEX_TIER1_SHARED}/apex/letsencrypt:/etc/letsencrypt/live:ro
@@ -26,30 +27,39 @@ services:
       - no-new-privileges:true
 
   # --- backups (host network, manual profile, one-shot) ---
-  restic:
-    image: restic/restic:0.18.1
-    container_name: apex-restic
+  # resticontainer (github.com/deytenit/resticontainer): a restic wrapper that
+  # discovers per-service backup intent from `restic.*` compose labels
+  # (restic.enable / restic.backup.paths / restic.backup.stop / restic.hooks.*).
+  # On `backup` it queries the docker daemon, runs each service's pre-hook (e.g.
+  # pg_dump), stops any container flagged restic.backup.stop, resolves the labeled
+  # container mounts to host paths (via /hostfs + mount source), snapshots the union
+  # in a single restic run, then restarts + runs post-hooks. Every other subcommand
+  # (init/snapshots/forget/…) passes straight through to restic. Needs the docker
+  # socket (list/inspect/exec/stop/start) and the host root read-only (to read the
+  # data) — a privileged manual one-shot, same trust surface as the old restic core.
+  resticontainer:
+    image: ghcr.io/deytenit/resticontainer:0.19.1
+    container_name: apex-resticontainer
     profiles: [manual]
     volumes:
       - ${APEX_TIER3}/restic/cache:/var/restic-cache
-      # FIX (Task 1.11): mount the backed-up tier roots read-only so `restic backup
-      # /data/@tier1 /data/@tier2` actually snapshots data (was missing today).
-      - ${APEX_TIER_ROOT1}:/data/@tier1:ro
-      - ${APEX_TIER_ROOT2}:/data/@tier2:ro
+      - /var/run/docker.sock:/var/run/docker.sock          # query + exec/stop/start
+      - /:/hostfs:ro                                        # read the data off the host
     environment:
       <<: *shared-environment
+      RESTIC_HOSTFS: /hostfs                                # prefix for resolved mount sources
       RESTIC_REPOSITORY: "${APEX_RESTIC_REPOSITORY?error}"
       RESTIC_PASSWORD: "${APEX_RESTIC_PASSWORD?error}"
       AWS_ACCESS_KEY_ID: "${APEX_RESTIC_AWS_ACCESS_KEY_ID?error}"
       AWS_SECRET_ACCESS_KEY: "${APEX_RESTIC_AWS_SECRET_ACCESS_KEY?error}"
       AWS_REGION: "ru-central1"
       AWS_ENDPOINT_URL: "https://storage.yandexcloud.net/"
-      # per-node override (icarus historically ran "max" for manual invocations;
-      # scheduled backups always force --compression auto on the CLI)
-      RESTIC_COMPRESSION: "${APEX_RESTIC_COMPRESSION:-auto}"
+      # max compression is fleet policy (backup/run also forces --compression max on
+      # the CLI; needs repo format v2, which a fresh restic init creates). Per-node
+      # override via APEX_RESTIC_COMPRESSION only if ever needed.
+      RESTIC_COMPRESSION: "${APEX_RESTIC_COMPRESSION:-max}"
       RESTIC_CACHE_DIR: /var/restic-cache
     network_mode: "host"
-    entrypoint: ["restic"]
 
   # --- docker socket proxy ---
   socket-proxy:
@@ -151,19 +161,48 @@ services:
     security_opt:
       - no-new-privileges:true
 
-  # --- optional: loki log aggregation (node-local, enclave) ---
-  loki:
-    image: grafana/loki:3.1.1
-    container_name: apex-loki
-    profiles: [loki]
-    command: ["-config.file=/etc/loki/config.yaml"]
+  # --- optional: observability agent (profile-gated) ---
+  # Grafana Alloy: one agent per node. Collects host metrics (node_exporter-
+  # equivalent) + container metrics (cadvisor) and PUSHES them via remote_write;
+  # collects systemd-journal + docker logs and PUSHES them to Loki. All targets
+  # are OUTBOUND to the central gfn stack (Prometheus + Loki live on icarus), so
+  # Alloy needs egress (direct, not enclave). Replaces the old store-only `loki`
+  # service (which had no feeder). Config is a committed, env-parameterized asset.
+  alloy:
+    image: grafana/alloy:v1.17.1
+    container_name: apex-alloy
+    profiles: [alloy]
+    user: "0:0"  # root: required for host metrics, journal, and docker.sock
     volumes:
-      - ${APEX_TIER1}/loki/config.yaml:/etc/loki/config.yaml:ro
-      - ${APEX_TIER2}/loki/data:/loki
+      - ${APEX_COMMONS}/assets/alloy/config.alloy:/etc/alloy/config.alloy:ro
+      - ${APEX_TIER2}/alloy/data:/var/lib/alloy
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /proc:/host/proc:ro
+      - /sys:/host/sys:ro
+      - /:/rootfs:ro
+      - /var/log/journal:/var/log/journal:ro
+      - /run/log/journal:/run/log/journal:ro
+      - /etc/machine-id:/etc/machine-id:ro
+    environment:
+      <<: *shared-environment
+      APEX_NODE: "${APEX_NODE?error}"
+      # push targets + machine auth (per-node scalars in apex.env / secret .env)
+      APEX_LOKI_PUSH_URL: "${APEX_LOKI_PUSH_URL?error}"
+      APEX_PROM_PUSH_URL: "${APEX_PROM_PUSH_URL?error}"
+      APEX_OBS_BASIC_AUTH_USER: "${APEX_OBS_BASIC_AUTH_USER?error}"
+      APEX_OBS_BASIC_AUTH_PASS: "${APEX_OBS_BASIC_AUTH_PASS?error}"
+    command:
+      - run
+      - --server.http.listen-addr=0.0.0.0:12345
+      - --storage.path=/var/lib/alloy
+      - /etc/alloy/config.alloy
     networks:
-      enclave: {}
+      direct: {}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
     restart: unless-stopped
-    user: "${APEX_UID?error}:${APEX_GID?error}"
+    security_opt:
+      - no-new-privileges:true
 
 networks:
   direct:

source/happ-subscription-generator/Dockerfile

@@ -1,7 +0,0 @@
-FROM python:3.11-alpine
-
-WORKDIR /app
-
-COPY generator.py .
-
-ENTRYPOINT ["python", "/app/generator.py"]

source/happ-subscription-generator/README.md

@@ -1,35 +0,0 @@
-# HAPP Subscription Generator
-
-**Disclaimer:** This tool is intended exclusively for **educational purposes** and **private networking** research. It is designed to demonstrate how to programmatically generate and manage subscription configurations for distributed network components.
-
-This tool generates `.psub` subscription files from a JSON configuration. It supports global routing via `happ://` links and multiple VLESS connection strings.
-
-## Features
-- **Global Routing**: Encodes your `routing` object into the `happ://routing/onadd/` link.
-- **Templates**: Define common Xray-core settings once and reuse them across users.
-- **Deep Merge**: User-specific `xray` overrides are merged with templates.
-- **Auto Remarks**: Generates readable titles based on country, protocol, and node names.
-- **ISO Lookup**: Automatically converts country codes (e.g., `US`) to names and emojis.
-
-## Usage
-
-The tool is designed to be run via the operator script:
-
-```bash
-./.operator/scripts/ops/generate-happ-subscriptions <input_json> <output_dir>
-```
-
-### JSON Format Specification
-
-The configuration file consists of three main top-level objects:
-
-1.  **`routing`**: Global routing settings (DNS, direct/proxy rules, etc.).
-2.  **`templates`**: Named connection templates containing shared protocol and stream settings.
-3.  **`users`**: List of users, each with a unique subscription key (`psub`) and a set of configurations derived from templates.
-
-#### Example Reference
-See [example.json](./example.json) for a comprehensive template of the configuration format.
-
-## Implementation Details
-- **Logic**: Written in Python 3.11 (standard library only).
-- **Isolation**: Runs inside a Docker container (`python:3.11-alpine`).

source/happ-subscription-generator/example.json

@@ -1,141 +0,0 @@
-{
-  "routing": {
-    "name": "enterprise-secure-access",
-    "GlobalProxy": "false",
-    "RouteOrder": "block-proxy-direct",
-    "RemoteDNSType": "DoU",
-    "RemoteDNSDomain": "internal.corp",
-    "RemoteDNSIP": "10.50.10.10",
-    "DomesticDNSType": "DoH",
-    "DomesticDNSDomain": "https://dns.example.com/query",
-    "DomesticDNSIP": "1.1.1.1",
-    "Geoipurl": "https://example.com/data/geoip.dat",
-    "Geositeurl": "https://example.com/data/geosite.dat",
-    "LastUpdated": "2026-04-14",
-    "DnsHosts": {
-      "dns.example.com": "1.1.1.1",
-      "gateway.internal.corp": "10.50.10.1"
-    },
-    "DirectSites": [
-      "geosite:google",
-      "geosite:microsoft",
-      "geosite:private",
-      "domain:internal.corp",
-      "regexp:.*\\.local$"
-    ],
-    "DirectIp": [
-      "geoip:private",
-      "10.0.0.0/8",
-      "172.16.0.0/12",
-      "192.168.0.0/16"
-    ],
-    "ProxyIp": [
-      "10.50.0.0/16",
-      "10.60.0.0/16",
-      "172.20.0.0/14"
-    ],
-    "DomainStrategy": "IPIfNonMatch",
-    "FakeDNS": "false",
-    "UseChunkFiles": "true"
-  },
-  "templates": {
-    "corp-vpn-v1": {
-      "country": "US",
-      "nodes": ["hq-gateway-01"],
-      "xray": {
-        "protocol": "vless",
-        "settings": {
-          "address": "vpn-hq.example.net",
-          "port": 443,
-          "encryption": "none",
-          "flow": "xtls-rprx-vision"
-        },
-        "streamSettings": {
-          "network": "raw",
-          "security": "tls",
-          "tlsSettings": {
-            "serverName": "vpn-hq.example.net",
-            "allowInsecure": false,
-            "fingerprint": "chrome",
-            "alpn": ["h2", "http/1.1"]
-          }
-        }
-      }
-    },
-    "corp-vpn-v2": {
-      "country": "UK",
-      "nodes": ["branch-office-uk"],
-      "xray": {
-        "protocol": "vless",
-        "settings": {
-          "address": "vpn-uk.example.net",
-          "port": 443,
-          "encryption": "none",
-          "flow": "xtls-rprx-vision"
-        },
-        "streamSettings": {
-          "network": "xhttp",
-          "security": "tls",
-          "tlsSettings": {
-            "serverName": "vpn-uk.example.net",
-            "fingerprint": "chrome",
-            "alpn": ["h3", "h2"]
-          },
-          "xhttpSettings": {
-            "mode": "stream-one",
-            "path": "/internal-access-bridge/"
-          }
-        }
-      }
-    },
-    "secure-enclave": {
-      "country": "DE",
-      "nodes": ["secure-zone-alpha"],
-      "xray": {
-        "protocol": "vless",
-        "settings": {
-          "address": "secure.example.net",
-          "port": 443,
-          "encryption": "none",
-          "flow": "xtls-rprx-vision"
-        },
-        "streamSettings": {
-          "network": "raw",
-          "security": "reality",
-          "realitySettings": {
-            "dest": "internal-app.example.net",
-            "fingerprint": "chrome",
-            "serverNames": [
-              "internal-app.example.net"
-            ],
-            "publicKey": "base64-encoded-public-key-placeholder"
-          }
-        },
-        "mux": {
-          "enabled": true,
-          "concurrency": 8
-        }
-      }
-    }
-  },
-  "users": [
-    {
-      "name": "employee-01",
-      "psub": "sub-key-dev-01",
-      "configs": [
-        { "template": "corp-vpn-v1", "xray": { "settings": { "id": "uuid-placeholder-01" } } },
-        { "template": "corp-vpn-v2", "xray": { "settings": { "id": "uuid-placeholder-01" } } },
-        { "template": "secure-enclave", "xray": { "settings": { "id": "uuid-placeholder-01" }, "streamSettings": { "realitySettings": { "shortId": "short-id-01" } } } }
-      ]
-    },
-    {
-      "name": "employee-02",
-      "psub": "sub-key-admin-02",
-      "configs": [
-        { "template": "corp-vpn-v1", "xray": { "settings": { "id": "uuid-placeholder-02" } } },
-        { "template": "corp-vpn-v2", "xray": { "settings": { "id": "uuid-placeholder-02" } } },
-        { "template": "secure-enclave", "xray": { "settings": { "id": "uuid-placeholder-02" }, "streamSettings": { "realitySettings": { "shortId": "short-id-02" } } } }
-      ]
-    }
-  ]
-}

source/happ-subscription-generator/generator.py

@@ -1,128 +0,0 @@
-#!/usr/bin/env python3
-"""
-HAPP subscription generator.
-
-Produces one .psub file per user. Each file contains:
-  line 1 : happ://routing/onadd/<base64>   (routing profile, unchanged behaviour)
-  line N : a share link for every config the user has
-
-Design
-------
-The config is written 1:1 against the HAPP / Xray share-link format. A template
-is just the literal pieces of a URI:
-
-    scheme   ->  vless:// , hy2:// , trojan:// , ss:// , socks:// , vmess://
-    host     ->  the @host part
-    port     ->  the :port part
-    id       ->  the userinfo before @  (uuid / password / auth)
-    params   ->  the ?a=b&c=d query, keys are the REAL link keys
-                 (type, security, sni, fp, alpn, flow, encryption,
-                  pbk, sid, spx, mode, path, host, serviceName,
-                  headerType, mux, obfs, ... whatever the link supports)
-
-Nothing is translated or guessed. Whatever you put in `params` is what goes in
-the link, url-encoded. This keeps the generator dumb and the config honest.
-
-Per-user `configs` entries deep-merge onto the named template, so a user only
-supplies the bits that differ (their `id`, their `params.sid`, etc.).
-
-The `#remark` is built from template metadata (country / nodes) + the user name.
-"""
-
-import json
-import base64
-import sys
-import os
-import copy
-from urllib.parse import urlencode, quote
-
-# ISO 3166-1 alpha-2 -> (flag, display name). Extend as needed.
-COUNTRY_MAP = {
-    "LV": ("\U0001F1F1\U0001F1FB", "Latvia"),
-    "RU": ("\U0001F1F7\U0001F1FA", "Russia"),
-    "KZ": ("\U0001F1F0\U0001F1FF", "Kazakhstan"),
-    "BY": ("\U0001F1E7\U0001F1FE", "Belarus"),
-}
-
-
-def deep_merge(base, override):
-    """Recursively merge `override` into a copy of `base`. Lists/scalars replace."""
-    out = copy.deepcopy(base)
-    for key, value in override.items():
-        if isinstance(out.get(key), dict) and isinstance(value, dict):
-            out[key] = deep_merge(out[key], value)
-        else:
-            out[key] = copy.deepcopy(value)
-    return out
-
-
-def routing_link(routing):
-    """happ://routing/onadd/<base64(json)>"""
-    payload = json.dumps(routing, separators=(",", ":")).encode()
-    return "happ://routing/onadd/" + base64.b64encode(payload).decode()
-
-
-def build_remark(node, user_name):
-    code = node.get("country", "")
-    flag, name = COUNTRY_MAP.get(code, ("\U0001F310", code or "??"))
-    transport = node.get("params", {}).get("type", "tcp")
-    chain = " > ".join(node.get("nodes", ["unknown"]))
-    return f"{flag} {name} [{transport}] | {chain} | {user_name}"
-
-
-def build_link(node, user_name):
-    """Assemble one share link from a fully-merged node dict."""
-    scheme = node["scheme"]
-    host = node["host"]
-    port = node["port"]
-    userinfo = node.get("id", "")
-
-    # Drop empty params so the query stays clean; preserve insertion order.
-    params = {k: v for k, v in node.get("params", {}).items()
-              if v is not None and v != ""}
-
-    # Lists (e.g. alpn) -> comma-joined; everything else stringified.
-    flat = {}
-    for k, v in params.items():
-        flat[k] = ",".join(map(str, v)) if isinstance(v, list) else str(v)
-
-    query = urlencode(flat, safe="/:")
-    remark = quote(build_remark(node, user_name))
-
-    base = f"{scheme}://{userinfo}@{host}:{port}"
-    return f"{base}?{query}#{remark}" if query else f"{base}#{remark}"
-
-
-def main():
-    if len(sys.argv) < 3:
-        print("Usage: generator.py <input_json> <output_dir>")
-        sys.exit(1)
-
-    input_path, output_dir = sys.argv[1], sys.argv[2]
-
-    with open(input_path, "r", encoding="utf-8") as f:
-        data = json.load(f)
-
-    routing = routing_link(data.get("routing", {}))
-    templates = data.get("templates", {})
-    users = data.get("users", [])
-
-    os.makedirs(output_dir, exist_ok=True)
-
-    for user in users:
-        name = user["name"]
-        out_path = os.path.join(output_dir, user.get("psub", name))
-
-        lines = [routing]
-        for entry in user.get("configs", []):
-            tmpl = entry.get("template")
-            node = deep_merge(templates.get(tmpl, {}), entry) if tmpl else entry
-            lines.append(build_link(node, name))
-
-        with open(out_path, "w", encoding="utf-8") as f:
-            f.write("\n".join(lines) + "\n")
-        print(f"Generated: {out_path}")
-
-
-if __name__ == "__main__":
-    main()