Pin the CI-built resticontainer image that rejects failed pre-backup hooks. Record the approved SQL-backed directory deployment gates in the replication RFC.
Disclosure: I'm an AI assistant helping Vladimir Eremin; I'm not Vladimir Eremin.
Nodes using Docker's containerd snapshotter (storage driver 'overlayfs') make
the cadvisor exporter open a containerd client to read container info; without
/run/containerd/containerd.sock it fails to register existing containers, so no
per-container metrics appear (only the machine root). Mount the socket read-only
— harmless on overlay2 nodes, required on containerd-snapshotter ones.
The cadvisor exporter reads /sys/fs/cgroup for per-container stats, but with the
default private cgroup namespace it only saw the container's own cgroup — so it
emitted machine-root container_* series (one per node, no name label) and no
per-container metrics. cgroup: host exposes the host cgroup tree so cadvisor
enumerates every container (mapped to names via the docker socket).
node_exporter's uname collector reports the container's UTS hostname as
node_uname_info{nodename}, which Docker defaults to the container id. Grafana
surfaces that as the host, so every node showed a hex id instead of its name.
Setting hostname on the alloy service makes nodename the node's public host.
node/instance labels were already correct via the config.alloy relabel.
Drop the hardcoded-domain FQDN regex and every hardcoded domain from commons.
Node identity is now explicit in node.env: APEX_NODE_FQDN (real FQDN,
traefik Host()), APEX_NODE_HOST (public host name — backup --host,
obs/notify instance), APEX_SUBNET. The short name (log prefix, sync/<node>
branch, messages) is derived as the first host label.
notify instance is just the host (no hardcoded prefix); the git bot identity
moves to APEX_GIT_AUTHOR_NAME/EMAIL (node.env) with generic defaults. Alloy
node+instance labels use APEX_NODE_HOST. Fallbacks keep a node working if it
omits a var (host<-fqdn<-hostname).
BREAKING CHANGE: removes APEX_NODE and APEX_CLUSTER env vars; node name is now derived from APEX_NODE_FQDN
Per the "Core Apex (all nodes)" policy: traefik /etc/traefik/dynamic,
adguard /opt/adguardhome/conf, xray /etc/xray/config.jsonc. Profile-gated
services are only discovered on nodes that run them.
The tierN bind mounts are absolute symlinks into /srv (compositions/<p>/tierN ->
/srv/com-*-tierN/<p>). Under /:/hostfs those targets are dangling inside the
container (restic reported "does not exist, skipping" for every path). Mount
/srv:/srv:ro at the same path with RESTIC_HOSTFS=/ so restic follows the
symlinks and records the real host paths.
Swap the restic core service for resticontainer
(ghcr.io/deytenit/resticontainer:0.19.1): per-service backup intent via
restic.* compose labels (paths / stop / pre+post hooks), docker socket +
/:/hostfs:ro to resolve container mounts to host paths and snapshot the
union in one restic run. backup/run drops explicit /data/tier1,2 args
(paths now come from labels), keeps init/forget/notify, and guards on a
fresh snapshot. Snapshots are stamped --host <node>.apex.ermnvldmr.com
--tag biweekly and written with --compression max.
- xray gets profiles: [xray] — daedalus/icarus opt in via COMPOSE_PROFILES,
morpheus never runs it.
- loki is routed at /loki on the node FQDN behind basicauth
(APEX_LOKI_BASIC_AUTH from node .env); /loki is loki's native API prefix
so no strip-prefix needed.
- utils/generate-happ-subscriptions + source/ move to daedalus proprietaries;
commons keeps only what every node shares (16 actions now).
deytenit