67 Commits
Author SHA1 Message Date
deytenit 9d733a8f5d fix: allow non-Compose commits on Docker-free hosts [skip ci] 2026-11-06 00:40:05 +03:00
deytenit 00d19c3008 fix: synchronize node main directly without capture CI [skip ci] 2026-11-06 00:17:54 +03:00
deytenit 7ad4473e93 refactor(provisioning): remove manual SSH bootstrap workflow 2026-11-04 23:20:47 +03:00
deytenit f9091d116a chore(ci): update static S3 fallback region to auto for Cloudflare R2 2026-10-29 23:54:37 +03:00
deytenit 0881013357 docs(apex): keep public guidance and examples deployment-neutral 2026-10-26 12:53:58 +03:00
deytenit 995ea9bdf9 fix(apex): preserve environments and restore secure standalone operations 2026-10-26 12:26:09 +03:00
deytenit 6463bad5ee feat(engine): de-slop commons, harden security, add targeted compose and env bridge
- Purge misplaced RFCs, scratch tokens, and node-proprietary assets
- Decouple remote provisioning to ci/scripts/init-node.py
- Add targeted composition orchestration in actions/compose.py
- Add host-side Xray stale socket cleanup in /dev/shm
- Add actions/env.py for shell exports and .env synchronization
- Harden UFW, backup retention, host tempfiles, and notify dispatcher
- Add pyproject.toml and Makefile with 78 passing unit tests
2026-10-26 00:03:33 +03:00
deytenit f044304ef8 fix(readiness): refresh dependency health outside request path 2026-10-11 23:38:40 +03:00
deytenit 2ba0b8bd39 fix(readiness): accept concurrent browser asset probes 2026-10-11 21:43:53 +03:00
deytenit 517b0a0d4f docs(failover): align probe deadline with deployed worker 2026-10-11 21:33:38 +03:00
deytenit e4308a2dd4 fix(readiness): allow measured native probe execution time 2026-10-11 21:30:14 +03:00
deytenit 31938681dc feat(failover): route multiple hostnames through one Worker
Add an allowlisted multi-host entrypoint that delegates each hostname to the
existing origin selector with its own required health status. Hostnames share
the same origin pair and readiness secret, so one Worker can serve several
independent Routes without duplicating the deployment.

Validation: 21 Worker tests and independent review passed.

Disclosure: I'm an AI assistant helping the author; I'm not the author.
2026-10-11 20:36:50 +03:00
deytenit 30760bb4f8 feat(replication): support application health and controlled two-node recovery
Allow explicit 200 health responses for application origins and share only
in-flight readiness probes without caching their results. Document the accepted
native primary/replica pattern over the existing Xray transport, with fenced
manual promotion and independent node repositories.

Validation: 18 Worker tests, 17 readiness tests, and git diff --check passed.
Production SSO migration and traffic activation remain separate rollout steps.

Disclosure: I'm an AI assistant helping the author; I'm not the author.
2026-10-11 20:30:08 +03:00
deytenit 2a09eac23b feat(replication): add authenticated readiness adapter
Run a fixed application probe behind an authenticated local endpoint with bounded concurrency, deadlines, and no-store responses. Keep application safety checks and production enrollment explicit integration gates.

Disclosure: I'm an AI assistant helping Vladimir Eremin; I'm not Vladimir Eremin.
2026-10-11 16:26:20 +03:00
deytenit 458ad5f930 fix(backup): pin verified hook failure release
Pin the CI-built resticontainer image that rejects failed pre-backup hooks. Record the approved SQL-backed directory deployment gates in the replication RFC.

Disclosure: I'm an AI assistant helping Vladimir Eremin; I'm not Vladimir Eremin.
2026-10-11 16:05:40 +03:00
deytenit dabfd8dbc8 feat: add free-tier composition origin failover asset
Select a ready origin before dispatch, preserve authentication requests without replay, and document Cloudflare compatibility and activation gates.

Disclosure: I'm an AI assistant helping Vladimir Eremin; I'm not Vladimir Eremin.
2026-10-11 02:25:37 +03:00
deytenit 3e26df259f fix: check running images with Docker Buildx
Compare matching runtime and registry digest types, and distinguish declared Compose builds from registry images. Report unavailable checks explicitly.

Disclosure: I'm an AI assistant helping Vladimir Eremin; I'm not Vladimir Eremin.
2026-10-11 02:18:46 +03:00
deytenit fe86e55f4a feat: apply node timezone before scheduled jobs
Read an optional configs/timezone declaration and refresh active cron after a timezone change. Report a failed cron restart instead of claiming success.

Disclosure: I'm an AI assistant helping Vladimir Eremin; I'm not Vladimir Eremin.
2026-10-11 02:07:13 +03:00
deytenit 99ff1f1433 fix: share Happ generation and correct protocol labels
Move the reusable generator into commons, add Netherlands display metadata, and label Hysteria links by protocol without changing URI parameters.

Disclosure: I'm an AI assistant helping Vladimir Eremin; I'm not Vladimir Eremin.
2026-10-11 01:49:44 +03:00
deytenit 65fef9a6b0 fix: support current Traefik ACME certificate export
Update the on-demand certificate dumper to v2.11.4 so Traefik accounts without KeyType no longer panic. Validated native export and all 260 tests.

Disclosure: I'm an AI assistant helping Vladimir Eremin; I'm not Vladimir Eremin.
2026-10-11 01:23:21 +03:00
deytenit 4106af4474 fix: preserve new tier permissions under restrictive umask
Explicitly apply the intended directory mode after creation while preserving existing tier and parent permissions. Regression tests cover enrollment's restrictive umask.

Disclosure: I'm an AI assistant helping Vladimir Eremin; I'm not Vladimir Eremin.
2026-10-10 21:33:51 +03:00
deytenit 5753d18440 fix: restore native security state across enrollment failures
Install CrowdSec before its bouncer and repair package placeholder credentials without changing custom YAML credentials. Match native Debian UFW and package-unit behavior during protected SSH enrollment.

Restore only owned Docker helper chains, cancel queued service starts, and recover failed owned services before reporting rollback complete. Preserve changed administrator policy for reconciliation.

Verified with 258 automated tests and an isolated Debian 13 VM: baseline repair/retry, SSH/security activation, committed retry, watchdog recovery, actual reboot, and pending-cleanup reconciliation. Native source bundle: 8ae1f9fa1bf3dfa1f6c3aeb1c96fe92f170cbcbb7afae664266a76975a1ce1c7.

Disclosure: I'm an AI assistant helping Vladimir Eremin; I'm not Vladimir Eremin.
2026-10-10 21:12:57 +03:00
deytenit 6be21acde1 feat: initialize Debian nodes through shared provisioning
Add local SSH enrollment for clean Debian 13 hosts using the same baseline and user-data contract as Packer. Preserve provider networking, validate storage and repository ownership, and protect final SSH and security-service activation with persistent access recovery.

Document composition-scoped replication patterns for independent node repositories and add behavioral coverage for provisioning, retries, and recovery. Native verification of the security activation transaction remains the production enrollment gate.

Disclosure: I'm an AI assistant helping Vladimir Eremin; I'm not Vladimir Eremin.
2026-10-10 20:32:12 +03:00
deytenit cbf3991147 fix: stabilize alloy config
BREAKING CHANGE: \
2026-10-04 15:36:12 +03:00
deytenit a9436ec385 feat: Harden allow on resources 2026-09-27 22:42:56 +03:00
deytenit 0748ab43a0 feat: Add memory retention policies for apt and systemd-journal 2026-09-27 21:41:02 +03:00
deytenit 48840c3022 chore(compositions/apex): bump core service image tags to latest minor/patch 2026-09-27 20:17:15 +03:00
deytenit c94745b7e7 feat(skills): add update-compositions skill and registry query reference 2026-09-27 17:54:47 +03:00
deytenit 5672e5d65c fix(alloy): enable docker_only in cadvisor exporter 2026-09-27 17:35:47 +03:00
deytenit 78ab580e6f fix(cloud-init): defer ssh deploy key write in master template 2026-09-13 15:42:38 +03:00
deytenit 7b4e40dba5 fix(configure): gracefully handle missing ufw dirs and skip redundant apt calls 2026-09-13 15:39:23 +03:00
deytenit 6123c3f6b4 feat: add Debian 13 golden image Packer pipeline and cloud-init bootstrap 2026-08-30 19:48:22 +03:00
deytenit b72b961968 fix: forward host proxy env vars and network to dclint container 2026-08-09 01:09:18 +03:00
deytenit 64d9138a8f fix: keep notification credentials out of process arguments 2026-08-02 15:07:52 +03:00
deytenit 6daabae415 fix: mount containerd socket into alloy for cadvisor per-container stats
Nodes using Docker's containerd snapshotter (storage driver 'overlayfs') make
the cadvisor exporter open a containerd client to read container info; without
/run/containerd/containerd.sock it fails to register existing containers, so no
per-container metrics appear (only the machine root). Mount the socket read-only
— harmless on overlay2 nodes, required on containerd-snapshotter ones.
2026-07-12 03:03:17 +03:00
deytenit 0a585cc061 fix: run alloy in the host cgroup namespace (cgroup: host)
The cadvisor exporter reads /sys/fs/cgroup for per-container stats, but with the
default private cgroup namespace it only saw the container's own cgroup — so it
emitted machine-root container_* series (one per node, no name label) and no
per-container metrics. cgroup: host exposes the host cgroup tree so cadvisor
enumerates every container (mapped to names via the docker socket).
2026-07-12 02:52:43 +03:00
deytenit 9bc97f196f feat: scrape CrowdSec agent metrics through Alloy
Add a prometheus.scrape for the CrowdSec agent's :6060 endpoint, forwarded to the
existing remote_write. CrowdSec runs on the host, so nodes that run it bind the
metrics endpoint on 0.0.0.0 (host firewall keeps it off the public interface) and
Alloy reaches it via the docker gateway; nodes without CrowdSec report the target
down. node/instance labels are stamped APEX_NODE_HOST like the other exporters.
2026-07-11 22:37:38 +03:00
deytenit e3749577d1 docs: rewrite README with a friendly what/why/how structure
Open with a centered logo and a plain-language description of what apex does for
a self-hoster, then Why, then How it works. Quick start shows a minimal
selfhost.example.com bootstrap. Links the published docs (en + Russian) and the
getting-started guide. Drops badges and the repo-layout table (covered in docs).
2026-07-11 20:23:58 +03:00
deytenit 94082ad9be chore: genericize deployment-specific defaults for public release
- core compose: parameterize TZ (default Etc/UTC), restic S3 region+endpoint
  (vendor-neutral default: AWS S3 us-east-1, no custom endpoint), and the
  xray-enclave allowlist (APEX_ENCLAVE_SUBNET, default 198.18.0.0/16).
  Cloudflare + resticontainer kept as-is.
- the reference fleet pins its real values (Europe/Moscow, ru-central1, Yandex
  Object Storage) in each node's apex.env, so deployed behavior is unchanged.
- drop PARITY.md (internal bash->python port audit; node-specific narrative).
- genericize node-name leftovers (icarus/daedalus, MSK) in tests + comments.

Tests: 26/26 pass. Verified: resticontainer resolves ru-central1/Yandex/Moscow on a node.
2026-07-11 18:00:13 +03:00
deytenit fc11217a90 chore: relicense to BSD 2-Clause; note AI-assisted contributions in README
Swap MIT for BSD 2-Clause (verbatim OSI/SPDX text) ahead of publishing apex as a
generic self-host framework. Add a README footer welcoming AI-assisted contributions
that meet the quality bar and are understood by the contributor.
2026-07-11 17:21:35 +03:00
deytenit 477c0eb5d1 fix: set alloy container hostname to APEX_NODE_HOST
node_exporter's uname collector reports the container's UTS hostname as
node_uname_info{nodename}, which Docker defaults to the container id. Grafana
surfaces that as the host, so every node showed a hex id instead of its name.
Setting hostname on the alloy service makes nodename the node's public host.
node/instance labels were already correct via the config.alloy relabel.
2026-07-11 16:49:56 +03:00
deytenit bd973d783d refactor: make engine domain-agnostic — identity from node.env, no hardcoded domain
Drop the hardcoded-domain FQDN regex and every hardcoded domain from commons.
Node identity is now explicit in node.env: APEX_NODE_FQDN (real FQDN,
traefik Host()), APEX_NODE_HOST (public host name — backup --host,
obs/notify instance), APEX_SUBNET. The short name (log prefix, sync/<node>
branch, messages) is derived as the first host label.

notify instance is just the host (no hardcoded prefix); the git bot identity
moves to APEX_GIT_AUTHOR_NAME/EMAIL (node.env) with generic defaults. Alloy
node+instance labels use APEX_NODE_HOST. Fallbacks keep a node working if it
omits a var (host<-fqdn<-hostname).

BREAKING CHANGE: removes APEX_NODE and APEX_CLUSTER env vars; node name is now derived from APEX_NODE_FQDN
2026-07-08 09:32:20 +03:00
deytenit 53d3bedca6 feat: add resticontainer backup labels for traefik/adguard/xray
Per the "Core Apex (all nodes)" policy: traefik /etc/traefik/dynamic,
adguard /opt/adguardhome/conf, xray /etc/xray/config.jsonc. Profile-gated
services are only discovered on nodes that run them.
2026-07-07 21:38:25 +03:00
deytenit c100db57fe fix: mount /srv in resticontainer so absolute tierN symlinks resolve
The tierN bind mounts are absolute symlinks into /srv (compositions/<p>/tierN ->
/srv/com-*-tierN/<p>). Under /:/hostfs those targets are dangling inside the
container (restic reported "does not exist, skipping" for every path). Mount
/srv:/srv:ro at the same path with RESTIC_HOSTFS=/ so restic follows the
symlinks and records the real host paths.
2026-07-07 21:15:15 +03:00
deytenit 07a53b9b2f feat: migrate to resticontainer with label-driven per-service backup
Swap the restic core service for resticontainer
(ghcr.io/deytenit/resticontainer:0.19.1): per-service backup intent via
restic.* compose labels (paths / stop / pre+post hooks), docker socket +
/:/hostfs:ro to resolve container mounts to host paths and snapshot the
union in one restic run. backup/run drops explicit /data/tier1,2 args
(paths now come from labels), keeps init/forget/notify, and guards on a
fresh snapshot. Snapshots are stamped --host <node>.apex.ermnvldmr.com
--tag biweekly and written with --compression max.
2026-07-07 20:37:31 +03:00
deytenit b41fc6097f refactor: replace loki store with Grafana Alloy push agent 2026-07-06 14:34:09 +03:00
deytenit 334ca196e0 feat: add .apex-root-owned opt-out for root-owned tier compositions
An empty .apex-root-owned marker in a composition dir makes tiers/useradd
skip its noroot user + .env UID injection, and tiers/chown force its tier
data (dir + children, recursively) to root:root instead of noroot-<proj>.
For services that run as root and reject non-root-owned data (e.g. seafile-mc).
2026-07-06 10:39:04 +03:00
deytenit 0b42ab1bfc docs: link published documentation in README 2026-07-05 12:40:17 +03:00
deytenit ce93f879e7 refactor: gate xray behind profile, expose loki via basicauth, move happ generator to daedalus
- xray gets profiles: [xray] — daedalus/icarus opt in via COMPOSE_PROFILES,
  morpheus never runs it.
- loki is routed at /loki on the node FQDN behind basicauth
  (APEX_LOKI_BASIC_AUTH from node .env); /loki is loki's native API prefix
  so no strip-prefix needed.
- utils/generate-happ-subscriptions + source/ move to daedalus proprietaries;
  commons keeps only what every node shares (16 actions now).
2026-07-05 12:16:51 +03:00
deytenit 1bc448961a fix: update sync/repository rebase target from trunk to main 2026-07-05 12:04:24 +03:00
deytenit c5b61a3294 fix: exclude commons/ from dclint --fix to prevent submodule dirtying
In node repos the shared core compose lives inside the pinned commons
submodule; letting dclint rewrite it dirties the submodule, which makes
sync/repository abort the nightly capture-up. Harmless in the standalone
commons repo (no commons/ dir).
2026-07-05 11:20:25 +03:00
deytenit 6aaef35780 fix: resolve 17 confirmed bash-parity drifts from port review
Adversarial parity review of every ported action against its bash source
confirmed 17 real behavioral drifts (of 22 claimed); all fixed here:

- tiers/chown: defaults were compositions/apex/tierN (one level too deep);
  now the tier roots. shared/ base reset is non-recursive again.
- compose: --dry-run was swallowed by argparse.REMAINDER; now stripped.
- host.write_file: rendered files land 0644 (tempfile's 0600 made fresh
  systemd units/crowdsec configs owner-only).
- systemd.deploy_units: top-level non-hidden files only.
- ufw: missing docker/host config dir exits 66; missing /etc/ufw target
  aborts; empty rules dir warns; dotfile .rules excluded.
- sync/packages: Telegram error-trap parity and failed notify exits 1.
  Same notify-failure exit for backup/run and sync/repository.
  repository regains the node-dir guard.
- tiers/useradd: no more secret-bearing .env.bak per run.
- tiers/link: EOF on confirm prompt declines and continues.
- extract-traefik-certs: sudo mkdir inside the notify envelope.
- generate-happ-subscriptions: realpath parent-exists guards + docker check.
- configure/crowdsec: bouncer install goes through install_packages.
- lint-docker-compose: malformed/unreadable anchor-IP inputs are lint errors.
- core compose: APEX_RESTIC_REPOSITORY now ?error; RESTIC_COMPRESSION
  per-node via APEX_RESTIC_COMPRESSION (default auto).
- sys.service_exists line-anchored; tpl docstring corrected.
2026-07-05 10:50:40 +03:00
deytenit ad47b73928 docs: add PARITY map and mark engine readiness gate as green 2026-07-04 21:31:05 +03:00
deytenit d97311a652 feat: add shared base core compose with env-parameterized restic tier mounts 2026-07-04 21:30:25 +03:00
deytenit ec54ab592f feat: port utils actions (extract-traefik-certs, generate-happ-subscriptions, lint-docker-compose) 2026-07-04 21:29:17 +03:00
deytenit b9c80b0212 feat: port tiers actions (link, chown, useradd) 2026-07-04 21:28:28 +03:00
deytenit f7fc0eb697 feat: port sync and backup/run actions 2026-07-04 21:27:39 +03:00
deytenit 05469332e8 feat: port configure actions (base, ufw, crowdsec, cron, systemd, routing) 2026-07-04 21:26:56 +03:00
deytenit 63c4b6c7eb feat: add apex launcher, test runner, and cli smoke test 2026-07-04 21:25:59 +03:00
deytenit 83e6bcfe31 feat: add engine cli (discover/dispatch/help, hostname identity) 2026-07-04 21:25:59 +03:00
deytenit 496ed01ede feat: add engine context module (ctx assembly, tier vars, commons wrap) 2026-07-04 21:25:14 +03:00
deytenit 9ad3c93644 feat: add engine lib modules (ufw, crowdsec, systemd, notify) 2026-07-04 21:24:44 +03:00
deytenit 72eec40ef3 feat: port engine lib modules (log, sys, tpl, host) 2026-07-04 21:23:48 +03:00
deytenit 12eac9772b feat: add engine overlay module (resolve/discover, ast scan, tombstone) 2026-07-04 21:21:58 +03:00
deytenit cc004f99f8 feat: add engine identity module (FQDN resolution via node.env, hostname fallback) 2026-07-04 21:21:14 +03:00
deytenit f96c53add9 feat: add engine descriptor module (Meta/Arg/Flag/Opt/Rest, argparse) 2026-07-04 21:20:28 +03:00
deytenit e9c7e13848 feat: scaffold apex commons project (assets, source, hooks, docs) 2026-07-04 21:19:28 +03:00