fix, feat, doc: massive build system refactorion, bug fixes and apex documentation (#19)
This commit is contained in:
deytenit authored and 
GitHub committed
1 parent
e8b42c60c2
commit
91f5245823
165 files changed
+8598
-1205
No files matched your search
@@ -0,0 +1,24 @@
|
||||
---
|
||||
title: Guides
|
||||
description: Task-oriented guides for building node repositories, actions, and compositions.
|
||||
weight: 20
|
||||
---
|
||||
|
||||
Task-oriented guides for day-to-day work with apex. Each guide solves one recurring problem: assembling a [node](/apex/glossary#node) repository, developing [actions](/apex/glossary#action) for the [engine](/apex/glossary#engine), defining service [compositions](/apex/glossary#compositions), and operating the underlying host.
|
||||
|
||||
## Available guides {#available-guides}
|
||||
|
||||
{{< cards >}}
|
||||
{{< card link="/apex/guides/node-repository" title="Node repository" subtitle="Assemble the per-node repository that declares a node's identity, configs, and compositions." >}}
|
||||
{{< card link="/apex/guides/action-development" title="Action development" subtitle="Write actions that the engine dispatches on a node." >}}
|
||||
{{< card link="/apex/guides/service-composition" title="Service composition" subtitle="Define project compositions and deploy them alongside the core composition." >}}
|
||||
{{< card link="/apex/guides/host-operations" title="Host operations" subtitle="Perform routine operations against a live host through the engine." >}}
|
||||
{{< /cards >}}
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Docs: Getting started](/apex/getting-started)
|
||||
- [Docs: Concepts](/apex/concepts)
|
||||
- [GitHub: apex.ermnvldmr.com](https://github.com/deytenit/apex.ermnvldmr.com)
|
||||
@@ -0,0 +1,90 @@
|
||||
---
|
||||
title: Develop actions
|
||||
description: Write, override, and test engine actions in commons or proprietaries.
|
||||
weight: 22
|
||||
---
|
||||
|
||||
**Context:** all automation ships as [actions](/apex/glossary#action) — one Python module each, dispatched by path. Shared behavior belongs in [commons](/apex/glossary#commons) `actions/`; anything node-specific belongs in that node's `proprietaries/actions/`. Both use the identical contract, and the launcher puts both trees on `sys.path`, so a proprietary action imports `engine.*` exactly like a commons one.
|
||||
|
||||
## A minimal action {#minimal}
|
||||
|
||||
```python {filename="proprietaries/actions/utils/example.py"}
|
||||
"""One-line module docstring describing the action."""
|
||||
import os
|
||||
from engine.descriptor import Meta, Arg, Flag
|
||||
|
||||
METADATA = Meta(
|
||||
summary="Copy a rendered config into place.",
|
||||
args=[Arg("target_dir", "Destination directory"),
|
||||
Flag("--dry-run", "Preview only.")],
|
||||
)
|
||||
|
||||
def run(ctx, args):
|
||||
if not os.path.isdir(args.target_dir):
|
||||
ctx.log.error(f"No such directory: {args.target_dir}")
|
||||
raise SystemExit(66)
|
||||
content = ctx.tpl.render("example/config", ctx.vars())
|
||||
ctx.host.write_file(os.path.join(args.target_dir, "config"), content,
|
||||
dry_run=args.dry_run, sudo=True)
|
||||
ctx.log.success("Config deployed.")
|
||||
```
|
||||
|
||||
The file's path is its name: the module above dispatches as `apex utils/example`. Return `0`/`None` for success, an integer for a specific exit code, or raise `SystemExit(code)`; the [exit-code convention](/apex/concepts/actions-and-overlay#exit-codes) is `64` usage, `65` data error, `66` missing input, `1` general.
|
||||
|
||||
## Conventions for unattended actions {#unattended}
|
||||
|
||||
Actions that run from cron follow two rules so failures page instead of vanishing:
|
||||
|
||||
- **Wrap the body** in `try/except Exception` and send `ctx.notify.error(...)` before re-raising as `SystemExit(1)` — the Python equivalent of a shell ERR trap.
|
||||
- **Gate the exit code on the final notification**: `ctx.notify.*` returns `False` on delivery failure; a completed run whose status message never arrived should still exit `1`.
|
||||
|
||||
```python
|
||||
try:
|
||||
...work...
|
||||
log.success("Done.")
|
||||
if not ctx.notify.success(TITLE, url, "Done.", node):
|
||||
raise SystemExit(1)
|
||||
except SystemExit:
|
||||
raise
|
||||
except Exception as e:
|
||||
log.error(f"{type(e).__name__}: {e}")
|
||||
ctx.notify.error(TITLE, url, f"Critical error: {e}", node)
|
||||
raise SystemExit(1)
|
||||
```
|
||||
|
||||
## Override, wrap, disable {#override}
|
||||
|
||||
A proprietary file at the same relative path as a commons action shadows it completely. Inside the shadowing module, `ctx.commons.run(args)` invokes the shadowed implementation — the wrap pattern:
|
||||
|
||||
```python {filename="proprietaries/actions/sync/packages.py"}
|
||||
from engine.descriptor import Meta, Arg
|
||||
|
||||
METADATA = Meta(summary="sync/packages with a node-local pre-step.",
|
||||
args=[Arg("telegram_bot_url", "Telegram Bot URL")])
|
||||
|
||||
def run(ctx, args):
|
||||
ctx.log.info("Node-local preparation...")
|
||||
return ctx.commons.run(args) # delegate to the commons implementation
|
||||
```
|
||||
|
||||
To retire an action on one node, assign `DISABLED = "reason"` at module top level: dispatch refuses it (exit `1`) and the listing shows `[DISABLED: reason]`.
|
||||
|
||||
## Testing {#testing}
|
||||
|
||||
The commons test suite is stdlib `unittest`:
|
||||
|
||||
```bash
|
||||
./tests/run.sh # python3 -m unittest discover -s tests -p 'test_*.py' -v
|
||||
```
|
||||
|
||||
Engine changes must keep the suite green. A cheap smoke gate for any action — commons or proprietary — is its parser: `apex <name> --help` must exit `0`, which proves the module parses, its `METADATA` is well-formed, and discovery sees it.
|
||||
|
||||
The commons pre-commit hook (wired by `init.sh` via `core.hooksPath`) runs `apex utils/lint-docker-compose --hook` against staged compose files, so composition changes are linted at commit time.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Concept: Actions and the overlay](/apex/concepts/actions-and-overlay)
|
||||
- [Concept: The execution context](/apex/concepts/execution-context)
|
||||
- [Reference: Engine API](/apex/reference/engine-api)
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
title: Operate a node
|
||||
description: "Day-2 operations: configure, compose, backups, and repository sync."
|
||||
weight: 24
|
||||
---
|
||||
|
||||
**Context:** once bootstrapped, a [node](/apex/glossary#node) is operated entirely through the [engine](/apex/glossary#engine). This guide covers the recurring operations and what each actually does on the host.
|
||||
|
||||
## Host configuration {#configure}
|
||||
|
||||
`apex configure` runs the full configuration suite in a fixed order; each step is individually invocable when only one layer changed:
|
||||
|
||||
| Step | Effect |
|
||||
|---|---|
|
||||
| `configure/base` | Ensures a Debian base: package index update, essential packages (`curl`, `wget`, `git`, `rsyslog`), rsyslog running, `ufw` installed, and the `ufw-docker` helper at `~/.local/bin/ufw-docker`. |
|
||||
| `configure/ufw` | Applies host rules, injects the managed kernel-rule blocks, applies container rules via `ufw-docker`. Supports `--dry-run`. |
|
||||
| `configure/crowdsec` | Installs CrowdSec if absent, deploys `configs/crowdsec/` to `/etc/crowdsec/`, installs collections, and the firewall bouncer when configured. Supports `--dry-run`. |
|
||||
| `configure/cron` | Renders `configs/cron/crontab` through the template layer and installs it — with preview, empty-render refusal (exit `65`), and rollback on a failed install. Supports `--dry-run`. |
|
||||
| `configure/systemd` | Renders the top-level files of `configs/systemd/` into `/etc/systemd/system/`, then daemon-reloads and enables each `.service`. Supports `--dry-run`. |
|
||||
| `configure/routing` | Enables IP forwarding and installs the fwmark policy-routing rule and table used by the enclave edge; idempotent. |
|
||||
|
||||
After `configure/ufw`, restart `ufw` deliberately (`sudo systemctl restart ufw`) — the action reminds you, and warns that the SSH port must already be allowed.
|
||||
|
||||
## Composition lifecycle {#compose}
|
||||
|
||||
```bash
|
||||
apex compose up # core first, then services alphabetically
|
||||
apex compose down # reverse order
|
||||
apex compose restart
|
||||
apex compose up --dry-run # print the commands only
|
||||
```
|
||||
|
||||
Extra arguments pass through to `docker compose` per project. Each project's `.env` and `apex.env` are supplied automatically when present.
|
||||
|
||||
## Backups {#backups}
|
||||
|
||||
```bash
|
||||
apex backup/run "$(cat <path-to-telegram-url-file>)"
|
||||
```
|
||||
|
||||
Runs [resticontainer](https://github.com/deytenit/resticontainer) one-shot under the `manual` compose profile: init-if-needed, then a label-driven backup that discovers each service's intent from its `restic.*` compose labels (`restic.enable`, `restic.backup.paths`, …), snapshots the union of the resolved host paths in a single restic run (`--host` = `APEX_NODE_HOST`, `--compression max`), applies retention (`forget --prune`), and notifies. It refuses to report success if no `restic.*`-labelled service produced a new snapshot. The full flow is described in [Concept: Tiered storage](/apex/concepts/tiered-storage#backups). Schedule it from `configs/cron/crontab` so the run and its Telegram outcome are unattended.
|
||||
|
||||
## Repository sync (capture-up) {#sync}
|
||||
|
||||
```bash
|
||||
apex sync/repository <telegram-bot-url>
|
||||
```
|
||||
|
||||
Commits the live checkout to `sync/<node>`, rebases onto `origin/main`, and force-pushes with lease; CI folds the branch back into `main`. The action refuses to run — with a paged error — when the commons submodule is dirty or the `compositions/` tree is missing, and aborts cleanly on a rebase conflict (manual intervention required). See [Concept: Framework architecture](/apex/concepts/architecture#capture-up) for the model.
|
||||
|
||||
## Update awareness {#updates}
|
||||
|
||||
```bash
|
||||
apex sync/packages <telegram-bot-url>
|
||||
```
|
||||
|
||||
Reports pending apt upgrades and out-of-date container images (local digests compared against the registry via `skopeo`) in a single Telegram digest. It changes nothing on the host — it is a scheduled awareness check.
|
||||
|
||||
## Certificate extraction {#certs}
|
||||
|
||||
```bash
|
||||
apex utils/extract-traefik-certs <acme.json> <dest-dir> [telegram-bot-url]
|
||||
```
|
||||
|
||||
Dumps the edge proxy's ACME store into per-domain `fullchain.pem`/`privkey.pem` pairs (via a containerized dumper), restricts key permissions to `600`, and re-owns the output to the invoking user — for services that need raw PEM material instead of proxy-terminated TLS.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Getting started with an apex node](/apex/getting-started)
|
||||
- [Reference: Actions](/apex/reference/actions)
|
||||
- [Docs: Troubleshooting](/apex/troubleshooting)
|
||||
@@ -0,0 +1,105 @@
|
||||
---
|
||||
title: Build a node repository
|
||||
description: Assemble the four pillars, pin commons, and wire the capture-up flow.
|
||||
weight: 21
|
||||
---
|
||||
|
||||
**Context:** every [node](/apex/glossary#node) gets exactly one repository. Pick any repository name — [identity](/apex/glossary#node-identity) is declared explicitly in `node.env`, so the repo name is a convention, not a functional input. This guide assembles one from nothing.
|
||||
|
||||
{{% steps %}}
|
||||
|
||||
### Initialize the repository and pin commons
|
||||
|
||||
```bash
|
||||
git init <node-repo> && cd <node-repo>
|
||||
git submodule add <commons-url> commons
|
||||
( cd commons && git fetch --tags && git checkout <release-tag> )
|
||||
git add .gitmodules commons
|
||||
mkdir -p proprietaries/actions proprietaries/lib configs compositions/apex .github/workflows
|
||||
```
|
||||
|
||||
The gitlink pins the exact engine version the node runs. Upgrading later means checking out a newer tag inside `commons/` and committing the moved pointer.
|
||||
|
||||
### Declare identity
|
||||
|
||||
```ini {filename="node.env"}
|
||||
APEX_NODE_FQDN=node1.example.com
|
||||
APEX_NODE_HOST=node1.example.com
|
||||
APEX_SUBNET=198.18.16.0/24
|
||||
APEX_GIT_AUTHOR_NAME=apex [bot]
|
||||
APEX_GIT_AUTHOR_EMAIL=apex@localhost
|
||||
```
|
||||
|
||||
`node.env` declares the node's identity explicitly — no domain shape is assumed. `APEX_NODE_FQDN` is the real FQDN used for traefik `Host()` routing (fallback: the OS hostname, with a warning). `APEX_NODE_HOST` is the public host name used for the backup `--host` and the observability/notify `instance` label (default: `APEX_NODE_FQDN`); the short node `name` is its first label. `APEX_SUBNET` is the node's /24 for the `direct` network and is **required** — a missing value exits `66` (see [Concept: Fleet addressing scheme](/apex/concepts/addressing)). `APEX_GIT_AUTHOR_NAME`/`APEX_GIT_AUTHOR_EMAIL` are the capture-up bot identity (generic defaults if unset).
|
||||
|
||||
### Write the core wrapper and its env files
|
||||
|
||||
```yaml {filename="compositions/apex/docker-compose.yml"}
|
||||
name: apex
|
||||
include:
|
||||
- ../../commons/compositions/apex/docker-compose.yml
|
||||
```
|
||||
|
||||
```ini {filename="compositions/apex/apex.env"}
|
||||
APEX_TRAEFIK_DIRECT_IP=198.18.16.10
|
||||
APEX_RESTIC_REPOSITORY=s3:<endpoint>/<bucket>/<node>
|
||||
COMPOSE_PROFILES=adguard,xray,alloy
|
||||
# observability push targets (only when the alloy profile is enabled)
|
||||
APEX_PROM_PUSH_URL=https://<central-host>/api/v1/push
|
||||
APEX_LOKI_PUSH_URL=https://<central-host>/loki/api/v1/push
|
||||
APEX_OBS_BASIC_AUTH_USER=<node-user>
|
||||
# optional scalars with compose defaults:
|
||||
# TZ=Etc/UTC
|
||||
# APEX_RESTIC_COMPRESSION=max
|
||||
# APEX_RESTIC_AWS_REGION=us-east-1 # set for any non-AWS S3 store
|
||||
# APEX_RESTIC_AWS_ENDPOINT= # e.g. an S3-compatible endpoint URL
|
||||
```
|
||||
|
||||
`apex.env` holds committed non-secret scalars only. `COMPOSE_PROFILES` selects which optional core services this node runs — `adguard`, `xray`, `alloy` (the observability push agent), and `manual` (one-shot backups) — and may be empty. Document the uncommitted secret keys in `compositions/apex/SECRETS.md` so an operator on a fresh host knows what to fill in `.env` (`APEX_RESTIC_PASSWORD`, the S3 access keys, `APEX_TRAEFIK_CF_DNS_API_TOKEN`, `APEX_TRAEFIK_ACME_EMAIL`, and `APEX_OBS_BASIC_AUTH_PASS` when observability is on).
|
||||
|
||||
### Ignore what never belongs in git
|
||||
|
||||
```gitignore {filename=".gitignore"}
|
||||
**/@tier1
|
||||
**/@tier2
|
||||
**/@tier3
|
||||
**/.env
|
||||
!compositions/apex/apex.env
|
||||
__pycache__/
|
||||
*.pyc
|
||||
```
|
||||
|
||||
### Add the bootstrap script
|
||||
|
||||
`init.sh` prepares a fresh clone on the host: `git submodule update --init --recursive`, `git config core.hooksPath commons/githooks`, `git config --global --add safe.directory <repo>`, and prepend `<repo>/commons` to `PATH` in the login shell so `apex` resolves. Mark it executable.
|
||||
|
||||
### Start the addressing ledger
|
||||
|
||||
Create `compositions/ADDRESSING.md` with the node's subnet (`APEX_SUBNET`) and the reserved ranges (`.10`–`.19` core, `.20`+ services). Every [anchor IP](/apex/glossary#anchor-ip) allocated later is recorded here.
|
||||
|
||||
### Wire capture-up CI
|
||||
|
||||
Add a scheduled workflow that merges `sync/<node>` into `main` when the branch exists, committing under a bot identity. Check out **without** submodules — the merge only moves the commons pointer, and an SSH submodule URL is not fetchable with the default CI token. See [Concept: Framework architecture](/apex/concepts/architecture) for the full [capture-up](/apex/glossary#capture-up) loop.
|
||||
|
||||
### Validate
|
||||
|
||||
```bash
|
||||
./commons/apex --help
|
||||
```
|
||||
|
||||
The listing must show every commons action (plus your proprietaries, once added). Off-host — before `node.env` is filled — expect the harmless warning that `APEX_NODE_FQDN` is unset (the engine falls back to the OS hostname); `APEX_SUBNET` must still be set or the engine exits `66`. Commit the assembled repository on `main`.
|
||||
|
||||
{{% /steps %}}
|
||||
|
||||
## Notes
|
||||
|
||||
- The `proprietaries/` pillar may start empty (keep it present with `.gitkeep` files); the engine treats a missing overlay gracefully.
|
||||
- Migrate host config sources into `configs/` (`ufw/`, `cron/crontab`, `systemd/`, `crowdsec/`) as the node adopts each `configure/*` step — the actions consume `configs/` verbatim through the template layer.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Getting started with an apex node](/apex/getting-started)
|
||||
- [Concept: Framework architecture](/apex/concepts/architecture)
|
||||
- [Guide: Add a service composition](/apex/guides/service-composition)
|
||||
@@ -0,0 +1,101 @@
|
||||
---
|
||||
title: Add a service composition
|
||||
description: "Wire a new project into a node: anchor IP, networks, tiers, and lint."
|
||||
weight: 23
|
||||
---
|
||||
|
||||
**Context:** a project joins a [node](/apex/glossary#node) as one directory under `compositions/` containing a hand-authored compose file. The framework's contract for that file is small: a static [anchor IP](/apex/glossary#anchor-ip) on the `direct` network, external network references by key, and tier-relative data paths.
|
||||
|
||||
{{% steps %}}
|
||||
|
||||
### Allocate the anchor address
|
||||
|
||||
Pick the next free address at `.20` or above in the node's /24 and record it in `compositions/ADDRESSING.md`. Addresses `.10`–`.19` are reserved for the [core composition](/apex/glossary#core-composition) (see [Concept: Fleet addressing scheme](/apex/concepts/addressing)).
|
||||
|
||||
### Author the compose file
|
||||
|
||||
```yaml {filename="compositions/<project>/docker-compose.yml"}
|
||||
name: <project>
|
||||
services:
|
||||
<anchor>:
|
||||
image: ...
|
||||
container_name: <project>-<anchor>
|
||||
user: "${APEX_UID?error}:${APEX_GID?error}"
|
||||
group_add:
|
||||
- "${APEX_SHARED_GID?error}"
|
||||
volumes:
|
||||
- ./@tier1/<anchor>/config:/config
|
||||
- ./@tier2/<anchor>/data:/data
|
||||
networks:
|
||||
direct:
|
||||
ipv4_address: 198.18.16.20
|
||||
enclave: {}
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.<project>.rule=Host(`...`)"
|
||||
- "traefik.http.services.<project>.loadbalancer.server.port=8080"
|
||||
# backup intent (resticontainer, discovered from restic.* labels)
|
||||
- "restic.enable=true"
|
||||
- "restic.backup.paths=/config"
|
||||
restart: unless-stopped
|
||||
|
||||
<sidecar>:
|
||||
image: ...
|
||||
network_mode: "service:<anchor>"
|
||||
|
||||
networks:
|
||||
direct:
|
||||
external: true
|
||||
enclave:
|
||||
external: true
|
||||
```
|
||||
|
||||
Conventions at work:
|
||||
|
||||
- **One anchor per project** holds the static `ipv4_address` on `direct`; sidecars share its network namespace via `network_mode: service:<anchor>`.
|
||||
- **Networks are referenced by key** (`direct`, `enclave`, `socket`) with `external: true` — the core composition owns their definitions.
|
||||
- **`enclave` membership plus traefik labels** publishes the service through the edge proxy; the proxy discovers it via the socket proxy, never via published ports.
|
||||
- **Data paths are tier-relative** (`./@tierN/...`) and resolve through the symlinks created by `tiers/link`.
|
||||
- **Backup is opt-in via `restic.*` labels.** `apex backup/run` drives [resticontainer](https://github.com/deytenit/resticontainer), which discovers what to snapshot from compose labels: `restic.enable=true` marks the service, `restic.backup.paths=<container-path>` names the mounts to back up (resticontainer resolves them to host paths). A service with no `restic.*` labels is simply not backed up. Back up config only where the data is regenerable.
|
||||
- The `${APEX_UID...}` trio comes from the project `.env` maintained by `tiers/useradd`.
|
||||
|
||||
### Provision storage and identity
|
||||
|
||||
```bash
|
||||
apex tiers/link <tier1> <tier2> <tier3>
|
||||
apex tiers/useradd
|
||||
apex tiers/chown
|
||||
```
|
||||
|
||||
Re-running the trio is idempotent; it picks up the new project, creates its tier directories and `.env`, provisions its `noroot-<project>` user, and applies the ownership matrix.
|
||||
|
||||
### Lint
|
||||
|
||||
```bash
|
||||
apex utils/lint-docker-compose
|
||||
```
|
||||
|
||||
Validates every anchor IP (well-formed, unique, inside the node subnet) and runs the compose linter over the repository. The pre-commit hook enforces the same lint on staged compose files.
|
||||
|
||||
### Deploy
|
||||
|
||||
```bash
|
||||
apex compose up
|
||||
```
|
||||
|
||||
The core composition is already up; the new project starts in its alphabetical slot. Verify with `docker ps` and, if the service is proxied, through its route on the edge.
|
||||
|
||||
{{% /steps %}}
|
||||
|
||||
## Notes
|
||||
|
||||
- Service-internal secrets (for example a database password) live in the project's own `.env` in tier-1 storage, alongside the auto-injected UID block.
|
||||
- Never define `direct`/`enclave`/`socket` inside a project file — a second definition would race the core composition for network ownership.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Concept: Compositions and the shared core](/apex/concepts/compositions)
|
||||
- [Concept: Tiered storage and privilege isolation](/apex/concepts/tiered-storage)
|
||||
- [Concept: Fleet addressing scheme](/apex/concepts/addressing)
|
||||
Reference in new issue
Block a user