fix, feat, doc: massive build system refactorion, bug fixes and apex documentation (#19)
This commit is contained in:
deytenit authored and 
GitHub committed
1 parent
e8b42c60c2
commit
91f5245823
165 files changed
+8598
-1205
No files matched your search
@@ -0,0 +1,25 @@
|
||||
---
|
||||
title: Concepts
|
||||
description: "Architecture, identity, actions, execution context, compositions, storage, addressing, and security."
|
||||
weight: 10
|
||||
---
|
||||
|
||||
Theoretical foundations of the apex fleet-operations framework: how a [node](/apex/glossary#node) declares its [identity](/apex/concepts/node-identity), how [actions](/apex/glossary#action) execute, and how a node organizes its services, storage, and addressing.
|
||||
|
||||
{{% cards %}}
|
||||
{{< card link="architecture" title="Architecture" icon="share" subtitle="The engine, commons, and configs layers and how they fit together." >}}
|
||||
{{< card link="node-identity" title="Node identity" icon="finger-print" subtitle="How a node declares who it is and where it belongs." >}}
|
||||
{{< card link="actions-and-overlay" title="Actions and overlay" icon="lightning-bolt" subtitle="Shared actions and node-local overrides." >}}
|
||||
{{< card link="execution-context" title="Execution context" icon="terminal" subtitle="The environment every action receives at run time." >}}
|
||||
{{< card link="compositions" title="Compositions" icon="cube" subtitle="How service stacks are declared and deployed per node." >}}
|
||||
{{< card link="tiered-storage" title="Tiered storage" icon="database" subtitle="Standardized storage classification for service data." >}}
|
||||
{{< card link="addressing" title="Addressing" icon="globe-alt" subtitle="The node subnet, anchor IPs, and the enclave convention." >}}
|
||||
{{< card link="security-model" title="Security model" icon="shield-check" subtitle="Privilege isolation and network hardening." >}}
|
||||
{{% /cards %}}
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Guides: Operating an apex fleet](/apex/guides)
|
||||
- [Reference: Commands and interfaces](/apex/reference)
|
||||
@@ -0,0 +1,111 @@
|
||||
---
|
||||
title: "Actions and the overlay"
|
||||
description: "The action contract and how node-local proprietaries shadow commons actions."
|
||||
weight: 13
|
||||
---
|
||||
|
||||
Every unit of automation in apex is an [action](/apex/glossary#action): a single Python module that the [engine](/apex/glossary#engine) dispatches by name. The [overlay](/apex/glossary#overlay) lets a node's [proprietaries](/apex/glossary#proprietaries) add, override, or disable [commons](/apex/glossary#commons) actions without forking them. This page defines the action contract, the declarative argument descriptors, and the resolution rules.
|
||||
|
||||
## The action contract {#contract}
|
||||
|
||||
An action is one `.py` file under an `actions/` tree. The file path **is** the action name: `actions/<group>/<action>.py` is invoked as `apex <group>/<action> [args...]`. There is no registry to update — dropping a file into the tree publishes the action.
|
||||
|
||||
The module must expose two things:
|
||||
|
||||
- **`METADATA = Meta(...)`** → the summary and argument descriptors that drive both the help text and parsing. The descriptor system succeeds the `action_metadata()` heredoc of the bash "root" commons framework, apex's predecessor.
|
||||
- **`run(ctx, args)`** → the entrypoint. `ctx` is the engine-built execution context; `args` is the parsed `argparse` namespace.
|
||||
|
||||
```python {filename="actions/<group>/<action>.py"}
|
||||
from engine.descriptor import Meta, Arg, Flag
|
||||
|
||||
METADATA = Meta(
|
||||
summary="One-line summary shown in the action listing",
|
||||
args=[
|
||||
Arg("target", help="A required positional argument"),
|
||||
Flag("--dry-run", help="A boolean flag"),
|
||||
],
|
||||
)
|
||||
|
||||
def run(ctx, args):
|
||||
...
|
||||
return 0
|
||||
```
|
||||
|
||||
The return value of `run` becomes the process exit code. A non-integer return (including `None`) is treated as success (`0`). An unhandled exception is logged with the context prefix and exits `1`.
|
||||
|
||||
> [!NOTE]
|
||||
> `METADATA` is technically optional — a module without it is dispatched with an empty `Meta` and accepts no arguments — but every real action should declare it, since the listing and `--help` output are generated from it.
|
||||
|
||||
## Declarative arguments {#arguments}
|
||||
|
||||
`Meta.args` is a list of descriptors. Each maps to exactly one `argparse` construct:
|
||||
|
||||
| Descriptor | Maps to | Behavior |
|
||||
| --- | --- | --- |
|
||||
| `Arg("name")` | positional argument | Required by default; `required=False` makes it an optional positional (`nargs="?"`, default `None`). |
|
||||
| `Opt("--name", default=...)` | value option | An optional flag that takes a value, with a declared default. |
|
||||
| `Flag("--name")` | boolean flag | `store_true`; present means `True`. |
|
||||
| `Rest("name")` | remainder | Greedy passthrough of all remaining argv (`argparse.REMAINDER`). |
|
||||
|
||||
The engine builds the parser as `apex <group>/<action>` and parses the remaining argv before calling `run`. `--help` is handled at parse time. A usage error — unknown flag, missing positional — prints usage and exits `64` (`EX_USAGE`) instead of argparse's default `2`.
|
||||
|
||||
## Listing without importing {#listing}
|
||||
|
||||
`apex` with no arguments (or `--help`) lists every available action. Listing never imports action modules: the engine statically scans each file with the `ast` module and extracts only the `METADATA` summary and any `DISABLED` constant. Action code therefore cannot run — or crash — at list time; a file that fails to parse is listed as `(unparseable)` rather than breaking the listing.
|
||||
|
||||
Each entry is tagged with its overlay source:
|
||||
|
||||
- no tag → a commons action,
|
||||
- `(local)` → exists only in proprietaries,
|
||||
- `(local override)` → a proprietaries action shadowing a commons one,
|
||||
- `[DISABLED: reason]` → the action declares `DISABLED`.
|
||||
|
||||
## The overlay {#overlay}
|
||||
|
||||
Action names are resolved across two trees, proprietaries first:
|
||||
|
||||
{{< filetree/container >}}
|
||||
{{< filetree/folder name="node repository" state="open" >}}
|
||||
{{< filetree/file name="node.env" >}}
|
||||
{{< filetree/folder name="commons" state="open" >}}
|
||||
{{< filetree/folder name="engine" state="closed" >}}{{< /filetree/folder >}}
|
||||
{{< filetree/folder name="actions" state="open" >}}
|
||||
{{< filetree/file name="<group>/<action>.py" >}}
|
||||
{{< /filetree/folder >}}
|
||||
{{< /filetree/folder >}}
|
||||
{{< filetree/folder name="proprietaries" state="open" >}}
|
||||
{{< filetree/folder name="actions" state="open" >}}
|
||||
{{< filetree/file name="<group>/<action>.py" >}}
|
||||
{{< /filetree/folder >}}
|
||||
{{< /filetree/folder >}}
|
||||
{{< /filetree/folder >}}
|
||||
{{< /filetree/container >}}
|
||||
|
||||
The engine detects this layout by its own location: when it runs from a `commons/` directory whose parent contains `node.env`, the parent is the [node](/apex/glossary#node) repository root and `proprietaries/actions/` is the overlay tree. A standalone commons checkout is its own root and has no overlay layer. Inside a node repository, [node identity](/apex/glossary#node-identity) is resolved from the host FQDN before dispatch.
|
||||
|
||||
Resolution for a given name checks `proprietaries/actions/<name>.py` first and `commons/actions/<name>.py` second; exactly one module is loaded by path. An unknown name is an error (exit `1`). The overlay supports four moves:
|
||||
|
||||
- **Add** → a file that exists only in proprietaries becomes a node-local action, listed as `(local)`.
|
||||
- **Override** → a proprietaries file at the same relative path shadows the commons implementation entirely; the commons path is recorded on the resolution as `shadowed`.
|
||||
- **Wrap** → an overriding module can delegate to the implementation it shadows via `ctx.commons.run(args)`, adding node-local behavior before or after the commons logic instead of replacing it.
|
||||
- **Disable** → a module that assigns `DISABLED = "reason"` is refused at dispatch: the engine logs the reason and exits `1`. The listing shows `[DISABLED: reason]`. A bare `DISABLED = True` also disables, with the generic reason `disabled`; an empty string does not.
|
||||
|
||||
## Exit codes {#exit-codes}
|
||||
|
||||
Actions and the engine follow a fixed exit-code convention:
|
||||
|
||||
| Code | Meaning | Typical cause |
|
||||
| --- | --- | --- |
|
||||
| `0` | success | `run` returned `0` or a non-integer. |
|
||||
| `1` | general error | Unknown action, disabled action, unhandled exception, non-integer `SystemExit`. |
|
||||
| `64` | usage error | Arguments rejected by the parser (`EX_USAGE`). |
|
||||
| `65` | data error | Malformed input data (`EX_DATAERR`), raised by engine libraries. |
|
||||
| `66` | missing input | A required input is absent (`EX_NOINPUT`), raised by engine libraries. |
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Concept: Framework architecture](/apex/concepts/architecture)
|
||||
- [Concept: The execution context](/apex/concepts/execution-context)
|
||||
- [GitHub: apex.ermnvldmr.com](https://github.com/deytenit/apex.ermnvldmr.com)
|
||||
@@ -0,0 +1,46 @@
|
||||
---
|
||||
title: "Fleet addressing scheme"
|
||||
description: "Per-node /24 subnets and static anchor IPs inside the enclave space (default 198.18.0.0/16)."
|
||||
weight: 17
|
||||
---
|
||||
|
||||
Apex assigns every service a fixed, predictable IPv4 address inside a per-node subnet. Each node declares its own `/24`, service anchors are static addresses within it, and the whole scheme is enforced mechanically by the anchor-IP lint.
|
||||
|
||||
## Address space {#address-space}
|
||||
|
||||
The [enclave](/apex/glossary#enclave) space defaults to `198.18.0.0/16`, the benchmarking range set aside by [RFC 2544: Benchmarking Methodology for Network Interconnect Devices](https://datatracker.ietf.org/doc/html/rfc2544). This range is never routed on the public Internet and does not collide with the common private ranges (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`) that cloud providers and home networks already consume. The operator may override it via `APEX_ENCLAVE_SUBNET`.
|
||||
|
||||
## Node subnets {#node-subnets}
|
||||
|
||||
Each [node](/apex/glossary#node) declares exactly one `/24` as `APEX_SUBNET` in its `node.env`, and that subnet is the `direct` network's IPAM range. The operator assigns it; conventionally it sits inside the enclave space (`198.18.0.0/16` by default, or whatever `APEX_ENCLAVE_SUBNET` is set to), but the framework imposes no arithmetic on the choice.
|
||||
|
||||
> [!NOTE]
|
||||
> As an optional operator convention, you may organize node `/24`s into contiguous blocks (for example, grouping related nodes) to keep allocations legible. This is a bookkeeping choice, not a framework rule.
|
||||
|
||||
## Allocation inside a node /24 {#node-allocation}
|
||||
|
||||
Within a node's `/24`, host addresses are partitioned by role:
|
||||
|
||||
| Range | Purpose |
|
||||
|---|---|
|
||||
| `.10`–`.19` | Reserved for [core composition](/apex/glossary#core-composition) anchors; `traefik` is always `.10` (via `APEX_TRAEFIK_DIRECT_IP`) |
|
||||
| `.20` and up | Service anchors, allocated per `<project>` |
|
||||
|
||||
Every service anchor is a static `ipv4_address` on the direct network — a fixed [anchor IP](/apex/glossary#anchor-ip) rather than a dynamically assigned lease. Allocations are recorded in `compositions/ADDRESSING.md`, which serves as the human-readable register for the node's subnet.
|
||||
|
||||
## Enforcement {#enforcement}
|
||||
|
||||
The compose lint validates anchor IPs across all [compositions](/apex/glossary#compositions) on a node. For every `ipv4_address` found in `compositions/*/docker-compose.yml` it checks that the address is:
|
||||
|
||||
- **Well-formed** — a malformed literal is reported as a lint error, not a crash.
|
||||
- **Unique** — the same anchor IP may not appear in two projects; duplicates are reported with both offending projects.
|
||||
- **Inside the node subnet** — every anchor must fall within the node's declared `/24`.
|
||||
|
||||
Any violation fails the lint with exit code `1` (general error). Passing validation logs `Anchor IP validation passed.` The implementation lives in the `utils/lint-docker-compose` [action](/apex/glossary#action); see [GitHub: apex.ermnvldmr.com](https://github.com/deytenit/apex.ermnvldmr.com).
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [GitHub: apex.ermnvldmr.com](https://github.com/deytenit/apex.ermnvldmr.com)
|
||||
- [RFC 2544: Benchmarking Methodology for Network Interconnect Devices](https://datatracker.ietf.org/doc/html/rfc2544)
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
title: Framework architecture
|
||||
description: Four-pillar node repositories, the commons submodule, and the dispatch flow.
|
||||
weight: 11
|
||||
---
|
||||
|
||||
apex splits fleet automation into one shared repository — the [commons](/apex/glossary#commons) — and one repository per [node](/apex/glossary#node). The commons carries everything every node shares: the [engine](/apex/glossary#engine), the shared [actions](/apex/glossary#action), and the [core composition](/apex/glossary#core-composition). A node repository carries everything that makes that node itself: its identity, host [configs](/apex/glossary#configs), service [compositions](/apex/glossary#compositions), and node-local [proprietaries](/apex/glossary#proprietaries).
|
||||
|
||||
## The four pillars {#pillars}
|
||||
|
||||
Every node repository has the same shape:
|
||||
|
||||
{{< filetree/container >}}
|
||||
{{< filetree/folder name="<node>.<your-domain>" state="open" >}}
|
||||
{{< filetree/folder name="commons" state="closed" >}}{{< /filetree/folder >}}
|
||||
{{< filetree/folder name="proprietaries" state="closed" >}}{{< /filetree/folder >}}
|
||||
{{< filetree/folder name="configs" state="closed" >}}{{< /filetree/folder >}}
|
||||
{{< filetree/folder name="compositions" state="closed" >}}{{< /filetree/folder >}}
|
||||
{{< filetree/file name="node.env" >}}
|
||||
{{< filetree/file name="init.sh" >}}
|
||||
{{< /filetree/folder >}}
|
||||
{{< /filetree/container >}}
|
||||
|
||||
| Pillar | Content | Ownership |
|
||||
|---|---|---|
|
||||
| `commons/` | Git submodule → `apex.ermnvldmr.com`, pinned to a release tag | Shared, versioned |
|
||||
| `proprietaries/` | Node-local actions, libraries, and tool sources ([overlaid](/apex/glossary#overlay) over commons) | Node |
|
||||
| `configs/` | Host configuration sources: `ufw/`, `cron/crontab`, `systemd/`, `crowdsec/` | Node |
|
||||
| `compositions/` | One directory per project plus the `apex/` core wrapper and the addressing ledger | Node |
|
||||
|
||||
The submodule pin is the framework's version contract: a node runs exactly the engine and core composition recorded in its gitlink. Upgrading a node is a deliberate act — check out a newer tag inside `commons/` and commit the moved pointer.
|
||||
|
||||
## Dispatch flow {#dispatch}
|
||||
|
||||
The launcher `commons/apex` acts on the local checkout; there is no node argument anywhere.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["apex <group>/<action> [args]"] --> B{Layout}
|
||||
B -- "commons/ dir with node.env beside it" --> C[Node checkout:<br/>overlay = proprietaries/actions]
|
||||
B -- otherwise --> D[Standalone commons:<br/>no overlay]
|
||||
C --> E["Overlay resolve<br/>(proprietaries first)"]
|
||||
D --> E
|
||||
E --> F["Identity resolve<br/>(FQDN + node.env)"]
|
||||
F --> G["Build ctx<br/>(log, sys, tpl, host, ufw,<br/>crowdsec, systemd, notify)"]
|
||||
G --> H["parse args (Meta)"]
|
||||
H --> I["run(ctx, args)"]
|
||||
```
|
||||
|
||||
The engine detects the layout from its own location: running from a directory named `commons/` whose parent contains `node.env` means "node checkout" — the parent becomes the repository root and `proprietaries/actions/` becomes the overlay tree. Any other location is a standalone commons checkout acting as its own root.
|
||||
|
||||
Listing (`apex --help`) never imports action modules; summaries and disabled markers are extracted by static `ast` scanning. Dispatch loads exactly one module by path. Details in [Concept: Actions and the overlay](/apex/concepts/actions-and-overlay).
|
||||
|
||||
## The capture-up model {#capture-up}
|
||||
|
||||
Node repositories are ledgers, not just sources: the host itself records what it runs.
|
||||
|
||||
1. A scheduled `sync/repository` run on the host commits the working tree to a `sync/<node>` branch (including the commons submodule pointer), rebases it onto `origin/main`, and pushes with `--force-with-lease`.
|
||||
2. A scheduled CI job in the node repository merges `sync/<node>` back into `main` under a bot identity.
|
||||
|
||||
This [capture-up](/apex/glossary#capture-up) loop means drift on a host becomes a commit — visible, diffable, and revertable — instead of silent divergence. The sync aborts rather than committing when the commons submodule is dirty or the `compositions/` tree is missing, so a broken checkout never becomes the recorded state.
|
||||
|
||||
> [!NOTE]
|
||||
> The CI merge job checks out the repository *without* submodules: the commons submodule uses an SSH URL that the default CI token cannot fetch, and the merge only moves the gitlink pointer anyway.
|
||||
|
||||
## Design constraints {#constraints}
|
||||
|
||||
- **Stdlib only.** The engine runs on a stock Debian Python 3 — no `pip`, no third-party imports. The optional `vendor/` directory exists for pinned pure-Python code but ships empty.
|
||||
- **Hand-authored composes.** There is no manifest layer or code generation; compose files are the source of truth.
|
||||
- **Evidence over convention.** Anything enforceable is enforced by an action (anchor-IP lint, crontab rollback, marker-block self-healing) rather than by documentation.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Concept: Actions and the overlay](/apex/concepts/actions-and-overlay)
|
||||
- [Concept: Node identity resolution](/apex/concepts/node-identity)
|
||||
- [Guide: Build a node repository](/apex/guides/node-repository)
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
title: Compositions and the shared core
|
||||
description: The include-based core composition, profiles, networks, and env layering.
|
||||
weight: 15
|
||||
---
|
||||
|
||||
A node's runtime is declared as [compositions](/apex/glossary#compositions): one docker-compose project per service under `compositions/<project>/`, plus the [core composition](/apex/glossary#core-composition) — the edge and infrastructure services every node needs. The core lives in the [commons](/apex/glossary#commons) and is consumed per node through a thin wrapper, so its definition is written once and versioned with the framework.
|
||||
|
||||
## The include wrapper {#include}
|
||||
|
||||
Each node's `compositions/apex/docker-compose.yml` is three lines:
|
||||
|
||||
```yaml {filename="compositions/apex/docker-compose.yml"}
|
||||
name: apex
|
||||
include:
|
||||
- ../../commons/compositions/apex/docker-compose.yml
|
||||
```
|
||||
|
||||
The shared file is fully `${APEX_*}`-parameterized and contains no relative host paths — every mount resolves through variables the [engine](/apex/glossary#engine) exports at invocation, so the same file is correct on every node.
|
||||
|
||||
A structural outlier — a node that genuinely needs to change a core service, such as publishing an extra host port — layers an override through the include path-list form instead of editing the shared file:
|
||||
|
||||
```yaml {filename="compositions/apex/docker-compose.yml"}
|
||||
name: apex
|
||||
include:
|
||||
- path:
|
||||
- ../../commons/compositions/apex/docker-compose.yml
|
||||
- ./traefik.override.yml
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> The `include:` directive requires Docker Compose v2.20 or newer.
|
||||
|
||||
## Core services and profiles {#core-services}
|
||||
|
||||
The core composition defines the fleet's shared service set. Always-on services:
|
||||
|
||||
- **traefik** — the edge reverse proxy and the anchor of the `direct` network (always `.10` in the node subnet). It terminates TLS via a DNS challenge, discovers backends through docker labels, and exposes its dashboard on the node FQDN behind an [enclave](/apex/glossary#enclave) IP allowlist.
|
||||
- **socket-proxy** — a filtered docker-socket proxy; traefik reads container state through it and never touches the socket directly.
|
||||
|
||||
Opt-in services, gated by compose profiles a node enables in `COMPOSE_PROFILES` (set in `apex.env`):
|
||||
|
||||
| Profile | Service | Role |
|
||||
|---|---|---|
|
||||
| `xray` | xray | Host-network proxy edge for the enclave. |
|
||||
| `adguard` | adguard | DNS filtering, sharing traefik's network namespace; routed at `/adguard` on the node FQDN. |
|
||||
| `alloy` | alloy | Per-node observability agent (Grafana Alloy). Collects host + container metrics and journal + container logs and **pushes** them outbound to a central Prometheus and Loki (see [Concept: Security model](/apex/concepts/security-model#observability)). Its container `hostname` is set to `APEX_NODE_HOST`, and that value stamps the `node` and `instance` labels on every metric and log stream. |
|
||||
| `manual` | resticontainer | One-shot label-driven backup runner (`ghcr.io/deytenit/resticontainer`) — never starts with `up`; invoked by the `backup/run` action, which discovers per-service backup intent from `restic.*` compose labels. |
|
||||
|
||||
## Networks {#networks}
|
||||
|
||||
The core composition owns the three node networks; service compositions reference them by key with `external: true`.
|
||||
|
||||
| Network | Type | Purpose |
|
||||
|---|---|---|
|
||||
| `direct` | bridge, subnet = `APEX_SUBNET` | Static-[anchor-IP](/apex/glossary#anchor-ip) network; every project's anchor holds a fixed address here (see [Concept: Fleet addressing scheme](/apex/concepts/addressing)). |
|
||||
| `enclave` | bridge, `internal: true` | Reverse-proxy backend network; traefik reaches labeled services here. |
|
||||
| `socket` | bridge, `internal: true` | Carries only traefik ↔ socket-proxy traffic. |
|
||||
|
||||
Within a project, sibling containers typically join the anchor's network namespace (`network_mode: service:<anchor>`) instead of attaching to networks themselves.
|
||||
|
||||
## Environment layering {#env-layering}
|
||||
|
||||
Values reach a composition from three layers, in increasing precedence:
|
||||
|
||||
1. **`.env`** — uncommitted secrets, a symlink into tier-1 storage (`APEX_RESTIC_*` credentials, `APEX_TRAEFIK_*` tokens, the UID/GID block).
|
||||
2. **`apex.env`** — committed non-secret scalars (`APEX_TRAEFIK_DIRECT_IP`, `APEX_RESTIC_REPOSITORY`, `COMPOSE_PROFILES`, optional `APEX_RESTIC_COMPRESSION`).
|
||||
3. **Engine exports** — the `APEX_*` identity and [tier](/apex/glossary#tier) paths from `ctx.vars()`, injected into the process environment by the orchestration actions and therefore winning over both files.
|
||||
|
||||
Critical variables carry `?error` markers in the shared file, so a missing secret fails composition config loudly instead of starting a misconfigured container.
|
||||
|
||||
## Orchestration {#orchestration}
|
||||
|
||||
`apex compose <action>` drives every composition on the node:
|
||||
|
||||
- **Order** — the core project first (it owns the networks), then every service composition in alphabetical order; `down` reverses the sequence.
|
||||
- **Env files** — for each project, `.env` and `apex.env` are passed as `--env-file` when present.
|
||||
- **`up`** implies `-d`; any extra arguments pass through to `docker compose`.
|
||||
- **`--dry-run`** prints each command with its working directory instead of executing — accepted anywhere on the command line.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Concept: Fleet addressing scheme](/apex/concepts/addressing)
|
||||
- [Guide: Add a service composition](/apex/guides/service-composition)
|
||||
- [Reference: Environment variables](/apex/reference/environment)
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
title: The execution context
|
||||
description: ctx — the single seam between an action and the host.
|
||||
weight: 14
|
||||
---
|
||||
|
||||
Every [action](/apex/glossary#action) receives a `ctx` object as its first argument. It is the only sanctioned way to touch the host: logging, process execution, template rendering, file mutation, firewall and service management, and notifications all go through it. Actions therefore stay declarative, and behavior that must be consistent fleet-wide (log format, backup discipline, notification payloads) lives in one place — the [engine](/apex/glossary#engine).
|
||||
|
||||
## Members {#members}
|
||||
|
||||
| Member | Purpose |
|
||||
|---|---|
|
||||
| `ctx.log` | Prefixed logging: `[<node>] [<action>] [LEVEL] message`; errors go to stderr, everything else to stdout. |
|
||||
| `ctx.sys` | Process execution (`run`, `sudo`, `ok`), systemd service helpers, and idempotent package installation. |
|
||||
| `ctx.tpl` | Template rendering over the `configs/` pillar with `string.Template.safe_substitute`. |
|
||||
| `ctx.host` | Idempotent host mutation: timestamped backups, atomic file writes, marker-block injection, crontab installation with rollback. |
|
||||
| `ctx.ufw` | Host and container firewall application (see [Concept: Security model](/apex/concepts/security-model)). |
|
||||
| `ctx.crowdsec` | CrowdSec installation, config deployment, and collection management. |
|
||||
| `ctx.systemd` | Rendering and enabling systemd units from `configs/systemd/`. |
|
||||
| `ctx.notify` | Telegram notifications; every method returns a boolean so actions can gate their exit code on delivery. |
|
||||
| `ctx.paths` | Resolved repository paths: `repo_root`, `commons`, `proprietaries`, `configs`, `compositions`, `core`, and the tier properties. |
|
||||
| `ctx.node` | The resolved [node identity](/apex/glossary#node-identity) record (`name`, `subnet`, `fqdn`, `host`); `None` in a standalone commons checkout. |
|
||||
| `ctx.commons` | Available only in an overriding proprietary action: `ctx.commons.run(args)` delegates to the shadowed commons implementation. |
|
||||
|
||||
Method-level signatures are listed in [Reference: Engine API](/apex/reference/engine-api).
|
||||
|
||||
## Template semantics {#templates}
|
||||
|
||||
`ctx.tpl` renders files from `configs/` with Python's `string.Template.safe_substitute`:
|
||||
|
||||
- Known `$VAR` and `${VAR}` references are replaced.
|
||||
- **Unknown references stay literal.** A typo in a template survives into the rendered output where it is visible, instead of silently collapsing to an empty string.
|
||||
- Shell command substitution (`$(...)`) passes through untouched, which keeps crontab entries with embedded `$(cat ...)` intact.
|
||||
|
||||
The substitution set is `ctx.vars()` — the full process environment plus the engine-provided `APEX_*` values below. Loops and conditionals never live in templates; they belong in the action.
|
||||
|
||||
## The variable export {#vars}
|
||||
|
||||
`ctx.vars()` returns the environment used for every template render, and its `APEX_*` subset is exported to `docker compose` by the orchestration actions. On a node it contains:
|
||||
|
||||
| Variable | Value |
|
||||
|---|---|
|
||||
| `APEX_COMMONS` | Absolute path of the commons directory (the launcher lives at `$APEX_COMMONS/apex`). |
|
||||
| `APEX_REPO_ROOT` | Absolute path of the node repository root. |
|
||||
| `APEX_NODE_HOST`, `APEX_NODE_FQDN`, `APEX_SUBNET` | The resolved identity fields. |
|
||||
| `APEX_TIER1`…`APEX_TIER3` | The core project's per-[tier](/apex/glossary#tier) data directories. |
|
||||
| `APEX_TIER1_SHARED`…`APEX_TIER3_SHARED` | The shared areas inside each tier. |
|
||||
| `APEX_TIER_ROOT1`…`APEX_TIER_ROOT3` | The tier storage roots (node-level links). |
|
||||
|
||||
The full origin/consumer matrix is in [Reference: Environment variables](/apex/reference/environment).
|
||||
|
||||
## Host mutation discipline {#host}
|
||||
|
||||
`ctx.host` encodes the framework's write rules:
|
||||
|
||||
- **Atomic writes.** `write_file` renders to a temporary file (mode `0644`, matching what a root shell redirect would create) and installs it with `mv` or `sudo cp`; overwriting an existing file preserves that file's mode.
|
||||
- **Timestamped backups.** When requested, the previous file is kept next to the target as `<name>.<epoch>.bak` before the write.
|
||||
- **Marker-block injection.** Managed blocks inside system files (for example `/etc/ufw/before.rules`) are delimited by start/end marker comments. Injection removes every prior managed block — including blocks left by predecessor tooling — before inserting the fresh one after its anchor line, so re-runs never accumulate.
|
||||
- **Crontab rollback.** `install_crontab` snapshots the current crontab, refuses an empty render (exit `65`), previews the result, and restores the snapshot if installation fails.
|
||||
|
||||
## Notifications {#notifications}
|
||||
|
||||
`ctx.notify` posts Telegram messages through the standard library (`urllib`). Payloads carry the instance name — the node's public host `APEX_NODE_HOST` — a status (`firing`/`resolved`), an escalation level, and the message truncated to 200 characters. Every method returns `True` only when the send succeeded; unattended actions treat a failed send of their final status message as a failure (exit `1`), so a broken notification channel is noticed instead of silently tolerated.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Reference: Engine API](/apex/reference/engine-api)
|
||||
- [Reference: Environment variables](/apex/reference/environment)
|
||||
- [Concept: Actions and the overlay](/apex/concepts/actions-and-overlay)
|
||||
@@ -0,0 +1,77 @@
|
||||
---
|
||||
title: "Node identity resolution"
|
||||
description: "How a node learns who it is: node.env identity keys and the FQDN fallback."
|
||||
weight: 12
|
||||
---
|
||||
|
||||
Every apex operation runs in the context of exactly one [node](/apex/glossary#node). The [engine](/apex/glossary#engine) derives that [node identity](/apex/glossary#node-identity) — node name, subnet, FQDN, and public host — from the `node.env` file at the repository root, with the OS hostname as a fallback for the FQDN only. The framework assumes no domain and no clustering: a node declares who it is explicitly. This resolution step succeeds the `root_require_node` mechanism of the bash "root" commons framework that apex replaces.
|
||||
|
||||
## Identity sources {#sources}
|
||||
|
||||
### node.env
|
||||
|
||||
`node.env` is a plain `KEY=value` file (shell-sourceable) located at the repository root. When parsed, comments and blank lines are ignored, and surrounding single or double quotes around values are stripped.
|
||||
|
||||
It declares identity explicitly through these keys:
|
||||
|
||||
- `APEX_NODE_FQDN` — the node's real FQDN, used for traefik `Host()` routing. If unset, the engine falls back to the OS hostname (and warns).
|
||||
- `APEX_NODE_HOST` — the node's public host name, used for the backup `--host` and the observability/notify `instance`. Defaults to `APEX_NODE_FQDN`.
|
||||
- `APEX_SUBNET` — the node's `/24` for the `direct` network. **Required**; there is no other source for it (see [Concept: Fleet addressing scheme](/apex/concepts/addressing)).
|
||||
|
||||
```ini {filename="node.env"}
|
||||
APEX_NODE_FQDN=node1.example.com
|
||||
APEX_NODE_HOST=node1.example.com
|
||||
APEX_SUBNET=198.18.16.0/24
|
||||
```
|
||||
|
||||
### The OS hostname
|
||||
|
||||
The OS hostname is consulted only as a fallback for `APEX_NODE_FQDN`. When `node.env` does not set `APEX_NODE_FQDN`, the engine uses the hostname and emits a warning. It is never parsed for a node name or any other coordinate.
|
||||
|
||||
## Resolution order {#resolution}
|
||||
|
||||
{{% steps %}}
|
||||
|
||||
### Resolve the FQDN
|
||||
|
||||
`fqdn` is `APEX_NODE_FQDN` from `node.env`, or the OS hostname when that key is unset. If neither yields a value, resolution fails (see [Failure modes](#failures)).
|
||||
|
||||
### Resolve the public host
|
||||
|
||||
`host` is `APEX_NODE_HOST` from `node.env`, or `fqdn` when that key is unset.
|
||||
|
||||
### Derive the short name
|
||||
|
||||
`name` is the **first label** of `host` (`host.split(".")[0]`) — for example, `node1` for `node1.example.com`. It is used for logs, the `sync/<node>` branch, and messages.
|
||||
|
||||
### Read the subnet
|
||||
|
||||
`subnet` is always read from `APEX_SUBNET` in `node.env`. It is required.
|
||||
|
||||
{{% /steps %}}
|
||||
|
||||
The result is a `Node` record with four fields: `name`, `subnet`, `fqdn`, and `host`.
|
||||
|
||||
## Warnings {#warnings}
|
||||
|
||||
Identity resolution emits exactly one warning; it signals a recoverable inconsistency that should be fixed.
|
||||
|
||||
| Condition | Warning message |
|
||||
|---|---|
|
||||
| `APEX_NODE_FQDN` is unset in `node.env` (the hostname is used instead) | `APEX_NODE_FQDN not set in node.env; using hostname '<fqdn>'.` |
|
||||
|
||||
## Failure modes {#failures}
|
||||
|
||||
Resolution refuses to guess. Two conditions are fatal, and both exit with code `66` — the engine's exit-code convention for missing input:
|
||||
|
||||
- **No FQDN at all.** `APEX_NODE_FQDN` is unset in `node.env` *and* the OS hostname is empty. The engine writes `identity: APEX_NODE_FQDN missing from node.env and no hostname` to stderr and exits.
|
||||
- **No subnet.** `APEX_SUBNET` is absent from `node.env` (including when the file itself is missing). The engine writes `identity: APEX_SUBNET missing from node.env` to stderr and exits.
|
||||
|
||||
> [!WARNING]
|
||||
> A valid `node.env` carrying `APEX_SUBNET` is required on every node. `APEX_NODE_FQDN` may be omitted only when the OS hostname supplies a usable FQDN.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [GitHub: apex.ermnvldmr.com](https://github.com/deytenit/apex.ermnvldmr.com)
|
||||
@@ -0,0 +1,53 @@
|
||||
---
|
||||
title: Security model
|
||||
description: Privilege isolation, firewall management, intrusion detection, and secret handling.
|
||||
weight: 18
|
||||
---
|
||||
|
||||
apex hardens a [node](/apex/glossary#node) in independent layers: filesystem privilege isolation, a managed host firewall, intrusion detection, container-level containment, and a strict secrets boundary. Each layer is applied by an [action](/apex/glossary#action) and is therefore reproducible and re-runnable.
|
||||
|
||||
## Privilege isolation {#privilege}
|
||||
|
||||
Every project runs as its own locked `noroot-<project>` system user; shared storage is group-writable with the sticky bit so projects can exchange files without being able to delete each other's. The full mechanism — users, groups, and the ownership matrix — is described in [Concept: Tiered storage](/apex/concepts/tiered-storage).
|
||||
|
||||
## Host firewall {#firewall}
|
||||
|
||||
`apex configure/ufw` applies the node's firewall from the `configs/ufw/` pillar in three parts:
|
||||
|
||||
- **Host rules** (`configs/ufw/host/host.rules`) — plain `ufw` commands, one per line, applied in order.
|
||||
- **Kernel-level rule blocks** (`configs/ufw/host/before.rules`, `after.rules`) — injected into `/etc/ufw/before.rules` and `/etc/ufw/after.rules` as managed marker blocks. Injection is **self-healing**: every previous managed block — including blocks left by predecessor tooling — is removed before the fresh block is inserted after ufw's `# End required lines` anchor, and a timestamped `.bak` of the target is kept beside it.
|
||||
- **Container rules** (`configs/ufw/docker/*.rules`) — applied through `ufw-docker`, which translates them into rules that survive Docker's own iptables manipulation. `configure/base` installs the helper to `~/.local/bin/ufw-docker`.
|
||||
|
||||
Missing configuration is an error, not a silent skip: an absent `docker/` or `host/` directory exits `66` when its counterpart exists, and a missing `/etc/ufw` target file aborts the run. The action finishes with a reminder that `ufw` must be restarted to take effect — and that the SSH port must be allowed first.
|
||||
|
||||
## Intrusion detection {#crowdsec}
|
||||
|
||||
`apex configure/crowdsec` installs CrowdSec through its official installer when `cscli` is absent, renders everything under `configs/crowdsec/` into `/etc/crowdsec/` (acquisition sources, parsers, bouncer configuration), and installs the `crowdsecurity/linux` and `crowdsecurity/traefik` collections. When the node's configs include a firewall-bouncer configuration, the matching bouncer package is installed as well, closing the loop from detection to blocking.
|
||||
|
||||
## Container containment {#containers}
|
||||
|
||||
The [core composition](/apex/glossary#core-composition) encodes container-level hardening:
|
||||
|
||||
- The edge proxy never touches the Docker socket: it discovers backends through a **filtered socket proxy** that exposes read-only container/service state on an isolated network.
|
||||
- `enclave` and `socket` are `internal: true` networks — no egress, no host port exposure.
|
||||
- Core services set `no-new-privileges` and run under the project's noroot identity wherever the workload allows; capabilities are granted per service, not globally.
|
||||
|
||||
## Observability {#observability}
|
||||
|
||||
Observability is an **outbound push**, not an exposed route. The optional `alloy` core service (Grafana Alloy, one agent per node) collects host metrics (a `node_exporter`-equivalent), container metrics (cadvisor), the systemd journal, and docker container logs, and pushes them to a **central** Prometheus and Loki that live on a designated node — the metrics via `prometheus.remote_write` to `APEX_PROM_PUSH_URL`, the logs via `loki.write` to `APEX_LOKI_PUSH_URL`. Both targets are authenticated with HTTP basic auth using `APEX_OBS_BASIC_AUTH_USER` / `APEX_OBS_BASIC_AUTH_PASS`. Because the push is outbound, Alloy attaches to the `direct` network (which has egress) rather than the `internal` `enclave`; there is no node-local log store and no `/loki` route to protect.
|
||||
|
||||
## Secrets boundary {#secrets}
|
||||
|
||||
- Secrets live only in per-project `.env` files, which are **symlinks into tier-1 storage** — outside the repository working tree and covered by `.gitignore` (`**/.env`).
|
||||
- Each node repository documents its expected keys in `SECRETS.md`; the file documents names, never values.
|
||||
- Committed configuration (`apex.env`) is restricted to non-secret scalars.
|
||||
- The shared composition marks critical secrets with `?error`, so a missing value fails fast at compose-config time.
|
||||
- [Capture-up](/apex/glossary#capture-up) commits are safe by construction: the ignored `.env` links and tier data never enter the git index.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Concept: Tiered storage and privilege isolation](/apex/concepts/tiered-storage)
|
||||
- [Concept: Compositions and the shared core](/apex/concepts/compositions)
|
||||
- [Reference: Actions](/apex/reference/actions)
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
title: Tiered storage and privilege isolation
|
||||
description: The @tier link structure, shared storage, and per-project noroot users.
|
||||
weight: 16
|
||||
---
|
||||
|
||||
Service data never lives inside a [node](/apex/glossary#node) repository. It lives in three storage [tiers](/apex/glossary#tier) — separate locations chosen per host — and reaches [compositions](/apex/glossary#compositions) through a standardized symlink structure. Ownership of that data is delegated to per-project system users, so no service container writes as root.
|
||||
|
||||
## The tier link structure {#links}
|
||||
|
||||
`apex tiers/link <tier1> <tier2> <tier3>` builds two layers of links:
|
||||
|
||||
- **Node-level:** `compositions/@tierN` → the tier storage root. These are what the engine exposes as `APEX_TIER_ROOT1..3`.
|
||||
- **Per-project:** for every `compositions/<project>/`, the action creates `<tier-root>/<project>/` and links `compositions/<project>/@tierN` → `<tier-root>/<project>`. Inside each tier it also maintains `shared/` with a per-project subdirectory, and a relative `shared` link inside every project's tier directory pointing back at it.
|
||||
|
||||
Each project additionally gets `compositions/<project>/.env` → `<tier1-root>/<project>/.env` (created empty if missing) — the uncommitted secrets file described in [Concept: Compositions](/apex/concepts/compositions). The `@tierN` links and `.env` files are ignored by git; the repository stays pure source.
|
||||
|
||||
If a path that should be a symlink exists as a real file or directory, the action warns and asks for confirmation before removing it; declining (including non-interactive EOF) skips that link and continues.
|
||||
|
||||
## Users and groups {#users}
|
||||
|
||||
`apex tiers/useradd` provisions one locked system user per project — `noroot-<project>` (name sanitized, its own primary group) — plus the fleet-wide `noroot-shared` group every project user joins. It then injects the resolved IDs into the project's `.env` between marker comments, replacing any previous block on re-runs:
|
||||
|
||||
```ini {filename="compositions/<project>/.env"}
|
||||
# START AUTO GENERATED APEX-NOROOT-USERS
|
||||
APEX_UID=998
|
||||
APEX_GID=998
|
||||
APEX_SHARED_GID=997
|
||||
# END AUTO GENERATED APEX-NOROOT-USERS
|
||||
```
|
||||
|
||||
Compositions consume these as `user: "${APEX_UID?error}:${APEX_GID?error}"` and `group_add: ["${APEX_SHARED_GID?error}"]`.
|
||||
|
||||
## The ownership matrix {#ownership}
|
||||
|
||||
`apex tiers/chown` enforces ownership across all three tiers. Without arguments it operates on the tier roots (`APEX_TIER_ROOT1..3`); explicit tier paths can override.
|
||||
|
||||
| Path | Ownership | Notes |
|
||||
|---|---|---|
|
||||
| `<tier-root>` | `root:root` | Directory inode only. |
|
||||
| `<tier-root>/shared` | `root:root` | Directory inode only — never recursive, so existing project data inside is untouched. |
|
||||
| `<tier-root>/node-infra` | `root:root` | Recursive. |
|
||||
| `<tier-root>/<project>` | `uid:gid` | Recursive; IDs read from the project `.env`. |
|
||||
| `<tier-root>/shared/<project>` | `uid:shared_gid` | Recursive, plus the sticky bit — collaborators can add files but only delete their own. |
|
||||
|
||||
Projects with an incomplete `APEX_UID`/`APEX_GID`/`APEX_SHARED_GID` set are skipped with a warning pointing at `tiers/useradd`.
|
||||
|
||||
## Engine tier variables {#variables}
|
||||
|
||||
| Variable | Resolves to |
|
||||
|---|---|
|
||||
| `APEX_TIER_ROOT1..3` | The tier storage roots (`compositions/@tierN`, fully resolved). |
|
||||
| `APEX_TIER1..3` | The core project's data directories inside each tier (`compositions/apex/@tierN`). |
|
||||
| `APEX_TIER1..3_SHARED` | The shared area inside the core project's tiers. |
|
||||
|
||||
The core composition mounts through these exclusively; the backup runner instead mounts the fleet data root `/srv` read-only and resolves each labeled service's container mounts back to their host paths.
|
||||
|
||||
## Backups {#backups}
|
||||
|
||||
`apex backup/run <telegram-bot-url>` drives the core `resticontainer` service (profile `manual`, one-shot) end to end. Backup intent is **label-driven**: resticontainer discovers every service carrying `restic.enable=true`, reads its `restic.backup.paths` (and optional `restic.backup.stop` / `restic.hooks.*`), resolves those container paths to host paths, and snapshots their union in a single restic run.
|
||||
|
||||
1. Checks repository reachability with `snapshots`; initializes the repository if needed.
|
||||
2. Runs `backup` with `--host <APEX_NODE_HOST>` and `--tag biweekly` — resticontainer runs each labeled service's pre-hook, stops any container flagged `restic.backup.stop`, snapshots the resolved host paths, then restarts and runs post-hooks. The run refuses to report success if no new snapshot lands (a guard against zero labels being discovered).
|
||||
3. Applies retention with `forget --prune` (7 daily, 4 weekly, 12 monthly).
|
||||
4. Sends a Telegram status either way; a success whose notification cannot be delivered still exits `1` so schedulers notice.
|
||||
|
||||
The repository location and credentials come entirely from `APEX_RESTIC_*` configuration (see [Reference: Environment variables](/apex/reference/environment)). Which data is captured is decided per service by its `restic.*` labels — for example the core services back up only their configuration (traefik dynamic rules, xray/adguard config), leaving reproducible tier-3 data such as caches and logs out of the set.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Concept: Security model](/apex/concepts/security-model)
|
||||
- [Reference: Actions](/apex/reference/actions)
|
||||
- [Guide: Add a service composition](/apex/guides/service-composition)
|
||||
Reference in new issue
Block a user