fix, feat, doc: massive build system refactorion, bug fixes and apex documentation (#19)
This commit is contained in:
deytenit authored and 
GitHub committed
1 parent
e8b42c60c2
commit
91f5245823
165 files changed
+8598
-1205
No files matched your search
@@ -0,0 +1,37 @@
|
||||
---
|
||||
title: Documentation of the apex fleet operations framework
|
||||
description: "Documentation for the apex fleet operations framework: a stdlib-python engine and a shared compose core for self-hosted nodes."
|
||||
weight: 1
|
||||
type: docs
|
||||
cascade:
|
||||
type: docs
|
||||
---
|
||||
|
||||
`apex` is the [commons](/apex/glossary#commons) of a fleet of self-hosted [nodes](/apex/glossary#node): a python3-stdlib [engine](/apex/glossary#engine) that runs grouped [actions](/apex/glossary#action) as `apex <group>/<action>`, paired with a shared, `${env}`-parameterized docker-compose core. Every node repository consumes the framework as the pinned `commons/` git submodule. apex succeeds the bash `root` commons framework.
|
||||
|
||||
The stack is deliberately small: `git` plus the Python 3 standard library. There is no `pip`, Node, Go, or CUE, and compose files are hand-authored — no code generation.
|
||||
|
||||
## Core features
|
||||
|
||||
- **Single entrypoint on the local checkout** — `apex` always acts on the checkout it runs from; there is no `<node>` argument to pass.
|
||||
- **Explicit identity from `node.env`** — [node identity](/apex/glossary#node-identity) is declared per node in `node.env` (`APEX_NODE_FQDN`, `APEX_NODE_HOST`, `APEX_SUBNET`); the framework assumes no particular domain, and the short node `name` is just the first label of the host.
|
||||
- **Four-pillar node repositories** — every node repository follows the same shape: the pinned `commons/` submodule, node-local [proprietaries](/apex/glossary#proprietaries), the node's [compositions](/apex/glossary#compositions), and its `node.env`.
|
||||
- **Proprietary overlay** — node-local proprietaries are [overlaid](/apex/glossary#overlay) over the shared actions; commons holds only what every node shares.
|
||||
- **Tiered storage with privilege isolation** — standardized storage [tiers](/apex/glossary#tier) with per-project privilege isolation.
|
||||
- **Shared core composition** — the [core composition](/apex/glossary#core-composition) (`compositions/apex/docker-compose.yml`) is `${env}`-parameterized and included in every node's compositions.
|
||||
|
||||
## Sections
|
||||
|
||||
{{% cards %}}
|
||||
{{< card link="getting-started" title="Getting started" icon="map" subtitle="Prerequisites, bootstrap, and the first actions on a fresh node." >}}
|
||||
{{< card link="concepts" title="Concepts" icon="book-open" subtitle="Architecture, identity, the overlay, and the storage model." >}}
|
||||
{{< card link="guides" title="Guides" icon="map" subtitle="Task-oriented guides for operating and extending a node." >}}
|
||||
{{< card link="reference" title="Reference" icon="code" subtitle="Actions, the engine helper library, and exit codes." >}}
|
||||
{{< card link="troubleshooting" title="Troubleshooting" icon="shield-check" subtitle="Symptoms, diagnosis, and fixes for common failures." >}}
|
||||
{{% /cards %}}
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [GitHub: apex.ermnvldmr.com](https://github.com/deytenit/apex.ermnvldmr.com)
|
||||
@@ -0,0 +1,25 @@
|
||||
---
|
||||
title: Concepts
|
||||
description: "Architecture, identity, actions, execution context, compositions, storage, addressing, and security."
|
||||
weight: 10
|
||||
---
|
||||
|
||||
Theoretical foundations of the apex fleet-operations framework: how a [node](/apex/glossary#node) declares its [identity](/apex/concepts/node-identity), how [actions](/apex/glossary#action) execute, and how a node organizes its services, storage, and addressing.
|
||||
|
||||
{{% cards %}}
|
||||
{{< card link="architecture" title="Architecture" icon="share" subtitle="The engine, commons, and configs layers and how they fit together." >}}
|
||||
{{< card link="node-identity" title="Node identity" icon="finger-print" subtitle="How a node declares who it is and where it belongs." >}}
|
||||
{{< card link="actions-and-overlay" title="Actions and overlay" icon="lightning-bolt" subtitle="Shared actions and node-local overrides." >}}
|
||||
{{< card link="execution-context" title="Execution context" icon="terminal" subtitle="The environment every action receives at run time." >}}
|
||||
{{< card link="compositions" title="Compositions" icon="cube" subtitle="How service stacks are declared and deployed per node." >}}
|
||||
{{< card link="tiered-storage" title="Tiered storage" icon="database" subtitle="Standardized storage classification for service data." >}}
|
||||
{{< card link="addressing" title="Addressing" icon="globe-alt" subtitle="The node subnet, anchor IPs, and the enclave convention." >}}
|
||||
{{< card link="security-model" title="Security model" icon="shield-check" subtitle="Privilege isolation and network hardening." >}}
|
||||
{{% /cards %}}
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Guides: Operating an apex fleet](/apex/guides)
|
||||
- [Reference: Commands and interfaces](/apex/reference)
|
||||
@@ -0,0 +1,111 @@
|
||||
---
|
||||
title: "Actions and the overlay"
|
||||
description: "The action contract and how node-local proprietaries shadow commons actions."
|
||||
weight: 13
|
||||
---
|
||||
|
||||
Every unit of automation in apex is an [action](/apex/glossary#action): a single Python module that the [engine](/apex/glossary#engine) dispatches by name. The [overlay](/apex/glossary#overlay) lets a node's [proprietaries](/apex/glossary#proprietaries) add, override, or disable [commons](/apex/glossary#commons) actions without forking them. This page defines the action contract, the declarative argument descriptors, and the resolution rules.
|
||||
|
||||
## The action contract {#contract}
|
||||
|
||||
An action is one `.py` file under an `actions/` tree. The file path **is** the action name: `actions/<group>/<action>.py` is invoked as `apex <group>/<action> [args...]`. There is no registry to update — dropping a file into the tree publishes the action.
|
||||
|
||||
The module must expose two things:
|
||||
|
||||
- **`METADATA = Meta(...)`** → the summary and argument descriptors that drive both the help text and parsing. The descriptor system succeeds the `action_metadata()` heredoc of the bash "root" commons framework, apex's predecessor.
|
||||
- **`run(ctx, args)`** → the entrypoint. `ctx` is the engine-built execution context; `args` is the parsed `argparse` namespace.
|
||||
|
||||
```python {filename="actions/<group>/<action>.py"}
|
||||
from engine.descriptor import Meta, Arg, Flag
|
||||
|
||||
METADATA = Meta(
|
||||
summary="One-line summary shown in the action listing",
|
||||
args=[
|
||||
Arg("target", help="A required positional argument"),
|
||||
Flag("--dry-run", help="A boolean flag"),
|
||||
],
|
||||
)
|
||||
|
||||
def run(ctx, args):
|
||||
...
|
||||
return 0
|
||||
```
|
||||
|
||||
The return value of `run` becomes the process exit code. A non-integer return (including `None`) is treated as success (`0`). An unhandled exception is logged with the context prefix and exits `1`.
|
||||
|
||||
> [!NOTE]
|
||||
> `METADATA` is technically optional — a module without it is dispatched with an empty `Meta` and accepts no arguments — but every real action should declare it, since the listing and `--help` output are generated from it.
|
||||
|
||||
## Declarative arguments {#arguments}
|
||||
|
||||
`Meta.args` is a list of descriptors. Each maps to exactly one `argparse` construct:
|
||||
|
||||
| Descriptor | Maps to | Behavior |
|
||||
| --- | --- | --- |
|
||||
| `Arg("name")` | positional argument | Required by default; `required=False` makes it an optional positional (`nargs="?"`, default `None`). |
|
||||
| `Opt("--name", default=...)` | value option | An optional flag that takes a value, with a declared default. |
|
||||
| `Flag("--name")` | boolean flag | `store_true`; present means `True`. |
|
||||
| `Rest("name")` | remainder | Greedy passthrough of all remaining argv (`argparse.REMAINDER`). |
|
||||
|
||||
The engine builds the parser as `apex <group>/<action>` and parses the remaining argv before calling `run`. `--help` is handled at parse time. A usage error — unknown flag, missing positional — prints usage and exits `64` (`EX_USAGE`) instead of argparse's default `2`.
|
||||
|
||||
## Listing without importing {#listing}
|
||||
|
||||
`apex` with no arguments (or `--help`) lists every available action. Listing never imports action modules: the engine statically scans each file with the `ast` module and extracts only the `METADATA` summary and any `DISABLED` constant. Action code therefore cannot run — or crash — at list time; a file that fails to parse is listed as `(unparseable)` rather than breaking the listing.
|
||||
|
||||
Each entry is tagged with its overlay source:
|
||||
|
||||
- no tag → a commons action,
|
||||
- `(local)` → exists only in proprietaries,
|
||||
- `(local override)` → a proprietaries action shadowing a commons one,
|
||||
- `[DISABLED: reason]` → the action declares `DISABLED`.
|
||||
|
||||
## The overlay {#overlay}
|
||||
|
||||
Action names are resolved across two trees, proprietaries first:
|
||||
|
||||
{{< filetree/container >}}
|
||||
{{< filetree/folder name="node repository" state="open" >}}
|
||||
{{< filetree/file name="node.env" >}}
|
||||
{{< filetree/folder name="commons" state="open" >}}
|
||||
{{< filetree/folder name="engine" state="closed" >}}{{< /filetree/folder >}}
|
||||
{{< filetree/folder name="actions" state="open" >}}
|
||||
{{< filetree/file name="<group>/<action>.py" >}}
|
||||
{{< /filetree/folder >}}
|
||||
{{< /filetree/folder >}}
|
||||
{{< filetree/folder name="proprietaries" state="open" >}}
|
||||
{{< filetree/folder name="actions" state="open" >}}
|
||||
{{< filetree/file name="<group>/<action>.py" >}}
|
||||
{{< /filetree/folder >}}
|
||||
{{< /filetree/folder >}}
|
||||
{{< /filetree/folder >}}
|
||||
{{< /filetree/container >}}
|
||||
|
||||
The engine detects this layout by its own location: when it runs from a `commons/` directory whose parent contains `node.env`, the parent is the [node](/apex/glossary#node) repository root and `proprietaries/actions/` is the overlay tree. A standalone commons checkout is its own root and has no overlay layer. Inside a node repository, [node identity](/apex/glossary#node-identity) is resolved from the host FQDN before dispatch.
|
||||
|
||||
Resolution for a given name checks `proprietaries/actions/<name>.py` first and `commons/actions/<name>.py` second; exactly one module is loaded by path. An unknown name is an error (exit `1`). The overlay supports four moves:
|
||||
|
||||
- **Add** → a file that exists only in proprietaries becomes a node-local action, listed as `(local)`.
|
||||
- **Override** → a proprietaries file at the same relative path shadows the commons implementation entirely; the commons path is recorded on the resolution as `shadowed`.
|
||||
- **Wrap** → an overriding module can delegate to the implementation it shadows via `ctx.commons.run(args)`, adding node-local behavior before or after the commons logic instead of replacing it.
|
||||
- **Disable** → a module that assigns `DISABLED = "reason"` is refused at dispatch: the engine logs the reason and exits `1`. The listing shows `[DISABLED: reason]`. A bare `DISABLED = True` also disables, with the generic reason `disabled`; an empty string does not.
|
||||
|
||||
## Exit codes {#exit-codes}
|
||||
|
||||
Actions and the engine follow a fixed exit-code convention:
|
||||
|
||||
| Code | Meaning | Typical cause |
|
||||
| --- | --- | --- |
|
||||
| `0` | success | `run` returned `0` or a non-integer. |
|
||||
| `1` | general error | Unknown action, disabled action, unhandled exception, non-integer `SystemExit`. |
|
||||
| `64` | usage error | Arguments rejected by the parser (`EX_USAGE`). |
|
||||
| `65` | data error | Malformed input data (`EX_DATAERR`), raised by engine libraries. |
|
||||
| `66` | missing input | A required input is absent (`EX_NOINPUT`), raised by engine libraries. |
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Concept: Framework architecture](/apex/concepts/architecture)
|
||||
- [Concept: The execution context](/apex/concepts/execution-context)
|
||||
- [GitHub: apex.ermnvldmr.com](https://github.com/deytenit/apex.ermnvldmr.com)
|
||||
@@ -0,0 +1,46 @@
|
||||
---
|
||||
title: "Fleet addressing scheme"
|
||||
description: "Per-node /24 subnets and static anchor IPs inside the enclave space (default 198.18.0.0/16)."
|
||||
weight: 17
|
||||
---
|
||||
|
||||
Apex assigns every service a fixed, predictable IPv4 address inside a per-node subnet. Each node declares its own `/24`, service anchors are static addresses within it, and the whole scheme is enforced mechanically by the anchor-IP lint.
|
||||
|
||||
## Address space {#address-space}
|
||||
|
||||
The [enclave](/apex/glossary#enclave) space defaults to `198.18.0.0/16`, the benchmarking range set aside by [RFC 2544: Benchmarking Methodology for Network Interconnect Devices](https://datatracker.ietf.org/doc/html/rfc2544). This range is never routed on the public Internet and does not collide with the common private ranges (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`) that cloud providers and home networks already consume. The operator may override it via `APEX_ENCLAVE_SUBNET`.
|
||||
|
||||
## Node subnets {#node-subnets}
|
||||
|
||||
Each [node](/apex/glossary#node) declares exactly one `/24` as `APEX_SUBNET` in its `node.env`, and that subnet is the `direct` network's IPAM range. The operator assigns it; conventionally it sits inside the enclave space (`198.18.0.0/16` by default, or whatever `APEX_ENCLAVE_SUBNET` is set to), but the framework imposes no arithmetic on the choice.
|
||||
|
||||
> [!NOTE]
|
||||
> As an optional operator convention, you may organize node `/24`s into contiguous blocks (for example, grouping related nodes) to keep allocations legible. This is a bookkeeping choice, not a framework rule.
|
||||
|
||||
## Allocation inside a node /24 {#node-allocation}
|
||||
|
||||
Within a node's `/24`, host addresses are partitioned by role:
|
||||
|
||||
| Range | Purpose |
|
||||
|---|---|
|
||||
| `.10`–`.19` | Reserved for [core composition](/apex/glossary#core-composition) anchors; `traefik` is always `.10` (via `APEX_TRAEFIK_DIRECT_IP`) |
|
||||
| `.20` and up | Service anchors, allocated per `<project>` |
|
||||
|
||||
Every service anchor is a static `ipv4_address` on the direct network — a fixed [anchor IP](/apex/glossary#anchor-ip) rather than a dynamically assigned lease. Allocations are recorded in `compositions/ADDRESSING.md`, which serves as the human-readable register for the node's subnet.
|
||||
|
||||
## Enforcement {#enforcement}
|
||||
|
||||
The compose lint validates anchor IPs across all [compositions](/apex/glossary#compositions) on a node. For every `ipv4_address` found in `compositions/*/docker-compose.yml` it checks that the address is:
|
||||
|
||||
- **Well-formed** — a malformed literal is reported as a lint error, not a crash.
|
||||
- **Unique** — the same anchor IP may not appear in two projects; duplicates are reported with both offending projects.
|
||||
- **Inside the node subnet** — every anchor must fall within the node's declared `/24`.
|
||||
|
||||
Any violation fails the lint with exit code `1` (general error). Passing validation logs `Anchor IP validation passed.` The implementation lives in the `utils/lint-docker-compose` [action](/apex/glossary#action); see [GitHub: apex.ermnvldmr.com](https://github.com/deytenit/apex.ermnvldmr.com).
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [GitHub: apex.ermnvldmr.com](https://github.com/deytenit/apex.ermnvldmr.com)
|
||||
- [RFC 2544: Benchmarking Methodology for Network Interconnect Devices](https://datatracker.ietf.org/doc/html/rfc2544)
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
title: Framework architecture
|
||||
description: Four-pillar node repositories, the commons submodule, and the dispatch flow.
|
||||
weight: 11
|
||||
---
|
||||
|
||||
apex splits fleet automation into one shared repository — the [commons](/apex/glossary#commons) — and one repository per [node](/apex/glossary#node). The commons carries everything every node shares: the [engine](/apex/glossary#engine), the shared [actions](/apex/glossary#action), and the [core composition](/apex/glossary#core-composition). A node repository carries everything that makes that node itself: its identity, host [configs](/apex/glossary#configs), service [compositions](/apex/glossary#compositions), and node-local [proprietaries](/apex/glossary#proprietaries).
|
||||
|
||||
## The four pillars {#pillars}
|
||||
|
||||
Every node repository has the same shape:
|
||||
|
||||
{{< filetree/container >}}
|
||||
{{< filetree/folder name="<node>.<your-domain>" state="open" >}}
|
||||
{{< filetree/folder name="commons" state="closed" >}}{{< /filetree/folder >}}
|
||||
{{< filetree/folder name="proprietaries" state="closed" >}}{{< /filetree/folder >}}
|
||||
{{< filetree/folder name="configs" state="closed" >}}{{< /filetree/folder >}}
|
||||
{{< filetree/folder name="compositions" state="closed" >}}{{< /filetree/folder >}}
|
||||
{{< filetree/file name="node.env" >}}
|
||||
{{< filetree/file name="init.sh" >}}
|
||||
{{< /filetree/folder >}}
|
||||
{{< /filetree/container >}}
|
||||
|
||||
| Pillar | Content | Ownership |
|
||||
|---|---|---|
|
||||
| `commons/` | Git submodule → `apex.ermnvldmr.com`, pinned to a release tag | Shared, versioned |
|
||||
| `proprietaries/` | Node-local actions, libraries, and tool sources ([overlaid](/apex/glossary#overlay) over commons) | Node |
|
||||
| `configs/` | Host configuration sources: `ufw/`, `cron/crontab`, `systemd/`, `crowdsec/` | Node |
|
||||
| `compositions/` | One directory per project plus the `apex/` core wrapper and the addressing ledger | Node |
|
||||
|
||||
The submodule pin is the framework's version contract: a node runs exactly the engine and core composition recorded in its gitlink. Upgrading a node is a deliberate act — check out a newer tag inside `commons/` and commit the moved pointer.
|
||||
|
||||
## Dispatch flow {#dispatch}
|
||||
|
||||
The launcher `commons/apex` acts on the local checkout; there is no node argument anywhere.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["apex <group>/<action> [args]"] --> B{Layout}
|
||||
B -- "commons/ dir with node.env beside it" --> C[Node checkout:<br/>overlay = proprietaries/actions]
|
||||
B -- otherwise --> D[Standalone commons:<br/>no overlay]
|
||||
C --> E["Overlay resolve<br/>(proprietaries first)"]
|
||||
D --> E
|
||||
E --> F["Identity resolve<br/>(FQDN + node.env)"]
|
||||
F --> G["Build ctx<br/>(log, sys, tpl, host, ufw,<br/>crowdsec, systemd, notify)"]
|
||||
G --> H["parse args (Meta)"]
|
||||
H --> I["run(ctx, args)"]
|
||||
```
|
||||
|
||||
The engine detects the layout from its own location: running from a directory named `commons/` whose parent contains `node.env` means "node checkout" — the parent becomes the repository root and `proprietaries/actions/` becomes the overlay tree. Any other location is a standalone commons checkout acting as its own root.
|
||||
|
||||
Listing (`apex --help`) never imports action modules; summaries and disabled markers are extracted by static `ast` scanning. Dispatch loads exactly one module by path. Details in [Concept: Actions and the overlay](/apex/concepts/actions-and-overlay).
|
||||
|
||||
## The capture-up model {#capture-up}
|
||||
|
||||
Node repositories are ledgers, not just sources: the host itself records what it runs.
|
||||
|
||||
1. A scheduled `sync/repository` run on the host commits the working tree to a `sync/<node>` branch (including the commons submodule pointer), rebases it onto `origin/main`, and pushes with `--force-with-lease`.
|
||||
2. A scheduled CI job in the node repository merges `sync/<node>` back into `main` under a bot identity.
|
||||
|
||||
This [capture-up](/apex/glossary#capture-up) loop means drift on a host becomes a commit — visible, diffable, and revertable — instead of silent divergence. The sync aborts rather than committing when the commons submodule is dirty or the `compositions/` tree is missing, so a broken checkout never becomes the recorded state.
|
||||
|
||||
> [!NOTE]
|
||||
> The CI merge job checks out the repository *without* submodules: the commons submodule uses an SSH URL that the default CI token cannot fetch, and the merge only moves the gitlink pointer anyway.
|
||||
|
||||
## Design constraints {#constraints}
|
||||
|
||||
- **Stdlib only.** The engine runs on a stock Debian Python 3 — no `pip`, no third-party imports. The optional `vendor/` directory exists for pinned pure-Python code but ships empty.
|
||||
- **Hand-authored composes.** There is no manifest layer or code generation; compose files are the source of truth.
|
||||
- **Evidence over convention.** Anything enforceable is enforced by an action (anchor-IP lint, crontab rollback, marker-block self-healing) rather than by documentation.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Concept: Actions and the overlay](/apex/concepts/actions-and-overlay)
|
||||
- [Concept: Node identity resolution](/apex/concepts/node-identity)
|
||||
- [Guide: Build a node repository](/apex/guides/node-repository)
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
title: Compositions and the shared core
|
||||
description: The include-based core composition, profiles, networks, and env layering.
|
||||
weight: 15
|
||||
---
|
||||
|
||||
A node's runtime is declared as [compositions](/apex/glossary#compositions): one docker-compose project per service under `compositions/<project>/`, plus the [core composition](/apex/glossary#core-composition) — the edge and infrastructure services every node needs. The core lives in the [commons](/apex/glossary#commons) and is consumed per node through a thin wrapper, so its definition is written once and versioned with the framework.
|
||||
|
||||
## The include wrapper {#include}
|
||||
|
||||
Each node's `compositions/apex/docker-compose.yml` is three lines:
|
||||
|
||||
```yaml {filename="compositions/apex/docker-compose.yml"}
|
||||
name: apex
|
||||
include:
|
||||
- ../../commons/compositions/apex/docker-compose.yml
|
||||
```
|
||||
|
||||
The shared file is fully `${APEX_*}`-parameterized and contains no relative host paths — every mount resolves through variables the [engine](/apex/glossary#engine) exports at invocation, so the same file is correct on every node.
|
||||
|
||||
A structural outlier — a node that genuinely needs to change a core service, such as publishing an extra host port — layers an override through the include path-list form instead of editing the shared file:
|
||||
|
||||
```yaml {filename="compositions/apex/docker-compose.yml"}
|
||||
name: apex
|
||||
include:
|
||||
- path:
|
||||
- ../../commons/compositions/apex/docker-compose.yml
|
||||
- ./traefik.override.yml
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> The `include:` directive requires Docker Compose v2.20 or newer.
|
||||
|
||||
## Core services and profiles {#core-services}
|
||||
|
||||
The core composition defines the fleet's shared service set. Always-on services:
|
||||
|
||||
- **traefik** — the edge reverse proxy and the anchor of the `direct` network (always `.10` in the node subnet). It terminates TLS via a DNS challenge, discovers backends through docker labels, and exposes its dashboard on the node FQDN behind an [enclave](/apex/glossary#enclave) IP allowlist.
|
||||
- **socket-proxy** — a filtered docker-socket proxy; traefik reads container state through it and never touches the socket directly.
|
||||
|
||||
Opt-in services, gated by compose profiles a node enables in `COMPOSE_PROFILES` (set in `apex.env`):
|
||||
|
||||
| Profile | Service | Role |
|
||||
|---|---|---|
|
||||
| `xray` | xray | Host-network proxy edge for the enclave. |
|
||||
| `adguard` | adguard | DNS filtering, sharing traefik's network namespace; routed at `/adguard` on the node FQDN. |
|
||||
| `alloy` | alloy | Per-node observability agent (Grafana Alloy). Collects host + container metrics and journal + container logs and **pushes** them outbound to a central Prometheus and Loki (see [Concept: Security model](/apex/concepts/security-model#observability)). Its container `hostname` is set to `APEX_NODE_HOST`, and that value stamps the `node` and `instance` labels on every metric and log stream. |
|
||||
| `manual` | resticontainer | One-shot label-driven backup runner (`ghcr.io/deytenit/resticontainer`) — never starts with `up`; invoked by the `backup/run` action, which discovers per-service backup intent from `restic.*` compose labels. |
|
||||
|
||||
## Networks {#networks}
|
||||
|
||||
The core composition owns the three node networks; service compositions reference them by key with `external: true`.
|
||||
|
||||
| Network | Type | Purpose |
|
||||
|---|---|---|
|
||||
| `direct` | bridge, subnet = `APEX_SUBNET` | Static-[anchor-IP](/apex/glossary#anchor-ip) network; every project's anchor holds a fixed address here (see [Concept: Fleet addressing scheme](/apex/concepts/addressing)). |
|
||||
| `enclave` | bridge, `internal: true` | Reverse-proxy backend network; traefik reaches labeled services here. |
|
||||
| `socket` | bridge, `internal: true` | Carries only traefik ↔ socket-proxy traffic. |
|
||||
|
||||
Within a project, sibling containers typically join the anchor's network namespace (`network_mode: service:<anchor>`) instead of attaching to networks themselves.
|
||||
|
||||
## Environment layering {#env-layering}
|
||||
|
||||
Values reach a composition from three layers, in increasing precedence:
|
||||
|
||||
1. **`.env`** — uncommitted secrets, a symlink into tier-1 storage (`APEX_RESTIC_*` credentials, `APEX_TRAEFIK_*` tokens, the UID/GID block).
|
||||
2. **`apex.env`** — committed non-secret scalars (`APEX_TRAEFIK_DIRECT_IP`, `APEX_RESTIC_REPOSITORY`, `COMPOSE_PROFILES`, optional `APEX_RESTIC_COMPRESSION`).
|
||||
3. **Engine exports** — the `APEX_*` identity and [tier](/apex/glossary#tier) paths from `ctx.vars()`, injected into the process environment by the orchestration actions and therefore winning over both files.
|
||||
|
||||
Critical variables carry `?error` markers in the shared file, so a missing secret fails composition config loudly instead of starting a misconfigured container.
|
||||
|
||||
## Orchestration {#orchestration}
|
||||
|
||||
`apex compose <action>` drives every composition on the node:
|
||||
|
||||
- **Order** — the core project first (it owns the networks), then every service composition in alphabetical order; `down` reverses the sequence.
|
||||
- **Env files** — for each project, `.env` and `apex.env` are passed as `--env-file` when present.
|
||||
- **`up`** implies `-d`; any extra arguments pass through to `docker compose`.
|
||||
- **`--dry-run`** prints each command with its working directory instead of executing — accepted anywhere on the command line.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Concept: Fleet addressing scheme](/apex/concepts/addressing)
|
||||
- [Guide: Add a service composition](/apex/guides/service-composition)
|
||||
- [Reference: Environment variables](/apex/reference/environment)
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
title: The execution context
|
||||
description: ctx — the single seam between an action and the host.
|
||||
weight: 14
|
||||
---
|
||||
|
||||
Every [action](/apex/glossary#action) receives a `ctx` object as its first argument. It is the only sanctioned way to touch the host: logging, process execution, template rendering, file mutation, firewall and service management, and notifications all go through it. Actions therefore stay declarative, and behavior that must be consistent fleet-wide (log format, backup discipline, notification payloads) lives in one place — the [engine](/apex/glossary#engine).
|
||||
|
||||
## Members {#members}
|
||||
|
||||
| Member | Purpose |
|
||||
|---|---|
|
||||
| `ctx.log` | Prefixed logging: `[<node>] [<action>] [LEVEL] message`; errors go to stderr, everything else to stdout. |
|
||||
| `ctx.sys` | Process execution (`run`, `sudo`, `ok`), systemd service helpers, and idempotent package installation. |
|
||||
| `ctx.tpl` | Template rendering over the `configs/` pillar with `string.Template.safe_substitute`. |
|
||||
| `ctx.host` | Idempotent host mutation: timestamped backups, atomic file writes, marker-block injection, crontab installation with rollback. |
|
||||
| `ctx.ufw` | Host and container firewall application (see [Concept: Security model](/apex/concepts/security-model)). |
|
||||
| `ctx.crowdsec` | CrowdSec installation, config deployment, and collection management. |
|
||||
| `ctx.systemd` | Rendering and enabling systemd units from `configs/systemd/`. |
|
||||
| `ctx.notify` | Telegram notifications; every method returns a boolean so actions can gate their exit code on delivery. |
|
||||
| `ctx.paths` | Resolved repository paths: `repo_root`, `commons`, `proprietaries`, `configs`, `compositions`, `core`, and the tier properties. |
|
||||
| `ctx.node` | The resolved [node identity](/apex/glossary#node-identity) record (`name`, `subnet`, `fqdn`, `host`); `None` in a standalone commons checkout. |
|
||||
| `ctx.commons` | Available only in an overriding proprietary action: `ctx.commons.run(args)` delegates to the shadowed commons implementation. |
|
||||
|
||||
Method-level signatures are listed in [Reference: Engine API](/apex/reference/engine-api).
|
||||
|
||||
## Template semantics {#templates}
|
||||
|
||||
`ctx.tpl` renders files from `configs/` with Python's `string.Template.safe_substitute`:
|
||||
|
||||
- Known `$VAR` and `${VAR}` references are replaced.
|
||||
- **Unknown references stay literal.** A typo in a template survives into the rendered output where it is visible, instead of silently collapsing to an empty string.
|
||||
- Shell command substitution (`$(...)`) passes through untouched, which keeps crontab entries with embedded `$(cat ...)` intact.
|
||||
|
||||
The substitution set is `ctx.vars()` — the full process environment plus the engine-provided `APEX_*` values below. Loops and conditionals never live in templates; they belong in the action.
|
||||
|
||||
## The variable export {#vars}
|
||||
|
||||
`ctx.vars()` returns the environment used for every template render, and its `APEX_*` subset is exported to `docker compose` by the orchestration actions. On a node it contains:
|
||||
|
||||
| Variable | Value |
|
||||
|---|---|
|
||||
| `APEX_COMMONS` | Absolute path of the commons directory (the launcher lives at `$APEX_COMMONS/apex`). |
|
||||
| `APEX_REPO_ROOT` | Absolute path of the node repository root. |
|
||||
| `APEX_NODE_HOST`, `APEX_NODE_FQDN`, `APEX_SUBNET` | The resolved identity fields. |
|
||||
| `APEX_TIER1`…`APEX_TIER3` | The core project's per-[tier](/apex/glossary#tier) data directories. |
|
||||
| `APEX_TIER1_SHARED`…`APEX_TIER3_SHARED` | The shared areas inside each tier. |
|
||||
| `APEX_TIER_ROOT1`…`APEX_TIER_ROOT3` | The tier storage roots (node-level links). |
|
||||
|
||||
The full origin/consumer matrix is in [Reference: Environment variables](/apex/reference/environment).
|
||||
|
||||
## Host mutation discipline {#host}
|
||||
|
||||
`ctx.host` encodes the framework's write rules:
|
||||
|
||||
- **Atomic writes.** `write_file` renders to a temporary file (mode `0644`, matching what a root shell redirect would create) and installs it with `mv` or `sudo cp`; overwriting an existing file preserves that file's mode.
|
||||
- **Timestamped backups.** When requested, the previous file is kept next to the target as `<name>.<epoch>.bak` before the write.
|
||||
- **Marker-block injection.** Managed blocks inside system files (for example `/etc/ufw/before.rules`) are delimited by start/end marker comments. Injection removes every prior managed block — including blocks left by predecessor tooling — before inserting the fresh one after its anchor line, so re-runs never accumulate.
|
||||
- **Crontab rollback.** `install_crontab` snapshots the current crontab, refuses an empty render (exit `65`), previews the result, and restores the snapshot if installation fails.
|
||||
|
||||
## Notifications {#notifications}
|
||||
|
||||
`ctx.notify` posts Telegram messages through the standard library (`urllib`). Payloads carry the instance name — the node's public host `APEX_NODE_HOST` — a status (`firing`/`resolved`), an escalation level, and the message truncated to 200 characters. Every method returns `True` only when the send succeeded; unattended actions treat a failed send of their final status message as a failure (exit `1`), so a broken notification channel is noticed instead of silently tolerated.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Reference: Engine API](/apex/reference/engine-api)
|
||||
- [Reference: Environment variables](/apex/reference/environment)
|
||||
- [Concept: Actions and the overlay](/apex/concepts/actions-and-overlay)
|
||||
@@ -0,0 +1,77 @@
|
||||
---
|
||||
title: "Node identity resolution"
|
||||
description: "How a node learns who it is: node.env identity keys and the FQDN fallback."
|
||||
weight: 12
|
||||
---
|
||||
|
||||
Every apex operation runs in the context of exactly one [node](/apex/glossary#node). The [engine](/apex/glossary#engine) derives that [node identity](/apex/glossary#node-identity) — node name, subnet, FQDN, and public host — from the `node.env` file at the repository root, with the OS hostname as a fallback for the FQDN only. The framework assumes no domain and no clustering: a node declares who it is explicitly. This resolution step succeeds the `root_require_node` mechanism of the bash "root" commons framework that apex replaces.
|
||||
|
||||
## Identity sources {#sources}
|
||||
|
||||
### node.env
|
||||
|
||||
`node.env` is a plain `KEY=value` file (shell-sourceable) located at the repository root. When parsed, comments and blank lines are ignored, and surrounding single or double quotes around values are stripped.
|
||||
|
||||
It declares identity explicitly through these keys:
|
||||
|
||||
- `APEX_NODE_FQDN` — the node's real FQDN, used for traefik `Host()` routing. If unset, the engine falls back to the OS hostname (and warns).
|
||||
- `APEX_NODE_HOST` — the node's public host name, used for the backup `--host` and the observability/notify `instance`. Defaults to `APEX_NODE_FQDN`.
|
||||
- `APEX_SUBNET` — the node's `/24` for the `direct` network. **Required**; there is no other source for it (see [Concept: Fleet addressing scheme](/apex/concepts/addressing)).
|
||||
|
||||
```ini {filename="node.env"}
|
||||
APEX_NODE_FQDN=node1.example.com
|
||||
APEX_NODE_HOST=node1.example.com
|
||||
APEX_SUBNET=198.18.16.0/24
|
||||
```
|
||||
|
||||
### The OS hostname
|
||||
|
||||
The OS hostname is consulted only as a fallback for `APEX_NODE_FQDN`. When `node.env` does not set `APEX_NODE_FQDN`, the engine uses the hostname and emits a warning. It is never parsed for a node name or any other coordinate.
|
||||
|
||||
## Resolution order {#resolution}
|
||||
|
||||
{{% steps %}}
|
||||
|
||||
### Resolve the FQDN
|
||||
|
||||
`fqdn` is `APEX_NODE_FQDN` from `node.env`, or the OS hostname when that key is unset. If neither yields a value, resolution fails (see [Failure modes](#failures)).
|
||||
|
||||
### Resolve the public host
|
||||
|
||||
`host` is `APEX_NODE_HOST` from `node.env`, or `fqdn` when that key is unset.
|
||||
|
||||
### Derive the short name
|
||||
|
||||
`name` is the **first label** of `host` (`host.split(".")[0]`) — for example, `node1` for `node1.example.com`. It is used for logs, the `sync/<node>` branch, and messages.
|
||||
|
||||
### Read the subnet
|
||||
|
||||
`subnet` is always read from `APEX_SUBNET` in `node.env`. It is required.
|
||||
|
||||
{{% /steps %}}
|
||||
|
||||
The result is a `Node` record with four fields: `name`, `subnet`, `fqdn`, and `host`.
|
||||
|
||||
## Warnings {#warnings}
|
||||
|
||||
Identity resolution emits exactly one warning; it signals a recoverable inconsistency that should be fixed.
|
||||
|
||||
| Condition | Warning message |
|
||||
|---|---|
|
||||
| `APEX_NODE_FQDN` is unset in `node.env` (the hostname is used instead) | `APEX_NODE_FQDN not set in node.env; using hostname '<fqdn>'.` |
|
||||
|
||||
## Failure modes {#failures}
|
||||
|
||||
Resolution refuses to guess. Two conditions are fatal, and both exit with code `66` — the engine's exit-code convention for missing input:
|
||||
|
||||
- **No FQDN at all.** `APEX_NODE_FQDN` is unset in `node.env` *and* the OS hostname is empty. The engine writes `identity: APEX_NODE_FQDN missing from node.env and no hostname` to stderr and exits.
|
||||
- **No subnet.** `APEX_SUBNET` is absent from `node.env` (including when the file itself is missing). The engine writes `identity: APEX_SUBNET missing from node.env` to stderr and exits.
|
||||
|
||||
> [!WARNING]
|
||||
> A valid `node.env` carrying `APEX_SUBNET` is required on every node. `APEX_NODE_FQDN` may be omitted only when the OS hostname supplies a usable FQDN.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [GitHub: apex.ermnvldmr.com](https://github.com/deytenit/apex.ermnvldmr.com)
|
||||
@@ -0,0 +1,53 @@
|
||||
---
|
||||
title: Security model
|
||||
description: Privilege isolation, firewall management, intrusion detection, and secret handling.
|
||||
weight: 18
|
||||
---
|
||||
|
||||
apex hardens a [node](/apex/glossary#node) in independent layers: filesystem privilege isolation, a managed host firewall, intrusion detection, container-level containment, and a strict secrets boundary. Each layer is applied by an [action](/apex/glossary#action) and is therefore reproducible and re-runnable.
|
||||
|
||||
## Privilege isolation {#privilege}
|
||||
|
||||
Every project runs as its own locked `noroot-<project>` system user; shared storage is group-writable with the sticky bit so projects can exchange files without being able to delete each other's. The full mechanism — users, groups, and the ownership matrix — is described in [Concept: Tiered storage](/apex/concepts/tiered-storage).
|
||||
|
||||
## Host firewall {#firewall}
|
||||
|
||||
`apex configure/ufw` applies the node's firewall from the `configs/ufw/` pillar in three parts:
|
||||
|
||||
- **Host rules** (`configs/ufw/host/host.rules`) — plain `ufw` commands, one per line, applied in order.
|
||||
- **Kernel-level rule blocks** (`configs/ufw/host/before.rules`, `after.rules`) — injected into `/etc/ufw/before.rules` and `/etc/ufw/after.rules` as managed marker blocks. Injection is **self-healing**: every previous managed block — including blocks left by predecessor tooling — is removed before the fresh block is inserted after ufw's `# End required lines` anchor, and a timestamped `.bak` of the target is kept beside it.
|
||||
- **Container rules** (`configs/ufw/docker/*.rules`) — applied through `ufw-docker`, which translates them into rules that survive Docker's own iptables manipulation. `configure/base` installs the helper to `~/.local/bin/ufw-docker`.
|
||||
|
||||
Missing configuration is an error, not a silent skip: an absent `docker/` or `host/` directory exits `66` when its counterpart exists, and a missing `/etc/ufw` target file aborts the run. The action finishes with a reminder that `ufw` must be restarted to take effect — and that the SSH port must be allowed first.
|
||||
|
||||
## Intrusion detection {#crowdsec}
|
||||
|
||||
`apex configure/crowdsec` installs CrowdSec through its official installer when `cscli` is absent, renders everything under `configs/crowdsec/` into `/etc/crowdsec/` (acquisition sources, parsers, bouncer configuration), and installs the `crowdsecurity/linux` and `crowdsecurity/traefik` collections. When the node's configs include a firewall-bouncer configuration, the matching bouncer package is installed as well, closing the loop from detection to blocking.
|
||||
|
||||
## Container containment {#containers}
|
||||
|
||||
The [core composition](/apex/glossary#core-composition) encodes container-level hardening:
|
||||
|
||||
- The edge proxy never touches the Docker socket: it discovers backends through a **filtered socket proxy** that exposes read-only container/service state on an isolated network.
|
||||
- `enclave` and `socket` are `internal: true` networks — no egress, no host port exposure.
|
||||
- Core services set `no-new-privileges` and run under the project's noroot identity wherever the workload allows; capabilities are granted per service, not globally.
|
||||
|
||||
## Observability {#observability}
|
||||
|
||||
Observability is an **outbound push**, not an exposed route. The optional `alloy` core service (Grafana Alloy, one agent per node) collects host metrics (a `node_exporter`-equivalent), container metrics (cadvisor), the systemd journal, and docker container logs, and pushes them to a **central** Prometheus and Loki that live on a designated node — the metrics via `prometheus.remote_write` to `APEX_PROM_PUSH_URL`, the logs via `loki.write` to `APEX_LOKI_PUSH_URL`. Both targets are authenticated with HTTP basic auth using `APEX_OBS_BASIC_AUTH_USER` / `APEX_OBS_BASIC_AUTH_PASS`. Because the push is outbound, Alloy attaches to the `direct` network (which has egress) rather than the `internal` `enclave`; there is no node-local log store and no `/loki` route to protect.
|
||||
|
||||
## Secrets boundary {#secrets}
|
||||
|
||||
- Secrets live only in per-project `.env` files, which are **symlinks into tier-1 storage** — outside the repository working tree and covered by `.gitignore` (`**/.env`).
|
||||
- Each node repository documents its expected keys in `SECRETS.md`; the file documents names, never values.
|
||||
- Committed configuration (`apex.env`) is restricted to non-secret scalars.
|
||||
- The shared composition marks critical secrets with `?error`, so a missing value fails fast at compose-config time.
|
||||
- [Capture-up](/apex/glossary#capture-up) commits are safe by construction: the ignored `.env` links and tier data never enter the git index.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Concept: Tiered storage and privilege isolation](/apex/concepts/tiered-storage)
|
||||
- [Concept: Compositions and the shared core](/apex/concepts/compositions)
|
||||
- [Reference: Actions](/apex/reference/actions)
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
title: Tiered storage and privilege isolation
|
||||
description: The @tier link structure, shared storage, and per-project noroot users.
|
||||
weight: 16
|
||||
---
|
||||
|
||||
Service data never lives inside a [node](/apex/glossary#node) repository. It lives in three storage [tiers](/apex/glossary#tier) — separate locations chosen per host — and reaches [compositions](/apex/glossary#compositions) through a standardized symlink structure. Ownership of that data is delegated to per-project system users, so no service container writes as root.
|
||||
|
||||
## The tier link structure {#links}
|
||||
|
||||
`apex tiers/link <tier1> <tier2> <tier3>` builds two layers of links:
|
||||
|
||||
- **Node-level:** `compositions/@tierN` → the tier storage root. These are what the engine exposes as `APEX_TIER_ROOT1..3`.
|
||||
- **Per-project:** for every `compositions/<project>/`, the action creates `<tier-root>/<project>/` and links `compositions/<project>/@tierN` → `<tier-root>/<project>`. Inside each tier it also maintains `shared/` with a per-project subdirectory, and a relative `shared` link inside every project's tier directory pointing back at it.
|
||||
|
||||
Each project additionally gets `compositions/<project>/.env` → `<tier1-root>/<project>/.env` (created empty if missing) — the uncommitted secrets file described in [Concept: Compositions](/apex/concepts/compositions). The `@tierN` links and `.env` files are ignored by git; the repository stays pure source.
|
||||
|
||||
If a path that should be a symlink exists as a real file or directory, the action warns and asks for confirmation before removing it; declining (including non-interactive EOF) skips that link and continues.
|
||||
|
||||
## Users and groups {#users}
|
||||
|
||||
`apex tiers/useradd` provisions one locked system user per project — `noroot-<project>` (name sanitized, its own primary group) — plus the fleet-wide `noroot-shared` group every project user joins. It then injects the resolved IDs into the project's `.env` between marker comments, replacing any previous block on re-runs:
|
||||
|
||||
```ini {filename="compositions/<project>/.env"}
|
||||
# START AUTO GENERATED APEX-NOROOT-USERS
|
||||
APEX_UID=998
|
||||
APEX_GID=998
|
||||
APEX_SHARED_GID=997
|
||||
# END AUTO GENERATED APEX-NOROOT-USERS
|
||||
```
|
||||
|
||||
Compositions consume these as `user: "${APEX_UID?error}:${APEX_GID?error}"` and `group_add: ["${APEX_SHARED_GID?error}"]`.
|
||||
|
||||
## The ownership matrix {#ownership}
|
||||
|
||||
`apex tiers/chown` enforces ownership across all three tiers. Without arguments it operates on the tier roots (`APEX_TIER_ROOT1..3`); explicit tier paths can override.
|
||||
|
||||
| Path | Ownership | Notes |
|
||||
|---|---|---|
|
||||
| `<tier-root>` | `root:root` | Directory inode only. |
|
||||
| `<tier-root>/shared` | `root:root` | Directory inode only — never recursive, so existing project data inside is untouched. |
|
||||
| `<tier-root>/node-infra` | `root:root` | Recursive. |
|
||||
| `<tier-root>/<project>` | `uid:gid` | Recursive; IDs read from the project `.env`. |
|
||||
| `<tier-root>/shared/<project>` | `uid:shared_gid` | Recursive, plus the sticky bit — collaborators can add files but only delete their own. |
|
||||
|
||||
Projects with an incomplete `APEX_UID`/`APEX_GID`/`APEX_SHARED_GID` set are skipped with a warning pointing at `tiers/useradd`.
|
||||
|
||||
## Engine tier variables {#variables}
|
||||
|
||||
| Variable | Resolves to |
|
||||
|---|---|
|
||||
| `APEX_TIER_ROOT1..3` | The tier storage roots (`compositions/@tierN`, fully resolved). |
|
||||
| `APEX_TIER1..3` | The core project's data directories inside each tier (`compositions/apex/@tierN`). |
|
||||
| `APEX_TIER1..3_SHARED` | The shared area inside the core project's tiers. |
|
||||
|
||||
The core composition mounts through these exclusively; the backup runner instead mounts the fleet data root `/srv` read-only and resolves each labeled service's container mounts back to their host paths.
|
||||
|
||||
## Backups {#backups}
|
||||
|
||||
`apex backup/run <telegram-bot-url>` drives the core `resticontainer` service (profile `manual`, one-shot) end to end. Backup intent is **label-driven**: resticontainer discovers every service carrying `restic.enable=true`, reads its `restic.backup.paths` (and optional `restic.backup.stop` / `restic.hooks.*`), resolves those container paths to host paths, and snapshots their union in a single restic run.
|
||||
|
||||
1. Checks repository reachability with `snapshots`; initializes the repository if needed.
|
||||
2. Runs `backup` with `--host <APEX_NODE_HOST>` and `--tag biweekly` — resticontainer runs each labeled service's pre-hook, stops any container flagged `restic.backup.stop`, snapshots the resolved host paths, then restarts and runs post-hooks. The run refuses to report success if no new snapshot lands (a guard against zero labels being discovered).
|
||||
3. Applies retention with `forget --prune` (7 daily, 4 weekly, 12 monthly).
|
||||
4. Sends a Telegram status either way; a success whose notification cannot be delivered still exits `1` so schedulers notice.
|
||||
|
||||
The repository location and credentials come entirely from `APEX_RESTIC_*` configuration (see [Reference: Environment variables](/apex/reference/environment)). Which data is captured is decided per service by its `restic.*` labels — for example the core services back up only their configuration (traefik dynamic rules, xray/adguard config), leaving reproducible tier-3 data such as caches and logs out of the set.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Concept: Security model](/apex/concepts/security-model)
|
||||
- [Reference: Actions](/apex/reference/actions)
|
||||
- [Guide: Add a service composition](/apex/guides/service-composition)
|
||||
@@ -0,0 +1,102 @@
|
||||
---
|
||||
title: Getting started with an apex node
|
||||
description: Bootstrap a node repository and bring a host under apex management.
|
||||
weight: 5
|
||||
---
|
||||
|
||||
This quick start takes a fresh host from a cloned [node](/apex/glossary#node) repository to a running service stack. It assumes the node repository already exists — to build one from scratch, see [Guide: Build a node repository](/apex/guides/node-repository).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- A Debian-based host. Any FQDN works — [node identity](/apex/glossary#node-identity) is declared explicitly in `node.env` (`APEX_NODE_FQDN`, `APEX_NODE_HOST`, `APEX_SUBNET`), not parsed from a required hostname shape.
|
||||
- `git` and Python 3 — the [engine](/apex/glossary#engine) uses the standard library only; `pip` is never required.
|
||||
- Docker with Compose v2.20 or newer (the [core composition](/apex/glossary#core-composition) is consumed through the `include:` directive).
|
||||
- A sudo-capable login user.
|
||||
- Three storage locations for the [tiers](/apex/glossary#tier) (for example, three mount points).
|
||||
|
||||
## Bootstrap
|
||||
|
||||
{{% steps %}}
|
||||
|
||||
### Clone the node repository
|
||||
|
||||
```bash
|
||||
git clone --recurse-submodules git@github.com:<owner>/<node-repo>.git
|
||||
cd <node-repo>
|
||||
./init.sh
|
||||
source ~/.bashrc
|
||||
```
|
||||
|
||||
`init.sh` initializes the `commons/` submodule, points `core.hooksPath` at `commons/githooks`, marks the checkout `safe.directory`, and prepends `commons/` to `PATH` — the launcher is `commons/apex`.
|
||||
|
||||
### Verify action discovery
|
||||
|
||||
```bash
|
||||
apex --help
|
||||
```
|
||||
|
||||
The listing shows every [commons](/apex/glossary#commons) action plus any node-local [proprietaries](/apex/glossary#proprietaries) tagged `(local)`. When `node.env` has no `APEX_NODE_FQDN`, the engine warns and falls back to the OS hostname — harmless on a workstation, but set `APEX_NODE_FQDN` explicitly on the node itself. (`APEX_SUBNET` is required: if it is missing the engine exits `66`.)
|
||||
|
||||
### Link the storage tiers
|
||||
|
||||
```bash
|
||||
apex tiers/link <tier1-path> <tier2-path> <tier3-path>
|
||||
```
|
||||
|
||||
Creates the node-level `compositions/@tierN` links, the per-project tier directories and `@tierN` links, `shared/` areas, and each project's `.env` symlink into tier 1. Success ends with `@tier setup completed.`
|
||||
|
||||
### Provision project users
|
||||
|
||||
```bash
|
||||
apex tiers/useradd
|
||||
apex tiers/chown
|
||||
```
|
||||
|
||||
`tiers/useradd` creates one `noroot-<project>` system user per composition plus the `noroot-shared` group, and injects `APEX_UID`/`APEX_GID`/`APEX_SHARED_GID` into each project `.env`. `tiers/chown` then applies the ownership matrix (see [Concept: Tiered storage](/apex/concepts/tiered-storage)) and ends with `Permissions applied.`
|
||||
|
||||
### Fill the core secrets
|
||||
|
||||
Edit `compositions/apex/.env` (a symlink into tier-1 storage, never committed) with the keys documented in the node repository's `SECRETS.md`:
|
||||
|
||||
```ini {filename="compositions/apex/.env"}
|
||||
APEX_RESTIC_PASSWORD=...
|
||||
APEX_RESTIC_AWS_ACCESS_KEY_ID=...
|
||||
APEX_RESTIC_AWS_SECRET_ACCESS_KEY=...
|
||||
APEX_TRAEFIK_CF_DNS_API_TOKEN=...
|
||||
APEX_TRAEFIK_ACME_EMAIL=...
|
||||
APEX_OBS_BASIC_AUTH_PASS=... # only when the alloy observability profile is enabled
|
||||
```
|
||||
|
||||
`APEX_UID`/`APEX_GID`/`APEX_SHARED_GID` are already present — `tiers/useradd` maintains them between marker comments. Non-secret scalars (`APEX_RESTIC_REPOSITORY`, the push URLs, `COMPOSE_PROFILES`, `TZ`, …) live in the committed `compositions/apex/apex.env` instead — see [Guide: Build a node repository](/apex/guides/node-repository).
|
||||
|
||||
### Configure the host
|
||||
|
||||
```bash
|
||||
apex configure
|
||||
```
|
||||
|
||||
Runs every `configure/*` action in fixed order: `base`, `ufw`, `crowdsec`, `cron`, `systemd`, `routing`. Each step is also individually invocable; several accept `--dry-run`. See [Guide: Operate a node](/apex/guides/host-operations).
|
||||
|
||||
### Bring the stack up
|
||||
|
||||
```bash
|
||||
apex compose up
|
||||
```
|
||||
|
||||
The core composition comes up first (it owns the networks), then every service composition in alphabetical order.
|
||||
|
||||
{{% /steps %}}
|
||||
|
||||
## Validation
|
||||
|
||||
- `apex --help` lists actions without warnings on the live host.
|
||||
- `docker ps` shows the core containers (`apex-traefik`, `apex-socket-proxy`, plus any profile-enabled services) and each project's anchor.
|
||||
- `apex utils/lint-docker-compose` reports `Anchor IP validation passed.`
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Guide: Build a node repository](/apex/guides/node-repository)
|
||||
- [Concept: Framework architecture](/apex/concepts/architecture)
|
||||
- [GitHub: apex.ermnvldmr.com](https://github.com/deytenit/apex.ermnvldmr.com)
|
||||
@@ -0,0 +1,69 @@
|
||||
---
|
||||
title: Glossary
|
||||
description: Terminology used within the apex fleet operations framework.
|
||||
weight: 100
|
||||
---
|
||||
|
||||
Terminology used within the apex fleet operations framework.
|
||||
|
||||
## Action {#action}
|
||||
|
||||
**Action** - An atomic automation task: one Python module exposing `METADATA = Meta(...)` and `run(ctx, args)`, dispatched by its file path as `apex <group>/<action>`.
|
||||
|
||||
## Anchor IP {#anchor-ip}
|
||||
|
||||
**Anchor IP** - The static `ipv4_address` a project's main container holds on the `direct` network. Anchors make service addresses predictable and are validated by the compose lint.
|
||||
|
||||
## Capture-up {#capture-up}
|
||||
|
||||
**Capture-up** - The flow that records a host's live state back into its repository: a scheduled commit to the `sync/<node>` branch, rebased onto `main` and folded back by CI. Drift becomes a diff instead of divergence.
|
||||
|
||||
## Commons {#commons}
|
||||
|
||||
**Commons** - The shared repository (`apex.ermnvldmr.com`) holding the engine, the shared actions, and the core composition. Every node consumes it as the pinned `commons/` git submodule.
|
||||
|
||||
## Compositions {#compositions}
|
||||
|
||||
**Compositions** - The node-repository pillar declaring the node's runtime: one docker-compose project per service plus the core wrapper and the addressing ledger.
|
||||
|
||||
## Configs {#configs}
|
||||
|
||||
**Configs** - The node-repository pillar holding host configuration sources (`ufw/`, `cron/`, `systemd/`, `crowdsec/`), rendered through the template layer by the `configure/*` actions.
|
||||
|
||||
## Core composition {#core-composition}
|
||||
|
||||
**Core composition** - The shared, `${APEX_*}`-parameterized docker-compose project defining the fleet's common services (edge proxy, socket proxy, and the opt-in profile services). Defined once in the commons, included per node.
|
||||
|
||||
## Enclave {#enclave}
|
||||
|
||||
**Enclave** - The fleet's private address space, `198.18.0.0/16`, and by extension the internal reverse-proxy backend network named `enclave` in the core composition.
|
||||
|
||||
## Engine {#engine}
|
||||
|
||||
**Engine** - The stdlib-Python runtime under `commons/engine/`: the launcher, identity resolution, the overlay, the execution context, and the `ctx.*` helper library.
|
||||
|
||||
## Node {#node}
|
||||
|
||||
**Node** - A host managed by the framework, defined by exactly one node repository and identified by the FQDN it declares in `node.env` (for example, `node1.example.com`).
|
||||
|
||||
## Node identity {#node-identity}
|
||||
|
||||
**Node identity** - The resolved record of who a node is — name, subnet, FQDN, public host — derived from `node.env`, with the OS hostname as a fallback for the FQDN.
|
||||
|
||||
## Overlay {#overlay}
|
||||
|
||||
**Overlay** - The resolution rule that lets a node's proprietaries add, override, wrap, or disable commons actions: `proprietaries/actions/` is consulted before `commons/actions/`.
|
||||
|
||||
## Proprietaries {#proprietaries}
|
||||
|
||||
**Proprietaries** - The node-repository pillar for node-local code: actions, libraries, and tool sources that apply to exactly one node and therefore do not belong in the commons.
|
||||
|
||||
## Tier {#tier}
|
||||
|
||||
**Tier** - One of the three standardized storage classes a node mounts (configuration, valuable data, reproducible data), reached from compositions through the `@tierN` symlink structure.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [GitHub: apex.ermnvldmr.com](https://github.com/deytenit/apex.ermnvldmr.com)
|
||||
@@ -0,0 +1,24 @@
|
||||
---
|
||||
title: Guides
|
||||
description: Task-oriented guides for building node repositories, actions, and compositions.
|
||||
weight: 20
|
||||
---
|
||||
|
||||
Task-oriented guides for day-to-day work with apex. Each guide solves one recurring problem: assembling a [node](/apex/glossary#node) repository, developing [actions](/apex/glossary#action) for the [engine](/apex/glossary#engine), defining service [compositions](/apex/glossary#compositions), and operating the underlying host.
|
||||
|
||||
## Available guides {#available-guides}
|
||||
|
||||
{{< cards >}}
|
||||
{{< card link="/apex/guides/node-repository" title="Node repository" subtitle="Assemble the per-node repository that declares a node's identity, configs, and compositions." >}}
|
||||
{{< card link="/apex/guides/action-development" title="Action development" subtitle="Write actions that the engine dispatches on a node." >}}
|
||||
{{< card link="/apex/guides/service-composition" title="Service composition" subtitle="Define project compositions and deploy them alongside the core composition." >}}
|
||||
{{< card link="/apex/guides/host-operations" title="Host operations" subtitle="Perform routine operations against a live host through the engine." >}}
|
||||
{{< /cards >}}
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Docs: Getting started](/apex/getting-started)
|
||||
- [Docs: Concepts](/apex/concepts)
|
||||
- [GitHub: apex.ermnvldmr.com](https://github.com/deytenit/apex.ermnvldmr.com)
|
||||
@@ -0,0 +1,90 @@
|
||||
---
|
||||
title: Develop actions
|
||||
description: Write, override, and test engine actions in commons or proprietaries.
|
||||
weight: 22
|
||||
---
|
||||
|
||||
**Context:** all automation ships as [actions](/apex/glossary#action) — one Python module each, dispatched by path. Shared behavior belongs in [commons](/apex/glossary#commons) `actions/`; anything node-specific belongs in that node's `proprietaries/actions/`. Both use the identical contract, and the launcher puts both trees on `sys.path`, so a proprietary action imports `engine.*` exactly like a commons one.
|
||||
|
||||
## A minimal action {#minimal}
|
||||
|
||||
```python {filename="proprietaries/actions/utils/example.py"}
|
||||
"""One-line module docstring describing the action."""
|
||||
import os
|
||||
from engine.descriptor import Meta, Arg, Flag
|
||||
|
||||
METADATA = Meta(
|
||||
summary="Copy a rendered config into place.",
|
||||
args=[Arg("target_dir", "Destination directory"),
|
||||
Flag("--dry-run", "Preview only.")],
|
||||
)
|
||||
|
||||
def run(ctx, args):
|
||||
if not os.path.isdir(args.target_dir):
|
||||
ctx.log.error(f"No such directory: {args.target_dir}")
|
||||
raise SystemExit(66)
|
||||
content = ctx.tpl.render("example/config", ctx.vars())
|
||||
ctx.host.write_file(os.path.join(args.target_dir, "config"), content,
|
||||
dry_run=args.dry_run, sudo=True)
|
||||
ctx.log.success("Config deployed.")
|
||||
```
|
||||
|
||||
The file's path is its name: the module above dispatches as `apex utils/example`. Return `0`/`None` for success, an integer for a specific exit code, or raise `SystemExit(code)`; the [exit-code convention](/apex/concepts/actions-and-overlay#exit-codes) is `64` usage, `65` data error, `66` missing input, `1` general.
|
||||
|
||||
## Conventions for unattended actions {#unattended}
|
||||
|
||||
Actions that run from cron follow two rules so failures page instead of vanishing:
|
||||
|
||||
- **Wrap the body** in `try/except Exception` and send `ctx.notify.error(...)` before re-raising as `SystemExit(1)` — the Python equivalent of a shell ERR trap.
|
||||
- **Gate the exit code on the final notification**: `ctx.notify.*` returns `False` on delivery failure; a completed run whose status message never arrived should still exit `1`.
|
||||
|
||||
```python
|
||||
try:
|
||||
...work...
|
||||
log.success("Done.")
|
||||
if not ctx.notify.success(TITLE, url, "Done.", node):
|
||||
raise SystemExit(1)
|
||||
except SystemExit:
|
||||
raise
|
||||
except Exception as e:
|
||||
log.error(f"{type(e).__name__}: {e}")
|
||||
ctx.notify.error(TITLE, url, f"Critical error: {e}", node)
|
||||
raise SystemExit(1)
|
||||
```
|
||||
|
||||
## Override, wrap, disable {#override}
|
||||
|
||||
A proprietary file at the same relative path as a commons action shadows it completely. Inside the shadowing module, `ctx.commons.run(args)` invokes the shadowed implementation — the wrap pattern:
|
||||
|
||||
```python {filename="proprietaries/actions/sync/packages.py"}
|
||||
from engine.descriptor import Meta, Arg
|
||||
|
||||
METADATA = Meta(summary="sync/packages with a node-local pre-step.",
|
||||
args=[Arg("telegram_bot_url", "Telegram Bot URL")])
|
||||
|
||||
def run(ctx, args):
|
||||
ctx.log.info("Node-local preparation...")
|
||||
return ctx.commons.run(args) # delegate to the commons implementation
|
||||
```
|
||||
|
||||
To retire an action on one node, assign `DISABLED = "reason"` at module top level: dispatch refuses it (exit `1`) and the listing shows `[DISABLED: reason]`.
|
||||
|
||||
## Testing {#testing}
|
||||
|
||||
The commons test suite is stdlib `unittest`:
|
||||
|
||||
```bash
|
||||
./tests/run.sh # python3 -m unittest discover -s tests -p 'test_*.py' -v
|
||||
```
|
||||
|
||||
Engine changes must keep the suite green. A cheap smoke gate for any action — commons or proprietary — is its parser: `apex <name> --help` must exit `0`, which proves the module parses, its `METADATA` is well-formed, and discovery sees it.
|
||||
|
||||
The commons pre-commit hook (wired by `init.sh` via `core.hooksPath`) runs `apex utils/lint-docker-compose --hook` against staged compose files, so composition changes are linted at commit time.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Concept: Actions and the overlay](/apex/concepts/actions-and-overlay)
|
||||
- [Concept: The execution context](/apex/concepts/execution-context)
|
||||
- [Reference: Engine API](/apex/reference/engine-api)
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
title: Operate a node
|
||||
description: "Day-2 operations: configure, compose, backups, and repository sync."
|
||||
weight: 24
|
||||
---
|
||||
|
||||
**Context:** once bootstrapped, a [node](/apex/glossary#node) is operated entirely through the [engine](/apex/glossary#engine). This guide covers the recurring operations and what each actually does on the host.
|
||||
|
||||
## Host configuration {#configure}
|
||||
|
||||
`apex configure` runs the full configuration suite in a fixed order; each step is individually invocable when only one layer changed:
|
||||
|
||||
| Step | Effect |
|
||||
|---|---|
|
||||
| `configure/base` | Ensures a Debian base: package index update, essential packages (`curl`, `wget`, `git`, `rsyslog`), rsyslog running, `ufw` installed, and the `ufw-docker` helper at `~/.local/bin/ufw-docker`. |
|
||||
| `configure/ufw` | Applies host rules, injects the managed kernel-rule blocks, applies container rules via `ufw-docker`. Supports `--dry-run`. |
|
||||
| `configure/crowdsec` | Installs CrowdSec if absent, deploys `configs/crowdsec/` to `/etc/crowdsec/`, installs collections, and the firewall bouncer when configured. Supports `--dry-run`. |
|
||||
| `configure/cron` | Renders `configs/cron/crontab` through the template layer and installs it — with preview, empty-render refusal (exit `65`), and rollback on a failed install. Supports `--dry-run`. |
|
||||
| `configure/systemd` | Renders the top-level files of `configs/systemd/` into `/etc/systemd/system/`, then daemon-reloads and enables each `.service`. Supports `--dry-run`. |
|
||||
| `configure/routing` | Enables IP forwarding and installs the fwmark policy-routing rule and table used by the enclave edge; idempotent. |
|
||||
|
||||
After `configure/ufw`, restart `ufw` deliberately (`sudo systemctl restart ufw`) — the action reminds you, and warns that the SSH port must already be allowed.
|
||||
|
||||
## Composition lifecycle {#compose}
|
||||
|
||||
```bash
|
||||
apex compose up # core first, then services alphabetically
|
||||
apex compose down # reverse order
|
||||
apex compose restart
|
||||
apex compose up --dry-run # print the commands only
|
||||
```
|
||||
|
||||
Extra arguments pass through to `docker compose` per project. Each project's `.env` and `apex.env` are supplied automatically when present.
|
||||
|
||||
## Backups {#backups}
|
||||
|
||||
```bash
|
||||
apex backup/run "$(cat <path-to-telegram-url-file>)"
|
||||
```
|
||||
|
||||
Runs [resticontainer](https://github.com/deytenit/resticontainer) one-shot under the `manual` compose profile: init-if-needed, then a label-driven backup that discovers each service's intent from its `restic.*` compose labels (`restic.enable`, `restic.backup.paths`, …), snapshots the union of the resolved host paths in a single restic run (`--host` = `APEX_NODE_HOST`, `--compression max`), applies retention (`forget --prune`), and notifies. It refuses to report success if no `restic.*`-labelled service produced a new snapshot. The full flow is described in [Concept: Tiered storage](/apex/concepts/tiered-storage#backups). Schedule it from `configs/cron/crontab` so the run and its Telegram outcome are unattended.
|
||||
|
||||
## Repository sync (capture-up) {#sync}
|
||||
|
||||
```bash
|
||||
apex sync/repository <telegram-bot-url>
|
||||
```
|
||||
|
||||
Commits the live checkout to `sync/<node>`, rebases onto `origin/main`, and force-pushes with lease; CI folds the branch back into `main`. The action refuses to run — with a paged error — when the commons submodule is dirty or the `compositions/` tree is missing, and aborts cleanly on a rebase conflict (manual intervention required). See [Concept: Framework architecture](/apex/concepts/architecture#capture-up) for the model.
|
||||
|
||||
## Update awareness {#updates}
|
||||
|
||||
```bash
|
||||
apex sync/packages <telegram-bot-url>
|
||||
```
|
||||
|
||||
Reports pending apt upgrades and out-of-date container images (local digests compared against the registry via `skopeo`) in a single Telegram digest. It changes nothing on the host — it is a scheduled awareness check.
|
||||
|
||||
## Certificate extraction {#certs}
|
||||
|
||||
```bash
|
||||
apex utils/extract-traefik-certs <acme.json> <dest-dir> [telegram-bot-url]
|
||||
```
|
||||
|
||||
Dumps the edge proxy's ACME store into per-domain `fullchain.pem`/`privkey.pem` pairs (via a containerized dumper), restricts key permissions to `600`, and re-owns the output to the invoking user — for services that need raw PEM material instead of proxy-terminated TLS.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Getting started with an apex node](/apex/getting-started)
|
||||
- [Reference: Actions](/apex/reference/actions)
|
||||
- [Docs: Troubleshooting](/apex/troubleshooting)
|
||||
@@ -0,0 +1,105 @@
|
||||
---
|
||||
title: Build a node repository
|
||||
description: Assemble the four pillars, pin commons, and wire the capture-up flow.
|
||||
weight: 21
|
||||
---
|
||||
|
||||
**Context:** every [node](/apex/glossary#node) gets exactly one repository. Pick any repository name — [identity](/apex/glossary#node-identity) is declared explicitly in `node.env`, so the repo name is a convention, not a functional input. This guide assembles one from nothing.
|
||||
|
||||
{{% steps %}}
|
||||
|
||||
### Initialize the repository and pin commons
|
||||
|
||||
```bash
|
||||
git init <node-repo> && cd <node-repo>
|
||||
git submodule add <commons-url> commons
|
||||
( cd commons && git fetch --tags && git checkout <release-tag> )
|
||||
git add .gitmodules commons
|
||||
mkdir -p proprietaries/actions proprietaries/lib configs compositions/apex .github/workflows
|
||||
```
|
||||
|
||||
The gitlink pins the exact engine version the node runs. Upgrading later means checking out a newer tag inside `commons/` and committing the moved pointer.
|
||||
|
||||
### Declare identity
|
||||
|
||||
```ini {filename="node.env"}
|
||||
APEX_NODE_FQDN=node1.example.com
|
||||
APEX_NODE_HOST=node1.example.com
|
||||
APEX_SUBNET=198.18.16.0/24
|
||||
APEX_GIT_AUTHOR_NAME=apex [bot]
|
||||
APEX_GIT_AUTHOR_EMAIL=apex@localhost
|
||||
```
|
||||
|
||||
`node.env` declares the node's identity explicitly — no domain shape is assumed. `APEX_NODE_FQDN` is the real FQDN used for traefik `Host()` routing (fallback: the OS hostname, with a warning). `APEX_NODE_HOST` is the public host name used for the backup `--host` and the observability/notify `instance` label (default: `APEX_NODE_FQDN`); the short node `name` is its first label. `APEX_SUBNET` is the node's /24 for the `direct` network and is **required** — a missing value exits `66` (see [Concept: Fleet addressing scheme](/apex/concepts/addressing)). `APEX_GIT_AUTHOR_NAME`/`APEX_GIT_AUTHOR_EMAIL` are the capture-up bot identity (generic defaults if unset).
|
||||
|
||||
### Write the core wrapper and its env files
|
||||
|
||||
```yaml {filename="compositions/apex/docker-compose.yml"}
|
||||
name: apex
|
||||
include:
|
||||
- ../../commons/compositions/apex/docker-compose.yml
|
||||
```
|
||||
|
||||
```ini {filename="compositions/apex/apex.env"}
|
||||
APEX_TRAEFIK_DIRECT_IP=198.18.16.10
|
||||
APEX_RESTIC_REPOSITORY=s3:<endpoint>/<bucket>/<node>
|
||||
COMPOSE_PROFILES=adguard,xray,alloy
|
||||
# observability push targets (only when the alloy profile is enabled)
|
||||
APEX_PROM_PUSH_URL=https://<central-host>/api/v1/push
|
||||
APEX_LOKI_PUSH_URL=https://<central-host>/loki/api/v1/push
|
||||
APEX_OBS_BASIC_AUTH_USER=<node-user>
|
||||
# optional scalars with compose defaults:
|
||||
# TZ=Etc/UTC
|
||||
# APEX_RESTIC_COMPRESSION=max
|
||||
# APEX_RESTIC_AWS_REGION=us-east-1 # set for any non-AWS S3 store
|
||||
# APEX_RESTIC_AWS_ENDPOINT= # e.g. an S3-compatible endpoint URL
|
||||
```
|
||||
|
||||
`apex.env` holds committed non-secret scalars only. `COMPOSE_PROFILES` selects which optional core services this node runs — `adguard`, `xray`, `alloy` (the observability push agent), and `manual` (one-shot backups) — and may be empty. Document the uncommitted secret keys in `compositions/apex/SECRETS.md` so an operator on a fresh host knows what to fill in `.env` (`APEX_RESTIC_PASSWORD`, the S3 access keys, `APEX_TRAEFIK_CF_DNS_API_TOKEN`, `APEX_TRAEFIK_ACME_EMAIL`, and `APEX_OBS_BASIC_AUTH_PASS` when observability is on).
|
||||
|
||||
### Ignore what never belongs in git
|
||||
|
||||
```gitignore {filename=".gitignore"}
|
||||
**/@tier1
|
||||
**/@tier2
|
||||
**/@tier3
|
||||
**/.env
|
||||
!compositions/apex/apex.env
|
||||
__pycache__/
|
||||
*.pyc
|
||||
```
|
||||
|
||||
### Add the bootstrap script
|
||||
|
||||
`init.sh` prepares a fresh clone on the host: `git submodule update --init --recursive`, `git config core.hooksPath commons/githooks`, `git config --global --add safe.directory <repo>`, and prepend `<repo>/commons` to `PATH` in the login shell so `apex` resolves. Mark it executable.
|
||||
|
||||
### Start the addressing ledger
|
||||
|
||||
Create `compositions/ADDRESSING.md` with the node's subnet (`APEX_SUBNET`) and the reserved ranges (`.10`–`.19` core, `.20`+ services). Every [anchor IP](/apex/glossary#anchor-ip) allocated later is recorded here.
|
||||
|
||||
### Wire capture-up CI
|
||||
|
||||
Add a scheduled workflow that merges `sync/<node>` into `main` when the branch exists, committing under a bot identity. Check out **without** submodules — the merge only moves the commons pointer, and an SSH submodule URL is not fetchable with the default CI token. See [Concept: Framework architecture](/apex/concepts/architecture) for the full [capture-up](/apex/glossary#capture-up) loop.
|
||||
|
||||
### Validate
|
||||
|
||||
```bash
|
||||
./commons/apex --help
|
||||
```
|
||||
|
||||
The listing must show every commons action (plus your proprietaries, once added). Off-host — before `node.env` is filled — expect the harmless warning that `APEX_NODE_FQDN` is unset (the engine falls back to the OS hostname); `APEX_SUBNET` must still be set or the engine exits `66`. Commit the assembled repository on `main`.
|
||||
|
||||
{{% /steps %}}
|
||||
|
||||
## Notes
|
||||
|
||||
- The `proprietaries/` pillar may start empty (keep it present with `.gitkeep` files); the engine treats a missing overlay gracefully.
|
||||
- Migrate host config sources into `configs/` (`ufw/`, `cron/crontab`, `systemd/`, `crowdsec/`) as the node adopts each `configure/*` step — the actions consume `configs/` verbatim through the template layer.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Getting started with an apex node](/apex/getting-started)
|
||||
- [Concept: Framework architecture](/apex/concepts/architecture)
|
||||
- [Guide: Add a service composition](/apex/guides/service-composition)
|
||||
@@ -0,0 +1,101 @@
|
||||
---
|
||||
title: Add a service composition
|
||||
description: "Wire a new project into a node: anchor IP, networks, tiers, and lint."
|
||||
weight: 23
|
||||
---
|
||||
|
||||
**Context:** a project joins a [node](/apex/glossary#node) as one directory under `compositions/` containing a hand-authored compose file. The framework's contract for that file is small: a static [anchor IP](/apex/glossary#anchor-ip) on the `direct` network, external network references by key, and tier-relative data paths.
|
||||
|
||||
{{% steps %}}
|
||||
|
||||
### Allocate the anchor address
|
||||
|
||||
Pick the next free address at `.20` or above in the node's /24 and record it in `compositions/ADDRESSING.md`. Addresses `.10`–`.19` are reserved for the [core composition](/apex/glossary#core-composition) (see [Concept: Fleet addressing scheme](/apex/concepts/addressing)).
|
||||
|
||||
### Author the compose file
|
||||
|
||||
```yaml {filename="compositions/<project>/docker-compose.yml"}
|
||||
name: <project>
|
||||
services:
|
||||
<anchor>:
|
||||
image: ...
|
||||
container_name: <project>-<anchor>
|
||||
user: "${APEX_UID?error}:${APEX_GID?error}"
|
||||
group_add:
|
||||
- "${APEX_SHARED_GID?error}"
|
||||
volumes:
|
||||
- ./@tier1/<anchor>/config:/config
|
||||
- ./@tier2/<anchor>/data:/data
|
||||
networks:
|
||||
direct:
|
||||
ipv4_address: 198.18.16.20
|
||||
enclave: {}
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.<project>.rule=Host(`...`)"
|
||||
- "traefik.http.services.<project>.loadbalancer.server.port=8080"
|
||||
# backup intent (resticontainer, discovered from restic.* labels)
|
||||
- "restic.enable=true"
|
||||
- "restic.backup.paths=/config"
|
||||
restart: unless-stopped
|
||||
|
||||
<sidecar>:
|
||||
image: ...
|
||||
network_mode: "service:<anchor>"
|
||||
|
||||
networks:
|
||||
direct:
|
||||
external: true
|
||||
enclave:
|
||||
external: true
|
||||
```
|
||||
|
||||
Conventions at work:
|
||||
|
||||
- **One anchor per project** holds the static `ipv4_address` on `direct`; sidecars share its network namespace via `network_mode: service:<anchor>`.
|
||||
- **Networks are referenced by key** (`direct`, `enclave`, `socket`) with `external: true` — the core composition owns their definitions.
|
||||
- **`enclave` membership plus traefik labels** publishes the service through the edge proxy; the proxy discovers it via the socket proxy, never via published ports.
|
||||
- **Data paths are tier-relative** (`./@tierN/...`) and resolve through the symlinks created by `tiers/link`.
|
||||
- **Backup is opt-in via `restic.*` labels.** `apex backup/run` drives [resticontainer](https://github.com/deytenit/resticontainer), which discovers what to snapshot from compose labels: `restic.enable=true` marks the service, `restic.backup.paths=<container-path>` names the mounts to back up (resticontainer resolves them to host paths). A service with no `restic.*` labels is simply not backed up. Back up config only where the data is regenerable.
|
||||
- The `${APEX_UID...}` trio comes from the project `.env` maintained by `tiers/useradd`.
|
||||
|
||||
### Provision storage and identity
|
||||
|
||||
```bash
|
||||
apex tiers/link <tier1> <tier2> <tier3>
|
||||
apex tiers/useradd
|
||||
apex tiers/chown
|
||||
```
|
||||
|
||||
Re-running the trio is idempotent; it picks up the new project, creates its tier directories and `.env`, provisions its `noroot-<project>` user, and applies the ownership matrix.
|
||||
|
||||
### Lint
|
||||
|
||||
```bash
|
||||
apex utils/lint-docker-compose
|
||||
```
|
||||
|
||||
Validates every anchor IP (well-formed, unique, inside the node subnet) and runs the compose linter over the repository. The pre-commit hook enforces the same lint on staged compose files.
|
||||
|
||||
### Deploy
|
||||
|
||||
```bash
|
||||
apex compose up
|
||||
```
|
||||
|
||||
The core composition is already up; the new project starts in its alphabetical slot. Verify with `docker ps` and, if the service is proxied, through its route on the edge.
|
||||
|
||||
{{% /steps %}}
|
||||
|
||||
## Notes
|
||||
|
||||
- Service-internal secrets (for example a database password) live in the project's own `.env` in tier-1 storage, alongside the auto-injected UID block.
|
||||
- Never define `direct`/`enclave`/`socket` inside a project file — a second definition would race the core composition for network ownership.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Concept: Compositions and the shared core](/apex/concepts/compositions)
|
||||
- [Concept: Tiered storage and privilege isolation](/apex/concepts/tiered-storage)
|
||||
- [Concept: Fleet addressing scheme](/apex/concepts/addressing)
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
title: Reference
|
||||
description: CLI, actions, environment variables, and the engine API.
|
||||
weight: 30
|
||||
---
|
||||
|
||||
Technical reference for the apex fleet-operations framework surface: the command-line interface, the built-in [actions](/apex/glossary#action), the environment contract they consume, and the Python [engine](/apex/glossary#engine) API.
|
||||
|
||||
{{% cards %}}
|
||||
{{< card link="cli" title="CLI reference" icon="terminal" subtitle="Command-line invocation, subcommands, and exit codes." >}}
|
||||
{{< card link="actions" title="Actions reference" icon="play" subtitle="Built-in actions and their arguments." >}}
|
||||
{{< card link="environment" title="Environment reference" icon="variable" subtitle="Environment variables exposed to actions and compositions." >}}
|
||||
{{< card link="engine-api" title="Engine API reference" icon="code" subtitle="Python modules that power the engine." >}}
|
||||
{{% /cards %}}
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [GitHub: apex.ermnvldmr.com](https://github.com/deytenit/apex.ermnvldmr.com)
|
||||
@@ -0,0 +1,123 @@
|
||||
---
|
||||
title: Actions reference
|
||||
description: "Every commons action: arguments and behavior."
|
||||
weight: 32
|
||||
---
|
||||
|
||||
The sixteen [commons](/apex/glossary#commons) actions, by group. Arguments are shown as the parser accepts them; optional arguments are bracketed. Exit codes follow the [engine convention](/apex/reference/cli#exit-codes).
|
||||
|
||||
## configure {#configure}
|
||||
|
||||
### configure
|
||||
|
||||
`apex configure [--dry-run]`
|
||||
|
||||
Runs the whole suite in fixed order — `base`, `ufw`, `crowdsec`, `cron`, `systemd`, `routing` — and stops at the first failing step, propagating its exit code. `--dry-run` is forwarded to the steps that support it.
|
||||
|
||||
### configure/base
|
||||
|
||||
`apex configure/base`
|
||||
|
||||
Debian-only guard (exit `1` elsewhere). Updates the package index, installs the essential packages (`curl`, `wget`, `git`, `rsyslog`), ensures rsyslog is running, installs `ufw` when absent, and downloads the `ufw-docker` helper to `~/.local/bin/ufw-docker` (skipped when already present).
|
||||
|
||||
### configure/ufw
|
||||
|
||||
`apex configure/ufw [--dry-run]`
|
||||
|
||||
Applies the node firewall from `configs/ufw/`: `host/host.rules` through plain `ufw`, `host/before.rules`/`after.rules` as self-healing managed blocks inside `/etc/ufw/`, and `docker/*.rules` through `ufw-docker`. Exits `1` when the config root is missing, `66` when exactly one of `docker/`/`host/` is absent, `1` when a `/etc/ufw` target file does not exist. Requires `configure/base` first (checks `ufw` and `ufw-docker`). Finishes with a restart reminder.
|
||||
|
||||
### configure/crowdsec
|
||||
|
||||
`apex configure/crowdsec [--dry-run]`
|
||||
|
||||
Installs CrowdSec via the official script when `cscli` is absent, renders `configs/crowdsec/` into `/etc/crowdsec/` (backing up overwritten files), installs the `crowdsecurity/linux` and `crowdsecurity/traefik` collections, and installs the iptables firewall bouncer when `configs/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml.local` exists and the package is missing. `--dry-run` renders without writing and skips install steps.
|
||||
|
||||
### configure/cron
|
||||
|
||||
`apex configure/cron [--dry-run]`
|
||||
|
||||
Renders `configs/cron/crontab` with `ctx.vars()` and installs it via `crontab -`. Missing source file: exit `1`. Empty render: exit `65`, previous crontab kept. Failed install: previous crontab restored, exit `1`. The rendered result is always printed for review.
|
||||
|
||||
### configure/systemd
|
||||
|
||||
`apex configure/systemd [--dry-run]`
|
||||
|
||||
Renders every top-level, non-hidden file of `configs/systemd/` into `/etc/systemd/system/<name>`; for each `.service` unit it daemon-reloads and enables `--now`. Missing config directory: exit `1`.
|
||||
|
||||
### configure/routing
|
||||
|
||||
`apex configure/routing`
|
||||
|
||||
Enables `net.ipv4.ip_forward`, installs the fwmark `0x1` → routing-table-100 policy rule and the local default route in that table. Both checks are idempotent — re-runs change nothing.
|
||||
|
||||
## sync {#sync}
|
||||
|
||||
### sync/repository
|
||||
|
||||
`apex sync/repository <telegram_bot_url>`
|
||||
|
||||
The [capture-up](/apex/glossary#capture-up) commit: switches to (or creates) `sync/<node>`, stages the whole tree plus the refreshed commons submodule pointer, commits under a bot identity when anything changed, rebases onto `origin/main`, updates submodules, and pushes with `--force-with-lease`. Aborts (exit `1`) when the commons submodule is dirty or `compositions/` is missing; on a rebase conflict it aborts the rebase and pages via Telegram. Success and every failure path notify; a failed success-notification also exits `1`.
|
||||
|
||||
### sync/packages
|
||||
|
||||
`apex sync/packages <telegram_bot_url>`
|
||||
|
||||
Debian-only. Collects pending apt upgrades (`--just-print`) and out-of-date running container images (local RepoDigest vs the registry digest via `skopeo`), then sends one Telegram digest. Read-only with respect to packages and images. Unhandled failures page via Telegram and exit `1`; so does a failed digest delivery.
|
||||
|
||||
## backup {#backup}
|
||||
|
||||
### backup/run
|
||||
|
||||
`apex backup/run <telegram_bot_url>`
|
||||
|
||||
Drives the label-driven `resticontainer` core service one-shot (profile `manual`): repository reachability check with init-if-needed, then a single `backup` in which resticontainer discovers every service carrying `restic.*` compose labels, runs their pre-hooks, stops any container flagged `restic.backup.stop`, resolves the labeled container mounts to host paths, and snapshots the union in one restic run. The snapshot is stamped `--host <APEX_NODE_HOST>`, `--tag biweekly`, `--compression max`. A guard then proves a fresh snapshot for this host actually landed — if no `restic.*` labels were discovered the backup is a silent no-op, so it refuses to report success. Retention follows via `forget --prune` (7 daily / 4 weekly / 12 monthly). Each restic failure pages and exits `1`; success pages too, and a failed success-notification exits `1`.
|
||||
|
||||
## tiers {#tiers}
|
||||
|
||||
### tiers/link
|
||||
|
||||
`apex tiers/link <tier1_path> <tier2_path> <tier3_path>`
|
||||
|
||||
Builds the whole [tier](/apex/glossary#tier) symlink structure: node-level `compositions/@tierN` links, per-project tier directories, `shared/` areas with back-links, per-project `@tierN` links, and the `.env` symlink into tier 1 (target created when missing). Existing non-symlink paths prompt for confirmation; declining skips that link.
|
||||
|
||||
### tiers/useradd
|
||||
|
||||
`apex tiers/useradd`
|
||||
|
||||
Ensures the `noroot-shared` group, then one locked `noroot-<project>` system user per composition project (own primary group, no home, `nologin` shell, member of the shared group), and injects `APEX_UID`/`APEX_GID`/`APEX_SHARED_GID` into each project `.env` between managed markers.
|
||||
|
||||
### tiers/chown
|
||||
|
||||
`apex tiers/chown [tier1_path] [tier2_path] [tier3_path]`
|
||||
|
||||
Applies the ownership matrix (see [Concept: Tiered storage](/apex/concepts/tiered-storage#ownership)) across the tier roots — defaults are the node-level tier links. Requires `noroot-shared` to exist (exit `1` otherwise); projects with incomplete `.env` identity are skipped with a warning.
|
||||
|
||||
## utils {#utils}
|
||||
|
||||
### utils/extract-traefik-certs
|
||||
|
||||
`apex utils/extract-traefik-certs <source_acme> <dest_dir> [telegram_bot_url]`
|
||||
|
||||
Extracts the ACME store into per-domain `fullchain.pem`/`privkey.pem` via a containerized dumper, then `chmod 600` on the keys and re-owns the output tree to the invoking user. Missing source: exit `1`. Failures inside the extraction page via Telegram when a URL was given.
|
||||
|
||||
### utils/lint-docker-compose
|
||||
|
||||
`apex utils/lint-docker-compose [--hook]`
|
||||
|
||||
Two passes: (1) [anchor-IP](/apex/glossary#anchor-ip) validation over `compositions/*/docker-compose.yml` — well-formed, unique, inside the node subnet; (2) the `dclint` compose linter with `--fix`, run in a container over the repository (tier links, `.env`, and the commons submodule excluded). `--hook` restricts pass 2 to staged compose files and re-stages what the linter fixed. The exit code reflects both passes.
|
||||
|
||||
## compose {#compose}
|
||||
|
||||
### compose
|
||||
|
||||
`apex compose <action> [--dry-run] [extra...]`
|
||||
|
||||
Orchestrates `docker compose <action>` across the node: the core project first, then services alphabetically (reversed for `down`). `up` implies `-d`; `.env`/`apex.env` are passed per project when present; extra arguments pass through; `--dry-run` (accepted in any position) prints commands instead of executing.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Reference: CLI](/apex/reference/cli)
|
||||
- [Guide: Operate a node](/apex/guides/host-operations)
|
||||
- [GitHub: apex.ermnvldmr.com](https://github.com/deytenit/apex.ermnvldmr.com)
|
||||
@@ -0,0 +1,56 @@
|
||||
---
|
||||
title: CLI reference
|
||||
description: "The apex launcher: listing, dispatch, help, and exit codes."
|
||||
weight: 31
|
||||
---
|
||||
|
||||
The launcher is a single stdlib-Python script, `commons/apex` (on `PATH` after `init.sh`). Everything it does targets the local checkout — no node argument exists.
|
||||
|
||||
## Invocation forms {#invocation}
|
||||
|
||||
| Command | Behavior |
|
||||
|---|---|
|
||||
| `apex` or `apex --help` | List every available [action](/apex/glossary#action) with its summary; exit `0`. |
|
||||
| `apex <group>/<action> [args...]` | Dispatch one action. |
|
||||
| `apex <group>/<action> --help` | Print the action's argparse help generated from its `METADATA`; exit `0`. |
|
||||
|
||||
The listing tags each entry with its [overlay](/apex/glossary#overlay) source — no tag (commons), `(local)`, `(local override)` — and appends `[DISABLED: reason]` where a module declares `DISABLED`. Listing uses static `ast` scanning, so no action code executes.
|
||||
|
||||
## Layout and path resolution {#layout}
|
||||
|
||||
At startup the launcher decides which of two layouts it is in:
|
||||
|
||||
- **Node checkout** — the launcher lives in a directory named `commons/` whose parent contains `node.env`. The parent is the repository root; `proprietaries/actions/` is the overlay tree; [node identity](/apex/glossary#node-identity) is resolved before dispatch, and any identity warnings are logged first.
|
||||
- **Standalone commons** — any other location. The commons directory is its own root, there is no overlay, and `ctx.node` is `None` (log lines carry the `global` prefix).
|
||||
|
||||
`sys.path` is assembled as: the commons directory (makes `engine` importable), then `commons/vendor/` when present, then `../proprietaries/` in a node checkout (makes proprietary `lib/` code importable).
|
||||
|
||||
## Exit codes {#exit-codes}
|
||||
|
||||
| Code | Source |
|
||||
|---|---|
|
||||
| `0` | Action returned `0` or a non-integer; `--help` paths. |
|
||||
| `1` | Unknown action; disabled action; unhandled exception (logged as `Type: message` with the context prefix); non-integer `SystemExit`. |
|
||||
| `64` | Usage error from argument parsing (`EX_USAGE`). |
|
||||
| `65` | Data error raised by engine libraries (`EX_DATAERR`) — for example an empty rendered crontab. |
|
||||
| `66` | Missing input (`EX_NOINPUT`) — for example missing identity fields or an absent ufw config directory. |
|
||||
|
||||
Any other code is whatever the action itself returned or raised.
|
||||
|
||||
## Log format {#log-format}
|
||||
|
||||
All engine and action output follows one line shape:
|
||||
|
||||
```text
|
||||
[<node>] [<group>/<action>] [LEVEL] message
|
||||
```
|
||||
|
||||
`LEVEL` is `INFO`, `WARN`, `ERROR`, or `SUCCESS`; `ERROR` lines go to stderr, everything else to stdout — cron-friendly for both capture and alerting.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Concept: Actions and the overlay](/apex/concepts/actions-and-overlay)
|
||||
- [Reference: Actions](/apex/reference/actions)
|
||||
- [Reference: Engine API](/apex/reference/engine-api)
|
||||
@@ -0,0 +1,88 @@
|
||||
---
|
||||
title: Engine API reference
|
||||
description: The descriptor and ctx APIs available to action authors.
|
||||
weight: 34
|
||||
---
|
||||
|
||||
The surface an [action](/apex/glossary#action) author programs against: the argument descriptors in `engine.descriptor` and the members of the execution context. Conceptual background in [Concept: The execution context](/apex/concepts/execution-context).
|
||||
|
||||
## Descriptors {#descriptors}
|
||||
|
||||
```python
|
||||
from engine.descriptor import Meta, Arg, Opt, Flag, Rest
|
||||
```
|
||||
|
||||
| Class | Signature | argparse mapping |
|
||||
|---|---|---|
|
||||
| `Meta` | `Meta(summary, args=[])` | Drives both the listing summary and the parser. |
|
||||
| `Arg` | `Arg(name, help="", required=True)` | Positional; `required=False` → `nargs="?"`, default `None`. |
|
||||
| `Opt` | `Opt(name, help="", default=None)` | Value option (`--name VALUE`) with a default. |
|
||||
| `Flag` | `Flag(name, help="")` | Boolean `store_true` flag. |
|
||||
| `Rest` | `Rest(name, help="")` | Greedy remainder (`argparse.REMAINDER`). |
|
||||
|
||||
Usage errors exit `64` instead of argparse's default `2`.
|
||||
|
||||
> [!WARNING]
|
||||
> `Rest` captures *everything* after the first positional — including flags. A `Flag` declared next to a `Rest` is only recognized *before* the positional; an action that must accept its flag anywhere has to strip it out of the remainder itself (the `compose` action does exactly this for `--dry-run`).
|
||||
|
||||
## ctx.log {#log}
|
||||
|
||||
`info(msg)` · `warn(msg)` · `success(msg)` → stdout; `error(msg)` → stderr. All lines carry the `[<node>] [<action>] [LEVEL]` prefix.
|
||||
|
||||
## ctx.sys {#sys}
|
||||
|
||||
| Method | Behavior |
|
||||
|---|---|
|
||||
| `run(cmd, check=True, capture=False, cwd=None, env=None, input=None)` | `subprocess.run` wrapper returning `CompletedProcess`; `capture=True` collects text stdout/stderr. |
|
||||
| `sudo(cmd, **kw)` | `run(["sudo", *cmd])`. |
|
||||
| `ok(cmd, **kw)` | `True` iff exit `0`; never raises. |
|
||||
| `service_exists(name)` / `service_is_active(name)` / `service_is_enabled(name)` | systemd unit queries. |
|
||||
| `start_and_enable(name)` | Start + enable when needed; returns final active state. |
|
||||
| `ensure_running(name)` | `enable --now` when inactive. |
|
||||
| `restart(name)` / `reload(name, dry_run=False)` / `daemon_reload()` | Service lifecycle. |
|
||||
| `wait_for(name, timeout=30, interval=2)` | Poll until active. |
|
||||
| `install_packages(packages)` | For each package: skip when its binary is on `PATH`, else `apt-get update -y` + `apt-get install -y`. |
|
||||
|
||||
## ctx.tpl {#tpl}
|
||||
|
||||
| Method | Behavior |
|
||||
|---|---|
|
||||
| `render(path, vars)` | Render one file (relative paths resolve against `configs/`); `safe_substitute` semantics — unknown variables stay literal. |
|
||||
| `render_dir(rel_dir, vars)` | Render every file under a `configs/` subtree; returns `{relative_path: content}`. |
|
||||
|
||||
## ctx.host {#host}
|
||||
|
||||
| Method | Behavior |
|
||||
|---|---|
|
||||
| `backup(path, sudo=False)` | Copy the file to `<path>.<epoch>.bak`; returns the backup path or `None`. |
|
||||
| `write_file(path, content, backup=True, dry_run=False, sudo=False)` | Atomic install via temp file (mode `0644`); optional prior backup; `sudo` variant preserves an existing target's mode. |
|
||||
| `inject_block(target, content, start, end, anchor=None, old_pairs=(), dry_run=False, sudo=True)` | Managed marker-block replacement; removes current and legacy blocks, inserts after the first anchor line; `False` when the anchor is missing; missing target is an error. |
|
||||
| `install_crontab(text, dry_run=False)` | Snapshot, refuse empty render (`SystemExit(65)`), preview, install via `crontab -`, restore snapshot on failure (`SystemExit(1)`). |
|
||||
|
||||
## ctx.ufw {#ufw}
|
||||
|
||||
`apply(rules_dir, config_dir=None, dry_run=False)` — the full firewall pass: directory guards (exit `66`), tool guards (`ufw`, `ufw-docker`), host rules, managed-block injection into `/etc/ufw/before.rules` and `after.rules`, container rules from `*.rules` files.
|
||||
|
||||
## ctx.crowdsec {#crowdsec}
|
||||
|
||||
`ensure_installed()` · `deploy(configs_subdir, vars, dry_run=False)` · `install_collections(collections)`.
|
||||
|
||||
## ctx.systemd {#systemd}
|
||||
|
||||
`deploy_units(configs_subdir, vars, dry_run=False)` — render the subtree's top-level non-hidden files into `/etc/systemd/system/`, then daemon-reload and `enable --now` each `.service`.
|
||||
|
||||
## ctx.notify {#notify}
|
||||
|
||||
`telegram(title, bot_url, message, node=None, level="INFO")` plus the shorthands `success` / `error` / `info` / `warn`. All return `bool` — `True` only when the message was delivered; an empty `bot_url` logs a warning and returns `False`. The alert `instance` defaults to the node's public host (`APEX_NODE_HOST`) — no domain or node prefix is applied.
|
||||
|
||||
## ctx.paths and ctx.node {#paths}
|
||||
|
||||
`ctx.paths`: `repo_root`, `commons`, `proprietaries`, `configs`, `compositions`, `core`, `tier1..3`, `stier1..3`, `root_tier1..3`. `ctx.node`: the identity record (`name`, `subnet`, `fqdn`, `host`) or `None` in a standalone commons checkout. `ctx.vars()`: the template/compose environment (see [Reference: Environment variables](/apex/reference/environment)). `ctx.commons.run(args)`: delegate to the shadowed commons action — only valid in an overriding proprietary module.
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Concept: The execution context](/apex/concepts/execution-context)
|
||||
- [Guide: Develop actions](/apex/guides/action-development)
|
||||
- [GitHub: apex.ermnvldmr.com](https://github.com/deytenit/apex.ermnvldmr.com)
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
title: Environment variables reference
|
||||
description: Every APEX_* variable, where it is set, and who consumes it.
|
||||
weight: 33
|
||||
---
|
||||
|
||||
The `APEX_*` namespace is the framework's entire configuration surface. Variables originate in one of four layers; later layers win when the orchestration actions assemble the process environment.
|
||||
|
||||
## node.env — identity {#node-env}
|
||||
|
||||
Committed at the repository root; read by [node identity](/apex/glossary#node-identity) resolution. The framework is domain-agnostic: a node declares its own FQDN and public host name — there is no cluster concept and no required FQDN shape.
|
||||
|
||||
| Variable | Meaning | Notes |
|
||||
|---|---|---|
|
||||
| `APEX_NODE_FQDN` | The node's real FQDN (Traefik `Host()` routing) | Falls back to the OS hostname (with a warning) when unset. |
|
||||
| `APEX_NODE_HOST` | The node's public host name (backup `--host`, obs/notify `instance`) | Defaults to `APEX_NODE_FQDN`. Its first label becomes the short `name`. |
|
||||
| `APEX_SUBNET` | The node's /24 for the `direct` network | Required, only source of the subnet. Missing = exit `66`. |
|
||||
| `APEX_GIT_AUTHOR_NAME` | Bot identity for capture-up commits | Generic default (`apex [bot]`). |
|
||||
| `APEX_GIT_AUTHOR_EMAIL` | Bot identity email for capture-up commits | Generic default (`apex@localhost`). |
|
||||
|
||||
## apex.env — committed scalars {#apex-env}
|
||||
|
||||
Committed at `compositions/apex/`; non-secret by definition.
|
||||
|
||||
| Variable | Meaning | Notes |
|
||||
|---|---|---|
|
||||
| `APEX_TRAEFIK_DIRECT_IP` | The edge proxy's [anchor IP](/apex/glossary#anchor-ip) in the node subnet | Required by the core composition (`?error`). |
|
||||
| `APEX_RESTIC_REPOSITORY` | Backup repository locator | Required (`?error`). |
|
||||
| `APEX_RESTIC_COMPRESSION` | Restic compression level | Optional; defaults to `max` (fleet policy — `backup/run` also forces `--compression max`). |
|
||||
| `APEX_RESTIC_AWS_REGION` | S3 backend region | Optional; defaults to `us-east-1`. |
|
||||
| `APEX_RESTIC_AWS_ENDPOINT` | S3 backend endpoint URL | Optional; defaults to empty (AWS S3). Set for any non-AWS S3-compatible store. |
|
||||
| `APEX_LOKI_PUSH_URL` | Loki ingest URL for the [Alloy](/apex/concepts/architecture) agent (`loki.write`) | Required by the `alloy` profile (`?error`). |
|
||||
| `APEX_PROM_PUSH_URL` | Prometheus `remote_write` URL for the Alloy agent | Required by the `alloy` profile (`?error`). |
|
||||
| `APEX_OBS_BASIC_AUTH_USER` | Basic-auth user for the observability push targets | Required by the `alloy` profile (`?error`). |
|
||||
| `COMPOSE_PROFILES` | Opt-in core profiles (`adguard`, `xray`, `alloy`) | May be empty; `manual` is activated by `backup/run`, not here. |
|
||||
| `TZ` | Container timezone | Optional; defaults to `Etc/UTC`. |
|
||||
|
||||
## .env — secrets {#secrets}
|
||||
|
||||
Uncommitted; a symlink into tier-1 storage. Keys are documented per node in `SECRETS.md`.
|
||||
|
||||
| Variable | Meaning | Notes |
|
||||
|---|---|---|
|
||||
| `APEX_UID`, `APEX_GID`, `APEX_SHARED_GID` | The project's noroot identity | Auto-injected by `tiers/useradd`; required (`?error`). |
|
||||
| `APEX_RESTIC_PASSWORD` | Backup repository password | Required (`?error`). |
|
||||
| `APEX_RESTIC_AWS_ACCESS_KEY_ID`, `APEX_RESTIC_AWS_SECRET_ACCESS_KEY` | Object-storage credentials | Required (`?error`). |
|
||||
| `APEX_TRAEFIK_CF_DNS_API_TOKEN` | DNS-challenge token for ACME | Required (`?error`). |
|
||||
| `APEX_TRAEFIK_ACME_EMAIL` | ACME account email | Required (`?error`). |
|
||||
| `APEX_OBS_BASIC_AUTH_PASS` | Basic-auth password for the observability push targets | Required by the `alloy` profile (`?error`). |
|
||||
| `APEX_LOKI_BASIC_AUTH` | htpasswd users for the central Loki/Prometheus ingest | Not core-required. Only on a node that itself fronts the central observability ingest. |
|
||||
|
||||
## Engine exports — ctx.vars() {#engine}
|
||||
|
||||
Computed by the [engine](/apex/glossary#engine) at invocation; exported to `docker compose` by the orchestration actions and available to every template render. Never set these by hand.
|
||||
|
||||
| Variable | Value |
|
||||
|---|---|
|
||||
| `APEX_NODE_HOST` | The node's public host name (`node.host`); backup `--host` and obs/notify `instance`. |
|
||||
| `APEX_NODE_FQDN` | The node's real FQDN (`node.fqdn`); used for edge routing rules. |
|
||||
| `APEX_SUBNET` | Node subnet. |
|
||||
| `APEX_COMMONS` | Absolute commons directory (`$APEX_COMMONS/apex` is the launcher — crontabs and systemd units use it). |
|
||||
| `APEX_REPO_ROOT` | Absolute node repository root. |
|
||||
| `APEX_TIER1`, `APEX_TIER2`, `APEX_TIER3` | Core project data directory per [tier](/apex/glossary#tier). |
|
||||
| `APEX_TIER1_SHARED`, `APEX_TIER2_SHARED`, `APEX_TIER3_SHARED` | Shared area per tier. |
|
||||
| `APEX_TIER_ROOT1`, `APEX_TIER_ROOT2`, `APEX_TIER_ROOT3` | Tier storage roots. |
|
||||
|
||||
> [!NOTE]
|
||||
> Variables marked `?error` above are enforced by the shared core composition: `docker compose` refuses to evaluate the model while they are unset, which turns a forgotten secret into an immediate, explicit failure.
|
||||
|
||||
Several core-composition variables use `${VAR:-default}` instead, so they are optional and overridable per node: `TZ` (`Etc/UTC`), `APEX_RESTIC_AWS_REGION` (`us-east-1`), `APEX_RESTIC_AWS_ENDPOINT` (empty), `APEX_RESTIC_COMPRESSION` (`max`), and `APEX_ENCLAVE_SUBNET` (`198.18.0.0/16`, the ipallowlist source range for the Traefik dashboard/adguard routes).
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Concept: The execution context](/apex/concepts/execution-context)
|
||||
- [Concept: Compositions and the shared core](/apex/concepts/compositions)
|
||||
- [Concept: Node identity resolution](/apex/concepts/node-identity)
|
||||
@@ -0,0 +1,79 @@
|
||||
---
|
||||
title: Troubleshooting
|
||||
description: Symptoms, diagnosis, and fixes for common apex failure modes.
|
||||
weight: 40
|
||||
---
|
||||
|
||||
Failure modes the [engine](/apex/glossary#engine) reports, with the exact log lines it emits.
|
||||
|
||||
## APEX_NODE_FQDN falls back to the hostname {#fqdn-fallback}
|
||||
|
||||
**Symptoms:** every action logs `APEX_NODE_FQDN not set in node.env; using hostname '<fqdn>'.`
|
||||
|
||||
**Diagnosis:** `node.env` does not set `APEX_NODE_FQDN`, so the engine derived the FQDN from the OS hostname. On a node whose hostname is the intended FQDN this is harmless; otherwise traefik `Host()` routing will use the wrong name.
|
||||
|
||||
**Solution:** set `APEX_NODE_FQDN` explicitly in `node.env` at the repository root (for example `APEX_NODE_FQDN=node1.example.com`).
|
||||
|
||||
## Identity exits 66 {#identity-66}
|
||||
|
||||
**Symptoms:** `identity: APEX_NODE_FQDN missing from node.env and no hostname` or `identity: APEX_SUBNET missing from node.env`, exit `66`.
|
||||
|
||||
**Diagnosis:** a required identity field cannot be resolved. The first line means neither `APEX_NODE_FQDN` nor an OS hostname yields an FQDN; the second means `APEX_SUBNET` is absent. The engine refuses to guess.
|
||||
|
||||
**Solution:** create or complete `node.env` at the repository root — set `APEX_SUBNET`, and set `APEX_NODE_FQDN` (or ensure the OS hostname is usable).
|
||||
|
||||
## configure/ufw exits 66 {#ufw-66}
|
||||
|
||||
**Symptoms:** `Rules directory does not exist: ...` or `Config directory does not exist: ...`, exit `66`.
|
||||
|
||||
**Diagnosis:** exactly one of `configs/ufw/docker/` or `configs/ufw/host/` is missing — a broken config layout rather than an intentionally rule-less node.
|
||||
|
||||
**Solution:** restore the missing directory (empty is acceptable: an empty `docker/` warns `No .rules files found ... nothing to apply.` and continues).
|
||||
|
||||
## ufw tooling missing {#ufw-tools}
|
||||
|
||||
**Symptoms:** `ufw not installed/working via sudo. Run configure/base first.` or `ufw-docker not found at ~/.local/bin/ufw-docker. Run configure/base first.`, exit `1`.
|
||||
|
||||
**Diagnosis:** the firewall layer's prerequisites are absent.
|
||||
|
||||
**Solution:** run `apex configure/base`, then retry `apex configure/ufw`.
|
||||
|
||||
## Crontab install refused or rolled back {#cron}
|
||||
|
||||
**Symptoms:** `Rendered crontab is empty; aborting (previous kept).` (exit `65`) or `crontab install failed; restoring previous.` (exit `1`).
|
||||
|
||||
**Diagnosis:** the rendered `configs/cron/crontab` was empty (template/variable problem — remember unknown `$VARS` stay literal, they do not blank the file) or `crontab` rejected the syntax.
|
||||
|
||||
**Solution:** inspect the printed preview, fix the template, re-run `apex configure/cron`. The previous crontab is intact in both cases.
|
||||
|
||||
## sync/repository aborts {#sync-aborts}
|
||||
|
||||
**Symptoms:** `Commons submodule at <path> has uncommitted changes. Aborting.` or `Directory '<repo>/compositions' does not exist. Cannot sync.`, exit `1`.
|
||||
|
||||
**Diagnosis:** the [capture-up](/apex/glossary#capture-up) commit refuses to record a broken checkout — a dirty pinned submodule or a missing [compositions](/apex/glossary#compositions) tree.
|
||||
|
||||
**Solution:** for a dirty submodule, `git -C commons checkout -- .` (or commit an intentional pin move); for a missing tree, restore the checkout before the next scheduled run.
|
||||
|
||||
## Telegram not delivered {#telegram}
|
||||
|
||||
**Symptoms:** `Telegram Bot URL not provided. Skipping notification.` or `Failed to send Telegram notification: <error>` — and notify-gated actions (`backup/run`, `sync/repository`, `sync/packages`) exit `1` even though their work succeeded.
|
||||
|
||||
**Diagnosis:** the bot URL argument was empty, or the send failed (network, proxy, revoked token). The non-zero exit is deliberate: an unattended run whose status cannot be reported is treated as failed.
|
||||
|
||||
**Solution:** verify the URL source file the cron line reads, network/proxy reachability, and the bot token; re-run manually to confirm delivery.
|
||||
|
||||
## docker compose rejects the core wrapper {#compose-include}
|
||||
|
||||
**Symptoms:** `docker compose` errors on the `include` keyword when evaluating `compositions/apex/docker-compose.yml`.
|
||||
|
||||
**Diagnosis:** the [core composition](/apex/glossary#core-composition) is consumed through `include:`, which requires Docker Compose v2.20+.
|
||||
|
||||
**Solution:** upgrade the Compose plugin (`docker compose version` to verify).
|
||||
|
||||
---
|
||||
|
||||
**See also:**
|
||||
|
||||
- [Reference: CLI](/apex/reference/cli)
|
||||
- [Guide: Operate a node](/apex/guides/host-operations)
|
||||
- [GitHub: apex.ermnvldmr.com](https://github.com/deytenit/apex.ermnvldmr.com)
|
||||
Reference in new issue
Block a user