4.3 KiB
title, description, weight
| title | description | weight |
|---|---|---|
| Troubleshooting | Symptoms, diagnosis, and fixes for common apex failure modes. | 40 |
Failure modes the engine reports, with the exact log lines it emits.
APEX_NODE_FQDN falls back to the hostname
Symptoms: every action logs APEX_NODE_FQDN not set in node.env; using hostname '<fqdn>'.
Diagnosis: node.env does not set APEX_NODE_FQDN, so the engine derived the FQDN from the OS hostname. On a node whose hostname is the intended FQDN this is harmless; otherwise traefik Host() routing will use the wrong name.
Solution: set APEX_NODE_FQDN explicitly in node.env at the repository root (for example APEX_NODE_FQDN=node1.example.com).
Identity exits 66
Symptoms: identity: APEX_NODE_FQDN missing from node.env and no hostname or identity: APEX_SUBNET missing from node.env, exit 66.
Diagnosis: a required identity field cannot be resolved. The first line means neither APEX_NODE_FQDN nor an OS hostname yields an FQDN; the second means APEX_SUBNET is absent. The engine refuses to guess.
Solution: create or complete node.env at the repository root — set APEX_SUBNET, and set APEX_NODE_FQDN (or ensure the OS hostname is usable).
configure/ufw exits 66
Symptoms: Rules directory does not exist: ... or Config directory does not exist: ..., exit 66.
Diagnosis: exactly one of configs/ufw/docker/ or configs/ufw/host/ is missing — a broken config layout rather than an intentionally rule-less node.
Solution: restore the missing directory (empty is acceptable: an empty docker/ warns No .rules files found ... nothing to apply. and continues).
ufw tooling missing
Symptoms: ufw not installed/working via sudo. Run configure/base first. or ufw-docker not found at ~/.local/bin/ufw-docker. Run configure/base first., exit 1.
Diagnosis: the firewall layer's prerequisites are absent.
Solution: run apex configure/base, then retry apex configure/ufw.
Crontab install refused or rolled back
Symptoms: Rendered crontab is empty; aborting (previous kept). (exit 65) or crontab install failed; restoring previous. (exit 1).
Diagnosis: the rendered configs/cron/crontab was empty (template/variable problem — remember unknown $VARS stay literal, they do not blank the file) or crontab rejected the syntax.
Solution: inspect the printed preview, fix the template, re-run apex configure/cron. The previous crontab is intact in both cases.
sync/repository aborts
Symptoms: Commons submodule at <path> has uncommitted changes. Aborting. or Directory '<repo>/compositions' does not exist. Cannot sync., exit 1.
Diagnosis: the capture-up commit refuses to record a broken checkout — a dirty pinned submodule or a missing compositions tree.
Solution: for a dirty submodule, git -C commons checkout -- . (or commit an intentional pin move); for a missing tree, restore the checkout before the next scheduled run.
Telegram not delivered
Symptoms: Telegram Bot URL not provided. Skipping notification. or Failed to send Telegram notification: <error> — and notify-gated actions (backup/run, sync/repository, sync/packages) exit 1 even though their work succeeded.
Diagnosis: the bot URL argument was empty, or the send failed (network, proxy, revoked token). The non-zero exit is deliberate: an unattended run whose status cannot be reported is treated as failed.
Solution: verify the URL source file the cron line reads, network/proxy reachability, and the bot token; re-run manually to confirm delivery.
docker compose rejects the core wrapper
Symptoms: docker compose errors on the include keyword when evaluating compositions/apex/docker-compose.yml.
Diagnosis: the core composition is consumed through include:, which requires Docker Compose v2.20+.
Solution: upgrade the Compose plugin (docker compose version to verify).
See also: