deytenit/apex.ermnvldmr.com
1
0 Fork 0
Code Issues Pull Requests Actions Releases Wiki Activity

chore(compositions/apex): bump core service image tags to latest minor/patch

Browse Source
This commit is contained in:
deytenit committed 2026-09-27 20:17:15 +03:00
1 parent c94745b7e7
commit 48840c3022
2 files changed
+132 -71
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.agents/skills/update-compositions/SKILL.md
+128 -67
View File
@@ -1,19 +1,21 @@
---
name: update-compositions
description: Use when updating, upgrading, or bumping Docker container images, tags, and composition services across one or more APEX fleet nodes.
description: Use when updating, upgrading, or bumping Docker container images, tags, composition services, and OS packages across one or more APEX fleet nodes.
---
# Updating APEX Compositions
# Updating APEX Compositions & System Packages
This skill guides the safe, structured update of container images and compose definitions across nodes in the APEX fleet.
This skill guides the safe, structured update of container images, compose definitions, and base operating system packages across nodes in an APEX fleet.
## Core Rules
1. **Verify access first**: Never execute commands against a node without confirming SSH connectivity and credentials.
2. **Hold critical services**: Always ask the user which services or compositions must NOT be touched before discovering or updating images.
3. **Classify update scope**: Separate updates into **Minor/Patch** (non-breaking) and **Major** (breaking / schema migration risks).
4. **Gate major updates**: Never apply a Major version bump without in-depth migration research and explicit user approval.
5. **Verify health post-recreation**: An update is not complete until containers reach `Up` and `healthy` status with clean logs.
3. **Classify update scope**: Separate updates into **Minor/Patch** (non-breaking), **Major** (breaking / schema migration risks), and **Unstable SemVer / CalVer** (requiring changelog scrutiny).
4. **Gate major and breaking updates**: Never apply a Major version bump, SemVer `0.x` minor change, or CalVer jump without migration research and explicit user approval.
5. **Preserve APEX engine context**: Always orchestrate compose actions through `./commons/apex compose` (or with derived `APEX_*` variables and layered env files) rather than bare `docker compose`.
6. **Safeguard in-tree patches**: Do not bump images for containers with host bind mounts that patch application source code without explicit review.
7. **Verify fleet observability post-rollout**: An update is not complete until containers reach `Up` and `healthy`, logs are clean, Prometheus scrape targets report `UP`, blackbox probes pass, and remote write streams are established.
---
@@ -21,11 +23,11 @@ This skill guides the safe, structured update of container images and compose de
### Step 1: Confirm Node Access
1. Identify the target nodes from the user request (e.g., `icarus`, `daedalus`, `morpheus`).
2. If access credentials, custom SSH ports, or key paths are not declared in the environment's SSH configuration, ask the user:
1. Identify the target nodes from the user request.
2. If access credentials, custom SSH ports, or key paths are not declared in the environment's SSH configuration, confirm:
- SSH hostname / IP
- SSH port (APEX default: `2222`)
- SSH user (APEX default: `adam`)
- SSH user
- Identity file / key
3. Test connectivity to each node before proceeding:
```bash
@@ -38,14 +40,14 @@ This skill guides the safe, structured update of container images and compose de
### Step 2: Inquire About Critical Services & Holds
1. Prompt the user explicitly before inspecting or modifying any services:
> "Which critical services or compositions on these nodes should **NOT** be updated during this run (e.g. databases, SSO/Authelia, Mail, Vaultwarden, Matrix)?"
> "Which critical services or compositions on these nodes should **NOT** be updated during this run (e.g. databases, SSO/Identity providers, Mail, Vaults, core routing)?"
2. Record the freeze list (held services/compositions).
3. Ensure all subsequent steps skip services on the freeze list.
4. **Completion check**: User response received, and the list of excluded services is confirmed.
---
### Step 3: Inventory Running Services and Images
### Step 3: Inventory Running Services, Images, and Custom Patches
For each target node:
@@ -56,13 +58,16 @@ For each target node:
2. Locate the compose files on the node:
- Shared core: `/srv/<node_fqdn>/compositions/apex/docker-compose.yml`
- Node applications: `/srv/<node_fqdn>/compositions/*/docker-compose.yml`
- Overrides: `docker-compose.override.yml` in each composition folder
- Overrides: `docker-compose.override.yml` in each composition directory
3. Map every running container to its compose declaration:
- Image repository (e.g. `authelia/authelia`, `postgres`, `jellyfin/jellyfin`)
- Pinned tag / SHA digest (e.g. `:4`, `:16-alpine`, `:v1.37.1`)
- Image repository
- Pinned tag / SHA digest
- Whether the image is locally built (`build:` context present)
4. Exclude any services identified on the freeze list from Step 2.
5. **Completion check**: A complete table of non-held compositions, running tags, and compose file locations is constructed.
4. **Inspect for in-tree patches & mounts**:
- Check container volume declarations for bind mounts into application code paths (e.g. `./patches/...:/app/...`, monkeypatched modules, or custom entrypoint scripts).
- If a service mounts patches into internal software paths, flag it: upstream image updates may alter internal APIs and break the patch. Add to hold or require explicit user confirmation.
5. Exclude any services identified on the freeze list from Step 2.
6. **Completion check**: A complete table of candidate compositions, running tags, compose file paths, and in-tree patch status is constructed.
---
@@ -72,56 +77,68 @@ For each candidate service:
1. Query the container registry (Docker Hub, GitHub Container Registry `ghcr.io`, Quay, etc.) for available release tags. Refer to [references/registry-queries.md](references/registry-queries.md) for registry API patterns.
2. Determine the latest stable versioned tag:
- Prefer semantic versioned tags (e.g. `1.38.0`, `16.4-alpine`) matching the existing packaging flavor.
- Prefer semantic versioned tags matching the existing packaging flavor (e.g. `-alpine`, `-bookworm`).
- Ignore pre-release, alpha, beta, or release-candidate (`-rc`) tags unless the running container already pins one.
- Do not replace a pinned version tag with bare `:latest`.
3. Compare the current tag with the latest tag using Semantic Versioning (SemVer: `MAJOR.MINOR.PATCH`):
3. Compare the current tag with the latest tag:
| Tag Transition | Classification | Action Path |
| :--- | :--- | :--- |
| `1.37.1` -> `1.37.2` | **Patch** | Proceed to Step 6 |
| `1.37.1` -> `1.38.0` | **Minor** | Proceed to Step 6 |
| Floating tag (e.g. `:4` or `:16`) with new digest | **Minor / In-place** | Proceed to Step 6 |
| `16` -> `17` or `1.x` -> `2.x` | **Major** | **Route to Step 5 (Strict Gate)** |
| Locally built image (`build:`) | **Local Build** | Git update + build (Step 6) |
| Tag Transition | Category | Classification | Action Path |
| :--- | :--- | :--- | :--- |
| `1.37.1` -> `1.37.2` | SemVer `1.x+` | **Patch** | Safe -> Proceed to Step 6 |
| `1.37.1` -> `1.38.0` | SemVer `1.x+` | **Minor** | Safe -> Proceed to Step 6 |
| `0.4.1` -> `0.4.2` | SemVer `0.x` | **0.x Patch** | Safe -> Proceed to Step 6 |
| `0.4.1` -> `0.7.0` | SemVer `0.x` | **0.x Minor** | **Route to Step 5 (Strict Gate)** |
| `YY.M.D` -> `YY.M.D` | CalVer / Date | **CalVer Jump** | Multi-month jump -> **Route to Step 5** |
| Floating tag (e.g. `:4`, `:16`) | Floating | **In-place Digest Bump** | Safe -> Proceed to Step 6 |
| `1.x` -> `2.x` or `16` -> `17` | SemVer `1.x+` | **Major** | **Route to Step 5 (Strict Gate)** |
| Locally built image (`build:`) | Local Build | **Local Build** | Git update + build (Step 6) |
4. **Completion check**: Every eligible service has a determined target tag and is classified as Major or Minor/Patch.
> [!IMPORTANT]
> **SemVer 0.x Rule**: Per SemVer specification Clause 4, version `0.x` software has no public API stability guarantees. Minor version bumps (e.g. `0.4` -> `0.7`) often contain breaking changes, renamed environment variables, or schema changes. Treat them with the same scrutiny as Major updates.
>
> **CalVer Rule**: Projects using date-based versioning (e.g. Xray, Ubuntu base images) do not follow SemVer breaking change rules. Releases spanning several months must be vetted against upstream changelogs for breaking configuration or default behavior shifts.
4. **Completion check**: Every eligible service has a determined target tag and update classification.
---
### Step 5: Major Update Protocol (Strict Gating)
### Step 5: Major & Breaking Update Protocol (Strict Gating)
For each service classified as **Major**:
For each service classified as **Major**, **0.x Minor**, or **CalVer Jump**:
1. **Research Update Strategy**:
1. **Research Upstream Changes**:
- Check upstream release notes, migration guides, and changelogs.
- Identify:
- Breaking configuration changes (renamed environment variables, CLI flags).
- Storage schema migrations (e.g., PostgreSQL major version pg_upgrade requirements).
- Deprecated dependencies or protocol changes.
2. **Formulate Migration Plan**:
- Outline pre-upgrade steps (e.g., explicit database dump `pg_dump` / `mysqldump` beyond normal snapshots).
- Outline required compose file changes (volumes, environment).
- Outline fallback/rollback steps if the upgrade fails.
- Breaking configuration changes (renamed environment variables, CLI flags, config format alterations).
- Altered default security behaviors (e.g. new access control rules, loopback/private IP blocking, required authentication headers).
- Storage schema migrations (e.g., PostgreSQL major upgrade requirements).
2. **Formulate Migration & Rollback Plan**:
- Outline pre-upgrade steps (e.g. explicit logical database dump `pg_dump` / `mysqldump` beyond normal snapshots).
- Outline required compose file changes (volumes, environment variables).
- Outline fallback/rollback steps if the update fails.
3. **Request Explicit User Approval**:
- Present the findings to the user:
- Present findings to the user:
- Target service and node.
- Current version vs New major version.
- Current version vs New version.
- Summary of breaking changes and migration strategy.
- Ask: *"Do you approve proceeding with the major update for `<service>` on `<node>` under this strategy?"*
- Ask: *"Do you approve proceeding with the update for `<service>` on `<node>` under this strategy?"*
4. **Gate**:
- If approved: Proceed to Step 6 for this service.
- If rejected or deferred: Add service to the freeze list and leave it untouched.
5. **Completion check**: Every Major update either has documented strategy + user approval or is explicitly skipped.
5. **Completion check**: Every breaking or major update has documented strategy + user approval or is explicitly skipped.
---
### Step 6: Apply Updates and Recreate Containers
### Step 6: Apply Container Updates via APEX Engine
Follow node rollout order to minimize blast radius:
1. Application nodes / workers first (e.g., `icarus`).
2. Communications / data nodes next (e.g., `morpheus`).
3. Core edge / gateway / SSO nodes last (e.g., `daedalus`), ensuring routing and auth remain stable during earlier rollouts.
Follow node and service rollout order to minimize blast radius:
1. **Node ordering**:
- Application & worker nodes first.
- Communications, data, and storage nodes next.
- Core edge / gateway / SSO nodes last, ensuring routing and auth remain stable during earlier rollouts.
2. **Service dependency sequencing**:
- Infrastructure providers first (DNS resolvers, database servers, mail exchangers).
- Verify provider health before updating or restarting dependent consumer services (e.g. SSO forwardAuth, telemetry agents).
For each node:
@@ -131,43 +148,87 @@ For each node:
ssh <node> "apex backup/run"
```
- For database services being updated, execute an in-container logical export to host storage.
2. **Update image tags in compose files**:
- Modify the target `docker-compose.yml` or `docker-compose.override.yml` with the new version tag.
3. **Pull images**:
```bash
ssh <node> "cd /srv/<node_fqdn>/compositions/<service> && docker compose pull"
```
4. **Recreate services**:
```bash
ssh <node> "cd /srv/<node_fqdn>/compositions/<service> && docker compose up -d"
```
5. **Completion check**: Compose commands finish with exit code 0.
2. **Update image tags in compose definitions**:
- Modify the target `docker-compose.yml`, `docker-compose.override.yml`, or node configuration repository.
3. **Pull and recreate services via APEX CLI**:
- Always run through the node's APEX CLI to preserve derived `APEX_*` variables and layered env files:
```bash
ssh <node> "cd /srv/<node_fqdn> && ./commons/apex compose pull"
ssh <node> "cd /srv/<node_fqdn> && ./commons/apex compose up -d"
```
- *Note on `--force-recreate`*: When updating containers using floating tags, recovering from host reboots, or handling services with `tmpfs` state (e.g. mail spools), pass `--force-recreate` to ensure the container entrypoint runs a clean initialization:
```bash
ssh <node> "cd /srv/<node_fqdn> && ./commons/apex compose up -d --force-recreate"
```
4. **Completion check**: Compose commands finish with exit code 0.
---
### Step 7: Verify Service Health & Stability
### Step 7: Apply System Packages and OS Updates
Immediately after recreation on each node:
When the update scope includes host operating system packages:
1. **Pre-upgrade check**:
- Check available disk space: `df -h /`
- Check for broken package states: `sudo dpkg --audit`
2. **Handle unattended package configuration (debconf)**:
- Prevent interactive prompts (e.g. GRUB drive selection on VPS instances) from blocking non-interactive execution:
```bash
# Example for grub-pc on virtualized hosts using /dev/sda:
echo 'grub-pc grub-pc/install_devices multiselect /dev/sda' | sudo debconf-set-selections
```
3. **Execute system package upgrade**:
```bash
ssh <node> "sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade -y"
```
4. **Clean package cache**:
```bash
ssh <node> "sudo apt-get autoremove -y && sudo apt-get clean"
```
5. **Reboot handling & coordination**:
- Check if a system reboot is required:
```bash
ssh <node> "test -f /var/run/reboot-required && echo 'REBOOT_REQUIRED' || echo 'OK'"
```
- If reboot is required:
1. Notify the user before initiating the reboot.
2. Reboot node: `ssh <node> "sudo reboot"`
3. Wait for SSH availability: poll connectivity with timeout.
4. After boot, verify systemd state (`systemctl is-system-running`) and check that Docker containers auto-started.
5. If services with `tmpfs` mounts fail startup checks (e.g. Postfix spool directories in mail containers), run `./commons/apex compose up -d --force-recreate`.
6. **Completion check**: Host packages are up to date and node is online.
---
### Step 8: Verify Service Health & Fleet Observability
Immediately after recreation and host reboots on each node:
1. **Inspect container state**:
```bash
ssh <node> "docker ps -a --filter name=<container_name> --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'"
ssh <node> "docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'"
```
- Verify status is `Up`.
- If a healthcheck is configured, verify it transitions to `(healthy)` and does NOT get stuck in `(unhealthy)` or `(starting)`.
- Verify healthchecks transition to `(healthy)` and do not hang in `(unhealthy)` or `(starting)`.
2. **Inspect recent logs**:
```bash
ssh <node> "docker logs --tail 50 <container_name>"
```
- Check for crash loops, panic traces, connection refused, or migration errors.
3. **Verify HTTP/network response (if edge/web service)**:
- Check for crash loops, panic traces, database connection errors, or authentication failures.
3. **Verify HTTP/network response (edge/web services)**:
```bash
curl -IfsS -k https://<service_fqdn> || true
```
4. **Rollback protocol**:
- If a container enters a crash loop (`Restarting`, `Exit 1`):
4. **Observability & Telemetry Verification**:
- If the fleet runs Prometheus:
- Check scrape targets: `curl -s http://<prom_internal_ip>:9090/prometheus/api/v1/targets | jq -r '.data.activeTargets[] | "\(.labels.job) | \(.health)"'` — verify all targets report `up`.
- Check blackbox probes: `curl -s 'http://<prom_internal_ip>:9090/prometheus/api/v1/query?query=probe_success==0'` — verify 0 failing probes.
- Check active alerts: `curl -s http://<prom_internal_ip>:9090/prometheus/api/v1/alerts | jq '.data.alerts[] | select(.state=="firing")'` — verify only expected watchdogs fire.
- Check remote write: confirm telemetry agent (e.g. Alloy) is actively shipping host/container metrics without push retry errors.
5. **Rollback protocol**:
- If a container enters a crash loop (`Restarting`, `Exit 1`) or breaks ingress:
1. Capture failure logs.
2. Revert the image tag in `docker-compose.yml` to the prior tag.
3. Re-run `docker compose up -d`.
2. Revert the image tag in the compose file to the prior tag.
3. Re-run `./commons/apex compose up -d --force-recreate`.
4. Notify user immediately with logs and root cause.
5. **Completion check**: All updated containers are confirmed `Up` and `healthy` with clean logs.
6. **Completion check**: All containers are confirmed `Up` and `healthy`, external ingress functions normally, and telemetry metrics/targets are green.
compositions/apex/docker-compose.yml
+4 -4
View File
@@ -10,7 +10,7 @@ services:
# --- optional: adguard (shares traefik netns) ---
adguard:
image: adguard/adguardhome:v0.107.72
image: adguard/adguardhome:v0.107.79
container_name: apex-adguard
volumes:
- ${APEX_TIER1}/adguard/data:/opt/adguardhome/work
@@ -39,7 +39,7 @@ services:
# Alloy needs egress (direct, not enclave). Replaces the old store-only `loki`
# service (which had no feeder). Config is a committed, env-parameterized asset.
alloy:
image: grafana/alloy:v1.17.1
image: grafana/alloy:v1.20.0
container_name: apex-alloy
volumes:
- ${APEX_COMMONS}/assets/alloy/config.alloy:/etc/alloy/config.alloy:ro
@@ -137,7 +137,7 @@ services:
# --- edge reverse proxy (the core anchor on direct) ---
traefik:
image: traefik:3.6
image: traefik:3.7
container_name: apex-traefik
depends_on:
- socket-proxy
@@ -205,7 +205,7 @@ services:
security_opt:
- no-new-privileges:true
xray:
image: ghcr.io/xtls/xray-core:26.3.27
image: ghcr.io/xtls/xray-core:26.9.9
container_name: apex-xray
volumes:
- ${APEX_TIER1}/xray/data/config.jsonc:/etc/xray/config.jsonc:ro