diff --git a/.agents/skills/update-compositions/SKILL.md b/.agents/skills/update-compositions/SKILL.md new file mode 100644 index 0000000..e5e1f5d --- /dev/null +++ b/.agents/skills/update-compositions/SKILL.md @@ -0,0 +1,173 @@ +--- +name: update-compositions +description: Use when updating, upgrading, or bumping Docker container images, tags, and composition services across one or more APEX fleet nodes. +--- + +# Updating APEX Compositions + +This skill guides the safe, structured update of container images and compose definitions across nodes in the APEX fleet. + +## Core Rules + +1. **Verify access first**: Never execute commands against a node without confirming SSH connectivity and credentials. +2. **Hold critical services**: Always ask the user which services or compositions must NOT be touched before discovering or updating images. +3. **Classify update scope**: Separate updates into **Minor/Patch** (non-breaking) and **Major** (breaking / schema migration risks). +4. **Gate major updates**: Never apply a Major version bump without in-depth migration research and explicit user approval. +5. **Verify health post-recreation**: An update is not complete until containers reach `Up` and `healthy` status with clean logs. + +--- + +## Workflow Steps + +### Step 1: Confirm Node Access + +1. Identify the target nodes from the user request (e.g., `icarus`, `daedalus`, `morpheus`). +2. If access credentials, custom SSH ports, or key paths are not declared in the environment's SSH configuration, ask the user: + - SSH hostname / IP + - SSH port (APEX default: `2222`) + - SSH user (APEX default: `adam`) + - Identity file / key +3. Test connectivity to each node before proceeding: + ```bash + ssh -p @ "hostname && uptime" + ``` +4. **Completion check**: All target nodes return exit code 0 on the connectivity probe. + +--- + +### Step 2: Inquire About Critical Services & Holds + +1. Prompt the user explicitly before inspecting or modifying any services: + > "Which critical services or compositions on these nodes should **NOT** be updated during this run (e.g. databases, SSO/Authelia, Mail, Vaultwarden, Matrix)?" +2. Record the freeze list (held services/compositions). +3. Ensure all subsequent steps skip services on the freeze list. +4. **Completion check**: User response received, and the list of excluded services is confirmed. + +--- + +### Step 3: Inventory Running Services and Images + +For each target node: + +1. List active containers, current images, and status: + ```bash + ssh "docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}'" + ``` +2. Locate the compose files on the node: + - Shared core: `/srv//compositions/apex/docker-compose.yml` + - Node applications: `/srv//compositions/*/docker-compose.yml` + - Overrides: `docker-compose.override.yml` in each composition folder +3. Map every running container to its compose declaration: + - Image repository (e.g. `authelia/authelia`, `postgres`, `jellyfin/jellyfin`) + - Pinned tag / SHA digest (e.g. `:4`, `:16-alpine`, `:v1.37.1`) + - Whether the image is locally built (`build:` context present) +4. Exclude any services identified on the freeze list from Step 2. +5. **Completion check**: A complete table of non-held compositions, running tags, and compose file locations is constructed. + +--- + +### Step 4: Seek Latest Tags and Classify Update Types + +For each candidate service: + +1. Query the container registry (Docker Hub, GitHub Container Registry `ghcr.io`, Quay, etc.) for available release tags. Refer to [references/registry-queries.md](references/registry-queries.md) for registry API patterns. +2. Determine the latest stable versioned tag: + - Prefer semantic versioned tags (e.g. `1.38.0`, `16.4-alpine`) matching the existing packaging flavor. + - Ignore pre-release, alpha, beta, or release-candidate (`-rc`) tags unless the running container already pins one. + - Do not replace a pinned version tag with bare `:latest`. +3. Compare the current tag with the latest tag using Semantic Versioning (SemVer: `MAJOR.MINOR.PATCH`): + +| Tag Transition | Classification | Action Path | +| :--- | :--- | :--- | +| `1.37.1` -> `1.37.2` | **Patch** | Proceed to Step 6 | +| `1.37.1` -> `1.38.0` | **Minor** | Proceed to Step 6 | +| Floating tag (e.g. `:4` or `:16`) with new digest | **Minor / In-place** | Proceed to Step 6 | +| `16` -> `17` or `1.x` -> `2.x` | **Major** | **Route to Step 5 (Strict Gate)** | +| Locally built image (`build:`) | **Local Build** | Git update + build (Step 6) | + +4. **Completion check**: Every eligible service has a determined target tag and is classified as Major or Minor/Patch. + +--- + +### Step 5: Major Update Protocol (Strict Gating) + +For each service classified as **Major**: + +1. **Research Update Strategy**: + - Check upstream release notes, migration guides, and changelogs. + - Identify: + - Breaking configuration changes (renamed environment variables, CLI flags). + - Storage schema migrations (e.g., PostgreSQL major version pg_upgrade requirements). + - Deprecated dependencies or protocol changes. +2. **Formulate Migration Plan**: + - Outline pre-upgrade steps (e.g., explicit database dump `pg_dump` / `mysqldump` beyond normal snapshots). + - Outline required compose file changes (volumes, environment). + - Outline fallback/rollback steps if the upgrade fails. +3. **Request Explicit User Approval**: + - Present the findings to the user: + - Target service and node. + - Current version vs New major version. + - Summary of breaking changes and migration strategy. + - Ask: *"Do you approve proceeding with the major update for `` on `` under this strategy?"* +4. **Gate**: + - If approved: Proceed to Step 6 for this service. + - If rejected or deferred: Add service to the freeze list and leave it untouched. +5. **Completion check**: Every Major update either has documented strategy + user approval or is explicitly skipped. + +--- + +### Step 6: Apply Updates and Recreate Containers + +Follow node rollout order to minimize blast radius: +1. Application nodes / workers first (e.g., `icarus`). +2. Communications / data nodes next (e.g., `morpheus`). +3. Core edge / gateway / SSO nodes last (e.g., `daedalus`), ensuring routing and auth remain stable during earlier rollouts. + +For each node: + +1. **Pre-upgrade backup**: + - Trigger a backup of stateful volumes: + ```bash + ssh "apex backup/run" + ``` + - For database services being updated, execute an in-container logical export to host storage. +2. **Update image tags in compose files**: + - Modify the target `docker-compose.yml` or `docker-compose.override.yml` with the new version tag. +3. **Pull images**: + ```bash + ssh "cd /srv//compositions/ && docker compose pull" + ``` +4. **Recreate services**: + ```bash + ssh "cd /srv//compositions/ && docker compose up -d" + ``` +5. **Completion check**: Compose commands finish with exit code 0. + +--- + +### Step 7: Verify Service Health & Stability + +Immediately after recreation on each node: + +1. **Inspect container state**: + ```bash + ssh "docker ps -a --filter name= --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'" + ``` + - Verify status is `Up`. + - If a healthcheck is configured, verify it transitions to `(healthy)` and does NOT get stuck in `(unhealthy)` or `(starting)`. +2. **Inspect recent logs**: + ```bash + ssh "docker logs --tail 50 " + ``` + - Check for crash loops, panic traces, connection refused, or migration errors. +3. **Verify HTTP/network response (if edge/web service)**: + ```bash + curl -IfsS -k https:// || true + ``` +4. **Rollback protocol**: + - If a container enters a crash loop (`Restarting`, `Exit 1`): + 1. Capture failure logs. + 2. Revert the image tag in `docker-compose.yml` to the prior tag. + 3. Re-run `docker compose up -d`. + 4. Notify user immediately with logs and root cause. +5. **Completion check**: All updated containers are confirmed `Up` and `healthy` with clean logs. diff --git a/.agents/skills/update-compositions/references/registry-queries.md b/.agents/skills/update-compositions/references/registry-queries.md new file mode 100644 index 0000000..3396134 --- /dev/null +++ b/.agents/skills/update-compositions/references/registry-queries.md @@ -0,0 +1,99 @@ +# Container Registry Tag Queries + +This reference details how to discover available image tags across common container registries without requiring external tooling. + +## 1. Docker Hub (`hub.docker.com`) + +Docker Hub provides an unauthenticated v2 API for public repositories. + +### Official Images (Library) +For top-level images like `postgres`, `redis`, `traefik`, `mariadb`: + +```bash +IMAGE="postgres" +curl -s "https://registry.hub.docker.com/v2/repositories/library/${IMAGE}/tags?page_size=100" \ + | python3 -c ' +import sys, json +data = json.load(sys.stdin) +tags = [t["name"] for t in data.get("results", [])] +print("\n".join(tags[:30])) +' +``` + +### Community / User Images +For namespace images like `adguard/adguardhome`, `prom/prometheus`, `grafana/alloy`: + +```bash +ORG="adguard" +IMAGE="adguardhome" +curl -s "https://registry.hub.docker.com/v2/repositories/${ORG}/${IMAGE}/tags?page_size=100" \ + | python3 -c ' +import sys, json +data = json.load(sys.stdin) +tags = [t["name"] for t in data.get("results", [])] +print("\n".join(tags[:30])) +' +``` + +--- + +## 2. GitHub Container Registry (`ghcr.io`) + +GHCR implements OCI distribution spec. Public image tags can be queried with an anonymous bearer token. + +```bash +IMAGE="xtls/xray-core" +TOKEN=$(curl -s "https://ghcr.io/token?scope=repository:${IMAGE}:pull" | python3 -c 'import sys, json; print(json.load(sys.stdin)["token"])') +curl -s -H "Authorization: Bearer ${TOKEN}" "https://ghcr.io/v2/${IMAGE}/tags/list" \ + | python3 -c ' +import sys, json +data = json.load(sys.stdin) +tags = data.get("tags", []) +print("\n".join(sorted(tags, reverse=True)[:30])) +' +``` + +--- + +## 3. Quay.io (`quay.io`) + +Quay provides a REST API: + +```bash +REPO="repository_name" +curl -s "https://quay.io/api/v1/repository/${REPO}/tag/?limit=50&onlyActiveTags=true" \ + | python3 -c ' +import sys, json +data = json.load(sys.stdin) +tags = [t["name"] for t in data.get("tags", [])] +print("\n".join(tags[:30])) +' +``` + +--- + +## 4. SemVer Matching & Sorting + +When filtering tags to find the latest stable version matching the current container's flavor: + +```python +import re + +def parse_semver(tag: str): + # Match patterns like v1.2.3, 1.2.3-alpine, 16.4 + m = re.match(r"^v?(\d+)(?:\.(\d+))?(?:\.(\d+))?(?:-(.+))?$", tag) + if not m: + return None + major = int(m.group(1)) + minor = int(m.group(2) or 0) + patch = int(m.group(3) or 0) + flavor = m.group(4) or "" + return (major, minor, patch, flavor) +``` + +Rules for candidate selection: +1. Preserve flavor: If currently running `16-alpine`, match only tags with suffix `-alpine`. +2. Ignore pre-releases: Filter out tags containing `alpha`, `beta`, `rc`, `dev`, `test`, or git SHAs. +3. Classify: + - Same `major`, higher `minor` or `patch` -> **Minor / Patch**. + - Higher `major` -> **Major** (requires Step 5 approval).