174 lines
7.7 KiB
Markdown
174 lines
7.7 KiB
Markdown
---
|
|
name: update-compositions
|
|
description: Use when updating, upgrading, or bumping Docker container images, tags, and composition services across one or more APEX fleet nodes.
|
|
---
|
|
|
|
# Updating APEX Compositions
|
|
|
|
This skill guides the safe, structured update of container images and compose definitions across nodes in the APEX fleet.
|
|
|
|
## Core Rules
|
|
|
|
1. **Verify access first**: Never execute commands against a node without confirming SSH connectivity and credentials.
|
|
2. **Hold critical services**: Always ask the user which services or compositions must NOT be touched before discovering or updating images.
|
|
3. **Classify update scope**: Separate updates into **Minor/Patch** (non-breaking) and **Major** (breaking / schema migration risks).
|
|
4. **Gate major updates**: Never apply a Major version bump without in-depth migration research and explicit user approval.
|
|
5. **Verify health post-recreation**: An update is not complete until containers reach `Up` and `healthy` status with clean logs.
|
|
|
|
---
|
|
|
|
## Workflow Steps
|
|
|
|
### Step 1: Confirm Node Access
|
|
|
|
1. Identify the target nodes from the user request (e.g., `icarus`, `daedalus`, `morpheus`).
|
|
2. If access credentials, custom SSH ports, or key paths are not declared in the environment's SSH configuration, ask the user:
|
|
- SSH hostname / IP
|
|
- SSH port (APEX default: `2222`)
|
|
- SSH user (APEX default: `adam`)
|
|
- Identity file / key
|
|
3. Test connectivity to each node before proceeding:
|
|
```bash
|
|
ssh -p <port> <user>@<node_fqdn> "hostname && uptime"
|
|
```
|
|
4. **Completion check**: All target nodes return exit code 0 on the connectivity probe.
|
|
|
|
---
|
|
|
|
### Step 2: Inquire About Critical Services & Holds
|
|
|
|
1. Prompt the user explicitly before inspecting or modifying any services:
|
|
> "Which critical services or compositions on these nodes should **NOT** be updated during this run (e.g. databases, SSO/Authelia, Mail, Vaultwarden, Matrix)?"
|
|
2. Record the freeze list (held services/compositions).
|
|
3. Ensure all subsequent steps skip services on the freeze list.
|
|
4. **Completion check**: User response received, and the list of excluded services is confirmed.
|
|
|
|
---
|
|
|
|
### Step 3: Inventory Running Services and Images
|
|
|
|
For each target node:
|
|
|
|
1. List active containers, current images, and status:
|
|
```bash
|
|
ssh <node> "docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}'"
|
|
```
|
|
2. Locate the compose files on the node:
|
|
- Shared core: `/srv/<node_fqdn>/compositions/apex/docker-compose.yml`
|
|
- Node applications: `/srv/<node_fqdn>/compositions/*/docker-compose.yml`
|
|
- Overrides: `docker-compose.override.yml` in each composition folder
|
|
3. Map every running container to its compose declaration:
|
|
- Image repository (e.g. `authelia/authelia`, `postgres`, `jellyfin/jellyfin`)
|
|
- Pinned tag / SHA digest (e.g. `:4`, `:16-alpine`, `:v1.37.1`)
|
|
- Whether the image is locally built (`build:` context present)
|
|
4. Exclude any services identified on the freeze list from Step 2.
|
|
5. **Completion check**: A complete table of non-held compositions, running tags, and compose file locations is constructed.
|
|
|
|
---
|
|
|
|
### Step 4: Seek Latest Tags and Classify Update Types
|
|
|
|
For each candidate service:
|
|
|
|
1. Query the container registry (Docker Hub, GitHub Container Registry `ghcr.io`, Quay, etc.) for available release tags. Refer to [references/registry-queries.md](references/registry-queries.md) for registry API patterns.
|
|
2. Determine the latest stable versioned tag:
|
|
- Prefer semantic versioned tags (e.g. `1.38.0`, `16.4-alpine`) matching the existing packaging flavor.
|
|
- Ignore pre-release, alpha, beta, or release-candidate (`-rc`) tags unless the running container already pins one.
|
|
- Do not replace a pinned version tag with bare `:latest`.
|
|
3. Compare the current tag with the latest tag using Semantic Versioning (SemVer: `MAJOR.MINOR.PATCH`):
|
|
|
|
| Tag Transition | Classification | Action Path |
|
|
| :--- | :--- | :--- |
|
|
| `1.37.1` -> `1.37.2` | **Patch** | Proceed to Step 6 |
|
|
| `1.37.1` -> `1.38.0` | **Minor** | Proceed to Step 6 |
|
|
| Floating tag (e.g. `:4` or `:16`) with new digest | **Minor / In-place** | Proceed to Step 6 |
|
|
| `16` -> `17` or `1.x` -> `2.x` | **Major** | **Route to Step 5 (Strict Gate)** |
|
|
| Locally built image (`build:`) | **Local Build** | Git update + build (Step 6) |
|
|
|
|
4. **Completion check**: Every eligible service has a determined target tag and is classified as Major or Minor/Patch.
|
|
|
|
---
|
|
|
|
### Step 5: Major Update Protocol (Strict Gating)
|
|
|
|
For each service classified as **Major**:
|
|
|
|
1. **Research Update Strategy**:
|
|
- Check upstream release notes, migration guides, and changelogs.
|
|
- Identify:
|
|
- Breaking configuration changes (renamed environment variables, CLI flags).
|
|
- Storage schema migrations (e.g., PostgreSQL major version pg_upgrade requirements).
|
|
- Deprecated dependencies or protocol changes.
|
|
2. **Formulate Migration Plan**:
|
|
- Outline pre-upgrade steps (e.g., explicit database dump `pg_dump` / `mysqldump` beyond normal snapshots).
|
|
- Outline required compose file changes (volumes, environment).
|
|
- Outline fallback/rollback steps if the upgrade fails.
|
|
3. **Request Explicit User Approval**:
|
|
- Present the findings to the user:
|
|
- Target service and node.
|
|
- Current version vs New major version.
|
|
- Summary of breaking changes and migration strategy.
|
|
- Ask: *"Do you approve proceeding with the major update for `<service>` on `<node>` under this strategy?"*
|
|
4. **Gate**:
|
|
- If approved: Proceed to Step 6 for this service.
|
|
- If rejected or deferred: Add service to the freeze list and leave it untouched.
|
|
5. **Completion check**: Every Major update either has documented strategy + user approval or is explicitly skipped.
|
|
|
|
---
|
|
|
|
### Step 6: Apply Updates and Recreate Containers
|
|
|
|
Follow node rollout order to minimize blast radius:
|
|
1. Application nodes / workers first (e.g., `icarus`).
|
|
2. Communications / data nodes next (e.g., `morpheus`).
|
|
3. Core edge / gateway / SSO nodes last (e.g., `daedalus`), ensuring routing and auth remain stable during earlier rollouts.
|
|
|
|
For each node:
|
|
|
|
1. **Pre-upgrade backup**:
|
|
- Trigger a backup of stateful volumes:
|
|
```bash
|
|
ssh <node> "apex backup/run"
|
|
```
|
|
- For database services being updated, execute an in-container logical export to host storage.
|
|
2. **Update image tags in compose files**:
|
|
- Modify the target `docker-compose.yml` or `docker-compose.override.yml` with the new version tag.
|
|
3. **Pull images**:
|
|
```bash
|
|
ssh <node> "cd /srv/<node_fqdn>/compositions/<service> && docker compose pull"
|
|
```
|
|
4. **Recreate services**:
|
|
```bash
|
|
ssh <node> "cd /srv/<node_fqdn>/compositions/<service> && docker compose up -d"
|
|
```
|
|
5. **Completion check**: Compose commands finish with exit code 0.
|
|
|
|
---
|
|
|
|
### Step 7: Verify Service Health & Stability
|
|
|
|
Immediately after recreation on each node:
|
|
|
|
1. **Inspect container state**:
|
|
```bash
|
|
ssh <node> "docker ps -a --filter name=<container_name> --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'"
|
|
```
|
|
- Verify status is `Up`.
|
|
- If a healthcheck is configured, verify it transitions to `(healthy)` and does NOT get stuck in `(unhealthy)` or `(starting)`.
|
|
2. **Inspect recent logs**:
|
|
```bash
|
|
ssh <node> "docker logs --tail 50 <container_name>"
|
|
```
|
|
- Check for crash loops, panic traces, connection refused, or migration errors.
|
|
3. **Verify HTTP/network response (if edge/web service)**:
|
|
```bash
|
|
curl -IfsS -k https://<service_fqdn> || true
|
|
```
|
|
4. **Rollback protocol**:
|
|
- If a container enters a crash loop (`Restarting`, `Exit 1`):
|
|
1. Capture failure logs.
|
|
2. Revert the image tag in `docker-compose.yml` to the prior tag.
|
|
3. Re-run `docker compose up -d`.
|
|
4. Notify user immediately with logs and root cause.
|
|
5. **Completion check**: All updated containers are confirmed `Up` and `healthy` with clean logs.
|