MIS — IT Operations

There are 13 deployments stuck in Queued state. Each one will try to rebuild a 777 MB image when it runs. Cancel them all from the Coolify UI before doing anything else.

R.1.1 Open Coolify deployments page:

http://168.231.115.117:8000/project/.../application/.../deployment

R.1.2 For each Queued entry, click into it and hit Cancel. Do this individually. Do NOT click "Restart" on Coolify itself — that risks restarting the Docker daemon mid-prune.

R.1.3 When all queued items are cancelled, the deployment list should show only the most recent successful build (or the most recent failed one). The 13 queued count drops to 0.

Blindly excluding folders from the Docker image will silently 404 any dashboard that references them. Run this audit first to see if any live HTML/JS/JSON references the paths we're about to exclude.

If you want to re-run the audit yourself (e.g. after months of changes), the commands are below for reference:

grep -rE '(creative-review/page-reviews|pipeline/snapshots|creative-review/images)' --include='*.html' --include='*.js' --include='*.json' . | grep -v '\.git/' | head -50

Based on R.2 findings, write the new .dockerignore. The default below assumes R.2 returned no live references. Adjust if R.2 found matches.

R.3.1 Open F:/brainzyme-git/.dockerignore in your editor.

R.3.2 Replace contents with this audit-verified safe baseline:

.git/
node_modules/
*.zip
*.mp4
*.mov
*.psd
tmp/
output/pending/
_archive/
creative-review/page-reviews/*/screenshots/

R.3.3 Commit and push:

git add .dockerignore
git commit -m 'fix(deploy): shrink build context via .dockerignore'
git push

R.3.4 In Coolify UI, manually trigger a deploy of the latest commit. Watch the deploy log for the build context line:

R.3.5 Open apps.nutritionalproducts.org/creative-review/ and apps.nutritionalproducts.org/google-ads-audit/v4.0-alt/ — both dashboards must still render images. If either is broken, run Rollback 3 immediately.

Single-paste health-check block. Run this on the VPS after R.4–R.6. Confirms every defence is actually active and not silently misconfigured.

R.7.1 SSH to the VPS if you've exited, then paste:

echo "=== R.4: Docker log rotation ==="
cat /etc/docker/daemon.json 2>/dev/null
python3 -m json.tool /etc/docker/daemon.json > /dev/null 2>&1 && echo "✓ valid JSON" || echo "✗ INVALID/MISSING"
echo "Docker: $(systemctl is-active docker)"
echo ""
echo "=== R.5: journald cap ==="
grep -E "^SystemMaxUse" /etc/systemd/journald.conf || echo "✗ NOT SET"
journalctl --disk-usage
echo ""
echo "=== Disk health ==="
df -h /
echo ""
echo "=== Docker storage ==="
docker system df
echo ""
echo "=== Containers running ==="
docker ps --format "table {{.Names}}\t{{.Status}}"
echo ""
echo "=== Recent cleanup activity ==="
journalctl -u docker --since "1 hour ago" --no-pager 2>/dev/null | tail -5

R.7.2 Also check Coolify UI: Servers → [your server] → Docker Cleanup → scroll to Recent Executions. Should show at least one Success entry within the last 10–15 minutes — that's the proof cleanup is firing on schedule.

Without this, the next disk-fill is silent until deploys start failing. A 5-minute cron that checks disk and pings a webhook (Slack / Discord / email) when above threshold catches it 24+ hours early.

E.1 Decide your webhook target. For a quick start, use a free Discord webhook or a Reamaze inbox endpoint. Replace YOUR_WEBHOOK_URL below with the actual URL.

E.2 Create the alert script:

cat > /usr/local/bin/disk-alert.sh << 'EOF'
#!/bin/bash
USAGE=$(df -h / | awk 'NR==2 {gsub(/%/,""); print $5}')
THRESHOLD=60
WEBHOOK_URL="YOUR_WEBHOOK_URL"
if [ "$USAGE" -gt "$THRESHOLD" ]; then
  curl -sS -X POST -H 'Content-Type: application/json' \
    -d "{\"content\":\"[VPS srv843884] Disk at ${USAGE}% — above ${THRESHOLD}% threshold\"}" \
    "$WEBHOOK_URL" > /dev/null
fi
EOF
chmod +x /usr/local/bin/disk-alert.sh

E.3 Add to crontab (5-min interval):

*/5 * * * * /usr/local/bin/disk-alert.sh

E.4 Test the alert by temporarily lowering THRESHOLD to 10 and running the script manually:

/usr/local/bin/disk-alert.sh

Should fire a webhook. Restore THRESHOLD to 60 once verified.

1092 historical deployments in the Coolify table is bloat. Cap retention so the database stays small and the UI stays fast.

F.1 In Coolify UI: Settings → Advanced → find "Deployment history retention" (or similar — exact label varies by version).

F.2 Set to 50. Save.

F.3 If the setting doesn't exist in your Coolify version, manually clean from Postgres on next quiet maintenance window. Hold for now.

The right answer for high-frequency small updates. Right now every commit triggers a full nginx rebuild — even if all that changed was one JSON file. The real fix: nginx image holds the static shell only; data files live in a Coolify-mounted volume that workflows write to directly without any Docker rebuild.

D.1 Architecture overview:

• nginx image holds: HTML, JS, CSS, templates, layouts (rebuild rarely — only on theme changes)
• VPS volume holds: JSON data, screenshots, snapshots, all fast-changing assets (mounted at /usr/share/nginx/html/data)
• GitHub Actions workflow rsyncs changed data files to the volume on every push — no Docker build, no image layer, no prune

D.2 Implementation steps (do this on a calm afternoon — ~30 min):

1. In Coolify: add a bind-mount volume to the application: VPS path /data/brainzyme-static → container path /usr/share/nginx/html/data
2. Move creative-review/, google-ads-audit/pipeline/snapshots/, and other fast-churning data dirs to /data/ path under the new mount
3. Update HTML/JS references to point at /data/... paths
4. Add a GitHub Actions step that rsyncs the moved dirs from repo to /data/brainzyme-static/ via SSH on push (skipping the Docker rebuild path)
5. For static-shell-only commits, deploy.yml continues to trigger a normal Coolify rebuild

D.3 Lighter interim option: if full volume migration is too much right now, use a multi-stage Dockerfile where stage 1 is COPY . /build, stage 2 copies only the served output. Combined with Fix A's .dockerignore, keeps the final image layer small without changing serving architecture. ~10 min.