From a13931f7201e8a42cc8972040ede1effb91a3258 Mon Sep 17 00:00:00 2001 From: Codex Date: Fri, 1 May 2026 12:35:53 +0800 Subject: [PATCH] chore: add capacity governance automation --- project-docs/运维手册-2026-05-01.md | 19 + .../闭环交付与深度开发方案-2026-05-01.md | 2 +- ...容量治理与备份保留策略执行记录-2026-05-01.md | 72 +++ ...7.3-容量治理与备份保留策略方案-2026-05-01.md | 90 ++++ scripts/run-capacity-governance.ps1 | 428 ++++++++++++++++++ 5 files changed, 610 insertions(+), 1 deletion(-) create mode 100644 project-docs/阶段7.3-容量治理与备份保留策略执行记录-2026-05-01.md create mode 100644 project-docs/阶段7.3-容量治理与备份保留策略方案-2026-05-01.md create mode 100644 scripts/run-capacity-governance.ps1 diff --git a/project-docs/运维手册-2026-05-01.md b/project-docs/运维手册-2026-05-01.md index a7df6c80..b37e1896 100644 --- a/project-docs/运维手册-2026-05-01.md +++ b/project-docs/运维手册-2026-05-01.md @@ -269,6 +269,25 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-production-o 当前容量基线：阶段 7.2 巡检显示服务器磁盘约 `87%-88%`，已触发 WARN。若继续增长到 `90%`，巡检门禁将转为 FAIL；应优先清理过期备份、Docker build cache 和未使用镜像。 +容量治理 dry run： + +```powershell +powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-capacity-governance.ps1 +``` + +容量治理正式执行： + +```powershell +powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-capacity-governance.ps1 -Apply +``` + +默认保留策略： + +1. 磁盘 `80%` WARN，`90%` FAIL。 +2. 发布备份保留最近 `10` 个，并且只删除超过 `14` 天的旧文件。 +3. 回滚备份保留最近 `10` 个，并且只删除超过 `14` 天的旧文件。 +4. Docker 清理只处理 builder cache 和 dangling images，不执行 `docker system prune -a`，不删除 volumes。 + 列出可回滚的 frontend 备份： ```powershell diff --git a/project-docs/闭环交付与深度开发方案-2026-05-01.md b/project-docs/闭环交付与深度开发方案-2026-05-01.md index c5ad4277..67ba8868 100644 --- a/project-docs/闭环交付与深度开发方案-2026-05-01.md +++ b/project-docs/闭环交付与深度开发方案-2026-05-01.md @@ -20,7 +20,7 @@ 阶段 1 已完成源码闭环与生产入口闭环，生产无人 E2E `18/18` 通过。阶段 2 已完成 Review 重试、Knowledge Base 文档重试、Review -> Chat -> Contract 上下文联动、首页交付工作台和阶段 2 自动化无人闭环，本地与生产 E2E 均 `7/7` 通过。阶段 3 已完成系统 readiness、试点交付准备页、默认账号说明、演示样本数据、依赖健康门禁和阶段 3 自动化无人闭环，本地与生产 E2E 均 `5/5` 通过。 -当前后续重点已从“生产运营自动化与发布门禁固化”推进到“生产运营常态化与一键发布”：阶段 7 首批一键发布闭环已完成，阶段 7.2 一键回滚与生产巡检门禁也已完成首批闭环。当前唯一持续 WARN 是服务器磁盘使用率约 `87%-88%`，下一步应进入备份保留、历史 Docker 资源清理和容量治理。 +当前后续重点已从“生产运营自动化与发布门禁固化”推进到“生产运营常态化与一键发布”：阶段 7 首批一键发布闭环已完成，阶段 7.2 一键回滚与生产巡检门禁也已完成首批闭环。当前正在进入阶段 7.3 容量治理：固化备份保留策略、Docker build cache/旧镜像清理和磁盘告警阈值。 --- diff --git a/project-docs/阶段7.3-容量治理与备份保留策略执行记录-2026-05-01.md b/project-docs/阶段7.3-容量治理与备份保留策略执行记录-2026-05-01.md new file mode 100644 index 00000000..4348f176 --- /dev/null +++ b/project-docs/阶段7.3-容量治理与备份保留策略执行记录-2026-05-01.md @@ -0,0 +1,72 @@ +# 阶段 7.3：容量治理与备份保留策略执行记录（2026-05-01） + +> 当前目标：处理阶段 7.2 暴露的磁盘容量 WARN，将备份保留策略、Docker build cache/旧镜像清理和磁盘告警阈值固化为自动化流程。 + +--- + +## 一、当前执行结论 + +| 主线 | 状态 | 备注 | +|---|---|---| +| gstack 安装校验 | PASS | 已返回 `GSTACK_OK` | +| 阶段 7.3 方案 | READY | 已新增容量治理方案文档 | +| 容量治理脚本 | READY | 已新增 `scripts/run-capacity-governance.ps1` | +| 初始容量状态 | WARN | 远端 `/` 磁盘 `88%` | +| dry run | PASS_WITH_CAPACITY_WARN | 策略计算通过，备份无删除候选，Docker 可回收空间明显 | +| 正式清理 | PENDING | 待执行 | +| 清理后巡检 | PENDING | 待执行 | +| 清理后 release gate | PENDING | 待执行 | + +--- + +## 二、初始现场状态 + +轻量盘点结果： + +```text +/dev/mapper/ol-root 70G total, 62G used, 8.5G available, 88% used +``` + +frontend 备份包当前很小，不是主要容量压力来源： + +```text +/opt/ai-contract-platform/backups/phase7-deploy/frontend-before-2026-05-01T11-21-50.tgz +/opt/ai-contract-platform/backups/phase7-deploy/frontend-before-2026-05-01T11-26-02.tgz +/opt/ai-contract-platform/backups/phase7-rollback/frontend-current-before-rollback-2026-05-01T12-11-51-035.tgz +``` + +`docker system df` 在服务器上会超时，因此容量治理脚本对 Docker 诊断和清理都加入超时保护。 + +--- + +## 三、dry run 验证结果 + +验证命令： + +```powershell +powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-capacity-governance.ps1 -AllowDirtyWorktree +``` + +验证产物： + +```text +project-docs/qa-artifacts/phase7-capacity-governance/2026-05-01T12-34-41-578/ +``` + +结论： + +1. remote capacity run PASS。 +2. disk threshold WARN：当前磁盘 `89%`，超过 `80%` 预警阈值，低于 `90%` 失败阈值。 +3. backup retention PASS：发布备份 `2` 个、回滚备份 `1` 个，均无删除候选。 +4. Docker prune SKIP：dry run 不执行清理。 +5. Docker 盘点显示 images 可回收约 `9.7GB`，build cache 可回收约 `17.6GB`，容量压力主要来自 Docker 历史资源。 + +--- + +## 四、待执行验证 + +正式清理将在脚本提交后、干净工作区中执行： + +```powershell +powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-capacity-governance.ps1 -Apply +``` diff --git a/project-docs/阶段7.3-容量治理与备份保留策略方案-2026-05-01.md b/project-docs/阶段7.3-容量治理与备份保留策略方案-2026-05-01.md new file mode 100644 index 00000000..43978f4b --- /dev/null +++ b/project-docs/阶段7.3-容量治理与备份保留策略方案-2026-05-01.md @@ -0,0 +1,90 @@ +# 阶段 7.3：容量治理与备份保留策略方案（2026-05-01） + +> 前置状态：阶段 7.2 已完成一键回滚与生产巡检门禁。生产巡检持续报告服务器磁盘 `87%-88%` WARN，因此阶段 7.3 的目标是把容量治理做成可审计、可复跑、可控清理的运维闭环。 + +--- + +## 一、阶段目标 + +1. 固化磁盘告警阈值：`80% WARN`，`90% FAIL`。 +2. 固化 frontend 发布备份保留策略。 +3. 固化 frontend 回滚前备份保留策略。 +4. 清理 Docker build cache 和 dangling images。 +5. 所有清理先 dry run，正式执行必须留下 JSON/Markdown 证据。 +6. 清理后复跑生产巡检门禁和 release gate。 + +--- + +## 二、交付范围 + +| 主线 | 目标 | 交付物 | +|---|---|---| +| A. 容量治理脚本 | 统一盘点、dry run、正式清理 | `scripts/run-capacity-governance.ps1` | +| B. 保留策略 | 备份保留数量、年龄和删除边界 | 本文档、运维手册 | +| C. Docker 清理 | builder cache、dangling images | 容量治理脚本 | +| D. 验证证据 | dry run、apply、ops check、release gate | `project-docs/qa-artifacts/phase7-capacity-governance//` | + +--- + +## 三、默认策略 + +| 项目 | 默认值 | 说明 | +|---|---:|---| +| 磁盘 WARN | `80%` | 生产巡检和容量治理均记录 WARN | +| 磁盘 FAIL | `90%` | 达到后阻断门禁 | +| 发布备份保留数量 | `10` | 始终保留最近 10 个 | +| 发布备份最大年龄 | `14 天` | 只有超过数量且超过年龄才删除 | +| 回滚备份保留数量 | `10` | 始终保留最近 10 个 | +| 回滚备份最大年龄 | `14 天` | 只有超过数量且超过年龄才删除 | +| Docker builder prune | `24h` | 清理 24 小时以前的 builder cache | +| Docker prune 超时 | `180 秒` | 避免 Docker 诊断或清理卡死 | + +--- + +## 四、使用方式 + +dry run： + +```powershell +powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-capacity-governance.ps1 +``` + +正式执行： + +```powershell +powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-capacity-governance.ps1 -Apply +``` + +只执行备份保留，不清理 Docker： + +```powershell +powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-capacity-governance.ps1 -Apply -SkipDockerPrune +``` + +只清理 Docker，不删除备份： + +```powershell +powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\run-capacity-governance.ps1 -Apply -SkipBackupCleanup +``` + +--- + +## 五、安全边界 + +1. 默认不删除任何远端文件。 +2. 备份删除只使用 `rm -f`，不使用递归删除。 +3. 备份删除路径必须位于配置的备份根目录内。 +4. 备份必须同时满足“超过保留数量”和“超过保留年龄”才会删除。 +5. Docker 清理只执行 builder cache prune 和 dangling image prune，不执行 `docker system prune -a`，不删除 volumes。 +6. 所有 Docker 命令带超时，避免容量治理脚本挂死。 + +--- + +## 六、验收标准 + +1. dry run 能输出当前磁盘、备份候选、Docker 诊断结果。 +2. 正式执行不会删除最近备份。 +3. 正式执行会尝试清理 Docker builder cache 和 dangling images。 +4. 执行后生产巡检 required 项 PASS。 +5. 执行后 release gate PASS。 +6. 若磁盘仍高于 `80%`，必须保留 WARN 结论和下一步容量治理建议。 diff --git a/scripts/run-capacity-governance.ps1 b/scripts/run-capacity-governance.ps1 new file mode 100644 index 00000000..25eafd3d --- /dev/null +++ b/scripts/run-capacity-governance.ps1 @@ -0,0 +1,428 @@ +param( + [string]$ServerHost = "192.168.3.90", + [string]$SshUser = "root", + [string]$SshKeyPath = "$HOME\.ssh\water_audit_ed25519", + [string]$RemoteProjectPath = "/opt/ai-contract-platform", + [string]$RemoteDeployBackupRoot = "/opt/ai-contract-platform/backups/phase7-deploy", + [string]$RemoteRollbackBackupRoot = "/opt/ai-contract-platform/backups/phase7-rollback", + [string]$ArtifactRoot = "", + [int]$DiskWarnPercent = 80, + [int]$DiskFailPercent = 90, + [int]$DeployBackupKeep = 10, + [int]$DeployBackupMaxAgeDays = 14, + [int]$RollbackBackupKeep = 10, + [int]$RollbackBackupMaxAgeDays = 14, + [int]$DockerBuilderPruneUntilHours = 24, + [int]$DockerCommandTimeoutSeconds = 180, + [switch]$Apply, + [switch]$SkipBackupCleanup, + [switch]$SkipDockerPrune, + [switch]$AllowDirtyWorktree +) + +$ErrorActionPreference = "Stop" + +$repoRoot = Resolve-Path (Join-Path $PSScriptRoot "..") +$timestamp = Get-Date -Format "yyyy-MM-ddTHH-mm-ss-fff" +if ([string]::IsNullOrWhiteSpace($ArtifactRoot)) { + $ArtifactRoot = Join-Path $repoRoot "project-docs\qa-artifacts\phase7-capacity-governance" +} +if (-not [System.IO.Path]::IsPathRooted($ArtifactRoot)) { + $ArtifactRoot = Join-Path $repoRoot $ArtifactRoot +} +$artifactDir = Join-Path $ArtifactRoot $timestamp +$summaryJson = Join-Path $artifactDir "capacity-governance-summary.json" +$summaryMd = Join-Path $artifactDir "capacity-governance-summary.md" +$gitStatusPath = Join-Path $artifactDir "git-status.txt" +$remoteCapacityPath = Join-Path $artifactDir "remote-capacity-output.txt" + +New-Item -ItemType Directory -Force -Path $artifactDir | Out-Null + +$checks = New-Object System.Collections.Generic.List[object] + +function Add-Check { + param( + [string]$Key, + [string]$Label, + [ValidateSet("pass", "fail", "warn", "skip")] + [string]$Status, + [string]$Detail, + [bool]$Required = $true, + [hashtable]$Meta = @{} + ) + + $checks.Add([ordered]@{ + key = $Key + label = $Label + status = $Status + required = $Required + detail = $Detail + meta = $Meta + }) | Out-Null +} + +function Save-Json { + param( + [string]$Path, + [object]$Value + ) + $Value | ConvertTo-Json -Depth 12 | Set-Content -Encoding UTF8 -Path $Path +} + +function Convert-KeyValueOutput { + param([string]$Text) + + $data = @{} + foreach ($line in ($Text -split "`r?`n")) { + if ($line -match '^([A-Z0-9_]+)=(.*)$') { + $data[$matches[1]] = $matches[2] + } + } + return $data +} + +function Invoke-Remote { + param( + [string]$Name, + [string]$Command + ) + + $target = "${SshUser}@${ServerHost}" + $logPath = Join-Path $artifactDir "$Name.log" + $args = @( + "-i", $SshKeyPath, + "-o", "BatchMode=yes", + "-o", "StrictHostKeyChecking=accept-new", + "-o", "ConnectTimeout=10", + $target, + $Command + ) + $previousErrorActionPreference = $ErrorActionPreference + $ErrorActionPreference = "Continue" + try { + $output = & ssh @args 2>&1 + $exitCode = $LASTEXITCODE + } finally { + $ErrorActionPreference = $previousErrorActionPreference + } + $outputText = ($output -join [Environment]::NewLine) + $outputText | Set-Content -Encoding UTF8 -Path $logPath + return [ordered]@{ + exit_code = $exitCode + output = $outputText + log = $logPath + } +} + +if (-not (Test-Path "$HOME\.Codex\skills\gstack\bin")) { + Add-Check -Key "gstack" -Label "gstack preflight" -Status "fail" -Detail "gstack is missing" +} else { + Add-Check -Key "gstack" -Label "gstack preflight" -Status "pass" -Detail "GSTACK_OK" +} + +if (-not (Test-Path $SshKeyPath)) { + Add-Check -Key "ssh_key" -Label "SSH key" -Status "fail" -Detail "SSH key not found: $SshKeyPath" +} else { + Add-Check -Key "ssh_key" -Label "SSH key" -Status "pass" -Detail "SSH key exists" +} + +$branch = (git -C $repoRoot rev-parse --abbrev-ref HEAD).Trim() +$commit = (git -C $repoRoot rev-parse HEAD).Trim() +$remoteUrl = (git -C $repoRoot remote get-url origin).Trim() +$gitStatus = git -C $repoRoot status --short +($gitStatus -join [Environment]::NewLine) | Set-Content -Encoding UTF8 -Path $gitStatusPath +if ($gitStatus) { + if ($AllowDirtyWorktree) { + Add-Check -Key "working_tree" -Label "working tree" -Status "warn" -Required $false -Detail "Working tree has uncommitted changes; capacity governance allowed by -AllowDirtyWorktree" -Meta @{ artifact = $gitStatusPath } + } else { + Add-Check -Key "working_tree" -Label "working tree" -Status "fail" -Detail "Working tree has uncommitted changes; pass -AllowDirtyWorktree for development validation" -Meta @{ artifact = $gitStatusPath } + } +} else { + Add-Check -Key "working_tree" -Label "working tree" -Status "pass" -Detail "Working tree is clean" -Meta @{ artifact = $gitStatusPath } +} + +if ($remoteUrl -match "192\.168\.3\.90") { + Add-Check -Key "origin_remote" -Label "origin remote" -Status "pass" -Detail "origin points to 192.168.3.90" -Meta @{ + branch = $branch + commit = $commit + remote = $remoteUrl + } +} else { + Add-Check -Key "origin_remote" -Label "origin remote" -Status "fail" -Detail "origin does not point to 192.168.3.90" -Meta @{ + branch = $branch + commit = $commit + remote = $remoteUrl + } +} + +$failedPreflight = @($checks | Where-Object { $_.required -and $_.status -eq "fail" }) +if ($failedPreflight.Count -eq 0) { + $applyFlag = if ($Apply) { "true" } else { "false" } + $skipBackupFlag = if ($SkipBackupCleanup) { "true" } else { "false" } + $skipDockerFlag = if ($SkipDockerPrune) { "true" } else { "false" } + $remoteScript = @" +set -euo pipefail +project='$RemoteProjectPath' +deploy_root='$RemoteDeployBackupRoot' +rollback_root='$RemoteRollbackBackupRoot' +apply='$applyFlag' +skip_backup='$skipBackupFlag' +skip_docker='$skipDockerFlag' +deploy_keep='$DeployBackupKeep' +deploy_days='$DeployBackupMaxAgeDays' +rollback_keep='$RollbackBackupKeep' +rollback_days='$RollbackBackupMaxAgeDays' +builder_until_hours='$DockerBuilderPruneUntilHours' +docker_timeout='$DockerCommandTimeoutSeconds' + +emit_disk() { + label="`$1" + used="`$(df -P "`$project" | awk 'NR==2 {print `$5}' | tr -d %)" + avail="`$(df -P "`$project" | awk 'NR==2 {print `$4}')" + used_kb="`$(df -P "`$project" | awk 'NR==2 {print `$3}')" + echo "DISK_`${label}_USED_PERCENT=`$used" + echo "DISK_`${label}_AVAILABLE_KB=`$avail" + echo "DISK_`${label}_USED_KB=`$used_kb" +} + +scan_backups() { + prefix="`$1" + root="`$2" + pattern="`$3" + keep="`$4" + max_age_days="`$5" + now="`$(date +%s)" + count=0 + candidate_count=0 + candidate_bytes=0 + deleted_count=0 + deleted_bytes=0 + mkdir -p "`$root" + while read -r epoch_float size path; do + [ -n "`$path" ] || continue + epoch="`$(printf '%s\n' "`$epoch_float" | cut -d. -f1)" + count="`$((count + 1))" + age_days="`$(((now - epoch) / 86400))" + should_delete=false + if [ "`$count" -gt "`$keep" ] && [ "`$age_days" -gt "`$max_age_days" ]; then + should_delete=true + candidate_count="`$((candidate_count + 1))" + candidate_bytes="`$((candidate_bytes + size))" + echo "`${prefix}_DELETE_CANDIDATE=`$path|`$size|`$age_days" + fi + if [ "`$apply" = "true" ] && [ "`$skip_backup" != "true" ] && [ "`$should_delete" = "true" ]; then + case "`$path" in + "`$root"/*) rm -f -- "`$path" ;; + *) echo "`${prefix}_DELETE_REFUSED=`$path" ;; + esac + if [ ! -e "`$path" ]; then + deleted_count="`$((deleted_count + 1))" + deleted_bytes="`$((deleted_bytes + size))" + fi + fi + done < <(find "`$root" -maxdepth 1 -type f -name "`$pattern" -printf '%T@ %s %p\n' 2>/dev/null | sort -nr) + echo "`${prefix}_TOTAL_COUNT=`$count" + echo "`${prefix}_DELETE_CANDIDATE_COUNT=`$candidate_count" + echo "`${prefix}_DELETE_CANDIDATE_BYTES=`$candidate_bytes" + echo "`${prefix}_DELETED_COUNT=`$deleted_count" + echo "`${prefix}_DELETED_BYTES=`$deleted_bytes" +} + +cd "`$project" +test "`$PWD" = "`$project" +echo "REMOTE_PROJECT_OK=1" +echo "APPLY_MODE=`$apply" +emit_disk BEFORE + +echo "DOCKER_SYSTEM_DF_BEFORE_BEGIN" +timeout 20 docker system df || true +echo "DOCKER_SYSTEM_DF_BEFORE_END" + +scan_backups DEPLOY_BACKUP "`$deploy_root" 'frontend-before-*.tgz' "`$deploy_keep" "`$deploy_days" +scan_backups ROLLBACK_BACKUP "`$rollback_root" 'frontend-current-before-rollback-*.tgz' "`$rollback_keep" "`$rollback_days" + +if [ "`$apply" = "true" ] && [ "`$skip_docker" != "true" ]; then + echo "DOCKER_BUILDER_PRUNE_BEGIN" + if timeout "`$docker_timeout" docker builder prune -f --filter "until=`${builder_until_hours}h"; then + echo "DOCKER_BUILDER_PRUNE_EXIT=0" + else + echo "DOCKER_BUILDER_PRUNE_EXIT=`$?" + fi + echo "DOCKER_BUILDER_PRUNE_END" + echo "DOCKER_IMAGE_PRUNE_BEGIN" + if timeout "`$docker_timeout" docker image prune -f; then + echo "DOCKER_IMAGE_PRUNE_EXIT=0" + else + echo "DOCKER_IMAGE_PRUNE_EXIT=`$?" + fi + echo "DOCKER_IMAGE_PRUNE_END" +else + echo "DOCKER_PRUNE_SKIPPED=1" +fi + +emit_disk AFTER +echo "DOCKER_SYSTEM_DF_AFTER_BEGIN" +timeout 20 docker system df || true +echo "DOCKER_SYSTEM_DF_AFTER_END" +"@ + + $capacityRun = Invoke-Remote -Name "01-remote-capacity-governance" -Command $remoteScript + $capacityRun.output | Set-Content -Encoding UTF8 -Path $remoteCapacityPath + $remoteData = Convert-KeyValueOutput -Text $capacityRun.output + if ($capacityRun.exit_code -eq 0 -and $remoteData.REMOTE_PROJECT_OK -eq "1") { + Add-Check -Key "remote_capacity_run" -Label "remote capacity run" -Status "pass" -Detail "Remote capacity governance script completed" -Meta @{ artifact = $remoteCapacityPath } + } else { + Add-Check -Key "remote_capacity_run" -Label "remote capacity run" -Status "fail" -Detail "Remote capacity governance script failed" -Meta @{ + exit_code = $capacityRun.exit_code + artifact = $remoteCapacityPath + } + } + + $beforeDisk = if ($remoteData.DISK_BEFORE_USED_PERCENT) { [int]$remoteData.DISK_BEFORE_USED_PERCENT } else { 100 } + $afterDisk = if ($remoteData.DISK_AFTER_USED_PERCENT) { [int]$remoteData.DISK_AFTER_USED_PERCENT } else { 100 } + if ($afterDisk -ge $DiskFailPercent) { + Add-Check -Key "disk_threshold" -Label "disk threshold" -Status "fail" -Detail "Disk usage is $afterDisk%, fail threshold is $DiskFailPercent%" -Meta @{ + before = $beforeDisk + after = $afterDisk + artifact = $remoteCapacityPath + } + } elseif ($afterDisk -ge $DiskWarnPercent) { + Add-Check -Key "disk_threshold" -Label "disk threshold" -Status "warn" -Required $false -Detail "Disk usage is $afterDisk%, warn threshold is $DiskWarnPercent%" -Meta @{ + before = $beforeDisk + after = $afterDisk + artifact = $remoteCapacityPath + } + } else { + Add-Check -Key "disk_threshold" -Label "disk threshold" -Status "pass" -Detail "Disk usage is $afterDisk%" -Meta @{ + before = $beforeDisk + after = $afterDisk + artifact = $remoteCapacityPath + } + } + + $deployCandidates = if ($remoteData.DEPLOY_BACKUP_DELETE_CANDIDATE_COUNT) { [int]$remoteData.DEPLOY_BACKUP_DELETE_CANDIDATE_COUNT } else { 0 } + $rollbackCandidates = if ($remoteData.ROLLBACK_BACKUP_DELETE_CANDIDATE_COUNT) { [int]$remoteData.ROLLBACK_BACKUP_DELETE_CANDIDATE_COUNT } else { 0 } + $deployDeleted = if ($remoteData.DEPLOY_BACKUP_DELETED_COUNT) { [int]$remoteData.DEPLOY_BACKUP_DELETED_COUNT } else { 0 } + $rollbackDeleted = if ($remoteData.ROLLBACK_BACKUP_DELETED_COUNT) { [int]$remoteData.ROLLBACK_BACKUP_DELETED_COUNT } else { 0 } + if ($SkipBackupCleanup) { + Add-Check -Key "backup_retention" -Label "backup retention" -Status "skip" -Required $false -Detail "Skipped by -SkipBackupCleanup" + } elseif ($Apply) { + Add-Check -Key "backup_retention" -Label "backup retention" -Status "pass" -Detail "Backup retention applied; deleted deploy=$deployDeleted rollback=$rollbackDeleted" -Meta @{ + deploy_candidates = $deployCandidates + rollback_candidates = $rollbackCandidates + artifact = $remoteCapacityPath + } + } elseif (($deployCandidates + $rollbackCandidates) -gt 0) { + Add-Check -Key "backup_retention" -Label "backup retention" -Status "warn" -Required $false -Detail "Dry run found backup deletion candidates" -Meta @{ + deploy_candidates = $deployCandidates + rollback_candidates = $rollbackCandidates + artifact = $remoteCapacityPath + } + } else { + Add-Check -Key "backup_retention" -Label "backup retention" -Status "pass" -Detail "No backup deletion candidates under current retention policy" -Meta @{ + deploy_total = $remoteData.DEPLOY_BACKUP_TOTAL_COUNT + rollback_total = $remoteData.ROLLBACK_BACKUP_TOTAL_COUNT + artifact = $remoteCapacityPath + } + } + + if ($SkipDockerPrune) { + Add-Check -Key "docker_prune" -Label "Docker prune" -Status "skip" -Required $false -Detail "Skipped by -SkipDockerPrune" + } elseif ($Apply) { + $builderExit = if ($remoteData.DOCKER_BUILDER_PRUNE_EXIT) { [int]$remoteData.DOCKER_BUILDER_PRUNE_EXIT } else { -1 } + $imageExit = if ($remoteData.DOCKER_IMAGE_PRUNE_EXIT) { [int]$remoteData.DOCKER_IMAGE_PRUNE_EXIT } else { -1 } + if ($builderExit -eq 0 -and $imageExit -eq 0) { + Add-Check -Key "docker_prune" -Label "Docker prune" -Status "pass" -Detail "Docker builder prune and dangling image prune completed" -Meta @{ artifact = $remoteCapacityPath } + } else { + Add-Check -Key "docker_prune" -Label "Docker prune" -Status "warn" -Required $false -Detail "Docker prune completed with non-zero exit or timeout" -Meta @{ + builder_exit = $builderExit + image_exit = $imageExit + artifact = $remoteCapacityPath + } + } + } else { + Add-Check -Key "docker_prune" -Label "Docker prune" -Status "skip" -Required $false -Detail "Dry run only; pass -Apply to prune Docker builder cache and dangling images" + } +} + +$failedRequired = @($checks | Where-Object { $_.required -and $_.status -eq "fail" }) +$warningsList = @($checks | Where-Object { $_.status -eq "warn" }) +$skippedList = @($checks | Where-Object { $_.status -eq "skip" }) +$overallStatus = if ($failedRequired.Count -gt 0) { "FAIL" } elseif ($warningsList.Count -gt 0) { "WARN" } else { "PASS" } + +$summary = [ordered]@{ + name = "phase7-capacity-governance" + status = $overallStatus + generated_at = (Get-Date).ToUniversalTime().ToString("o") + artifact_dir = $artifactDir + branch = $branch + commit = $commit + remote = $remoteUrl + targets = [ordered]@{ + server_host = $ServerHost + remote_project_path = $RemoteProjectPath + deploy_backup_root = $RemoteDeployBackupRoot + rollback_backup_root = $RemoteRollbackBackupRoot + } + policy = [ordered]@{ + disk_warn_percent = $DiskWarnPercent + disk_fail_percent = $DiskFailPercent + deploy_backup_keep = $DeployBackupKeep + deploy_backup_max_age_days = $DeployBackupMaxAgeDays + rollback_backup_keep = $RollbackBackupKeep + rollback_backup_max_age_days = $RollbackBackupMaxAgeDays + docker_builder_prune_until_hours = $DockerBuilderPruneUntilHours + docker_command_timeout_seconds = $DockerCommandTimeoutSeconds + } + options = [ordered]@{ + apply = [bool]$Apply + skip_backup_cleanup = [bool]$SkipBackupCleanup + skip_docker_prune = [bool]$SkipDockerPrune + allow_dirty_worktree = [bool]$AllowDirtyWorktree + } + checks = $checks + totals = [ordered]@{ + total = $checks.Count + failed_required = $failedRequired.Count + warnings = $warningsList.Count + skipped = $skippedList.Count + } +} +Save-Json -Path $summaryJson -Value $summary + +$markdown = New-Object System.Collections.Generic.List[string] +$markdown.Add("# Phase 7 Capacity Governance") | Out-Null +$markdown.Add("") | Out-Null +$markdown.Add("- Status: $overallStatus") | Out-Null +$markdown.Add("- Generated at: $($summary.generated_at)") | Out-Null +$markdown.Add("- Branch: $branch") | Out-Null +$markdown.Add("- Commit: $commit") | Out-Null +$markdown.Add("- Artifact dir: $artifactDir") | Out-Null +$markdown.Add("- Apply mode: $([bool]$Apply)") | Out-Null +$markdown.Add("") | Out-Null +$markdown.Add("| Check | Required | Status | Detail |") | Out-Null +$markdown.Add("|---|---:|---|---|") | Out-Null +foreach ($check in $checks) { + $markdown.Add("| $($check.label) | $($check.required) | $($check.status.ToUpperInvariant()) | $($check.detail) |") | Out-Null +} +$markdown.Add("") | Out-Null +$markdown.Add("Policy:") | Out-Null +$markdown.Add("") | Out-Null +$markdown.Add("- Disk warn/fail: $DiskWarnPercent% / $DiskFailPercent%") | Out-Null +$markdown.Add("- Deploy backup retention: keep $DeployBackupKeep and only delete files older than $DeployBackupMaxAgeDays days") | Out-Null +$markdown.Add("- Rollback backup retention: keep $RollbackBackupKeep and only delete files older than $RollbackBackupMaxAgeDays days") | Out-Null +$markdown.Add("- Docker builder prune: until ${DockerBuilderPruneUntilHours}h") | Out-Null +$markdown.Add("") | Out-Null +$markdown.Add("Local artifacts:") | Out-Null +$markdown.Add("") | Out-Null +$markdown.Add('- `capacity-governance-summary.json`') | Out-Null +$markdown.Add('- `capacity-governance-summary.md`') | Out-Null +$markdown.Add('- `git-status.txt`') | Out-Null +$markdown.Add('- `remote-capacity-output.txt`') | Out-Null +$markdown.Add('- `*.log`') | Out-Null +$markdown | Set-Content -Encoding UTF8 -Path $summaryMd + +Write-Host "PHASE7_CAPACITY_GOVERNANCE_$overallStatus $artifactDir" +if ($overallStatus -eq "FAIL") { + throw "Phase 7 capacity governance failed. See $summaryMd" +}