diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000000000000000000000000000000000000..54858240b69a9a456a3abf5b384ed46c087e6d96 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,4 @@ +*.sh text eol=lf +Makefile text eol=lf +*.md text eol=lf +*.tsv text eol=lf diff --git a/README.md b/README.md index 8597cde318ca878902ff2980c0e1c9c1ef26c70a..78af5ec97f9294df9483c38a6ee2bd0d0e2896fe 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,10 @@ -# TencentOS发行版检查 -检查项目包括 -- KABI -- 软件包安装 -- 内核模块加载 -- systemd服务启动 \ No newline at end of file +# TencentOS发行版检查 +检查项目包括 +- KABI +- 软件包安装 +- 内核模块加载 +- systemd服务启动 + +并发软件包功能测试： + +- [基于 chroot 的并发安装、命令、服务和卸载测试](testcase/README-command-execute-chroot.md) diff --git a/testcase/Makefile b/testcase/Makefile index 1b4db5bcbf65a82a67d32917028e601b2dd629ba..760eb97ae79072b2e3447bcc2e20b115726d4347 100644 --- a/testcase/Makefile +++ b/testcase/Makefile @@ -19,3 +19,8 @@ clean: rm -rfv $(ALL_OBJ) $(ALL_BIN) cleanall: clean + +.PHONY: check-command-execute-chroot +check-command-execute-chroot: + bash tests/test-command-execute-chroot.sh + bash tests/test-chroot-regressions.sh diff --git a/testcase/README-command-execute-chroot.md b/testcase/README-command-execute-chroot.md new file mode 100644 index 0000000000000000000000000000000000000000..5dd97b1964e00a8b23b509583e47959ca900449e --- /dev/null +++ b/testcase/README-command-execute-chroot.md @@ -0,0 +1,122 @@ +# 基于 chroot 的并发软件包测试 + +`command-execute-chroot.sh` 是独立于现有串行用例的新入口。它为每个软件包创建隔离 rootfs，并发执行： + +```text +安装 → ELF 命令探测 → systemd 服务测试 → 卸载 → 分类汇总 +``` + +## 环境 + +- OpenCloudOS 8/9 或 OpenCloudOS Stream； +- root 权限； +- Bash 4.4+、DNF、RPM、util-linux（需支持 `unshare --kill-child`）、coreutils； +- 可用的目标版本软件源； +- `systemd-container` 可选，用于在 rootfs 内真实启动 systemd 服务。 + +请在专用测试机使用。本脚本会安装软件包并执行其二进制命令，不应在生产主机运行。 + +## 使用 + +三个位置参数依次为：待测包列表、桌面白名单、特殊配置服务包列表。每行读取第一列，忽略空行和 `#` 注释。 + +```bash +sudo bash testcase/command-execute-chroot.sh \ + -j 4 \ + --base-rootfs /var/lib/yum-ci/base-oc9 \ + --output-dir ./artifacts/j4 \ + ./pkg.list ./desktop.list ./service.list +``` + +第一次运行会在 `--base-rootfs` 创建基础系统，后续并发度测试可直接复用。基础 +rootfs 会记录 releasever、架构和 repo/vars 指纹，并使用文件锁防止并发创建；任一 +信息不一致时脚本会拒绝复用，需换用空目录重新创建。未指定该参数时，基础系统随 +临时工作目录一起清理。 + +主要选项： + +| 选项 | 默认值 | 说明 | +| --- | --- | --- | +| `-j N` | `min(CPU, 8)` | 同时运行的包级 worker 数量，硬上限 64 | +| `--root-mode` | `auto` | 优先 overlay，不可用时回退 copy | +| `--service-mode` | `auto` | `auto`、`nspawn`、`skip` | +| `--package-timeout` | 1200 秒 | 单包完整生命周期超时 | +| `--stage-timeout` | 300 秒 | 安装/卸载阶段超时 | +| `--command-timeout` | 5 秒 | 单个命令探测超时 | +| `--service-timeout` | 60 秒 | nspawn 启动及单次 unit 操作超时 | +| `--releasever` | 宿主版本 | DNF 目标版本 | +| `--keep-work` | 关闭 | 保留临时 rootfs 用于排障 | + +## 隔离与并发 + +- 每个 worker 进入私有 mount/PID namespace；PID 1 退出后，内核会回收包括 + `setsid()` 后台进程在内的全部后代； +- rootfs 使用独立 overlay upper/work，失败时复制基础 rootfs； +- `/dev`、`/proc`、`/sys`、`/run` 只挂载在 worker 的私有命名空间； +- FIFO 令牌限制并发度，单个 worker 失败不会停止其他包，并发上限避免 FIFO 预填阻塞； +- 外层生命周期超时配合 `unshare --kill-child`，超时后清理整个 PID namespace； +- 每轮 nspawn 使用独立 machine 名；中断处理按 PID 启动时间核对并回收 worker session，避免 PID 复用误杀； +- 结果先写临时文件再原子重命名，汇总顺序始终与输入一致。 +- 汇总会严格校验结果的 17 列结构、包序号/名称、状态和数值字段；损坏或错配结果 + 会转换为基础设施错误，不能计为通过。 + +## 阶段规则 + +### 安装 + +使用 chroot 内 DNF 安装；基础 rootfs 已包含的包使用 `reinstall`。DNF 非零、超时、scriptlet/file 错误或安装后 RPM 查询失败均为 `FAIL`。 + +### 命令 + +所有包会先独立保存 RPM 文件清单供服务发现使用；桌面白名单的命令阶段为 `SKIP`。 +其他包枚举 `/usr/bin`、`/usr/sbin` 下的 ELF 文件，用 +`strace` 验证 `execve`，并检查超时、126/127、动态加载器缺库信息、异常非零退出 +及致命信号。没有 ELF 命令为 `PASS`。 + +### 服务 + +识别 `.service`、`.socket`、`.target`： + +- 特殊配置列表：`SKIP` 并写明原因； +- `auto`：存在 `systemd-nspawn` 时启动独立 systemd，执行 `start → is-active → stop`；否则明确 `SKIP`； +- `nspawn`：强制真实服务测试，环境不支持即 `FAIL`； +- `skip`：显式跳过； +- template unit 缺少实例名时单独标记 `SKIP`。 + +如需真实服务测试： + +```bash +sudo dnf install -y systemd-container +sudo bash testcase/command-execute-chroot.sh --service-mode nspawn ... +``` + +### 卸载 + +DNF remove 后由宿主 `rpm --root` 再次查询准确的已安装 NEVRA。仍存在、验证器 +异常或命令失败均为 `FAIL`。基础 rootfs 预装包先 `reinstall`，随后也执行真实卸载； +worker 的结果记录和验证工具不依赖被测 rootfs 内的命令。 + +## 输出 + +- `package-command-result-chroot-*.txt`：六列兼容报告与中文分类汇总； +- `package-command-detail-chroot-*.tsv`：每阶段状态、原因、耗时和计数； +- `yum-ci-chroot-*/packages/`：按包拆分的 install/command/service/remove 日志； +- `yum-ci-chroot-*/run-manifest.txt`：环境、参数和输入哈希。 + +包级结论优先级为 `FAIL > SKIP > PASS`，并强制满足： + +```text +通过 + 失败 + 跳过 = 待测总数 +``` + +## 检查与基准 + +```bash +make -C testcase check-command-execute-chroot + +sudo JOBS_LIST="1 2 4 8" REPEAT=1 \ + bash testcase/benchmark-command-execute-chroot.sh \ + /var/lib/yum-ci/base-oc9 pkg.list desktop.list service.list ./benchmark +``` + +正式提交性能数据时，应固定基础 rootfs、包列表和软件源，并记录冷/热缓存条件。 diff --git a/testcase/VALIDATION-command-execute-chroot.md b/testcase/VALIDATION-command-execute-chroot.md new file mode 100644 index 0000000000000000000000000000000000000000..281f26cceb15961952f6f210435ed1bb38e1b636 --- /dev/null +++ b/testcase/VALIDATION-command-execute-chroot.md @@ -0,0 +1,103 @@ +# IJV5MW 真实环境验证记录 + +验证时间：2026-07-29（UTC+8）。所有测试均在独立测试服务器执行，不使用生产环境。 + +2026-07-31 安全审查修复后，重新执行了全部静态检查和 32 项宿主回归测试；下方 +功能与性能数据仍是原测试服务器的实测记录。持久 rootfs 元数据已升级为 schema 2， +再次实机验证时需要使用空目录重建基础 rootfs。 + +## 环境 + +| 项目 | 值 | +| --- | --- | +| 操作系统 | OpenCloudOS 9.6 x86_64 | +| 内核 | 6.6.119-49.23.oc9.x86_64 | +| CPU | 8 vCPU，AMD EPYC 9754 | +| 内存 | 15 GiB | +| 磁盘 | 50 GiB，测试前可用 46 GiB | +| Bash | 5.2.15 | +| ShellCheck | 0.11.0 | +| 基础 rootfs | 359 MiB，所有性能轮次复用同一份 | + +## 静态与无 root 回归 + +以下检查全部通过： + +```bash +bash -n testcase/command-execute-chroot.sh \ + testcase/lib/chroot-package-worker.sh \ + testcase/tests/test-command-execute-chroot.sh \ + testcase/tests/test-chroot-regressions.sh \ + testcase/benchmark-command-execute-chroot.sh + +shellcheck testcase/command-execute-chroot.sh \ + testcase/lib/chroot-package-worker.sh \ + testcase/tests/test-command-execute-chroot.sh \ + testcase/tests/test-chroot-regressions.sh \ + testcase/benchmark-command-execute-chroot.sh + +make -C testcase check-command-execute-chroot # 32/32 PASS +git diff --check +``` + +## 功能验证 + +### overlay 与真实服务测试 + +输入：`bc`、`cronie`、`openssh-server`，并发度 2，强制 `--service-mode nspawn`。 + +| 包 | install | command | service | remove | +| --- | --- | --- | --- | --- | +| bc | PASS | PASS（2 个 ELF） | PASS（无 unit） | PASS | +| cronie | PASS | PASS（3 个 ELF） | PASS（`crond.service` start/is-active/stop） | PASS | +| openssh-server | PASS | PASS（1 个 ELF） | SKIP（特殊配置服务列表） | PASS | + +结果：通过 2、失败 0、跳过 1，对账为 3。每个包使用独立 overlay upper/work。 + +### copy 回退模式 + +使用相同输入和 `--root-mode copy` 再次运行，结果与 overlay 一致：通过 2、失败 0、跳过 1。证明 overlay 不可用时的 copy/reflink 路径可工作。 + +### 失败与超时 + +- 不存在包 `package-that-does-not-exist-ijv5mw`：退出码 1，自动归入“安装失败”，原因保留为 `No match for argument`； +- `--package-timeout 1`：退出码 1，原因明确为 `package lifecycle timed out after 1s`； +- 两次负向测试均满足 `PASS + FAIL + SKIP = total`。 + +## 并发基准 + +包列表：`bc which zip unzip jq tree cronie openssh-server`。`which` 位于桌面白名单，`openssh-server` 位于特殊服务列表。所有轮次复用相同 rootfs、软件源和输入，单次热缓存测试结果如下： + +| 并发度 | 耗时 | 加速比 | PASS/FAIL/SKIP | +| ---: | ---: | ---: | ---: | +| 1 | 10.379 s | 1.000x | 6/0/2 | +| 2 | 6.361 s | 1.632x | 6/0/2 | +| 4 | 4.387 s | 2.366x | 6/0/2 | +| 8 | 3.457 s | 3.002x | 6/0/2 | + +`-j 8` 相比 `-j 1` 减少约 66.7% 的包测试耗时。四轮中 `crond.service` 均真实通过 `start → is-active → stop`。 + +输入 SHA-256： + +```text +packages 5727bee73c74d606a7b8fb0689f3cf8f6d7f171425e394384aba1cf335dfe5d5 +desktop 44a4c727a56f128ff84cc51751a341a2ae03b0d540ec29f1db7a0e84e6222a80 +service 2fa9c6d58068eb9f6817ecdb962f237777094cc9daf196d9e443c70e5c7be92b +``` + +## 清理审计 + +在正常、安装失败、生命周期超时、overlay、copy 及四轮并发基准后分别检查： + +```text +yum-ci 挂载残留: 0 +systemd-nspawn machine 残留: 0 +worker 进程残留: 0 +/var/tmp 临时 rootfs 残留: 0 +``` + +持久保留的只有显式指定的 `/var/lib/yum-ci/base-oc9` 基础 rootfs和输出报告。 + +## 验证边界 + +本记录覆盖 OpenCloudOS 9.6。OpenCloudOS 8 和 Stream 的包名、Bash 版本及仓库差异仍建议由对应环境或社区 CI 复核；脚本避免使用 Bash 5 专属的 `wait -n -p`。 diff --git a/testcase/benchmark-command-execute-chroot.sh b/testcase/benchmark-command-execute-chroot.sh new file mode 100755 index 0000000000000000000000000000000000000000..d6694a752e681d78e78b14459db2f2fb44a8ea7b --- /dev/null +++ b/testcase/benchmark-command-execute-chroot.sh @@ -0,0 +1,72 @@ +#!/usr/bin/env bash + +set -uo pipefail + +if [ "$#" -ne 5 ]; then + cat >&2 <<'EOF' +Usage: benchmark-command-execute-chroot.sh BASE_ROOTFS PKG_LIST DESKTOP_LIST SERVICE_LIST OUTPUT_DIR + +Environment: + JOBS_LIST="1 2 4 8" concurrency values + REPEAT=1 repetitions per value + EXTRA_ARGS="..." extra runner options +EOF + exit 2 +fi + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +RUNNER="$SCRIPT_DIR/command-execute-chroot.sh" +BASE_ROOTFS=$1 +PKG_LIST=$2 +DESKTOP_LIST=$3 +SERVICE_LIST=$4 +OUTPUT_DIR=$5 +JOBS_LIST=${JOBS_LIST:-"1 2 4 8"} +REPEAT=${REPEAT:-1} +EXTRA_ARGS=${EXTRA_ARGS:-} + +[[ $REPEAT =~ ^[1-9][0-9]*$ ]] || { printf 'REPEAT must be positive\n' >&2; exit 2; } +[ ! -e "$OUTPUT_DIR" ] || { printf 'output directory already exists: %s\n' "$OUTPUT_DIR" >&2; exit 2; } +mkdir -p "$OUTPUT_DIR" + +RESULT="$OUTPUT_DIR/benchmark.tsv" +printf 'jobs\trepeat\texit_code\tduration_ms\tpass\tfail\tskip\tspeedup\n' >"$RESULT" +BASELINE="" + +for jobs in $JOBS_LIST; do + [[ $jobs =~ ^[1-9][0-9]*$ ]] || { printf 'invalid jobs value: %s\n' "$jobs" >&2; exit 2; } + for ((round = 1; round <= REPEAT; round++)); do + run_dir="$OUTPUT_DIR/j${jobs}-r${round}" + start=$(date +%s%3N) + rc=0 + # EXTRA_ARGS is intentionally word-split to allow documented runner options. + # shellcheck disable=SC2086 + bash "$RUNNER" -j "$jobs" --base-rootfs "$BASE_ROOTFS" --output-dir "$run_dir" \ + $EXTRA_ARGS "$PKG_LIST" "$DESKTOP_LIST" "$SERVICE_LIST" >"$run_dir.log" 2>&1 || rc=$? + end=$(date +%s%3N) + duration=$((end - start)) + summary=$(grep -E '环境: chroot .*待测包总数:' "$run_dir.log" | tail -n 1 || true) + pass=$(printf '%s' "$summary" | sed -n 's/.*通过: \([0-9][0-9]*\).*/\1/p') + fail=$(printf '%s' "$summary" | sed -n 's/.*失败: \([0-9][0-9]*\).*/\1/p') + skip=$(printf '%s' "$summary" | sed -n 's/.*跳过: \([0-9][0-9]*\).*/\1/p') + pass=${pass:-0}; fail=${fail:-0}; skip=${skip:-0} + if [ -z "$BASELINE" ] && [ "$jobs" -eq 1 ]; then BASELINE=$duration; fi + speedup=NA + if [ -n "$BASELINE" ]; then + speedup=$(awk -v base="$BASELINE" -v current="$duration" 'BEGIN {printf "%.3f", base/current}') + fi + printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \ + "$jobs" "$round" "$rc" "$duration" "$pass" "$fail" "$skip" "$speedup" >>"$RESULT" + done +done + +{ + printf 'generated=%s\n' "$(date -Is)" + printf 'kernel=%s\n' "$(uname -r)" + printf 'cpu=%s\n' "$(getconf _NPROCESSORS_ONLN)" + free -h 2>/dev/null || true + df -h "$BASE_ROOTFS" 2>/dev/null || true + sha256sum "$PKG_LIST" "$DESKTOP_LIST" "$SERVICE_LIST" 2>/dev/null || true +} >"$OUTPUT_DIR/environment.txt" + +cat "$RESULT" diff --git a/testcase/command-execute-chroot.sh b/testcase/command-execute-chroot.sh new file mode 100755 index 0000000000000000000000000000000000000000..a343b17bc08067dc1aee117a15ac4aa85df6fd77 --- /dev/null +++ b/testcase/command-execute-chroot.sh @@ -0,0 +1,716 @@ +#!/usr/bin/env bash +############################################################################### +# @用例ID: IJV5MW +# @用例名称: command-execute-chroot +# @用例级别: 2 +# @用例类型: 功能测试 +# @自动化: 1 +############################################################################### + +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +WORKER_DEFAULT="$SCRIPT_DIR/lib/chroot-package-worker.sh" + +JOBS=0 +MAX_JOBS=64 +PACKAGE_TIMEOUT=1200 +STAGE_TIMEOUT=300 +COMMAND_TIMEOUT=5 +SERVICE_TIMEOUT=60 +BOOTSTRAP_TIMEOUT=1200 +ROOT_MODE=auto +SERVICE_MODE=auto +RELEASEVER=auto +REPO_DIR=/etc/yum.repos.d +WORK_PARENT=/var/tmp +OUTPUT_DIR="$PWD" +BASE_ROOTFS="" +KEEP_WORK=0 +VALIDATE_ONLY=0 +WORKER="${YUM_CI_CHROOT_WORKER:-$WORKER_DEFAULT}" + +WORK_ROOT="" +RUN_DIR="" +NORMALIZED_PACKAGES="" +NORMALIZED_DESKTOP="" +NORMALIZED_SERVICES="" +INTERRUPTED=0 +INFRA_FAILURE=0 +BASE_ARCH="" +REPO_FINGERPRINT="" +RUN_TOKEN="" +SESSION_DIR="" +declare -a CHILD_PIDS=() +declare -A CHILD_STARTS=() + +usage() { + cat <<'EOF' +Usage: + command-execute-chroot.sh [options] PKG_LIST DESKTOP_LIST SERVICE_LIST + +Required inputs: + PKG_LIST packages to test (first non-comment field per line) + DESKTOP_LIST packages whose command stage must be skipped + SERVICE_LIST packages whose service stage needs special configuration + +Options: + -j, --jobs N concurrent package workers, maximum 64 (default: min(CPU, 8)) + --package-timeout SEC full lifecycle timeout per package (default: 1200) + --stage-timeout SEC install/remove timeout (default: 300) + --command-timeout SEC timeout for each command probe (default: 5) + --service-timeout SEC service container/unit timeout (default: 60) + --bootstrap-timeout SEC base rootfs creation timeout (default: 1200) + --root-mode MODE auto, overlay, or copy (default: auto) + --service-mode MODE auto, nspawn, or skip (default: auto) + --releasever VERSION DNF releasever (default: host release) + --repo-dir DIR repository definitions (default: /etc/yum.repos.d) + --base-rootfs DIR create/reuse a persistent base rootfs + --work-dir DIR temporary work parent (default: /var/tmp) + --output-dir DIR reports and logs (default: current directory) + --keep-work retain temporary rootfs data for debugging + --validate-only parse inputs and print the plan without root access + -h, --help show this help + +Exit codes: 0 = PASS/SKIP only, 1 = package test failure, 2 = usage/infrastructure. +EOF +} + +log() { + printf '[%s] %s\n' "$(date '+%F %T')" "$*" >&2 +} + +die() { + log "ERROR: $*" + exit 2 +} + +is_positive_integer() { + [[ ${1:-} =~ ^[1-9][0-9]*$ ]] +} + +sanitize_field() { + printf '%s' "${1:-}" | tr '\t\r\n' ' ' | sed 's/[[:space:]][[:space:]]*/ /g; s/^ //; s/ $//' +} + +normalize_list() { + local input=$1 output=$2 kind=$3 line value line_no=0 + : >"$output" + while IFS= read -r line || [ -n "$line" ]; do + line_no=$((line_no + 1)) + line=${line%$'\r'} + line=${line%%#*} + line=${line#"${line%%[![:space:]]*}"} + value=${line%%[[:space:]]*} + [ -n "$value" ] || continue + if [[ ! $value =~ ^[A-Za-z0-9][A-Za-z0-9_.+~:@%-]*$ ]]; then + die "$kind contains an unsafe package token at line $line_no: $value" + fi + printf '%s\n' "$value" >>"$output" + done <"$input" +} + +tree_fingerprint() { + local root=$1 rel digest + if [ ! -d "$root" ]; then + printf 'absent\n' | sha256sum | awk '{print $1}' + return + fi + ( + cd "$root" || exit 1 + while IFS= read -r -d '' rel; do + if [ -L "$rel" ]; then + printf 'L\t%s\t%s\n' "$rel" "$(readlink -- "$rel")" + elif [ -f "$rel" ]; then + digest=$(sha256sum -- "$rel" | awk '{print $1}') || exit 1 + printf 'F\t%s\t%s\n' "$rel" "$digest" + elif [ -d "$rel" ]; then + printf 'D\t%s\n' "$rel" + else + printf 'O\t%s\n' "$rel" + fi + done < <(find . -mindepth 1 -printf '%P\0' | LC_ALL=C sort -z) + ) | sha256sum | awk '{print $1}' +} + +repo_fingerprint() { + local repo_hash dnf_vars_hash yum_vars_hash + repo_hash=$(tree_fingerprint "$REPO_DIR") || return 1 + dnf_vars_hash=$(tree_fingerprint /etc/dnf/vars) || return 1 + yum_vars_hash=$(tree_fingerprint /etc/yum/vars) || return 1 + printf 'repo=%s\ndnf_vars=%s\nyum_vars=%s\n' \ + "$repo_hash" "$dnf_vars_hash" "$yum_vars_hash" | + sha256sum | awk '{print $1}' +} + +detect_releasever() { + local file + for file in /etc/dnf/vars/releasever /etc/yum/vars/releasever; do + if [ -s "$file" ]; then + head -n 1 "$file" + return 0 + fi + done + # shellcheck disable=SC1091 + . /etc/os-release + printf '%s\n' "${VERSION_ID%%.*}" +} + +default_jobs() { + local cpu + cpu=$(getconf _NPROCESSORS_ONLN 2>/dev/null || printf '1') + is_positive_integer "$cpu" || cpu=1 + [ "$cpu" -le 8 ] || cpu=8 + printf '%s\n' "$cpu" +} + +require_command() { + command -v "$1" >/dev/null 2>&1 || die "required command is missing: $1" +} + +safe_remove_tree() { + local target=${1:-} + [ -n "$WORK_ROOT" ] || return 1 + case "$target" in + "$WORK_ROOT" | "$WORK_ROOT"/*) rm -rf --one-file-system -- "$target" ;; + *) log "refusing to remove path outside work root: $target"; return 1 ;; + esac +} + +session_matches() { + local pid=$1 expected_start=$2 identity sid start + [[ $pid =~ ^[1-9][0-9]*$ && $expected_start =~ ^[0-9]+$ ]] || return 1 + identity=$(awk '{print $6, $22}' "/proc/$pid/stat" 2>/dev/null) || return 1 + read -r sid start <<<"$identity" + [ "$sid" = "$pid" ] && [ "$start" = "$expected_start" ] +} + +process_matches() { + local pid=$1 expected_start=$2 start + [[ $pid =~ ^[1-9][0-9]*$ && $expected_start =~ ^[0-9]+$ ]] || return 1 + start=$(awk '{print $22}' "/proc/$pid/stat" 2>/dev/null) || return 1 + [ "$start" = "$expected_start" ] +} + +signal_worker_sessions() { + local signal=$1 file pid start + [ -n "$SESSION_DIR" ] && [ -d "$SESSION_DIR" ] || return 0 + for file in "$SESSION_DIR"/*.session; do + [ -f "$file" ] || continue + read -r pid start <"$file" || continue + if session_matches "$pid" "$start"; then + kill -"$signal" -- "-$pid" 2>/dev/null || true + else + rm -f -- "$file" + fi + done +} + +stop_children() { + local pid start + local -a wait_pids=() + signal_worker_sessions TERM + for pid in "${CHILD_PIDS[@]}"; do + start=${CHILD_STARTS[$pid]:-} + if process_matches "$pid" "$start"; then + wait_pids+=("$pid") + kill -TERM "$pid" 2>/dev/null || true + fi + done + sleep 1 + signal_worker_sessions KILL + for pid in "${wait_pids[@]}"; do + start=${CHILD_STARTS[$pid]} + if process_matches "$pid" "$start"; then + kill -KILL "$pid" 2>/dev/null || true + fi + done + for pid in "${wait_pids[@]}"; do + wait "$pid" 2>/dev/null || true + done + CHILD_PIDS=() + CHILD_STARTS=() +} + +on_signal() { + INTERRUPTED=1 + log "signal received; stopping package workers" + stop_children + exit 130 +} + +cleanup() { + local rc=$? + trap - EXIT INT TERM HUP + if [ "$INTERRUPTED" -eq 1 ]; then + stop_children + rc=2 + fi + if [ -n "$WORK_ROOT" ] && [ -d "$WORK_ROOT" ] && [ "$KEEP_WORK" -eq 0 ]; then + safe_remove_tree "$WORK_ROOT" || rc=2 + elif [ -n "$WORK_ROOT" ] && [ -d "$WORK_ROOT" ]; then + log "temporary work retained at $WORK_ROOT" + fi + exit "$rc" +} + +prepare_repo_config() { + local root=$1 + mkdir -p "$root/etc/yum.repos.d" "$root/etc/dnf/vars" "$root/etc/yum/vars" || return 1 + cp -a "$REPO_DIR"/. "$root/etc/yum.repos.d/" || return 1 + [ ! -d /etc/dnf/vars ] || cp -a /etc/dnf/vars/. "$root/etc/dnf/vars/" || return 1 + [ ! -d /etc/yum/vars ] || cp -a /etc/yum/vars/. "$root/etc/yum/vars/" || return 1 + printf '%s\n' "$RELEASEVER" >"$root/etc/dnf/vars/releasever" || return 1 + printf '%s\n' "$RELEASEVER" >"$root/etc/yum/vars/releasever" || return 1 +} + +metadata_value() { + local file=$1 key=$2 + awk -F= -v key="$key" '$1 == key { sub(/^[^=]*=/, ""); print; exit }' "$file" +} + +base_rootfs_matches() { + local root=$1 metadata + metadata="$root/.yum-ci-base-rootfs" + [ -f "$metadata" ] || return 1 + [ "$(metadata_value "$metadata" schema)" = 2 ] || return 1 + [ "$(metadata_value "$metadata" releasever)" = "$RELEASEVER" ] || return 1 + [ "$(metadata_value "$metadata" arch)" = "$BASE_ARCH" ] || return 1 + [ "$(metadata_value "$metadata" repo_fingerprint)" = "$REPO_FINGERPRINT" ] || return 1 + [ "$(head -n 1 "$root/etc/dnf/vars/releasever" 2>/dev/null)" = "$RELEASEVER" ] || return 1 + [ "$(head -n 1 "$root/etc/yum/vars/releasever" 2>/dev/null)" = "$RELEASEVER" ] || return 1 +} + +prepare_base_rootfs_unlocked() { + local root=$1 package_manager=$2 log_file="$RUN_DIR/base-rootfs-bootstrap.log" + if [ -x "$root/usr/bin/bash" ] && [ -x "$root/usr/bin/dnf" ]; then + if ! base_rootfs_matches "$root"; then + log "base rootfs metadata does not match release=$RELEASEVER arch=$BASE_ARCH or current repositories" + return 2 + fi + log "reusing base rootfs: $root" + return 0 + fi + if [ -d "$root" ] && [ -n "$(find "$root" -mindepth 1 -maxdepth 1 -print -quit 2>/dev/null)" ]; then + die "base rootfs is non-empty but incomplete: $root" + fi + mkdir -p "$root" || return 2 + prepare_repo_config "$root" || { log "failed to prepare repository configuration"; return 2; } + log "creating base rootfs for release $RELEASEVER" + if ! timeout -k 30 "$BOOTSTRAP_TIMEOUT" "$package_manager" -y \ + --installroot="$root" --releasever="$RELEASEVER" \ + --setopt=install_weak_deps=False --setopt=keepcache=False \ + install bash coreutils dnf rpm strace util-linux systemd grep findutils \ + file procps-ng shadow-utils >"$log_file" 2>&1; then + log "base rootfs creation failed; see $log_file" + return 1 + fi + mkdir -p "$root/etc" "$root/tmp" "$root/run" "$root/proc" "$root/sys" "$root/dev" || return 2 + chmod 1777 "$root/tmp" || return 2 + rm -f "$root/etc/resolv.conf" || return 2 + cp -L /etc/resolv.conf "$root/etc/resolv.conf" || return 2 + : >"$root/etc/machine-id" || return 2 + local metadata_tmp="$root/.yum-ci-base-rootfs.tmp.$$" + printf 'schema=2\nreleasever=%s\narch=%s\nrepo_fingerprint=%s\ncreated=%s\n' \ + "$RELEASEVER" "$BASE_ARCH" "$REPO_FINGERPRINT" "$(date -Is)" >"$metadata_tmp" || return 2 + mv -f "$metadata_tmp" "$root/.yum-ci-base-rootfs" || return 2 +} + +prepare_base_rootfs() { + local root=$1 package_manager=$2 lock_fd rc=0 + mkdir -p "$root" || return 2 + exec {lock_fd}<"$root" || { log "cannot open base rootfs directory for locking: $root"; return 2; } + if ! flock -w "$BOOTSTRAP_TIMEOUT" "$lock_fd"; then + log "timed out waiting for base rootfs lock: $root" + exec {lock_fd}>&- + return 2 + fi + prepare_base_rootfs_unlocked "$root" "$package_manager" || rc=$? + flock -u "$lock_fd" || rc=2 + exec {lock_fd}>&- + return "$rc" +} + +write_manifest() { + local manifest=$RUN_DIR/run-manifest.txt + local os_name + # shellcheck disable=SC1091 + . /etc/os-release + os_name=${PRETTY_NAME:-unknown} + { + printf 'issue=IJV5MW\n' + printf 'started=%s\n' "$(date -Is)" + printf 'os=%s\n' "$os_name" + printf 'kernel=%s\n' "$(uname -r)" + printf 'host_arch=%s\n' "$(uname -m)" + printf 'bash=%s\n' "$BASH_VERSION" + printf 'jobs=%s\n' "$JOBS" + printf 'root_mode=%s\n' "$ROOT_MODE" + printf 'service_mode=%s\n' "$SERVICE_MODE" + printf 'releasever=%s\n' "$RELEASEVER" + printf 'base_arch=%s\n' "$BASE_ARCH" + printf 'repo_fingerprint=%s\n' "$REPO_FINGERPRINT" + printf 'base_rootfs=%s\n' "$BASE_ROOTFS" + printf 'package_timeout=%s\n' "$PACKAGE_TIMEOUT" + printf 'stage_timeout=%s\n' "$STAGE_TIMEOUT" + printf 'command_timeout=%s\n' "$COMMAND_TIMEOUT" + printf 'service_timeout=%s\n' "$SERVICE_TIMEOUT" + printf 'package_count=%s\n' "$(wc -l <"$NORMALIZED_PACKAGES")" + command -v sha256sum >/dev/null 2>&1 && \ + printf 'package_list_sha256=%s\n' "$(sha256sum "$NORMALIZED_PACKAGES" | awk '{print $1}')" + } >"$manifest" +} + +synthesize_result() { + local index=$1 request=$2 reason=$3 result_file=$4 tmp + reason=$(sanitize_field "$reason") + tmp="$result_file.tmp.$$" + printf '%s\t%s\tunknown\t%s\tFAIL\t%s\tSKIP\tinstall failed\tSKIP\tinstall failed\tSKIP\tinstall failed\t0\t0\t0\tnone\tnone\n' \ + "$index" "$request" "$request" "$reason" >"$tmp" || return 1 + mv -f "$tmp" "$result_file" +} + +run_one() { + local index=$1 request=$2 package_dir result_file worker_root rc=0 + local session_file session_tmp session_pid session_start + package_dir="$RUN_DIR/packages/$(printf '%05d' "$index")-$(printf '%s' "$request" | tr -c 'A-Za-z0-9_.-' '_')" + result_file="$RUN_DIR/results/$(printf '%05d' "$index").tsv" + worker_root="$WORK_ROOT/roots/$(printf '%05d' "$index")" + mkdir -p "$package_dir" || return 2 + + session_file="$SESSION_DIR/$(printf '%05d' "$index").session" + session_tmp="$session_file.tmp.$BASHPID" + session_pid="" + trap 'if [ -n "${session_pid:-}" ]; then kill -TERM -- "-$session_pid" 2>/dev/null || true; sleep 1; kill -KILL -- "-$session_pid" 2>/dev/null || true; wait "$session_pid" 2>/dev/null || true; fi; rm -f -- "$session_tmp" "$session_file"; exit 143' INT TERM HUP + setsid timeout -k 20 "$PACKAGE_TIMEOUT" \ + unshare --mount --pid --fork --kill-child=KILL --mount-proc --propagation private \ + "$WORKER" \ + "$index" "$request" "$BASE_ROOTFS" "$WORK_ROOT" "$package_dir" "$result_file" \ + "$NORMALIZED_DESKTOP" "$NORMALIZED_SERVICES" "$ROOT_MODE" "$SERVICE_MODE" \ + "$STAGE_TIMEOUT" "$COMMAND_TIMEOUT" "$SERVICE_TIMEOUT" "$RUN_TOKEN" & + session_pid=$! + session_start=$(awk '{print $22}' "/proc/$session_pid/stat" 2>/dev/null || true) + if [[ ! $session_start =~ ^[0-9]+$ ]]; then + wait "$session_pid" || rc=$? + else + printf '%s %s\n' "$session_pid" "$session_start" >"$session_tmp" && \ + mv -f "$session_tmp" "$session_file" || rc=2 + if [ "$rc" -eq 0 ]; then + wait "$session_pid" || rc=$? + else + kill -KILL -- "-$session_pid" 2>/dev/null || true + wait "$session_pid" 2>/dev/null || true + fi + fi + trap - INT TERM HUP + rm -f -- "$session_tmp" "$session_file" + + if [ "$KEEP_WORK" -eq 0 ] && [ -d "$worker_root" ]; then + if ! safe_remove_tree "$worker_root"; then + log "failed to remove worker root after namespace teardown: $worker_root" + rc=2 + fi + fi + + case "$rc" in + 0 | 1) + if [ ! -s "$result_file" ]; then + synthesize_result "$index" "$request" \ + "worker exited with status $rc before writing a result" "$result_file" || return 2 + return 2 + fi + return 0 + ;; + 2) + if [ ! -s "$result_file" ] || ! awk -F '\t' \ + 'NF == 17 && ($5 == "FAIL" || $7 == "FAIL" || $9 == "FAIL" || $11 == "FAIL") { found=1 } END { exit !found }' \ + "$result_file"; then + synthesize_result "$index" "$request" \ + "worker infrastructure or cleanup failure" "$result_file" || return 2 + fi + return 2 + ;; + 124 | 137 | 143) + synthesize_result "$index" "$request" \ + "package lifecycle timed out after ${PACKAGE_TIMEOUT}s" "$result_file" || return 2 + return 0 + ;; + *) + [ -s "$result_file" ] || synthesize_result "$index" "$request" \ + "worker exited with unexpected status $rc" "$result_file" || return 2 + return 2 + ;; + esac +} + +valid_stage_status() { + case "$1" in PASS | FAIL | SKIP) return 0 ;; *) return 1 ;; esac +} + +valid_result_file() { + local file=$1 expected_index=$2 expected_request=$3 + local index request source binary install install_reason command command_reason + local service service_reason remove remove_reason duration command_count unit_count used_root used_service + [ "$(wc -l <"$file")" -eq 1 ] || return 1 + [ "$(awk -F '\t' 'NR == 1 { print NF }' "$file")" -eq 17 ] || return 1 + IFS=$'\t' read -r index request source binary install install_reason command command_reason \ + service service_reason remove remove_reason duration command_count unit_count used_root used_service <"$file" || return 1 + [ "$index" = "$expected_index" ] && [ "$request" = "$expected_request" ] || return 1 + [ -n "$source" ] && [ -n "$binary" ] || return 1 + valid_stage_status "$install" && valid_stage_status "$command" && \ + valid_stage_status "$service" && valid_stage_status "$remove" || return 1 + [[ $duration =~ ^[0-9]+$ && $command_count =~ ^[0-9]+$ && $unit_count =~ ^[0-9]+$ ]] || return 1 + case "$used_root" in none | overlay | copy) ;; *) return 1 ;; esac + case "$used_service" in none | nspawn | skip) ;; *) return 1 ;; esac +} + +append_category() { + local file=$1 request=$2 reason=$3 + printf '%s\t%s\n' "$request" "$(sanitize_field "$reason")" >>"$file" +} + +print_category() { + local report=$1 title=$2 file=$3 count + count=$(wc -l <"$file") + printf '【%s】(%s)\n' "$title" "$count" >>"$report" + if [ "$count" -eq 0 ]; then + printf '(无)\n' >>"$report" + else + awk -F '\t' '{printf "- %s 原因: %s\n", $1, $2}' "$file" >>"$report" + fi +} + +aggregate_results() { + local stamp=$1 + local report="$OUTPUT_DIR/package-command-result-chroot-$stamp.txt" + local detail="$OUTPUT_DIR/package-command-detail-chroot-$stamp.tsv" + local install_fail="$RUN_DIR/install-fail.tsv" + local remove_fail="$RUN_DIR/remove-fail.tsv" + local command_fail="$RUN_DIR/command-fail.tsv" + local service_fail="$RUN_DIR/service-fail.tsv" + local skipped="$RUN_DIR/skipped.tsv" + local total=0 pass=0 fail=0 skip=0 + local expected_request index request source binary install install_reason command command_reason + local service service_reason remove remove_reason _duration _command_count _unit_count _used_root _used_service result_file + + : >"$install_fail" && : >"$remove_fail" && : >"$command_fail" && \ + : >"$service_fail" && : >"$skipped" || return 2 + printf '# source-package\tbinary_package\tinstall\tcommand\tservice\tremove\n' >"$report" || return 2 + printf '# index\trequest\tsource-package\tbinary-package\tinstall\tinstall-reason\tcommand\tcommand-reason\tservice\tservice-reason\tremove\tremove-reason\tduration-ms\tcommand-count\tunit-count\troot-mode\tservice-mode\n' >"$detail" || return 2 + + while IFS= read -r expected_request || [ -n "$expected_request" ]; do + total=$((total + 1)) + result_file="$RUN_DIR/results/$(printf '%05d' "$total").tsv" + if [ ! -s "$result_file" ]; then + synthesize_result "$total" "$expected_request" "missing worker result" "$result_file" || return 2 + INFRA_FAILURE=1 + elif ! valid_result_file "$result_file" "$total" "$expected_request"; then + log "invalid worker result for index $total ($expected_request)" + synthesize_result "$total" "$expected_request" "malformed or mismatched worker result" "$result_file" || return 2 + INFRA_FAILURE=1 + fi + IFS=$'\t' read -r index request source binary install install_reason command command_reason \ + service service_reason remove remove_reason _duration _command_count _unit_count _used_root _used_service \ + <"$result_file" || return 2 + cat "$result_file" >>"$detail" || return 2 + printf '%s\t%s\t%s\t%s\t%s\t%s\n' \ + "$source" "$binary" "$install" "$command" "$service" "$remove" >>"$report" || return 2 + + [ "$install" != FAIL ] || append_category "$install_fail" "$request" "$install_reason" || return 2 + [ "$command" != FAIL ] || append_category "$command_fail" "$request" "$command_reason" || return 2 + [ "$service" != FAIL ] || append_category "$service_fail" "$request" "$service_reason" || return 2 + [ "$remove" != FAIL ] || append_category "$remove_fail" "$request" "$remove_reason" || return 2 + + if [ "$install" = FAIL ] || [ "$command" = FAIL ] || [ "$service" = FAIL ] || [ "$remove" = FAIL ]; then + fail=$((fail + 1)) + elif [ "$install" = SKIP ] || [ "$command" = SKIP ] || [ "$service" = SKIP ] || [ "$remove" = SKIP ]; then + skip=$((skip + 1)) + append_category "$skipped" "$request" \ + "install=$install($install_reason); command=$command($command_reason); service=$service($service_reason); remove=$remove($remove_reason)" || return 2 + else + pass=$((pass + 1)) + fi + done <"$NORMALIZED_PACKAGES" + + { + printf '\n========== 测试结果汇总 (%s) ==========\n' "$(date '+%F %T')" + printf '环境: chroot | 并发度: %s | 待测包总数: %s | 通过: %s | 失败: %s | 跳过: %s\n' \ + "$JOBS" "$total" "$pass" "$fail" "$skip" + } >>"$report" || return 2 + print_category "$report" "安装失败" "$install_fail" || return 2 + print_category "$report" "卸载失败" "$remove_fail" || return 2 + print_category "$report" "命令测试失败" "$command_fail" || return 2 + print_category "$report" "服务测试失败" "$service_fail" || return 2 + print_category "$report" "跳过项" "$skipped" || return 2 + printf '对账: %s + %s + %s = %s\n' "$pass" "$fail" "$skip" "$total" >>"$report" || return 2 + printf '=======================================================\n' >>"$report" || return 2 + + cat "$report" || return 2 + log "main report: $report" + log "detail report: $detail" + [ "$fail" -eq 0 ] || return 1 + return 0 +} + +parse_args() { + while [ "$#" -gt 0 ]; do + case "$1" in + -j | --jobs) [ "$#" -ge 2 ] || die "$1 requires a value"; JOBS=$2; shift 2 ;; + --package-timeout) [ "$#" -ge 2 ] || die "$1 requires a value"; PACKAGE_TIMEOUT=$2; shift 2 ;; + --stage-timeout) [ "$#" -ge 2 ] || die "$1 requires a value"; STAGE_TIMEOUT=$2; shift 2 ;; + --command-timeout) [ "$#" -ge 2 ] || die "$1 requires a value"; COMMAND_TIMEOUT=$2; shift 2 ;; + --service-timeout) [ "$#" -ge 2 ] || die "$1 requires a value"; SERVICE_TIMEOUT=$2; shift 2 ;; + --bootstrap-timeout) [ "$#" -ge 2 ] || die "$1 requires a value"; BOOTSTRAP_TIMEOUT=$2; shift 2 ;; + --root-mode) [ "$#" -ge 2 ] || die "$1 requires a value"; ROOT_MODE=$2; shift 2 ;; + --service-mode) [ "$#" -ge 2 ] || die "$1 requires a value"; SERVICE_MODE=$2; shift 2 ;; + --releasever) [ "$#" -ge 2 ] || die "$1 requires a value"; RELEASEVER=$2; shift 2 ;; + --repo-dir) [ "$#" -ge 2 ] || die "$1 requires a value"; REPO_DIR=$2; shift 2 ;; + --base-rootfs) [ "$#" -ge 2 ] || die "$1 requires a value"; BASE_ROOTFS=$2; shift 2 ;; + --work-dir) [ "$#" -ge 2 ] || die "$1 requires a value"; WORK_PARENT=$2; shift 2 ;; + --output-dir) [ "$#" -ge 2 ] || die "$1 requires a value"; OUTPUT_DIR=$2; shift 2 ;; + --keep-work) KEEP_WORK=1; shift ;; + --validate-only) VALIDATE_ONLY=1; shift ;; + -h | --help) usage; exit 0 ;; + --) shift; break ;; + -*) die "unknown option: $1" ;; + *) break ;; + esac + done + [ "$#" -eq 3 ] || { usage >&2; die "exactly three input files are required"; } + PKG_INPUT=$1 + DESKTOP_INPUT=$2 + SERVICE_INPUT=$3 +} + +main() { + local stamp package_manager token_fifo index=0 request pid child_start rc=0 aggregate_rc package_count worker_rc + parse_args "$@" + + if [ "$JOBS" = 0 ]; then + JOBS=$(default_jobs) + fi + for value in "$JOBS" "$PACKAGE_TIMEOUT" "$STAGE_TIMEOUT" "$COMMAND_TIMEOUT" "$SERVICE_TIMEOUT" "$BOOTSTRAP_TIMEOUT"; do + is_positive_integer "$value" || die "numeric options must be positive integers" + done + [ "$JOBS" -le "$MAX_JOBS" ] || die "jobs must not exceed $MAX_JOBS" + case "$ROOT_MODE" in auto | overlay | copy) ;; *) die "root mode must be auto, overlay, or copy" ;; esac + case "$SERVICE_MODE" in auto | nspawn | skip) ;; *) die "service mode must be auto, nspawn, or skip" ;; esac + [ -f "$PKG_INPUT" ] || die "package list not found: $PKG_INPUT" + [ -f "$DESKTOP_INPUT" ] || die "desktop list not found: $DESKTOP_INPUT" + [ -f "$SERVICE_INPUT" ] || die "service list not found: $SERVICE_INPUT" + + WORK_ROOT=$(mktemp -d "${TMPDIR:-/tmp}/yum-ci-validate.XXXXXX") || die "cannot create validation directory" + trap cleanup EXIT + trap on_signal INT TERM HUP + NORMALIZED_PACKAGES="$WORK_ROOT/packages.list" + NORMALIZED_DESKTOP="$WORK_ROOT/desktop.list" + NORMALIZED_SERVICES="$WORK_ROOT/services.list" + normalize_list "$PKG_INPUT" "$NORMALIZED_PACKAGES" "package list" + normalize_list "$DESKTOP_INPUT" "$NORMALIZED_DESKTOP" "desktop list" + normalize_list "$SERVICE_INPUT" "$NORMALIZED_SERVICES" "service list" + [ -s "$NORMALIZED_PACKAGES" ] || die "package list is empty after normalization" + package_count=$(wc -l <"$NORMALIZED_PACKAGES") + if [ "$JOBS" -gt "$package_count" ]; then + JOBS=$package_count + fi + + if [ "$VALIDATE_ONLY" -eq 1 ]; then + printf 'packages=%s jobs=%s root_mode=%s service_mode=%s\n' \ + "$(wc -l <"$NORMALIZED_PACKAGES")" "$JOBS" "$ROOT_MODE" "$SERVICE_MODE" + cat "$NORMALIZED_PACKAGES" + return 0 + fi + + [ "$(id -u)" -eq 0 ] || die "this test must run as root" + for command_name in realpath timeout unshare setsid mount umount mountpoint chroot awk sed grep \ + find cp sort readlink sha256sum flock rpm; do + require_command "$command_name" + done + unshare --help 2>&1 | grep -q -- '--kill-child' || \ + die "unshare does not support --kill-child; util-linux is too old" + [ -x "$WORKER" ] || die "worker is not executable: $WORKER" + [ -d "$REPO_DIR" ] || die "repo directory not found: $REPO_DIR" + mkdir -p "$WORK_PARENT" "$OUTPUT_DIR" || die "cannot create work/output directory" + OUTPUT_DIR=$(realpath "$OUTPUT_DIR") || die "cannot resolve output directory" + WORK_PARENT=$(realpath "$WORK_PARENT") || die "cannot resolve work directory" + + # Replace the validation directory with a work directory on the requested filesystem. + safe_remove_tree "$WORK_ROOT" || true + WORK_ROOT=$(mktemp -d "$WORK_PARENT/yum-ci-chroot.XXXXXX") || die "cannot create work directory" + NORMALIZED_PACKAGES="$WORK_ROOT/packages.list" + NORMALIZED_DESKTOP="$WORK_ROOT/desktop.list" + NORMALIZED_SERVICES="$WORK_ROOT/services.list" + normalize_list "$PKG_INPUT" "$NORMALIZED_PACKAGES" "package list" + normalize_list "$DESKTOP_INPUT" "$NORMALIZED_DESKTOP" "desktop list" + normalize_list "$SERVICE_INPUT" "$NORMALIZED_SERVICES" "service list" + [ -s "$NORMALIZED_PACKAGES" ] || die "package list became empty during preparation" + package_count=$(wc -l <"$NORMALIZED_PACKAGES") + if [ "$JOBS" -gt "$package_count" ]; then + JOBS=$package_count + fi + + RELEASEVER=$([ "$RELEASEVER" = auto ] && detect_releasever || printf '%s' "$RELEASEVER") + [[ $RELEASEVER =~ ^[A-Za-z0-9._-]+$ ]] || die "releasever contains unsafe characters" + BASE_ARCH=$(uname -m) + REPO_FINGERPRINT=$(repo_fingerprint) || die "cannot fingerprint repository configuration" + stamp="$(date '+%Y%m%d-%H%M%S')-$$-${RANDOM:-0}" + RUN_TOKEN=$stamp + RUN_DIR="$OUTPUT_DIR/yum-ci-chroot-$stamp" + SESSION_DIR="$WORK_ROOT/sessions" + mkdir -p "$RUN_DIR/packages" "$RUN_DIR/results" "$WORK_ROOT/roots" "$SESSION_DIR" || \ + die "cannot create run directories" + + if [ -n "$BASE_ROOTFS" ]; then + mkdir -p "$BASE_ROOTFS" || die "cannot create base rootfs directory" + BASE_ROOTFS=$(realpath "$BASE_ROOTFS") || die "cannot resolve base rootfs path" + else + BASE_ROOTFS="$WORK_ROOT/base-rootfs" + fi + package_manager=$(command -v dnf || command -v yum || true) + [ -n "$package_manager" ] || die "dnf/yum is required" + prepare_base_rootfs "$BASE_ROOTFS" "$package_manager" || die "base rootfs preparation failed" + write_manifest || die "cannot write run manifest" + + token_fifo="$WORK_ROOT/tokens" + mkfifo "$token_fifo" || die "cannot create scheduler FIFO" + exec 9<>"$token_fifo" || die "cannot open scheduler FIFO" + rm -f "$token_fifo" || die "cannot unlink scheduler FIFO" + for ((index = 0; index < JOBS; index++)); do printf '.' >&9; done + + index=0 + while IFS= read -r request || [ -n "$request" ]; do + index=$((index + 1)) + IFS= read -r -n 1 -u 9 + # Defer signal cleanup across spawn/identity registration so every + # asynchronous worker is either fully tracked or already exited. + trap 'INTERRUPTED=1' INT TERM HUP + ( + worker_rc=0 + run_one "$index" "$request" || worker_rc=$? + printf '.' >&9 + exit "$worker_rc" + ) & + pid=$! + CHILD_PIDS+=("$pid") + child_start=$(awk '{print $22}' "/proc/$pid/stat" 2>/dev/null || true) + [[ $child_start =~ ^[0-9]+$ ]] && CHILD_STARTS["$pid"]=$child_start + trap on_signal INT TERM HUP + [ "$INTERRUPTED" -eq 0 ] || on_signal + done <"$NORMALIZED_PACKAGES" + + for pid in "${CHILD_PIDS[@]}"; do + wait "$pid" || INFRA_FAILURE=1 + done + CHILD_PIDS=() + CHILD_STARTS=() + exec 9>&- + + aggregate_rc=0 + aggregate_results "$stamp" || aggregate_rc=$? + case "$aggregate_rc" in 0) ;; 1) rc=1 ;; *) rc=2 ;; esac + [ "$INFRA_FAILURE" -eq 0 ] || rc=2 + printf 'finished=%s\nexit_code=%s\n' "$(date -Is)" "$rc" >>"$RUN_DIR/run-manifest.txt" || rc=2 + return "$rc" +} + +main "$@" diff --git a/testcase/lib/chroot-package-worker.sh b/testcase/lib/chroot-package-worker.sh new file mode 100755 index 0000000000000000000000000000000000000000..00aa26dcc438474b973e1de30ce1e632aba689b6 --- /dev/null +++ b/testcase/lib/chroot-package-worker.sh @@ -0,0 +1,661 @@ +#!/usr/bin/env bash + +set -uo pipefail + +if [ "$#" -ne 14 ]; then + printf 'worker expects 14 arguments, got %s\n' "$#" >&2 + exit 2 +fi + +INDEX=$1 +REQUEST=$2 +BASE_ROOTFS=$3 +WORK_ROOT=$4 +PACKAGE_DIR=$5 +RESULT_FILE=$6 +DESKTOP_LIST=$7 +SPECIAL_SERVICE_LIST=$8 +ROOT_MODE_REQUESTED=$9 +SERVICE_MODE_REQUESTED=${10} +STAGE_TIMEOUT=${11} +COMMAND_TIMEOUT=${12} +SERVICE_TIMEOUT=${13} +RUN_TOKEN=${14} + +[[ $RUN_TOKEN =~ ^[A-Za-z0-9][A-Za-z0-9-]*$ ]] || { + printf 'worker run token contains unsafe characters\n' >&2 + exit 2 +} + +ROOT_DIR="$WORK_ROOT/roots/$(printf '%05d' "$INDEX")" +ROOT="$ROOT_DIR/root" +UPPER="$ROOT_DIR/upper" +OVERLAY_WORK="$ROOT_DIR/overlay-work" +START_MS=$(date +%s%3N) + +SOURCE_PACKAGE=unknown +BINARY_PACKAGE=$REQUEST +PACKAGE_NAME=$REQUEST +INSTALL_STATUS=SKIP +INSTALL_REASON="not run" +COMMAND_STATUS=SKIP +COMMAND_REASON="install not completed" +SERVICE_STATUS=SKIP +SERVICE_REASON="install not completed" +REMOVE_STATUS=SKIP +REMOVE_REASON="install not completed" +COMMAND_COUNT=0 +UNIT_COUNT=0 +ROOT_MODE_USED=none +SERVICE_MODE_USED=none +PREINSTALLED=0 +RESULT_WRITTEN=0 +OVERLAY_MOUNTED=0 +RUNTIME_MOUNTED=0 +DEV_MOUNTED=0 +PROC_MOUNTED=0 +SYS_MOUNTED=0 +RUN_MOUNTED=0 +INFRA_FAILURE=0 +MACHINE_NAME="" +NSPAWN_PID="" + +log() { + printf '[%s] %s\n' "$(date '+%F %T')" "$*" >>"$PACKAGE_DIR/worker.log" +} + +sanitize() { + printf '%s' "${1:-}" | tr '\t\r\n' ' ' | sed 's/[[:space:]][[:space:]]*/ /g; s/^ //; s/ $//' +} + +elapsed_ms() { + local now + now=$(date +%s%3N) + printf '%s\n' "$((now - START_MS))" +} + +result_write() { + local duration tmp + [ "$RESULT_WRITTEN" -eq 0 ] || return 0 + duration=$(elapsed_ms) + tmp="$RESULT_FILE.tmp.$$" + printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \ + "$INDEX" "$(sanitize "$REQUEST")" "$(sanitize "$SOURCE_PACKAGE")" "$(sanitize "$BINARY_PACKAGE")" \ + "$INSTALL_STATUS" "$(sanitize "$INSTALL_REASON")" \ + "$COMMAND_STATUS" "$(sanitize "$COMMAND_REASON")" \ + "$SERVICE_STATUS" "$(sanitize "$SERVICE_REASON")" \ + "$REMOVE_STATUS" "$(sanitize "$REMOVE_REASON")" \ + "$duration" "$COMMAND_COUNT" "$UNIT_COUNT" "$ROOT_MODE_USED" "$SERVICE_MODE_USED" >"$tmp" || return 1 + if ! mv -f "$tmp" "$RESULT_FILE"; then + rm -f -- "$tmp" + return 1 + fi + RESULT_WRITTEN=1 +} + +nspawn_running() { + local state + [ -n "$NSPAWN_PID" ] || return 1 + kill -0 "$NSPAWN_PID" 2>/dev/null || return 1 + state=$(awk '{print $3}' "/proc/$NSPAWN_PID/stat" 2>/dev/null || true) + [ -n "$state" ] || return 1 + [ "$state" != Z ] +} + +stop_nspawn() { + local rc=0 running=0 deadline remaining + [ -n "$MACHINE_NAME" ] || return 0 + deadline=$((SECONDS + SERVICE_TIMEOUT)) + timeout -k 2 "$SERVICE_TIMEOUT" machinectl terminate "$MACHINE_NAME" \ + >>"$PACKAGE_DIR/service.log" 2>&1 || rc=1 + if [ -n "$NSPAWN_PID" ]; then + if nspawn_running; then running=1; else running=0; fi + while [ "$running" -eq 1 ] && [ "$SECONDS" -lt "$deadline" ]; do + remaining=$((deadline - SECONDS)) + if [ "$remaining" -gt 1 ]; then sleep 1; else break; fi + nspawn_running || running=0 + done + if [ "$running" -eq 1 ]; then + kill -TERM "$NSPAWN_PID" 2>/dev/null || true + sleep 1 + if nspawn_running; then + kill -KILL "$NSPAWN_PID" 2>/dev/null || true + fi + fi + if nspawn_running; then + rc=1 + else + wait "$NSPAWN_PID" 2>/dev/null || true + rc=0 + fi + fi + if [ "$rc" -eq 0 ]; then + MACHINE_NAME="" + NSPAWN_PID="" + fi + return "$rc" +} + +unmount_one() { + local target=$1 + mountpoint -q "$target" 2>/dev/null || return 0 + umount -R "$target" >/dev/null 2>&1 || umount -R -l "$target" >/dev/null 2>&1 || return 1 + ! mountpoint -q "$target" 2>/dev/null +} + +unmount_runtime() { + local rc=0 + if [ "$RUN_MOUNTED" -eq 1 ]; then + if unmount_one "$ROOT/run"; then RUN_MOUNTED=0; else rc=1; fi + fi + if [ "$SYS_MOUNTED" -eq 1 ]; then + if unmount_one "$ROOT/sys"; then SYS_MOUNTED=0; else rc=1; fi + fi + if [ "$PROC_MOUNTED" -eq 1 ]; then + if unmount_one "$ROOT/proc"; then PROC_MOUNTED=0; else rc=1; fi + fi + if [ "$DEV_MOUNTED" -eq 1 ]; then + if unmount_one "$ROOT/dev"; then DEV_MOUNTED=0; else rc=1; fi + fi + if [ "$RUN_MOUNTED" -eq 0 ] && [ "$SYS_MOUNTED" -eq 0 ] && \ + [ "$PROC_MOUNTED" -eq 0 ] && [ "$DEV_MOUNTED" -eq 0 ]; then + RUNTIME_MOUNTED=0 + fi + return "$rc" +} + +runtime_mounts_present() { + mountpoint -q "$ROOT/run" 2>/dev/null || mountpoint -q "$ROOT/sys" 2>/dev/null || \ + mountpoint -q "$ROOT/proc" 2>/dev/null || mountpoint -q "$ROOT/dev" 2>/dev/null +} + +cleanup() { + local rc=$? cleanup_failed=0 + trap - EXIT INT TERM HUP + stop_nspawn || cleanup_failed=1 + unmount_runtime || cleanup_failed=1 + if [ "$OVERLAY_MOUNTED" -eq 1 ]; then + if unmount_one "$ROOT"; then OVERLAY_MOUNTED=0; else cleanup_failed=1; fi + fi + if [ "$RESULT_WRITTEN" -eq 0 ]; then + INSTALL_STATUS=FAIL + INSTALL_REASON="worker terminated unexpectedly with status $rc" + COMMAND_STATUS=SKIP + COMMAND_REASON="worker terminated" + SERVICE_STATUS=SKIP + SERVICE_REASON="worker terminated" + REMOVE_STATUS=SKIP + REMOVE_REASON="worker terminated" + result_write || cleanup_failed=1 + fi + log "root deletion is deferred until the mount/PID namespace has exited: $ROOT_DIR" + [ "$cleanup_failed" -eq 0 ] || rc=2 + exit "$rc" +} + +trap cleanup EXIT INT TERM HUP + +list_matches() { + local file=$1 item candidate + shift + [ -s "$file" ] || return 1 + while IFS= read -r item || [ -n "$item" ]; do + for candidate in "$@"; do + [ -n "$candidate" ] && [ "$item" = "$candidate" ] && return 0 + done + done <"$file" + return 1 +} + +first_error() { + local file=$1 fallback=$2 line + line=$(grep -Eim1 'error:|failed|failure|conflict|nothing to do|no match|timed out|cannot|not found' "$file" 2>/dev/null || true) + [ -n "$line" ] && sanitize "$line" || printf '%s' "$fallback" +} + +run_with_timeout() { + local timeout_value=$1 log_file=$2 + shift 2 + timeout -k 10 "$timeout_value" "$@" >"$log_file" 2>&1 +} + +prepare_root() { + mkdir -p "$ROOT_DIR" "$ROOT" "$UPPER" "$OVERLAY_WORK" + if [ "$ROOT_MODE_REQUESTED" != copy ]; then + if mount -t overlay overlay \ + -o "lowerdir=$BASE_ROOTFS,upperdir=$UPPER,workdir=$OVERLAY_WORK" "$ROOT" \ + >>"$PACKAGE_DIR/rootfs.log" 2>&1; then + OVERLAY_MOUNTED=1 + ROOT_MODE_USED=overlay + return 0 + fi + if [ "$ROOT_MODE_REQUESTED" = overlay ]; then + return 1 + fi + log "overlay unavailable; falling back to copy mode" + fi + ROOT_MODE_USED=copy + cp -a --reflink=auto "$BASE_ROOTFS"/. "$ROOT"/ >>"$PACKAGE_DIR/rootfs.log" 2>&1 +} + +mount_runtime() { + if runtime_mounts_present; then + unmount_runtime || return 1 + fi + mkdir -p "$ROOT/dev" "$ROOT/proc" "$ROOT/sys" "$ROOT/run" "$ROOT/tmp" || return 1 + chmod 1777 "$ROOT/tmp" || return 1 + mount --rbind /dev "$ROOT/dev" || return 1 + DEV_MOUNTED=1 + mount --make-rslave "$ROOT/dev" || return 1 + mount -t proc proc "$ROOT/proc" || return 1 + PROC_MOUNTED=1 + mount --rbind /sys "$ROOT/sys" || return 1 + SYS_MOUNTED=1 + mount --make-rslave "$ROOT/sys" || return 1 + mount -t tmpfs -o mode=0755,nosuid,nodev tmpfs "$ROOT/run" || return 1 + RUN_MOUNTED=1 + rm -f "$ROOT/etc/resolv.conf" || return 1 + cp -L /etc/resolv.conf "$ROOT/etc/resolv.conf" || return 1 + RUNTIME_MOUNTED=1 +} + +query_installed_package() { + local query + query=$(chroot "$ROOT" rpm -q \ + --qf '%{SOURCERPM}\t%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}\t%{NAME}\n' \ + "$REQUEST" 2>/dev/null | head -n 1) || return 1 + [ -n "$query" ] || return 1 + IFS=$'\t' read -r SOURCE_PACKAGE BINARY_PACKAGE PACKAGE_NAME <<<"$query" + SOURCE_PACKAGE=${SOURCE_PACKAGE%.src.rpm} +} + +install_package() { + local verb=install rc=0 + if chroot "$ROOT" rpm -q "$REQUEST" >/dev/null 2>&1; then + PREINSTALLED=1 + verb=reinstall + fi + log "install stage: dnf $verb $REQUEST" + run_with_timeout "$STAGE_TIMEOUT" "$PACKAGE_DIR/install.log" \ + chroot "$ROOT" dnf -y --setopt=install_weak_deps=False "$verb" "$REQUEST" || rc=$? + if [ "$rc" -ne 0 ]; then + INSTALL_STATUS=FAIL + if [ "$rc" -eq 124 ]; then + INSTALL_REASON="installation timed out after ${STAGE_TIMEOUT}s" + else + INSTALL_REASON=$(first_error "$PACKAGE_DIR/install.log" "dnf $verb exited with status $rc") + fi + return 1 + fi + if grep -Eqi 'scriptlet failed|cpio:.*failed|No such file or directory' "$PACKAGE_DIR/install.log"; then + INSTALL_STATUS=FAIL + INSTALL_REASON=$(first_error "$PACKAGE_DIR/install.log" "install log contains a scriptlet/file error") + return 1 + fi + if ! query_installed_package; then + INSTALL_STATUS=FAIL + INSTALL_REASON="package manager succeeded but RPM query did not find $REQUEST" + return 1 + fi + INSTALL_STATUS=PASS + INSTALL_REASON="installed as $BINARY_PACKAGE" + return 0 +} + +probe_command() { + local command_path=$1 trace_inside=$2 arg=$3 attempt_log=$4 rc=0 + : >"$ROOT$trace_inside" + if [ -n "$arg" ]; then + timeout -k 2 "$COMMAND_TIMEOUT" chroot "$ROOT" \ + strace -f -qq -e trace=execve -o "$trace_inside" "$command_path" "$arg" \ + "$attempt_log" 2>&1 || rc=$? + else + timeout -k 2 "$COMMAND_TIMEOUT" chroot "$ROOT" \ + strace -f -qq -e trace=execve -o "$trace_inside" "$command_path" \ + "$attempt_log" 2>&1 || rc=$? + fi + cat "$attempt_log" >>"$PACKAGE_DIR/command.log" + if [ "$rc" -eq 124 ] || [ "$rc" -eq 137 ]; then + return 2 + fi + if [ "$rc" -eq 126 ] || [ "$rc" -eq 127 ] || \ + grep -Eqi 'error while loading shared libraries|cannot open shared object file' "$attempt_log"; then + return 4 + fi + if ! grep -Eq 'execve\(.*\)[[:space:]]*=[[:space:]]*0' "$ROOT$trace_inside"; then + return 1 + fi + if grep -Eq '\+\+\+ killed by (SIGSEGV|SIGABRT|SIGBUS|SIGILL)' "$ROOT$trace_inside"; then + return 3 + fi + [ "$rc" -eq 0 ] && return 0 + grep -Eqi 'usage:|invalid (argument|option)|unknown (argument|option)|unrecognized option|try .*--help' \ + "$attempt_log" +} + +capture_package_files() { + chroot "$ROOT" rpm -ql "$PACKAGE_NAME" >"$PACKAGE_DIR/rpm-files.list" \ + 2>"$PACKAGE_DIR/rpm-files.log" +} + +test_commands() { + local files="$PACKAGE_DIR/rpm-files.list" commands="$PACKAGE_DIR/commands.list" + local path type arg trace attempt_log rc failed=0 timeouts=0 crashes=0 load_errors=0 + local command_index=0 attempt_index + : >"$commands" + while IFS= read -r path || [ -n "$path" ]; do + case "$path" in /usr/bin/* | /usr/sbin/*) ;; *) continue ;; esac + chroot "$ROOT" test -x "$path" >/dev/null 2>&1 || continue + type=$(chroot "$ROOT" file -Lb "$path" 2>/dev/null || true) + [[ $type == *ELF* ]] || continue + printf '%s\n' "$path" >>"$commands" + done <"$files" + + COMMAND_COUNT=$(wc -l <"$commands") + if [ "$COMMAND_COUNT" -eq 0 ]; then + COMMAND_STATUS=PASS + COMMAND_REASON="package contains no executable ELF command under /usr/bin or /usr/sbin" + return 0 + fi + + while IFS= read -r path || [ -n "$path" ]; do + command_index=$((command_index + 1)) + printf '\n===== %s =====\n' "$path" >>"$PACKAGE_DIR/command.log" + rc=1 + attempt_index=0 + for arg in --help --version ""; do + attempt_index=$((attempt_index + 1)) + trace="/tmp/yum-ci-command-${INDEX}-${command_index}-${attempt_index}.trace" + attempt_log="$PACKAGE_DIR/command-${command_index}-${attempt_index}.log" + probe_command "$path" "$trace" "$arg" "$attempt_log" && { rc=0; break; } + rc=$? + [ "$rc" -ne 2 ] || timeouts=$((timeouts + 1)) + [ "$rc" -ne 3 ] || crashes=$((crashes + 1)) + [ "$rc" -ne 4 ] || load_errors=$((load_errors + 1)) + case "$rc" in 3 | 4) break ;; esac + done + if [ "$rc" -ne 0 ]; then + failed=$((failed + 1)) + printf '%s\trc=%s\n' "$path" "$rc" >>"$PACKAGE_DIR/command-failures.tsv" + fi + done <"$commands" + + if [ "$failed" -gt 0 ]; then + COMMAND_STATUS=FAIL + COMMAND_REASON="$failed/$COMMAND_COUNT commands failed executable smoke tests (timeouts=$timeouts crashes=$crashes load-errors=$load_errors)" + return 1 + fi + COMMAND_STATUS=PASS + COMMAND_REASON="all $COMMAND_COUNT executable ELF commands reached execve successfully" +} + +discover_units() { + local units=$1 path + [ -r "$PACKAGE_DIR/rpm-files.list" ] || return 1 + : >"$units" || return 1 + while IFS= read -r path || [ -n "$path" ]; do + case "$path" in + /usr/lib/systemd/system/*.service | /usr/lib/systemd/system/*.socket | /usr/lib/systemd/system/*.target | \ + /lib/systemd/system/*.service | /lib/systemd/system/*.socket | /lib/systemd/system/*.target) + basename "$path" ;; + esac + done <"$PACKAGE_DIR/rpm-files.list" | sort -u >"$units" || return 1 + UNIT_COUNT=$(wc -l <"$units") || return 1 +} + +nspawn_available() { + command -v systemd-nspawn >/dev/null 2>&1 && + command -v machinectl >/dev/null 2>&1 && + [ -S /run/systemd/private ] +} + +machine_name_for() { + local prefix=yumci- suffix="-${INDEX}" token_length + token_length=$((64 - ${#prefix} - ${#suffix})) + [ "$token_length" -gt 0 ] || return 1 + printf '%s%s%s\n' "$prefix" "${RUN_TOKEN:0:token_length}" "$suffix" +} + +start_nspawn() { + local deadline remaining + MACHINE_NAME=$(machine_name_for) + : >"$ROOT/etc/machine-id" + systemd-nspawn --quiet --boot --directory="$ROOT" --machine="$MACHINE_NAME" \ + >"$PACKAGE_DIR/nspawn.log" 2>&1 & + NSPAWN_PID=$! + deadline=$((SECONDS + SERVICE_TIMEOUT)) + while [ "$SECONDS" -lt "$deadline" ]; do + kill -0 "$NSPAWN_PID" 2>/dev/null || return 1 + remaining=$((deadline - SECONDS)) + if timeout -k 2 "$remaining" systemctl --no-pager --machine="$MACHINE_NAME" list-units \ + >>"$PACKAGE_DIR/nspawn.log" 2>&1; then + return 0 + fi + sleep 1 + done + return 1 +} + +test_services_nspawn() { + local units=$1 unit rc=0 tested=0 skipped_templates=0 failed=0 + SERVICE_MODE_USED=nspawn + if ! unmount_runtime; then + SERVICE_STATUS=FAIL + SERVICE_REASON="failed to unmount chroot runtime filesystems before nspawn" + INFRA_FAILURE=1 + return 2 + fi + if ! start_nspawn; then + stop_nspawn || INFRA_FAILURE=1 + if [ "$SERVICE_MODE_REQUESTED" = nspawn ]; then + SERVICE_STATUS=FAIL + SERVICE_REASON="systemd-nspawn failed to boot the chroot rootfs" + INFRA_FAILURE=1 + return 2 + fi + SERVICE_STATUS=SKIP + SERVICE_REASON="systemd-nspawn is present but could not boot the chroot rootfs" + return 0 + fi + + while IFS= read -r unit || [ -n "$unit" ]; do + if [[ $unit == *@.* ]]; then + skipped_templates=$((skipped_templates + 1)) + printf '%s\tSKIP\ttemplate unit requires an instance/configuration\n' "$unit" >>"$PACKAGE_DIR/service-results.tsv" + continue + fi + tested=$((tested + 1)) + printf '\n===== %s =====\n' "$unit" >>"$PACKAGE_DIR/service.log" + rc=0 + timeout -k 5 "$SERVICE_TIMEOUT" systemctl --no-pager --machine="$MACHINE_NAME" start "$unit" \ + >>"$PACKAGE_DIR/service.log" 2>&1 || rc=$? + if [ "$rc" -eq 0 ]; then + timeout -k 5 "$SERVICE_TIMEOUT" systemctl --no-pager --machine="$MACHINE_NAME" is-active "$unit" \ + >>"$PACKAGE_DIR/service.log" 2>&1 || rc=$? + fi + if [ "$rc" -eq 0 ]; then + timeout -k 5 "$SERVICE_TIMEOUT" systemctl --no-pager --machine="$MACHINE_NAME" stop "$unit" \ + >>"$PACKAGE_DIR/service.log" 2>&1 || rc=$? + else + timeout -k 5 "$SERVICE_TIMEOUT" systemctl --no-pager --machine="$MACHINE_NAME" status "$unit" \ + >>"$PACKAGE_DIR/service.log" 2>&1 || true + timeout -k 5 "$SERVICE_TIMEOUT" systemctl --no-pager --machine="$MACHINE_NAME" stop "$unit" \ + >>"$PACKAGE_DIR/service.log" 2>&1 || true + fi + if [ "$rc" -eq 0 ]; then + printf '%s\tPASS\tstart/is-active/stop succeeded\n' "$unit" >>"$PACKAGE_DIR/service-results.tsv" + else + failed=$((failed + 1)) + printf '%s\tFAIL\tstart/is-active/stop returned %s\n' "$unit" "$rc" >>"$PACKAGE_DIR/service-results.tsv" + fi + done <"$units" + if ! stop_nspawn; then + SERVICE_STATUS=FAIL + SERVICE_REASON="systemd-nspawn did not terminate within ${SERVICE_TIMEOUT}s" + INFRA_FAILURE=1 + return 2 + fi + + if [ "$failed" -gt 0 ]; then + SERVICE_STATUS=FAIL + SERVICE_REASON="$failed/$tested testable units failed start/is-active/stop" + return 1 + fi + if [ "$tested" -eq 0 ] && [ "$skipped_templates" -gt 0 ]; then + SERVICE_STATUS=SKIP + SERVICE_REASON="all $skipped_templates units are templates requiring an instance/configuration" + return 0 + fi + if [ "$skipped_templates" -gt 0 ]; then + SERVICE_STATUS=SKIP + SERVICE_REASON="$tested units passed; $skipped_templates template units skipped" + return 0 + fi + SERVICE_STATUS=PASS + SERVICE_REASON="all $tested units passed start/is-active/stop in systemd-nspawn" +} + +test_services() { + local units="$PACKAGE_DIR/units.list" + if ! discover_units "$units"; then + SERVICE_STATUS=FAIL + SERVICE_REASON="package file inventory is missing or unreadable" + SERVICE_MODE_USED=none + INFRA_FAILURE=1 + return 2 + fi + if [ "$UNIT_COUNT" -eq 0 ]; then + SERVICE_STATUS=PASS + SERVICE_REASON="package provides no systemd service/socket/target units" + SERVICE_MODE_USED=none + return 0 + fi + if list_matches "$SPECIAL_SERVICE_LIST" "$REQUEST" "$PACKAGE_NAME" "$BINARY_PACKAGE"; then + SERVICE_STATUS=SKIP + SERVICE_REASON="package is in the special-configuration service list" + SERVICE_MODE_USED=skip + awk '{printf "%s\tSKIP\trequires special configuration\n", $0}' "$units" >"$PACKAGE_DIR/service-results.tsv" + return 0 + fi + if [ "$SERVICE_MODE_REQUESTED" = skip ]; then + SERVICE_STATUS=SKIP + SERVICE_REASON="service mode skip was requested" + SERVICE_MODE_USED=skip + return 0 + fi + if ! nspawn_available; then + SERVICE_MODE_USED=skip + if [ "$SERVICE_MODE_REQUESTED" = nspawn ]; then + SERVICE_STATUS=FAIL + SERVICE_REASON="systemd-nspawn/machinectl or host systemd is unavailable" + INFRA_FAILURE=1 + return 2 + fi + SERVICE_STATUS=SKIP + SERVICE_REASON="plain chroot has no PID1 and systemd-nspawn is unavailable" + return 0 + fi + test_services_nspawn "$units" +} + +remove_package() { + local rc=0 verify_rc=0 + [ "$RUNTIME_MOUNTED" -eq 1 ] || mount_runtime || { + REMOVE_STATUS=FAIL + REMOVE_REASON="could not restore chroot runtime mounts before removal" + INFRA_FAILURE=1 + return 2 + } + run_with_timeout "$STAGE_TIMEOUT" "$PACKAGE_DIR/remove.log" \ + chroot "$ROOT" dnf -y --setopt=clean_requirements_on_remove=False \ + --setopt=protected_packages= remove "$BINARY_PACKAGE" || rc=$? + if [ "$rc" -ne 0 ]; then + REMOVE_STATUS=FAIL + if [ "$rc" -eq 124 ]; then + REMOVE_REASON="removal timed out after ${STAGE_TIMEOUT}s" + else + REMOVE_REASON=$(first_error "$PACKAGE_DIR/remove.log" "dnf remove exited with status $rc") + fi + return 1 + fi + rpm --root "$ROOT" -q "$BINARY_PACKAGE" >"$PACKAGE_DIR/remove-verify.log" 2>&1 || verify_rc=$? + case "$verify_rc" in + 0) + REMOVE_STATUS=FAIL + REMOVE_REASON="dnf remove succeeded but host RPM query still reports $BINARY_PACKAGE installed" + return 1 + ;; + 1) ;; + *) + REMOVE_STATUS=FAIL + REMOVE_REASON="host RPM removal verification failed with status $verify_rc" + INFRA_FAILURE=1 + return 2 + ;; + esac + REMOVE_STATUS=PASS + if [ "$PREINSTALLED" -eq 1 ]; then + REMOVE_REASON="preinstalled package removed and host RPM query confirms absence" + else + REMOVE_REASON="package removed and host RPM query confirms absence" + fi +} + +main() { + mkdir -p "$PACKAGE_DIR" + : >"$PACKAGE_DIR/worker.log" + log "testing $REQUEST in isolated rootfs" + if ! prepare_root; then + INSTALL_STATUS=FAIL + INSTALL_REASON="failed to create isolated rootfs using mode $ROOT_MODE_REQUESTED" + INFRA_FAILURE=1 + result_write || return 2 + return 2 + fi + if ! mount_runtime; then + INSTALL_STATUS=FAIL + INSTALL_REASON="failed to mount /dev, /proc, /sys, or /run in chroot" + INFRA_FAILURE=1 + result_write || return 2 + return 2 + fi + if ! install_package; then + result_write || return 2 + return 1 + fi + if ! capture_package_files; then + COMMAND_STATUS=FAIL + COMMAND_REASON="rpm -ql failed for $PACKAGE_NAME" + SERVICE_STATUS=SKIP + SERVICE_REASON="service discovery skipped because package file inventory failed" + REMOVE_STATUS=SKIP + REMOVE_REASON="removal skipped after package inventory infrastructure failure" + INFRA_FAILURE=1 + result_write || return 2 + return 2 + fi + + if list_matches "$DESKTOP_LIST" "$REQUEST" "$PACKAGE_NAME" "$BINARY_PACKAGE"; then + COMMAND_STATUS=SKIP + COMMAND_REASON="package is in the desktop whitelist" + else + test_commands || true + fi + test_services || true + if [ "$INFRA_FAILURE" -eq 0 ]; then + remove_package || true + else + REMOVE_STATUS=SKIP + REMOVE_REASON="removal skipped after infrastructure failure" + fi + result_write || return 2 + + [ "$INFRA_FAILURE" -eq 0 ] || return 2 + if [ "$INSTALL_STATUS" = FAIL ] || [ "$COMMAND_STATUS" = FAIL ] || \ + [ "$SERVICE_STATUS" = FAIL ] || [ "$REMOVE_STATUS" = FAIL ]; then + return 1 + fi + return 0 +} + +main diff --git a/testcase/tests/test-chroot-regressions.sh b/testcase/tests/test-chroot-regressions.sh new file mode 100755 index 0000000000000000000000000000000000000000..a56d25929b1bdb2faeb5478c4224811db6998180 --- /dev/null +++ b/testcase/tests/test-chroot-regressions.sh @@ -0,0 +1,295 @@ +#!/usr/bin/env bash +# shellcheck disable=SC2034,SC2317 + +set -uo pipefail + +TEST_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +RUNNER=$(realpath "$TEST_DIR/../command-execute-chroot.sh") +WORKER_SCRIPT=$(realpath "$TEST_DIR/../lib/chroot-package-worker.sh") +TMP=$(mktemp -d) +PASS=0 +FAIL=0 + +test_cleanup() { rm -rf -- "$TMP"; } +trap test_cleanup EXIT + +ok() { PASS=$((PASS + 1)); printf 'ok %s - %s\n' "$((PASS + FAIL))" "$1"; } +not_ok() { FAIL=$((FAIL + 1)); printf 'not ok %s - %s\n' "$((PASS + FAIL))" "$1"; } + +check() { + local name=$1 + shift + if "$@"; then ok "$name"; else not_ok "$name"; fi +} + +printf '1..20\n' + +# Load the runner functions without executing main. +# shellcheck disable=SC1090 +source <(sed '$d' "$RUNNER") +trap test_cleanup EXIT + +mkdir -p "$TMP/aggregate/run/results" "$TMP/aggregate/run/packages" "$TMP/aggregate/out" +printf 'alpha\n' >"$TMP/aggregate/packages.list" +printf 'corrupt\n' >"$TMP/aggregate/run/results/00001.tsv" +NORMALIZED_PACKAGES="$TMP/aggregate/packages.list" +RUN_DIR="$TMP/aggregate/run" +OUTPUT_DIR="$TMP/aggregate/out" +JOBS=1 +INFRA_FAILURE=0 +aggregate_rc=0 +aggregate_results regression >"$TMP/aggregate/stdout" 2>"$TMP/aggregate/stderr" || aggregate_rc=$? +check "truncated result becomes a package failure" test "$aggregate_rc" -eq 1 +check "truncated result is also marked infrastructure failure" test "$INFRA_FAILURE" -eq 1 +check "malformed result is synthesized with the original package identity" \ + grep -q $'^unknown\talpha\tFAIL' "$TMP/aggregate/out/package-command-result-chroot-regression.txt" + +printf '1\tother\tother-src\tother\tPASS\tok\tPASS\tok\tPASS\tok\tPASS\tok\t1\t0\t0\tcopy\tnone\n' \ + >"$TMP/aggregate/mismatch.tsv" +if valid_result_file "$TMP/aggregate/mismatch.tsv" 1 alpha; then + not_ok "result identity mismatch is rejected" +else + ok "result identity mismatch is rejected" +fi + +mkdir -p "$TMP/bin" "$TMP/run-one/results" "$TMP/run-one/packages" \ + "$TMP/run-one/work/roots" "$TMP/run-one/sessions" +cat >"$TMP/bin/setsid" <<'EOF' +#!/usr/bin/env bash +printf '%s\n' "$*" >"$FAKE_SETSID_LOG" +exit 2 +EOF +chmod +x "$TMP/bin/setsid" +OLD_PATH=$PATH +PATH="$TMP/bin:$PATH" +export FAKE_SETSID_LOG="$TMP/run-one/setsid.log" +RUN_DIR="$TMP/run-one" +WORK_ROOT="$TMP/run-one/work" +SESSION_DIR="$TMP/run-one/sessions" +RUN_TOKEN=test-run +BASE_ROOTFS="$TMP/run-one/base" +NORMALIZED_DESKTOP="$TMP/run-one/desktop.list" +NORMALIZED_SERVICES="$TMP/run-one/service.list" +ROOT_MODE=copy +SERVICE_MODE=skip +STAGE_TIMEOUT=5 +COMMAND_TIMEOUT=1 +SERVICE_TIMEOUT=1 +PACKAGE_TIMEOUT=5 +WORKER=/does/not/matter +: >"$NORMALIZED_DESKTOP" +: >"$NORMALIZED_SERVICES" +run_rc=0 +run_one 1 alpha || run_rc=$? +PATH=$OLD_PATH +check "worker infrastructure errors propagate as exit 2" test "$run_rc" -eq 2 +check "worker runs in a kill-on-exit PID namespace" \ + grep -q -- 'unshare --mount --pid --fork --kill-child=KILL --mount-proc' "$TMP/run-one/setsid.log" + +BASE_ARCH=x86_64 +REPO_FINGERPRINT=repo-test +RELEASEVER=9 +BOOTSTRAP_TIMEOUT=2 +RUN_DIR="$TMP/base-run" +mkdir -p "$RUN_DIR" "$TMP/base/usr/bin" "$TMP/base/etc/dnf/vars" "$TMP/base/etc/yum/vars" +: >"$TMP/base/usr/bin/bash" +: >"$TMP/base/usr/bin/dnf" +chmod +x "$TMP/base/usr/bin/bash" "$TMP/base/usr/bin/dnf" +printf 'schema=2\nreleasever=8\narch=x86_64\nrepo_fingerprint=repo-test\n' >"$TMP/base/.yum-ci-base-rootfs" +printf '8\n' >"$TMP/base/etc/dnf/vars/releasever" +printf '8\n' >"$TMP/base/etc/yum/vars/releasever" +base_rc=0 +prepare_base_rootfs "$TMP/base" /bin/false || base_rc=$? +check "mismatched persistent base rootfs is rejected" test "$base_rc" -eq 2 +printf 'schema=2\nreleasever=9\narch=x86_64\nrepo_fingerprint=repo-test\n' >"$TMP/base/.yum-ci-base-rootfs" +printf '9\n' >"$TMP/base/etc/dnf/vars/releasever" +printf '9\n' >"$TMP/base/etc/yum/vars/releasever" +check "matching persistent base rootfs is reusable" prepare_base_rootfs "$TMP/base" /bin/false + +# Load worker functions without its final main call. It expects 14 positional arguments. +mkdir -p "$TMP/worker-work/roots" "$TMP/worker-package" "$TMP/worker-root/etc" +: >"$TMP/desktop.list" +: >"$TMP/service.list" +set -- 1 alpha "$TMP/base" "$TMP/worker-work" "$TMP/worker-package" \ + "$TMP/worker-result.tsv" "$TMP/desktop.list" "$TMP/service.list" auto auto 5 1 2 test-run +# shellcheck disable=SC1090 +source <(sed '$d' "$WORKER_SCRIPT") +trap test_cleanup EXIT +trap - INT TERM HUP +mkdir -p "$ROOT/tmp" "$ROOT/etc" "$PACKAGE_DIR" + +timeout() { + printf 'error while loading shared libraries: libmissing.so.1: cannot open shared object file\n' + return 1 +} +probe_rc=0 +probe_command /usr/bin/demo /tmp/missing.trace --help "$PACKAGE_DIR/missing.log" || probe_rc=$? +unset -f timeout +check "missing dynamic libraries fail command probing" test "$probe_rc" -eq 4 + +run_with_timeout() { return 0; } +rpm() { return 1; } +PREINSTALLED=1 +RUNTIME_MOUNTED=1 +BINARY_PACKAGE=alpha-1.0-1.x86_64 +PACKAGE_NAME=alpha +REMOVE_STATUS=SKIP +REMOVE_REASON="not run" +remove_rc=0 +remove_package || remove_rc=$? +unset -f run_with_timeout rpm +check "preinstalled packages execute and verify the remove stage" \ + test "$remove_rc:$REMOVE_STATUS" = "0:PASS" + +FAKE_DEV_MOUNTED=0 +mount() { + case "$*" in + "--rbind /dev $ROOT/dev") FAKE_DEV_MOUNTED=1; return 0 ;; + "--make-rslave $ROOT/dev") return 0 ;; + "-t proc proc $ROOT/proc") return 1 ;; + *) return 0 ;; + esac +} +mountpoint() { + [ "${2:-}" = "$ROOT/dev" ] && [ "$FAKE_DEV_MOUNTED" -eq 1 ] +} +umount() { + [ "${*: -1}" = "$ROOT/dev" ] && FAKE_DEV_MOUNTED=0 + return 0 +} +RUNTIME_MOUNTED=0 +DEV_MOUNTED=0 +PROC_MOUNTED=0 +SYS_MOUNTED=0 +RUN_MOUNTED=0 +partial_rc=0 +mount_runtime || partial_rc=$? +unmount_rc=0 +unmount_runtime || unmount_rc=$? +check "partial runtime mounts are tracked and unmounted" \ + test "$partial_rc:$unmount_rc:$DEV_MOUNTED:$FAKE_DEV_MOUNTED" = "1:0:0:0" + +if grep -Eq 'rm -rf[^[:cntrl:]]*ROOT_DIR' "$WORKER_SCRIPT"; then + not_ok "worker defers root deletion until namespace teardown" +else + ok "worker defers root deletion until namespace teardown" +fi +unset -f mount mountpoint umount + +TIMEOUT_CALLED=0 +timeout() { TIMEOUT_CALLED=1; return 0; } +MACHINE_NAME=test-machine +NSPAWN_PID="" +stop_nspawn_rc=0 +stop_nspawn || stop_nspawn_rc=$? +unset -f timeout +check "nspawn termination is covered by the service timeout wrapper" \ + test "$stop_nspawn_rc:$TIMEOUT_CALLED" = "0:1" + +chroot() { + if [ "${2:-} ${3:-}" = "rpm -ql" ]; then + printf '/usr/lib/systemd/system/alpha.service\n' + return 0 + fi + return 1 +} +inventory_rc=0 +capture_package_files || inventory_rc=$? +units="$PACKAGE_DIR/inventory-units.list" +discover_units "$units" +unset -f chroot +check "package files are captured independently of command probing" \ + test "$inventory_rc" -eq 0 +check "captured inventory remains available to service discovery" \ + test "$UNIT_COUNT:$(cat "$units")" = "1:alpha.service" +rm -f "$PACKAGE_DIR/rpm-files.list" +SERVICE_STATUS=SKIP +INFRA_FAILURE=0 +missing_inventory_rc=0 +test_services || missing_inventory_rc=$? +check "missing service inventory fails closed as infrastructure" \ + test "$missing_inventory_rc:$SERVICE_STATUS:$INFRA_FAILURE" = "2:FAIL:1" + +RUN_TOKEN=runner-one +INDEX=7 +machine_one=$(machine_name_for) +RUN_TOKEN=runner-two +machine_two=$(machine_name_for) +RUN_TOKEN=$(printf 'x%.0s' {1..100}) +INDEX=98765 +machine_long=$(machine_name_for) +if [ "$machine_one" != "$machine_two" ] && [ "${#machine_long}" -le 64 ] && \ + [[ $machine_long == *-98765 ]]; then + ok "nspawn names include the unique runner token" +else + not_ok "nspawn names include the unique runner token" +fi + +SESSION_DIR="$TMP/sessions" +mkdir -p "$SESSION_DIR" +setsid sleep 30 & +session_pid=$! +session_start=$(awk '{print $22}' "/proc/$session_pid/stat") +printf '%s %s\n' "$session_pid" "$session_start" >"$SESSION_DIR/test.session" +CHILD_PIDS=() +CHILD_STARTS=() +stop_children +if kill -0 "$session_pid" 2>/dev/null; then + kill -KILL -- "-$session_pid" 2>/dev/null || true + wait "$session_pid" 2>/dev/null || true + not_ok "registered worker sessions are terminated and reaped" +else + wait "$session_pid" 2>/dev/null || true + ok "registered worker sessions are terminated and reaped" +fi + +setsid sleep 30 & +reused_pid=$! +reused_start=$(awk '{print $22}' "/proc/$reused_pid/stat") +printf '%s %s\n' "$reused_pid" "$((reused_start + 1))" >"$SESSION_DIR/stale.session" +CHILD_PIDS=("$reused_pid") +CHILD_STARTS=(["$reused_pid"]="$((reused_start + 1))") +stop_children +if kill -0 "$reused_pid" 2>/dev/null; then + ok "stale session records cannot kill a reused PID" +else + not_ok "stale session records cannot kill a reused PID" +fi +kill -KILL -- "-$reused_pid" 2>/dev/null || true +wait "$reused_pid" 2>/dev/null || true + +mkdir -p "$TMP/signal-bin" "$TMP/signal/run/results" "$TMP/signal/run/packages" \ + "$TMP/signal/work/roots" "$TMP/signal/sessions" +cat >"$TMP/signal-bin/timeout" <<'EOF' +#!/usr/bin/env bash +exec sleep 30 +EOF +chmod +x "$TMP/signal-bin/timeout" +OLD_PATH=$PATH +PATH="$TMP/signal-bin:$PATH" +RUN_DIR="$TMP/signal/run" +WORK_ROOT="$TMP/signal/work" +SESSION_DIR="$TMP/signal/sessions" +RUN_TOKEN=signal-run +KEEP_WORK=1 +CHILD_PIDS=() +CHILD_STARTS=() +run_one 2 beta & +tracked_child=$! +tracked_start=$(awk '{print $22}' "/proc/$tracked_child/stat") +CHILD_PIDS+=("$tracked_child") +CHILD_STARTS["$tracked_child"]=$tracked_start +for _ in {1..50}; do + [ -s "$SESSION_DIR/00002.session" ] && break + sleep 0.02 +done +stop_children +PATH=$OLD_PATH +if [ ! -e "/proc/$tracked_child" ] && [ ! -e "$SESSION_DIR/00002.session" ]; then + ok "interrupt cleanup follows the registered setsid session leader" +else + not_ok "interrupt cleanup follows the registered setsid session leader" +fi + +printf '# pass=%s fail=%s\n' "$PASS" "$FAIL" +[ "$FAIL" -eq 0 ] diff --git a/testcase/tests/test-command-execute-chroot.sh b/testcase/tests/test-command-execute-chroot.sh new file mode 100755 index 0000000000000000000000000000000000000000..efbb67300a0a01a1b1bd4120fc861ab1f2393c7a --- /dev/null +++ b/testcase/tests/test-command-execute-chroot.sh @@ -0,0 +1,86 @@ +#!/usr/bin/env bash + +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +RUNNER="$(realpath "$SCRIPT_DIR/../command-execute-chroot.sh")" +TMP=$(mktemp -d) +PASS=0 +FAIL=0 + +cleanup() { rm -rf -- "$TMP"; } +trap cleanup EXIT + +ok() { + PASS=$((PASS + 1)) + printf 'ok %s - %s\n' "$((PASS + FAIL))" "$1" +} + +not_ok() { + FAIL=$((FAIL + 1)) + printf 'not ok %s - %s\n' "$((PASS + FAIL))" "$1" +} + +expect_success() { + local name=$1 + shift + if "$@" >"$TMP/stdout" 2>"$TMP/stderr"; then ok "$name"; else not_ok "$name"; fi +} + +expect_failure() { + local name=$1 + shift + if "$@" >"$TMP/stdout" 2>"$TMP/stderr"; then not_ok "$name"; else ok "$name"; fi +} + +printf '1..12\n' + +expect_success "help is available without root" bash "$RUNNER" --help + +cat >"$TMP/packages.list" <<'EOF' +# comment +grep + +which extra-column-is-ignored +grep # duplicate requests remain independent + leading-space-package trailing-column +EOF +: >"$TMP/desktop.list" +printf 'openssh-server\n' >"$TMP/service.list" + +expect_success "valid lists are accepted" bash "$RUNNER" --validate-only -j 2 \ + "$TMP/packages.list" "$TMP/desktop.list" "$TMP/service.list" +if grep -q 'packages=4 jobs=2' "$TMP/stdout"; then ok "validation reports package count and jobs"; else not_ok "validation reports package count and jobs"; fi +if [ "$(grep -c '^grep$' "$TMP/stdout")" -eq 2 ]; then ok "duplicate package requests are preserved"; else not_ok "duplicate package requests are preserved"; fi +if grep -q '^leading-space-package$' "$TMP/stdout"; then ok "leading whitespace is normalized"; else not_ok "leading whitespace is normalized"; fi + +printf 'bad;token\n' >"$TMP/unsafe.list" +expect_failure "unsafe package tokens are rejected" bash "$RUNNER" --validate-only \ + "$TMP/unsafe.list" "$TMP/desktop.list" "$TMP/service.list" + +: >"$TMP/empty.list" +expect_failure "empty package lists are rejected" bash "$RUNNER" --validate-only \ + "$TMP/empty.list" "$TMP/desktop.list" "$TMP/service.list" + +expect_failure "invalid concurrency is rejected" bash "$RUNNER" --validate-only -j nope \ + "$TMP/packages.list" "$TMP/desktop.list" "$TMP/service.list" + +expect_failure "concurrency above the hard limit is rejected" bash "$RUNNER" --validate-only -j 65 \ + "$TMP/packages.list" "$TMP/desktop.list" "$TMP/service.list" + +expect_failure "invalid service mode is rejected" bash "$RUNNER" --validate-only --service-mode bad \ + "$TMP/packages.list" "$TMP/desktop.list" "$TMP/service.list" + +expect_failure "missing input file is rejected" bash "$RUNNER" --validate-only \ + "$TMP/missing.list" "$TMP/desktop.list" "$TMP/service.list" + +if timeout 2 bash "$RUNNER" --package-timeout >"$TMP/stdout" 2>"$TMP/stderr"; then + not_ok "missing option values are rejected without looping" +elif [ "$?" -eq 2 ]; then + ok "missing option values are rejected without looping" +else + not_ok "missing option values are rejected without looping" +fi + +printf '# pass=%s fail=%s\n' "$PASS" "$FAIL" +[ "$FAIL" -eq 0 ]