fix(kyber): preserve host recovery through SMART failures

[shunkakinoki]

A nonzero SMART health status previously terminated the host-health script before CRI checks and freezer recovery. Bound each SMART read, distinguish command failures from health-status bits, validate the normalized endurance value, and retain existing wear warnings when a fresh reading is unavailable. Run CRI and freezer recovery before SMART probes.

Validation at head 9df7c4c97564dfa80a107da64af96fc4570d8bba: reproduced the previous exit-16 failure; all 2,509 ShellSpec examples pass, including 118 focused Kyber examples. ShellCheck and whitespace validation pass. Tests cover SMART warning bitmasks, command failures/timeouts, malformed or absent measurements, alert recovery, and execution order.

Runtime activation is pending. This fixes a monitor failure path; it does not establish the cause of recurring host I/O pressure or resolve the existing SSD-wear warning. Freeze thresholds and containment remain unchanged.

[indent]

Warning

This organization's free trial has ended, so Indent couldn't continue this review. Manage billing to resume reviews on this repository.

CI Checks

All required CI checks passed on 9df7c4c.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 1dbc0fe2-00fc-4691-9f69-82e3b5008d14

📥 Commits

Reviewing files that changed from the base of the PR and between 1143760 and 9df7c4c.

📒 Files selected for processing (2)

• config/k3s/kyber-host-health.sh
• spec/activate_kyber_spec.sh

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Summary

Summary by CodeRabbit

• Bug Fixes
  • Improved disk health monitoring to detect unreadable, timed-out, malformed, or worn SMART data.
  • Prevented disk-check failures from interrupting other host recovery checks.
  • Ensured disk endurance warnings remain active until a valid healthy reading is received.
  • Adjusted recovery check order so container runtime and freezer recovery occur before disk endurance checks.

Walkthrough

The host health script now runs timeout-protected SMART wear checks per disk, alerts on read or invalid-data failures, and updates endurance alerts only for valid measurements. Recovery checks run before disk checks. ShellSpec tests cover status bits, failures, invalid values, healthy readings, and ordering.

Changes

Disk wear health checks

Layer / File(s)	Summary
SMART wear validation and alert handling config/k3s/kyber-host-health.sh	The new helper runs smartctl -A with timeouts, validates Wear_Leveling_Count, and updates read and endurance alerts.
Check ordering and validation coverage config/k3s/kyber-host-health.sh, spec/activate_kyber_spec.sh	The script runs disk checks after DNS checks. Tests cover SMART statuses, read failures, timeouts, malformed values, healthy measurements, and execution order.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 9df7c

The host-health script now bounds and validates SMART reads while allowing CRI and freezer recovery to run first. Current coverage supports the intended alert and recovery behavior, with no remaining merge-blocking risk identified.

Poem

A rabbit checks each disk with care
SMART data hops through timed-out air
Bad readings raise a warning bright
Good wear clears alerts from sight
Recovery runs before the night

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the Kyber SMART failure fix and its recovery-preservation purpose.
Description check	✅ Passed	The description directly explains the SMART failure handling changes, recovery ordering, validation, and test results.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/kyber-smart-health-recovery

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[pullfrog]

Reviewed changes

This PR adds SMART health monitoring to the kyber host health script to prevent premature termination of host recovery. Previously, a nonzero SMART health status would terminate the host-health script before CRI checks and freezer recovery. The fix introduces a check_disk_wear() function that captures SMART health status separately, allowing the system to continue host recovery even when SMART readings fail (due to timeouts, malformed output, or hardware unavailability). Existing wear warnings are preserved when SMART reads are unavailable.

Changes:

• config/k3s/kyber-host-health.sh: Added check_disk_wear_device() function and integrated SMART health checks into manage_orchestration_circuit_breaker() to run before SMART probes.
• spec/activate_kyber_spec.sh: Updated the activation script to include SMART health check handling in the test specifications.

These changes ensure that host recovery proceeds normally even when SMART health status is problematic, while still capturing wear warnings and alerting appropriately.

  ^{｜ View workflow run ｜ Using openrouter/openrouter/free ｜ 𝕏}

[linear-code]

SHUN-4004