JMS-consuming services have no livenessProbe, so a dead consumer looks healthy

[jcschaff]

Summary

None of the JMS-consuming services declare a livenessProbe, so a service that has stopped
consuming looks perfectly healthy to Kubernetes. Dev's submit sat Running, 1/1 Ready,
0 restarts for 6h50m while consuming nothing at all (#2031).

$ grep -c livenessProbe kustomize/base/{submit,sched,data,db}.yaml
submit.yaml:0
sched.yaml:0
data.yaml:0
db.yaml:0

The only thing that noticed was the sim health check going red, and only indirectly — it reported
"simulation took longer than 480000 to complete", which reads as a compute problem rather than a
messaging one. Diagnosing it took a log-volume comparison between namespaces:

	dev	prod
submit lines / 6h	0	1212

Why this is worth fixing separately from #2031

#2031 is one way a consumer can die. A liveness probe is the backstop for all of them, including
the ones we have not found yet. The two failures in question were both invisible from outside:

• the consumer thread exiting (JMS consumer thread dies silently when a broker restarts, and the pod stays Ready #2031), and
• the earlier JMS wedge that JmsFailoverWatchdog was written for — which needed an in-process
  watchdog precisely because nothing external was watching.

JmsFailoverWatchdog is a good mechanism but it only guards the connection, and it can only act if
the JVM notices. An external probe does not depend on the process's own view of its health.

Suggested shape

A probe that asserts liveness of consumption, not merely that the JVM is up — the latter would
have passed happily throughout this incident. Options, roughly in increasing order of effort:

1. Consumer-thread liveness: a small HTTP endpoint (or file touched by the poll loop) reporting
   whether each registered consumer thread is alive and when it last completed a poll cycle. Fails
   if any consumer has been dead, or silent, beyond a threshold.
2. Broker session check: probe verifies the JMS connection and session are valid.
3. exec probe on a marker file the poll loop updates, if adding an HTTP listener to these
   services is unattractive.

Option 1 is the one that would have caught this specific failure immediately, since the thread was
gone rather than merely idle. Note that "idle" and "dead" must be distinguishable — dev is legitimately
idle much of the time, so a probe keyed on "no messages processed recently" would false-positive
there. That distinction is the main design question.

Also worth considering

readinessProbe as well as livenessProbe: a service that cannot consume should arguably not be
counted as ready, though for these queue consumers there is no traffic being load-balanced to them,
so liveness is the one that matters.

Files: kustomize/base/{submit,sched,data,db}.yaml in
vcell-fluxcd.

Refs #2031