Skip to content

feat(log): Capture bigtable_rs PingAndWarm failures as logs - #734

Merged
jan-auer merged 2 commits into
mainfrom
junior/log-bigtable-pingandwarm-failures
Oct 27, 2026
Merged

jan-auer merged 2 commits into
mainfrom
junior/log-bigtable-pingandwarm-failures

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

bigtable_rs logs failed background PingAndWarm requests at DEBUG. The default filter dropped them, so we could not see if warm pings also timed out during Bigtable incidents.

The default filter now enables DEBUG for bigtable_rs::bigtable. The Sentry layer sends these as Sentry logs. Stderr output stays at the configured level (info in prod). The filter targets the module and not the whole crate because bigtable_rs::auth_service logs at DEBUG on every request. The module also logs channel refresh start and end at DEBUG, about twice per pod every 50 minutes.

Ops does not set RUST_LOG for objectstore, so this applies in all regions.

via @lcian.

--

View Junior Session [Sentry]

Co-Authored-By: Lorenzo Cian <17258265+lcian@users.noreply.github.com>
@sentry-junior
sentry-junior Bot requested a review from a team as a code owner October 23, 2026 15:22
@codecov

codecov Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.91%. Comparing base (9a9c788) to head (43cdcd1).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #734   +/-   ##
=======================================
  Coverage   91.91%   91.91%           
=======================================
  Files         117      117           
  Lines       24141    24141           
=======================================
  Hits        22188    22188           
  Misses       1953     1953           
Components Coverage Δ
Rust Backend 94.92% <ø> (ø)
Rust Client 82.86% <ø> (ø)
Python Client 93.73% <ø> (ø)

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sentry-junior sentry-junior Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At bigtable_rs 072cb58, bigtable_rs::bigtable=DEBUG adds only three debug! logs:

  • Background PingAndWarm failed: {error}: the one we want. This happens at most once per channel per 10s warm tick, and only while pings are failing.
  • Refreshing {n} channels / Refreshed {n} channels: about two per pod every 50 minutes.

Nothing else changes. The module's info! and warn! logs already pass the default filter. read_rows only uses trace!, which stays filtered. The per-request debug! logs in bigtable_rs::auth_service are excluded because the directive is scoped to the bigtable module.

Comment thread objectstore-log/src/subscriber.rs Resolved
@jan-auer
jan-auer merged commit d682c53 into main Oct 27, 2026
24 checks passed
@jan-auer
jan-auer deleted the junior/log-bigtable-pingandwarm-failures branch October 27, 2026 10:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants