Skip to content

BE-562: Bundle CA certificates in hash-graph Docker image#8713

Merged
TimDiekmann merged 1 commit into
mainfrom
t/be-562-fix-sentry-tls-backend-after-reqwest-upgrade
May 12, 2026
Merged

BE-562: Bundle CA certificates in hash-graph Docker image#8713
TimDiekmann merged 1 commit into
mainfrom
t/be-562-fix-sentry-tls-backend-after-reqwest-upgrade

Conversation

@TimDiekmann
Copy link
Copy Markdown
Member

🌟 What is the purpose of this PR?

Fix a runtime panic in the hash-graph Docker image in AWS where Sentry initialisation fails because the FROM scratch runner image lacks a system CA trust store. After the recent reqwest 0.13 upgrade, rustls-platform-verifier is the only trust source available for the rustls feature, and it requires /etc/ssl/certs/ca-certificates.crt to exist on disk at runtime.

🔗 Related links

  • Linear: BE-562 (internal)
  • Trigger: Update reqwest Rust crates #8672 (reqwest 0.13 upgrade)
  • Underlying: reqwest 0.13 removed rustls-tls-native-roots / rustls-tls-webpki-roots features. rustls-platform-verifier is now a mandatory dep on every rustls* feature flag of reqwest; there is no opt-out.

🚫 Blocked by

  • none

🔍 What does this change?

  • Copy /etc/ssl/certs/ca-certificates.crt from the Debian-based builder stage into /out/etc/ssl/certs/ so the final FROM scratch image has a system trust store available for rustls-platform-verifier.
  • Image size grows by ~220 KB (the size of the CA bundle).
  • Only hash-graph is affected — all other workload images (hash-api, hash-frontend, AI/integration workers) are based on full Debian images and already have ca-certificates installed.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

  • does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

  • are internal and do not require a docs change

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

  • do not affect the execution graph

⚠️ Known issues

The panic message emitted by sentry-0.47's reqwest transport is misleading — it claims a TLS backend feature is missing (Enable either the native-tls or the rustls feature of the sentry crate), but the actual cause is a missing system trust store. The features are correctly enabled; only the CA bundle is absent. Nothing to fix on our end.

🐾 Next steps

None planned. If future Rust workloads adopt the FROM scratch pattern, they will need the same two-line COPY.

🛡 What tests cover this?

No automated tests — Docker image content is not part of any CI test suite. Verified manually:

  1. Built hash-graph:repro locally with this fix using yarn build:docker:dev.
  2. Confirmed /etc/ssl/certs/ca-certificates.crt is present in the image (docker export | tar -tv).
  3. Reproduced the original AWS panic on the fixed image by simulating the unfixed state via docker run --tmpfs /etc/ssl/certs:size=1m ... with HASH_GRAPH_SENTRY_DSN set — got the exact Failed to build reqwest client ... No CA certificates were loaded from the system panic.
  4. Without the tmpfs override, Sentry transport builds successfully and the TLS handshake to o0.ingest.sentry.io:443 completes (verified via h2 debug logs); failure happens later on Postgres connect, which is expected.

❓ How to test this?

  1. Check out the branch.
  2. From apps/hash-graph, run yarn build:docker:dev (or yarn build:docker for release).
  3. docker run --rm -e HASH_GRAPH_SENTRY_DSN='https://aaa@o0.ingest.sentry.io/1' --entrypoint=/hash-graph hash-graph server — should fail later on Postgres connect, not on the Sentry TLS panic.

📹 Demo

n/a

`reqwest` 0.13 removed all `rustls-tls-*-roots` features and unconditionally
depends on `rustls-platform-verifier`, which reads the system trust store at
runtime. The `hash-graph` runner image is `FROM scratch`, so the trust store
is empty and `sentry::init()` panics with
`No CA certificates were loaded from the system`.

Copy `/etc/ssl/certs/ca-certificates.crt` from the Debian-based builder stage
into the final image (~220 KB) so outbound TLS works again.
@vercel
Copy link
Copy Markdown

vercel Bot commented May 12, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hash Ready Ready Preview, Comment May 12, 2026 4:41pm
2 Skipped Deployments
Project Deployment Actions Updated (UTC)
hashdotdesign-tokens Ignored Ignored Preview May 12, 2026 4:41pm
petrinaut Skipped Skipped May 12, 2026 4:41pm

@github-actions github-actions Bot added area/apps > hash* Affects HASH (a `hash-*` app) type/eng > backend Owned by the @backend team area/apps area/apps > hash-graph labels May 12, 2026
@TimDiekmann TimDiekmann requested a review from a team May 12, 2026 16:33
@TimDiekmann TimDiekmann enabled auto-merge May 12, 2026 16:33
Copy link
Copy Markdown

@augmentcode augmentcode Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed. No suggestions at this time.

Comment augment review to trigger a new review at any time.

@TimDiekmann TimDiekmann added this pull request to the merge queue May 12, 2026
Merged via the queue into main with commit add3967 May 12, 2026
57 checks passed
@TimDiekmann TimDiekmann deleted the t/be-562-fix-sentry-tls-backend-after-reqwest-upgrade branch May 12, 2026 17:17
@github-actions
Copy link
Copy Markdown
Contributor

Benchmark results

@rust/hash-graph-benches – Integrations

policy_resolution_large

Function Value Mean Flame graphs
resolve_policies_for_actor user: empty, selectivity: high, policies: 2002 $$28.4 \mathrm{ms} \pm 150 \mathrm{μs}\left({\color{gray}2.44 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: low, policies: 1 $$3.51 \mathrm{ms} \pm 19.5 \mathrm{μs}\left({\color{gray}1.29 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: medium, policies: 1001 $$13.3 \mathrm{ms} \pm 99.6 \mathrm{μs}\left({\color{gray}0.904 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: high, policies: 3314 $$44.1 \mathrm{ms} \pm 318 \mathrm{μs}\left({\color{gray}0.814 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: low, policies: 1 $$14.7 \mathrm{ms} \pm 120 \mathrm{μs}\left({\color{gray}1.86 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: medium, policies: 1526 $$24.9 \mathrm{ms} \pm 167 \mathrm{μs}\left({\color{gray}0.286 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: high, policies: 2078 $$29.9 \mathrm{ms} \pm 186 \mathrm{μs}\left({\color{gray}4.35 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: low, policies: 1 $$3.79 \mathrm{ms} \pm 17.9 \mathrm{μs}\left({\color{gray}-0.021 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: medium, policies: 1033 $$14.8 \mathrm{ms} \pm 94.2 \mathrm{μs}\left({\color{gray}3.73 \mathrm{\%}}\right) $$ Flame Graph

policy_resolution_medium

Function Value Mean Flame graphs
resolve_policies_for_actor user: empty, selectivity: high, policies: 102 $$3.83 \mathrm{ms} \pm 23.5 \mathrm{μs}\left({\color{gray}-0.198 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: low, policies: 1 $$3.01 \mathrm{ms} \pm 18.3 \mathrm{μs}\left({\color{gray}-0.455 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: medium, policies: 51 $$3.37 \mathrm{ms} \pm 17.8 \mathrm{μs}\left({\color{gray}-0.394 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: high, policies: 269 $$5.27 \mathrm{ms} \pm 35.4 \mathrm{μs}\left({\color{gray}1.11 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: low, policies: 1 $$3.60 \mathrm{ms} \pm 19.4 \mathrm{μs}\left({\color{gray}0.653 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: medium, policies: 107 $$4.22 \mathrm{ms} \pm 28.3 \mathrm{μs}\left({\color{gray}1.60 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: high, policies: 133 $$4.52 \mathrm{ms} \pm 31.5 \mathrm{μs}\left({\color{gray}0.752 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: low, policies: 1 $$3.52 \mathrm{ms} \pm 21.8 \mathrm{μs}\left({\color{gray}0.701 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: medium, policies: 63 $$4.20 \mathrm{ms} \pm 29.8 \mathrm{μs}\left({\color{gray}2.44 \mathrm{\%}}\right) $$ Flame Graph

policy_resolution_none

Function Value Mean Flame graphs
resolve_policies_for_actor user: empty, selectivity: high, policies: 2 $$2.69 \mathrm{ms} \pm 12.1 \mathrm{μs}\left({\color{gray}3.04 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: low, policies: 1 $$2.53 \mathrm{ms} \pm 17.5 \mathrm{μs}\left({\color{gray}-0.222 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: medium, policies: 1 $$2.67 \mathrm{ms} \pm 18.3 \mathrm{μs}\left({\color{gray}3.79 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: high, policies: 8 $$2.96 \mathrm{ms} \pm 16.0 \mathrm{μs}\left({\color{gray}2.93 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: low, policies: 1 $$2.76 \mathrm{ms} \pm 14.9 \mathrm{μs}\left({\color{gray}3.46 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: medium, policies: 3 $$2.93 \mathrm{ms} \pm 16.6 \mathrm{μs}\left({\color{gray}2.58 \mathrm{\%}}\right) $$ Flame Graph

policy_resolution_small

Function Value Mean Flame graphs
resolve_policies_for_actor user: empty, selectivity: high, policies: 52 $$3.06 \mathrm{ms} \pm 18.6 \mathrm{μs}\left({\color{gray}-0.028 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: low, policies: 1 $$2.78 \mathrm{ms} \pm 14.7 \mathrm{μs}\left({\color{gray}0.704 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: medium, policies: 25 $$3.05 \mathrm{ms} \pm 16.9 \mathrm{μs}\left({\color{gray}1.34 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: high, policies: 94 $$3.48 \mathrm{ms} \pm 17.3 \mathrm{μs}\left({\color{gray}0.332 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: low, policies: 1 $$2.97 \mathrm{ms} \pm 16.4 \mathrm{μs}\left({\color{gray}0.037 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: medium, policies: 26 $$3.32 \mathrm{ms} \pm 22.5 \mathrm{μs}\left({\color{gray}0.376 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: high, policies: 66 $$3.37 \mathrm{ms} \pm 20.5 \mathrm{μs}\left({\color{gray}-0.532 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: low, policies: 1 $$2.96 \mathrm{ms} \pm 15.9 \mathrm{μs}\left({\color{gray}-0.380 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: medium, policies: 29 $$3.35 \mathrm{ms} \pm 19.6 \mathrm{μs}\left({\color{gray}0.370 \mathrm{\%}}\right) $$ Flame Graph

read_scaling_complete

Function Value Mean Flame graphs
entity_by_id;one_depth 1 entities $$54.6 \mathrm{ms} \pm 287 \mathrm{μs}\left({\color{gray}-0.959 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;one_depth 10 entities $$46.9 \mathrm{ms} \pm 233 \mathrm{μs}\left({\color{gray}1.35 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;one_depth 25 entities $$52.3 \mathrm{ms} \pm 288 \mathrm{μs}\left({\color{gray}3.96 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;one_depth 5 entities $$45.8 \mathrm{ms} \pm 398 \mathrm{μs}\left({\color{gray}1.60 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;one_depth 50 entities $$63.7 \mathrm{ms} \pm 349 \mathrm{μs}\left({\color{gray}0.833 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;two_depth 1 entities $$64.3 \mathrm{ms} \pm 407 \mathrm{μs}\left({\color{gray}1.88 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;two_depth 10 entities $$56.6 \mathrm{ms} \pm 315 \mathrm{μs}\left({\color{gray}-1.135 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;two_depth 25 entities $$107 \mathrm{ms} \pm 566 \mathrm{μs}\left({\color{gray}1.91 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;two_depth 5 entities $$58.5 \mathrm{ms} \pm 3.77 \mathrm{ms}\left({\color{red}25.1 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;two_depth 50 entities $$278 \mathrm{ms} \pm 830 \mathrm{μs}\left({\color{lightgreen}-6.604 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;zero_depth 1 entities $$19.3 \mathrm{ms} \pm 120 \mathrm{μs}\left({\color{gray}-0.477 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;zero_depth 10 entities $$20.3 \mathrm{ms} \pm 97.7 \mathrm{μs}\left({\color{gray}0.349 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;zero_depth 25 entities $$20.8 \mathrm{ms} \pm 145 \mathrm{μs}\left({\color{gray}0.339 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;zero_depth 5 entities $$19.3 \mathrm{ms} \pm 117 \mathrm{μs}\left({\color{gray}0.158 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;zero_depth 50 entities $$25.7 \mathrm{ms} \pm 144 \mathrm{μs}\left({\color{gray}-0.220 \mathrm{\%}}\right) $$ Flame Graph

read_scaling_linkless

Function Value Mean Flame graphs
entity_by_id 1 entities $$19.5 \mathrm{ms} \pm 117 \mathrm{μs}\left({\color{gray}2.51 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 10 entities $$19.4 \mathrm{ms} \pm 114 \mathrm{μs}\left({\color{gray}-0.032 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 100 entities $$19.8 \mathrm{ms} \pm 107 \mathrm{μs}\left({\color{gray}0.255 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 1000 entities $$20.5 \mathrm{ms} \pm 114 \mathrm{μs}\left({\color{gray}0.645 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 10000 entities $$26.9 \mathrm{ms} \pm 221 \mathrm{μs}\left({\color{gray}0.022 \mathrm{\%}}\right) $$ Flame Graph

representative_read_entity

Function Value Mean Flame graphs
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/block/v/1 $$36.2 \mathrm{ms} \pm 326 \mathrm{μs}\left({\color{gray}3.86 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/book/v/1 $$36.0 \mathrm{ms} \pm 319 \mathrm{μs}\left({\color{gray}4.05 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/building/v/1 $$36.9 \mathrm{ms} \pm 325 \mathrm{μs}\left({\color{gray}-0.211 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/organization/v/1 $$37.1 \mathrm{ms} \pm 291 \mathrm{μs}\left({\color{gray}1.79 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/page/v/2 $$37.4 \mathrm{ms} \pm 317 \mathrm{μs}\left({\color{gray}1.30 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/person/v/1 $$35.5 \mathrm{ms} \pm 280 \mathrm{μs}\left({\color{gray}2.26 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/playlist/v/1 $$37.4 \mathrm{ms} \pm 311 \mathrm{μs}\left({\color{gray}3.97 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/song/v/1 $$36.8 \mathrm{ms} \pm 304 \mathrm{μs}\left({\color{gray}-1.575 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/uk-address/v/1 $$35.6 \mathrm{ms} \pm 340 \mathrm{μs}\left({\color{gray}2.26 \mathrm{\%}}\right) $$ Flame Graph

representative_read_entity_type

Function Value Mean Flame graphs
get_entity_type_by_id Account ID: bf5a9ef5-dc3b-43cf-a291-6210c0321eba $$8.70 \mathrm{ms} \pm 49.0 \mathrm{μs}\left({\color{gray}0.197 \mathrm{\%}}\right) $$ Flame Graph

representative_read_multiple_entities

Function Value Mean Flame graphs
entity_by_property traversal_paths=0 0 $$96.8 \mathrm{ms} \pm 591 \mathrm{μs}\left({\color{gray}0.893 \mathrm{\%}}\right) $$
entity_by_property traversal_paths=255 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true $$153 \mathrm{ms} \pm 679 \mathrm{μs}\left({\color{gray}1.60 \mathrm{\%}}\right) $$
entity_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false $$105 \mathrm{ms} \pm 638 \mathrm{μs}\left({\color{gray}1.62 \mathrm{\%}}\right) $$
entity_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true $$117 \mathrm{ms} \pm 565 \mathrm{μs}\left({\color{gray}2.02 \mathrm{\%}}\right) $$
entity_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true $$126 \mathrm{ms} \pm 617 \mathrm{μs}\left({\color{gray}1.85 \mathrm{\%}}\right) $$
entity_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true $$131 \mathrm{ms} \pm 562 \mathrm{μs}\left({\color{gray}1.40 \mathrm{\%}}\right) $$
link_by_source_by_property traversal_paths=0 0 $$108 \mathrm{ms} \pm 466 \mathrm{μs}\left({\color{gray}2.74 \mathrm{\%}}\right) $$
link_by_source_by_property traversal_paths=255 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true $$138 \mathrm{ms} \pm 583 \mathrm{μs}\left({\color{gray}2.00 \mathrm{\%}}\right) $$
link_by_source_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false $$116 \mathrm{ms} \pm 576 \mathrm{μs}\left({\color{gray}2.44 \mathrm{\%}}\right) $$
link_by_source_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true $$123 \mathrm{ms} \pm 546 \mathrm{μs}\left({\color{gray}1.05 \mathrm{\%}}\right) $$
link_by_source_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true $$125 \mathrm{ms} \pm 513 \mathrm{μs}\left({\color{gray}1.23 \mathrm{\%}}\right) $$
link_by_source_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true $$128 \mathrm{ms} \pm 629 \mathrm{μs}\left({\color{gray}3.46 \mathrm{\%}}\right) $$

scenarios

Function Value Mean Flame graphs
full_test query-limited $$167 \mathrm{ms} \pm 2.40 \mathrm{ms}\left({\color{lightgreen}-11.221 \mathrm{\%}}\right) $$ Flame Graph
full_test query-unlimited $$149 \mathrm{ms} \pm 551 \mathrm{μs}\left({\color{lightgreen}-14.948 \mathrm{\%}}\right) $$ Flame Graph
linked_queries query-limited $$41.2 \mathrm{ms} \pm 237 \mathrm{μs}\left({\color{gray}1.06 \mathrm{\%}}\right) $$ Flame Graph
linked_queries query-unlimited $$570 \mathrm{ms} \pm 904 \mathrm{μs}\left({\color{gray}3.25 \mathrm{\%}}\right) $$ Flame Graph

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/apps > hash* Affects HASH (a `hash-*` app) area/apps > hash-graph area/apps type/eng > backend Owned by the @backend team

Development

Successfully merging this pull request may close these issues.

2 participants