Skip to content

fix(profiling): fix Lock dropping all samples when gevent monkey-patches#18261

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 6 commits into
mainfrom
vlad/lock-fix-gevent-monkey-patched
May 27, 2026
Merged

fix(profiling): fix Lock dropping all samples when gevent monkey-patches#18261
gh-worker-dd-mergequeue-cf854d[bot] merged 6 commits into
mainfrom
vlad/lock-fix-gevent-monkey-patched

Conversation

@vlad-scherbich
Copy link
Copy Markdown
Contributor

@vlad-scherbich vlad-scherbich commented May 26, 2026

Description

Bug

When gevent.monkey.patch_all (or patch_thread) runs after the profiler has started, it replaces threading's lock primitives in-place with gevent's cooperative lock types, destroying the profiler's _LockAllocatorWrapper. This causes the lock profiler to silently drop all lock profiling samples for the rest of the process lifetime.

This is a follow-up to the module-cloning bug fixed in PR #16775, where cleanup_loaded_modules re-imported threading; this bug is related to gevent's monkey-patching and is a separate code path that overwrites lock attributes after the re-import fix runs.

Fix

Adds a class-level hook on LockCollector that wraps gevent.monkey.patch_all and gevent.monkey.patch_thread with post-callbacks. After gevent finishes replacing lock types, the hook checks if any active collector's _LockAllocatorWrapper was overwritten and re-applies the profiler patch.

Testing

BEFORE

Screenshot 2026-05-27 at 9 14 21 AM

AFTER

Screenshot 2026-05-27 at 9 15 26 AM

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

Codeowners resolved as

ddtrace/profiling/collector/_lock.pyx                                   @DataDog/profiling-python
releasenotes/notes/fix-lock-profiler-gevent-monkey-patch-12dfaebf75e05535.yaml  @DataDog/apm-python
tests/profiling/collector/test_threading.py                             @DataDog/profiling-python

@datadog-prod-us1-5
Copy link
Copy Markdown
Contributor

datadog-prod-us1-5 Bot commented May 26, 2026

Pipelines  Tests

Fix all issues with BitsAI

⚠️ Warnings

🚦 8 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | build linux serverless: [arm64, cp315-cp315, v113741357-d2b8243-manylinux2014_aarch64, 1]   View in Datadog   GitLab

🔧 Fix in code (Fix with Cursor). NotImplementedError: This version of CPython is not supported yet.

DataDog/apm-reliability/dd-trace-py | build linux: [amd64, cp315-cp315, v113741238-d2b8243-manylinux2014_x86_64]   View in Datadog   GitLab

🔧 Fix in code (Fix with Cursor). NotImplementedError: This version of CPython is not supported yet

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v113741238-d2b8243-manylinux2014_x86_64, 1]   View in Datadog   GitLab

🛟 This job is unlikely to succeed on retry. Please review your pipeline configuration. NotImplementedError: This version of CPython is not supported yet

View all 8 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: eed9169 | Docs | Datadog PR Page | Give us feedback!

@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented May 26, 2026

Benchmarks

Benchmark execution time: 2026-05-26 03:58:39

Comparing candidate commit 7c996de in PR branch vlad/lock-fix-gevent-monkey-patched with baseline commit 1777513 in branch main.

Found 0 performance improvements and 5 performance regressions! Performance is the same for 617 metrics, 10 unstable metrics.

scenario:iastaspects-lstrip_aspect

  • 🟥 execution_time [+71.356µs; +76.646µs] or [+25.749%; +27.659%]

scenario:iastaspects-translate_aspect

  • 🟥 execution_time [+52.625µs; +59.714µs] or [+10.496%; +11.909%]

scenario:iastaspectsospath-ospathbasename_aspect

  • 🟥 execution_time [+100.172µs; +110.350µs] or [+23.669%; +26.074%]

scenario:span-start

  • 🟥 execution_time [+1.516ms; +1.663ms] or [+9.691%; +10.630%]

scenario:telemetryaddmetric-1-count-metric-1-times

  • 🟥 execution_time [+255.886ns; +298.012ns] or [+11.882%; +13.838%]

@vlad-scherbich vlad-scherbich force-pushed the vlad/lock-fix-gevent-monkey-patched branch from 7c996de to 9952009 Compare May 27, 2026 13:59
@vlad-scherbich vlad-scherbich marked this pull request as ready for review May 27, 2026 14:12
@vlad-scherbich vlad-scherbich requested review from a team as code owners May 27, 2026 14:12
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d0ff52d3c3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread ddtrace/profiling/collector/_lock.pyx Outdated
Comment thread ddtrace/profiling/collector/_lock.pyx
Comment thread ddtrace/profiling/collector/_lock.pyx
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a profiling reliability issue where gevent.monkey.patch_all() / patch_thread() can overwrite the lock-profiler’s patched lock allocators after the profiler is already running, causing lock samples to be dropped.

Changes:

  • Add a LockCollector class-level mechanism to detect/wrap gevent.monkey.patch_all / patch_thread and re-apply lock patches if they were overwritten.
  • Add subprocess tests that simulate gevent’s in-place replacement behavior (plus an optional integration test when gevent is available).
  • Add a release note describing the fix.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
ddtrace/profiling/collector/_lock.pyx Adds gevent monkey-patch wrapping and repatching logic tied to active lock collectors.
tests/profiling/collector/test_threading.py Adds new subprocess tests simulating gevent lock replacement and verifying repatching behavior.
releasenotes/notes/fix-lock-profiler-gevent-monkey-patch-12dfaebf75e05535.yaml Documents the fix in release notes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread ddtrace/profiling/collector/_lock.pyx
Comment thread tests/profiling/collector/test_threading.py Outdated
Comment thread releasenotes/notes/fix-lock-profiler-gevent-monkey-patch-12dfaebf75e05535.yaml Outdated
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment thread ddtrace/profiling/collector/_lock.pyx
@gh-worker-dd-mergequeue-cf854d gh-worker-dd-mergequeue-cf854d Bot merged commit 241958d into main May 27, 2026
532 of 534 checks passed
@gh-worker-dd-mergequeue-cf854d gh-worker-dd-mergequeue-cf854d Bot deleted the vlad/lock-fix-gevent-monkey-patched branch May 27, 2026 22:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Profiling Continous Profling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants