Emit Quent events for cudf-polars memory reservation. - #24038
Emit Quent events for cudf-polars memory reservation.#24038TomAugspurger wants to merge 3 commits into
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
github-actions
Bot
added
Python
| This emits the following events: | ||
|
|
||
| - queueing | ||
| - allocating (with the Quent Processor for the current thread) |
There was a problem hiding this comment.
Just a note: making a memory reservation is async in python, so this will to be running on the asyncio event loop, concurrently with many other things. I think this is the right way to model this in Quent, but we'll want to double check that. We aren't really consuming CPU resources to run this task.
| purpose | ||
| What the memory is reserved for (e.g. ``"scan"``). Distinguishes | ||
| reservations made by a single operator. |
There was a problem hiding this comment.
I'd like to understand this better: what operators make multiple reservations?
Requiring this might not be smart. Then we'll end up with redundant info in the telemetry like "The Scan operator reserved memory for a scan."
And perhaps this could be an enum rather than an arbitrary string, to make things easier for consumers of the output.
1 participant
Description
This updates our emitted Quent telemetry to better track memory reservations. I've added a wrapper around
rapidsmpf.streaming.core.memory_reserve_or_wait.reserve_memorythat includesLike any other Quent event, this can be tied back to an individual Operator (cudf-polars Actor).