Skip to content

Jobs: document that GPU performance counters are unavailable #2714

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pjmalandrino wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Jobs: document that GPU performance counters are unavailable #2714

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/hub/jobs-configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,9 @@ l40sx4 4x Nvidia L40S 48 vCPU 382 GB 3200 GB 4x L40S (1
l40sx8 8x Nvidia L40S 192 vCPU 1534 GB 6500 GB 8x L40S (384 GB) $0.3917 $23.50
```

> [!NOTE]
> NVIDIA GPU performance counters are not accessible from inside a Job. Profilers that read them, such as Nsight Compute, fail with `ERR_NVGPUCTRPERM` even after attaching to the process. The restriction is enforced on the host, so it cannot be enabled from the job's image, command, or environment (see [NVIDIA's documentation](https://developer.nvidia.com/nvidia-development-tools-solutions-err_nvgpuctrperm-permission-issue-performance-counters)). Tools that do not read performance counters, such as `nvidia-smi` and CUDA events, work normally.

## Expose Ports

Jobs can expose container ports through the public jobs proxy using `--expose <port>` (CLI) or `expose=[<port>]` (Python API). Each exposed port is reachable at `https://<job_id>--<port>.hf.jobs` and requires an HF token with `read` access to the job's namespace:
Expand Down