feat: profile and time deploy operations and facts#1723
Draft
wowi42 wants to merge 1 commit intopyinfra-dev:3.xfrom
Draft
feat: profile and time deploy operations and facts#1723wowi42 wants to merge 1 commit intopyinfra-dev:3.xfrom
wowi42 wants to merge 1 commit intopyinfra-dev:3.xfrom
Conversation
Add per-host timing for the prepare and execute phases of every operation, plus per-host per-fact timing. Surface the data via two new flags: - --timings: print a human-readable summary of the slowest operations and facts at the end of the run - --timings-json: emit structured timing data to stdout The total elapsed wall time is always printed as "Finished, took X" once a deploy has executed. Timestamps on log lines can be opted in via --log-timestamps or PYINFRA_LOG_TIMESTAMPS=1. Closes pyinfra-dev#1717. Supersedes pyinfra-dev#469 and pyinfra-dev#640.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds per-host timing for the prepare and execute phases of every operation, plus per-host per-fact timing. Surfaces the data via two new flags and always prints total elapsed wall time at the end of a run.
--timings: human-readable summary of the slowest operations and facts at the end of the run--timings-json: structured timing data emitted to stdout for external tooling--log-timestamps(alsoPYINFRA_LOG_TIMESTAMPS=1): wall-clock prefix on every log lineCloses #1717. Supersedes #469 and #640.
Implementation notes
StateTimingsdataclass onState.timingsaccumulates timings as the deploy progresses. Per-host data is stored under the concreteHost, with a small helper resolving gevent context proxies before storing them as dict keys.command_generator()) and execute time (the retry loop in_run_host_op)._get_factbody so the network call andprocess()parsing are both included. Multiple samples per fact key are kept since facts can be re-fetched.--timings.Example
Test plan
scripts/dev-lint.sh(ruff, ruff format, mypy, arguments sync) passesscripts/dev-test.shpasses (1675 tests)StateTimingsaccumulators, end-to-end op timing population, and JSON output shapepyinfra @local exec -- echo hi --timings,--timings-json,--log-timestamps)