Skip to content

[v2] Implement conditional autoprompt importing#10231

Open
aemous wants to merge 10 commits intov2from
lazy-initialization-v2
Open

[v2] Implement conditional autoprompt importing#10231
aemous wants to merge 10 commits intov2from
lazy-initialization-v2

Conversation

@aemous
Copy link
Copy Markdown
Contributor

@aemous aemous commented Apr 20, 2026

Description of changes:

  • Refactored autoprompt code so that resolving the autoprompt mode does not require importing autoprompt or instantiating a AutoPromptDriver object. The mode resolution code has been moved to clidriver.py as the standalone functions resolve_auto_prompt_mode and validate_auto_prompt_args_are_mutually_exclusive. AutoPromptDriver is now only imported lazily when autoprompt mode is actually active.
  • Moved resolve_mode and validate_auto_prompt_args_are_mutually_exclusive tests from tests/unit/autoprompt/test_core.py to tests/unit/test_clidriver.py to reflect the new location of those functions.
  • Separated PrompterKeyboardInterrupt into a new minimal awscli/autoprompt/exceptions.py module, so that awscli.errorhandler can import it without pulling in awscli.autoprompt.factory (which has many top-level prompt_toolkit imports).
  • Made the Display import in ecs/monitorexpressgatewayservice.py lazy, deferring the import of prompt_toolkit_display (and therefore prompt_toolkit) until the interactive ECS monitoring display is actually used.
  • Made the imports of PTKPrompt and RequiredInputValidator from configure/sso.py in customizations/login/login.py lazy, deferring the import of configure/sso.py until the login region prompt is actually needed.
  • Made imports of StartLiveTailCommand and TailCommand in customizations/logs/__init__.py lazy, deferring the import of prompt_toolkit until the aws logs subcommand is invoked (i.e. the logs subcommand table is built).
  • Extracted ConfigureSSOCommand, ConfigureSSOSessionCommand, and their shared base class BaseSSOConfigurationCommand into a new customizations/configure/sso_commands.py module, which has no prompt_toolkit imports. This allows configure/configure.py to keep its reference to these SSO commands and defer the import of prompt_toolkit (via configure/sso.py) until aws configure sso is actually invoked.
  • Made the import of select_menu from wizard/ui/selectmenu in configure/sso.py lazy, deferring the import of wizard/ui/__init__.py (which imports prompt_toolkit) until ConfigureSSOCommand is actually instantiated.
  • Made the import of create_wizard_app from wizard/factory in wizard/devcommands.py lazy, deferring the import of the wizard UI chain (which imports prompt_toolkit) until a wizard dev command is actually run.
  • Made the import of wizard/factory in wizard/commands.py lazy by introducing a _get_runner method on TopLevelWizardCommand, deferring the import of the wizard factory and UI chain until a wizard command is actually executed.
  • Moved the import of devcommands in awscli/customizations/wizard/commands.py from module-level to within the register_wizard_commands function. Although this currently gives no behavioral difference since register_wizard_commands is always called for every command, in the future when plugin/customization initialization is only called when a relevant command is entered (lazy plugin loading), this change will lead to lazy-importing of devcommands only when it's needed.

Description of tests:

  • Successfully ran pre-prod build workflow.
  • Benchmarked before and after this patch (results below)

Benchmark Results: Performance Improvement Analysis

The following benchmarking analysis evaluates the impact of the Conditional Autoprompt feature across two standard CLI operations. Tests were performed with 2,000 samples per state to ensure statistical precision.

Tested on m7i.xlarge EC2 instance, x86 architecture, AL2023 os, 64 GB storage, 2,000 iterations each.

Consolidated Performance Gains

CLI Command Metric Before Patch After Patch Change
aws --version Mean Time 609.883 ms 528.471 ms -81.412 ms
stdev. 14.9329 ms 11.6255 ms -3.3074 ms
SE 0.3339 ms 0.2600 ms
aws sts get-caller-identity Mean Time 697.245 ms 615.703 ms -81.542 ms
stdev. 9.62246 ms 9.41794 ms -0.20452 ms
SE 0.2152 ms 0.2106 ms

Statistical Significance & Reliability

To filter out environmental noise (CPU spikes, OS scheduling), we measured the performance shift in terms of Standard Error (SE).

  • Distance in Standard Errors: The improvements represent shifts of ~243.8 SEs for aws --version and ~379 SEs for aws sts get-caller-identity.
  • P-Value: In both tests, $p < 0.001$. This indicates that the probability of these improvements being a result of random chance is effectively zero.
  • Precision: With a sample size of $n=2000$, our margin of error is extremely small ($<0.3$ ms). The performance gains are systemic and statistically "locked-in."

Visual Analysis

Average execution times compared
graph (4)

Significant time savings are shown. We can expect similar improvements across all modeled commands, and some improvement(s) to customization commands.

Probability density functions of execution time
graph (2)

aws --version execution time probability.

graph (3)

aws sts get-caller-identity execution time probability.

The probability density curves in both graphs show no overlap between the "Before" and "After" states. The narrow, tall peaks confirm that the large sample size has provided a high-confidence estimate of the true performance improvement. For the version command, we note that the peak is taller and narrower in the "After" state, signaling that the patch not only improves execution time but also reduces noise.


Summary: The feature provides a consistent and significant performance reduction (11.70-13.35%) across different command types with statistical certainty.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@aemous aemous marked this pull request as ready for review April 20, 2026 18:03
@aemous aemous requested a review from a team April 20, 2026 18:05
@aemous aemous added the v2 label Apr 20, 2026
@AndrewAsseily AndrewAsseily requested a review from hssyoo April 20, 2026 18:44
Copy link
Copy Markdown
Contributor

@hssyoo hssyoo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The measurements in the PR description show a ~5ms improvement, which is far from the ~66ms mean import time we measured. Based on these numbers, it seems highly likely that prompt_toolkit is still being imported elsewhere.

I was able to confirm this by checking sys.modules at the end of a aws --version call.

@aemous aemous requested a review from hssyoo April 20, 2026 22:04
@aemous
Copy link
Copy Markdown
Contributor Author

aemous commented Apr 20, 2026

The measurements in the PR description show a ~5ms improvement, which is far from the ~66ms mean import time we measured. Based on these numbers, it seems highly likely that prompt_toolkit is still being imported elsewhere.

I was able to confirm this by checking sys.modules at the end of a aws --version call.

Great catch! I have analyzed sys.modules repeatedly to find where to switch eager imports to lazy imports across various files, the resulting sys.modules show no more prompt_toolkit for unrelated commands, when autoprompt is off.

@aemous aemous force-pushed the lazy-initialization-v2 branch from e8d5797 to e82544d Compare April 20, 2026 22:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants