Skip to content

add VS code system tests and manual test plans#19208

Merged
seanbudd merged 30 commits intomasterfrom
vsCodeTests
Nov 14, 2025
Merged

add VS code system tests and manual test plans#19208
seanbudd merged 30 commits intomasterfrom
vsCodeTests

Conversation

@seanbudd
Copy link
Copy Markdown
Member

@seanbudd seanbudd commented Nov 12, 2025

Link to issue number:

Part of #18762
Split from #18419

Summary of the issue:

We should expand our system tests to test against more applications.
Starting with open source applications that are easily available within GitHub actions runners,

Description of user facing changes:

None

Description of developer facing changes:

None

Description of development approach:

Perform manual tests and implement equivalent system tests where possible

Testing strategy:

See changes

Known issues with pull request:

Code Review Checklist:

  • Documentation:
    • Change log entry
    • User Documentation
    • Developer / Technical Documentation
    • Context sensitive help for GUI changes
  • Testing:
    • Unit tests
    • System (end to end) tests
    • Manual testing
  • UX of all users considered:
    • Speech
    • Braille
    • Low Vision
    • Different web browsers
    • Localization in other languages / culture than English
  • API is compatible with existing add-ons.
  • Security precautions taken.

@seanbudd seanbudd requested a review from a team as a code owner November 12, 2025 06:12
@seanbudd seanbudd marked this pull request as draft November 12, 2025 06:13
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds system tests and a manual testing plan for VS Code (and VSCodium derivatives) to expand NVDA's test coverage for applications available in GitHub Actions runners. The PR introduces automated tests for basic VS Code functionality and provides a comprehensive manual smoke testing guide for IDE accessibility verification.

  • Adds 4 new automated test functions for VS Code (sidebar toggle, command palette, file navigation, and search panel)
  • Creates a comprehensive manual testing plan covering 6 key areas of VS Code accessibility
  • Updates copyright headers and imports to support the new test infrastructure

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 8 comments.

File Description
tests/system/robot/vscodeTests.py Adds 4 new automated test functions for VS Code features; renames existing function and updates imports with type hints
tests/manual/applications/IDEs/vsCodium.md Creates new manual smoke testing plan covering application launch, file operations, editor accessibility, command palette, extensions, and terminal

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/system/robot/vscodeTests.py Outdated
Comment thread tests/system/robot/vscodeTests.py
Comment thread tests/manual/applications/IDEs/vsCodium.md Outdated
Comment thread tests/manual/applications/IDEs/vsCodium.md Outdated
Comment thread tests/system/robot/vscodeTests.py
Comment thread tests/system/robot/vscodeTests.py
Comment thread tests/system/robot/vscodeTests.py Outdated
Comment thread tests/system/robot/vscodeTests.py Outdated
seanbudd added a commit that referenced this pull request Nov 12, 2025
@seanbudd seanbudd marked this pull request as ready for review November 13, 2025 01:53
@seanbudd seanbudd requested a review from Copilot November 13, 2025 01:54
Comment thread tests/system/robot/vscodeTests.py Outdated
@seanbudd seanbudd marked this pull request as draft November 13, 2025 01:56
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 19 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/manual/applications/IDEs/vsCodium.md Outdated
Comment thread tests/system/robot/vscodeTests.py Outdated
Comment thread tests/system/robot/vscodeTests.robot Outdated
Comment thread tests/system/robot/vscodeTests.py Outdated
Comment thread tests/system/robot/vscodeTests.py Outdated
Comment thread tests/system/robot/vscodeTests.py Outdated
Comment thread tests/system/robot/vscodeTests.py Outdated
Comment thread tests/system/robot/vscodeTests.robot Outdated
Comment thread tests/system/robot/vscodeTests.robot Outdated
Comment thread tests/system/robot/vscodeTests.robot Outdated
@seanbudd seanbudd marked this pull request as ready for review November 13, 2025 04:16
@seanbudd seanbudd requested a review from Copilot November 13, 2025 05:02
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/system/robot/vscodeTests.py
Comment thread tests/system/libraries/VSCodeLib.py Outdated
Comment thread tests/manual/applications/IDEs/vsCodium.md Outdated
Copy link
Copy Markdown
Member

@SaschaCowley SaschaCowley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks fine to me, though I don't understand why we're adding accessibility tests for Codium's UI. Most, if not all, of these tests are based on Codium's implementation, not on our ARIA/Electron support. They're also going to be quite brittle and prone to break with Codeium updates.

Comment thread tests/system/robot/vscodeTests.py Outdated
@seanbudd seanbudd merged commit 57f2e9d into master Nov 14, 2025
29 checks passed
@seanbudd seanbudd deleted the vsCodeTests branch November 14, 2025 03:30
@github-actions github-actions Bot added this to the 2026.1 milestone Nov 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants