[3.0] Add multi-platform generation and finish initial OpenXR bindings (#2628)

* Start of feature/openxr-multiplatform-generation

* Replace SkipScrapeIf with ScrapeIf to avoid double negative

* Add CacheKey option to ClangScraper config

* Move ScrapeIf to top of Microsoft job ClangScraper config

* Lock each job's ClangScraper to Windows to ensure generation consistency

* Sort documents before adding to project to ensure consistent ordering between scraped/cached results

* Initial attempt at multi-platform RSPs

* Fix Vulkan includes

* Get OpenXR scraping working on Linux

* Get OpenXR scraping working on Windows

* Fix incorrect Silk.NET.SilkTouch.Jobs path

* Begin working on Bindings Regeneration workflow

* Initialize submodules in workflow

* Ensure correct exit code if scraping fails

* Try to install PowerShell

* Add additional error logging

* Try Console.WriteLine instead?

Not sure why CI is hiding the error. Logger probably isn't getting flushed, but idk why WriteLine wouldn't.

* Also use a specific number as the exit code (temporary)

* Add more logging

* Also run on Linux and Mac to see if I get different errors

* Change the exit code back to 1

* Ensure scrape step exits with code 0

* Upload the .silktouch cache as an artifact

* Set include-hidden-files: true since .silktouch is considered hidden

* Add Mac OpenXR cache

* Csharpier format

* Edit doc comment

* Cleanup

* Remove unused field

* Use discard var for unused return

* Edit docs and rename the BakeSourceSets.Rewriter.Visit methods to Report

This is so I don't confuse the Visit methods with the actual Visit methods
defined by the base class.

* Remove unnecessary null check (impossible for SourceSet to be null at this point)

* Work on documenting BakeSourceSets

* Move non-null assertion to where the value comes from for clarity/correctness

* Remove null is returned comments since null is actually not returned

* Begin rewriting BakeSourceSets to fully create the BakeSets before merging

* Add IBakeStrategy.Cleanup

* Replace BakeMember with Merge

* Remove used of Humanizer in DefaultBakeStrategy

* Temporarily output the bakesets extracted by BakeSourceSets.Visitor

* Add initial Bake method

* Add docs about intended usage of IBakeStrategy methods

* Identify which sets need to be merged

* Attempt to implement bakeset merging

* Fix duplicate members

* Update error message to refer to the correct profile

* Always attempt to merge regardless if there is an existing member

This should the original behavior prior to this PR, meaning that functions are
now properly kept. However, I don't think this merges fields properly since the
my rewrite adds an additional case that the original DefaultBakeStrategy didn't
handle.

* Fix non-deterministic iteration

* Rename variables and add todo comments

* Fix issue we read the members from the node we just cleared and rename first/second to existing/incoming

* Ignore static fields when checking for struct field ABI

* Regenerate OpenXR bindings on Linux and rename xr source set to OpenXR for diff purposes

* Rename output source set back to "xr"

* Regenerate OpenXR on Linux for source set rename

* Rename rest of variables for consistency with existing/incoming naming convention

* Fix incorrect Concat with self

* Check ClassOrStructKeyword instead of Modifiers for if a record is a struct

* Fix incorrect base list concat

* Fix possible missing namespace for top level delegate types

* Add GetTypeDiscrim to reduce code dupe between the BaseType/Delegate cases

* Fix missing AddApiProfiles attributes

* Add combine caches step

* Remove setup powershell step (not needed for public runners)

* Regenerate OpenGL on Linux

* Add setup dependencies step

* Also install libgl-dev

* Add dependencies corresponding to the other headers required by OpenXR

* Change artifact naming convention

* Add concurrency restriction for bindings-regeneration

* Add regenerate bindings step

* Fix typo in git status command

* Fix ClangScraper not failing when scrape is skipped with no fallback cache

* Add IModContext.ContinueGeneration to allow mods to early exit the job

* Work on implementing use/generate cache only runs for ClangScraper

* Add ClangScraper.ReadBindingsCacheAsync

* Implement ClangScraper support for GenerateCacheOnly runs

* Update bindings-regeneration workflow to use --use/generate-cache-only options

* Set OpenXR Mac ClangScraper to exit after generation on GenerateCacheOnly runs

* Fix missing doc comment error

* Separate IModContext disposal from bindings output

* Also init submodules for regenerate bindings step since OpenXR still needs the XML

* Don't commit if no bindings changes

* Remove "I/O not available" error and change WorkingDirectory to be non-nullable

* Fix compile error

* Add BakeSourceSetsTests.MergeEquivalent

* Update snapshots to include the doc file path if set

* Add BakeSourceSetsTests.ThrowForStructMismatch

* Also test for base type handling

* Rework BakeSourceSets to handle (null, non-null) and (non-null, null) similarly

* Attempt to validate fields (fail because parent is null)

* Also provide the parent of the merged members since modifying a node sets parent to null

* Fix struct field not being validated

* Add BakeSourceSetsTests.ThrowForStructMismatch3

* CSharpier format

* Scope field validation back to only structs

Reason was because GL_ACTIVE_PROGRAM_EXT differs and that becomes a constant field defined in GL.

* Fix incorrect value for GL_ACTIVE_PROGRAM_EXT

* Remove unused property in AddVTables

* Change SourceProject to be non-nullable

* Remove duplicate GetCommittedFileAsync call and add await using to the remaining one

* Remove need to new() trivial configs by making the config param nullable

* Require field types and initializers to always exactly match

* Commit the OpenGL cache on Linux (didn't want to switch devices)

* Create builds.json

* Move native workflow sh scripts to new workflows folder

* Add workflows/bindings folder and work on matrix generation

* Implement workflow-matrix script

* Attempt to implement matrix dispatch in workflow

* Probably should checkout the repo

* Fix incorrect path

* Use Write-Output instead

* Use matrix.include instead

* Update checkout action version

* Add regenerateMatrix

* Use regenerateMatrix in bindings regen workflow

* Add push step

* Use actions checkout v7 (Rider lied)

* Pull before push

* Update .silktouch cache for 10a31e1

* Dedupe inputs passed to workflow-matrix.ps1

* Work on comment trigger

* Go back to workflow dispatch for now since I have to merge to test the workflow

* Format native.yml

* Use matrix.include instead of flat

* Add job existence check

* Remove old native workflow scripts

* Add new workflow-parse for native workflow

* Prefer env vars instead of cli args

* Move all inputs to the top for clarity

* Update checkout actions in native.yml

* Remove native config env from native.yml

* Rename prerequisites job to dispatch

* Update dispatch job names

* Add new workflow-matrix.ps1 for native workflow

* Update native.yml workflow to work with new ps1 scripts

* Resolve job names to their canonical names

* Fix a few issues

* Use a header instead for the PR comment

* Update action versions

* Restore the conditional comment behavior in native.yml

* Title case the heading

* Update .silktouch cache for 7d1cc1e

* Add todos

* Check if any matrix jobs are requested and skip if none

* Update GuillaumeFalourd/setup-windows10-sdk-action

* Fix typo and accidental overwrite of mustComment variable

* Update BakeSourceSets section in generator mods docs

* Add build definitions for all bindings regeneration jobs

* Add all jobs to hardcoded bindings regeneration command

* Update .silktouch cache for 678a4ce

* Regenerate SDL bindings for 678a4ce

* Rename entrypoint header files for simplicity/clarity

* Disable OpenXR DirectX integration for now

* Rename bindings output folders for consistency

* Update .silktouch cache for 143b869

* Regenerate OpenCL bindings for 143b869

* Regenerate SDL bindings for 143b869

* Regenerate Vulkan bindings for 143b869

* Ensure cache archives are deterministic

* Update .silktouch cache for 9b07fc2

* Move OpenXR Windows disable to be in Jobs.OpenXR.cs instead

* Add remaps for Vulkan types in OpenXR bindings and add Vulkan as compile time dep

* Change the disable to be in the openxr-windows header instead

Disabling ClangScraper breaks CI since CI expects the cache file

* Update .silktouch cache for 08e1417

* Regenerate OpenXR bindings for 08e1417

* Use lowercase sdl3 for rsps for consistency

* Limit regeneration to SDL for now

* Remove unnecessary null check

* Revert MSBuildModContext project handling changes for testing

* Update .silktouch cache for 54520f2

* Regenerate SDL bindings for 54520f2

* Revert "Revert MSBuildModContext project handling changes for testing"

This reverts commit 54520f2.

* Try fixing SDL test project bug

* Revert "Try fixing SDL test project bug"

This reverts commit ef12aa4.

* Set modifiedSolution to originalSolution instead for consistency

* Add sanity check

* Temporary: Disable everything after TransformEnums for SDL

* Fix commit error when no cache was modified

* Revert "Temporary: Disable everything after TransformEnums for SDL"

This reverts commit cc0a4c9.

* Dump test project doc states

* Temporary: Add more logs

* Revert "Temporary: Add more logs"

This reverts commit 4d28a14.

Revert "Dump test project doc states"

This reverts commit 137bed0.

* Temporary: Add more logs

* Remove old logs and add new logs

* Add dotnet restore step to workflow

* Also add dotnet workload restore

* Regenerate SDL bindings for f2b9de3

* Remove logs added to fix SDL test generation issue

* Revert "Rename bindings output folders for consistency"

This reverts commit 143b869.

* Update .silktouch cache for 9191d8c

* Regenerate SDL bindings for 9191d8c

* Re-enable generation of all bindings

* Update .silktouch cache for f970a6d

* Regenerate OpenCL bindings for f970a6d

* Regenerate Vulkan bindings for f970a6d

* Map Vulkan types used in OpenXR to their primitives

* Temporary: Rename xr folder to OpenXR to avoid diff noise

* Re-enable OpenXR bindings scraping on Windows

* Update .silktouch cache for f10177d

* Regenerate OpenXR bindings for f10177d

* Add type mappings for OpenXR Windows types

Type mappings were based on the Silk 2 mappings used here: https://github.com/dotnet/Silk.NET/blob/fd0c2871dd08019b9e6a166f13f7bfe44fb4f11a/src/OpenXR/Silk.NET.OpenXR/Structs/GraphicsRequirementsD3D11KHR.gen.cs#L65C1-L70

* Update .silktouch cache for d52ecdc

* Regenerate OpenXR bindings for d52ecdc

* Add comment on what the timestamp used by cache file entries means

* Remove commit message handling from bindings-regeneration commit message

The commit message wasn't being populated and I don't think it's worth retrieving it from it logs.

* Remove another unnecessary null check

* Add additional docs to IBakeStrategy

* Add doc comment on why parent syntax nodes are provided

* Update ClangSharp

* Update .silktouch cache for 5820cfa

* Regenerate OpenCL bindings for 5820cfa

* Regenerate OpenXR bindings for 5820cfa

* Remove another unnecessary null check

* Remap handle types external to OpenXR to void*

I'm not actually sure if this works since it's not ClangScraper creating these...

* Update .silktouch cache for dba7450

* Regenerate OpenXR bindings for dba7450

* Map the handle pointer to void* instead

* Update .silktouch cache for bd8b6fc

* Regenerate OpenXR bindings for bd8b6fc

* Try with the _T suffix

* Update .silktouch cache for 70f0375

* Regenerate OpenXR bindings for 70f0375

* Reapply "Rename bindings output folders for consistency"

This reverts commit 9191d8c.

* Update .silktouch cache for 7672cc1

* Regenerate OpenCL bindings for 7672cc1

* Regenerate SDL bindings for 7672cc1

* Regenerate Vulkan bindings for 7672cc1

* OpenXR: Rename output source set to xr and use xr as input directory for AddApiProfiles

* Vulkan: Fix incorrect AddApiProfiles.SourceSubdirectory

* Regenerate OpenXR bindings for 9255ab1

* Regenerate Vulkan bindings for 9255ab1

* [3.0] Add PR comment based native/bindings workflow (Part 1/2) (#2635)

* Support wildcard for specifying all jobs

(cherry picked from commit ed742c6b585d2bf7e736759c7a08d8f7fe71de08)

* Disable bindings/native workflows now that they have been reimplemented on main

* Update native build reminder wording

* Include required submodules as part of native/workflow-matrix output

* Remove submodule init from native build scripts (now handled by CI)

* Revert "Remove submodule init from native build scripts (now handled by CI)"

This reverts commit 127c3de.

* Remove submodules from native/workflow-matrix output since native submodules need more specific handling

Their build system might rely on git history, etc.

* Add workflow-affected-comment and workflow-dependencies native scripts

* Add similar workflow deps/comment scripts for bindings

* Add ability to specify additional dependencies per job

* Add dependencies to each bindings job

* Add script for testing workflow scripts

* Add snapshot testing to workflows scripts

* Adjust snapshot output, but apparently pwsh iteration order isn't stable

* Use ordered sets

* Work on check affected workflow

* Add artifact upload steps

* Only trigger check affected on synchronize

* Wrap all job definitions with a jobs object

* Add support for shared dependencies

* Fix dependency paths format

* Remove unnecessary env var

* Fix issue where workflow-reminder did not parse JSON

(cherry picked from commit 75bada7)

* Remove old code

* Also output the message to logs for debugging

* Fix wrong script

Probably explains why it didn't work doesn't it

* Always upload an artifact since download artifact fails on missing artifact

(cherry picked from commit 9b76580)

* Revert "Always upload an artifact since download artifact fails on missing artifact"

This reverts commit f69e6bd.

* Also upload PR number

(cherry picked from commit 8098981)

* Use overwrite: true

(cherry picked from commit 80fe34c)

* Adjust reminder message wording

* Adjust formatting to avoid ugly line breaks

(cherry picked from commit a9a8743)

* Adjust wording again

* Format publish site

* Format dotnet workflow

* Update snapshot

* Clarify that push permissions are required to run the command

* Edit build-system.md wording/formatting

* Update changelog now that OpenXR bindings are complete

Still untested though.

* Further edit wording in build-system.md

* Work on writing the new natives/bindings workflow doc section

* Finish updating the build-system docs to explain the new native/bindings workflows

* Add also a "Creating new bindings projects" section

* Remove "isolated" job that outputs the PR number since it's not actually isolated

* Use matrix in check-affected.yml

* Use Silk.NET instead of Silk in reminder comment and update snapshot

* Shorten workflow script names since the folder name implies that they are workflow scripts

* Work on prerequisites scripts and alphabetize test calls

* Enable snapshot testing for prerequisites script

* Fix duplicate lines in test snapshot

* Implement native/prerequisites.ps1

* Remove submodule inits from the individual build scripts since they are handled centrally now

* Remove todos since we now validate the PR number given on the workflow_run side

* Remove push permission requirement since we decided it was unnecessary

* Add step for installing dependencies and update snapshot

* cd to repo root to avoid relative path issues

* Fix $command not being ran

* Reapply "Rename bindings output folders for consistency"

This reverts commit 9191d8c.

* Fix extra " \"

* Don't use \ at the end of a line at all

Apparently you don't need them for GitHub and Discord doesn't render them properly.

* Use ubuntu-latest in dotnet.yml test job for consistency

I think literally everything else uses latest instead of a locked version and there's seemingly no reason not to.
I'm guessing this was a copy-paste/typo.

---------

Co-authored-by: The Silk.NET Automaton <9011267+dotnet-bot@users.noreply.github.com>

Author: Exanite

.github/workflows/check-affected.yml

@@ -0,0 +1,91 @@
+name: Check Affected Natives/Bindings
+
+on:
+  pull_request:
+    types:
+      - synchronize
+
+permissions:
+  contents: read
+
+concurrency:
+  group: "${{ github.workflow }}-${{ github.event.number }}"
+  cancel-in-progress: false
+
+jobs:
+
+  # Unprivileged - Executes untrusted code
+  check:
+    name: Check affected (${{ matrix.type }})
+    runs-on: ubuntu-latest
+
+    strategy:
+      fail-fast: false
+      matrix:
+        type:
+          - native
+          - bindings
+
+    steps:
+
+      - name: Checkout repo
+        uses: actions/checkout@v7
+        with:
+          submodules: false
+          path: repo
+
+      - name: Identify job dependencies
+        id: dependencies
+        shell: pwsh
+        working-directory: repo/eng/workflows/${{ matrix.type }}
+        run: ./dependencies.ps1 >> $env:GITHUB_OUTPUT
+
+      - name: Determine which jobs need to be reran
+        id: affected
+        uses: dorny/paths-filter@v4
+        with:
+          filters: ${{ steps.dependencies.outputs.workflowFilters }}
+
+      - name: Generate reminder message
+        id: reminder
+        shell: pwsh
+        working-directory: repo/eng/workflows/${{ matrix.type }}
+        env:
+          AFFECTED_JOBS: ${{ steps.affected.outputs.changes }}
+        run: |
+          # Duplicate is to output to logs
+          ./reminder.ps1
+          ./reminder.ps1 >> $env:GITHUB_OUTPUT
+
+      - name: Output reminder to file
+        if: ${{ steps.reminder.outputs.comment }}
+        shell: pwsh
+        env:
+          COMMENT: ${{ steps.reminder.outputs.comment }}
+        run: $env:COMMENT > ${{ matrix.type }}.txt
+
+      - name: Upload reminder
+        if: ${{ steps.reminder.outputs.comment }}
+        uses: actions/upload-artifact@v7
+        with:
+          name: ${{ matrix.type }}-reminder-comment
+          path: ${{ matrix.type }}.txt
+
+  prNumber:
+    name: Output PR number
+    runs-on: ubuntu-latest
+
+    needs: [ check ]
+
+    steps:
+
+      - name: Output PR number
+        shell: pwsh
+        run: ${{ github.event.number }} > pr.txt
+
+      - name: Upload PR number
+        uses: actions/upload-artifact@v7
+        with:
+          name: pr-number
+          path: pr.txt
+          overwrite: true