GenerateRuntimeGraph fails when building runtime in source-build mode on Alpine

[omajid]

Description

The dotnet/runtime build fails when running on Alpine in source-build mode with a portable configuration:

  /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(63,5): error MSB4018: The "GenerateRuntimeGraph" task failed unexpectedly. [TargetFramework=net8.0]
  /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(63,5): error MSB4018: System.InvalidOperationException: AdditionalRuntimeIdentifier x64 was specified, which could not be found in any existing RuntimeGroup, and no parent was specified. [TargetFramework=net8.0]
  /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(63,5): error MSB4018:    at Microsoft.NETCore.Platforms.BuildTasks.RuntimeGroupCollection.AddRuntimeIdentifier(RID rid, String parent) in /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/RuntimeGroupCollection.cs:line 116 [TargetFramework=net8.0]
  /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(63,5): error MSB4018:    at Microsoft.NETCore.Platforms.BuildTasks.RuntimeGroupCollection.AddRuntimeIdentifier(RID rid, String parent) in /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/RuntimeGroupCollection.cs:line 140 [TargetFramework=net8.0]
  /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(63,5): error MSB4018:    at Microsoft.NETCore.Platforms.BuildTasks.RuntimeGroupCollection.AddRuntimeIdentifier(RID rid, String parent) in /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/RuntimeGroupCollection.cs:line 140 [TargetFramework=net8.0]
  /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(63,5): error MSB4018:    at Microsoft.NETCore.Platforms.BuildTasks.RuntimeGroupCollection.AddRuntimeIdentifier(String runtimeIdentifier, String parent) in /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/RuntimeGroupCollection.cs:line 39 [TargetFramework=net8.0]
  /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(63,5): error MSB4018:    at Microsoft.NETCore.Platforms.BuildTasks.GenerateRuntimeGraph.AddRuntimeIdentifiers(ICollection`1 runtimeGroups) in /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/GenerateRuntimeGraph.cs:line 327 [TargetFramework=net8.0]
  /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(63,5): error MSB4018:    at Microsoft.NETCore.Platforms.BuildTasks.GenerateRuntimeGraph.Execute() in /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/GenerateRuntimeGraph.cs:line 157 [TargetFramework=net8.0]
  /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(63,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [TargetFramework=net8.0]
  /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(63,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [TargetFramework=net8.0]

The problem is most visible with .NET 6, because dotnet/runtime is built in portable mode as part of the end-to-end source-build process.

With .NET 7 and .NET 8, source-build only builds dotnet/runtime in non-portable mode, so this problem doesn't appear there out of the box. It only appears when building dotnet/runtime by itself, in a configuration that's not used by source-build.

Reproduction Steps

Create an alpine container using a Dockerfile

# Dockerfile suitable for building .NET on Alpine

FROM alpine:3.17

RUN apk update && \
    apk add \
    bash \
    binutils \
    clang \
    cmake \
    git \
    gcc \
    icu-dev \
    krb5-dev \
    llvm \
    libstdc++ \
    linux-headers \
    lttng-ust-dev \
    make \
    musl-dev \
    openssl-dev \
    python3 \
    zlib-dev \

Then git clone https://github.com/dotnet/runtime in the container and try and build runtime in source-build mode in the appropriate branch:

main:

./build.sh -p:TargetRid=linux-musl-x64 -c Release --restore --build --pack /p:ArcadeBuildFromSource=true -bl

release/7.0 and release/6.0

./build.sh -c Release --restore --build --pack /p:ArcadeBuildFromSource=true -bl

Expected behavior

Build works

Actual behavior

Build fails

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

[omajid]

cc @ayakael @crummel @mthalman @MichaelSimons @tmds

[tmds]

This is probably due to #77508 being merged in 6.0 for runtime, but not the changes in installer: dotnet/installer#14816.

The fix is to revert #77508 and maybe some other PRs that got merged for eliminating the portable build with 6.0.

[ayakael]

This is probably due to #77508 being merged in 6.0 for runtime, but not the changes in installer: dotnet/installer#14816.

The fix is to revert #77508 and maybe some other PRs that got merged for eliminating the portable build with 6.0.

Reverting this particular one would break a workaround we use for #73525 as TargetRid would never be passed to __DistroRid. I'm looking into this right now to propose a better fix.

[tmds]

Reverting this particular one would break a workaround we use for #73525 as TargetRid would never be passed to __DistroRid

The fix for this can backported.

[ayakael]

tmds

Could you point to the fix for this so I can backport?

[omajid]

This is probably due to #77508 being merged in 6.0 for runtime, but not the changes in installer: dotnet/installer#14816.

I am not sure I follow. I ran into this issue when building dotnet/runtime standalone. How is dotnet/installer involved?

[tmds]

I am not sure I follow. I ran into this issue when building dotnet/runtime standalone.

SourceBuild.props no longer worked standalone with the changes for the non-portable build. That is fixed on main.

If you want to make it work standalone on 6.0 as-is, you can add /p:BaseOS=.

Could you point to the fix for this so I can backport?

I think that would be #80901 and #81497

[ayakael]

This is probably due to #77508 being merged in 6.0 for runtime, but not the changes in installer: dotnet/installer#14816.

I am not sure I follow. I ran into this issue when building dotnet/runtime standalone. How is dotnet/installer involved?

Applying the following diff yields the same behaviour, as well:

diff --git a/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj b/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj
index 84d4bb88604..9fc8774c083 100644
--- a/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj
+++ b/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj
@@ -18,7 +18,7 @@
     <PackageDescription>Provides runtime information required to resolve target framework, platform, and runtime specific implementations of .NETCore packages.</PackageDescription>
 
     <!-- When building from source, ensure the RID we're building for is part of the RID graph -->
-    <AdditionalRuntimeIdentifiers Condition="'$(DotNetBuildFromSource)' == 'true'">$(AdditionalRuntimeIdentifiers);$(OutputRID)</AdditionalRuntimeIdentifiers>
+    <AdditionalRuntimeIdentifiers Condition="'$(DotNetBuildFromSource)' == 'true'">linux-musl-x64</AdditionalRuntimeIdentifiers>
     <ServicingVersion>8</ServicingVersion>
     <GeneratePackageOnBuild>true</GeneratePacm kageOnBuild>
   </PropertyGroup>

Thus, this seems to have to do something with how GenerateRuntimeGraph parses AdditionalRuntimeIdentifiers. If it had to do with how OutputRid was set (or not, in the case of lack of the installer backport), this would've fixed the error.

Also, re-reading omajid's tests, they also tested against main. So, there's someting wrong with the code regardless of the backport.

I am not sure I follow. I ran into this issue when building dotnet/runtime standalone.

SourceBuild.props no longer worked standalone with the changes for the non-portable build. That is fixed on main.

If you want to make it work standalone on 6.0 as-is, you can add /p:BaseOS=.

Could you point to the fix for this so I can backport?

I think that would be #80901 and #81497

Awesome, thank you for pointing that out :)

[tmds]

Thus, this seems to have to do something with how GenerateRuntimeGraph parses AdditionalRuntimeIdentifiers. If it had to do with how OutputRid was set (or not, in the case of lack of the installer backport), this would've fixed the error.

There were other PRs backported to 6.0 for eliminating portable build, like #77510.

I think we should start by reverting those, and seeing if there is still an issue to be solved.

Also, re-reading omajid's tests, they also tested against main. So, there's someting wrong with the code regardless of the backport.

afaik everything works well on main.