[release/6.0] Add alpine to CI for source-build by omajid · Pull Request #13074 · dotnet/installer

omajid · 2022-01-18T17:30:23Z

Please add description for changes you are making.
If there is an issue related to this PR, please add the reference.

omajid · 2022-01-18T20:32:41Z

This is the current set of errors:

##[error]/tarball/src/source-build-reference-packages.f5bc8bd3d3a2738be1095b2ca4f928504acb4801/artifacts/source-build/self/src/src/targetPacks/Directory.Build.targets(55,5): error MSB3073: (NETCORE_ENGINEERING_TELEMETRY=Build) The command "/tarball/src/source-build-reference-packages.f5bc8bd3d3a2738be1095b2ca4f928504acb4801/artifacts/source-build/self/src/src/targetPacks/build-il.sh /tarball/src/source-build-reference-packages.f5bc8bd3d3a2738be1095b2ca4f928504acb4801/artifacts/source-build/self/package-cache/runtime.linux-x64.microsoft.netcore.ilasm/6.0.0/runtimes/linux-x64/native/ /tarball/src/source-build-reference-packages.f5bc8bd3d3a2738be1095b2ca4f928504acb4801/artifacts/source-build/self/src/src/targetPacks/ILsrc/microsoft.aspnetcore.app.ref/3.0.1/ref/netcoreapp3.0/Microsoft.AspNetCore.Antiforgery.il /tarball/src/source-build-reference-packages.f5bc8bd3d3a2738be1095b2ca4f928504acb4801/artifacts/source-build/self/src/artifacts/bin/Microsoft.AspNetCore.App.Ref.3.0.1/pack/ref/netcoreapp3.0/Microsoft.AspNetCore.Antiforgery.dll" exited with code 127.

Which should get fixed by dotnet/source-build-assets#319, hopefully. Though not sure how *musl* artifacts would end up in previously-built source-built-artifacts.

omajid · 2022-02-11T16:22:17Z

Build fails because dotnet/runtime#64304 is not included in the SDK being used to build. I guess this will have to wait unit the next patch release.

omajid · 2022-03-11T17:17:26Z

Error is in building dotnet/runtime:

 Build:
 Build Environment: x64 Release Linux alpine.3.15-x64
 Project "/tarball/build.proj" (1) is building "/tarball/repos/known-good.proj" (3) on node 1 (Build target(s)).
 BuildRepoReferences:
...
 runtime ->
 Commencing CoreCLR Repo build
 __DistroRid: alpine.3.15.0-x64
 Setting up directories for build
  Checking prerequisites...

The RID should have been alpine-3.15, like it appears first in the output. The runtime build computes the wrong RID (alpine-3.15.0). Should get fixed by dotnet/runtime#62942

This also makes me wonder: if source-build knows the correct RID value, shouldn't we just pass that onto runtime instead of letting it compute a possibly different value?

ayakael · 2022-10-17T14:20:36Z

Error is in building dotnet/runtime:
 Build:
 Build Environment: x64 Release Linux alpine.3.15-x64
 Project "/tarball/build.proj" (1) is building "/tarball/repos/known-good.proj" (3) on node 1 (Build target(s)).
 BuildRepoReferences:
...
 runtime ->
 Commencing CoreCLR Repo build
 __DistroRid: alpine.3.15.0-x64
 Setting up directories for build
  Checking prerequisites...
The RID should have been alpine-3.15, like it appears first in the output. The runtime build computes the wrong RID (alpine-3.15.0). Should get fixed by dotnet/runtime#62942

This also makes me wonder: if source-build knows the correct RID value, shouldn't we just pass that onto runtime instead of letting it compute a possibly different value?

This should be fixed now that dotnet/runtime#62942 is merged and more definitively with dotnet/runtime#74504

omajid · 2023-03-29T20:24:10Z

Hey, @ayakael have you seen this error on alpine?

/runtime/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(60,5): error MSB4018: The "GenerateRuntimeGraph" task failed unexpectedly.
/runtime/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(60,5): error MSB4018: System.InvalidOperationException: AdditionalRuntimeIdentifier x64 was specified, which could not be found in any existing RuntimeGroup, and no parent was specified.
/runtime/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(60,5): error MSB4018:    at Microsoft.NETCore.Platforms.BuildTasks.RuntimeGroupCollection.AddRuntimeIdentifier(RID rid, String parent) in /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/RuntimeGroupCollection.cs:line 116
/runtime/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(60,5): error MSB4018:    at Microsoft.NETCore.Platforms.BuildTasks.RuntimeGroupCollection.AddRuntimeIdentifier(RID rid, String parent) in /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/RuntimeGroupCollection.cs:line 140
/runtime/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(60,5): error MSB4018:    at Micros oft.NETCore.Platforms.BuildTasks.RuntimeGroupCollection.AddRuntimeIdentifier(RID rid, String parent) in /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/RuntimeGroupCollection.cs:line 140
/runtime/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(60,5): error MSB4018:    at Microsoft.NETCore.Platforms.BuildTasks.GenerateRuntimeGraph.AddRuntimeIdentifiers(ICollection`1 runtimeGroups) in /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/GenerateRuntimeGraph.cs:line 325
/runtime/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(60,5): error MSB4018:    at Micros oft.NETCore.Platforms.BuildTasks.GenerateRuntimeGraph.Execute() in /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/Gene rateRuntimeGraph.cs:line 157
/runtime/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(60,5): error MSB4018:    at Micros oft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()

ayakael · 2023-03-29T21:06:50Z

Hey, @ayakael have you seen this error on alpine?

/runtime/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(60,5): error MSB4018: The "GenerateRuntimeGraph" task failed unexpectedly.
/runtime/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(60,5): error MSB4018: System.InvalidOperationException: AdditionalRuntimeIdentifier x64 was specified, which could not be found in any existing RuntimeGroup, and no parent was specified.
/runtime/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(60,5): error MSB4018:    at Microsoft.NETCore.Platforms.BuildTasks.RuntimeGroupCollection.AddRuntimeIdentifier(RID rid, String parent) in /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/RuntimeGroupCollection.cs:line 116
/runtime/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(60,5): error MSB4018:    at Microsoft.NETCore.Platforms.BuildTasks.RuntimeGroupCollection.AddRuntimeIdentifier(RID rid, String parent) in /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/RuntimeGroupCollection.cs:line 140
/runtime/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(60,5): error MSB4018:    at Micros oft.NETCore.Platforms.BuildTasks.RuntimeGroupCollection.AddRuntimeIdentifier(RID rid, String parent) in /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/RuntimeGroupCollection.cs:line 140
/runtime/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(60,5): error MSB4018:    at Microsoft.NETCore.Platforms.BuildTasks.GenerateRuntimeGraph.AddRuntimeIdentifiers(ICollection`1 runtimeGroups) in /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/GenerateRuntimeGraph.cs:line 325
/runtime/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(60,5): error MSB4018:    at Micros oft.NETCore.Platforms.BuildTasks.GenerateRuntimeGraph.Execute() in /runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/Gene rateRuntimeGraph.cs:line 157
/runtime/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj(60,5): error MSB4018:    at Micros oft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()

Yes, it was fixed by dotnet/aspnetcore#46735.

omajid · 2023-03-30T11:06:32Z

Yes, it was fixed by dotnet/aspnetcore#46735.

Strange. I am seeing the error while building runtime, not aspnetcore 😖

ayakael · 2023-03-30T11:19:46Z

Yes, it was fixed by dotnet/aspnetcore#46735.

Strange. I am seeing the error while building runtime, not aspnetcore confounded

Due to dotnet/runtime#73525, we actually manually set TargetRid when building with sourcebuild. This is maybe why I don't see this error.

omajid · 2023-03-30T12:21:53Z

Oh, the error is when building the portable version of runtime. RID is linux-musl-x64 at this point.

ayakael · 2023-03-30T12:28:48Z

Oh, the error is when building the portable version of runtime. RID is linux-musl-x64 at this point.

Apologies, early morning brain fog haha: Yeah, I encountered this since version 6.0.109. The following patch is a workaround:

From 79f02a53316f90543d60269d7c06727c376f423b Mon Sep 17 00:00:00 2001
From: Antoine Martin <dev@ayakael.net>
Date: Wed, 17 Sep 2022 18:41:08 +0000
Subject: [PATCH 1/1] No additional runtime id

For some reason, AdditionalRuntimeIdentifiers gets set as '$arch' rather than 'linux-musl-$arch'
Since we know our portable RID exists, just skip ensuring RID existence

---here's something wrong with 
diff --git a/src/runtime/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj.orig b/src/runtime/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj
index 84d4bb8..58f7e01 100644
--- a/src/runtime/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj.orig
+++ b/src/runtime/src/libraries/Microsoft.NETCore.Platforms/src/Microsoft.NETCore.Platforms.csproj
@@ -17,8 +17,6 @@
     <NoWarn>$(NoWarn);NU5128</NoWarn> <!-- No Dependencies-->
     <PackageDescription>Provides runtime information required to resolve target framework, platform, and runtime specific implementations of .NETCore packages.</PackageDescription>
 
-    <!-- When building from source, ensure the RID we're building for is part of the RID graph -->
-    <AdditionalRuntimeIdentifiers Condition="'$(DotNetBuildFromSource)' == 'true'">$(AdditionalRuntimeIdentifiers);$(OutputRID)</AdditionalRuntimeIdentifiers>
     <ServicingVersion>8</ServicingVersion>
     <GeneratePackageOnBuild>true</GeneratePackageOnBuild>
   </PropertyGroup>
-- 
2.37.1

I never took the time to debug what's wrong.

omajid · 2023-03-30T14:18:00Z

Thanks. I filed dotnet/runtime#84127 since I could reproduce this on all branches of dotnet/runtime using an alpine container.

ayakael · 2023-04-06T15:34:45Z

Thanks. I filed dotnet/runtime#84127 since I could reproduce this on all branches of dotnet/runtime using an alpine container.

To unlock this, do you want me to backport whatever fix comes out of dotnet/runtime#84127 to release/6.0 of runtime? Else, we can set /p:AddtionalRuntimeIdentifiers= from within repos/runtime-portable.proj. This workaround works for me.

ayakael · 2023-04-06T20:22:23Z

Backported: dotnet/runtime#84442

omajid · 2023-09-08T15:00:44Z

Current error is:

/bin/sh: /tmp/MSBuildTemproot/tmpf178499c4a434ba6b97e9c7129a8b137.exec.cmd: line 2: /tarball/src/aspnetcore/artifacts/source-build/self/package-cache/microsoft.netcore.app.crossgen2.linux-x64/6.0.21/tools/crossgen2: not found
/tarball/src/aspnetcore/artifacts/source-build/self/src/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj(455,5): error MSB3073: The command ""/tarball/src/aspnetcore/artifacts/source-build/self/package-cache/microsoft.netcore.app.crossgen2.linux-x64/6.0.21/tools/crossgen2" --targetarch:x64 --targetos:linux -O @"/tarball/src/aspnetcore/artifacts/source-build/self/src/artifacts/obj/Microsoft.AspNetCore.App.Runtime/Release/net6.0/linux-x64/crossgen/PlatformAssembliesPathsCrossgen2.rsp" --perfmap --perfmap-format-version:1 --perfmap-path:"/tarball/src/aspnetcore/artifacts/source-build/self/src/artifacts/bin/Microsoft.AspNetCore.App.Runtime/Release/net6.0/linux-x64/" -o:"/tarball/src/aspnetcore/artifacts/source-build/self/src/artifacts/bin/Microsoft.AspNetCore.App.Runtime/Release/net6.0/linux-x64/Microsoft.Extensions.Caching.Abstractions.dll" "/tarball/src/aspnetcore/artifacts/source-build/self/package-cache/microsoft.extensions.caching.abstractions/6.0.0/lib/netstandard2.0/Microsoft.Extensions.Caching.Abstractions.dll"" exited with code 127.

Looks like aspnetcore is incorrectly using linux-x64 crossgen instead of linux-musl-x64 crossgen.

@ayakael do you have a fix for that ?

ayakael · 2023-09-08T15:17:20Z

Yes! This is caused by dotnet/aspnetcore#39822 Backporting the portable build elimination patch should've fixed this, but as that was scrapped the problem still exists to this day when building aspnetcore with Rid linux-musl-$arch We've eliminated runtime-portable on Alpine's dotnet6. The fix before the backport was the following: https://lab.ilot.io/mirrors/aports/-/blob/87d14a808f8a227bfb7e5f0fde955eb0f15dbd03/community/dotnet6-build/build_musl-build-fix.patch It doesn't work though. It breaks build for non-musl as I never got the musl detection fix to work.

ayakael · 2023-09-09T16:09:49Z

@omajid Could you try this patch from within your PR?:

diff --git a/src/SourceBuild/tarball/content/Directory.Build.props b/src/SourceBuild/tarball/content/Directory.Build.props
index 797e54fa6..8be93da95 100644
--- a/src/SourceBuild/tarball/content/Directory.Build.props
+++ b/src/SourceBuild/tarball/content/Directory.Build.props
@@ -181,6 +181,7 @@
     <PortableRid Condition="'$(TargetOS)' == 'FreeBSD'">freebsd-$(Platform)</PortableRid>
     <PortableRid Condition="'$(TargetOS)' == 'OSX'">osx-$(Platform)</PortableRid>
     <PortableRid Condition="'$(TargetOS)' == 'Linux'">linux-$(Platform)</PortableRid>
+    <PortableRid Condition="$(TargetRid.StartsWith('linux-musl')) or $(TargetRid.StartsWith('alpine'))">linux-musl-$(Platform)</PortableRid>
     <PortableRid Condition="'$(TargetOS)' == 'Windows_NT'">win-$(Platform)</PortableRid>
     <TargetRid Condition="'$(PortableBuild)' == 'true' AND '$(PortableRid)' != ''">$(PortableRid)</TargetRid>
   </PropertyGroup>
diff --git a/src/SourceBuild/tarball/content/repos/aspnetcore.proj b/src/SourceBuild/tarball/content/repos/aspnetcore.proj
index 480f3c713..9b25187ab 100644
--- a/src/SourceBuild/tarball/content/repos/aspnetcore.proj
+++ b/src/SourceBuild/tarball/content/repos/aspnetcore.proj
@@ -7,6 +7,7 @@
     <!-- The arch flag (defaults to x64) overrides any value of TargetArchitecture that we might set -->
     <BuildCommandArgs>$(BuildCommandArgs) --arch $(Platform)</BuildCommandArgs>
     <BuildCommandArgs>$(BuildCommandArgs) --no-build-repo-tasks</BuildCommandArgs>
+    <BuildCommandArgs Condition="$(PortableRid.StartsWith('linux-musl'))">$(BuildCommandArgs) --os-name linux-musi</BuildCommandArgs>
     <BuildCommandArgs>$(BuildCommandArgs) /p:BuildNodeJs=false</BuildCommandArgs>
     <BuildCommandArgs>$(BuildCommandArgs) /p:PublishCompressedFilesPathPrefix=$(SourceBuiltAspNetCoreRuntime)</BuildCommandArgs>
     <!-- Update to 1.0.0 version of reference assemblies which are built in SBRP instead of the preview.2 version
diff --git a/src/SourceBuild/tarball/content/repos/installer.proj b/src/SourceBuild/tarball/content/repos/installer.proj
index e34337b85..9b42de398 100644
--- a/src/SourceBuild/tarball/content/repos/installer.proj
+++ b/src/SourceBuild/tarball/content/repos/installer.proj
@@ -25,7 +25,7 @@
     <BuildCommandArgs Condition="'$(TargetOS)' == 'Linux'">$(BuildCommandArgs) /p:Rid=$(TargetRid)</BuildCommandArgs>
     <BuildCommandArgs>$(BuildCommandArgs) /p:DOTNET_INSTALL_DIR=$(DotNetCliToolDir)</BuildCommandArgs>
 
-    <BuildCommandArgs Condition="'$(TargetOS)' == 'Linux'">$(BuildCommandArgs) /p:AspNetCoreInstallerRid=linux-$(Platform)</BuildCommandArgs>
+    <BuildCommandArgs Condition="'$(TargetOS)' == 'Linux'">$(BuildCommandArgs) /p:AspNetCoreInstallerRid=$(PortableRid)</BuildCommandArgs>
     <!-- core-sdk always wants to build portable on OSX and FreeBSD -->
     <BuildCommandArgs Condition="'$(TargetOS)' == 'FreeBSD'">$(BuildCommandArgs) /p:CoreSetupRid=freebsd-x64 /p:PortableBuild=true</BuildCommandArgs>
     <BuildCommandArgs Condition="'$(TargetOS)' == 'OSX'">$(BuildCommandArgs) /p:CoreSetupRid=osx-x64</BuildCommandArgs>

I can't test it locally.

ayakael · 2023-09-09T21:18:04Z

Okay, I've created #17324 that should unblock this. These changes fix building dotnet6 on Alpine Linux without eliminating runtime-portable.

mthalman

I'm testing these changes in the internal tarball build pipeline to verify things work. This doesn't get run by the PR.

mthalman · 2023-11-02T17:42:05Z


  # The following parameters aren't expected to be passed in rather they are used for encapsulation
  # -----------------------------------------------------------------------------------------------
+  alpine317Container: mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-20230214152931-7f13c75


We want this to be latest. Following the pattern of the other image names is not what we want. I'll update those separately.

Suggested change

alpine317Container: mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-20230214152931-7f13c75

alpine317Container: mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17

mthalman · 2023-11-06T13:51:54Z

@omajid - Finally got everything tested internally. I've pushed the necessary changes that will make this work.

omajid · 2023-11-06T15:11:43Z

Thanks, @mthalman !

@ayakael Any concerns with merging this?

ayakael · 2023-11-07T01:42:55Z

Thanks, @mthalman !

@ayakael Any concerns with merging this?

Looks good, thanks!

omajid force-pushed the 6.0.1xx-source-build-alpine-ci branch from c763019 to 6d414d6 Compare January 18, 2022 18:15

omajid force-pushed the 6.0.1xx-source-build-alpine-ci branch from 6d414d6 to ad828b1 Compare February 10, 2022 21:36

omajid force-pushed the 6.0.1xx-source-build-alpine-ci branch from ad828b1 to 0fb4ec6 Compare March 10, 2022 21:30

ayakael mentioned this pull request Mar 16, 2022

Initial Alpine support redhat-developer/dotnet-regular-tests#192

Merged

omajid changed the title ~~Add alpine to CI for source-build~~ [release/6.0] Add alpine to CI for source-build Aug 29, 2022

MichaelSimons added the Area-SourceBuild label Nov 10, 2022

omajid force-pushed the 6.0.1xx-source-build-alpine-ci branch from 0fb4ec6 to 01527ce Compare March 28, 2023 16:23

ayakael mentioned this pull request Apr 6, 2023

[release/6.0] Microsoft.NETCore.Platforms: support adding rids with '-' in the base part (backport #84413) dotnet/runtime#84442

Merged

carlossanlop mentioned this pull request May 15, 2023

[release/7.0-staging] Microsoft.NETCore.Platforms: support adding rids with '-' in the base part. dotnet/runtime#86282

Merged

omajid force-pushed the 6.0.1xx-source-build-alpine-ci branch from 41e1f4e to c8be71c Compare September 7, 2023 20:46

ayakael mentioned this pull request Sep 9, 2023

[release/6.0.1xx] Fix aspnetcore build on musl-libc platforms #17324

Merged

Add alpine to CI for source-build

2cda917

omajid force-pushed the 6.0.1xx-source-build-alpine-ci branch from c8be71c to 2cda917 Compare October 26, 2023 17:05

omajid marked this pull request as ready for review November 2, 2023 16:52

omajid requested a review from a team as a code owner November 2, 2023 16:52

mthalman reviewed Nov 2, 2023

View reviewed changes

mthalman mentioned this pull request Nov 3, 2023

[7.0] SDK content test for Alpine is not comparing against linux-musl-x64 version of MS SDK tarball dotnet/source-build#3714

Closed

mthalman added 2 commits November 6, 2023 07:49

Use floating image tag

51df9a0

Set MS SDK tarball to musl version for Alpine leg

c2ed503

mthalman requested a review from a team November 6, 2023 13:52

MichaelSimons approved these changes Nov 6, 2023

View reviewed changes

mthalman approved these changes Nov 7, 2023

View reviewed changes

mthalman merged commit 7905f1b into dotnet:release/6.0.1xx Nov 7, 2023

	alpine317Container: mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-20230214152931-7f13c75
	alpine317Container: mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17

Conversation

omajid commented Jan 18, 2022

Uh oh!

omajid commented Jan 18, 2022

Uh oh!

omajid commented Feb 11, 2022

Uh oh!

omajid commented Mar 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ayakael commented Oct 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

omajid commented Mar 29, 2023

Uh oh!

ayakael commented Mar 29, 2023

Uh oh!

omajid commented Mar 30, 2023

Uh oh!

ayakael commented Mar 30, 2023

Uh oh!

omajid commented Mar 30, 2023

Uh oh!

ayakael commented Mar 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

omajid commented Mar 30, 2023

Uh oh!

ayakael commented Apr 6, 2023

Uh oh!

ayakael commented Apr 6, 2023

Uh oh!

omajid commented Sep 8, 2023

Uh oh!

ayakael commented Sep 8, 2023 via email • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ayakael commented Sep 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ayakael commented Sep 9, 2023

Uh oh!

mthalman left a comment

Choose a reason for hiding this comment

Uh oh!

mthalman Nov 2, 2023

Choose a reason for hiding this comment

Uh oh!

mthalman commented Nov 6, 2023

Uh oh!

omajid commented Nov 6, 2023

Uh oh!

ayakael commented Nov 7, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

omajid commented Mar 11, 2022 •

edited

Loading

ayakael commented Oct 17, 2022 •

edited

Loading

ayakael commented Mar 30, 2023 •

edited

Loading

ayakael commented Sep 8, 2023 via email •

edited

Loading

ayakael commented Sep 9, 2023 •

edited

Loading