Outage on aix72-ppc64 host(s)

[aduh95]

00:04:28.132 gmake[2]: *** [tools/v8_gypfiles/abseil.target.mk:264: /home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-ppc64/out/Release/obj.target/tools/v8_gypfiles/libabseil.a] Error 2
00:04:28.132 gmake[2]: *** Waiting for unfinished jobs....
00:04:28.172 ar: No such file or directory
00:04:28.172 ar: 0707-124 The object file ares_getenv.o is corrupted.
00:04:28.172 	Symbol table entries for ares_getenv.o can not be added to the archive file.
00:04:28.198 ar: Error 0
00:04:28.198 ar: 0707-124 The object file ares_set_socket_functions.o is corrupted.
00:04:28.198 	Symbol table entries for ares_set_socket_functions.o can not be added to the archive file.
00:04:28.272 ar: Error 0
00:04:28.272 ar: 0707-124 The object file ares_htable_vpvp.o is corrupted.
00:04:28.272 	Symbol table entries for ares_htable_vpvp.o can not be added to the archive file.
00:04:28.377 ar: Error 0
00:04:28.377 ar: 0707-124 The object file ares_str.o is corrupted.
00:04:28.377 	Symbol table entries for ares_str.o can not be added to the archive file.
00:04:28.410 gmake[2]: *** [deps/cares/cares.target.mk:242: /home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-ppc64/out/Release/obj.target/deps/cares/libcares.a] Error 4
00:04:29.233 rm 1106dd2ba945a0d9d0090cb9141acb97dbd092fdd7bc46e17a77101e0f2b561e.intermediate
00:04:29.238 gmake[1]: *** [Makefile:143: node] Error 2
00:04:29.238 gmake: *** [Makefile:629: build-ci] Error 2
00:04:29.293 Build step 'Execute shell' marked build as failure

Source: https://ci.nodejs.org/job/node-test-commit-aix/62562/nodes=aix72-ppc64/console

Looking at https://ci.nodejs.org/computer/test%2Dosuosl%2Daix72%2Dppc64%5Fbe%2D4/builds, all builds have been failing for the past 24 hours.

[targos]

@sxa

[sxa]

That's weird. I'm not near a computer today but will try and take a look this evening.

[sxa]

That's odd unless the workspaces are somehow corrupted though since those should be filed generated by the build I believe 🤔

[targos]

I can SSH on the machine if you want me to try or check something

[sxa]

Hard to say what to check. Maybe look at one of the files it says are corrupted. Check it's not zero size and confirm if 'file' shows that it's valid.

I presume from the "all build are failing" that is occurring across multiple machines and versions?

@abmusse was also going to look at getting cnake/rust installed on the OSUOSL ones to bring them in sync with some of the other AIX agents of those machines but I'm not sure he'd started looking at that yet.

[targos]

There's almost nothing under out/Release/obj. The files referenced by the error messages don't exist. There's only one online machine with the https://ci.nodejs.org/label/aix72-ppc64/ label.

[aduh95]

Would it make sense to temporarily remove that platform while this is being investigated?

[abmusse]

There are 2 sub-projects under node-test-commit-aix

• aix-72-power9
  This looks to be building just fine - one machine is offline for Java update https://ci.nodejs.org/label/aix72-power9/

• aix72-ppc64
  Thse builds are having issues and as @targos mentioned - only one of these machines is online https://ci.nodejs.org/label/aix72-ppc64/

I know @sxa had issues bootstrapping some of the down OSUL machines and mentioned that here: #4326

I will work on applying the manual steps required on these (installing Clang backend) so ansible can bootstrap.

Would it make sense to temporarily remove that platform while this is being investigated?

Maybe it makes sense to disable the subproject https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix72-ppc64/ that is having issues but we should keep the working subproject up https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix72-power9/.

Does https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix72-ppc64/ run a build on every PR?

[sxa]

The playbook execution is unlikely to fix whatever it is. Just taking my brains trying to think what might have triggered it. Jenkins agent JVM had been updated to Java 25 but that should have been the only change.

I'll be able to log on shortly and see if I can reproduce it from the command line on an affected machine ...

[sxa]

-3 seems to be missing its /home filesystem. This has occurred on machines before ...
-4 (the one that has run all the recent ones) has an issue writing to its workspace directory - the error can be replicated as follows:

iojs@nodejs04:[/home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-ppc64/out] g++-10 -o /home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-ppc64/out/Release/obj.target/abseil/deps/v8/third_party/abseil-cpp/absl/crc/internal/crc_cord_state.o ../deps/v8/third_party/abseil-cpp/absl/crc/internal/crc_cord_state.cc '-D_GLIBCXX_USE_CXX11_ABI=1' '-D_FILE_OFFSET_BITS=64' '-DNODE_OPENSSL_CONF_NAME=nodejs_conf' '-DICU_NO_USER_DATA_OVERRIDE' '-D__STDC_FORMAT_MACROS' '-DOPENSSL_NO_PINSHARED' '-DOPENSSL_THREADS' '-DV8_TARGET_ARCH_PPC64' '-DV8_TARGET_ARCH_PPC_BE' '-D_LINUX_SOURCE_COMPAT=1' '-D_ALL_SOURCE=1' -I../deps/v8/third_party/abseil-cpp  -pthread -Wno-unused-parameter -maix64 -Wno-attributes -Wno-strict-overflow -Wno-return-type -Wno-int-in-bool-context -Wno-deprecated -Wno-stringop-overflow -Wno-stringop-overread -Wno-restrict -Wno-array-bounds -Wno-nonnull -Wno-dangling-pointer -flax-vector-conversions -ffp-contract=off -mcpu=power5+ -mfprnd -mno-popcntb -maix64 -fdollars-in-identifiers -fno-extern-tls-init -O3 -fno-omit-frame-pointer -fdata-sections -ffunction-sections -O3 -fno-rtti -fno-exceptions -fno-strict-aliasing -std=gnu++20 -Wno-invalid-offsetof -MMD -MF /home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-ppc64/out/Release/.deps//home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-ppc64/out/Release/obj.target/abseil/deps/v8/third_party/abseil-cpp/absl/crc/internal/crc_cord_state.o.d.raw   -c
Assembler:
/tmp//ccbwLvqp.s: line 4071: Cannot create /home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-ppc64/out/Release/obj.target/abseil/deps/v8/third_party/abseil-cpp/absl/crc/internal/crc_cord_state.o
iojs@nodejs04:[/home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-ppc64/out] mkdir /home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-ppc64/out/Release/obj.target/abseil/deps/v8/third_party/abseil-cpp/absl/crc/internal
iojs@nodejs04:[/home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-ppc64/out] g++-10 -o /home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-ppc64/out/Release/obj.target/abseil/deps/v8/third_party/abseil-cpp/absl/crc/internal/crc_cord_state.o ../deps/v8/third_party/abseil-cpp/absl/crc/internal/crc_cord_state.cc '-D_GLIBCXX_USE_CXX11_ABI=1' '-D_FILE_OFFSET_BITS=64' '-DNODE_OPENSSL_CONF_NAME=nodejs_conf' '-DICU_NO_USER_DATA_OVERRIDE' '-D__STDC_FORMAT_MACROS' '-DOPENSSL_NO_PINSHARED' '-DOPENSSL_THREADS' '-DV8_TARGET_ARCH_PPC64' '-DV8_TARGET_ARCH_PPC_BE' '-D_LINUX_SOURCE_COMPAT=1' '-D_ALL_SOURCE=1' -I../deps/v8/third_party/abseil-cpp  -pthread -Wno-unused-parameter -maix64 -Wno-attributes -Wno-strict-overflow -Wno-return-type -Wno-int-in-bool-context -Wno-deprecated -Wno-stringop-overflow -Wno-stringop-overread -Wno-restrict -Wno-array-bounds -Wno-nonnull -Wno-dangling-pointer -flax-vector-conversions -ffp-contract=off -mcpu=power5+ -mfprnd -mno-popcntb -maix64 -fdollars-in-identifiers -fno-extern-tls-init -O3 -fno-omit-frame-pointer -fdata-sections -ffunction-sections -O3 -fno-rtti -fno-exceptions -fno-strict-aliasing -std=gnu++20 -Wno-invalid-offsetof -MMD -MF /home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-ppc64/out/Release/.deps//home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-ppc64/out/Release/obj.target/abseil/deps/v8/third_party/abseil-cpp/absl/crc/internal/crc_cord_state.o.d.raw   -c
iojs@nodejs04:[/home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-ppc64/out]

This may to indicate that build isn't creating the directories required.
On the basis that the solutoin is non-obvious and a subsequent reboot hasn't resolved it either I have temporarily disabled the aix72-ppc64 axis as per Antoine's suggestion.

[sxa]

-3 seems to be missing its /home filesystem. This has occurred on machines before ...

Since it's a completely differnet issue I've split off #4330 for the -3 machine which appears to have a corrupt /home file system and was unable to mount it (Spoiler: I've recreated the file system to resolve it).

I'll continue to work the problem on -4 in this issue.

[sxa]

Hmmm comparing the build referenced earlier with 62507 (Another v22 which succeeded on the same machine):

• https://ci.nodejs.org/job/node-test-commit-aix/62562/nodes=aix72-ppc64/console
• https://ci.nodejs.org/job/node-test-commit-aix/62507/nodes=aix72-ppc64/console
  The newer (failing) log has a lot of these during the compile steps (unclear at the moment if it's directly related to the problem):

grep_64: (standard input): binary file matches

[sxa]

I went in and tried to replicate the problem with one object file thatwas causing problems. When I try to recompile it directly with the same g++-12 line it built in a way that was acceptable to ar. The "bad" version and the "good" version were the same size but with differing contents. The use of ccache did not affect the output when compiling manually.

Continuing to analyse. Noting that this machine has not has it's jenkins agent updated to v25 yet so that is not a contributing factor here. I'm attempting another build after the ccache just to fully eliminate corruption there as a cause.

[sxa]

The failures in node-test-commit-aix were mostly occurring within he first ten minutes of the build starting. The current build I have (node-stress-single-test#692) after clearing the cache has been running for an hour without hitting any such problems. It therefore seems likely at this point that the problem was ccache corruption (while I tried to test this in the last comment it's possible it didn't meet the criteria somehow for pulling from the cache in my standalone test despite pulling the command from the logs). Once my current build has finished I will attempt to re-enable the aix72-ppc64 combination with osuosl machines -3 and -4 reactivated.