feat: automatic obtaining of tests packages via --project-name in command line

[NathanMalenge]

Currently, the test runner requires all packages to be listed manually in the command line or CI scripts. This makes maintenance difficult when new test packages are added to a project.

This PR introduces an automated way to resolve test packages using Metacello project baselines.:

• Added flag --project-name to the test command
• Implemented testPackagesForProject: to extract packages from the testGroup of a given project's baseline.
• Updated the packages method to merge manually specified packages with those found via the project name.

I kept the old way too by merging the two possibilities but we can think about completely deleting the parameter specifying the testPackages if it is useless

[NathanMalenge]

I replaced the hardcoded package lists with dynamic resolution from Metacello Baselines.

Instead of using Metacello... record: 'Tests' (which was my first approach), which is heavy, I chose to read the metadatas directly from MetacelloProject.

The runner now looks up the BaselineOf for the given project and extracts the packages defined in the 'Tests' group:

version := (Smalltalk at: #BaselineOfMyProject) project version.
testsGroup := version packageNamed: 'Tests' ifAbsent: [ nil ].
^ testsGroup ifNotNil: [ testsGroup includes ] ifNil: [ #() ]

[NathanMalenge]

I noticed that some projects (like Shift) use nested groups in their Baselines (e.g., the 'Tests' group points to a 'shift-tests' group instead of packages directly). My current implementation retrieves these intermediate group names instead of the final package names.

[NathanMalenge]

I think I fixed the problem but, for now, 2 kernel tests and nearly all compiler tests are red.

[NathanMalenge]

I investigated the red tests, and it actually highlights an interesting point about the modularity of the Kernel and Compiler.

When the packages were hardcoded in the Jenkinsfile, we were only running a filtered subset of the tests (only 2 packages for the Kernel, and 4 for the Compiler).

By dynamically resolving the Tests group directly from the baselines, it now identifies and executes the full perimeter: 17 test packages for the Kernel, and 8 for the Compiler.

The fails happen because these additional tests depend on classes or methods that are not loaded in the minimal image. It shows the current limits of the Kernel's modularity.

Since the dynamic resolution seems to works as intended, the next step has to be to is to improve the modularity of the Kernel/Compiler baselines.

[jecisc]

I noticed that some projects (like Shift) use nested groups in their Baselines (e.g., the 'Tests' group points to a 'shift-tests' group instead of packages directly). My current implementation retrieves these intermediate group names instead of the final package names.

The model of Metacello was a mess in the past, but with Guille we cleaned it quite a bit and it should now be easier to manipulate the model. I'm not at the office before May but if needed I can help once I'm back if needed.

This PR is a great step in order to keep things clean :)

As you said, for now we did not run all tests. Guille started to implement this new layer of testing and there are more work to do to have all kernel and compilers tests to run. It's great to see some progress here :)

Maybe as a first step this PR could use your new parameter for baseline that already launch all tests. And later you could enable all tests of kernel and compiler?

Anyway, you are extracting a lot of tests from the kernel baseline, so it will help you to fix most of the broken tests with those steps :)

[NathanMalenge]

I've added unit tests to ClapTestRunnerTest;

I am aware that these tests introduce implicit dependencies on existing projects (Shift, Calypso, etc.) to verify the baseline reading. Since the TestRunner's job is precisely to handle Baselines, I chose to use real examples already present in the image rather than mocking some baselines.

I wonder if this is an acceptable trade-off, or if we should consider creating a Fake baseline class dedicated to these tests to keep the runner modular?

[jecisc]

I've added unit tests to ClapTestRunnerTest;

I am aware that these tests introduce implicit dependencies on existing projects (Shift, Calypso, etc.) to verify the baseline reading. Since the TestRunner's job is precisely to handle Baselines, I chose to use real examples already present in the image rather than mocking some baselines.

I wonder if this is an acceptable trade-off, or if we should consider creating a Fake baseline class dedicated to these tests to keep the runner modular?

We have a lot of tests using existing stuff in the image for tests but those test often end up failing when we update the system.

If possible, it would be nice to use ClassFactoryForTestCase to generate the baseline when the test is executed. Like this, the test data will always be alligned with the test and we do not polluate our source base with a package here only for a test.

[guillep]

If possible, it would be nice to use ClassFactoryForTestCase to generate the baseline when the test is executed. Like this, the test data will always be alligned with the test and we do not polluate our source base with a package here only for a test.

No need, just write a class! :)

[NathanMalenge]

I think everything is good now

[guillep]

I'm looking at the failing tests

• the vast majority (FileReferenceTest, FileVersionnerTest, PathTest) happens because the tests depend on the memory FileSystem. We should add FileSystem-Memory to the baseline, make the dependency explicit, and factor the other baselines using it
• Other tests (e.g., FileStringTest) fail with Can't find the requested origin while resolving FileLocator home. We should debug this one!