How to test on different operating systems?

[niccokunzmann]

Currently, we test icalendar on Ubuntu with a test matrix of different Python versions.
However, I want to keep it up-to-date and also be able to introduce breaking changes in the future. One of them was moving to zoneinfo with version 6.

These big changes lead to test failures in OS package distributions:

• Test failure on Fedora with 6.1.0 #763
• Tests fail on Alpine Linux for v6.1.3 #812
• [BUG] problem with timezone tests on NetBSD - 6.0.1 #731
• Test windows & macos #1062
• probably more to follow ...

By enabling tests for other Operating Systems, we can make sure that the updates can be shipped fast.

Questions:

• But how do we test on other OS?
• Do they exist as docker images?
• Also, making sure icalendar works on Windows and MacOS seems important.
• It seems like a huge test matrix to me. How do we make it efficient?

Your input is much appreciated, especially if you are a package distributor.

[0-wiz-0]

There's a NetBSD github action available (running inside a VM inside Ubuntu I think). An example workflow which includes installing python and a python module (from pkgsrc) are here.

[niccokunzmann]

Nice! I do not see how you run a command like echo inside netbsd. Would you consider creating a github action that runs one of the tox commands?

[0-wiz-0]

All the commands in the 'name: NetBSD test' section are run inside the NetBSD VM.

[stevepiercy]

An example of running multiple architecture, OS, and Python versions from Pyramid: https://github.com/Pylons/pyramid/blob/main/.github/workflows/ci-tests.yml#L47

Another example from plone.restapi, for running multiple OS, Python, and Plone versions: https://github.com/plone/plone.restapi/blob/main/.github/workflows/test-matrix.yml#L22-L33

Unfortunately GitHub doesn't provide direct support of NetBSD, but the suggested example from @0-wiz-0 should work. For all supported OSs and architectures, see https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories.

[limburgher]

In the Fedora build, we run: python3 -m pytest src/icalendar/tests

[gjabell]

I can't speak for other OS but most bespoke Linux distros probably have their own Dockerfile so it wouldn't be too hard to write a matrix which runs across a test set. I guess the most difficult might be adjusting to the different package managers, but it also depends whether the tests would use native packages vs. just pip installing everything.

One option might be to make it a dispatch workflow that can be run on-demand (e.g. before cutting a release), maybe having an option which OS to target. That way it would be OK if it isn't as efficient, since it wouldn't need to run every single commit / PR.

[niccokunzmann]

Hm. True. If we e.g. install icalendar with pip, we should probably install it without dependencies and prefer the installation of tzdata and dateutil through the package manager. That could assure better compliance. I would say, we can see this as an option and if this helps you as the maintainer at all, then it is worth trying it the way it helps best :)

[niccokunzmann]

Conda is another packager. They have this starting points for Linux, Windows, Mac: https://github.com/conda-forge/staged-recipes

[niccokunzmann]

WebAssembly is another test environment: #1073