Github actions overhaul

[ponte-vecchio]

The existing actions suffer from two issues which I delineate below. For each issue, I will also comment on possible fixes or changes going forward.

1. Tests take too long

The regression tests against default and supporthook are done serially, meaning there are in total 170 tests to run per action at the time of writing. This means that the duration of the actions are akin to that of latex2e, despite being a much smaller project. There is also (perhaps) a configuration mishap within the yml file itself that invokes the runner twice for the same push for the purpose of a PR. To this end, we may consider some of these options:

• Parallelise the benchmarks (splitting them up semantically, by numbers, etc.)
• Consider whether the regression test for support hooks must be run in duplicates as the default config
• Modify the github actions yml for possible speed-up/optimisation.

2. What of benchmarks?

I'm unsure what the purpose of the benchmarking is, seeing as it compares the time taken for a variety of TeXLive versions against some commits in main. Ever since the maintainership being transferred, the outputs of the benchmark test do not go anywhere, and are not accessible. In fact, #562 mentions combining the outputs for these, but it's unclear what the original author envisioned. To this end, we may consider:

• removing the benchmarking altogether,
• keep the benchmarks to the dev branch, with tweaks

---

[muzimuzhi]

I'm unsure what the purpose of the benchmarking is, seeing as it compares the time taken for a variety of TeXLive versions against some commits in main.

See

• Write some benchmark tests #480.

Ever since the maintainership being transferred, the outputs of the benchmark test do not go anywhere, and are not accessible.

The gh-pages branch held the benchmark outputs. Before, the GitHub Pages were built from gh-pages branch and deployed to <owner>.github.io/<repo>. Currently dev-version branch is used to build GitHub Pages and deployed to https://texackers.dev/tabularray, but the benchmark outputs were still on gh-pages branch.

https://github.com/TeXackers/tabularray/actions/workflows/pages/pages-build-deployment

[ponte-vecchio]

I'm unsure what the purpose of the benchmarking is, seeing as it compares the time taken for a variety of TeXLive versions against some commits in main.

See

* [Write some benchmark tests #480](https://github.com/TeXackers/tabularray/issues/480).

Thanks for your reply. However, the setup from #480 does not solve the problem we're addressing here. In fact, it was precisely #480 that this issue came to light.

Having already examined #480, the original motivation seems to be "just because", but it is still not clear what ongoing purpose it serves on main besides retrospective necessitation when some performance differences were found between 2023 and 2024.

There are also methodological concerns. For benchmarks to be meaningful, there needs to be a stable baseline/controlled environment in which they are performed. Without these being met, the numbers lack robustness and difficult to turn into actionables beyond rough ideas. Because all of the discourse around performance and benchmarks are on the firm assumption that the benchmark itself is accurately representative of capturing the nuance of performance changes, we will need to consider carefully the directionality of it.

This notion is further exacerbated by the fact that benchmarks are only performed on main. To me, benchmarks that doesn't track per-commit regressions (e.g., via git bisect) are not very meaningful. A potential move here is to move benchmarking to dev-version.

Ever since the maintainership being transferred, the outputs of the benchmark test do not go anywhere, and are not accessible.

The gh-pages branch held the benchmark outputs. Before, the GitHub Pages were built from gh-pages branch and deployed to <owner>.github.io/<repo>. Currently dev-version branch is used to build GitHub Pages and deployed to https://texackers.dev/tabularray, but the benchmark outputs were still on gh-pages branch.

https://github.com/TeXackers/tabularray/actions/workflows/pages/pages-build-deployment

#562 proposed consolidating the results from each TeXLive version of the benchmark into a single page/output, but it remains untouched. Deployment is one task, but specifically what purpose it serves for our development on top of our usual duties is another. For example, testing against multiple years of TeXLive versions assumes these comparison are warranted and thus meaningful, but if the benchmark methodology is already under scrutiny, a polished output page will not page this. Consider also the sacle of the project cf. other projects who also utilise the benchmark deployment action - I feel that a simple, multiplicate \the\pdfelapsedtime is at this stage more maintainable than the burden of maintenance while we enter a new era for tabularray.

So the question remains the same: what decision does any of this infrastructure actually inform? Until that's answered, the case for keeping it on main in its current form is weak.

[muzimuzhi]

I didn't intend to support the current benchmark settings, just providing some relevant information.

From my understanding, current benchmark settings could merely show when (at which commit pushed to the set-up branch) a big running time drop/increase happens.