Workflows on GitHub Enterprise are no longer processed by actions-runner-controller after upgrade to v0.14.0

[verdel]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.14.0

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. Upgrade actions-runner-controller to version 0.14.0
2. Run a GitHub Workflow specifying the RunnerSet name
3. The listener does not start any runners
4. If the minimum number of runners is greater than 0, existing runners do not process jobs from the queue

Describe the bug

We are using GitHub Enterprise version 3.18.3. After upgrading actions-runner-controller from version 0.13.1 to version 0.14.0, we encountered an issue caused by the transition to the scaleset library.

The issue is described in detail in an issue in the actions/scaleset repository.

After the upgrade, the listener stops creating runners when starting a GitHub Workflow that uses the RunnerSet label. If the minimum number of runners is set to greater than 0, the already started runners do not process GitHub Workflow jobs from the queue. However, if a label is manually assigned to a registered runner in the GitHub Enterprise settings, it starts processing jobs from the queue.

In practice, the transition to scaleset is a breaking change when using actions-runner-controller with GitHub Enterprise. This should have been documented in the changelog, along with the minimum GitHub Enterprise version compatible with actions-runner-controller v0.14.0.

Describe the expected behavior

After upgrading the controller to version 0.14.0, GitHub workflows in GitHub Enterprise are processed and do not remain in the queue indefinitely.

Additional Context

GitHub Enterprise version: 3.18.3
Actions Runner Controller version: 0.14.0

Controller Logs

https://gist.github.com/verdel/213efa232aa8f409efe753c18d4762bd

Runner Pod Logs

https://gist.github.com/verdel/213efa232aa8f409efe753c18d4762bd

[nikola-jokic]

Hey @verdel, The controller log is the same as the listener log, please provide the controller log

[verdel]

@nikola-jokic

The link provides a gist containing two files: listener.log and runner.log.

I conducted additional research and described the situation in detail in an issue in the scaleset repository. The problem is most likely caused by an outdated version of the Actions API used in GHES 3.18 and 3.19. As I understand it, the job acquisition procedure is now supposed to be handled automatically on the GitHub/GHES side, but these GHES versions do not support it.

As a result, in actions-runner-controller version 0.14.0, the ghalistener does not create a runner pod, and if the minimum number of runners is set to more than 0, the idle runners do not pick up workflows from the queue.

If, in this situation, an Acquire Job request is executed via curl, the idle runner will execute the workflow and then terminate, and the listener will start a new runner pod.