Add support of QOS in Slurm batch system

[ctriquet-cs]

This PR implement the resolution of the issue #5513.

Changelog Entry

• Added support for separate Slurm QOS configuration for CPU and GPU jobs so workflow nodes can use the appropriate QOS based on their resource requirements (fixes Support CPU and GPU-specific QOS options in Slurm batch system #5513)

Reviewer Checklist

• Make sure it is coming from issues/XXXX-fix-the-thing in the Toil repo, or from an external repo.
  • If it is coming from an external repo, make sure to pull it in for CI with:

    contrib/admin/test-pr otheruser theirbranchname issues/XXXX-fix-the-thing
    

  • If there is no associated issue, create one.
• Read through the code changes. Make sure that it doesn't have:
  • Addition of trailing whitespace.
  • New variable or member names in camelCase that want to be in snake_case.
  • New functions without type hints.
  • New functions or classes without informative docstrings.
  • Changes to semantics not reflected in the relevant docstrings.
  • New or changed command line options for Toil workflows that are not reflected in docs/running/{cliOptions,cwl,wdl}.rst
  • New features without tests.
• Comment on the lines of code where problems exist with a review comment. You can shift-click the line numbers in the diff to select multiple lines.
• Finish the review with an overall description of your opinion.

Merger Checklist

• Make sure the PR passed tests, including the Gitlab tests, for the most recent commit in its branch.
• Make sure the PR has been reviewed. If not, review it. If it has been reviewed and any requested changes seem to have been addressed, proceed.
• Merge with the Github "Squash and merge" feature.
  • If there are multiple authors' commits, add Co-authored-by to give credit to all contributing authors.
• Copy its recommended changelog entry to the Draft Changelog.
• Append the issue number in parentheses to the changelog entry.

[adamnovak]

This looks like it should work.

The documentation for this uses the phrasing of "sending" a job "to" a QOS. But the documentation at https://slurm.schedmd.com/qos.html says that jobs "request" a QOS. So I think we should say that e.g. --slurmQOS is the "QOS to request for Slurm jobs".

Also the option parsing logic ought to be redone as a reusable piece since we're copy-pasting it a fourth time here, but I don't know if you want to take that on.

[adamnovak]

I'd apply my suggestion but I don't have edit permission on the PR source branch.

[ctriquet-cs]

Hi @adamnovak, I had missed that occurrence in the help text. I’ll take your suggestion into account.

I'm also testing an update regarding your remark about the code redundancy. More to come...

[adamnovak]

This looks good to me! Thanks!

[adamnovak]

I've pulled this into the issues/5514-slurm-qos-support branch so tests can run.