Skip to content

feat: LHCb modules migration#97

Draft
AcquaDiGiorgio wants to merge 33 commits into
DIRACGrid:mainfrom
AcquaDiGiorgio:issue-87-LHCb-UploadLogFile
Draft

feat: LHCb modules migration#97
AcquaDiGiorgio wants to merge 33 commits into
DIRACGrid:mainfrom
AcquaDiGiorgio:issue-87-LHCb-UploadLogFile

Conversation

@AcquaDiGiorgio

@AcquaDiGiorgio AcquaDiGiorgio commented Feb 4, 2026

Copy link
Copy Markdown
Contributor

See #87

First approach, needs review.

I tried to use the current Mock classes (MockDataManager), but I don't think this is the correct way. However, it should not vary much.

I open it as a draft because I still need to create 2 more tests and change the interactions with DIRAC.

@AcquaDiGiorgio AcquaDiGiorgio self-assigned this Feb 4, 2026
@AcquaDiGiorgio AcquaDiGiorgio linked an issue Feb 4, 2026 that may be closed by this pull request
4 tasks
@AcquaDiGiorgio

Copy link
Copy Markdown
Contributor Author

There is an issue with the FailoverRequest.
The DIRAC's workflow, during the finalize phase calls sendFailoverRequest, which through the ReqClient sends the stored requests at the workflow_commons dictionary to DIRAC.
Currently, in dirac-cwl we don't have persitency between command executions, so the operations created are lost.

@AcquaDiGiorgio AcquaDiGiorgio force-pushed the issue-87-LHCb-UploadLogFile branch from c3f5c70 to 93cb40c Compare February 16, 2026 11:34
@aldbr aldbr linked an issue Apr 15, 2026 that may be closed by this pull request
@AcquaDiGiorgio AcquaDiGiorgio changed the title feat: UploadLogFile command implementation feat: LHCb modules migration Apr 16, 2026
@AcquaDiGiorgio

Copy link
Copy Markdown
Contributor Author

This PR will now tackle the migration of every single module (starting from the ones used in MCSim) by importing LHCb specific functions instead of rewritting them from the ground up

Comment thread src/dirac_cwl/commands/upload_output_data.py Outdated
Comment thread src/dirac_cwl/commands/upload_output_data.py Outdated
Comment thread src/dirac_cwl/commands/report_bookkeeping.py
Comment thread src/dirac_cwl/commands/report_bookkeeping.py
Comment thread src/dirac_cwl/commands/upload_log_file.py Outdated
value = returnValueOrRaise(workflow_commons.file_report.generateForwardDISET())
if not value:
logger.info("On second attempt, files correctly reported to TransformationDB")
elif workflow_commons.step_status == StepStatus.Done:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also based on my previous comment in the WorkflowAccounting: it would be interesting to see whether there is a way to get the status of the cwl execution vs a step execution.

Because if this is possible, then may be there is a way to reproduce exactly what we have in the workflow modules with the conditions like if workflowStatus and jobStatus...

f"Values for StepAccounting are wrong. Here are the given data: {data_dict}"
) from e

workflow_commons.dsc.addRegister(job_step)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure it works?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why wouldn't it?
When the command execution ends, the registers are extracted from the client and stored for later use.

Comment thread src/dirac_cwl/commands/workflow_commons.py Outdated
Failed = "Failed"


class Step(BaseModel):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There might be a way of having one step per CWL workflow step.
Check the workflow examples we have here (this is not MCSim but it's very similar): https://gitlab.cern.ch/lhcb-dpa/analysis-productions/lbapcommon/-/blob/master/tests/example_workflows/complex_workflow_with_filtering/AnaProd_example_workflows_job_tuple%2Csplit%2Cfilter_B0.cwl?ref_type=heads

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I will look into this

Comment thread test/test_commands.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Integrate real LHCb workflow commands (Pre/PostExecution) LHCb Workflow: UploadLogFile command

2 participants