-
Notifications
You must be signed in to change notification settings - Fork 101
This allows the user, during a run, to export it as a compressed arch… #1520
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,8 @@ | ||
| """Module provides generic interface to running Galaxy tools and workflows.""" | ||
|
|
||
| import os | ||
| import requests | ||
| import shutil | ||
| import sys | ||
| import tempfile | ||
| import time | ||
|
|
@@ -42,7 +44,11 @@ | |
| ) | ||
|
|
||
| from planemo.galaxy.api import summarize_history | ||
| from planemo.io import wait_on | ||
| from planemo.io import ( | ||
| error, | ||
| wait_on, | ||
| warn, | ||
| ) | ||
| from planemo.runnable import ( | ||
| ErrorRunResponse, | ||
| get_outputs, | ||
|
|
@@ -236,6 +242,47 @@ def _execute( # noqa C901 | |
| ctx.vlog("collecting outputs from run...") | ||
| run_response.collect_outputs(output_directory) | ||
| ctx.vlog("collecting outputs complete") | ||
|
|
||
| if kwds.get("upload_instance_url") or kwds.get("archive_file"): | ||
| ctx.vlog(f"Preparing galaxy run export, history {history_id}.") | ||
| archive_file = kwds.get("archive_file") | ||
|
|
||
| jeha_id = user_gi.histories.export_history( | ||
| history_id=history_id, | ||
| wait=True, | ||
| maxwait=3600, | ||
| ) | ||
|
|
||
| with tempfile.TemporaryDirectory() as temp_dir: | ||
| archive_file_output = os.path.join(temp_dir, "archive.tar.gz") | ||
|
|
||
| with open(archive_file_output, 'bw') as archive: | ||
| user_gi.histories.download_history( | ||
| history_id=history_id, | ||
| jeha_id=jeha_id, | ||
| outf=archive | ||
| ) | ||
|
|
||
| if kwds.get("arhicve_file"): | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There's a typo here. Can you add tests ?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, that's a type. I have added a test for the download option. I'm considering of writing a test that involves uploading a history to a separate Galaxy instance, and I'd appreciate some guidance on how to approach this. Would it make sense to look at the existing serve tests and try spinning up a Galaxy instance manually? Or would it be better to use the instance that Planemo starts during testing?
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please work with invocations though, the history will not include the invocation, so for the live use-case that's not so interesting since we can't show much there.
Yes, the test should manage the instance(s), probably best done using
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I have looked into using invocations instead, didn't realize that everything wasn't exported. I have no problems exporting the invocation using: 1 - This should maybe be implement in bioblend, similar the download_history function? However, I haven't managed to import the export invocation. Looked at the galaxy client which seem to be using
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Looks like I can import an invocation exported as a RO-Crate, but not when I have exported it as "Compressed File: Export the invocation to a compressed File containing the invocation data in Galaxy native format."
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @Smeds If you'd like to implement invocation export in BioBlend, it should be pretty similar to https://github.com/galaxyproject/bioblend/blob/9bdc2ce57c4443d29699ece7437d2bdf3692d4dc/bioblend/galaxy/invocations/__init__.py#L441 , I think.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Interesting, the tar.gz export/import worked for me. Do you still have the archive ?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could be an issue with test.galaxyproject.org that I was using, seem to work on usegalaxy.org. Is there any preference what kind of export format to use, RO-crate or Compressed Galaxy native format?
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Go with RO-crate imo, it's essentially the native format with a bit of extra metadata |
||
| shutil.copy(archive_file_output, archive_file) | ||
| ctx.vlog(f"Archive {kwds.get('arhicve_file')} created.") | ||
|
|
||
| if kwds.get("upload_instance_url"): | ||
| upload_url = kwds.get("upload_instance_url", None) | ||
| upload_key = kwds.get("upload_api_key", None) | ||
|
|
||
| upload_url = f"{upload_url}/api/histories" | ||
| if upload_key is None: | ||
| warn("No API key provided") | ||
| else: | ||
| upload_url = f"{upload_url}?key={upload_key}" | ||
|
|
||
| response = requests.post(upload_url, files={'archive_file': open(archive_file_output, 'rb')}) | ||
| if response.status_code == 200: | ||
| ctx.vlog(f"Upload run to {kwds.get('upload_instance_url')}") | ||
| else: | ||
| error(f"Failed to upload run to {kwds.get('upload_instance_url')}, status code: {response.status_code}, error {response.text}") | ||
|
|
||
| return run_response | ||
|
|
||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We'll want to export the invocation here.