This allows the user, during a run, to export it as a compressed arch…

[Smeds]

…ive and/or upload it to a Galaxy instance.

[mvdbeek]

There's a typo here. Can you add tests ?

[Smeds]

Yes, that's a type.

I have added a test for the download option.

I'm considering of writing a test that involves uploading a history to a separate Galaxy instance, and I'd appreciate some guidance on how to approach this. Would it make sense to look at the existing serve tests and try spinning up a Galaxy instance manually? Or would it be better to use the instance that Planemo starts during testing?

[mvdbeek]

Please work with invocations though, the history will not include the invocation, so for the live use-case that's not so interesting since we can't show much there.

Would it make sense to look at the existing serve tests and try spinning up a Galaxy instance manually? Or would it be better to use the instance that Planemo starts during testing?

Yes, the test should manage the instance(s), probably best done using _launch_thread_and_wait as in the serve tests. I would implement a separate export (where export takes an existing invocation id) and an import command, so then you can export from the first instance to a file and import to a new instance.

[Smeds]

I have looked into using invocations instead, didn't realize that everything wasn't exported. I have no problems exporting the invocation using:

1 - /api/invocations/{invocation_id}/prepare_store_download
2 - /api/short_term_storage/{storage_id}/ready
3 - /api/short_term_storage/{storage_id}

This should maybe be implement in bioblend, similar the download_history function?

However, I haven't managed to import the export invocation. Looked at the galaxy client which seem to be using /api/histories to import a invocation, but haven't been able to get it to work. And also not been able to import it using the web interface even though I export it using the same interface.

[Smeds]

Looks like I can import an invocation exported as a RO-Crate, but not when I have exported it as "Compressed File: Export the invocation to a compressed File containing the invocation data in Galaxy native format."

[nsoranzo]

@Smeds If you'd like to implement invocation export in BioBlend, it should be pretty similar to https://github.com/galaxyproject/bioblend/blob/9bdc2ce57c4443d29699ece7437d2bdf3692d4dc/bioblend/galaxy/invocations/__init__.py#L441 , I think.
Pull requests welcome!

[mvdbeek]

Interesting, the tar.gz export/import worked for me. Do you still have the archive ?

[Smeds]

Could be an issue with test.galaxyproject.org that I was using, seem to work on usegalaxy.org.

Is there any preference what kind of export format to use, RO-crate or Compressed Galaxy native format?

[mvdbeek]

Go with RO-crate imo, it's essentially the native format with a bit of extra metadata

[mvdbeek]

We'll want to export the invocation here.

[Smeds]

@mvdbeek I seem to have problem importing the invocation using the API. I get the following error message, if I look at the galaxy logs:

Traceback (most recent call last):
  File "/galaxy/lib/galaxy/jobs/__init__.py", line 2039, in finish
    task_wrapper = self.tool.exec_after_process(
  File "/galaxy/lib/galaxy/tools/__init__.py", line 3100, in exec_after_process
    JobImportHistoryArchiveWrapper(self.app, job.id).cleanup_after_job()
  File "/galaxy/lib/galaxy/tools/imp_exp/__init__.py", line 81, in cleanup_after_job
    model_store.perform_import(new_history, job=job, new_history=True)
  File "/galaxy/lib/galaxy/model/store/__init__.py", line 417, in perform_import
    self._reassign_hids(object_import_tracker, history)
  File "/galaxy/lib/galaxy/model/store/__init__.py", line 1019, in _reassign_hids
    raise Exception("Optional history is required here.")

https://github.com/galaxyproject/galaxy/blob/dd11f859364b0c3ff3e078c422963c2901bc776c/lib/galaxy/model/store/__init__.py#L1019-L1035

I can import the same file using the Web UI and from the code I guess it is using the current loaded history that can be found in my session, with planemo I don't really have a session. What's the best solution here to handle the problem?

• Setting up a session?
• Modify galaxy to either:
  • forward a history_id using a form variable?
  • create a new history if none is provided?
  • other solution?

[mvdbeek]

The API takes a history_id parameter, see the API docs in https://usegalaxy.org/api/docs#/workflows/create_invocations_from_store_api_invocations_from_store_post and the test method here: https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy_test/base/populators.py#L2050-L2061

[Smeds]

Thanks for pointing me in the right direction! I'm able to import the invocation but the datasets aren't included, all data have this message This dataset has been discarded and its files are not available to Galaxy.. If I import the same file using the UI all files will be included. Is there a setting I can pass along to have the datasets imported?

Import using create_invocations_from_store_api_invocations_from_store_post

Import using Web UI

Payload to the request example:

{
    "history_id": "1e8ab44153008be8",
    "model_store_format": "rocrate.zip",
    "store_content_uri": "base64://XXXXXXX"
}

[mvdbeek]

That's controlled at the export:

[Smeds]

That's controlled at the export:

I had included_files set to True, here the form data I provide to the export function:

{
                "model_store_format": "rocrate.zip",
                "include_files": True,
                "include_deleted": False,
                "include_hidden": False,
}

When I import the same file with the web UI the datasets get imported.

[Smeds]

I believe I've found the part of the Galaxy code that marks the dataset as discarded when data is imported using the /api/invocations/from_store endpoint.

https://github.com/galaxyproject/galaxy/blob/5a1b0ba517d0d9a6cf32c18f81b0e7cd8cf5c58f/lib/galaxy/managers/model_stores.py#L333

[Smeds]

Importing a history using from_store should behave the same way. Is this how we wants it to behave? Import via the Web UI seem to be using `/api/histories' as an endpoint which behaves a bit differently.

[Smeds]

@mvdbeek Do you believe that hat we'll have enough time to add these updates to the API and deploy them before GBCC. For the purposes of GBCC, I could potentially use a custom version of Planemo that relies on the current API. That would allow uploading a small workflow instance.

However, this wouldn't address the import issue, where datasets are being discarded—likely due to discarded_data=ImportDiscardedDataType.FORCE. L333C9-L333C54