[FR] Copying from Fileshare DP to JCDS - download then upload vs mount

[grahampugh]

Hi Harry, for your consideration:

We've been encountering slowness and frequent failures when copying from an FSDP to JCDS.

Direct copying

In our experience, copying from a mounted remote location directly to another remote location via API has always had the potential to be problematic, and the process generally works faster and more reliably if the files are first downloaded to a temporary location before being uploaded to the destination. I wonder if Jamf Sync could do this?

HTTPS vs SMB

For the same scenario, we have found it unnecessary to mount the SMB repo, and instead use HTTPS with Basic Auth to perform the download and upload.

We have created scripts that use the HTTPS URL rather than the SMB URL, and download before uploading which we have started to use internally and found great speed and reliability improvements, but I think these changes would improve the reliability of Jamf Sync, especially in areas of challenging network speed. In this year of The Great Migrate To Cloud, it would also be a timely improvement!

[neilmartin83]

Hey Harry,

Just to follow up on this very constructive work from Graham, as I went down the route of creating a purpose built Go based CLI binary for our specific internal use case. 99% driven by Claude Code. The team has been using it successfully to migrate FSDPs to JCDS on lots of instances over the last few days. It's internal only, just for a few people on our team - no intention of developing it further or releasing it publicly. I actually have it in a local Git repo and wasn't going to commit it to GitHub, as we only need it for this one-off migration piece for around 140 instances.

I wanted to point out the main areas we focused on and learned from during the very brief development cycle of this, that really achieved reliability and performance. I'm not sure how it would translate to Swift (I don't know Swift) or if Sync is already adopting these practices - but since we can articulate exactly what we want and know the underlying API/infrastructure we're dealing with, I wonder if AI might also assist here and if you'd be happy to receive a contribution that is AI assisted? (this comment is not AI generated btw - 100% human Neil!)

Really happy to get involved and help any way we can if any of the below items are of particular interest:

• Concurrency - simultaneous downloads and uploads (but not downloading/uploading the same file simultaneously) - we have 3 download workers and 2 upload workers. User can specify a -sequential flag to disable this if their bandwidth is not sufficient.
• Persistent progress state - we record progress of successful transfers in an instance-specific JSON file. If the tool is restarted after a failure, it will carry on where it left off and not transfer any packages that were successful before
• Resumable HTTP downloads from the source FSDP - as above - if the tool was interrupted/restarted and a download was in progress, it's resumed which is helpful for large packages
• Retry logic - in case of 50x errors or rate limiting etc, we just retry downloads/uploads up to 5 times. I've found we do hit an occasional 50x error when uploading.
• Hashing/verification - we calculate the hashes (MD5, SHA256, SHA3512) of the source file once its downloaded, so we can add them to the metadata before uploading. After upload is complete, we loop over POSTing to /v1/cloud-distribution-point/refresh-inventory and GETting /v1/packages/{id} until hashType = SHA3_512 and hashValue matches our calculated SHA5312 value. It fails if the hashValue is different.
• HTTP optimisations - connection pooling, keep-alive, 1MiB I/O buffers in the client, uploads streaming with a pre-calculated Content-Length header. Adding all this really sped up uploads for at least one colleague - he was getting 10-15Mbyte/sec before, and now close to 100Mbyte/sec, which is at the top end of his gigabit connection.

Thank you!