Skip to content

feat: replace Git LFS pushes with direct Batch API migration - #11

Merged
cvega merged 6 commits into
mainfrom
feature/direct-lfs-migration
Aug 29, 2026
Merged

cvega merged 6 commits into
mainfrom
feature/direct-lfs-migration

Conversation

@cvega

@cvega cvega commented Aug 29, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Replace git lfs push-based synchronization with a direct Git LFS Batch API uploader.

This avoids Git history traversal, uploads only objects missing from the destination, removes credentials from remote URLs, and provides an end-to-end migrate command.

What changed

  • Upload LFS objects directly through the destination Batch API.
  • Stream local objects in bounded batches with configurable concurrency.
  • Add optional hash validation and final remote reconciliation.
  • Add migrate orchestration for export → pull → sync.
  • Keep tokens in scoped authorization headers rather than Git URLs.
  • Add destination-scoped reports, error history, and advisory locking.
  • Centralize terminal, plain-text, quiet, verbose, and JSON output.
  • Add GitHub.com and GHES source/target hostname support. Thanks @janv8000 for the original GHES support work in feat: add custom host support when syncing #10.
  • Add a protected manual live-transfer workflow and deterministic fixtures.
  • Split detailed usage, operations, and development guidance into docs/.
  • Update Go dependencies and pinned workflow actions.

Behavior

sync no longer invokes Git or Git LFS. It scans the local LFS object store, negotiates each OID with the destination, uploads missing objects, and verifies destination state.

It does not create or push commits, branches, or tags. Destination repositories and refs must already exist for normal migrations.

Reruns are idempotent: remotely present objects are negotiated and skipped.

Security

  • Tokens are never embedded in clone, remote, or LFS URLs.
  • Authentication is sent through host-scoped headers.
  • Credentials are not forwarded to external storage hosts.
  • Credential-bearing and non-HTTPS transfer URLs are rejected.
  • Privileged live tests run manually against trusted default-branch code using short-lived GitHub App tokens.

Validation

  • 77 automated tests pass.
  • Overall coverage: 79.2%.
  • go vet, Staticcheck, and duplicate detection pass.
  • Windows, Linux, and macOS builds pass.
  • GitHub Actions workflows pass actionlint.
  • Real transfer validated 3 objects totaling 352 MB:
    • Missing-object negotiation
    • Direct upload
    • Remote reconciliation
    • Exact fresh-fetch OID/size comparison
    • Zero-upload idempotent rerun

The new multi-repository live workflow awaits its first post-merge run.

cvega added 6 commits August 28, 2026 20:30
…ng and state management

- Added `runReporter` to track sync process, including error logging and summary generation.
- Introduced tests for `runReporter` to ensure correct state persistence and error handling.
- Refactored sync logic to utilize `runReporter` for reporting issues and summarizing results.
- Implemented environment variable checks to secure sensitive information during sync.
- Enhanced error handling in sync operations to continue past corrupt objects and log issues.
- Created unit tests for repository syncing, including handling of valid and corrupt LFS objects.
- Updated the go-github dependency from v66 to v90 in api_test.go.
- Introduced a new Summary struct in the worker pool to encapsulate processing results.
- Replaced PrintSummary method with a Summary method in Stats struct to return a structured summary.
- Enhanced the export package to utilize a new output renderer for better logging and status updates.
- Refactored the migrate package to integrate the output renderer for consistent messaging.
- Added JSON output support in the renderer for structured logging.
- Improved error handling and reporting in the pull and sync packages.
- Created a new output renderer to manage console output and error reporting more effectively.
- Added tests for the new output renderer to ensure correct functionality and output formatting.
@cvega cvega self-assigned this Aug 29, 2026
@cvega cvega added bug Something isn't working documentation Improvements or additions to documentation enhancement New feature or request labels Aug 29, 2026
@cvega
cvega merged commit 52fdd88 into main Aug 29, 2026
1 check passed
@cvega
cvega deleted the feature/direct-lfs-migration branch August 29, 2026 06:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working documentation Improvements or additions to documentation enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant