Add capability for cli to scan by repo url

[omkar-foss]

The cli could take a repo url as input and scan and return the result. This would be a convenient option to have for folks who'd want to use disclosure for scanning multiple repos.

Example:

disclosure scan --format=text https://github.com/chaoss/disclosure

or simply:

disclosure scan --format=text github.com/chaoss/disclosure

[MoralCode]

i think i like the general vibe of this for improving the user experience. this is something the scorecard tool (for Openssf security scoring) does too

I will add though that this has also caused CollectOSS problems because scorecard is making clones of repos that could easily be VERY big into temporary directory locations, so i think if we do this we should:

1. keep the "use an existing cloned repo" as the default behavior (only try to clone if a url is provided)
2. make sure we have limits set - i.e. do an API call to check the repo clone size or something and fail if its above some threshold with a clear message (unless the user overrides it with a command line flag)

maybe we could also have options for things like:

• the user to specify the clone location (otherwise itll just use a tmp directory)
• the user to specify whether the repo clone dir should be deleted after the run completes

[omkar-foss]

Thanks @MoralCode, we're aligned on these constraints. I'll pick this up after #55.

Based on your comment, I'm thinking of these defaults:

• repo should be cloned into tmpdir, unless user specified another dir
• max repo size to not exceed 1GB, unless user specified another repo size limit
• deep clone the repo, unless user specified another depth or recent n number of commits to scan
• repo should be deleted after the scan is complete, unless user specifies to retain it

[andrew]

https://github.com/git-pkgs/clone might be helpful, I extracted it from a pattern in a couple of my repos

[MoralCode]

deep clone the repo

id say we should deep clone only as far as cloning every commit. so yeah i agree, although this also made me think of submodules. IMO recursing submodules wont help because then we might mix detections from one repo into another, so we should NOT do that (and i dont think we should put effort into supporting that as an option right now, but we shouldnt prevent it from being possible if we later decide its needed)

[omkar-foss]

id say we should deep clone only as far as cloning every commit.

Yes totally agree, so git clone by default deep clones with entire commit history and no submodules, so we could rely on that, and could use git-pkgs clone (@andrew mentioned above) with full=true something like this: clone.Ensure(ctx, clone.Retry{}, url, dst, "main", true);

Also we'd need branch name to default to main unless user specifies a different branch. I guess we could attempt master branch if main fails, and if master too fails we can show an error message to the user.

[MoralCode]

I believe I was working on logic in collectoss to detect the default branch name