Add support for pass-through of aws flags

[zstumgoren]

The initial implementation of the push and pull commands supports pass-through of boolean commands.

Would be nice to also add support for passing through aws s3 sync command flags that accept values, in particular --include and --exclude.

A convention for passing in sync flags is required because their native format (using leading double dashes such as in --delete) causes errors in argparse at the datakit layer, which interprets such commands as missing arguments for its own parser.

A proposed syntax for pass-through arguments is below.

Implementation

Will likely need a custom argument parser class to properly parse and escape incoming command-line arguments, based on our custom convention (see below).

The argparser for the datakit-data commands (push and pull) will use the argparse.REMAINDER strategy, as below, to gather all pass-through flags:

  class Push(ProjectMixin, CommandHelpers, Command):

      "Push local data to S3"

      def get_parser(self, prog_name):
          parser = super(Push, self).get_parser(prog_name)
          parser.add_argument('args', nargs=argparse.REMAINDER)
          return parser

Custom pass-through args syntax

Below is a first pass at syntax for passing through cli flags for the aws s3 sync command. Basically, it involves:

• dropping the leading -- on boolean arguments
• for args that accept values, using an = sign to separate flag from values (This will require some careful handling in the arg parser because the --grants flag accepts values that include the equals sign
• for args that accept multiple values, use commas to separate the values (--grants is potentially multivalue option)

We'll need to ensure proper escaping/handling of matching patterns for --include and --exclude

# Boolean flags
datakit data:push dryrun # => --dry-run

# Flags with values
datakit data:push include=.* # => --include .*