Add plugin-level file exclusion filter?

[zstumgoren]

Should we add support for plugin-level file excludes? These would be passed through using the --exclude flag (#6) for the underlying aws s3 sync command.

Mac OS's tendency to litter directories with the .DS_Store file is the immediate motivation for this, although it could be useful in other scenarios.

[michelleminkoff]

I really like this idea -- the .DS_Store always really annoys me, but it strikes me there might be other large files we don't want to move up and down -- could see large raw video being data at some point and mucking up the upload/download. But the .DS_Store problem in particular is ugh.

[michelleminkoff]

Follow-up use case I just noticed -- excluding the files that start with ~ that Excel generates while files are being used. I notice if I push while any spreadsheets are open (which is common workflow for me), these temp files are uploaded.)

[larryfenn]

In the same vein:

• Emacs autosave: #filename#
• Emacs file lock symlinks: .#filename
• Emacs autobackup: filename~
• Dropbox: .dropbox

[larryfenn]

See https://gitlab.inside.ap.org/newsapps/datakit-ap/issues/5

[zstumgoren]

Based on some off-line discussions, we seem to have agreement on the following implementation:

• Update datakit-data to automatically exclude any patterns specified in a project-level .dkitignore file. This would pass the flags to the underlying aws s3 sync utility's --exclude filter.
• Update our team-specific project templates (e.g. cookiecutter-r-project to include a .dkitignore file

We didn't discuss, but could also provide a flag to data:init that would supply a default .dkitignore with some generally useful patterns (e.g. .DS_Store, vim and emacs files, etc.). This way non-AP users of the software who may not be using cookiecutter and/or datakit-project plugin would have an easy way to bootstrap the file.

/cc @michelleminkoff @larryfenn @tthibo @myersjustinc going to remove Question label on this issue and include as deliverable for version 0.2.0

[myersjustinc]

Does the dkit phrase appear anywhere else in Datakit or its config files? Seems like .datakitignore (or even .datakit-ignore) would be more consistent. Sure, it's a bit of a mouthful, but it's a hidden file—so you'd really only have to deal with it once per project, if even that often. (More likely, you'd only deal with it if you were developing project templates.)

[zstumgoren]

dkit is used in top-level package namespace, but .datakit-ignore works fine.

[zstumgoren]

@myersjustinc Woops. actually forgot that we're not using dkit in package name anymore. either way, let's go with full name in dotfile