-
Notifications
You must be signed in to change notification settings - Fork 165
feat: add datacompy compare CLI with multi-backend support #534
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
fdosani
wants to merge
50
commits into
capitalone:main
from
capitalone-contributions:datacompy-cli-feature
Closed
Changes from 22 commits
Commits
Show all changes
50 commits
Select commit
Hold shift + click to select a range
6db0385
feat: implement CLI entry point and add shared fixtures for testing
fdosani 6221267
feat: Implement CLI for dataset comparison with multiple backends
fdosani 029b617
Add CLI integration tests for Polars, Snowflake, and Spark backends
fdosani 727d4bf
feat: Add CLI usage documentation and update index for new section
fdosani 6532a8c
feat: Add testing and documentation conventions to CLAUDE.md
fdosani e67ce84
feat: Enhance load_snowflake function to support custom CSV delimiter…
fdosani fec9e11
feat: Update Snowflake reference regex to allow valid identifiers and…
fdosani 23c368e
feat: Add validation for csv-delimiter argument and corresponding tests
fdosani eafd5f2
feat: Remove unused Spark session fixture and related imports from CL…
fdosani 895f3fe
feat: Remove unused metavar for input file format in compare subparser
fdosani 2d68fc9
feat: Add tests for absolute and relative tolerance, whitespace, and …
fdosani 7efb5e5
chore: refactor tests, enhance error handling and parameterization
fdosani 5b871b9
feat: Refactor comparison logic and enhance Snowflake file loading
fdosani 1c67fbe
feat: Enhance CLI error handling and add tests for argument validation
fdosani e049345
feat: import classes locally
fdosani 5d50aab
feat: Improve help messages for join column and backend options in CL…
fdosani 7cc5161
feat: Improve backend error handling and enhance main function except…
fdosani 8fc238a
feat: pin action versions
fdosani 11cc307
feat: update action versions in workflow files
fdosani 7d9cbb9
feat: remove datacompy.cli.commands package from setup
fdosani d0cb953
feat: skip snowflake tests if snowflake.snowpark is not installed
fdosani 321084b
feat: implement CLI entry point and add shared fixtures for testing
fdosani 32d466c
feat: Implement CLI for dataset comparison with multiple backends
fdosani c525047
Add CLI integration tests for Polars, Snowflake, and Spark backends
fdosani e01794e
feat: Add CLI usage documentation and update index for new section
fdosani c2d23c0
feat: Add testing and documentation conventions to CLAUDE.md
fdosani 62b44ad
feat: Enhance load_snowflake function to support custom CSV delimiter…
fdosani 4e2e02e
feat: Update Snowflake reference regex to allow valid identifiers and…
fdosani f641dff
feat: Add validation for csv-delimiter argument and corresponding tests
fdosani 788262b
feat: Remove unused Spark session fixture and related imports from CL…
fdosani 6485441
feat: Remove unused metavar for input file format in compare subparser
fdosani 0bd29a9
feat: Add tests for absolute and relative tolerance, whitespace, and …
fdosani a204867
chore: refactor tests, enhance error handling and parameterization
fdosani 89c9c39
feat: Refactor comparison logic and enhance Snowflake file loading
fdosani 8ddf0f1
feat: Enhance CLI error handling and add tests for argument validation
fdosani 3690733
feat: import classes locally
fdosani af93bb1
feat: Improve help messages for join column and backend options in CL…
fdosani f5eb682
feat: Improve backend error handling and enhance main function except…
fdosani 608d5e7
feat: pin action versions
fdosani bf4386f
feat: update action versions in workflow files
fdosani 291a063
feat: remove datacompy.cli.commands package from setup
fdosani da92835
feat: skip snowflake tests if snowflake.snowpark is not installed
fdosani 141b91e
Merge branch 'main' into datacompy-cli-feature
fdosani eb7095e
feat: enhance CLI functionality and error handling, add support for J…
fdosani 5141d46
feat: update Snowflake backend to enforce table reference validation …
fdosani 7f9b798
Merge pull request #6 from cof-contributions/feat-cli
d0d9ad9
feat: enhance CLI functionality and error handling, add support for J…
fdosani 7406ca0
feat: update Snowflake backend to enforce table reference validation …
fdosani 008e1b0
Merge pull request #9 from cof-contributions/sync-cli-feature
OSPO-CapitalOne fedb05c
Merge branch 'main' into datacompy-cli-feature
fdosani File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| # | ||
| # Copyright 2026 Capital One Services, LLC | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| """DataComPy command-line interface. | ||
|
|
||
| Entry point: ``datacompy`` (installed via ``[project.scripts]``) or | ||
| ``python -m datacompy``. | ||
|
|
||
| Examples | ||
| -------- | ||
| Compare two CSV files using the Polars backend (default): | ||
|
|
||
| .. code-block:: bash | ||
|
|
||
| datacompy compare --left a.csv --right b.csv --on id | ||
|
|
||
| Emit a JSON report to stdout: | ||
|
|
||
| .. code-block:: bash | ||
|
|
||
| datacompy compare --left a.csv --right b.csv --on id --json | ||
|
|
||
| Use Pandas and compare on the DataFrame index: | ||
|
|
||
| .. code-block:: bash | ||
|
|
||
| datacompy compare --left a.csv --right b.csv --on-index --backend pandas | ||
| """ | ||
|
|
||
| from datacompy.cli.main import main | ||
| from datacompy.cli.parser import build_parser | ||
|
|
||
| __all__ = ["build_parser", "main"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| # | ||
| # Copyright 2026 Capital One Services, LLC | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| """Allows ``python -m datacompy`` to invoke the CLI.""" | ||
|
|
||
| from datacompy.cli import main | ||
|
|
||
| raise SystemExit(main()) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,204 @@ | ||
| # | ||
| # Copyright 2026 Capital One Services, LLC | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| """Backend factory functions. | ||
|
|
||
| Each ``make_*_compare`` function accepts a typed ``CompareArgs`` and the | ||
| already-loaded DataFrames, then constructs and returns the appropriate | ||
| backend-specific ``Compare`` instance. The factories centralise the | ||
| constructor-signature differences between backends (e.g. Pandas supports | ||
| ``on_index``; Snowflake has no ``cast_column_names_lower``). | ||
| """ | ||
|
|
||
| from dataclasses import dataclass | ||
| from pathlib import Path | ||
| from typing import Any | ||
|
|
||
| from datacompy.cli.loaders import is_snowflake_ref | ||
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class CompareArgs: | ||
| """Typed mirror of the ``compare`` subcommand arguments.""" | ||
|
|
||
| left: str | ||
| right: str | ||
| format: str | None | ||
| on: list[str] | None | ||
| on_index: bool | ||
| backend: str | ||
| abs_tol: float | ||
| rel_tol: float | ||
| ignore_spaces: bool | ||
| ignore_case: bool | ||
| ignore_extra_columns: bool | ||
| ignore_unique_rows: bool | ||
| cast_column_names_lower: bool | ||
| csv_delimiter: str | ||
| df1_name: str | ||
| df2_name: str | ||
| sample_count: int | ||
| column_count: int | ||
| max_unequal_rows: int | None | ||
| json: bool | ||
| quiet: bool | ||
| spark_app_name: str | ||
| snowflake_config: Path | None | ||
|
|
||
|
|
||
| def _default_name(ref: str) -> str: | ||
| """Derive a human-readable dataset label from a file path or Snowflake table ref. | ||
|
|
||
| For file paths ``Path.stem`` is used (``"sales_data.parquet"`` → ``"sales_data"``). | ||
| For Snowflake table refs the table name (last segment) is used so that | ||
| ``"PROD.ANALYTICS.SALES_FACT"`` → ``"SALES_FACT"`` rather than the | ||
| misleading ``"PROD.ANALYTICS"`` that ``Path.stem`` would produce. | ||
| """ | ||
| if is_snowflake_ref(ref): | ||
| return ref.rsplit(".", 1)[-1] | ||
| return Path(ref).stem | ||
|
|
||
|
|
||
| def to_compare_args(ns: Any) -> CompareArgs: | ||
| """Convert an :class:`argparse.Namespace` to a typed :class:`CompareArgs`.""" | ||
| return CompareArgs( | ||
| left=ns.left, | ||
| right=ns.right, | ||
| format=ns.format, | ||
| on=ns.on, | ||
| on_index=ns.on_index, | ||
| backend=ns.backend, | ||
| abs_tol=ns.abs_tol, | ||
| rel_tol=ns.rel_tol, | ||
| ignore_spaces=ns.ignore_spaces, | ||
| ignore_case=ns.ignore_case, | ||
| ignore_extra_columns=ns.ignore_extra_columns, | ||
| ignore_unique_rows=ns.ignore_unique_rows, | ||
| cast_column_names_lower=ns.cast_column_names_lower, | ||
| csv_delimiter=ns.csv_delimiter, | ||
| df1_name=ns.df1_name if ns.df1_name is not None else _default_name(ns.left), | ||
| df2_name=ns.df2_name if ns.df2_name is not None else _default_name(ns.right), | ||
| sample_count=ns.sample_count, | ||
| column_count=ns.column_count, | ||
| max_unequal_rows=ns.max_unequal_rows, | ||
| json=ns.json, | ||
| quiet=ns.quiet, | ||
| spark_app_name=ns.spark_app_name, | ||
| snowflake_config=ns.snowflake_config, | ||
| ) | ||
|
|
||
|
|
||
| def make_pandas_compare(args: CompareArgs, df1: Any, df2: Any) -> Any: | ||
| """Construct a :class:`~datacompy.pandas.PandasCompare`.""" | ||
| from datacompy.pandas import PandasCompare | ||
|
|
||
| if args.on_index: | ||
| return PandasCompare( | ||
| df1, | ||
| df2, | ||
| on_index=True, | ||
| abs_tol=args.abs_tol, | ||
| rel_tol=args.rel_tol, | ||
| df1_name=args.df1_name, | ||
| df2_name=args.df2_name, | ||
| ignore_spaces=args.ignore_spaces, | ||
| ignore_case=args.ignore_case, | ||
| cast_column_names_lower=args.cast_column_names_lower, | ||
| ) | ||
| return PandasCompare( | ||
| df1, | ||
| df2, | ||
| join_columns=args.on, | ||
| abs_tol=args.abs_tol, | ||
| rel_tol=args.rel_tol, | ||
| df1_name=args.df1_name, | ||
| df2_name=args.df2_name, | ||
| ignore_spaces=args.ignore_spaces, | ||
| ignore_case=args.ignore_case, | ||
| cast_column_names_lower=args.cast_column_names_lower, | ||
| ) | ||
|
|
||
|
|
||
| def make_polars_compare(args: CompareArgs, df1: Any, df2: Any) -> Any: | ||
| """Construct a :class:`~datacompy.polars.PolarsCompare`.""" | ||
| from datacompy.polars import PolarsCompare | ||
|
|
||
| return PolarsCompare( | ||
| df1, | ||
| df2, | ||
| join_columns=args.on or [], | ||
| abs_tol=args.abs_tol, | ||
| rel_tol=args.rel_tol, | ||
| df1_name=args.df1_name, | ||
| df2_name=args.df2_name, | ||
| ignore_spaces=args.ignore_spaces, | ||
| ignore_case=args.ignore_case, | ||
| cast_column_names_lower=args.cast_column_names_lower, | ||
| ) | ||
|
|
||
|
|
||
| def make_spark_compare(args: CompareArgs, spark: Any, df1: Any, df2: Any) -> Any: | ||
| """Construct a :class:`~datacompy.spark.SparkSQLCompare`.""" | ||
| try: | ||
| from datacompy.spark import SparkSQLCompare | ||
| except ImportError as exc: | ||
| from datacompy.cli.errors import MissingExtraError | ||
|
|
||
| raise MissingExtraError( | ||
| "Spark backend requires 'datacompy[spark]'. " | ||
| "Install it with: pip install datacompy[spark]" | ||
| ) from exc | ||
|
|
||
| return SparkSQLCompare( | ||
| spark, | ||
| df1, | ||
| df2, | ||
| join_columns=args.on or [], | ||
| abs_tol=args.abs_tol, | ||
| rel_tol=args.rel_tol, | ||
| df1_name=args.df1_name, | ||
| df2_name=args.df2_name, | ||
| ignore_spaces=args.ignore_spaces, | ||
| ignore_case=args.ignore_case, | ||
| cast_column_names_lower=args.cast_column_names_lower, | ||
| ) | ||
|
|
||
|
|
||
| def make_snowflake_compare( | ||
| args: CompareArgs, session: Any, ref1: Any, ref2: Any | ||
| ) -> Any: | ||
| """Construct a :class:`~datacompy.snowflake.SnowflakeCompare`.""" | ||
| try: | ||
| from datacompy.snowflake import SnowflakeCompare | ||
| except ImportError as exc: | ||
| from datacompy.cli.errors import MissingExtraError | ||
|
|
||
| raise MissingExtraError( | ||
| "Snowflake backend requires 'datacompy[snowflake]'. " | ||
| "Install it with: pip install datacompy[snowflake]" | ||
| ) from exc | ||
|
|
||
| return SnowflakeCompare( | ||
| session, | ||
| ref1, | ||
| ref2, | ||
| join_columns=args.on, | ||
| abs_tol=args.abs_tol, | ||
| rel_tol=args.rel_tol, | ||
| df1_name=args.df1_name, | ||
| df2_name=args.df2_name, | ||
| ignore_spaces=args.ignore_spaces, | ||
| ignore_case=args.ignore_case, | ||
| ) | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could probably keep this to a single return call by making on_index false if args.on isn't set.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good call. I'll build the shared
kwargsonce and only add the join key that applies, so there's a single returnPandasCompare(...):