Skip to content
Closed
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
6db0385
feat: implement CLI entry point and add shared fixtures for testing
fdosani Jun 22, 2026
6221267
feat: Implement CLI for dataset comparison with multiple backends
fdosani Jun 23, 2026
029b617
Add CLI integration tests for Polars, Snowflake, and Spark backends
fdosani Jun 23, 2026
727d4bf
feat: Add CLI usage documentation and update index for new section
fdosani Jun 23, 2026
6532a8c
feat: Add testing and documentation conventions to CLAUDE.md
fdosani Jun 23, 2026
e67ce84
feat: Enhance load_snowflake function to support custom CSV delimiter…
fdosani Jun 23, 2026
fec9e11
feat: Update Snowflake reference regex to allow valid identifiers and…
fdosani Jun 23, 2026
23c368e
feat: Add validation for csv-delimiter argument and corresponding tests
fdosani Jun 23, 2026
eafd5f2
feat: Remove unused Spark session fixture and related imports from CL…
fdosani Jun 23, 2026
895f3fe
feat: Remove unused metavar for input file format in compare subparser
fdosani Jun 23, 2026
2d68fc9
feat: Add tests for absolute and relative tolerance, whitespace, and …
fdosani Jun 23, 2026
7efb5e5
chore: refactor tests, enhance error handling and parameterization
fdosani Jun 24, 2026
5b871b9
feat: Refactor comparison logic and enhance Snowflake file loading
fdosani Jun 24, 2026
1c67fbe
feat: Enhance CLI error handling and add tests for argument validation
fdosani Jun 24, 2026
e049345
feat: import classes locally
fdosani Jun 24, 2026
5d50aab
feat: Improve help messages for join column and backend options in CL…
fdosani Jun 24, 2026
7cc5161
feat: Improve backend error handling and enhance main function except…
fdosani Jun 24, 2026
8fc238a
feat: pin action versions
fdosani Jun 25, 2026
11cc307
feat: update action versions in workflow files
fdosani Jun 25, 2026
7d9cbb9
feat: remove datacompy.cli.commands package from setup
fdosani Jun 25, 2026
d0cb953
feat: skip snowflake tests if snowflake.snowpark is not installed
fdosani Jun 25, 2026
321084b
feat: implement CLI entry point and add shared fixtures for testing
fdosani Jun 22, 2026
32d466c
feat: Implement CLI for dataset comparison with multiple backends
fdosani Jun 23, 2026
c525047
Add CLI integration tests for Polars, Snowflake, and Spark backends
fdosani Jun 23, 2026
e01794e
feat: Add CLI usage documentation and update index for new section
fdosani Jun 23, 2026
c2d23c0
feat: Add testing and documentation conventions to CLAUDE.md
fdosani Jun 23, 2026
62b44ad
feat: Enhance load_snowflake function to support custom CSV delimiter…
fdosani Jun 23, 2026
4e2e02e
feat: Update Snowflake reference regex to allow valid identifiers and…
fdosani Jun 23, 2026
f641dff
feat: Add validation for csv-delimiter argument and corresponding tests
fdosani Jun 23, 2026
788262b
feat: Remove unused Spark session fixture and related imports from CL…
fdosani Jun 23, 2026
6485441
feat: Remove unused metavar for input file format in compare subparser
fdosani Jun 23, 2026
0bd29a9
feat: Add tests for absolute and relative tolerance, whitespace, and …
fdosani Jun 23, 2026
a204867
chore: refactor tests, enhance error handling and parameterization
fdosani Jun 24, 2026
89c9c39
feat: Refactor comparison logic and enhance Snowflake file loading
fdosani Jun 24, 2026
8ddf0f1
feat: Enhance CLI error handling and add tests for argument validation
fdosani Jun 24, 2026
3690733
feat: import classes locally
fdosani Jun 24, 2026
af93bb1
feat: Improve help messages for join column and backend options in CL…
fdosani Jun 24, 2026
f5eb682
feat: Improve backend error handling and enhance main function except…
fdosani Jun 24, 2026
608d5e7
feat: pin action versions
fdosani Jun 25, 2026
bf4386f
feat: update action versions in workflow files
fdosani Jun 25, 2026
291a063
feat: remove datacompy.cli.commands package from setup
fdosani Jun 25, 2026
da92835
feat: skip snowflake tests if snowflake.snowpark is not installed
fdosani Jun 25, 2026
141b91e
Merge branch 'main' into datacompy-cli-feature
fdosani Jul 24, 2026
eb7095e
feat: enhance CLI functionality and error handling, add support for J…
fdosani Jul 29, 2026
5141d46
feat: update Snowflake backend to enforce table reference validation …
fdosani Jul 29, 2026
7f9b798
Merge pull request #6 from cof-contributions/feat-cli
Jul 29, 2026
d0d9ad9
feat: enhance CLI functionality and error handling, add support for J…
fdosani Jul 29, 2026
7406ca0
feat: update Snowflake backend to enforce table reference validation …
fdosani Jul 29, 2026
008e1b0
Merge pull request #9 from cof-contributions/sync-cli-feature
OSPO-CapitalOne Jul 29, 2026
fedb05c
Merge branch 'main' into datacompy-cli-feature
fdosani Jul 29, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,16 @@ Tolerances (`abs_tol`, `rel_tol`) can be a single float (applied globally) or a
- **Imports**: Only absolute imports (relative imports banned via ruff TID252)
- **Pre-commit hooks**: ruff (lint + format), trailing whitespace, debug statements, end-of-file fixer, pyproject-fmt

## Testing Conventions

- Write plain pytest functions, not class-based test suites. Use `def test_*()` at module level.
- Do not group tests into `class Test*` unless the upstream codebase already does so in the same file.

## Documentation Conventions

- Do not use em dashes ("--" or "---") in documentation or docstrings; rewrite the sentence instead.
- Do not use emojis in documentation, docstrings, or commit messages.

## Branching

- `develop` is the active development branch for v1
Expand Down
45 changes: 45 additions & 0 deletions datacompy/cli/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
#
# Copyright 2026 Capital One Services, LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""DataComPy command-line interface.

Entry point: ``datacompy`` (installed via ``[project.scripts]``) or
``python -m datacompy``.

Examples
--------
Compare two CSV files using the Polars backend (default):

.. code-block:: bash

datacompy compare --left a.csv --right b.csv --on id

Emit a JSON report to stdout:

.. code-block:: bash

datacompy compare --left a.csv --right b.csv --on id --json

Use Pandas and compare on the DataFrame index:

.. code-block:: bash

datacompy compare --left a.csv --right b.csv --on-index --backend pandas
"""

from datacompy.cli.main import main
from datacompy.cli.parser import build_parser

__all__ = ["build_parser", "main"]
20 changes: 20 additions & 0 deletions datacompy/cli/__main__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
#
# Copyright 2026 Capital One Services, LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""Allows ``python -m datacompy`` to invoke the CLI."""

from datacompy.cli import main

raise SystemExit(main())
204 changes: 204 additions & 0 deletions datacompy/cli/backends.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,204 @@
#
# Copyright 2026 Capital One Services, LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""Backend factory functions.

Each ``make_*_compare`` function accepts a typed ``CompareArgs`` and the
already-loaded DataFrames, then constructs and returns the appropriate
backend-specific ``Compare`` instance. The factories centralise the
constructor-signature differences between backends (e.g. Pandas supports
``on_index``; Snowflake has no ``cast_column_names_lower``).
"""

from dataclasses import dataclass
from pathlib import Path
from typing import Any

from datacompy.cli.loaders import is_snowflake_ref


@dataclass(frozen=True)
class CompareArgs:
"""Typed mirror of the ``compare`` subcommand arguments."""

left: str
right: str
format: str | None
on: list[str] | None
on_index: bool
backend: str
abs_tol: float
rel_tol: float
ignore_spaces: bool
ignore_case: bool
ignore_extra_columns: bool
ignore_unique_rows: bool
cast_column_names_lower: bool
csv_delimiter: str
df1_name: str
df2_name: str
sample_count: int
column_count: int
max_unequal_rows: int | None
json: bool
quiet: bool
spark_app_name: str
snowflake_config: Path | None


def _default_name(ref: str) -> str:
"""Derive a human-readable dataset label from a file path or Snowflake table ref.

For file paths ``Path.stem`` is used (``"sales_data.parquet"`` → ``"sales_data"``).
For Snowflake table refs the table name (last segment) is used so that
``"PROD.ANALYTICS.SALES_FACT"`` → ``"SALES_FACT"`` rather than the
misleading ``"PROD.ANALYTICS"`` that ``Path.stem`` would produce.
"""
if is_snowflake_ref(ref):
return ref.rsplit(".", 1)[-1]
return Path(ref).stem


def to_compare_args(ns: Any) -> CompareArgs:
"""Convert an :class:`argparse.Namespace` to a typed :class:`CompareArgs`."""
return CompareArgs(
left=ns.left,
right=ns.right,
format=ns.format,
on=ns.on,
on_index=ns.on_index,
backend=ns.backend,
abs_tol=ns.abs_tol,
rel_tol=ns.rel_tol,
ignore_spaces=ns.ignore_spaces,
ignore_case=ns.ignore_case,
ignore_extra_columns=ns.ignore_extra_columns,
ignore_unique_rows=ns.ignore_unique_rows,
cast_column_names_lower=ns.cast_column_names_lower,
csv_delimiter=ns.csv_delimiter,
df1_name=ns.df1_name if ns.df1_name is not None else _default_name(ns.left),
df2_name=ns.df2_name if ns.df2_name is not None else _default_name(ns.right),
sample_count=ns.sample_count,
column_count=ns.column_count,
max_unequal_rows=ns.max_unequal_rows,
json=ns.json,
quiet=ns.quiet,
spark_app_name=ns.spark_app_name,
snowflake_config=ns.snowflake_config,
)


def make_pandas_compare(args: CompareArgs, df1: Any, df2: Any) -> Any:
"""Construct a :class:`~datacompy.pandas.PandasCompare`."""
from datacompy.pandas import PandasCompare

if args.on_index:
return PandasCompare(
df1,
df2,
on_index=True,
abs_tol=args.abs_tol,
rel_tol=args.rel_tol,
df1_name=args.df1_name,
df2_name=args.df2_name,
ignore_spaces=args.ignore_spaces,
ignore_case=args.ignore_case,
cast_column_names_lower=args.cast_column_names_lower,
)
return PandasCompare(
df1,
df2,
join_columns=args.on,
abs_tol=args.abs_tol,
rel_tol=args.rel_tol,
df1_name=args.df1_name,
df2_name=args.df2_name,
ignore_spaces=args.ignore_spaces,
ignore_case=args.ignore_case,
cast_column_names_lower=args.cast_column_names_lower,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could probably keep this to a single return call by making on_index false if args.on isn't set.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call. I'll build the shared kwargs once and only add the join key that applies, so there's a single return PandasCompare(...):



def make_polars_compare(args: CompareArgs, df1: Any, df2: Any) -> Any:
"""Construct a :class:`~datacompy.polars.PolarsCompare`."""
from datacompy.polars import PolarsCompare

return PolarsCompare(
df1,
df2,
join_columns=args.on or [],
abs_tol=args.abs_tol,
rel_tol=args.rel_tol,
df1_name=args.df1_name,
df2_name=args.df2_name,
ignore_spaces=args.ignore_spaces,
ignore_case=args.ignore_case,
cast_column_names_lower=args.cast_column_names_lower,
)


def make_spark_compare(args: CompareArgs, spark: Any, df1: Any, df2: Any) -> Any:
"""Construct a :class:`~datacompy.spark.SparkSQLCompare`."""
try:
from datacompy.spark import SparkSQLCompare
except ImportError as exc:
from datacompy.cli.errors import MissingExtraError

raise MissingExtraError(
"Spark backend requires 'datacompy[spark]'. "
"Install it with: pip install datacompy[spark]"
) from exc

return SparkSQLCompare(
spark,
df1,
df2,
join_columns=args.on or [],
abs_tol=args.abs_tol,
rel_tol=args.rel_tol,
df1_name=args.df1_name,
df2_name=args.df2_name,
ignore_spaces=args.ignore_spaces,
ignore_case=args.ignore_case,
cast_column_names_lower=args.cast_column_names_lower,
)


def make_snowflake_compare(
args: CompareArgs, session: Any, ref1: Any, ref2: Any
) -> Any:
"""Construct a :class:`~datacompy.snowflake.SnowflakeCompare`."""
try:
from datacompy.snowflake import SnowflakeCompare
except ImportError as exc:
from datacompy.cli.errors import MissingExtraError

raise MissingExtraError(
"Snowflake backend requires 'datacompy[snowflake]'. "
"Install it with: pip install datacompy[snowflake]"
) from exc

return SnowflakeCompare(
session,
ref1,
ref2,
join_columns=args.on,
abs_tol=args.abs_tol,
rel_tol=args.rel_tol,
df1_name=args.df1_name,
df2_name=args.df2_name,
ignore_spaces=args.ignore_spaces,
ignore_case=args.ignore_case,
)
Loading
Loading