diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md deleted file mode 100644 index cddcdb28..00000000 --- a/.github/copilot-instructions.md +++ /dev/null @@ -1,63 +0,0 @@ -# DataComPy AI Assistant Guide - -This guide provides essential context for AI coding agents to be productive in the DataComPy codebase. - -## Architecture Overview - -- **Strategy Pattern**: The core logic uses the Strategy design pattern. The abstract base class `datacompy.base.BaseCompare` defines the interface for all comparison operations. Each backend (Pandas, Spark, Polars, Snowflake) has a concrete implementation (`datacompy.pandas.PandasCompare`, etc.) that inherits from `BaseCompare` and implements its methods. -- **Comparison Reports**: Reports are generated via Jinja2 templates in `datacompy/templates/`, with the main template being `report_template.j2`. All backends use a similar reporting interface. -- **Extensibility**: To add or modify comparison logic for a backend, update the corresponding class. For changes affecting all backends, start with `BaseCompare`. - -## Developer Workflow - -- **Environment Setup**: - ```bash - pip install -e ".[dev]" - pre-commit install - ``` -- **Testing**: - - Run all tests: `pytest` - - Backend-specific tests: see `tests/test_pandas.py`, `tests/test_spark.py`, etc. -- **Linting & Formatting**: - - Lint: `ruff check` - - Format: `ruff format --check` - - Type-check: `mypy .` (strict mode) - - All are enforced via pre-commit hooks. -- **Documentation**: - - Build docs: `make -C docs html` - - Docs source: `docs/source/`, output: `docs/build/html/` - -## Code Conventions - -- **Typing**: All code must be fully type-hinted and pass `mypy --strict`. -- **Docstrings**: Use [NumPy style](https://numpydoc.readthedocs.io/en/latest/format.html). -- **Imports**: Only absolute imports are allowed (see `pyproject.toml`). -- **Templates**: All reporting uses Jinja2 templates in `datacompy/templates/`. -- **Backend-specific logic**: Each backend file (`pandas.py`, `spark.py`, `polars.py`, `snowflake.py`) implements the same interface and reporting pattern. - -## Patterns & Examples - -- **Comparison Usage**: - ```python - from datacompy import PandasCompare - - compare = PandasCompare(df1, df2, join_columns=[...]) - print(compare.report()) - ``` -- **Custom Templates**: - ```python - compare.report(template_path="custom_report.j2") - ``` -- **Tolerance Handling**: Tolerances can be set globally or per-column (see `validate_tolerance_parameter`). -- **Unique/Intersect Rows**: Each backend exposes `df1_unq_rows`, `df2_unq_rows`, and `intersect_rows` for advanced analysis. - -## Integration Points - -- **Dependencies**: Core dependencies are in `pyproject.toml` (Jinja2, pandas, polars, pyspark, snowflake-snowpark-python, etc.). -- **Pre-commit**: Linting, formatting, and type-checking are enforced via pre-commit hooks. -- **Builds**: Use the Makefile for docs (`make sphinx`). -- **CI**: See `.github/workflows/` for test and lint automation. - ---- - -If any section is unclear or missing, please provide feedback to iterate and improve these instructions. diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 493d9b14..eb141e75 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -17,6 +17,6 @@ repos: - id: end-of-file-fixer exclude: ^tests/snapshots/ - repo: https://github.com/tox-dev/pyproject-fmt - rev: "v2.5.0" + rev: "v2.26.0" hooks: - id: pyproject-fmt diff --git a/CLAUDE.md b/CLAUDE.md index 9266d8dd..03b9b12b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,9 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ## Project Overview -DataComPy is a Python library for comparing two DataFrames/tables across multiple backends: Pandas, Polars, Spark, and Snowflake. It originated as a replacement for SAS's `PROC COMPARE`. Currently at v1.0.0 beta (pre-release on the `develop` branch). +DataComPy is a Python library for comparing two DataFrames/tables across multiple backends: Pandas, Polars, Spark, and Snowflake. It originated as a replacement for SAS's `PROC COMPARE`. v1 is GA; the version lives in `datacompy/__version__` and `pyproject.toml` derives the distribution version from it. + +This file is the single AI agent guide for the repository. `.github/copilot-instructions.md` used to duplicate it and was removed; put new guidance here rather than starting a second copy. ## Common Commands @@ -16,13 +18,24 @@ pre-commit install ### Testing ```bash -pytest # all tests -pytest tests/test_pandas.py # single backend -pytest tests/test_pandas.py::TestPandasCompare::test_method # single test -pytest --cov=datacompy --cov-report=term-missing # with coverage +pytest # all tests +pytest tests/test_pandas.py # single backend +pytest tests/test_pandas.py::test_numeric_columns_equal_abs # single test +pytest -k "tolerance and not spark" # by expression +pytest --cov=datacompy --cov-report=term-missing # with coverage +pytest -c pytest-ansi.ini # Spark ANSI mode ``` -Spark tests require Java 17 and `pyspark` installed. Snowflake tests require a live Snowflake session (or `--snowflake-session local` for local testing). +CI runs the suite twice, once with the default `pytest.ini` and once with `-c pytest-ansi.ini`, which only differs by `spark.sql.ansi.enabled`. A change touching Spark casting or null handling needs both. + +**Coverage:** always target the top-level package (`--cov=datacompy`). Passing a dotted submodule such as `--cov=datacompy.cli` triggers a numpy double-import in some environments and fails ~100 otherwise-passing tests with a confusing `_NoValueType` `TypeError`. + +**Spark** needs `pyspark` and **Java 17** (newer JDKs fail with `py4j.protocol` errors). If the JDK came from conda (`conda install openjdk=17`, as `[edgetest.envs.core]` does), it is at `$CONDA_PREFIX/lib/jvm` and `JAVA_HOME` must point there. Activating the env normally sets this; a non-interactive shell will not inherit it: +```bash +export JAVA_HOME=$CONDA_PREFIX/lib/jvm +``` + +**Snowflake** tests need a live session, or `--snowflake-session local` for Snowpark's local testing mode. Local mode is an emulator, not Snowflake: `eqNullSafe` returns `True` for every row and high-precision decimals are truncated on DataFrame creation. Tests that depend on either must request the `requires_live_snowflake_session` fixture (`tests/conftest.py`), which skips them in local mode. ### Linting & Formatting ```bash @@ -33,6 +46,10 @@ ruff format # apply formatting mypy . # type-check (strict mode) ``` +**Use the `ruff` version pinned in `.pre-commit-config.yaml`.** The config uses recent selectors, and an older ruff fails to parse `pyproject.toml` at all rather than degrading gracefully. `pre-commit run --all-files` fetches the right version itself. + +**`mypy .` has a large pre-existing error baseline** (~185, mostly in `snowflake.py`, `polars.py`, and `pandas.py`) and is enforced by neither CI nor pre-commit; CI lint runs only `ruff check` and `ruff format --check`. New code is still expected to be clean, so check that your diff introduces no *new* errors rather than that the run is empty, and do not refactor unrelated modules to chase the baseline. Missing-stub errors for `pyspark` and `snowflake.snowpark` mean those extras are not installed in the current environment, not a code defect. + ### Documentation ```bash make sphinx # build docs (runs in docs/ subdirectory) @@ -44,15 +61,17 @@ make sphinx # build docs (runs in docs/ subdirectory) The core design uses the **Strategy pattern** with two abstraction layers: -1. **`BaseCompare`** (`datacompy/base.py`) — ABC defining the comparison interface. All backends implement: `_compare`, `_dataframe_merge`, `_intersect_compare`, `report`, `matches`, `subset`, `sample_mismatch`, `all_mismatch`, etc. +1. **`BaseCompare`** (`datacompy/base.py`), the ABC defining the comparison interface. All backends implement: `_compare`, `_dataframe_merge`, `_intersect_compare`, `report`, `matches`, `subset`, `sample_mismatch`, `all_mismatch`, etc. -2. **Backend implementations** — Each in its own module: +2. **Backend implementations**, each in its own module: - `datacompy/pandas.py` → `PandasCompare` - `datacompy/polars.py` → `PolarsCompare` - `datacompy/spark.py` → `SparkSQLCompare` - `datacompy/snowflake.py` → `SnowflakeCompare` -Spark and Snowflake are optional imports (try/except in `__init__.py`). +Spark and Snowflake are optional imports (try/except in `__init__.py`), so `datacompy.SparkSQLCompare` simply does not exist when the extra is missing. Because that try/except runs at package import time, importing *any* datacompy submodule pulls in pyspark when it is installed; there is no lazy path around it. + +Beyond the report, each backend exposes `df1_unq_rows`, `df2_unq_rows`, and `intersect_rows` for programmatic analysis, and `build_report_data()` returns a typed `ReportData` (`datacompy/report.py`) with `render()`, `to_html()`, `save()`, and `to_dict()`. Prefer these over parsing the string report. ### Comparator Subpackage @@ -73,6 +92,17 @@ Reports use Jinja2 templates from `datacompy/templates/report_template.j2`. The Tolerances (`abs_tol`, `rel_tol`) can be a single float (applied globally) or a dict mapping column names to per-column values. Validated by `validate_tolerance_parameter()` in `base.py`. +### Command Line Interface + +`datacompy/cli/` implements the `datacompy` console script (entry point `datacompy.cli:main`, also reachable as `python -m datacompy`). + +- `parser.py` holds the `OPTIONS` tuple, the **single source of truth** for the argument surface. Each `Opt` row records the argparse flags *and* the `*Compare` constructor keyword it maps to, so `build_parser()` and `backends.compare_kwargs()` are both generated from it. **Adding a library kwarg to the CLI is one new row.** Do not hand-write it in two places. `tests/cli/test_parser.py` asserts every `Opt.kwarg` against the real constructor signature via `inspect.signature`, so drift fails the build. +- Options are registered with `default=argparse.SUPPRESS` and defaults live on `Opt.default`. That is what makes `Opt.was_given()` meaningful. `validate_arguments()` must run **before** `fill_defaults()`. +- `backends.py` holds the `CLIBackend` ABC plus one implementation per backend, mirroring the `BaseCompare` strategy pattern. A backend owns its session, its loaders, and its constructor call. `pyspark` and `snowflake.snowpark` imports stay inside methods. +- Backend applicability is data (`Opt.backends`), not an `if` chain. An option passed with a backend that does not accept it is rejected generically. +- With `--backend snowflake`, `--left` / `--right` are **always** table references. There is deliberately no file-versus-table heuristic. +- Exit codes are the contract: `0` match, `1` mismatch, `2` error, `130` interrupt. Expected failures raise `CLIError` subclasses; anything else propagates as a traceback. + ## Code Conventions - **Typing**: All code must be fully type-hinted and pass `mypy --strict` @@ -80,8 +110,20 @@ Tolerances (`abs_tol`, `rel_tol`) can be a single float (applied globally) or a - **Imports**: Only absolute imports (relative imports banned via ruff TID252) - **Pre-commit hooks**: ruff (lint + format), trailing whitespace, debug statements, end-of-file fixer, pyproject-fmt +## Testing Conventions + +- Write plain pytest functions, not class-based suites. Use `def test_*()` at module level. +- Do not group tests into `class Test*` unless the file already does so. + +## Documentation Conventions + +- Do not use em dashes in documentation or docstrings; rewrite the sentence instead. +- Do not use emojis in documentation, docstrings, or commit messages. + ## Branching -- `develop` is the active development branch for v1 -- `main` is the release branch -- `support/0.19.x` maintained for v0 users (bug fixes only) +- `main` is the release branch and currently the most advanced one. Recent release commits land here, so branch from `main` unless told otherwise. +- `develop` predates the v1 GA and lags `main`. Do not assume it is the integration branch without checking `git log origin/main origin/develop`. +- `support/0.19.x` is maintained for v0 users (bug fixes only). + +CI (`.github/workflows/test-package.yml`) runs on `develop`, `main`, `release/*`, `release-*`, and `support/*`. diff --git a/README.md b/README.md index 90d89b3e..3b6ba1ed 100644 --- a/README.md +++ b/README.md @@ -55,6 +55,28 @@ pip install datacompy[snowflake] - Snowflake/Snowpark: ([See documentation](https://capitalone.github.io/datacompy/snowflake_usage.html)) +## Command Line Interface + +DataComPy ships a `datacompy` command, so ad hoc checks and CI pipelines do not +need a throw-away script ([see documentation](https://capitalone.github.io/datacompy/cli.html)): + +```bash +# Compare two files and print a report +datacompy compare --left before.csv --right after.csv --on id + +# Fail a build on any difference, with a JSON report saved as an artifact +datacompy compare \ + --left before.parquet --right after.parquet \ + --on account_id,as_of_date \ + --abs-tol balance=0.01 \ + --max-unequal-rows 0 \ + --report-format json --output reports/diff.json --quiet +``` + +It exits `0` when the datasets match, `1` when they differ, and `2` on error. +CSV, Parquet, and JSON inputs are supported on the pandas, polars, and Spark +backends, and Snowflake tables can be compared in place. + ## Programmatic Report Access Every compare object exposes `build_report_data()` which returns a typed diff --git a/datacompy/__main__.py b/datacompy/__main__.py new file mode 100644 index 00000000..369d8b12 --- /dev/null +++ b/datacompy/__main__.py @@ -0,0 +1,21 @@ +# +# Copyright 2026 Capital One Services, LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Allow ``python -m datacompy`` to invoke the CLI.""" + +from datacompy.cli import main + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/datacompy/cli/__init__.py b/datacompy/cli/__init__.py new file mode 100644 index 00000000..37a75437 --- /dev/null +++ b/datacompy/cli/__init__.py @@ -0,0 +1,85 @@ +# +# Copyright 2026 Capital One Services, LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +r"""DataComPy command line interface. + +Invoked as ``datacompy`` once installed, or as ``python -m datacompy``. + +Examples +-------- +Compare two CSV files with the polars backend, which is the default: + +.. code-block:: bash + + datacompy compare --left before.csv --right after.csv --on id + +Emit a machine readable report for a CI pipeline and rely on the exit code: + +.. code-block:: bash + + datacompy compare --left a.parquet --right b.parquet --on id,date \\ + --report-format json --max-unequal-rows 0 + +Write an HTML report to a file: + +.. code-block:: bash + + datacompy compare --left a.csv --right b.csv --on id \\ + --report-format html --output report.html +""" + +import argparse +from collections.abc import Callable, Sequence + +from datacompy.cli.compare import run_compare +from datacompy.cli.errors import CLIError +from datacompy.cli.output import print_error +from datacompy.cli.parser import build_parser + +#: Subcommand name to handler. Adding a command is additive. +COMMANDS: dict[str, Callable[[argparse.Namespace], int]] = {"compare": run_compare} + +__all__ = ["COMMANDS", "build_parser", "main"] + + +def main(argv: Sequence[str] | None = None) -> int: + """Parse *argv*, dispatch the subcommand, and return the exit code. + + Parameters + ---------- + argv : sequence of str, optional + Argument list. When ``None``, argparse reads :data:`sys.argv`. + + Returns + ------- + int + ``0`` on a match, ``1`` on a mismatch, ``2`` on an expected error, and + ``130`` on interrupt. Argparse exits with ``2`` itself on a parse + failure, before this function returns. + """ + parser = build_parser() + args = parser.parse_args(list(argv) if argv is not None else None) + debug = getattr(args, "debug", False) + try: + return COMMANDS[args.command](args) + except CLIError as exc: + if debug: + raise + print_error(str(exc)) + return exc.exit_code + except KeyboardInterrupt: + print_error("interrupted") + return 130 + # Anything else is an unexpected bug and propagates as a traceback. diff --git a/datacompy/cli/__main__.py b/datacompy/cli/__main__.py new file mode 100644 index 00000000..972bd2de --- /dev/null +++ b/datacompy/cli/__main__.py @@ -0,0 +1,21 @@ +# +# Copyright 2026 Capital One Services, LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Allow ``python -m datacompy.cli`` to invoke the CLI.""" + +from datacompy.cli import main + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/datacompy/cli/backends.py b/datacompy/cli/backends.py new file mode 100644 index 00000000..fe61e05d --- /dev/null +++ b/datacompy/cli/backends.py @@ -0,0 +1,467 @@ +# +# Copyright 2026 Capital One Services, LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Backend strategies for the DataComPy CLI. + +Each :class:`CLIBackend` owns the three things that vary between backends: +opening a session (Spark and Snowflake only), turning a ``--left`` or +``--right`` reference into something the comparison accepts, and calling the +right ``*Compare`` constructor. + +The constructor keyword arguments themselves are not written out here. They are +derived from :data:`datacompy.cli.parser.OPTIONS` by :func:`compare_kwargs`, so +the parser and the constructor call site cannot drift apart. + +Imports of ``pyspark`` and ``snowflake.snowpark`` are deferred into methods, +following the same pattern as ``datacompy/__init__.py``. +""" + +import argparse +import importlib +import json +import os +import re +from abc import ABC, abstractmethod +from collections.abc import Callable +from contextlib import ExitStack +from pathlib import Path +from typing import Any, ClassVar + +import pandas as pd +import polars as pl + +from datacompy.base import BaseCompare +from datacompy.cli.errors import BadArgsError, LoadError, MissingExtraError +from datacompy.cli.parser import OPTIONS + +#: File extension to canonical format name. +#: +#: ``.tsv`` is deliberately absent. Mapping it to ``csv`` would pick the right +#: reader but not the right delimiter, which stays comma unless +#: ``--csv-delimiter`` says otherwise, so a ``.tsv`` file would be recognised +#: and then misparsed. Until the delimiter is inferred per file, a tab +#: separated file is read with an explicit ``--input-format csv``. +_EXTENSION_FORMATS = { + ".csv": "csv", + ".parquet": "parquet", + ".pq": "parquet", + ".json": "json", + ".jsonl": "json", + ".ndjson": "json", +} + +_NDJSON_EXTENSIONS = frozenset({".jsonl", ".ndjson"}) + +#: A two or three part dotted Snowflake identifier, e.g. ``DB.SCHEMA.TABLE``. +_TABLE_REF = re.compile(r"^[A-Za-z_$][\w$]*(\.[A-Za-z_$][\w$]*){1,2}$") + + +def infer_format(ref: str, override: str | None) -> str: + """Return the input format for *ref*. + + Parameters + ---------- + ref : str + File path or URI. + override : str, optional + Explicit ``--input-format`` value. Returned as is when provided. + + Returns + ------- + str + One of ``"csv"``, ``"parquet"``, or ``"json"``. + + Raises + ------ + BadArgsError + When the extension is unrecognised and no override was given. + """ + if override is not None: + return override + extension = Path(ref).suffix.lower() + try: + return _EXTENSION_FORMATS[extension] + except KeyError: + raise BadArgsError( + f"cannot infer the format of {ref!r} from its extension " + f"{extension or '(none)'!r}. Pass --input-format csv|parquet|json." + ) from None + + +def _is_ndjson(ref: str) -> bool: + """Return ``True`` when *ref* looks like newline delimited JSON.""" + return Path(ref).suffix.lower() in _NDJSON_EXTENSIONS + + +def compare_kwargs(namespace: argparse.Namespace, backend: str) -> dict[str, Any]: + """Build the ``*Compare`` constructor keyword arguments for *backend*. + + Walks :data:`datacompy.cli.parser.OPTIONS`, keeping the rows that name a + constructor keyword and that apply to this backend. Values that resolve to + ``None`` are omitted so the library default applies. + """ + kwargs: dict[str, Any] = {} + for opt in OPTIONS: + if opt.kwarg is None or backend not in opt.backends: + continue + value = opt.resolved(namespace) + if value is None: + continue + kwargs[opt.kwarg] = value + return kwargs + + +def _missing_extra(backend: str, extra: str | None) -> MissingExtraError: + """Return the error raised when *backend* cannot be imported. + + Kept in one place because the same wording is needed wherever an optional + dependency is imported. *extra* is ``None`` for backends with no optional + dependency, where a failed import is not something ``pip install`` fixes. + """ + if extra is None: + return MissingExtraError(f"the {backend} backend could not be imported.") + return MissingExtraError( + f"the {backend} backend requires 'datacompy[{extra}]'. " + f"Install it with: pip install 'datacompy[{extra}]'" + ) + + +class CLIBackend(ABC): + """Strategy describing how the CLI drives one comparison backend.""" + + #: Value accepted by ``--backend``. + name: ClassVar[str] + #: Module holding the ``*Compare`` class, imported on demand. + module: ClassVar[str] + #: Name of the ``*Compare`` class within :attr:`module`. + class_name: ClassVar[str] + #: Optional dependency extra required by this backend, if any. + extra: ClassVar[str | None] = None + + @property + def compare_cls(self) -> Callable[..., BaseCompare]: + """Import and return the backend's ``*Compare`` class. + + Typed as a callable rather than ``type[BaseCompare]`` because the + constructor signatures differ between backends, which is exactly what + :func:`compare_kwargs` and the drift guard in ``tests/cli`` exist to + reconcile. + + Raises + ------ + MissingExtraError + When the backend needs an optional dependency that is not installed. + """ + try: + module = importlib.import_module(self.module) + except ImportError as exc: + raise _missing_extra(self.name, self.extra) from exc + return getattr(module, self.class_name) # type: ignore[no-any-return] + + def open_session(self, namespace: argparse.Namespace, stack: ExitStack) -> Any: + """Return a session for this backend, or ``None`` when none is needed. + + Implementations register teardown on *stack* so the session is closed on + both the success and the failure path. + """ + return None + + @abstractmethod + def load(self, session: Any, ref: str, namespace: argparse.Namespace) -> Any: + """Turn *ref* into whatever the backend's comparison accepts.""" + + def build( + self, namespace: argparse.Namespace, session: Any, left: Any, right: Any + ) -> BaseCompare: + """Construct the ``*Compare`` instance.""" + return self.compare_cls(left, right, **compare_kwargs(namespace, self.name)) + + +class PandasBackend(CLIBackend): + """Compare two files in memory with pandas.""" + + name = "pandas" + module = "datacompy.pandas" + class_name = "PandasCompare" + + def load( + self, session: Any, ref: str, namespace: argparse.Namespace + ) -> pd.DataFrame: + """Read *ref* into a :class:`pandas.DataFrame`.""" + fmt = infer_format(ref, namespace.input_format) + try: + if fmt == "csv": + return pd.read_csv(ref, sep=namespace.csv_delimiter) + if fmt == "parquet": + return pd.read_parquet(ref) + return pd.read_json(ref, lines=_is_ndjson(ref)) + except FileNotFoundError as exc: + raise LoadError(f"file not found: {ref}") from exc + except Exception as exc: + raise LoadError(f"cannot read {ref}: {exc}") from exc + + +class PolarsBackend(CLIBackend): + """Compare two files in memory with polars.""" + + name = "polars" + module = "datacompy.polars" + class_name = "PolarsCompare" + + def load( + self, session: Any, ref: str, namespace: argparse.Namespace + ) -> pl.DataFrame: + """Read *ref* into a :class:`polars.DataFrame`.""" + fmt = infer_format(ref, namespace.input_format) + try: + if fmt == "csv": + return pl.read_csv(ref, separator=namespace.csv_delimiter) + if fmt == "parquet": + return pl.read_parquet(ref) + if _is_ndjson(ref): + return pl.read_ndjson(ref) + return pl.read_json(ref) + except FileNotFoundError as exc: + raise LoadError(f"file not found: {ref}") from exc + except Exception as exc: + raise LoadError(f"cannot read {ref}: {exc}") from exc + + +class SparkBackend(CLIBackend): + """Compare two files with Spark SQL.""" + + name = "spark" + module = "datacompy.spark" + class_name = "SparkSQLCompare" + extra = "spark" + + def open_session(self, namespace: argparse.Namespace, stack: ExitStack) -> Any: + """Return a :class:`pyspark.sql.SparkSession`, stopping it only if we made it. + + A SparkSession is process wide, so ``getOrCreate`` returns the caller's + existing session when there is one. :func:`datacompy.cli.main` is a + public function that can be called in process from a notebook or an + Airflow task, and stopping a session the CLI did not create would kill + the caller's ``SparkContext`` along with it. Teardown is therefore + registered only when this call is the one that created the session. + + For the same reason ``--spark-app-name`` has no effect when a session + already exists: an application name cannot be changed once the + ``SparkContext`` is running. + + The log level defaults to ``ERROR`` so PySpark's INFO and WARN chatter + stays out of the CLI's own output. Override it with + ``DATACOMPY_SPARK_LOG_LEVEL``. + """ + try: + from pyspark.sql import SparkSession + except ImportError as exc: + raise _missing_extra(self.name, self.extra) from exc + + # Read before getOrCreate, which installs an active session as a side + # effect. This is thread local, so a session created on another thread + # and never activated on this one is not detected. + borrowed = SparkSession.getActiveSession() + spark = SparkSession.builder.appName(namespace.spark_app_name).getOrCreate() + if borrowed is None: + stack.callback(spark.stop) + try: + spark.sparkContext.setLogLevel( + os.environ.get("DATACOMPY_SPARK_LOG_LEVEL", "ERROR") + ) + except Exception: + # Log level is cosmetic and is not available on every deployment, + # so never let it prevent the session from being returned. + pass + return spark + + def load(self, session: Any, ref: str, namespace: argparse.Namespace) -> Any: + """Read *ref* into a PySpark DataFrame.""" + fmt = infer_format(ref, namespace.input_format) + try: + if fmt == "csv": + return session.read.csv( + ref, + header=True, + inferSchema=True, + sep=namespace.csv_delimiter, + ) + if fmt == "parquet": + return session.read.parquet(ref) + return session.read.json(ref, multiLine=not _is_ndjson(ref)) + except Exception as exc: + raise LoadError(f"Spark cannot read {ref}: {exc}") from exc + + def build( + self, namespace: argparse.Namespace, session: Any, left: Any, right: Any + ) -> BaseCompare: + """Construct a :class:`~datacompy.spark.SparkSQLCompare`.""" + return self.compare_cls( + session, left, right, **compare_kwargs(namespace, self.name) + ) + + +class SnowflakeBackend(CLIBackend): + """Compare two Snowflake tables in place. + + ``--left`` and ``--right`` are always table references for this backend, + either ``db.schema.table`` or ``schema.table``. Local files are read with + ``--backend pandas`` or ``--backend polars`` instead. + """ + + name = "snowflake" + module = "datacompy.snowflake" + class_name = "SnowflakeCompare" + extra = "snowflake" + + def open_session(self, namespace: argparse.Namespace, stack: ExitStack) -> Any: + """Return a Snowpark session built from ``--snowflake-config`` or the environment.""" + try: + from snowflake.snowpark.session import Session + except ImportError as exc: + raise _missing_extra(self.name, self.extra) from exc + + params = _snowflake_params(namespace.snowflake_config) + session = Session.builder.configs(params).create() + stack.callback(session.close) + return session + + def load(self, session: Any, ref: str, namespace: argparse.Namespace) -> str: + """Validate *ref* and return it as a fully qualified table name. + + Raises + ------ + BadArgsError + When *ref* is not a two or three part dotted identifier, or when a + two part reference cannot be qualified because the session has no + current database. + """ + if not _TABLE_REF.match(ref): + raise BadArgsError( + f"{ref!r} is not a Snowflake table reference. Expected " + "db.schema.table or schema.table. The snowflake backend " + "compares tables in place; use --backend pandas or " + "--backend polars to compare local files." + ) + if ref.count(".") == 2: + return ref + database = session.get_current_database() + if not database: + raise BadArgsError( + f"cannot qualify {ref!r}: the Snowflake session has no current " + "database. Use the db.schema.table form or set SNOWFLAKE_DATABASE." + ) + return f"{database}.{ref}" + + def build( + self, namespace: argparse.Namespace, session: Any, left: Any, right: Any + ) -> BaseCompare: + """Construct a :class:`~datacompy.snowflake.SnowflakeCompare`.""" + return self.compare_cls( + session, left, right, **compare_kwargs(namespace, self.name) + ) + + +def _snowflake_params(config_path: Path | None) -> dict[str, str]: + """Build Snowpark connection parameters from a JSON file or the environment. + + Raises + ------ + BadArgsError + When the config file is missing, unreadable, or not a JSON object, or + when the environment is missing a required variable. + """ + if config_path is not None: + try: + raw = json.loads(config_path.read_text()) + except FileNotFoundError as exc: + raise BadArgsError( + f"--snowflake-config file not found: {config_path}" + ) from exc + except (OSError, json.JSONDecodeError) as exc: + raise BadArgsError(f"--snowflake-config {config_path}: {exc}") from exc + if not isinstance(raw, dict): + raise BadArgsError( + "--snowflake-config must contain a JSON object of connection " + f"parameters, got {type(raw).__name__}." + ) + return raw + + account = os.environ.get("SNOWFLAKE_ACCOUNT") + user = os.environ.get("SNOWFLAKE_USER") + password = os.environ.get("SNOWFLAKE_PASSWORD") + authenticator = os.environ.get("SNOWFLAKE_AUTHENTICATOR") + token = os.environ.get("SNOWFLAKE_TOKEN") + + # The connector only reads ``token`` as an OAuth access token when the + # authenticator says so, so a bare SNOWFLAKE_TOKEN implies OAuth. + if token and not authenticator: + authenticator = "oauth" + oauth = authenticator is not None and authenticator.lower() == "oauth" + + required = [("SNOWFLAKE_ACCOUNT", account)] + if not oauth: + # Under OAuth the access token carries the identity, so a user name is + # optional. Every other authenticator still needs one. + required.append(("SNOWFLAKE_USER", user)) + missing = [name for name, value in required if not value] + if missing: + raise BadArgsError( + f"missing required environment variable(s): {', '.join(missing)}. " + "Set them or pass --snowflake-config path/to/connection.json." + ) + if oauth and not token: + raise BadArgsError( + "SNOWFLAKE_AUTHENTICATOR=oauth requires an access token in " + "SNOWFLAKE_TOKEN, or pass --snowflake-config for full control over " + "the connection parameters." + ) + if not password and not authenticator: + raise BadArgsError( + "set SNOWFLAKE_PASSWORD, SNOWFLAKE_TOKEN, or SNOWFLAKE_AUTHENTICATOR, " + "or pass --snowflake-config for full control over the connection " + "parameters." + ) + + params: dict[str, str] = {"account": str(account)} + if user: + params["user"] = user + if password: + params["password"] = password + if token: + params["token"] = token + if authenticator: + params["authenticator"] = authenticator + for variable, key in ( + ("SNOWFLAKE_ROLE", "role"), + ("SNOWFLAKE_WAREHOUSE", "warehouse"), + ("SNOWFLAKE_DATABASE", "database"), + ("SNOWFLAKE_SCHEMA", "schema"), + ): + value = os.environ.get(variable) + if value: + params[key] = value + return params + + +BACKENDS: dict[str, CLIBackend] = { + backend.name: backend + for backend in ( + PandasBackend(), + PolarsBackend(), + SparkBackend(), + SnowflakeBackend(), + ) +} diff --git a/datacompy/cli/compare.py b/datacompy/cli/compare.py new file mode 100644 index 00000000..2fd73375 --- /dev/null +++ b/datacompy/cli/compare.py @@ -0,0 +1,157 @@ +# +# Copyright 2026 Capital One Services, LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""The ``datacompy compare`` subcommand.""" + +import argparse +from contextlib import ExitStack + +from datacompy.cli.backends import BACKENDS +from datacompy.cli.errors import BadArgsError +from datacompy.cli.output import emit +from datacompy.cli.parser import OPTIONS, OPTIONS_BY_FLAG, fill_defaults +from datacompy.report import ReportData + + +def run_compare(namespace: argparse.Namespace) -> int: + """Run a comparison and return the process exit code. + + Parameters + ---------- + namespace : argparse.Namespace + Parsed arguments from the ``compare`` subparser. + + Returns + ------- + int + ``0`` when the datasets match or stay within ``--max-unequal-rows``, + and ``1`` when they differ. Problems with the arguments, the input + files, or the backend raise a + :class:`~datacompy.cli.errors.CLIError`, which + :func:`datacompy.cli.main` turns into exit code ``2``. + """ + validate_arguments(namespace) + fill_defaults(namespace) + + backend = BACKENDS[namespace.backend] + with ExitStack() as stack: + # The session is registered on the stack by the backend, so Spark and + # Snowflake shut down on the failure path as well as the success path. + session = backend.open_session(namespace, stack) + left = backend.load(session, namespace.left, namespace) + right = backend.load(session, namespace.right, namespace) + try: + comparison = backend.build(namespace, session, left, right) + except ValueError as exc: + # The comparison classes validate user supplied configuration such + # as join columns and tolerances with a plain ValueError. That is a + # bad argument, not a bug, so report it as one instead of dumping a + # traceback on the user. + raise BadArgsError(str(exc)) from exc + + report_data = comparison.build_report_data( + sample_count=namespace.sample_count, + column_count=namespace.column_count, + ) + emit( + report_data, + namespace.report_format, + namespace.output, + quiet=namespace.quiet, + ) + matched = within_threshold(namespace, report_data) + return 0 if matched else 1 + + +def validate_arguments(namespace: argparse.Namespace) -> None: + """Reject argument combinations that argparse cannot express. + + Per value rules (delimiter length, non negative counts, tolerance syntax) + are enforced by the ``type=`` callables in + :mod:`datacompy.cli.parser`. This function covers the rules that need to + look at more than one argument at a time. + + Must run before :func:`datacompy.cli.parser.fill_defaults`, which is what + makes :meth:`~datacompy.cli.parser.Opt.was_given` distinguish an explicitly + passed flag from an absent one. + + Raises + ------ + BadArgsError + On any invalid combination. + """ + backend = OPTIONS_BY_FLAG["--backend"].value(namespace) + + for opt in OPTIONS: + if opt.was_given(namespace) and backend not in opt.backends: + applies_to = ", ".join(sorted(opt.backends)) + raise BadArgsError( + f"{opt.flags[0]} is not supported with --backend {backend}. " + f"It applies to: {applies_to}." + ) + + join_columns = OPTIONS_BY_FLAG["--on"].value(namespace) + on_index = OPTIONS_BY_FLAG["--on-index"].value(namespace) + if join_columns and on_index: + raise BadArgsError("--on and --on-index are mutually exclusive.") + if not join_columns and not on_index: + raise BadArgsError( + "--on is required. Specify at least one join column with --on COL, " + "or use --on-index with --backend pandas to join on the index." + ) + + ignore_unique_rows = OPTIONS_BY_FLAG["--ignore-unique-rows"].value(namespace) + max_unequal_rows = OPTIONS_BY_FLAG["--max-unequal-rows"].value(namespace) + if ignore_unique_rows and max_unequal_rows is None: + raise BadArgsError( + "--ignore-unique-rows only has an effect together with " + "--max-unequal-rows N." + ) + + +def within_threshold( + namespace: argparse.Namespace, + report_data: ReportData, +) -> bool: + """Return ``True`` when the comparison counts as a pass. + + Without ``--max-unequal-rows`` this reproduces + :meth:`~datacompy.base.BaseCompare.matches`. With it, the datasets pass + while the number of differing rows stays at or below the threshold, and + while the columns line up unless ``--ignore-extra-columns`` was given. + + Both branches read *report_data*, which ``build_report_data`` has already + computed. Calling ``matches()`` here instead would recount straight from the + DataFrames, and on Spark and Snowflake each of those counts is a distributed + action, so the default invocation would scan the data a second time after + the report had already been rendered. + """ + rows = report_data.row_summary + columns_ok = namespace.ignore_extra_columns or ( + report_data.column_summary.df1_unique == 0 + and report_data.column_summary.df2_unique == 0 + ) + + if namespace.max_unequal_rows is None: + rows_overlap = rows.df1_unique == 0 and rows.df2_unique == 0 + # An empty intersection is a non-match for every backend, so the + # common_rows test is what keeps two empty datasets from passing. + intersect_matches = rows.common_rows > 0 and rows.unequal_rows == 0 + return columns_ok and rows_overlap and intersect_matches + + differing_rows = rows.unequal_rows + if not namespace.ignore_unique_rows: + differing_rows += rows.df1_unique + rows.df2_unique + return columns_ok and differing_rows <= namespace.max_unequal_rows diff --git a/datacompy/cli/errors.py b/datacompy/cli/errors.py new file mode 100644 index 00000000..02e40441 --- /dev/null +++ b/datacompy/cli/errors.py @@ -0,0 +1,43 @@ +# +# Copyright 2026 Capital One Services, LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Exception hierarchy for the DataComPy command line interface. + +Every exception defined here carries an ``exit_code`` so that +:func:`datacompy.cli.main` can catch :class:`CLIError` at a single site and +translate it into a friendly message plus the right process exit code. +""" + + +class CLIError(Exception): + """Base class for expected CLI failures. Always maps to exit code 2.""" + + exit_code: int = 2 + + +class BadArgsError(CLIError): + """Raised when arguments are individually valid but invalid in combination.""" + + +class LoadError(CLIError): + """Raised when a dataset cannot be read.""" + + +class MissingExtraError(CLIError): + """Raised when a backend needs an optional dependency that is not installed.""" + + +class OutputError(CLIError): + """Raised when the report cannot be written to the requested destination.""" diff --git a/datacompy/cli/output.py b/datacompy/cli/output.py new file mode 100644 index 00000000..4de904de --- /dev/null +++ b/datacompy/cli/output.py @@ -0,0 +1,105 @@ +# +# Copyright 2026 Capital One Services, LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Report rendering and delivery for the DataComPy CLI. + +Rendering and destination are independent: ``--report-format`` chooses between +text, JSON, and HTML, and ``--output`` chooses between stdout and a file. All +three renderings come from :class:`datacompy.report.ReportData`, so the CLI adds +no templating of its own. +""" + +import json +import sys +from pathlib import Path +from typing import Any + +import numpy as np + +from datacompy.cli.errors import OutputError +from datacompy.report import ReportData + + +def _json_default(obj: Any) -> Any: + """Coerce numpy scalars, which the stdlib JSON encoder cannot serialise.""" + if isinstance(obj, np.integer): + return int(obj) + if isinstance(obj, np.floating): + return float(obj) + if isinstance(obj, np.bool_): + return bool(obj) + return str(obj) + + +def render(report_data: ReportData, report_format: str) -> str: + """Render *report_data* in the requested format. + + Parameters + ---------- + report_data : datacompy.report.ReportData + Structured comparison result from ``compare.build_report_data()``. + report_format : {"text", "json", "html"} + Rendering to produce. + + Returns + ------- + str + The rendered report. + """ + if report_format == "json": + return json.dumps(report_data.to_dict(), indent=2, default=_json_default) + if report_format == "html": + return report_data.to_html() + return report_data.render() + + +def emit( + report_data: ReportData, + report_format: str, + output: Path | None, + *, + quiet: bool, +) -> None: + """Write the report to stdout, to *output*, or to both. + + ``quiet`` suppresses stdout only. A file requested with ``--output`` is + always written, since asking for a file is an explicit request for it. + + Raises + ------ + OutputError + When the destination file cannot be written. + """ + if quiet and output is None: + return + + # Rendered once and reused, so asking for a file and stdout together does + # not template the same ReportData twice. + rendered = render(report_data, report_format) + + if output is not None: + try: + output.parent.mkdir(parents=True, exist_ok=True) + output.write_text(rendered, encoding="utf-8") + except OSError as exc: + raise OutputError(f"cannot write {output}: {exc}") from exc + + if not quiet: + print(rendered) + + +def print_error(message: str) -> None: + """Write *message* to stderr with a ``datacompy:`` prefix.""" + print(f"datacompy: {message}", file=sys.stderr) diff --git a/datacompy/cli/parser.py b/datacompy/cli/parser.py new file mode 100644 index 00000000..bce17b66 --- /dev/null +++ b/datacompy/cli/parser.py @@ -0,0 +1,601 @@ +# +# Copyright 2026 Capital One Services, LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Declarative argument specification for the DataComPy CLI. + +The module holds a single source of truth, :data:`OPTIONS`, describing every +option the ``compare`` subcommand accepts. Each :class:`Opt` records both how +argparse should register the flag and which ``*Compare`` constructor keyword it +maps to, so :func:`build_parser` and +:func:`datacompy.cli.backends.compare_kwargs` are generated from the same data +rather than maintained as two parallel hand-written lists. + +Adding a library keyword argument to the CLI is therefore a single new row, and +``tests/cli/test_parser.py`` checks every :attr:`Opt.kwarg` against the real +constructor signature so the two can never silently drift apart. +""" + +import argparse +from collections.abc import Callable +from dataclasses import dataclass, field +from functools import partial +from pathlib import Path +from typing import Any + +from datacompy.cli.errors import BadArgsError + +ALL_BACKENDS = frozenset({"pandas", "polars", "spark", "snowflake"}) +FILE_BACKENDS = frozenset({"pandas", "polars", "spark"}) +INPUT_FORMATS = ("csv", "parquet", "json") +REPORT_FORMATS = ("text", "json", "html") + +#: Argument group headings, in the order they appear in ``--help``. +GROUP_INPUT = "input" +GROUP_JOIN = "join keys" +GROUP_BACKEND = "backend" +GROUP_COMPARISON = "comparison" +GROUP_NAMING = "naming" +GROUP_REPORT = "report" +GROUP_OUTPUT = "output" +GROUP_BACKEND_SPECIFIC = "backend specific" + + +@dataclass(frozen=True) +class Opt: + """One command line option and its mapping onto a ``*Compare`` keyword. + + Attributes + ---------- + flags : tuple of str + Option strings passed to ``argparse.ArgumentParser.add_argument``. + help : str + Help text shown in ``--help``. + group : str + Argument group heading the option is listed under. + kwarg : str, optional + Name of the ``*Compare`` constructor keyword this option supplies. + ``None`` marks the option as CLI only (for example ``--output``). + backends : frozenset of str + Backends that accept this option. Passing the flag with any other + backend is rejected by + :func:`datacompy.cli.compare.validate_arguments`. + resolve : callable, optional + Post-processing applied to the raw parsed value before it is handed to + the constructor. Receives ``(raw_value, namespace)``. Used where the + parsed shape differs from the library shape, such as flattening + repeated ``--on`` groups into a single list. + default : Any + Value used when the flag is absent. Options are registered with + ``argparse.SUPPRESS`` so that "left at the default" stays + distinguishable from "explicitly passed". + options : dict + Extra keyword arguments forwarded to ``add_argument`` (``action``, + ``type``, ``choices``, ``metavar``, ``required``). + """ + + flags: tuple[str, ...] + help: str + group: str + kwarg: str | None = None + backends: frozenset[str] = ALL_BACKENDS + resolve: Callable[[Any, argparse.Namespace], Any] | None = None + default: Any = None + options: dict[str, Any] = field(default_factory=dict) + + @property + def dest(self) -> str: + """The ``argparse`` destination attribute, derived from the first flag.""" + return self.flags[0].lstrip("-").replace("-", "_") + + def was_given(self, namespace: argparse.Namespace) -> bool: + """Return ``True`` when the user actually passed this flag.""" + return hasattr(namespace, self.dest) + + def value(self, namespace: argparse.Namespace) -> Any: + """Return the parsed value, falling back to :attr:`default`.""" + return getattr(namespace, self.dest, self.default) + + def resolved(self, namespace: argparse.Namespace) -> Any: + """Return the value in the shape the library constructor expects.""" + raw = self.value(namespace) + if self.resolve is None: + return raw + return self.resolve(raw, namespace) + + +# --------------------------------------------------------------------------- +# argparse ``type=`` callables +# --------------------------------------------------------------------------- + + +def join_column_group(value: str) -> list[str]: + """Split a single ``--on`` value on commas. + + Combined with ``action="append"`` this makes ``--on id,date``, + ``--on id --on date`` and any mix of the two equivalent. Column names that + genuinely contain a comma must use the repeated form. + """ + columns = [col.strip() for col in value.split(",") if col.strip()] + if not columns: + raise argparse.ArgumentTypeError("--on requires at least one column name") + return columns + + +def tolerance(value: str) -> float | tuple[str, float]: + """Parse a tolerance as either a bare number or a ``COLUMN=VALUE`` pair. + + A bare number applies to every numeric column. Repeated ``COLUMN=VALUE`` + pairs are collected into the per column dictionary that + :func:`datacompy.base.validate_tolerance_parameter` accepts. + """ + column, sep, raw = value.partition("=") + text = raw if sep else value + try: + number = float(text) + except ValueError as exc: + raise argparse.ArgumentTypeError( + f"expected a number or COLUMN=NUMBER, got {value!r}" + ) from exc + if number < 0: + raise argparse.ArgumentTypeError( + f"tolerance must not be negative, got {number}" + ) + if not sep: + return number + if not column.strip(): + raise argparse.ArgumentTypeError(f"missing column name in {value!r}") + return column.strip(), number + + +def single_char(value: str) -> str: + r"""Accept a one character delimiter, translating a literal ``\t`` to a tab.""" + translated = value.replace("\\t", "\t") + if len(translated) != 1: + raise argparse.ArgumentTypeError( + f"expected a single character, got {value!r}. " + r"Use '\t' (or the shell escape $'\t') for tab separated files." + ) + return translated + + +def non_negative_int(value: str) -> int: + """Accept a non negative integer.""" + try: + number = int(value) + except ValueError as exc: + raise argparse.ArgumentTypeError( + f"expected a non negative integer, got {value!r}" + ) from exc + if number < 0: + raise argparse.ArgumentTypeError( + f"expected a non negative integer, got {number}" + ) + return number + + +# --------------------------------------------------------------------------- +# ``resolve=`` callables +# --------------------------------------------------------------------------- + + +def _flatten_join_columns( + raw: list[list[str]] | None, namespace: argparse.Namespace +) -> list[str] | None: + """Flatten repeated ``--on`` groups into a single ordered column list.""" + if not raw: + return None + return [column for group in raw for column in group] + + +def _combine_tolerances( + raw: list[float | tuple[str, float]] | None, + namespace: argparse.Namespace, + *, + flag: str, +) -> float | dict[str, float] | None: + """Reduce repeated tolerance values to a single float or a per column dict. + + The library accepts ``float | Dict[str, float]`` but not a mixture, so + combining a bare number with ``COLUMN=VALUE`` pairs is rejected here with a + message naming the flag. + """ + if not raw: + return None + pairs = [item for item in raw if isinstance(item, tuple)] + scalars = [item for item in raw if not isinstance(item, tuple)] + if pairs and scalars: + raise BadArgsError( + f"{flag} takes either a single number or one or more COLUMN=VALUE " + "pairs, not both." + ) + if scalars: + if len(scalars) > 1: + raise BadArgsError( + f"{flag} was given a bare number more than once. Use " + f"{flag} COLUMN=VALUE to set per column tolerances." + ) + return scalars[0] + return dict(pairs) + + +def default_dataset_name(ref: str, backend: str) -> str: + """Derive a report label from a file path or Snowflake table reference. + + File paths use the stem, so ``sales_data.parquet`` becomes ``sales_data``. + Snowflake references use the final segment, so ``PROD.ANALYTICS.SALES`` + becomes ``SALES`` rather than the misleading ``PROD.ANALYTICS`` that + ``Path.stem`` would produce. + """ + if backend == "snowflake": + return ref.rsplit(".", 1)[-1] + return Path(ref).stem + + +def _resolve_dataset_name( + raw: str | None, namespace: argparse.Namespace, *, side: str +) -> str: + """Return an explicit dataset label, or one derived from the input reference. + + When both sides derive the same label, which happens when a file is compared + against itself or against a file of the same name in another directory, the + labels are suffixed so the two columns of the report stay distinguishable. + """ + if raw is not None: + return raw + left = default_dataset_name(namespace.left, namespace.backend) + right = default_dataset_name(namespace.right, namespace.backend) + if left != right: + return left if side == "left" else right + return f"{left}_1" if side == "left" else f"{right}_2" + + +# --------------------------------------------------------------------------- +# The specification +# --------------------------------------------------------------------------- + +OPTIONS: tuple[Opt, ...] = ( + Opt( + flags=("--left",), + help="Path, URI, or Snowflake table reference for the left dataset.", + group=GROUP_INPUT, + options={"required": True, "metavar": "REF"}, + ), + Opt( + flags=("--right",), + help="Path, URI, or Snowflake table reference for the right dataset.", + group=GROUP_INPUT, + options={"required": True, "metavar": "REF"}, + ), + Opt( + flags=("--input-format",), + help=( + "Force the input file format for both datasets. Omit to infer it " + "from each file extension, which also handles mixed format inputs." + ), + group=GROUP_INPUT, + backends=FILE_BACKENDS, + options={"choices": list(INPUT_FORMATS)}, + ), + Opt( + flags=("--csv-delimiter",), + help=( + "Field delimiter for CSV input (default: comma). " + r"Use '\t' for tab separated files." + ), + group=GROUP_INPUT, + backends=FILE_BACKENDS, + default=",", + options={"type": single_char, "metavar": "CHAR"}, + ), + Opt( + flags=("--on",), + help=( + "Join column. Accepts a comma separated list (--on id,date) or " + "repeated flags (--on id --on date). Use the repeated form for " + "column names that contain a comma." + ), + group=GROUP_JOIN, + kwarg="join_columns", + resolve=_flatten_join_columns, + options={ + "action": "append", + "type": join_column_group, + "metavar": "COL[,COL...]", + }, + ), + Opt( + flags=("--on-index",), + help="Join on the DataFrame index instead of columns. Pandas backend only.", + group=GROUP_JOIN, + kwarg="on_index", + backends=frozenset({"pandas"}), + default=False, + options={"action": "store_true"}, + ), + Opt( + flags=("--backend",), + help=( + "Comparison backend. Polars is fast and the default. Use pandas for " + "index based joins, spark for distributed data, or snowflake to " + "compare tables in place." + ), + group=GROUP_BACKEND, + default="polars", + options={"choices": sorted(ALL_BACKENDS)}, + ), + Opt( + flags=("--abs-tol",), + help=( + "Absolute tolerance for numeric comparisons (default 0). Accepts a " + "single number applied to every column, or repeated COLUMN=VALUE " + "pairs for per column tolerances." + ), + group=GROUP_COMPARISON, + kwarg="abs_tol", + resolve=partial(_combine_tolerances, flag="--abs-tol"), + options={"action": "append", "type": tolerance, "metavar": "N|COL=N"}, + ), + Opt( + flags=("--rel-tol",), + help=( + "Relative tolerance for numeric comparisons (default 0). Accepts a " + "single number or repeated COLUMN=VALUE pairs." + ), + group=GROUP_COMPARISON, + kwarg="rel_tol", + resolve=partial(_combine_tolerances, flag="--rel-tol"), + options={"action": "append", "type": tolerance, "metavar": "N|COL=N"}, + ), + Opt( + flags=("--ignore-spaces",), + help="Ignore leading and trailing whitespace in string columns.", + group=GROUP_COMPARISON, + kwarg="ignore_spaces", + default=False, + options={"action": "store_true"}, + ), + Opt( + flags=("--ignore-case",), + help="Ignore case in string columns.", + group=GROUP_COMPARISON, + kwarg="ignore_case", + default=False, + options={"action": "store_true"}, + ), + Opt( + flags=("--cast-column-names-lower",), + help=( + "Cast column names to lowercase before comparing (default: enabled). " + "Not applicable to snowflake, which normalises identifiers to uppercase." + ), + group=GROUP_COMPARISON, + kwarg="cast_column_names_lower", + backends=FILE_BACKENDS, + default=True, + options={"action": argparse.BooleanOptionalAction}, + ), + Opt( + flags=("--ignore-extra-columns",), + help=( + "Treat the datasets as matching even when one side has columns the " + "other does not." + ), + group=GROUP_COMPARISON, + default=False, + options={"action": "store_true"}, + ), + Opt( + flags=("--df1-name",), + help="Label for the left dataset in the report (default: derived from --left).", + group=GROUP_NAMING, + kwarg="df1_name", + resolve=partial(_resolve_dataset_name, side="left"), + options={"metavar": "NAME"}, + ), + Opt( + flags=("--df2-name",), + help="Label for the right dataset in the report (default: derived from --right).", + group=GROUP_NAMING, + kwarg="df2_name", + resolve=partial(_resolve_dataset_name, side="right"), + options={"metavar": "NAME"}, + ), + Opt( + flags=("--sample-count",), + help="Maximum number of sample mismatch rows to show per column (default 10).", + group=GROUP_REPORT, + default=10, + options={"type": non_negative_int, "metavar": "N"}, + ), + Opt( + flags=("--column-count",), + help="Maximum number of columns to show in unique row samples (default 10).", + group=GROUP_REPORT, + default=10, + options={"type": non_negative_int, "metavar": "N"}, + ), + Opt( + flags=("--max-unequal-rows",), + help=( + "Exit 0 when the number of differing rows is at most N, and 1 " + "otherwise. Counts value mismatches plus rows present in only one " + "dataset; pass --ignore-unique-rows to count value mismatches only." + ), + group=GROUP_REPORT, + options={"type": non_negative_int, "metavar": "N"}, + ), + Opt( + flags=("--ignore-unique-rows",), + help=( + "With --max-unequal-rows, exclude rows that exist in only one " + "dataset from the difference count." + ), + group=GROUP_REPORT, + default=False, + options={"action": "store_true"}, + ), + Opt( + flags=("--report-format",), + help="Report rendering (default: text).", + group=GROUP_OUTPUT, + default="text", + options={"choices": list(REPORT_FORMATS)}, + ), + Opt( + flags=("--output",), + help=( + "Write the report to this file instead of stdout. Parent " + "directories are created as needed." + ), + group=GROUP_OUTPUT, + options={"type": Path, "metavar": "PATH"}, + ), + Opt( + flags=("--quiet",), + help=( + "Do not print the report to stdout. A file named by --output is " + "still written. The exit code still reflects the result." + ), + group=GROUP_OUTPUT, + default=False, + options={"action": "store_true"}, + ), + Opt( + flags=("--spark-app-name",), + help="Spark application name.", + group=GROUP_BACKEND_SPECIFIC, + backends=frozenset({"spark"}), + default="datacompy-cli", + options={"metavar": "NAME"}, + ), + Opt( + flags=("--cache-intermediates",), + help=( + "Cache intermediate DataFrames (default: enabled). Pass " + "--no-cache-intermediates on Databricks Serverless and other " + "environments that do not support caching." + ), + group=GROUP_BACKEND_SPECIFIC, + kwarg="cache_intermediates", + backends=frozenset({"spark"}), + default=True, + options={"action": argparse.BooleanOptionalAction}, + ), + Opt( + flags=("--snowflake-config",), + help=( + "Path to a JSON file of Snowflake connection parameters. When " + "omitted the session is built from SNOWFLAKE_ACCOUNT plus one of " + "SNOWFLAKE_TOKEN (OAuth), SNOWFLAKE_AUTHENTICATOR, or " + "SNOWFLAKE_PASSWORD. SNOWFLAKE_USER is required for everything " + "except OAuth, and SNOWFLAKE_ROLE, SNOWFLAKE_WAREHOUSE, " + "SNOWFLAKE_DATABASE, and SNOWFLAKE_SCHEMA are optional." + ), + group=GROUP_BACKEND_SPECIFIC, + backends=frozenset({"snowflake"}), + options={"type": Path, "metavar": "PATH"}, + ), +) + + +#: Lookup from primary flag to specification row, for targeted validation. +OPTIONS_BY_FLAG: dict[str, Opt] = {opt.flags[0]: opt for opt in OPTIONS} + + +def fill_defaults(namespace: argparse.Namespace) -> None: + """Populate *namespace* in place with the default for every absent option. + + Options are registered with ``argparse.SUPPRESS`` so that an absent flag + leaves no attribute behind, which is what makes :meth:`Opt.was_given` + meaningful. Call :func:`datacompy.cli.compare.validate_arguments` before + this function, because afterwards every option looks as though it was + explicitly passed. + """ + for opt in OPTIONS: + if not hasattr(namespace, opt.dest): + setattr(namespace, opt.dest, opt.default) + if not hasattr(namespace, "debug"): + namespace.debug = False + + +def package_version() -> str: + """Return the datacompy version. + + Read from the package itself rather than from installed distribution + metadata, because ``pyproject.toml`` derives the distribution version from + ``datacompy.__version__`` and an editable install can carry stale metadata. + """ + from datacompy import __version__ + + return __version__ + + +def _debug_parent() -> argparse.ArgumentParser: + """Return a parent parser supplying ``--debug``. + + Sharing it between the top level parser and the subcommand means ``--debug`` + is accepted on either side of the subcommand name. ``SUPPRESS`` stops the + subparser from overwriting a value already set at the top level. + """ + parent = argparse.ArgumentParser(add_help=False) + parent.add_argument( + "--debug", + action="store_true", + default=argparse.SUPPRESS, + help=( + "Re-raise unexpected exceptions with a full traceback instead of a " + "short message. Useful when filing a bug report." + ), + ) + return parent + + +def build_parser() -> argparse.ArgumentParser: + """Build the top level ``datacompy`` argument parser.""" + debug_parent = _debug_parent() + parser = argparse.ArgumentParser( + prog="datacompy", + description="Compare two datasets across pandas, polars, Spark, or Snowflake.", + parents=[debug_parent], + ) + parser.add_argument( + "--version", + action="version", + version=f"%(prog)s {package_version()}", + ) + + subcommands = parser.add_subparsers(dest="command", required=True) + compare = subcommands.add_parser( + "compare", + help="Compare two datasets and report the differences.", + description=( + "Load --left and --right, compare them with --backend, and exit 0 " + "when they match, 1 when they differ or a threshold is exceeded, " + "and 2 on error." + ), + parents=[debug_parent], + ) + + groups: dict[str, argparse._ArgumentGroup] = {} + for opt in OPTIONS: + if opt.group not in groups: + groups[opt.group] = compare.add_argument_group(opt.group) + groups[opt.group].add_argument( + *opt.flags, + help=opt.help, + default=argparse.SUPPRESS, + **opt.options, + ) + return parser diff --git a/docs/source/cli.rst b/docs/source/cli.rst new file mode 100644 index 00000000..406a7823 --- /dev/null +++ b/docs/source/cli.rst @@ -0,0 +1,271 @@ +Command Line Interface +====================== + +DataComPy ships a ``datacompy`` command so you can compare two datasets without +writing a script. It is aimed at ad hoc checks from a shell and at CI pipelines +that run as shell tasks, such as an Airflow ``BashOperator``, a GitHub Actions +step, or a GitLab CI job. + +Quick start +----------- + +.. code-block:: bash + + datacompy compare --left before.csv --right after.csv --on id + +The command is also available as a module, which is handy when the console +script is not on ``PATH``: + +.. code-block:: bash + + python -m datacompy compare --left before.csv --right after.csv --on id + +Exit codes +---------- + +The exit code is the contract for automation. + +======= ============================================================== +Code Meaning +======= ============================================================== +``0`` The datasets match, or stay within ``--max-unequal-rows`` +``1`` The datasets differ, or the threshold was exceeded +``2`` Bad arguments, unreadable input, or a missing optional backend +``130`` Interrupted +======= ============================================================== + +Anything unexpected propagates as a traceback. Pass ``--debug`` to see the full +traceback for an error that would otherwise be reported as a short message. + +Choosing a backend +------------------ + +``--backend`` selects the comparison engine. Polars is the default. + +============== ================================================================ +Backend Use it for +============== ================================================================ +``polars`` The default. Fast, in memory, no extra install +``pandas`` Index based joins (``--on-index``), or wider ecosystem parity +``spark`` Distributed data. Needs ``datacompy[spark]`` and Java 17 +``snowflake`` Comparing two tables in place. Needs ``datacompy[snowflake]`` +============== ================================================================ + +Inputs +------ + +``--left`` and ``--right`` accept local paths and cloud URIs. CSV, Parquet, and +JSON are supported, including newline delimited JSON via a ``.jsonl`` or +``.ndjson`` extension. + +The format is inferred per file from its extension, so mixed inputs work without +any extra flags: + +.. code-block:: bash + + datacompy compare --left snapshot.csv --right snapshot.parquet --on id + +The extensions recognised are ``.csv``, ``.parquet``, ``.json``, ``.jsonl``, +and ``.ndjson``. Use ``--input-format`` when the extension is missing or +unusual, and ``--csv-delimiter`` for anything other than a comma: + +.. code-block:: bash + + datacompy compare --left extract.dat --right extract2.dat --on id \ + --input-format csv --csv-delimiter '\t' + +.. note:: + + ``--csv-delimiter`` applies to both datasets, and ``.tsv`` is not inferred + as a format for that reason. Read tab separated files with an explicit + ``--input-format csv --csv-delimiter '\t'``, which requires both sides to + use the same delimiter. Comparing a comma separated file against a tab + separated one is not currently supported. + +Cloud URIs such as ``s3://``, ``gs://``, and ``abfs://`` are handed straight to +the underlying reader, so they work once the matching filesystem library +(``s3fs``, ``gcsfs``, ``adlfs``) is installed. + +For ``--backend snowflake``, ``--left`` and ``--right`` are always table +references, either ``db.schema.table`` or ``schema.table``. A two part reference +is qualified with the session's current database. The CLI does not read local +files into Snowflake; use the pandas or polars backend for files, or load the +data into a table first. + +.. code-block:: bash + + datacompy compare --backend snowflake \ + --left PROD.ANALYTICS.SALES \ + --right STAGE.ANALYTICS.SALES \ + --on sale_id + +Join keys +--------- + +``--on`` accepts a comma separated list, a repeated flag, or a mix of the two: + +.. code-block:: bash + + datacompy compare --left a.csv --right b.csv --on id,date + datacompy compare --left a.csv --right b.csv --on id --on date + +Use the repeated form for column names that contain a comma. + +``--on-index`` joins on the DataFrame index instead, and is only available with +``--backend pandas``. + +Tolerances +---------- + +``--abs-tol`` and ``--rel-tol`` take either a single number that applies to every +numeric column, or repeated ``COLUMN=VALUE`` pairs for per column tolerances: + +.. code-block:: bash + + datacompy compare --left a.parquet --right b.parquet --on account_id \ + --abs-tol 0.01 --rel-tol 0.001 + + datacompy compare --left a.parquet --right b.parquet --on account_id \ + --abs-tol price=0.01 --abs-tol quantity=0 + +The two forms cannot be mixed on the same flag, because the library takes either +a single tolerance or a per column mapping. + +Normalisation +------------- + +.. code-block:: bash + + datacompy compare --left a.csv --right b.csv --on id \ + --ignore-spaces --ignore-case + +``--ignore-extra-columns`` treats the datasets as matching even when one side has +columns the other does not. Column names are lowercased before comparison by +default; pass ``--no-cast-column-names-lower`` to compare them as written. That +flag does not apply to Snowflake, which normalises identifiers to uppercase +itself. + +Reports +------- + +Rendering and destination are separate. ``--report-format`` picks between +``text`` (the default), ``json``, and ``html``. ``--output`` writes to a file +instead of, or as well as, stdout. + +.. code-block:: bash + + # Human readable, to the terminal + datacompy compare --left a.csv --right b.csv --on id + + # Machine readable, piped into another tool + datacompy compare --left a.csv --right b.csv --on id \ + --report-format json | jq '.row_summary.unequal_rows' + + # An HTML report saved for a build artifact, nothing on stdout + datacompy compare --left a.csv --right b.csv --on id \ + --report-format html --output reports/diff.html --quiet + +``--quiet`` suppresses stdout only. A file named by ``--output`` is always +written, and parent directories are created as needed. The exit code is +unaffected by either flag. + +``--sample-count`` and ``--column-count`` control how many sample rows and +columns the report shows. + +Failing a build +--------------- + +Without a threshold, any difference exits ``1``. ``--max-unequal-rows`` lets a +known amount of drift pass: + +.. code-block:: bash + + # Fail on any difference at all + datacompy compare --left before.parquet --right after.parquet --on id \ + --max-unequal-rows 0 --quiet + + # Tolerate up to 5 differing rows + datacompy compare --left before.parquet --right after.parquet --on id \ + --max-unequal-rows 5 --quiet + +By default the count includes both value mismatches and rows present in only one +dataset. Add ``--ignore-unique-rows`` to count value mismatches in common rows +only: + +.. code-block:: bash + + datacompy compare --left before.parquet --right after.parquet --on id \ + --max-unequal-rows 0 --ignore-unique-rows --quiet + +A threshold run also fails when one side has extra columns, unless +``--ignore-extra-columns`` is given. + +A GitHub Actions step looks like this: + +.. code-block:: yaml + + - name: Check the nightly load against the previous snapshot + run: | + datacompy compare \ + --left s3://warehouse/snapshots/previous.parquet \ + --right s3://warehouse/snapshots/current.parquet \ + --on account_id,as_of_date \ + --abs-tol balance=0.01 \ + --max-unequal-rows 0 \ + --report-format json \ + --output reports/diff.json + +Backend credentials +------------------- + +Spark +~~~~~ + +The CLI creates its own session and stops it when the command finishes, on both +the success and the failure path. A session that already exists is borrowed and +left running, so calling the CLI from a process that owns a session is safe. +``--spark-app-name`` sets the application name, and has no effect when a session +already exists. PySpark's INFO and WARN logging is suppressed so it does not mix +with the report; set ``DATACOMPY_SPARK_LOG_LEVEL`` to override that. + +Intermediate DataFrames are cached by default. Pass +``--no-cache-intermediates`` on Databricks Serverless and other environments +that do not support caching. + +Snowflake +~~~~~~~~~ + +Connection parameters come either from a JSON file or from the environment. + +.. code-block:: bash + + datacompy compare --backend snowflake \ + --left PROD.ANALYTICS.SALES --right STAGE.ANALYTICS.SALES --on sale_id \ + --snowflake-config ~/.snowflake/connection.json + +The JSON file holds Snowpark connection parameters as top level keys, for +example ``account``, ``user``, ``password``, ``role``, ``warehouse``, +``database``, and ``schema``. + +Without ``--snowflake-config``, the session is built from the environment: + +================================== =============================================== +Variable Notes +================================== =============================================== +``SNOWFLAKE_ACCOUNT`` Required +``SNOWFLAKE_USER`` Required, except under OAuth +``SNOWFLAKE_PASSWORD`` Required unless a token or authenticator is set +``SNOWFLAKE_TOKEN`` OAuth access token; implies OAuth on its own +``SNOWFLAKE_AUTHENTICATOR`` ``oauth``, or SSO such as ``externalbrowser`` +``SNOWFLAKE_ROLE`` Optional +``SNOWFLAKE_WAREHOUSE`` Optional +``SNOWFLAKE_DATABASE`` Optional, qualifies two part references +``SNOWFLAKE_SCHEMA`` Optional +================================== =============================================== + +Full option list +---------------- + +.. code-block:: bash + + datacompy compare --help diff --git a/docs/source/developer_instructions.rst b/docs/source/developer_instructions.rst index 8d722825..34976b3f 100644 --- a/docs/source/developer_instructions.rst +++ b/docs/source/developer_instructions.rst @@ -38,27 +38,46 @@ Just make sure Sphinx 1.3 or above is installed. Run unit tests -------------- -Run ``python -m pytest`` to run all unittests defined in the subfolder -``tests`` with the help of `py.test `_ and -`pytest-runner `_. +Run ``python -m pytest`` to run all tests defined in the ``tests`` subfolder. + +CI runs the suite twice, once with the default ``pytest.ini`` and once with +``pytest-ansi.ini``, which differs only by enabling ``spark.sql.ansi.enabled``. +A change touching Spark casting or null handling should be run both ways:: + + python -m pytest + python -m pytest -c pytest-ansi.ini + +The Spark tests need the ``spark`` extra and Java 17. Newer JDKs fail with +``py4j.protocol`` errors. If the JDK came from conda, ``JAVA_HOME`` has to point +at it, which a non-interactive shell will not inherit:: + + export JAVA_HOME=$CONDA_PREFIX/lib/jvm Snowflake testing ----------------- -Testing the Snowflake compare requires the use of a Snowflake cluster, as Snowflake does not support local running. -This means that Snowflake tests do not get run in CICD, and changes to the Snowflake Compare must be validated by -the process of running these tests locally. -Note that you must have the following environment variables set in order to instantiate a Snowflake Connection (for testing purposes): +The Snowflake tests run either against a live Snowflake session or against +Snowpark's local testing mode:: + + python -m pytest tests/test_snowflake.py + python -m pytest tests/test_snowflake.py --snowflake-session local + +Local testing mode is an emulator rather than Snowflake, and two of its +limitations matter here: ``eqNullSafe`` returns ``True`` for every row, and +high-precision decimals are truncated when a DataFrame is created. Tests that +depend on either request the ``requires_live_snowflake_session`` fixture, which +skips them in local mode. Changes to ``SnowflakeCompare`` still need a live +session to be fully validated, and that validation does not happen in CI. -- "SF_ACCOUNT": with your SF account -- "SF_UID": with your SF username -- "SF_PWD": with your SF password -- "SF_WAREHOUSE": with your desired SF warehouse -- "SF_DATABASE": with a valid database with which you have access -- "SF_SCHEMA": with a valid schema belonging to the provided database +A live session is built from the following environment variables, using +external browser authentication rather than a password: -Once these are set, you are free to run the suite of Snowflake tests. +- ``SF_ACCOUNT``: your Snowflake account +- ``SF_UID``: your Snowflake username +- ``SF_WAREHOUSE``: the warehouse to use +- ``SF_DATABASE``: a database you have access to +- ``SF_SCHEMA``: a schema belonging to that database Management of Requirements @@ -75,8 +94,8 @@ edgetest edgetest is a utility to help keep requirements up to date and ensure a subset of testing requirements still work. More on edgetest `here `_. -The ``pyproject.toml`` has configuration details on how to run edgetest. This process can be automated via GitHub Actions. -(A future addition, which will come soon). +The ``pyproject.toml`` has configuration details on how to run edgetest. The process is automated by the +``edgetest`` GitHub Actions workflow, which opens a pull request with any dependency bumps it finds. In order to execute edgetest locally you can run the following after install ``edgetest``: diff --git a/docs/source/index.rst b/docs/source/index.rst index 46ee380a..e1eaec13 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -9,6 +9,7 @@ Contents :maxdepth: 2 Installation + Command Line Interface Comparator Framework Usage Pandas Usage Spark Usage diff --git a/docs/source/install.rst b/docs/source/install.rst index 9176c324..73f3c251 100644 --- a/docs/source/install.rst +++ b/docs/source/install.rst @@ -2,11 +2,8 @@ Installation ============ -.. important:: - - If you are using Python 3.12 and above, please note that not all functioanlity will be supported. - Pandas and Polars support should work fine and are tested. - +DataComPy requires Python 3.10 or later. Every backend is tested against each +supported version. PyPI (basic) ------------ @@ -15,6 +12,22 @@ PyPI (basic) pip install datacompy +Installing extras +----------------- + +Pandas and Polars work out of the box. Spark and Snowflake are optional:: + + pip install datacompy[spark] + pip install datacompy[snowflake] + +.. note:: + + On Python 3.12 and above the ``spark`` extra resolves to PySpark 4. The + dependency markers pick the right version, so nothing extra is needed. + +Installing the package also provides the ``datacompy`` command line tool. See +:doc:`cli`. + A Conda environment or virtual environment is highly recommended: diff --git a/docs/source/polars_usage.rst b/docs/source/polars_usage.rst index ae4a3f98..d22bd7c3 100644 --- a/docs/source/polars_usage.rst +++ b/docs/source/polars_usage.rst @@ -1,11 +1,6 @@ Polars Usage ============ -.. important:: - - Please note that Polars support is experimental and new in ``datacompy`` - as of v0.11.0 - Overview -------- diff --git a/docs/source/spark_usage.rst b/docs/source/spark_usage.rst index 84da8baa..b67e6779 100644 --- a/docs/source/spark_usage.rst +++ b/docs/source/spark_usage.rst @@ -58,6 +58,27 @@ join column(s). print(compare.report()) +Caching +------- + +``SparkSQLCompare`` caches intermediate DataFrames by default, which avoids +recomputing the joined data for every part of the report. Some environments, +Databricks Serverless among them, do not support caching. Pass +``cache_intermediates=False`` there: + +.. code-block:: python + + compare = SparkSQLCompare( + spark, + df1, + df2, + join_columns='acct_id', + cache_intermediates=False, + ) + +The command line equivalent is ``--no-cache-intermediates``. See :doc:`cli`. + + Reports ------- diff --git a/docs/source/template_guide.rst b/docs/source/template_guide.rst index 44388685..bf09c255 100644 --- a/docs/source/template_guide.rst +++ b/docs/source/template_guide.rst @@ -10,7 +10,7 @@ Template Basics Custom templates are Jinja2 templates that receive comparison data and format it according to your needs. The template context is produced by calling ``dataclasses.asdict()`` on the :class:`~datacompy.report.ReportData` instance, so every field is passed in -with its **typed value** — no pre-formatting is applied. All formatting +with its **typed value**, and no pre-formatting is applied. All formatting decisions belong in the template. Available Template Variables diff --git a/pyproject.toml b/pyproject.toml index fbae1e2c..095f3319 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -33,10 +33,10 @@ classifiers = [ dynamic = [ "version" ] dependencies = [ "jinja2>=3", - "numpy<2.6,>=1.26.4", - "ordered-set<=4.1,>=4.0.2", - "pandas<3.1,>=2.2", - "polars[pandas]<1.44,>=0.20.4", + "numpy>=1.26.4,<2.6", + "ordered-set>=4.0.2,<=4.1", + "pandas>=2.2,<3.1", + "polars[pandas]>=0.20.4,<1.44", ] optional-dependencies.build = [ "build", "twine", "wheel" ] optional-dependencies.dev = [ @@ -51,9 +51,9 @@ optional-dependencies.dev = [ optional-dependencies.docs = [ "furo", "myst-parser", "sphinx" ] optional-dependencies.edgetest = [ "edgetest", "edgetest-conda" ] optional-dependencies.qa = [ "mypy", "pandas-stubs", "pre-commit", "ruff" ] -optional-dependencies.snowflake = [ "snowflake-snowpark-python<1.55,>=1.37" ] +optional-dependencies.snowflake = [ "snowflake-snowpark-python>=1.37,<1.55" ] optional-dependencies.spark = [ - "pyspark[connect]!=4,<=4.2,>=3.5; python_version<='3.11'", + "pyspark[connect]>=3.5,!=4,<=4.2; python_version<='3.11'", "pyspark[connect]>=4; python_version>='3.12'", ] optional-dependencies.tests = [ @@ -64,18 +64,23 @@ optional-dependencies.tests = [ ] optional-dependencies.tests-spark = [ "pytest-spark" ] urls."Bug Tracker" = "https://github.com/capitalone/datacompy/issues" +urls."Source Code" = "https://github.com/capitalone/datacompy" urls.Documentation = "https://capitalone.github.io/datacompy/" urls.Homepage = "https://github.com/capitalone/datacompy" urls.Repository = "https://github.com/capitalone/datacompy.git" - -urls."Source Code" = "https://github.com/capitalone/datacompy" +scripts.datacompy = "datacompy.cli:main" [tool.setuptools] -packages = [ "datacompy", "datacompy.comparator", "datacompy.templates" ] -zip-safe = false +packages = [ + "datacompy", + "datacompy.cli", + "datacompy.comparator", + "datacompy.templates", +] include-package-data = true -dynamic.version = { attr = "datacompy.__version__" } package-data."*" = [ "templates/*.j2", "templates/*.txt" ] +dynamic.version = { attr = "datacompy.__version__" } +zip-safe = false [tool.distutils] bdist_wheel.python-tag = "py3" @@ -85,18 +90,18 @@ target-version = "py312" src = [ "src" ] extend-include = [ "*.ipynb" ] lint.select = [ - "B", # flake8-bugbear + "B", # flake8-bugbear # "A", # flake8-builtins - "C4", # flake8-comprehensions - "D", # pydocstyle - "E", # pycodestyle errors - "F", # pyflakes - "I", # isort - "LOG", # flake8-logging - "NPY", # numpy rules - "RUF", # Ruff errors + "C4", # flake8-comprehensions + "D", # pydocstyle + "E", # pycodestyle errors + "F", # pyflakes + "I", # isort + "LOG", # flake8-logging + "NPY", # numpy rules + "RUF", # Ruff errors # "ARG", # flake8-unused-arguments - "SIM", # flake8-simplify + "SIM", # flake8-simplify # "C901", # mccabe complexity # "G", # flake8-logging-format "T20", # flake8-print @@ -122,6 +127,8 @@ lint.ignore = [ ] lint.per-file-ignores."**/{tests,docs}/*" = [ "ARG", "D", "E402", "F841" ] lint.per-file-ignores."__init__.py" = [ "E402" ] +# The CLI writes its report to stdout; `print` is the interface, not a debug statement. +lint.per-file-ignores."datacompy/cli/output.py" = [ "T20" ] lint.flake8-tidy-imports.ban-relative-imports = "all" lint.pydocstyle.convention = "numpy" lint.preview = true diff --git a/tests/cli/__init__.py b/tests/cli/__init__.py new file mode 100644 index 00000000..f126d79e --- /dev/null +++ b/tests/cli/__init__.py @@ -0,0 +1,16 @@ +# +# Copyright 2026 Capital One Services, LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for the DataComPy command line interface.""" diff --git a/tests/cli/conftest.py b/tests/cli/conftest.py new file mode 100644 index 00000000..a7d09d6b --- /dev/null +++ b/tests/cli/conftest.py @@ -0,0 +1,97 @@ +# +# Copyright 2026 Capital One Services, LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Shared fixtures for the CLI tests. + +The datasets are deliberately tiny and their differences are known exactly, so +the threshold tests can assert on specific counts: + +- ``id`` 1 matches on every column +- ``id`` 2 differs in ``amount`` by 0.005 +- ``id`` 3 exists only on the left +- ``id`` 4 exists only on the right + +That is 1 unequal row plus 2 unique rows, so 3 differing rows in total. +""" + +import pandas as pd +import pytest + +LEFT_ROWS = [ + {"id": 1, "name": "alice", "amount": 10.0}, + {"id": 2, "name": "bob", "amount": 20.0}, + {"id": 3, "name": "carol", "amount": 30.0}, +] + +RIGHT_ROWS = [ + {"id": 1, "name": "alice", "amount": 10.0}, + {"id": 2, "name": "bob", "amount": 20.005}, + {"id": 4, "name": "dave", "amount": 40.0}, +] + +#: Number of rows that differ between LEFT_ROWS and RIGHT_ROWS. +UNEQUAL_ROWS = 1 +UNIQUE_ROWS = 2 +TOTAL_DIFFERING_ROWS = UNEQUAL_ROWS + UNIQUE_ROWS + + +@pytest.fixture +def left_frame() -> pd.DataFrame: + """Return the left hand dataset.""" + return pd.DataFrame(LEFT_ROWS) + + +@pytest.fixture +def right_frame() -> pd.DataFrame: + """Return the right hand dataset, which differs from the left.""" + return pd.DataFrame(RIGHT_ROWS) + + +@pytest.fixture +def left_csv(tmp_path, left_frame): + """Write the left dataset to a CSV file and return its path.""" + path = tmp_path / "left.csv" + left_frame.to_csv(path, index=False) + return path + + +@pytest.fixture +def right_csv(tmp_path, right_frame): + """Write the right dataset to a CSV file and return its path.""" + path = tmp_path / "right.csv" + right_frame.to_csv(path, index=False) + return path + + +@pytest.fixture +def cli(left_csv, right_csv): + """Return a callable that runs ``datacompy compare`` on the CSV fixtures. + + Extra arguments are appended, and ``--left`` / ``--right`` / ``--on`` can be + overridden by passing them explicitly. + """ + from datacompy.cli import main + + def _run(*extra: str) -> int: + argv = ["compare"] + if "--left" not in extra: + argv += ["--left", str(left_csv)] + if "--right" not in extra: + argv += ["--right", str(right_csv)] + if "--on" not in extra and "--on-index" not in extra: + argv += ["--on", "id"] + return main([*argv, *extra]) + + return _run diff --git a/tests/cli/test_compare.py b/tests/cli/test_compare.py new file mode 100644 index 00000000..0719ea23 --- /dev/null +++ b/tests/cli/test_compare.py @@ -0,0 +1,666 @@ +# +# Copyright 2026 Capital One Services, LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""End to end tests for ``datacompy compare`` on the in memory backends. + +Every test drives :func:`datacompy.cli.main`, which returns the process exit +code, so the assertions are on the contract a CI pipeline actually sees. +""" + +import sys + +import pytest +from datacompy.cli import main + +from tests.cli.conftest import TOTAL_DIFFERING_ROWS, UNEQUAL_ROWS + +MATCH = 0 +MISMATCH = 1 +ERROR = 2 + +IN_MEMORY_BACKENDS = ["pandas", "polars"] + + +@pytest.fixture(params=IN_MEMORY_BACKENDS) +def backend(request): + """Run the test once per in memory backend.""" + return request.param + + +# --------------------------------------------------------------------------- +# Exit codes +# --------------------------------------------------------------------------- + + +def test_identical_datasets_exit_zero(cli, left_csv, backend, capsys): + assert cli("--right", str(left_csv), "--backend", backend) == MATCH + + +def test_differing_datasets_exit_one(cli, backend, capsys): + assert cli("--backend", backend) == MISMATCH + + +def test_missing_input_file_exits_two(cli, tmp_path, backend, capsys): + assert cli("--left", str(tmp_path / "nope.csv"), "--backend", backend) == ERROR + assert "file not found" in capsys.readouterr().err + + +def test_unreadable_input_exits_two(cli, tmp_path, backend, capsys): + corrupt = tmp_path / "corrupt.parquet" + corrupt.write_text("this is not parquet") + assert cli("--left", str(corrupt), "--backend", backend) == ERROR + assert "cannot read" in capsys.readouterr().err + + +def test_keyboard_interrupt_exits_130(cli, monkeypatch, capsys): + def boom(*args, **kwargs): + raise KeyboardInterrupt + + monkeypatch.setattr("datacompy.cli.COMMANDS", {"compare": boom}) + assert cli() == 130 + assert "interrupted" in capsys.readouterr().err + + +def test_debug_reraises_instead_of_printing(cli, tmp_path): + from datacompy.cli.errors import LoadError + + with pytest.raises(LoadError): + cli("--left", str(tmp_path / "nope.csv"), "--debug") + + +# --------------------------------------------------------------------------- +# Argument validation +# --------------------------------------------------------------------------- + + +def test_join_columns_are_required(left_csv, right_csv, capsys): + exit_code = main(["compare", "--left", str(left_csv), "--right", str(right_csv)]) + assert exit_code == ERROR + assert "--on is required" in capsys.readouterr().err + + +def test_on_and_on_index_are_mutually_exclusive(cli, capsys): + assert cli("--on", "id", "--on-index", "--backend", "pandas") == ERROR + assert "mutually exclusive" in capsys.readouterr().err + + +def test_on_index_is_rejected_for_non_pandas_backends(cli, capsys): + assert cli("--on-index", "--backend", "polars") == ERROR + message = capsys.readouterr().err + assert "--on-index is not supported with --backend polars" in message + assert "pandas" in message + + +def test_backend_specific_flags_are_rejected_elsewhere(cli, capsys): + assert cli("--spark-app-name", "x", "--backend", "polars") == ERROR + assert "--spark-app-name is not supported" in capsys.readouterr().err + + +def test_cache_intermediates_is_rejected_for_non_spark_backends(cli, capsys): + assert cli("--no-cache-intermediates", "--backend", "polars") == ERROR + assert "--cache-intermediates is not supported" in capsys.readouterr().err + + +def test_cast_column_names_lower_is_rejected_for_snowflake(cli, capsys): + assert cli("--no-cast-column-names-lower", "--backend", "snowflake") == ERROR + assert "--cast-column-names-lower is not supported" in capsys.readouterr().err + + +def test_ignore_unique_rows_needs_a_threshold(cli, backend, capsys): + assert cli("--ignore-unique-rows", "--backend", backend) == ERROR + assert "--max-unequal-rows" in capsys.readouterr().err + + +def test_unknown_extension_needs_an_explicit_format( + cli, tmp_path, left_csv, backend, capsys +): + renamed = tmp_path / "left.data" + renamed.write_text(left_csv.read_text()) + + assert cli("--left", str(renamed), "--backend", backend) == ERROR + assert "--input-format" in capsys.readouterr().err + + assert ( + cli("--left", str(renamed), "--input-format", "csv", "--backend", backend) + == MISMATCH + ) + + +# --------------------------------------------------------------------------- +# Joining +# --------------------------------------------------------------------------- + + +def test_on_index_joins_on_the_dataframe_index(cli, capsys): + assert cli("--on-index", "--backend", "pandas") == MISMATCH + + +@pytest.mark.parametrize( + "on_args", + [["--on", "id,name"], ["--on", "id", "--on", "name"]], +) +def test_multi_column_joins(cli, backend, on_args, capsys): + assert cli(*on_args, "--backend", backend) == MISMATCH + + +def test_unknown_join_column_exits_two(cli, backend, capsys): + assert cli("--on", "not_a_column", "--backend", backend) == ERROR + assert "not_a_column" in capsys.readouterr().err + + +def test_comparing_a_file_against_itself_gets_distinct_report_labels( + cli, left_csv, backend, capsys +): + # Identical dataset labels would make the report ambiguous, and the pandas + # backend cannot merge two frames that share a name. + assert cli("--right", str(left_csv), "--backend", backend) == MATCH + out = capsys.readouterr().out + assert "left_1" in out + assert "left_2" in out + + +# --------------------------------------------------------------------------- +# Tolerances +# --------------------------------------------------------------------------- + + +def test_global_absolute_tolerance_absorbs_the_difference(cli, backend, capsys): + assert ( + cli( + "--abs-tol", + "0.01", + "--max-unequal-rows", + "0", + "--ignore-unique-rows", + "--backend", + backend, + ) + == MATCH + ) + + +def test_per_column_tolerance_absorbs_the_difference(cli, backend, capsys): + assert ( + cli( + "--abs-tol", + "amount=0.01", + "--max-unequal-rows", + "0", + "--ignore-unique-rows", + "--backend", + backend, + ) + == MATCH + ) + + +def test_per_column_tolerance_on_another_column_does_not(cli, backend, capsys): + assert ( + cli( + "--abs-tol", + "name=99", + "--max-unequal-rows", + "0", + "--ignore-unique-rows", + "--backend", + backend, + ) + == MISMATCH + ) + + +def test_relative_tolerance(cli, backend, capsys): + assert ( + cli( + "--rel-tol", + "0.001", + "--max-unequal-rows", + "0", + "--ignore-unique-rows", + "--backend", + backend, + ) + == MATCH + ) + + +# --------------------------------------------------------------------------- +# Thresholds +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "threshold, expected", + [ + (0, MISMATCH), + (TOTAL_DIFFERING_ROWS - 1, MISMATCH), + (TOTAL_DIFFERING_ROWS, MATCH), + (TOTAL_DIFFERING_ROWS + 1, MATCH), + ], +) +def test_max_unequal_rows_counts_unique_rows_by_default( + cli, backend, threshold, expected, capsys +): + assert cli("--max-unequal-rows", str(threshold), "--backend", backend) == expected + + +@pytest.mark.parametrize( + "threshold, expected", + [(UNEQUAL_ROWS - 1, MISMATCH), (UNEQUAL_ROWS, MATCH)], +) +def test_ignore_unique_rows_counts_only_value_mismatches( + cli, backend, threshold, expected, capsys +): + assert ( + cli( + "--max-unequal-rows", + str(threshold), + "--ignore-unique-rows", + "--backend", + backend, + ) + == expected + ) + + +def test_extra_columns_fail_the_threshold_unless_ignored( + cli, tmp_path, left_frame, backend, capsys +): + wider = tmp_path / "wider.csv" + left_frame.assign(extra=1).to_csv(wider, index=False) + + threshold = ["--max-unequal-rows", str(TOTAL_DIFFERING_ROWS)] + assert cli("--left", str(wider), *threshold, "--backend", backend) == MISMATCH + assert ( + cli( + "--left", + str(wider), + *threshold, + "--ignore-extra-columns", + "--backend", + backend, + ) + == MATCH + ) + + +def test_ignore_extra_columns_without_a_threshold( + cli, tmp_path, left_frame, left_csv, backend, capsys +): + wider = tmp_path / "wider.csv" + left_frame.assign(extra=1).to_csv(wider, index=False) + + assert ( + cli("--left", str(wider), "--right", str(left_csv), "--backend", backend) + == MISMATCH + ) + assert ( + cli( + "--left", + str(wider), + "--right", + str(left_csv), + "--ignore-extra-columns", + "--backend", + backend, + ) + == MATCH + ) + + +# --------------------------------------------------------------------------- +# Empty and non-overlapping inputs +# --------------------------------------------------------------------------- + + +@pytest.fixture +def empty_csv(tmp_path, left_frame): + """The same columns as the fixtures, with no rows.""" + path = tmp_path / "empty.csv" + left_frame.iloc[:0].to_csv(path, index=False) + return path + + +def test_two_empty_datasets_are_not_a_match(cli, empty_csv, backend, capsys): + """An empty intersection is a non-match, not a match with nothing to report. + + Every backend's ``intersect_rows_match`` returns False when no rows overlap, + so "no rows differ" and "the datasets match" are different answers here. + ``within_threshold`` derives its verdict from the report data rather than + calling ``matches()``, and this pins the one case where the shortest reading + of those counts would disagree with the library. + """ + assert ( + cli("--left", str(empty_csv), "--right", str(empty_csv), "--backend", backend) + == MISMATCH + ) + + +def test_datasets_with_no_overlapping_join_keys_are_not_a_match( + cli, tmp_path, left_frame, backend, capsys +): + """Every row is unique to one side, so nothing intersects.""" + disjoint = tmp_path / "disjoint.csv" + left_frame.assign(id=left_frame["id"] + 100).to_csv(disjoint, index=False) + + assert cli("--right", str(disjoint), "--backend", backend) == MISMATCH + + +def test_empty_datasets_satisfy_an_explicit_zero_threshold( + cli, empty_csv, backend, capsys +): + """``--max-unequal-rows`` asks a different question, and gets a different answer. + + The threshold is a bound on how many rows differ, and zero rows differ, so an + empty comparison clears it even though it is not a match. Keeping the two + branches distinct is deliberate. + """ + assert ( + cli( + "--left", + str(empty_csv), + "--right", + str(empty_csv), + "--max-unequal-rows", + "0", + "--backend", + backend, + ) + == MATCH + ) + + +# --------------------------------------------------------------------------- +# Normalisation flags +# --------------------------------------------------------------------------- + + +def test_ignore_case(cli, tmp_path, left_frame, left_csv, backend, capsys): + upper = tmp_path / "upper.csv" + left_frame.assign(name=left_frame["name"].str.upper()).to_csv(upper, index=False) + + assert ( + cli("--left", str(upper), "--right", str(left_csv), "--backend", backend) + == MISMATCH + ) + assert ( + cli( + "--left", + str(upper), + "--right", + str(left_csv), + "--ignore-case", + "--backend", + backend, + ) + == MATCH + ) + + +def test_ignore_spaces(cli, tmp_path, left_frame, left_csv, backend, capsys): + padded = tmp_path / "padded.csv" + left_frame.assign(name=" " + left_frame["name"] + " ").to_csv(padded, index=False) + + assert ( + cli("--left", str(padded), "--right", str(left_csv), "--backend", backend) + == MISMATCH + ) + assert ( + cli( + "--left", + str(padded), + "--right", + str(left_csv), + "--ignore-spaces", + "--backend", + backend, + ) + == MATCH + ) + + +def test_cast_column_names_lower(cli, tmp_path, left_frame, left_csv, backend, capsys): + # Only the non-join column is uppercased, so --on id resolves either way and + # the flag alone decides whether the two frames line up. + shouty = tmp_path / "shouty.csv" + left_frame.rename(columns={"name": "NAME"}).to_csv(shouty, index=False) + + assert ( + cli("--left", str(shouty), "--right", str(left_csv), "--backend", backend) + == MATCH + ) + assert ( + cli( + "--left", + str(shouty), + "--right", + str(left_csv), + "--no-cast-column-names-lower", + "--backend", + backend, + ) + == MISMATCH + ) + + +# --------------------------------------------------------------------------- +# Input formats +# --------------------------------------------------------------------------- + + +def test_parquet_input(tmp_path, left_frame, right_frame, backend, capsys): + left = tmp_path / "left.parquet" + right = tmp_path / "right.parquet" + left_frame.to_parquet(left) + right_frame.to_parquet(right) + + assert ( + main( + [ + "compare", + "--left", + str(left), + "--right", + str(right), + "--on", + "id", + "--backend", + backend, + ] + ) + == MISMATCH + ) + + +def test_json_input(tmp_path, left_frame, right_frame, backend, capsys): + left = tmp_path / "left.json" + right = tmp_path / "right.json" + left_frame.to_json(left, orient="records") + right_frame.to_json(right, orient="records") + + assert ( + main( + [ + "compare", + "--left", + str(left), + "--right", + str(right), + "--on", + "id", + "--backend", + backend, + ] + ) + == MISMATCH + ) + + +def test_newline_delimited_json_input( + tmp_path, left_frame, right_frame, backend, capsys +): + left = tmp_path / "left.jsonl" + right = tmp_path / "right.jsonl" + left_frame.to_json(left, orient="records", lines=True) + right_frame.to_json(right, orient="records", lines=True) + + assert ( + main( + [ + "compare", + "--left", + str(left), + "--right", + str(right), + "--on", + "id", + "--backend", + backend, + ] + ) + == MISMATCH + ) + + +def test_mixed_input_formats_are_inferred_per_file( + tmp_path, left_frame, backend, capsys +): + csv_path = tmp_path / "left.csv" + parquet_path = tmp_path / "right.parquet" + left_frame.to_csv(csv_path, index=False) + left_frame.to_parquet(parquet_path) + + assert ( + main( + [ + "compare", + "--left", + str(csv_path), + "--right", + str(parquet_path), + "--on", + "id", + "--backend", + backend, + ] + ) + == MATCH + ) + + +def test_custom_csv_delimiter(tmp_path, left_frame, backend, capsys): + left = tmp_path / "left.csv" + right = tmp_path / "right.csv" + left_frame.to_csv(left, index=False, sep="\t") + left_frame.to_csv(right, index=False, sep="\t") + + assert ( + main( + [ + "compare", + "--left", + str(left), + "--right", + str(right), + "--on", + "id", + "--csv-delimiter", + r"\t", + "--backend", + backend, + ] + ) + == MATCH + ) + + +def test_tsv_extension_is_not_inferred(cli, tmp_path, left_frame, backend, capsys): + """``.tsv`` is not in the extension table, so it asks for an explicit format. + + Inferring it as CSV would pick the right reader and the wrong delimiter, + since ``--csv-delimiter`` applies to both sides at once and defaults to a + comma. Failing with a message that names the flag beats parsing the file + into a single mangled column. + """ + left = tmp_path / "left.tsv" + right = tmp_path / "right.tsv" + left_frame.to_csv(left, index=False, sep="\t") + left_frame.to_csv(right, index=False, sep="\t") + + assert ( + cli("--left", str(left), "--right", str(right), "--backend", backend) == ERROR + ) + assert "--input-format" in capsys.readouterr().err + + # Forcing both the format and the delimiter still works. + assert ( + cli( + "--left", + str(left), + "--right", + str(right), + "--input-format", + "csv", + "--csv-delimiter", + r"\t", + "--backend", + backend, + ) + == MATCH + ) + + +# --------------------------------------------------------------------------- +# Entry point wiring +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("module", ["datacompy", "datacompy.cli"]) +def test_module_entry_points_report_the_package_version(module): + import subprocess + + from datacompy import __version__ + + result = subprocess.run( + [sys.executable, "-m", module, "--version"], + capture_output=True, + text=True, + check=True, + ) + assert __version__ in result.stdout + + +def test_module_entry_point_propagates_the_exit_code(left_csv, right_csv): + import subprocess + + result = subprocess.run( + [ + sys.executable, + "-m", + "datacompy", + "compare", + "--left", + str(left_csv), + "--right", + str(right_csv), + "--on", + "id", + "--quiet", + ], + capture_output=True, + text=True, + ) + assert result.returncode == MISMATCH + assert result.stdout == "" diff --git a/tests/cli/test_output.py b/tests/cli/test_output.py new file mode 100644 index 00000000..80792f3f --- /dev/null +++ b/tests/cli/test_output.py @@ -0,0 +1,178 @@ +# +# Copyright 2026 Capital One Services, LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for report rendering and delivery. + +Rendering (``--report-format``) and destination (``--output``) are independent +axes, so the tests cover the combinations rather than a single flag each. +""" + +import json + +import pytest + +from tests.cli.conftest import TOTAL_DIFFERING_ROWS, UNEQUAL_ROWS, UNIQUE_ROWS + +MISMATCH = 1 +ERROR = 2 + + +# --------------------------------------------------------------------------- +# Rendering +# --------------------------------------------------------------------------- + + +def test_text_is_the_default_rendering(cli, capsys): + assert cli() == MISMATCH + out = capsys.readouterr().out + assert "DataComPy Comparison" in out + assert not out.lstrip().startswith("{") + + +def test_json_rendering_is_parseable_and_carries_the_counts(cli, capsys): + assert cli("--report-format", "json") == MISMATCH + payload = json.loads(capsys.readouterr().out) + + assert payload["row_summary"]["unequal_rows"] == UNEQUAL_ROWS + assert payload["row_summary"]["df1_unique"] == 1 + assert payload["row_summary"]["df2_unique"] == 1 + assert payload["column_summary"]["common_columns"] == 3 + assert payload["df1_name"] == "left" + assert payload["df2_name"] == "right" + + +def test_json_rendering_survives_numpy_scalars(cli, capsys): + """``max_diff`` and friends arrive as numpy scalars, which stdlib JSON rejects.""" + assert cli("--report-format", "json") == MISMATCH + payload = json.loads(capsys.readouterr().out) + stats = payload["mismatch_stats"]["stats"] + assert stats, "expected at least one mismatching column" + assert stats[0]["max_diff"] == pytest.approx(0.005) + + +def test_json_default_coerces_types_the_stdlib_encoder_rejects(): + """Spark and Snowflake surface numpy scalars that ``json.dumps`` cannot encode.""" + import datetime + + import numpy as np + from datacompy.cli.output import _json_default + + assert _json_default(np.int64(7)) == 7 + assert isinstance(_json_default(np.int64(7)), int) + assert _json_default(np.float64(1.5)) == pytest.approx(1.5) + assert isinstance(_json_default(np.float64(1.5)), float) + assert _json_default(np.bool_(True)) is True + # Anything else degrades to its string form rather than blowing up a report. + assert _json_default(datetime.date(2026, 7, 31)) == "2026-07-31" + + +def test_html_rendering_wraps_the_text_report(cli, capsys): + assert cli("--report-format", "html") == MISMATCH + out = capsys.readouterr().out + assert out.lstrip().startswith("") + assert "DataComPy Comparison" in out + + +# --------------------------------------------------------------------------- +# Destination +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("report_format", ["text", "json", "html"]) +def test_output_writes_a_file_and_creates_parent_directories( + cli, tmp_path, report_format, capsys +): + destination = tmp_path / "nested" / "deeper" / f"report.{report_format}" + assert ( + cli("--report-format", report_format, "--output", str(destination)) == MISMATCH + ) + + content = destination.read_text() + assert content + if report_format == "json": + assert json.loads(content)["row_summary"]["unequal_rows"] == UNEQUAL_ROWS + elif report_format == "html": + assert content.lstrip().startswith("") + else: + assert "DataComPy Comparison" in content + + +def test_output_still_prints_to_stdout_by_default(cli, tmp_path, capsys): + destination = tmp_path / "report.txt" + assert cli("--output", str(destination)) == MISMATCH + assert "DataComPy Comparison" in capsys.readouterr().out + assert "DataComPy Comparison" in destination.read_text() + + +def test_quiet_suppresses_stdout_but_still_writes_the_file(cli, tmp_path, capsys): + destination = tmp_path / "report.html" + assert ( + cli("--quiet", "--report-format", "html", "--output", str(destination)) + == MISMATCH + ) + assert capsys.readouterr().out == "" + assert destination.read_text().lstrip().startswith("") + + +def test_quiet_without_output_prints_nothing_at_all(cli, capsys): + assert cli("--quiet") == MISMATCH + captured = capsys.readouterr() + assert captured.out == "" + assert captured.err == "" + + +def test_quiet_and_json_still_suppresses_stdout(cli, capsys): + """PR 534's --quiet was a no-op alongside --json; format and silence are separate.""" + assert cli("--quiet", "--report-format", "json") == MISMATCH + assert capsys.readouterr().out == "" + + +def test_unwritable_output_exits_two(cli, tmp_path, capsys): + blocker = tmp_path / "blocker" + blocker.write_text("not a directory") + assert cli("--output", str(blocker / "report.txt")) == ERROR + assert "cannot write" in capsys.readouterr().err + + +# --------------------------------------------------------------------------- +# Report shaping +# --------------------------------------------------------------------------- + + +def test_sample_count_zero_suppresses_sample_rows(cli, capsys): + assert cli("--sample-count", "0", "--report-format", "json") == MISMATCH + payload = json.loads(capsys.readouterr().out) + assert payload["mismatch_stats"]["has_samples"] is False + assert payload["df1_unique_rows"]["has_rows"] is False + + +def test_column_count_is_reflected_in_the_report_data(cli, capsys): + assert cli("--column-count", "2", "--report-format", "json") == MISMATCH + assert json.loads(capsys.readouterr().out)["column_count"] == 2 + + +def test_dataset_labels_can_be_overridden(cli, capsys): + assert cli("--df1-name", "before", "--df2-name", "after") == MISMATCH + out = capsys.readouterr().out + assert "before" in out + assert "after" in out + + +def test_the_fixture_counts_are_what_the_report_says(cli, capsys): + assert cli("--report-format", "json") == MISMATCH + summary = json.loads(capsys.readouterr().out)["row_summary"] + total = summary["unequal_rows"] + summary["df1_unique"] + summary["df2_unique"] + assert total == TOTAL_DIFFERING_ROWS + assert summary["df1_unique"] + summary["df2_unique"] == UNIQUE_ROWS diff --git a/tests/cli/test_parser.py b/tests/cli/test_parser.py new file mode 100644 index 00000000..b3ecf74e --- /dev/null +++ b/tests/cli/test_parser.py @@ -0,0 +1,366 @@ +# +# Copyright 2026 Capital One Services, LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for the argument specification and parser construction.""" + +import argparse +import inspect + +import pytest +from datacompy.cli.backends import BACKENDS, compare_kwargs +from datacompy.cli.errors import BadArgsError, MissingExtraError +from datacompy.cli.parser import ( + ALL_BACKENDS, + OPTIONS, + OPTIONS_BY_FLAG, + build_parser, + fill_defaults, + join_column_group, + non_negative_int, + single_char, + tolerance, +) + + +def _parse(*args: str) -> argparse.Namespace: + """Parse a ``compare`` invocation without running it.""" + return build_parser().parse_args(["compare", *args]) + + +def _minimal(*extra: str) -> argparse.Namespace: + """Parse a valid minimal invocation plus *extra*, with defaults filled in.""" + namespace = _parse("--left", "a.csv", "--right", "b.csv", "--on", "id", *extra) + fill_defaults(namespace) + return namespace + + +# --------------------------------------------------------------------------- +# Specification integrity +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("backend_name", sorted(ALL_BACKENDS)) +def test_every_option_kwarg_exists_on_its_backend_constructor(backend_name): + """Guard against the parser drifting from the library constructor signatures. + + This is the reason the option table exists. If someone renames a keyword on + a ``*Compare`` class, or adds a CLI row with a typo, this fails immediately + instead of at runtime for a user. + """ + backend = BACKENDS[backend_name] + try: + compare_cls = backend.compare_cls + except MissingExtraError: + pytest.skip(f"datacompy[{backend.extra}] is not installed") + + parameters = inspect.signature(compare_cls.__init__).parameters + for opt in OPTIONS: + if opt.kwarg is None or backend_name not in opt.backends: + continue + assert opt.kwarg in parameters, ( + f"{opt.flags[0]} maps to {opt.kwarg!r}, which " + f"{compare_cls.__name__}.__init__ does not accept" + ) + + +#: Constructor keywords the CLI deliberately does not expose, and why. +UNEXPOSED_KWARGS = { + "custom_comparators": "takes comparator instances, which a flag cannot express", +} + +#: Constructor parameters that carry the data or the session, not configuration. +_NOT_CONFIGURATION = frozenset({"self", "df1", "df2", "spark_session", "session"}) + + +@pytest.mark.parametrize("backend_name", sorted(ALL_BACKENDS)) +def test_every_backend_constructor_kwarg_is_exposed_or_allowlisted(backend_name): + """The drift guard pointed the other way. + + The forward guard catches a CLI row that names a keyword the library does + not have. This one catches the library growing a keyword the CLI never + surfaces, which is the failure that stays invisible: the command keeps + working, it just quietly cannot reach the new behaviour. Anything genuinely + not expressible as a flag belongs in ``UNEXPOSED_KWARGS`` with a reason. + """ + backend = BACKENDS[backend_name] + try: + compare_cls = backend.compare_cls + except MissingExtraError: + pytest.skip(f"datacompy[{backend.extra}] is not installed") + + exposed = {opt.kwarg for opt in OPTIONS if opt.kwarg} + for name in inspect.signature(compare_cls.__init__).parameters: + if name in _NOT_CONFIGURATION: + continue + assert name in exposed or name in UNEXPOSED_KWARGS, ( + f"{compare_cls.__name__}.__init__ accepts {name!r}, which no CLI " + f"option supplies. Add an Opt row for it, or add it to " + f"UNEXPOSED_KWARGS with the reason it cannot be a flag." + ) + + +def test_the_unexposed_allowlist_has_no_stale_entries(): + """An allowlist that outlives the parameter it excuses is just misleading.""" + known = set() + for backend in BACKENDS.values(): + try: + compare_cls = backend.compare_cls + except MissingExtraError: + continue + known |= set(inspect.signature(compare_cls.__init__).parameters) + + for name in UNEXPOSED_KWARGS: + assert name in known, f"{name!r} is allowlisted but no constructor takes it" + + +def test_option_flags_and_destinations_are_unique(): + flags = [flag for opt in OPTIONS for flag in opt.flags] + assert len(flags) == len(set(flags)) + dests = [opt.dest for opt in OPTIONS] + assert len(dests) == len(set(dests)) + + +def test_option_backends_are_known_backend_names(): + for opt in OPTIONS: + assert opt.backends <= ALL_BACKENDS, opt.flags + assert opt.backends, f"{opt.flags[0]} applies to no backend" + + +def test_every_backend_name_has_a_backend_implementation(): + assert set(BACKENDS) == ALL_BACKENDS + + +def test_compare_kwargs_omits_options_the_backend_does_not_accept(): + namespace = _minimal() + assert "cast_column_names_lower" in compare_kwargs(namespace, "polars") + assert "cast_column_names_lower" not in compare_kwargs(namespace, "snowflake") + assert "on_index" in compare_kwargs(namespace, "pandas") + assert "on_index" not in compare_kwargs(namespace, "polars") + assert compare_kwargs(namespace, "spark")["cache_intermediates"] is True + assert "cache_intermediates" not in compare_kwargs(namespace, "polars") + + +def test_cache_intermediates_can_be_turned_off(): + """Databricks Serverless and similar environments cannot cache.""" + namespace = _minimal("--no-cache-intermediates") + assert compare_kwargs(namespace, "spark")["cache_intermediates"] is False + + +def test_compare_kwargs_omits_unset_values_so_library_defaults_apply(): + namespace = _minimal() + kwargs = compare_kwargs(namespace, "polars") + assert "abs_tol" not in kwargs + assert "rel_tol" not in kwargs + assert kwargs["join_columns"] == ["id"] + + +# --------------------------------------------------------------------------- +# SUPPRESS defaults +# --------------------------------------------------------------------------- + + +def test_absent_flags_leave_no_attribute_until_defaults_are_filled(): + namespace = _parse("--left", "a.csv", "--right", "b.csv", "--on", "id") + assert not OPTIONS_BY_FLAG["--ignore-case"].was_given(namespace) + assert not hasattr(namespace, "ignore_case") + + fill_defaults(namespace) + assert namespace.ignore_case is False + assert namespace.backend == "polars" + assert namespace.debug is False + + +def test_explicitly_passed_flag_is_distinguishable_from_its_default(): + namespace = _parse( + "--left", "a.csv", "--right", "b.csv", "--on", "id", "--cast-column-names-lower" + ) + opt = OPTIONS_BY_FLAG["--cast-column-names-lower"] + assert opt.was_given(namespace) + assert opt.value(namespace) is True + + namespace = _parse("--left", "a.csv", "--right", "b.csv", "--on", "id") + assert not opt.was_given(namespace) + assert opt.value(namespace) is True + + +# --------------------------------------------------------------------------- +# ``--on`` +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "args, expected", + [ + (["--on", "id"], ["id"]), + (["--on", "id,date"], ["id", "date"]), + (["--on", "id", "--on", "date"], ["id", "date"]), + (["--on", "id,date", "--on", "region"], ["id", "date", "region"]), + (["--on", " id , date "], ["id", "date"]), + ], +) +def test_on_accepts_comma_separated_and_repeated_forms(args, expected): + namespace = _parse("--left", "a.csv", "--right", "b.csv", *args) + fill_defaults(namespace) + assert OPTIONS_BY_FLAG["--on"].resolved(namespace) == expected + + +def test_on_rejects_an_empty_value(): + with pytest.raises(argparse.ArgumentTypeError): + join_column_group(",,") + + +# --------------------------------------------------------------------------- +# Tolerances +# --------------------------------------------------------------------------- + + +def test_tolerance_parses_a_bare_number_and_a_column_pair(): + assert tolerance("0.01") == pytest.approx(0.01) + assert tolerance("price=0.01") == ("price", pytest.approx(0.01)) + assert tolerance(" price =0.01") == ("price", pytest.approx(0.01)) + + +@pytest.mark.parametrize("value", ["nope", "price=nope", "=0.1", "-0.5"]) +def test_tolerance_rejects_bad_values(value): + with pytest.raises(argparse.ArgumentTypeError): + tolerance(value) + + +def test_tolerance_resolves_to_a_float_or_a_per_column_dict(): + namespace = _minimal("--abs-tol", "0.01") + assert OPTIONS_BY_FLAG["--abs-tol"].resolved(namespace) == pytest.approx(0.01) + + namespace = _minimal("--abs-tol", "price=0.01", "--abs-tol", "qty=0") + assert OPTIONS_BY_FLAG["--abs-tol"].resolved(namespace) == pytest.approx( + {"price": 0.01, "qty": 0.0} + ) + + +def test_mixing_a_bare_tolerance_with_column_pairs_is_rejected(): + namespace = _minimal("--abs-tol", "0.01", "--abs-tol", "price=0.02") + with pytest.raises(BadArgsError, match="not both"): + OPTIONS_BY_FLAG["--abs-tol"].resolved(namespace) + + +def test_repeating_a_bare_tolerance_is_rejected(): + namespace = _minimal("--rel-tol", "0.01", "--rel-tol", "0.02") + with pytest.raises(BadArgsError, match="more than once"): + OPTIONS_BY_FLAG["--rel-tol"].resolved(namespace) + + +# --------------------------------------------------------------------------- +# Other type callables +# --------------------------------------------------------------------------- + + +def test_single_char_translates_an_escaped_tab(): + assert single_char(",") == "," + assert single_char("\\t") == "\t" + assert single_char("\t") == "\t" + + +@pytest.mark.parametrize("value", ["", ";;", "ab"]) +def test_single_char_rejects_anything_but_one_character(value): + with pytest.raises(argparse.ArgumentTypeError): + single_char(value) + + +def test_non_negative_int(): + assert non_negative_int("0") == 0 + assert non_negative_int("42") == 42 + for value in ("-1", "1.5", "many"): + with pytest.raises(argparse.ArgumentTypeError): + non_negative_int(value) + + +# --------------------------------------------------------------------------- +# Parser wiring +# --------------------------------------------------------------------------- + + +def test_dataset_names_default_to_the_input_reference(): + namespace = _minimal("--left", "/data/sales_2024.parquet") + # The later --left wins, so the default label comes from that file. + assert OPTIONS_BY_FLAG["--df1-name"].resolved(namespace) == "sales_2024" + assert OPTIONS_BY_FLAG["--df2-name"].resolved(namespace) == "b" + + +def test_snowflake_dataset_names_use_the_table_not_the_schema(): + namespace = _parse( + "--left", + "PROD.ANALYTICS.SALES", + "--right", + "STAGE.ANALYTICS.ORDERS", + "--on", + "id", + "--backend", + "snowflake", + ) + fill_defaults(namespace) + assert OPTIONS_BY_FLAG["--df1-name"].resolved(namespace) == "SALES" + assert OPTIONS_BY_FLAG["--df2-name"].resolved(namespace) == "ORDERS" + + +def test_colliding_default_dataset_names_are_disambiguated(): + """Two sides sharing a label would make the report ambiguous. + + Comparing the same table across environments is the common Snowflake case, + and comparing a file against itself is the common file case. Both derive the + same label, so the sides are numbered instead. + """ + namespace = _parse( + "--left", + "PROD.ANALYTICS.SALES", + "--right", + "STAGE.ANALYTICS.SALES", + "--on", + "id", + "--backend", + "snowflake", + ) + fill_defaults(namespace) + assert OPTIONS_BY_FLAG["--df1-name"].resolved(namespace) == "SALES_1" + assert OPTIONS_BY_FLAG["--df2-name"].resolved(namespace) == "SALES_2" + + +def test_explicit_dataset_names_win(): + namespace = _minimal("--df1-name", "before", "--df2-name", "after") + assert OPTIONS_BY_FLAG["--df1-name"].resolved(namespace) == "before" + assert OPTIONS_BY_FLAG["--df2-name"].resolved(namespace) == "after" + + +@pytest.mark.parametrize( + "argv", + [ + ["--debug", "compare", "--left", "a.csv", "--right", "b.csv", "--on", "id"], + ["compare", "--left", "a.csv", "--right", "b.csv", "--on", "id", "--debug"], + ], +) +def test_debug_is_accepted_on_either_side_of_the_subcommand(argv): + assert build_parser().parse_args(argv).debug is True + + +def test_subcommand_is_required(): + with pytest.raises(SystemExit): + build_parser().parse_args([]) + + +def test_version_exits_cleanly(capsys): + from datacompy import __version__ + + with pytest.raises(SystemExit) as excinfo: + build_parser().parse_args(["--version"]) + assert excinfo.value.code == 0 + assert __version__ in capsys.readouterr().out diff --git a/tests/cli/test_snowflake.py b/tests/cli/test_snowflake.py new file mode 100644 index 00000000..7c71b7bd --- /dev/null +++ b/tests/cli/test_snowflake.py @@ -0,0 +1,267 @@ +# +# Copyright 2026 Capital One Services, LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for the Snowflake backend's reference handling and session parameters. + +None of these need Snowpark installed. Reference resolution only needs +``get_current_database``, so a stub session stands in, and the connection +parameter builder is pure. The comparison itself is exercised by the library's +own Snowflake tests. +""" + +import json + +import pytest +from datacompy.cli.backends import SnowflakeBackend, _snowflake_params +from datacompy.cli.errors import BadArgsError +from datacompy.cli.parser import build_parser, fill_defaults + +SNOWFLAKE_ENV = ( + "SNOWFLAKE_ACCOUNT", + "SNOWFLAKE_USER", + "SNOWFLAKE_PASSWORD", + "SNOWFLAKE_AUTHENTICATOR", + "SNOWFLAKE_TOKEN", + "SNOWFLAKE_ROLE", + "SNOWFLAKE_WAREHOUSE", + "SNOWFLAKE_DATABASE", + "SNOWFLAKE_SCHEMA", +) + + +class StubSession: + """Minimal stand in exposing only what reference resolution touches.""" + + def __init__(self, database=None): + self._database = database + + def get_current_database(self): + return self._database + + +@pytest.fixture +def namespace(): + """A parsed Snowflake invocation with defaults filled in.""" + parsed = build_parser().parse_args( + [ + "compare", + "--left", + "PROD.ANALYTICS.SALES", + "--right", + "STAGE.ANALYTICS.SALES", + "--on", + "id", + "--backend", + "snowflake", + ] + ) + fill_defaults(parsed) + return parsed + + +@pytest.fixture +def clean_env(monkeypatch): + """Remove every Snowflake environment variable so tests start from nothing.""" + for name in SNOWFLAKE_ENV: + monkeypatch.delenv(name, raising=False) + + +# --------------------------------------------------------------------------- +# Reference resolution +# --------------------------------------------------------------------------- + + +def test_three_part_reference_is_used_as_is(namespace): + backend = SnowflakeBackend() + session = StubSession(database="OTHER") + assert backend.load(session, "PROD.ANALYTICS.SALES", namespace) == ( + "PROD.ANALYTICS.SALES" + ) + + +def test_two_part_reference_is_qualified_with_the_current_database(namespace): + backend = SnowflakeBackend() + session = StubSession(database="PROD") + assert backend.load(session, "ANALYTICS.SALES", namespace) == "PROD.ANALYTICS.SALES" + + +def test_two_part_reference_without_a_current_database_is_rejected(namespace): + backend = SnowflakeBackend() + with pytest.raises(BadArgsError, match="no current database"): + backend.load(StubSession(database=None), "ANALYTICS.SALES", namespace) + + +@pytest.mark.parametrize( + "ref", + [ + "/data/sales.parquet", + "s3://bucket/sales.parquet", + "relative/sales.parquet", + "SALES", + "A.B.C.D", + "1BAD.TABLE", + "@stage/file.parquet", + "", + ], +) +def test_non_table_references_are_rejected(namespace, ref): + """The backend never guesses whether a reference is a file or a table. + + With ``--backend snowflake`` a reference is always a table, so anything that + is not a two or three part identifier is a clear argument error rather than + something to fall back on. + """ + backend = SnowflakeBackend() + with pytest.raises(BadArgsError, match="not a Snowflake table reference"): + backend.load(StubSession(database="PROD"), ref, namespace) + + +def test_a_bare_filename_is_read_as_a_table_reference(namespace): + """``data.csv`` is a syntactically valid ``schema.table``, and is treated as one. + + This is the deliberate consequence of dropping file versus table guessing. + Snowflake reports the missing table, which is clearer than a heuristic that + is sometimes wrong in the other direction. Path-like and URI references are + still rejected here, because they can never be identifiers. + """ + backend = SnowflakeBackend() + assert backend.load(StubSession(database="PROD"), "data.csv", namespace) == ( + "PROD.data.csv" + ) + + +@pytest.mark.parametrize( + "ref", ["DB.SCHEMA.TABLE", "db.schema.table", "_DB.$SCHEMA.T1", "SCHEMA.TABLE"] +) +def test_valid_table_reference_shapes(namespace, ref): + backend = SnowflakeBackend() + assert backend.load(StubSession(database="PROD"), ref, namespace) + + +# --------------------------------------------------------------------------- +# Connection parameters +# --------------------------------------------------------------------------- + + +def test_config_file_is_used_verbatim(tmp_path): + config = tmp_path / "connection.json" + config.write_text(json.dumps({"account": "acct", "user": "u", "password": "p"})) + assert _snowflake_params(config) == { + "account": "acct", + "user": "u", + "password": "p", + } + + +def test_missing_config_file_is_reported(tmp_path): + with pytest.raises(BadArgsError, match="file not found"): + _snowflake_params(tmp_path / "absent.json") + + +def test_malformed_config_file_is_reported(tmp_path): + config = tmp_path / "connection.json" + config.write_text("{not json") + with pytest.raises(BadArgsError, match="snowflake-config"): + _snowflake_params(config) + + +def test_config_file_must_hold_an_object(tmp_path): + config = tmp_path / "connection.json" + config.write_text(json.dumps(["account", "user"])) + with pytest.raises(BadArgsError, match="JSON object"): + _snowflake_params(config) + + +def test_parameters_are_built_from_the_environment(monkeypatch, clean_env): + monkeypatch.setenv("SNOWFLAKE_ACCOUNT", "acct") + monkeypatch.setenv("SNOWFLAKE_USER", "u") + monkeypatch.setenv("SNOWFLAKE_PASSWORD", "p") + monkeypatch.setenv("SNOWFLAKE_WAREHOUSE", "wh") + + assert _snowflake_params(None) == { + "account": "acct", + "user": "u", + "password": "p", + "warehouse": "wh", + } + + +def test_sso_authenticator_replaces_the_password(monkeypatch, clean_env): + monkeypatch.setenv("SNOWFLAKE_ACCOUNT", "acct") + monkeypatch.setenv("SNOWFLAKE_USER", "u") + monkeypatch.setenv("SNOWFLAKE_AUTHENTICATOR", "externalbrowser") + + params = _snowflake_params(None) + assert params["authenticator"] == "externalbrowser" + assert "password" not in params + + +def test_a_bare_token_selects_oauth_without_a_user(monkeypatch, clean_env): + monkeypatch.setenv("SNOWFLAKE_ACCOUNT", "acct") + monkeypatch.setenv("SNOWFLAKE_TOKEN", "tok") + + assert _snowflake_params(None) == { + "account": "acct", + "token": "tok", + "authenticator": "oauth", + } + + +def test_an_explicit_oauth_authenticator_is_left_alone(monkeypatch, clean_env): + monkeypatch.setenv("SNOWFLAKE_ACCOUNT", "acct") + monkeypatch.setenv("SNOWFLAKE_AUTHENTICATOR", "OAuth") + monkeypatch.setenv("SNOWFLAKE_TOKEN", "tok") + + params = _snowflake_params(None) + assert params["authenticator"] == "OAuth" + assert params["token"] == "tok" + + +def test_a_user_is_still_passed_through_under_oauth(monkeypatch, clean_env): + monkeypatch.setenv("SNOWFLAKE_ACCOUNT", "acct") + monkeypatch.setenv("SNOWFLAKE_TOKEN", "tok") + monkeypatch.setenv("SNOWFLAKE_USER", "u") + + assert _snowflake_params(None)["user"] == "u" + + +def test_oauth_without_a_token_is_rejected(monkeypatch, clean_env): + monkeypatch.setenv("SNOWFLAKE_ACCOUNT", "acct") + monkeypatch.setenv("SNOWFLAKE_USER", "u") + monkeypatch.setenv("SNOWFLAKE_AUTHENTICATOR", "oauth") + + with pytest.raises(BadArgsError, match="SNOWFLAKE_TOKEN"): + _snowflake_params(None) + + +def test_missing_required_environment_variables_are_named(monkeypatch, clean_env): + monkeypatch.setenv("SNOWFLAKE_PASSWORD", "p") + with pytest.raises(BadArgsError, match="SNOWFLAKE_ACCOUNT, SNOWFLAKE_USER"): + _snowflake_params(None) + + +def test_a_user_is_still_required_outside_oauth(monkeypatch, clean_env): + monkeypatch.setenv("SNOWFLAKE_ACCOUNT", "acct") + monkeypatch.setenv("SNOWFLAKE_AUTHENTICATOR", "externalbrowser") + + with pytest.raises(BadArgsError, match="SNOWFLAKE_USER"): + _snowflake_params(None) + + +def test_missing_credentials_are_reported(monkeypatch, clean_env): + monkeypatch.setenv("SNOWFLAKE_ACCOUNT", "acct") + monkeypatch.setenv("SNOWFLAKE_USER", "u") + with pytest.raises(BadArgsError, match="SNOWFLAKE_PASSWORD"): + _snowflake_params(None) diff --git a/tests/cli/test_spark.py b/tests/cli/test_spark.py new file mode 100644 index 00000000..f1e140b9 --- /dev/null +++ b/tests/cli/test_spark.py @@ -0,0 +1,241 @@ +# +# Copyright 2026 Capital One Services, LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for the Spark backend of the CLI. + +These need ``datacompy[spark]`` and Java 17. The session is created and stopped +by the CLI itself, which is part of what is under test, so no session fixture is +shared with the rest of the suite. +""" + +import pytest +from datacompy.cli import main + +pytest.importorskip("pyspark", reason="requires datacompy[spark]") + +MATCH = 0 +MISMATCH = 1 +ERROR = 2 + + +@pytest.fixture +def no_borrowed_session(): + """Assert the CLI will be the one creating the session. + + A SparkSession is process wide, so the ownership tests below can only say + anything if nothing else in the run has one open. ``pytest-spark`` supplies + a session scoped ``spark_session`` fixture that would be exactly that, and + today it is simply not instantiated yet when ``tests/cli`` runs. Skipping + makes that dependency visible instead of turning it into a confusing + failure if the collection order ever changes. + """ + from pyspark.sql import SparkSession + + if SparkSession.getActiveSession() is not None: + pytest.skip("a SparkSession is already active, so ownership cannot be asserted") + yield + + +@pytest.fixture +def borrowed_session(): + """A session the CLI did not create, standing in for a notebook or Airflow task.""" + from pyspark.sql import SparkSession + + session = SparkSession.builder.appName("datacompy-cli-borrowed").getOrCreate() + try: + yield session + finally: + session.stop() + + +def test_spark_compares_csv_files(left_csv, right_csv, capsys): + exit_code = main( + [ + "compare", + "--left", + str(left_csv), + "--right", + str(right_csv), + "--on", + "id", + "--backend", + "spark", + ] + ) + assert exit_code == MISMATCH + assert "DataComPy Comparison" in capsys.readouterr().out + + +def test_spark_matches_identical_files(left_csv, tmp_path, left_frame, capsys): + copy = tmp_path / "copy.csv" + left_frame.to_csv(copy, index=False) + + exit_code = main( + [ + "compare", + "--left", + str(left_csv), + "--right", + str(copy), + "--on", + "id", + "--backend", + "spark", + ] + ) + assert exit_code == MATCH + + +def test_spark_session_is_stopped_even_when_loading_fails( + no_borrowed_session, tmp_path, capsys +): + """The session is registered on an ExitStack, so it closes on the error path too.""" + from pyspark.sql import SparkSession + + exit_code = main( + [ + "compare", + "--left", + str(tmp_path / "absent.csv"), + "--right", + str(tmp_path / "absent.csv"), + "--on", + "id", + "--backend", + "spark", + ] + ) + assert exit_code == ERROR + assert SparkSession.getActiveSession() is None + + +def test_spark_app_name_is_accepted(left_csv, right_csv, capsys): + exit_code = main( + [ + "compare", + "--left", + str(left_csv), + "--right", + str(right_csv), + "--on", + "id", + "--backend", + "spark", + "--spark-app-name", + "datacompy-cli-test", + ] + ) + assert exit_code == MISMATCH + + +def test_a_borrowed_session_survives_the_comparison( + borrowed_session, left_csv, right_csv, capsys +): + """``main`` must not stop a session it did not create. + + ``getOrCreate`` hands back the caller's session, and a SparkContext is + process wide, so stopping it here would break a notebook or an Airflow task + that called ``main`` in process and carried on afterwards. + """ + from pyspark.sql import SparkSession + + exit_code = main( + [ + "compare", + "--left", + str(left_csv), + "--right", + str(right_csv), + "--on", + "id", + "--backend", + "spark", + ] + ) + + assert exit_code == MISMATCH + assert SparkSession.getActiveSession() is not None + # Still usable, not merely still referenced. + assert borrowed_session.createDataFrame([(1,)], ["x"]).count() == 1 + + +def test_cache_intermediates_changes_what_spark_does( + borrowed_session, left_csv, right_csv, monkeypatch, capsys +): + """The flag has to reach Spark, not just survive parsing. + + ``SparkSQLCompare`` caches ``intersect_rows`` and unpersists it again before + the comparison returns, so nothing is left to inspect once the CLI exits. + Counting ``cache()`` calls is what separates the two runs. + + The class to patch is taken from a DataFrame the session actually produces. + ``cache`` is overridden on the concrete class, so patching the + ``pyspark.sql.DataFrame`` base would never intercept, and the concrete + class is not at the same import path across PySpark versions. + """ + df_cls = type(borrowed_session.createDataFrame([(1,)], ["x"])) + original = df_cls.cache + calls = [] + + def spy(self): + calls.append(self) + return original(self) + + monkeypatch.setattr(df_cls, "cache", spy) + + argv = [ + "compare", + "--left", + str(left_csv), + "--right", + str(right_csv), + "--on", + "id", + "--backend", + "spark", + "--quiet", + ] + + assert main(argv) == MISMATCH + cached_by_default = len(calls) + + calls.clear() + assert main([*argv, "--no-cache-intermediates"]) == MISMATCH + cached_when_disabled = len(calls) + + assert cached_by_default > 0, "expected caching to be enabled by default" + assert cached_when_disabled == 0 + + +def test_a_borrowed_session_survives_a_failed_comparison( + borrowed_session, tmp_path, capsys +): + """The error path unwinds the ExitStack too, and must not stop it either.""" + exit_code = main( + [ + "compare", + "--left", + str(tmp_path / "absent.csv"), + "--right", + str(tmp_path / "absent.csv"), + "--on", + "id", + "--backend", + "spark", + ] + ) + + assert exit_code == ERROR + assert borrowed_session.createDataFrame([(1,)], ["x"]).count() == 1