Follow-up from #545 and #549.
Context
When a CSV family input is read with the wrong delimiter, every row collapses into one column whose name is the entire header line. The comparison then fails on the join columns, and the only thing the user sees is the library's message:
$ datacompy compare --left left.tsv --right right.tsv --on id --backend pandas
datacompy: df1 must have all columns from join_columns: {'id'}
That message is correct and points nowhere near the cause. id really is absent, because the frame has a single column named id,name,amount. The user is left looking at their join columns rather than at the delimiter.
#545 makes this rarer by inferring the delimiter from the extension, but it also removes the guard that used to catch the case. Before that change a .tsv input without --input-format failed with an actionable "cannot infer the format" error. Afterwards, a comma delimited file that happens to be named .tsv, or a tab delimited file with no distinguishing extension, lands on the message above instead. Under --on-index there is no error at all: the comparison runs on one mangled string column per side and reports on that.
The message comes from the comparison classes (datacompy/pandas.py:262, datacompy/polars.py:254, datacompy/spark.py:322) as a plain ValueError, and datacompy/cli/compare.py:62 translates it into a BadArgsError verbatim.
What to do
Keep the library messages as they are and add the hint in the CLI, which is the only layer that knows the input was a file, what extension it had, and which delimiter was chosen for it.
When the missing join columns error fires, and the offending side was read as csv, and the resulting frame has exactly one column whose name contains a plausible delimiter (,, \t, ;, |), append a second line:
datacompy: df1 must have all columns from join_columns: {'id'}
Hint: 'left.tsv' parsed into a single column named 'id,name,amount'. The
delimiter may be wrong. Pass --csv-delimiter to override the extension based
default.
The ValueError is raised inside backend.build, well after the loaders have run, so the loaders need to record what they did. A small per side record (reference, resolved format, resolved delimiter, column count and names) carried through run() is enough, and all four backends expose .columns on the frame they return.
The same check is worth applying on the --on-index path, where nothing raises today. A single column frame on either side whose name contains a plausible delimiter should produce a warning on stderr before the report, since a silent comparison of two mangled strings is the worst version of this failure.
Acceptance criteria
- A comma delimited file named
.tsv compared with --on id reports the missing join column and the delimiter hint, naming the offending input
- The hint names the file whose parse looks wrong, not both sides, when only one side is affected
- A genuine missing join column on a correctly parsed file, for example
--on nope against a normal .csv, is unchanged and gains no hint
- A legitimate single column CSV whose header contains no delimiter character gains no hint
- The
--on-index path warns on stderr when either side parsed into a single suspicious column, and still exits on the comparison result rather than failing
- Library exception messages and types are unchanged, so this stays a CLI concern
Out of scope
Sniffing the delimiter from file content, either to guess it or to validate the extension based choice. The hint only reads what was already parsed. Detecting the delimiter properly is a larger change and would need its own discussion about precedence against --csv-delimiter and the extension.
Follow-up from #545 and #549.
Context
When a CSV family input is read with the wrong delimiter, every row collapses into one column whose name is the entire header line. The comparison then fails on the join columns, and the only thing the user sees is the library's message:
That message is correct and points nowhere near the cause.
idreally is absent, because the frame has a single column namedid,name,amount. The user is left looking at their join columns rather than at the delimiter.#545 makes this rarer by inferring the delimiter from the extension, but it also removes the guard that used to catch the case. Before that change a
.tsvinput without--input-formatfailed with an actionable "cannot infer the format" error. Afterwards, a comma delimited file that happens to be named.tsv, or a tab delimited file with no distinguishing extension, lands on the message above instead. Under--on-indexthere is no error at all: the comparison runs on one mangled string column per side and reports on that.The message comes from the comparison classes (
datacompy/pandas.py:262,datacompy/polars.py:254,datacompy/spark.py:322) as a plainValueError, anddatacompy/cli/compare.py:62translates it into aBadArgsErrorverbatim.What to do
Keep the library messages as they are and add the hint in the CLI, which is the only layer that knows the input was a file, what extension it had, and which delimiter was chosen for it.
When the missing join columns error fires, and the offending side was read as
csv, and the resulting frame has exactly one column whose name contains a plausible delimiter (,,\t,;,|), append a second line:The
ValueErroris raised insidebackend.build, well after the loaders have run, so the loaders need to record what they did. A small per side record (reference, resolved format, resolved delimiter, column count and names) carried throughrun()is enough, and all four backends expose.columnson the frame they return.The same check is worth applying on the
--on-indexpath, where nothing raises today. A single column frame on either side whose name contains a plausible delimiter should produce a warning on stderr before the report, since a silent comparison of two mangled strings is the worst version of this failure.Acceptance criteria
.tsvcompared with--on idreports the missing join column and the delimiter hint, naming the offending input--on nopeagainst a normal.csv, is unchanged and gains no hint--on-indexpath warns on stderr when either side parsed into a single suspicious column, and still exits on the comparison result rather than failingOut of scope
Sniffing the delimiter from file content, either to guess it or to validate the extension based choice. The hint only reads what was already parsed. Detecting the delimiter properly is a larger change and would need its own discussion about precedence against
--csv-delimiterand the extension.