Skip to content

extract_numbers.py: a CORRECT multi-year deck reports 5 'high severity' inconsistencies — grouping ignores the period #339

Description

@AB2006-personal

Summary

find_inconsistencies in skills/ib-check-deck/scripts/extract_numbers.py
groups figures by category alone and flags anything more than 5% from the
largest group. A deck carrying FY2023, FY2024 and FY2025E revenue is therefore
self-contradictory by construction — which is every deck with a financial
history.

Measured on a four-line deck: a correct deck produced 5 "high severity"
inconsistencies
, against 1 for a deck containing a genuine contradiction.
Precision ≈ 1/6. The true finding is indistinguishable from the noise.

Reproduce

clean.md — correct, no contradiction:

## Slide 2 — Historical Performance
Revenue grew from $100.0 million in FY2023 to $120.0 million in FY2024.
EBITDA was $20.0 million in FY2023 and $25.0 million in FY2024.
## Slide 5 — Projections
Revenue of $145.0 million is forecast for FY2025E.

planted.md — one genuine contradiction (FY2024 revenue stated twice):

## Slide 2 — Historical Performance
Revenue was $120.0 million in FY2024.
## Slide 7 — Transaction Summary
FY2024 revenue of $135.0 million supports the valuation.
$ python extract_numbers.py clean.md   --check   ->  5 inconsistencies (all severity "high")
$ python extract_numbers.py planted.md --check   ->  1 inconsistency

On the clean deck it reports revenue $100.0million vs $120.0million,
vs $145.0million, and ebitda $20.0million vs $25.0million — all correct
figures for different periods. (One of the five is 2023t, the separate
parsing defect filed alongside this.)

Cause

# lines ~192–210
for num in numbers:
    if num.category != 'other':
        by_category[num.category].append(num)   # period is discarded
...
if diff_pct < 0.05:                              # 5% tolerance on VALUE only

Nothing distinguishes "the same metric stated twice inconsistently" from "the
same metric across three periods".

Suggested fix

Key the comparison on (category, period), not category alone:

  • parse a period marker near each figure — FY2024, FY24, 2025E, 2023A;
  • attribute a figure to the period that follows it before the next figure,
    falling back to the preceding one. Nearest-by-distance is wrong on the
    commonest sentence there is: in "EBITDA was $20.0m in FY2023 and $25.0m in
    FY2024"
    , 25.0 sits 13 chars after FY2023 and 18 before FY2024, so
    distance alone assigns both figures to FY2023 and re-creates the false
    positive;
  • only report when two values share a category and a period;
  • skip figures with no resolvable period rather than pooling them.

With that change the two fixtures above give 0 and 1 respectively.

Happy to open a PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions