Score, diagnose, and fix tabular data before it hits your pipeline. DataLens produces an auditable quality score (0–100), sector-specific data contracts, column-level diagnostics, interactive dashboards, one-click fixes, and schema drift detection — all grounded in deterministic Python profiling.
DataLens is a local-first data quality workbench for analysts, researchers, and ML engineers who need a fast, repeatable answer to: “Is this dataset trustworthy enough to use?”
Profiling runs in Python (profiler.py + quality_profiles.py). Every score, contract rule, and violation count comes from pandas/scipy on your machine.
| Layer | What it does |
|---|---|
| Profiler | Column stats, null/outlier/mixed-type detection, email/date/encoding checks |
| Sector profiles | Weighted dimension scores + pass/fail contracts for retail, healthcare, financial, survey, ML training |
| Dashboards | Overview, columns, distributions, BI analytics, fixes, drift, correlation, report |
| Fixes | Allowlisted transforms (drop_nulls, impute, strip whitespace, dedupe) with revision tracking |
- Survey / research teams — duplicate respondents, item missingness, mixed-type Likert columns
- Retail / ops — transaction keys, negative sales, duplicate rows
- ML practitioners — label presence, feature completeness, correlation views
- Anyone with CSV/Excel/Parquet — one upload, eight dashboards, exportable markdown report
- Deterministic truth — Python computes scores and rule pass/fail
- Auditable — revision history, sample failures, exportable reports
- Local-first — in-memory sessions for dev; no account required
- Bounded sessions — upload limits and row caps protect local memory use
git clone https://github.com/interfluve-wav/DataLens.git
cd DataLens
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
# API (port 8000) + Vite frontend (port 5173)
./dev.shOpen http://localhost:5173 — upload a file, pick a sector profile, explore dashboards.
source venv/bin/activate
streamlit run app.py
# → http://localhost:8501venv/bin/python3 -m pytest tests/ -vCSV, TSV, Excel (.xlsx, .xlsm), ODS, JSON, Parquet — via tabular_loader.py.
| Profile | Focus |
|---|---|
generic |
Balanced hygiene scoring |
retail |
Store/date/SKU uniqueness, non-negative sales |
healthcare |
Clinical completeness, timeliness, precision |
financial |
Timeliness, email format (KYC-oriented) |
survey |
Respondent uniqueness, high per-row item missingness, required columns |
ml_training |
Label presence, duplicates, outliers |
Each profile produces dimension scores (weighted 0–100) and contract rules (critical vs warning) with sample_failures for failed checks.
Shown on Quality Overview as a preview card labeled Coming in the next update. The control is disabled in this release; deterministic profiling and fixes are fully available.
┌─────────────────────────────────────────────────────────────┐
│ frontend/ (React + Vite + shadcn + Recharts + GSAP) │
│ Upload → 8 dashboards → fixes / report export │
└───────────────────────────┬─────────────────────────────────┘
│ /api
┌───────────────────────────▼─────────────────────────────────┐
│ api.py (FastAPI, in-memory sessions, revision tracking) │
└───────────────────────────┬─────────────────────────────────┘
│
┌───────────────────────────▼─────────────────────────────────┐
│ profiler.py tabular_loader.py quality_profiles.py │
│ hardening.py (limits, VALID_FIX_TYPES, session TTL) │
└─────────────────────────────────────────────────────────────┘
Stack: Python 3.12+, FastAPI, pandas, scipy, React 19, TypeScript, Vite
Legacy UI: Streamlit (app.py) — kept for reference, not the primary surface
| Endpoint | Purpose |
|---|---|
POST /api/upload |
Upload dataset + optional baseline + quality profile |
GET /api/session/{id} |
Full session payload (scores, rules, analytics) |
POST /api/fixes |
Apply allowlisted column fixes, bump revision |
GET /api/session/{id}/report |
Markdown quality report |
DataLens/
├── api.py # FastAPI backend
├── profiler.py # Core profiling engine
├── quality_profiles.py # Sector profiles + DQ contracts
├── tabular_loader.py # Multi-format ingest
├── hardening.py # Upload limits, sessions, fix allowlist
├── app.py # Legacy Streamlit UI
├── frontend/ # React primary UI
├── tests/ # pytest suite
├── scripts/dev.sh # Local dev orchestration
└── requirements.txt
Scores and contract results trace to pandas/scipy on real data.
venv/bin/python3 -m pytest tests/ -v- Fork the repo
- Create a feature branch (
git checkout -b feature/your-feature) - Commit with a clear message
- Push and open a Pull Request
- Run
pytestbefore submitting
MIT License — free to use, modify, and distribute.
Suhaas Chitturi — @interfluve-wav