E-commerce A/B Test Analysis Platform

A comprehensive, production-ready A/B testing framework that combines statistical rigor with business impact analysis. Built to help data teams avoid costly mistakes and make confident, data-driven decisions.

Try the Live App Here!

Live Demo: https://ecommerce-ab-testing-platform.streamlit.app/

Screenshots

Main Dashboard Overview

Get instant insights with key metrics, conversion rates, and automated recommendations.

---

Configuration Options

Flexible settings for different experiments and confidence levels.

---

Deep Statistical Analysis

Comprehensive hypothesis testing, confidence intervals, and power analysis.

---

Bayesian Analysis

The recommendation by bayesian

---

Posterior Distributions

The overlapping curves

---

Lift Distributions

Distribution of possible lift values based on my analysis

---

Bayesian comparison

The comparison table- Bayesian vs Frequentist

---

Problem Statement

Companies waste millions on incorrect A/B test conclusions due to:

• Peeking problems - Stopping tests early based on promising results
• Multiple comparison errors - Testing too many segments without correction
• Simpson's Paradox - Missing confounding variables in segment analysis
• Underpowered tests - Insufficient sample sizes leading to false negatives
• Novelty effects - Mistaking temporary engagement spikes for lasting change

This platform addresses these challenges with built-in statistical safeguards and automated recommendations.

Key Features

Multi-Experiment Analysis

• Analyze 4 realistic experiment scenarios simultaneously:
  • Homepage redesign (UI/UX changes)
  • Checkout flow optimization
  • Pricing strategy testing
  • Recommendation algorithm comparison

Statistical Rigor

• Proper hypothesis testing with Z-tests for proportions
• Confidence intervals with visualization
• Sample size calculation and power analysis
• Multiple testing corrections (Bonferroni)
• Sequential testing detection (peeking problem alerts)

Segment Analysis

• Automatic user segmentation (New/Returning/VIP)
• Device-level analysis (Mobile/Desktop/Tablet)
• Simpson's Paradox detection and warnings

Business Impact

• Revenue per user analysis
• Annual revenue projections
• ROI calculations
• Statistical tests for revenue differences

"What Could Go Wrong" Module

Automated detection of:

• Peeking bias
• Multiple comparison problems
• Simpson's Paradox
• Novelty effects
• Underpowered tests

Bayesian Analysis (Advanced)

• Direct probability statements: "95% probability Treatment is better"
• Posterior distributions with visual overlap analysis
• Credible intervals (90% and 95%)
• Risk assessment: Probability and magnitude of potential loss
• Prior selection: Incorporate domain knowledge
• Safe early stopping: No peeking problem
• Side-by-side comparison with Frequentist results

Why Bayesian?

• Answers the question stakeholders actually ask: "What's the chance B is better?"
• More intuitive for business decision-making
• Allows continuous monitoring without statistical penalties
• Incorporates prior knowledge from past experiments

Live Demo

Try it yourself:

git clone https://github.com/YOUR_USERNAME/ecommerce-ab-testing-platform.git
cd ecommerce-ab-testing-platform
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install -r requirements.txt
python src/data_generator.py
streamlit run src/app.py

Then open http://localhost:8501 in your browser.

Tech Stack

• Python 3.10+ - Core programming language
• Streamlit - Interactive web framework
• Pandas & NumPy - Data manipulation
• SciPy & Statsmodels - Statistical analysis
• Plotly - Interactive visualizations

Project Structure

ecommerce-ab-testing/
├── src/
│   ├── app.py                 # Main Streamlit dashboard
│   ├── data_generator.py      # Synthetic data generation
│   ├── statistical_tests.py   # Core statistical functions
│   └── visualizations.py      # Plotly chart functions
├── data/
│   └── ab_test_data.csv       # Generated sample data
├── requirements.txt           # Python dependencies
└── README.md                  # This file

Statistical Methods

Hypothesis Testing

• Z-test for proportions to compare conversion rates
• Two-tailed tests for detecting both positive and negative effects
• 95% confidence intervals for effect size estimation

Power Analysis

• Pre-test sample size calculations using Cohen's h
• Post-test power analysis to validate results
• MDE (Minimum Detectable Effect) calculations

Multiple Comparison Correction

• Bonferroni correction for segment analysis
• Family-wise error rate control

What I Learned

This project demonstrates expertise in:

• Statistical inference - Hypothesis testing, confidence intervals, power analysis
• Experimental design - A/B testing best practices, avoiding common pitfalls
• Data visualization - Creating intuitive, interactive dashboards
• Software engineering - Modular code, documentation, version control
• Business analytics - Translating statistical results into actionable recommendations

Future Enhancements

• Bayesian A/B testing with prior distributions
• Multi-armed bandit algorithms
• Sequential analysis (always-valid p-values)
• Heterogeneous treatment effects (CATE estimation)
• Database integration for real-time data
• Automated email reports
• Docker containerization

Sample Data

The platform includes realistic synthetic data with:

• 40,000 total users across 4 experiments
• Realistic conversion rates (10-25%)
• Revenue simulation with gamma distribution
• User segments (New/Returning/VIP)
• Device categories (Mobile/Desktop/Tablet)
• Time-series data over 10 days

License

MIT License - feel free to use this project for learning or professional purposes.

Author

SAKSHI

• Email: sakshchavan30@gmail.com

---

**Note:**The data is synthetically generated for educational purposes.