Skip to content

README.md

Latest commit

 

History

History
177 lines (139 loc) · 7.21 KB

README.md

File metadata and controls

177 lines (139 loc) · 7.21 KB
Raw

🎧 AASIST Speech-Only Audio Deepfake Detector 🛡️

A production-ready API and web platform for detecting AI-generated synthetic speech (deepfakes) in audio and video files. Built on a custom AASIST (Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention) architecture trained exclusively on human speech — no music, no shortcuts.

🌟 Overview

This project detects synthetic speech generated by modern TTS (Text-to-Speech) and VC (Voice Conversion) systems including ElevenLabs, Amazon Polly, RVC v2, XTTS-v2, FishSpeech, OpenVoice, and others. Unlike standard deepfake detectors that accidentally learn to classify "music vs speech," this model is trained strictly on speech-only data, ensuring it learns genuine neural vocoder spectral artifacts and phase discontinuities.

📊 Benchmark Metrics:

  • 🎯 In-Domain EER (Equal Error Rate): ~22.15%
  • 📈 In-Domain AUC (Area Under Curve): 0.8411 (84.11%)
  • ⚡ In-Domain Accuracy: 77.87%
  • 🧠 Architecture: Raw-waveform AASIST with SincConvFast + Residual CNN Stack + Spectro-Temporal Attention Pooling

🏗️ Model Architecture

                       🎙️ Raw Audio Waveform (16kHz, Mono)
                                      │
                                      ▼
                   🎛️ SincConv (128 Learnable Band-Pass Filters)
                                      │
                                      ▼
                           🧱 Residual CNN Stack
    ├── 🔹 Block 1: 128 ➔ 32  ➔ 32  (MaxPool1d)
    ├── 🔹 Block 2: 32  ➔ 64  ➔ 64  (MaxPool1d)
    └── 🔹 Block 3: 64  ➔ 128 ➔ 128 (MaxPool1d)
                                      │
                                      ▼
                   🧠 Spectro-Temporal Attention Pooling (Mean + Std)
                                      │
                                      ▼
                     📊 FC Layer (256 ➔ 64) ➔ SELU ➔ Dropout(0.25)
                                      │
                                      ▼
                             🎯 FC Classifier (64 ➔ 2)
                                      │
                                      ▼
                   ⚖️ Softmax Output ➔ [P(Fake), P(Real)]

💡 Why AASIST?

  • 📡 Raw Waveform Processing: Operates directly on raw audio waveforms — avoiding hand-crafted STFT/Mel-spectrogram information loss.
  • 🎛️ SincConv Filters: Learns physiologically-motivated auditory filterbanks directly during backpropagation.
  • 🎯 Attention Pooling: Captures sparse, localized artifacts (vocoder phase glitches, linear predictive encoding errors, spectral tilt).

📚 Training Data (Speech-Only)

📁 Dataset 🌐 Language 🎭 Type 👨‍🦱 Real 🤖 Fake ⚡ Voice Generators
Mohammed01/ArFake 🇸🇦 Arabic TTS/VC ✅ ✅ XTTS-v2, FishSpeech, ArTST, VITS
Arabic_Audio_Deepfake 🇸🇦 Arabic VC ✅ ✅ RVC v2 Voice Conversion
MLAAD-tiny 🌍 Multilingual TTS ✅ ✅ 64 Neural TTS Engines
garystafford/deepfake-audio 🇬🇧 English TTS ✅ ✅ ElevenLabs, Polly, Speechify, Hume AI

📌 Total Dataset: ~3,138 balanced audio samples (1,569 Real vs 1,569 Fake).

Important

Critical Design Choice: No music datasets (such as Suno AI or Udio) were used. Including generative music introduces a trivial binary "speech vs music" shortcut that completely breaks generalization to real-world voice clones.

⚙️ Training Configuration

Parameter Value / Setting
🛠️ Optimizer AdamW (lr=1e-3, weight_decay=0.01)
📈 Scheduler CosineAnnealingLR (15 Epochs, eta_min=1e-5)
🎯 Loss Function CrossEntropyLoss (Label Smoothing = 0.10)
📦 Batch Size 8
🛑 Early Stopping Patience = 4 epochs on Validation EER
✂️ Crop Duration 4.0 seconds (64,000 samples @ 16kHz)
🔊 Normalization Peak Amplitude Normalization (x / max(abs(x)) * 0.95)

🚀 API Endpoints

1️⃣ POST /api/analyze-file

Upload any local audio or video file from your device.

  • Supported Formats: .mp3, .wav, .m4a, .flac, .ogg, .aac, .opus, .wma, .mp4, .avi, .mkv, .mov, .webm.
  • Request: multipart/form-data with file field.

Response Schema (JSON):

{
  "prediction": "FAKE",
  "confidence": 0.8234,
  "real_probability": 0.1766,
  "fake_probability": 0.8234,
  "segments_count": 5,
  "threshold": 0.5,
  "model_loaded": true,
  "checkpoint": "aasist_speech_only_best.pt",
  "timeline": [
    {
      "segment": 1,
      "start_sec": 0.0,
      "end_sec": 4.0,
      "real_prob": 0.2103,
      "fake_prob": 0.7897,
      "prediction": "FAKE"
    }
  ]
}

2️⃣ POST /api/analyze-youtube

Directly scan and evaluate audio from any public YouTube URL.

  • Request: application/x-www-form-urlencoded with youtube_url field.
  • Response Schema: Same format as /api/analyze-file.

3️⃣ GET /api/health

Check API service health status, active compute device (CUDA/CPU), and loaded checkpoint.

💻 Local Setup & Installation

1. Requirements

pip install fastapi uvicorn python-multipart imageio-ffmpeg numpy scipy torch torchaudio

2. Optional: YouTube Extraction Engine

pip install -U yt-dlp

3. Project Structure

your_project/
├── 📄 app.py                     ← Complete Self-Contained FastAPI Engine
├── ⚖️ aasist_speech_only_best.pt ← Trained AASIST Weights
└── 🎨 static/
    ├── 📄 index.html             ← Cyberpunk Obsidian Web UI
    ├── 🎨 style.css              ← Glassmorphism & Animations
    └── 📜 app.js                 ← Chart.js & Gauge Controller

4. Run Server

python app.py
# or
uvicorn app:app --host 127.0.0.1 --port 8000 --reload

Open your browser at: http://127.0.0.1:8000 🌐

🔬 Key Technical Decisions

Technical Choice Rationale & Benefit
🎛️ SincConv over STFT Learnable parametric filterbanks adapt dynamically to neural vocoder phase artifacts rather than fixed FFT bins.
🚫 No Music in Training Forces the Sinc filters to inspect vocal tract acoustic properties rather than background instrumental acoustics.
📊 Peak Normalization Preserves natural waveform amplitude dynamics and energy envelopes across pause boundaries.
✂️ 4-Sec Sliding Window Provides optimal temporal windowing for neural vocoder artifact detection with 2-second stride overlap.

⚠️ Limitations & Considerations

  • 🗣️ Language Distribution: Dataset contains ~40% Arabic, ~35% English, and ~25% Multilingual. Performance may vary on underrepresented dialects.
  • ⏱️ Short Audio Clips: Clips under 4 seconds are zero-padded. Utterances shorter than 1 second may provide reduced confidence.
  • 📈 Research EER: Equal Error Rate is ~22.15% on non-trivial speech deepfakes. Production commercial deployments should aim for ensemble models for sub-10% EER.