⚓ Maritime AI Commander

Production-Grade Real-Time Voice Interface for Naval & Maritime Operations

The Maritime AI Commander is a next-generation, multi-modal dashboard designed for ship crews, cargo operators, and naval command. It delivers low-latency, bi-directional voice interaction powered by the Google Gemini Live API (WebSocket) and autonomously generates visual assets — diagrams, charts, images, and video — in response to conversational context.

---

🚀 Key Features

🧠 Intelligent Voice Core

• Real-time Voice Interaction: Powered by gemini-2.5-flash-native-audio-preview-09-2025 via the Gemini Live WebSocket API. Supports natural interruption, voice pacing, and emotive speech with the Kore voice.
• Live RAG via Function Calling: Gemini itself triggers a search_knowledge_base tool call mid-conversation, which queries an external Python RAG API in real-time and feeds ranked, cited results back into the audio response.
• Inline Document Mode: An alternative RAG mode injects uploaded documents directly into the system prompt context (up to 25,000 characters).
• Background Analyst Stream: After each conversational turn, a stateless gemini-2.5-flash call analyzes the exchange to detect out-of-domain queries, missing knowledge gaps, and whether a visual asset should be generated.
• Auto-Reconnect: Persistent sessions with automatic reconnection on connection loss (after ≥10s uptime).

🎨 Multi-Modal Asset Generation

The system autonomously determines when and what to visualize based on the conversation:

• Dynamic Diagrams: Mermaid.js flowcharts, sequences, and checklists for SOPs and operational procedures.
• Live Charting: Recharts-powered bar/line charts for engine diagnostics, fuel metrics, or cargo data.
• Reference Images: Generated via gemini-2.5-flash-image for equipment identification or scenario visualization.
• Cinematic Video: veo-3.1-fast-generate-preview for training simulations (async polling with retry).
• Asset Approval Flow: System-suggested assets are held in a proposal queue for user approval before rendering. Explicit user requests skip the queue and generate immediately.

🖥️ Immersive UI

• AI Core Visualizer: A real-time Canvas 2D particle orb system with orbiting particles, scanning rings, waveform rings, and data stream lines — all responding live to audio levels and connection state (idle / listening / speaking / processing).
• Night Ops Mode: A full red-palette HUD mode (persisted in localStorage) for low-light bridge environments. Toggleable at runtime.
• Glassmorphism Design: Dark-mode UI with scanline and grain overlays, designed for control room conditions.
• Admin Control Plane: Manage the Knowledge Base (upload/delete .txt documents), monitor out-of-domain alerts, and toggle RAG mode live without restarting the session.

---

🛠️ Technology Stack

Category	Technology
Framework	React 19, TypeScript, Vite 7
Styling	Tailwind CSS 3
AI SDK	@google/genai v1.33+
Live Voice	gemini-2.5-flash-native-audio-preview-09-2025
Text / Analysis	gemini-2.5-flash
Image Generation	gemini-2.5-flash-image
Video Generation	veo-3.1-fast-generate-preview
Diagrams	mermaid v11
Charts	recharts v3
Audio	Native Web Audio API with custom PCM pipeline

---

⚙️ Architecture

1. Hybrid Audio Pipeline

To support cross-platform compatibility (iOS / macOS enforce 44.1kHz or 48kHz hardware rates):

• Input: Captures mic audio at the system's native hardware sample rate (no forced rate).
• Downsampling: A custom downsampleTo16k Linear PCM resampler converts audio to 16kHz in real-time before sending to the Gemini API.
• Output: Receives raw PCM 24kHz from Gemini, aligns bytes to Int16Array, and schedules playback via the Web Audio API with a 25ms safety margin to prevent buffer starvation clicks on Mac/Safari.

2. Dual-Stream Architecture

The app runs two concurrent logic streams:

1. Voice Stream — A persistent WebSocket session (MaritimeLiveClient) handling microphone input, audio output, and real-time transcription with interrupt support.
2. Analyst Stream — A stateless gemini-2.5-flash text call that fires after each turn completes. It classifies the interaction (maritime domain check, missing knowledge, visual asset need) and triggers side effects without blocking the voice session.

3. Function-Calling RAG

When RAG mode is active, the Live session is configured with a search_knowledge_base function declaration. Gemini decides when to invoke it, passing a semantically optimized query. The client intercepts the toolCall message, queries the Python RAG backend (POST /search), ranks results by relevance score, formats a structured context response with synthesis instructions, and returns it to the session via sendToolResponse.

4. Asset Management

useAssetManager maintains a typed asset registry and a generation queue. Assets are generated with exponential-backoff retry (retryWithBackoff). Mermaid code is sanitized (markdown blocks stripped, valid start tokens enforced) before rendering. Chart JSON is schema-validated. All assets are addressable by ID in the panel display.

5. Session Lifecycle

useMaritimeSession manages the full lifecycle:

• Creates a fresh MaritimeLiveClient per session with all event handlers.
• Persists maritime_autoconnect in localStorage to survive page refreshes.
• Schedules automatic reconnection (3s delay) if a session drops after running for >10s.
• Cleanly tears down audio nodes, media streams, and WebSocket connections on disconnect.

---

📁 Project Structure

.
├── frontend/                  # React Vite Application
│   ├── src/
│   │   ├── components/        # UI Components (AiCoreVisualizer, AdminPanel, etc.)
│   │   ├── hooks/             # Custom React Hooks
│   │   ├── services/          # API, WebSocket, and RAG clients
│   │   ├── types.ts           # Shared TypeScript interfaces
│   │   └── App.tsx            # Main Application Entry
│   └── package.json
│
├── backend/
│   └── rag/                   # Python FastAPI RAG Server
│       ├── ingest.py          # Document embedding and indexing
│       ├── rag_server.py      # Main FastAPI server
│       ├── retrieval/         # Vector search logic
│       └── requirements.txt
│
└── package.json               # Root monorepo workspace and scripts

---

📦 Installation & Setup

Prerequisites

• Node.js 18+
• A Google AI Studio API Key with access to Gemini 2.5 Flash, Gemini Live, and Veo APIs.
• (Optional) A running Python RAG API server at http://localhost:8000 for the function-calling knowledge base.

Steps

1. Clone the repository

   git clone https://github.com/your-username/maritime-ai-commander.git
   cd maritime-ai-commander

2. Install dependencies and setup environments

   npm install
   npm run setup

   Note: npm run setup automatically initializes both the React frontend and sets up a Python virtual environment for the backend.

3. Configure environment Create a .env file in the frontend directory:

   VITE_API_KEY=your_gemini_api_key_here
   VITE_RAG_API_URL=http://localhost:8000

   VITE_RAG_API_URL defaults to http://localhost:8000 if not set. The app operates normally without the RAG server — searches will return empty and the AI will fall back to general maritime standards.

4. Run the application Launch both the React development server and the FastAPI backend concurrently:

   npm run dev:all

🎮 Usage Guide

Voice Dashboard

1. Connect: Click the Microphone button in the footer to initialize the neural connection.
2. Speak: Ask anything maritime-related:
   • "What is the procedure for dropping anchor?"
   • "Show me a checklist for fire emergency response."
   • "Plot a chart of fuel consumption over the last 4 hours."
   • "Generate a collision avoidance scenario video."
3. Observe: The AI Core Visualizer reacts in real-time. If a visual was requested, it appears in the Asset Panel overlay.
4. Interrupt: Speak at any time during a response to interrupt — the session handles it gracefully.

RAG Mode

• Toggle RAG Mode in the sidebar to enable function-calling knowledge base search.
• The session restarts automatically with the new configuration.
• With the Python RAG server running, the AI will cite specific documents in its responses.

Admin Panel

1. Click the Settings (⚙) icon in the header.
2. Knowledge Base: Upload .txt files (e.g., ShipManual.txt, CargoManifest.txt) to feed ship-specific knowledge.
3. Alerts: View queries flagged as out-of-maritime-domain or missing knowledge — useful for identifying knowledge base gaps.

Night Ops Mode

• Click the moon icon in the header to switch to a full red-palette HUD mode for low-light bridge environments.
• Mode is persisted across sessions via localStorage.

---

🔧 Troubleshooting

No audio / visualizer not responding?

• Ensure the browser has microphone permission.
• If on macOS/iOS, the system auto-handles sample rate conversion — ensure no other app has exclusive control of the audio input.
• Check browser console for [MaritimeLiveClient] state logs.

"Billing Quota Exceeded"

• The project uses gemini-2.5-flash, gemini-2.5-flash-image, and veo-3.1. Ensure your Google Cloud project has billing enabled and sufficient Live API and Veo quotas.

Connection drops / reconnecting loop?

• The Live API requires a stable internet connection (WebSocket wss://). Firewalls blocking wss:// will cause failures.
• If a session runs for less than 10 seconds before dropping, auto-reconnect is suppressed to prevent boot loops.

RAG search returning no results?

• Verify the Python RAG server is running at the URL in VITE_RAG_API_URL.
• The RAG client has a 5s timeout; slow server responses will be treated as empty results.
• Upload documents via the Admin Panel if the knowledge base is empty.

Video generation is slow?

• veo-3.1-fast-generate-preview can take 30–90 seconds. The app polls the operation every 5 seconds with retry logic. The asset will appear in the panel once ready.

---

📜 License

MIT — Free for academic and commercial use.

---

Built with the Google Gemini API · React