Crystal Vibe Coders

Mumbai Kahaani - AI-Powered Interactive Storytelling Platform
Project Description
Mumbai Kahaani transforms Mumbai’s map into a living, breathing storyteller that brings the city’s rich history to life through cutting-edge AI voice technology. The platform addresses the critical problem of urban history dying in the digital age—while GPS coordinates abound, the stories, memories, and cultural heritage that make cities truly alive are being lost.
User Problem Solved: Traditional audio guides are static, one-size-fits-all, and can’t answer questions. Mumbai Kahaani creates dynamic, personalized stories for ANY location in Mumbai with real-time research and interactive voice conversations.
Core Flow: Users click anywhere on Mumbai’s interactive map → AI gathers location data (POIs, area info) → Perplexity AI conducts real-time web research → Claude Sonnet 4.5 crafts engaging narratives → ElevenLabs brings stories to life with natural voice synthesis → Users can listen passively or interrupt for live Q&A conversations.
Key Demos:
Podcast Mode: 2-3 minute narrated stories with visual audio waves
Conversation Mode: Interrupt-friendly AI agent that tells stories continuously but pauses to answer questions
Customization: Choose narrative lens (dark/bright/balanced), time periods (colonial to contemporary), languages (English/Hindi/Marathi/Hinglish), and voice styles
End-to-End Execution: The app runs as a full-stack web application with concurrent frontend (Vite/React) and backend (Express) servers. Users authenticate via Clerk, interact with an interactive Leaflet map, customize story preferences, then experience either pre-generated TTS narration or live conversational AI. All data flows through a pipeline: location selection → parallel API calls (Overpass/Nominatim/Perplexity) → Claude script generation → ElevenLabs voice synthesis → WebRTC real-time delivery.
Judging Criteria Alignment
Working Prototype
✅ Fully functional end-to-end application deployed on Render with production-grade architecture
✅ Dual interaction modes: Podcast narration with TTS audio + live conversational AI agent
✅ Real-time data pipeline: Location → Research → Script → Voice → User interaction
✅ Error handling and fallbacks for API outages
✅ Responsive UI with smooth animations and mobile support
Technical Complexity & Integration
✅ Multi-API orchestration: 5+ APIs integrated (Overpass, Nominatim, Perplexity, Claude, ElevenLabs) with parallel execution and error recovery
✅ Advanced AI stack: Latest Claude Sonnet 4.5 (Jan 2025) for creative writing, Perplexity AI for real-time research, ElevenLabs Conversational AI for interrupt-friendly voice agents
✅ Real-time voice technology: WebRTC integration for bidirectional audio with speech interruption detection
✅ Multilingual AI: Hinglish code-switching with Devanagari script support
✅ Production deployment: One-click Render deployment with environment management
Innovation & Creativity
✅ First AI storytelling map for Mumbai with dynamic content generation
✅ Interrupt-friendly conversations: Agent stops mid-sentence to answer questions then resumes naturally
✅ Bilingual AI narratives: Natural Hinglish mixing Devanagari and English
✅ Contextual intelligence: Claude understands POI relationships and user preferences
✅ Fact-based storytelling: Perplexity ensures all content is verified with sources
Real-World Impact
✅ Cultural preservation: Makes Mumbai’s history accessible and engaging for locals and tourists
✅ Educational value: Teaches history through immersive storytelling rather than textbooks
✅ Accessibility: Voice-first interface helps visually impaired users experience heritage
✅ Tourism enhancement: Creates personalized, interactive audio guides for any location
✅ Community building: Platform for sharing local stories and historical knowledge
Theme Alignment
✅ AI Agent theme: Core innovation is conversational AI that acts as a knowledgeable Mumbai storyteller
✅ Voice integration: Primary interaction is through natural voice conversations
✅ Tool use: Integrates multiple AI tools (Claude, Perplexity, ElevenLabs) in sophisticated pipeline
✅ Browser-based: WebRTC voice chat runs entirely in browser without native apps
✅ Practical utility: Solves real problem of inaccessible urban history through AI assistance
Technologies & Setup
Frontend Technologies
React 18 with TypeScript for type-safe component architecture
Vite for lightning-fast development and optimized production builds
Tailwind CSS v4 for utility-first styling and responsive design
Leaflet + React-Leaflet for interactive Mumbai map with OpenStreetMap tiles
Framer Motion for smooth animations and transitions
@11labs/react for official ElevenLabs Conversational AI SDK integration
Backend Technologies
Node.js + Express for REST API server with CORS handling
Environment-aware configuration (dev/prod modes)
Concurrent server execution (frontend + backend together)
AI & ML Services
Claude Sonnet 4.5 (Anthropic) - Latest frontier model for creative script generation with 200K context window
Perplexity AI - Real-time web research with Sonar models for factual accuracy
ElevenLabs TTS - Multilingual voice synthesis with Adam voice (deep Indian accent)
ElevenLabs Conversational AI - Real-time voice agent with interrupt detection and context retention
Geospatial & Data APIs
Overpass API - OpenStreetMap data for Points of Interest within 2km radius
Nominatim (OSM) - Reverse geocoding for location names and neighborhoods
OpenStreetMap - Base map tiles and geographic data
Authentication & Deployment
Clerk - Secure user authentication with React integration
Render - Web service hosting with auto-deployment from GitHub
Git & GitHub - Version control and CI/CD pipeline
Voice/Browser/Tool Integrations
WebRTC - Real-time bidirectional audio for conversation mode
Speech interruption detection - Agent pauses when user speaks
Multilingual voice synthesis - Hindi, English, Marathi, Hinglish support
Context-aware conversations - Agent remembers location data and conversation history
Dynamic prompt engineering - AI adapts based on user preferences and research data
Setup & Demo Steps
Clone & Install:
git clone https://github.com/Arzaan-k/Map-narrator-Mumbai-Kahani-Eleven-Labs.gitcd Map-narrator-Mumbai-Kahani-Eleven-Labsnpm install
Configure Environment:
cp .env.example .env.local# Add API keys:# ANTHROPIC_API_KEY=sk-ant-xxxxx# PERPLEXITY_API_KEY=pplx-xxxxx # VITE_ELEVENLABS_API_KEY=sk_xxxxx# VITE_ELEVENLABS_AGENT_ID=agent_xxxxx# VITE_CLERK_PUBLISHABLE_KEY=pk_test_xxxxx (optional)
Get API Keys:
Anthropic: console.anthropic.com (Claude API access)
Perplexity: perplexity.ai/settings/api (free tier available)
ElevenLabs: elevenlabs.io (TTS free tier + Agent $11/month)
Clerk: dashboard.clerk.com (optional for auth)
Run Development:
npm run dev# Frontend: http://localhost:5173# Backend: http://localhost:3001
Demo Flow:
Open app → Sign in with Clerk
Click any location on Mumbai map (e.g., Gateway of India)
Customize: Choose “Balanced” lens, “All Eras”, Hinglish language, “Friendly” voice
Select “Conversation Mode”
Agent starts talking → Try interrupting with “Tell me more about the British Raj”
Switch to “Podcast Mode” → Listen to pre-generated narration
Production Deployment:
Push to GitHub → Create Render web service
Add environment variables in Render dashboard
Deploy automatically → Get live URL
Live Demo: Deployed at [your-render-url.onrender.com] with full functionality for judges to test end-to-end user flows, voice interactions, and AI responsiveness. The platform demonstrates sophisticated AI agent behavior with natural conversation capabilities and real-world utility for cultural preservation.