Human-1 by Josh Talks: A Full-Duplex Conversational Modeling Framework in Hindi using Real-World Conversations
cs.CL, cs.AI
Submitted: 2026-04-25
Updated: 2026-09-24
Comments: Preprint. Submitted to ICASSP 2027
License: http://creativecommons.org/licenses/by/4.0/
The gist: Full-duplex spoken dialogue systems can model natural conversational behaviours such as interruptions, overlaps, and backchannels, yet such systems remain largely unexplored for Indian languages.
Terminology
Abstract
Full-duplex spoken dialogue systems can model natural conversational behaviours such as interruptions, overlaps, and backchannels, yet such systems remain largely unexplored for Indian languages. We present the first open, reproducible full-duplex spoken dialogue system for Hindi by adapting Moshi, a state-of-the-art duplex speech architecture, using a custom Hindi tokeniser and training on 26,000 hours of real spontaneous conversations collected from 14,695 speakers with separate speaker channels, enabling direct learning of turn-taking and overlap patterns from natural interactions. To support Hindi text generation, we replace the original English tokeniser and reinitialise text-vocabulary-dependent parameters while retaining the pre-trained audio components. We propose a two-stage training recipe -- large-scale pre-training followed by fine-tuning on 1,000 hours of conversational data. Evaluation through the prompted dialogue continuation paradigm with both automatic metrics and human judgments demonstrates that the resulting model generates natural and meaningful full-duplex conversational behaviour in Hindi. This work serves as a first step toward real-time duplex spoken dialogue systems for Hindi and other Indian languages.
Sources
- Language Model Can Listen While Speaking
- WhisperX: Time-Accurate Speech Transcription of Long-Form Audio
- Moshi: a speech-text foundation model for real-time dialogue
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
- SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation
- AudioPaLM: A Large Language Model That Can Speak and Listen
- Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering