Speech-to-Speech Summarization System

Advanced S2S summarization preserving speaker characteristics and prosodic features

Project Overview

Speech-to-Speech (S2S) Summarization System is a novel approach that addresses the limitations of traditional speech-to-text summarization methods. While conventional systems often lose critical paralinguistic features such as speaker identity and prosody, our S2S system generates spoken summaries that preserve the target speaker’s vocal characteristics, creating more nuanced and engaging outputs.

Key Features

🎤 Advanced Speech Processing Pipeline

Cascaded ASR-LLM-TTS Architecture

  • Automatic Speech Recognition (ASR): High-accuracy speech-to-text conversion preserving contextual information
  • Large Language Model (LLM): Intelligent text summarization maintaining semantic coherence
  • Speaker-Adaptive Text-to-Speech (TTS): Voice synthesis preserving original speaker characteristics

Speaker Identity Preservation

  • Zero-shot Speaker Adaptation: Maintain target speaker’s voice without additional training
  • Few-shot Voice Replication: High-fidelity voice cloning with minimal reference audio
  • Prosodic Feature Retention: Preserve natural speech patterns, intonation, and rhythm

🔍 Beyond Traditional Summarization

Paralinguistic Feature Preservation

  • Speaker Identity Maintenance: Consistent voice characteristics throughout the summary
  • Prosodic Naturalness: Natural speech flow and intonation patterns
  • Acoustic Diversity Handling: Robust performance across diverse recording conditions

Comprehensive Evaluation Framework

  • Linguistic Quality Assessment: Summary coherence and informativeness metrics
  • Speaker Similarity Measurement: Voice characteristic preservation evaluation
  • Speech Naturalness Analysis: Prosodic quality and listening experience assessment

Technical Stack

Dataset & Evaluation

  • Dataset: SLUE-TED - Standardized benchmark for speech summarization tasks

ASR Models

  • Whisper Large-v3: OpenAI’s state-of-the-art multilingual speech recognition model
  • OWSM v4 Medium: Open Whisper-style Speech Model for robust transcription
  • OWSM-CTC v4: Connectionist Temporal Classification variant for improved alignment

TTS Systems

  • VITS: Conditional Variational Autoencoder for high-quality speech synthesis
  • YourTTS: Zero-shot multi-speaker text-to-speech model
  • XTTS-v2: Cross-lingual text-to-speech with voice cloning capabilities
  • CosyVoice 2: Advanced neural vocoder for natural speech generation

Supporting Technologies

  • Language Models: Large-scale transformer-based summarization models
  • Speaker Encoding: Deep learning models for voice characteristic extraction
  • Audio Processing: Signal processing pipelines for quality enhancement

System Architecture & Demo

Project Demo Video

Complete demonstration of the Speech-to-Speech Summarization System showcasing real-time processing, speaker preservation, and summarization quality.

System Architecture

Research Contributions

Novel System Architecture

  • Integrated Pipeline: Seamless integration of ASR, LLM, and speaker-adaptive TTS components
  • Speaker Preservation: Advanced techniques for maintaining speaker identity across summarization process
  • End-to-End Processing: Complete speech-to-speech workflow without intermediate text dependencies

Evaluation Framework

  • Multi-Dimensional Assessment: Comprehensive metrics covering linguistic, acoustic, and perceptual quality
  • SLUE-TED Benchmark: Evaluation on standardized speech summarization datasets
  • Comparative Analysis: Performance comparison with traditional S2T and text-to-text baselines

Experimental Results

Left: Comprehensive performance comparison across different model configurations on SLUE-TED dataset. Right: Detailed analysis of evaluation metrics including summary quality, speaker similarity, and speech naturalness scores.