Robust and Data-Efficient End-to-End Speech Translation: Segmentation, Cross-Modal Alignment, and Prosody-Aware Evaluation

End-to-end (E2E) Speech Translation (ST) offers a streamlined alternative to traditional cascaded systems, promising lower latency and reduced error propagation. However, its adoption in real-world scenarios is currently hindered by three critical bottlenecks: input processing challenges (mismatch between static data and continuous audio), training data scarcity, and evaluation limitations regarding paralinguistic information. This dissertation addresses these limitations through a cohesive set of methodological and architectural innovations. First, we tackle the segmentation and input processing challenge. We introduce Supervised Hybrid Audio Segmentation (SHAS), a method that effectively bridges the gap between manual training segmentation and automatic inference segmentation, allowing E2E models to process continuous audio streams with minimal performance loss. Building on this, we propose SegAugment, a data augmentation strategy that utilizes training data re-segmentation to maximize the utility of existing datasets, significantly improving performance in low-resource settings. Second, we tackle data scarcity by developing methods for zero-shot speech translation through cross-modal alignment. We introduce ZeroSwot, which leverages Optimal Transport to align speech encoders with massively multilingual machine translation models at the subword level, enabling translation without paired ST data. We further refine this approach with charSONAR, demonstrating that character-level modeling significantly improves cross-lingual and cross-modal transfer, achieving state-of-the-art results, despite being a zero-shot method. Finally, we investigate the semantic richness of E2E translations. We present a focused evaluation on prosody, revealing that while current E2E architectures possess the internal capacity to represent paralinguistic features like intonation and stress, they often fail to manifest these in the final translation. Collectively, these contributions advance the state of the art by creating E2E ST systems that are more robust to unsegmented inputs and more data-efficient, while also providing insights into the limitations of current E2E systems in preserving the nuances of spoken communication.

PhD Thesis, Universitat Politècnica de Catalunya