The Speech Prosody SIG Lecture Series, a new initiative of the Speech Prosody SIG Officers, aims to (1) offer to the Speech Prosody community a well-covered view of themes and methods in speech prosody; (2) introduce new perspectives and foster debate; (3) stimulate collaborations among speech prosody researchers, including by making known to the community the existence of public repositories with data, corpora, joint projects asking for collaboration and other resources that can be freely shared. Lectures will be presented live in YouTube, with Q&A, handled through the YouTube's chat feature.

Upcoming Talks

Speech prosody and social meaning

Robert Xu, Harvard University

Wednesday, Dec. 18th

Viewing Links: at Youtube, at Bilibili

Abstract: This talk explores how third-wave sociolinguistic theories can deepen our understanding of the structure and function of speech prosody, particularly in social interactions. Key concepts such as indexicality, style, stancetaking, and enregisterment will be introduced and exemplified through an ecologically-minded study of prominent social types in Beijing. I will demonstrate how prosodic features—such as pitch variation, voice quality, and timing—serve as semiotic resources for constructing these social types within a socio-cultural landscape. These prosodic elements not only interact with each other but also conspire with the body and conversation structures to convey dialogical social meanings. These meanings enable the constructed social types to mediate broader social relationships, structures, and transformations.

Plan:

TBD. Sasha Calhoun, TBD.

Archived Lectures

Tackling prosodic phenomena at their roots

Professor Yi Xu, University College London. October 25, 2023.

Abstract: Rather than being a coherent whole, speech prosody consists of highly diverse phenomena that are best understood in terms of their communicative functions, together with specific mechanisms of articulatory encoding and perceptual decoding. The understanding of these root causes is therefore key to further advances in prosody research.

archived talk at YouTube and at bilibili

Segmental Articulations and Prosody

Malin Svensson Lundmark, Lund University, November 23rd, 2023

Abstract: This lecture will be on an aspect of the articulatory-acoustics relationship that is rarely addressed but which is both stable and robust across, e.g., places of articulation, tonal context and prosodic levels. It’s about the acceleration and deceleration of articulatory movements and how they coincide with acoustic segment boundaries.

archived talk at Youtube and at biliblili

December 14, 2023.

How to handle variability in the study of intonation

Amalia Arvaniti, Radboud University, Netherlands.

Abstract: This talk will give an overview of the issue of variability in intonation and present methodological approaches that render variability easier to handle. These methodologies are presented by means of a case study, the English pitch accents H* and L+H*, which are treated as distinct phonological entities in some accounts but as endpoints of a continuum in others. The research that will be presented sheds light on the reasons for the disagreement between analyses and the discrepancies between analyses and empirical evidence, by examining both production data from British English unscripted speech and perceptual data, which also link the processing of the two accents to the participants’ levels of empathy, musicality, and autistic-like traits.

archived talk at Youtube and at bilibili

The speech synthesis phoneticians need is both realistic and controllable: A survey and a roadmap towards modern synthesis tools for phonetics.

Zofia Malisz, KTH Royal Institute of Technology.

April 17th, 2024. archived talk, also at bilibili.

ABSTRACT
In the last decade, data and machine learning-driven methods to speech synthesis have greatly improved its quality. So much so, that the realism achievable by current neural synthesisers can rival natural speech. However, modern neural synthesis methods have not yet transferred as tools for experimentation in the speech and language sciences. This is because modern systems still lack the ability to manipulate low-level acoustic characteristics of the signal such as e.g. formant frequencies.
In this talk, I survey recent advances in speech synthesis and discuss their potential as experimental tools for phonetic research. I argue that speech scientists and speech engineers would benefit from working more with each other again: in particular, in the pursuit of prosodic and acoustic parameter control in neural speech synthesis. I showcase several approaches to fine synthesis control that I have implemented with colleagues: the WavebenderGAN and a system that mimicks the source-filter model of speech production. These systems allow to manipulate formant frequencies and other acoustic parameters with the same (or better) accuracy as e.g. Praat but with a far superior signal quality.
Finally, I discuss ways to improve synthesis evaluation paradigms, so that not only industry but also speech science experimentation benchmarks are met. My hope is to inspire more students and researchers to take up these research challenges and explore the potential of working at the intersection of the speech technology and speech science.

Outline: 1. I discuss briefly the history of advancements in speech synthesis starting in the formant synthesis era and explain where the improvements came from. 2. I show experiments that I have done that prove modern synthesis is processed not differently than natural speech by humans in a lexical decision task as evidence that the realism (“naturalness”) goal has been largely achieved. 3. I explain how realism came at the expense of controllability. I show how controllability is an indispensable feature for speech synthesis to be adopted in phonetic experimentation. I survey the current state of research on controllability in speech engineering - concentrating on prosodic and formant control. 4. I propose how we can fix this by explaining the work I have done with colleagues on several systems that feature both realism and control. 5. I sketch a roadmap to improve synthesis tools for phonetics - by placing focus on benchmarking systems according to scientific criteria.

Predictive Modelling of Turn-taking in Spoken Conversation

Gabriel Skantze, KTH

May 15, 2024. archived talk at Youtube, and at Bilibili

Abstract: Conversational interfaces, in the form of voice assistants, smart speakers, and social robots are becoming ubiquitous. This development is partly fuelled by the recent developments in large language models. While this progress is very exciting, human-machine conversation is currently limited in many ways. In this talk, I will specifically address the modelling of conversational turn-taking. As current systems lack the sophisticated coordination mechanisms found in human-human interaction, they are often plagued by interruptions or sluggish responses. I will present our recent work on predictive modelling of turn-taking, which allows the system to not only react to turn-taking cues, but also predict upcoming turn-taking events and produce relevant cues to facilitate real-time coordination of spoken interaction. Through analysis of the model, we also learn about which cues are relevant to turn-taking, including prosody and filled pauses.

Plan:

  1. Introduction to conversational systems and human-robot interaction
  2. Why turn-taking is problematic in current systems
  3. Voice Activity Projection: A predictive, data-driven model of turn-taking
  4. Analysis of the model (prosody and filled pauses)
  5. Towards better turn-taking in conversational systems

Syntagmatic Prominence Relations in Prosodic Focus Marking

Lecturer: Simon Roessig (University of York, UK)

Sept 24th, 2024. Archived talk at YouTube, and at Bilibil.

Abstract: This talk is about the role of prenuclear prominences and their relation to nuclear accents in German and English. The production results (German) that I will present show that the realization of the prenuclear domain depends on whether it is focal or prefocal. The prenuclear noun is characterized by larger F0 excursions, higher F0 maxima, and longer durations when it is in broad focus than when it precedes a narrow focus. Furthermore, the realization of the prenuclear domain depends on the following focus type: The prenuclear noun is produced with smaller F0 excursions, lower F0 maxima and shorter durations before a corrective focus than before a non-corrective narrow focus. The findings suggest that the phonetic manifestation of information structure is distributed over larger prosodic domains with an inverse relationship in the syntagmatic dimension. In addition, the study contributes further evidence that continuous phonetic detail is used to encode information structural categories. An important question that arises from the production data is whether this phonetic detail can be used by listeners in perception. I will present first results from a series of perception experiments (German and English) to investigate this question.

Plan: 1. I will begin by outlining what we know about focus prosody in the nuclear and prenuclear domains. 2. I will then present findings from a production study that examines the prosody of the prenuclear domain in different types of focus. 3. These results show that there are interesting strength relations between prenuclear and nuclear prosody in the encoding of focus types. 4. I will present preliminary findings from perception experiments investigating the question whether listeners use prenuclear prominence modulations in identifying focus types. 5. Finally, I will conclude with a discussion of the results and future directions.

Sam Tilsen, Department of Linguistics, Cornell University

On the intermittency of speech and the lack of compelling evidence for phrasal rhythm or hierarchical prosodic phrase structure

Oct. 23rd, 2024

archived at YouTube and at Bilibili

Abstract: On the timescale of prosodic phrases, temporal patterns in conversational speech are not very regular. To the contrary, spurts of fluent speech tend to be highly irregular and intermittent. Hesitations and pauses are common. What is the mechanism behind this pattern? In this talk I consider and reject two possible explanations. First, I examine the possibility that a hierarchical organization of relatively long-timescale prosodic units might explain intermittency. Several predictions of hierarchical prosodic structure accounts are examined in an analysis of the Switchboard NXT corpus, but the empirical patterns are not very consistent with those predictions. Moreover, I argue that even laboratory studies that purport to find evidence for hierarchical phrase structure suffer from flawed argumentation. For these reasons, a hierarchical structure-based account of intermittency is suspect. Second, I examine the possibility that there is a phrase-timescale oscillator that governs phrase initiation. I critique recent studies that have argued that such an oscillator is involved in speech production. Through model simulations I show that in order to adequately capture empirical timing patterns, such an oscillator would need an overly powerful ability to change frequency from cycle to cycle. On top of this, the neurophysiological basis for a role of oscillation in governing phrasal timing is called into question. Instead of structural or oscillation-based mechanisms being responsible for phrasal timing, I argue that intermittency arises due to mechanisms responsible for the organization of syntactic and conceptual systems. I present a model in which phrase initiation is contingent on the achievement of a coherent state among those systems, and show how stochastic influences can generate hesitative phenomena that may be the basis for the intermittency of speech.

Plan:

  1. Speech activity is intermittent on the timescale of prosodic phrases.
  2. Hierarchical prosodic phrase structure does not explain intermittency.
  3. An oscillatory production mechanism does not explain intermittency.
  4. A model that accounts for intermittency is presented, in which syntactic and conceptual systems must achieve a coherent state before production is initiated.
  5. Implications of the model are discussed.
Controlling and Probing Generative End-to-end Models: New Opportunities for Research on Prosody

Gerard Bailly, CNRS, Grenoble-Alps University, November 27, 2024.

Archived talk: at YouTube, and at bilibili

Abstract: During decades, the interplay of phonetic forms, phonological structures and communicative functions was mainly questioned via meticulous analysis of acoustic or multimodal performance of speakers and listeners... with the ambition of providing technology with principles, controls and constraints emerging from our human creativity. This golden age of human intelligence is now largely defeated by model-free generative AI, in particular text-to-speech systems that provide signals or videos of speaking faces often misconfused with natural data!

In this presentation, I will argue for a positive attitude: consider these high quality end2end models as a proxy to capture lawful data variability and develop new tools to explore internal representations (so-called latent spaces) built by these successful models. I will further detail two works started with my colleagues Olivier Perrotin and Martin Lenglet: (1) exploration and control of phonetic and phonological embeddings via causal regression (Lenglet et al, Interspeech 2022 & submitted to CSL); (2) exploration and fine control of audiovisual attitudes via verbal tags (Bailly et al, LREC/COLING 2024).

Plan: 1. Brief review of latent space analysis and causal regression 2. Brief review of end-to-end text-to-speech technology 3. Exploration and control of phonetic and phonological embeddings via causal regression 4. Exploration and fine control of audiovisual emotion via verbal tags 5. Take-home message: Generative models as a proxy to data mining

Lecture Series Organizer: Plinio A. Barbosa, University of Campinas, Brazil

Of related interest: Ward and Levow’s Prosody Tutorial Video Series