Decoding Emotion and Personality from Speech
If you want to attend in person or are not at Queen's, please register to attend.
Register to Attend →
When you speak, you reveal more than your words. The rhythm, tone, and texture of your voice encode psychological traits — your emotional state, your personality, even your behavioral tendencies.
In this talk, Dr. Dhamyal discusses building computational systems to decode two types of psychological traits from speech: emotion and personality. She introduces the use of CLAP, a contrastive audio-language model that uses natural-language descriptions of speech (rather than discrete category labels) to supervise emotion learning. This architecture allows the model to leverage the richness of natural language in addition to the speech signal. She then covers SELM, which extends this approach to address CLAP's limitations.
For personality, she revisits the classical OCEAN model — a 60-year-old psychological framework for human personality — by re-analyzing its basis labels using modern LLM-derived word representations, revealing interesting patterns in the data as looked at through the lens of large language models. She will also speak about her work utilizing contrastive audio-language models with acoustic prompts that achieve state-of-the-art zero-shot emotion recognition results.
In the last part of the talk, Dr. Dhamyal will briefly discuss her work on Siri at Apple and explore the future of speech processing in the age of foundational models. She will also reflect on her journey as a Ph.D. student at Carnegie Mellon and research internships at Microsoft and Meta, offering valuable advice and perspectives for students interested in pursuing careers in research and industry.
Dr. Dhamyal is a Machine Learning Engineer at Apple, in the Siri team. She received her Ph.D. in Language Technologies from Carnegie Mellon University's School of Computer Science in 2024, where she was a member of the Machine Learning for Signal Processing group and the Center for Voice Intelligence and Security.
Her research has focused on developing computational methods for decoding psychological traits from the human voice, utilizing contrastive audio-language models with acoustic prompts that achieve state-of-the-art zero-shot emotion recognition results. Her research has been applied in real-world settings, including voice forensics technology demonstrated live at the World Economic Forum.
She has publications in top conferences including ICASSP, Interspeech, EMNLP, and ACL, and has completed research internships at Microsoft and Meta.
