Larry Heck, Ph.D.
AI and speech and language processing
Georgia Institute of Technology
Recruited: 2021
For nearly 30 years, Larry Heck pioneered technology and processes that gave rise to AI-driven speech and language technologies. He did this in industry, working for Microsoft, Google, Samsung and other global brands. In 2021, he brought his expertise to his alma mater, Georgia Tech, opening a new chapter of conducting research and preparing students to take the mantle of AI and machine learning innovation.
A primary focus of Heck’s research is optimizing the interactions between humans and AI-driven assistants. Heck envisions a day in the not-distant future when a conversation with a digital avatar is as natural as that with another person. Such interaction, he says, would make digital assistants far more productive and valuable.
To arrive at this day, Heck is working to evolve large language models (LLMs) to interpret nuance — tone of voice, gestures, facial expressions — and communicate likewise. This involves deepening understanding of what LLMs can and can’t do (and why); combining language with audio, video and imagery to broaden the skill sets of digital assistants; and training assistants to reason and better understand context over longer conversations.
One area of Heck’s focus has been the capacity for AI assistants to both understand and generate emotion. He co-authored a pair of academic papers in 2026 that helped peel back layers in LLMs that are relevant to emotion. In one study, he learned that the emotional tone of a text passage significantly affects how an LLM pays attention to different parts of the text and adjusts its response accordingly. The team also experimented with a way to train the LLM to better manage emotional interpretation.
A second paper, “Emotions Where Art Thou?,” explored the internal steps LLMs take to understand sentences. Heck discovered that LLMs have an organized, universal emotional “subspace” across layers of the model, rather than a single locus. He also developed a module to control how the LLM perceives text – as happier or sadder, for instance – without changing its meaning. The project offered a new understanding of how LLMs internalize and process emotions.
Another area of exploration involves helping LLMs “unlearn” knowledge they’ve acquired in the past. Heck created a kind of road map to make it easier to remove knowledge from an LLM without having to go through the costly and time-consuming process of retraining it. Though still in early development, this unlearning process can help reduce harmful or biased responses; meet legal requirements, such as GDPR; and make LLMs more efficient by allowing them to shed unnecessary information. Through his work on the technology, Heck filed a U.S. patent.
These efforts at Georgia Tech build on an illustrious career in industry: Heck played a significant role in developing many of the virtual assistants on the market. He founded the Cortana™ virtual assistant at Microsoft, led Samsung’s virtual assistant Bixby™ in North America, powered Viv Labs technology (founded by the inventors of Siri), served as a technical advisor to Yap Inc. (acquired by Amazon to initiate the Alexa™ virtual assistant) and founded Google’s Deep Dialogue group, a research effort behind the Google Assistant™.
While pursuing his Ph.D. at Georgia Tech, Heck built his first speech recognizer in 1987. His dissertation explored how to leverage speech algorithms for other signals such as machinery sounds and vibrations. For example, humans can listen to a metal-cutting tool and know it is about to wear out — could a machine also do this?
The idea of building a voice-enabled virtual office assistant was seeded in discussions between Heck and Adam Cheyer at the Stanford Research Institute (SRI), which Heck joined in 1991. Cheyer, who later founded Siri (Apple’s assistant) and Viv Labs, had developed the core platform that could alert a user about an email arriving from a specific sender. Heck extended this work to voicemail by creating a speaker verification/voice recognition technology.
Over the years, Heck stayed in touch with Georgia Tech, serving on various advisory boards and receiving a Distinguished Engineering Alumni Award. Before being recruited as an Eminent Scholar, he was elected to the National Academy of Inventors and named Fellow of the Institute of Electrical and Electronics Engineers (IEEE) for his leadership in applying machine learning to spoken and text language processing.
Another area of Heck’s current research involves smaller AI engines that live on the “edge” of a digital assistant device. He and team members in his lab experiment with pruning LLMs to create SLMs, or small language models – then aggregating these SLMs to run on a wider range of devices. He’s also applied a process called speculative decoding to LLMs, which speeds the prediction of language in the models. By lessening reliance on LLMs to power all interactions with a digital assistant, Heck aims to increase privacy and decrease latency, the time it takes to interact with the assistant.
Research
- Next-generation virtual assistants
- Human-computer interaction design
- Multimodal and situated interactions over screens and mixed reality (AR/VR)
- Task-oriented conversations to open-domain chit-chat to both, explicit to implicit (commonsense) knowledge-driven conversations and higher-level inference and reasoning.
- AI engines at the edge of devices
Choosing Georgia
“Georgia Tech’s College of Engineering and College of Computing have profound strengths in academics and research. I look forward to building on GT’s success in a fast-changing field of engineering and advanced computing, as well as participating in the larger innovation and startup ecosystem in Atlanta and Georgia.”
