The rise of digital assistants like Siri and Alexa is owed in no small part to Larry Heck. His three decades in industry – at Google, Microsoft and other digital powerhouses – brought numerous technologies that make up the foundation of today’s conversational AI.
In 2021, Heck was recruited to his alma mater, Georgia Tech, as a GRA Eminent Scholar. Since then, he’s continued to push the boundaries of AI-driven, digital assistant technology, not only through his research but also as a mentor to students.
As Heck marks his fifth year as an Eminent Scholar, he provided a high-level view of what he’s working on in his lab – and how it might impact everyday life in the years to come.
You’ve been working to improve the quality of interaction between people and digital assistants. What does that look like?
The goal I set was to build this digital avatar that you can talk to and have full, rich conversations. The avatar is able to interpret not just your words, but your voice tone, prosody, facial expressions, hand gestures — all of it. One thing I learned is how hard this problem is to solve. For one, there are not very good data sets out there to work with. So, my team is initiating a process to collect, create and annotate the data we need for face-to-face conversations that we can build on.
What kind of data?
It’s multi-modal, so not just text, but video and audio that shows conversational moves and actions that people take. We’ve learned that we can actually identify these conversational moves in the form of subspaces in large language models – and we can control them, too. If a digital avatar were equipped with the ability to both interpret and generate these moves and actions, that would help create a meeting of the minds, so to speak, when conversing with people.
What else is required to achieve this ‘meeting of the minds?’
Well, one is getting digital assistants to better recognize emotion. It’s prohibitive to do this with just training data – there’s just way too many combinations. So our first question has been: Is there a subspace within a large language model where this emotional component of language lives? We probed the layers of the transformer to find it. It turns out, there’s not one place, like a single neuron for “angry.” It’s a series of interconnected matrices inside the model, many neurons in many places.
Another is addressing latency. When you ask a digital assistant a question, it has to go all the way up and connect to the data center run by Open AI, Anthropic or another frontier model – then come all the way back down. That kind of a delay really slows down a conversation.
So, what’s an alternative?
One answer is more computing at the edge of the device. Open weight models, which allow the user to run and control a limited version of an LLM, are getting better and better. They’re starting to converge with the frontier lab models on tasks like coding and agentic activities. This “edge AI” can save companies a lot of money as well as improve privacy.
The question is, how do you get more tokens, or words, per second out of these open weight models to speed the digital assistant’s comprehension and response? We’ve been exploring something called speculative decoding, which guesses several words ahead and verifies these guesses. This is much different than the current model of one token at a time.
Describe what you see as the kind of end game here for these technologies.
I think the next three to five years will be spent building this foundation of what I call a conversational operating system. But what do you use it for? We’re looking closely at two options, tutoring and assisting with AI research.
So let’s take the AI research assistant. Imagine a researcher who has an Iron Man suit and is then able to solve AI research problems faster and faster. The suit has to fit tightly, just as the interface between Tony Stark and Iron Man is seamless. If that interface has latency or works a little clumsily, he falls out of the sky.
Not a lot of people are working on that interface. Because to tighten it is to solve the thorny, messy interactive issues with humans. But we’re talking about augmented human intelligence, AHI, and that’s what I’m trying to make happen.
What else have you been working on since coming to Georgia Tech?
We're doing a lot of interesting work on machine unlearning. It costs hundreds of millions of dollars to build a foundational large language model. Once you build it, if there's information inside of it that somebody down the road wants to remove, it's untenable to have to go back and try to retrain the entire model.
So the question is, what's an efficient method for removing that knowledge from the model without disturbing the rest of the knowledge that's there? We’ve made a lot of progress on answering that question and have published papers and received a patent.
Being able to remove information faster, and at less cost, brings many benefits. Fewer responses that are harmful to users, or biased in their information, for example. And this unlearning also contributes to LLMs meeting privacy and security requirements, such as in the General Data Protection Regulation in the EU.
