(Editor’s note: This post on AI is part of our Tech Tuesday series. Dispatches covers tech because so many of our highly skilled internationals are scientists, researchers and entrepreneurs.)
Some skills remain with us even though we cannot remember how we learned them. You pick up a broom to sweep your floor. You don’t pause to remember what sweeping is or how to do it. Your hands naturally grasp the handle, your body assumes the right posture, and you begin the familiar back-and-forth motion. This simple act, something we rarely give a second thought, holds a key to understanding human cognition – and remarkably to potentially revolutionizing artificial intelligence.
The physical artifacts in our environment, from family photos to well-worn household tools, act as powerful memory cues, reducing the cognitive load of recall. They become anchors for our experiences, helping us navigate the complex terrain of our personal histories. Each salient object or event has the potential to add to the home’s “memory,” changing how we interact with and remember the space. Our memories are not static records but dynamic interactions between our minds and our surroundings.
The power of memories
Game designers have long understood the power of objects to assist our memories. They call these objects “landmarks.” Good landmarks can make specific areas of a game world memorable. Players may recall experiences based on landmarks (e.g., “the forest with the ancient tree” or “the city with the grand castle.”) This principle applies equally to our real-world environments, where distinctive objects help anchor our memories and experiences. With artificial intelligence, we are still learning this.
Recent years have seen significant developments in AI, particularly with the rise of Large Language Models (LLMs) such as the GPT-4 and Llama families of algorithms. These models can be seen as incredibly well-read quiz show contestants who can quickly retrieve and combine information to answer questions or complete tasks.
The best of these AI models are built on what’s called a transformer architecture, where each piece of information (or “token”) in a sentence is a word, or part of a word. The transformer can look at all the tokens at once, predicting how they relate to each other, rather than reading them one by one. This allows it to grasp context and nuance much more effectively than older methods, like LSTMs, which relied on memory cells.
Anyone who has worked with these models will however be familiar with the problem that their lack of memory can lead to problems. As the context window of our conversations with them grow, they become confused and lose salient facts. In extreme cases, they jump from one piece of a discussion to another. They lose their coherence. Fixing this by re-incorporating memories is a key objective of newer AI models, such as the Mamba algorithms.
While all these models are impressive, they still fundamentally work by processing vast amounts of data to create internal representations of knowledge. It’s as if they’re trying to memorize entire libraries, rather than learning how to use reason through facts effectively. As you think about this, you might wonder how different this is from how you acquire and use knowledge in your daily life.
Broom sticks for AI
What if we took a cue from human cognition and tried a different approach with AI? What if it could learn to cognitively offload some of the memory work to its environment?
Imagine AI entities existing not as disembodied algorithms processing abstract data, but as agents interacting with digital environments that mirror our physical world. These digital spaces, better described as digital twins, could be practical representations of real-world locations imbued with instructional meaning and filled with landmarks. The key is that the AI would form “memories” through re-making its interactions with its digital surroundings, much as we do in the physical world.
This approach could lead to AI systems that are more adaptable and contextually aware. Rather than relying solely on vast databases or complex internal representations, these embodied AI agents would learn and remember through experience, developing a form of “muscle memory” for tasks within their digital domains. As they move from task to task, different memories are recalled.
By using an updated environment to guide and constrain the potential choices of an algorithm, we can influence its behaviour and understanding without necessarily changing its inner workings. This method leverages the rich, contextual information provided by the environment – whether physical or digital – to shape the AI’s responses and learning.
This perspective opens up new possibilities for creating AI systems that are more adaptable, contextually aware, and aligned with human cognition. It could help address some of the current limitations in AI, particularly in areas requiring common-sense reasoning or adaptability to new situations. It would also make working with AI much safer to embody it in a dependable digital medium, complete with guardrails and interpretable observation of its activities.
Rethinking AI: Learning from human cognition
This approach fits very well with the upcoming and popular use of several LLMs in multi-agent environments. Examples of multi-agent frameworks include CrewAI and AutoGen. Combining AI agents with digital twins takes an important cue from human cognition. Instead of trying to cram all possible knowledge into an AI’s “brain,” we teach it to interact with its environment to access and create memories, just like we do.
In practice, instead of trying to teach an AI everything by having it memorize countless tasks, we could place it in a digital kitchen where it learns by doing. It would form “memories” through its interactions, moving digital objects to well-known locations.
Importantly, this approach also recognizes how we actively remake our environment to help us remember. We arrange our kitchens for efficiency, we add birthdays to calendars, and we place clothing in familiar drawers. This behavior of modifying our surroundings to support memory and cognition is deeply ingrained in human nature – perhaps this is why our ancestors created cave art, showing a collective hunt or what constituted a family unit.
This approach could lead to AI systems that are more adaptable and contextually aware. Just as we don’t need to consciously recall how to use a broom, these AIs wouldn’t need to search through vast databases for every action. Instead, they would develop a kind of salient memory for tasks within their digital domains. Your insights into how you develop such intuitions in your own life could prove valuable in refining this approach to AI.
Implications and Applications
The implications of this approach extend far beyond AI development, creating a fascinating interplay between artificial and human intelligence.
As we develop AI systems that learn from digital representations of environments, we simultaneously gain invaluable insights into how humans interact with their surroundings. This reciprocal relationship offers a unique opportunity to enhance both AI capabilities and human well-being.
By observing how AI interacts with, learns from, and modifies digital twins of real-world environments, we can glean new understanding about the ways humans embed memories in their surroundings. These insights can then be applied to improve our physical environments, making them more conducive to memory formation, recall, and overall cognitive function.
For instance, in healthcare, this approach suggests new ways of supporting memory in patients with dementia. Instead of relying solely on internal memory aids, we can develop digital assistants and emphasizing the importance of familiar environments and objects. The AI’s interactions with a digital twin of a living space could inform the design of real-world spaces that better support recall and cognition based on recent and distant memories.
In education, learning environments that act as dynamic memory scaffolds could evolve with students’ growing knowledge. Imagine a virtual universe that adapts its layout and tools based on a student’s learning progress while they move through subjects like history and geography, being taught by the student themselves to store their insights from regular lessons.
This bidirectional flow of insights from data science to real-world challenges could extend to urban planning, and even personal healthcare. As we learn how to create digital environments that enhance AI learning and memory, we simultaneously discover principles for designing physical spaces that augment human cognition and well-being.
Conclusion
As we continue to explore the nature of memory and cognition, the view of memories as local, personal and embodied in our environment offers a compelling alternative to traditional approaches in AI.
This perspective challenges us to rethink not just how we develop AI, but how we understand intelligence itself.
The journey from a simple act of sweeping a floor to reimagining the future of AI underscores a fundamental truth: intelligence, whether human or artificial, doesn’t exist in isolation. It’s shaped by and deeply intertwined with the environment in which it operates. By recognizing this, we open up new avenues for AI development that could lead to systems that are more adaptable, intuitive, and aligned with human cognition.
The principle insight from this article should be that, while mathematics is important to the next generation of AI, our real-world insights and needs are equally important. Any one of us can become the next brilliant data scientist, or at least the key to unlocking a world where AI is able to tackle safely many of the social and environmental problems we are faced with. Your experiences could have far-reaching implications in fields like healthcare and education as we develop better approaches to handling knowledge.
How do you see the interplay between memory, environment, and cognition in your daily life? In what ways does your environment support your memory and cognition? Your real-world examples and insights could be the key to unlocking new approaches in AI, healthcare, education, and beyond. Looking forward to your contributions. They could be the spark that ignites the next breakthrough in cognitive science or AI development.
Please email your thoughts and experiences at: terry@dispatcheseurope.com.
––––––––––
See more from Shane here in Dispatches’ archives.
Shane Ó Seasnáin
Shane Ó Seasnáin is an AI expert, originally from Ireland, who works within various organizations in the Netherlands and across Europe. He has three simple steps for successful AI use: be curious about the world, ask questions about whether this is the best we can do, and determine how AI can deliver a better world. He believes that there are many ways we can radically transform our environment, healthcare, and industry using AI, but it is always with the goal of making people more capable and happy.
